Web Site of Rajiv Pant

In 2010, Cloud Computing is likely to see increasing adoption. Migrating Web applications from one data center to another is a complex project. To assist you in migrating Web applications from your hosting facilities to cloud hosting solutions like Amazon EC2, Microsoft Azure or RackSpace’s Cloud offerings, I’ve published a set of checklists for migrating Web applications to the Cloud.

These are not meant to be comprehensive step-by-step, ordered project plans with task dependencies. These are checklists in the style of those used in other industries like Aviation and Surgery where complex projects need to be performed. Their goal is get the known tasks covered so that you can spend your energies on any unexpected ones. To learn more about the practice of using checklists in complex projects, I recommend the book Checklist Manifesto by Atul Gawande.

The checklists are published on the RevolutionCloud book Web site at:

www.revolutioncloud.com/2010/01/checklists-migration/

and on the Checklists Wiki Web site at:

www.listswiki.com/wiki/IT_Web_Application_Migration

If you haven’t heard the news about the earthquake tragedy in Haiti, you can learn about it at: http://www.reddit.com/r/Haiti

The Reddit team at Conde Nast is working with a charity called DirectRelief to help the earthquake victims in Haiti. DirectRelief can use your help. 100% of the donations made to them are going to Haiti response programs — no administrative overhead or fundraising charged against your gift. It’s all going to Haiti.

From: http://www.reddit.com/tb/apnsu

We’ve all been shocked and saddened by the earthquake that’s hit Haiti and based on the trending reddit links since the quake hit, it’s clear many of you are anxious to help. I got to talking with a redditor at DirectRelief, a non-profit specializing in just this kind of disaster recovery and sending 100% of donations to help Haiti – redditors needn’t worry about any money siphoned off for administrative or fundraising costs.

…

My wife and I made a donation. If you are able to, please join us in helping those in need.

Born to Run by Christopher McDougall is one of the most inspiring books I’ve read in a long time. It is a book about adventures, travels, cultures, customs, ancient wisdom, evolution, anatomy, biology, footwear, anthropology, friendships and human nature. It is also a book about long distance running.

Through the stories of his warm characters, Christopher McDougall teaches that long distance running is more about cooperation, camaraderie and caring than about competition. It reminded me of my favorite quote: Victory is winning others, not defeating others.

Rating:

I’ve been reading the book SuperFreakonomics by Steven Levitt and Stephen Dubner. Like its predecessor, Freakonomics by the same authors, SuperFreakonomics is unputdownable once you start reading it.

What’s great about this book is that it argues against many things we believe to be true. Using statistics and challenging established wisdom, Levitt and Dubner make compelling arguments that give us fresh perspectives on knowledge we take for granted.

The authors cover a range of seemingly unrelated but interesting topics ranging from the economics of prostitution to the identifying potential terrorists using statistical data. The work of their colleague Sudhir Venkatesh, author of the gripping book Gang Leader for a Day is also featured in this book. They discuss the effectiveness of complex solutions like child safety seats (“car seats”) compared to simple solutions like seat belts. In their discussion of simple solutions to seemingly insurmountable challenges like preventing and controlling hurricanes, they remind us that simple solutions are sometimes the most powerful and reliable.

There are numerous things in this book that many people will not agree with. This book does not claim to be an encyclopedia of correct, proven information. If you are looking for established science and facts, compared to the hypothesis and opinions presented in this book, even WikiPedia would be a more authoritative source of facts.  However, what I liked about this book is that reading it inspires and challenges the reader to think in new ways. In fact, the same methods this book uses to challenge other wisdom could be used to dispute the viewpoints of the people whose opinions are expressed in this book.

Whether you agree with all, some or none of their findings, reading their book will benefit you: It presents and encourages ways of finding different and better ways of conducting experiments and studies. It sparks new ways of thinking, even if you use the methods to challenge some of their own findings.

The findings and statistical data are presented in their approachable storytelling that everyone can enjoy. It is an entertaining, enlightening and even educational read. I recommend reading this book.

Rating:

I thank HarperCollins for sending me a pre-release copy of the upcoming book for review. Below is a link to the book on Amazon. It will be released on October 20, 2009.

SuperFreakonomics: Global Cooling, Patriotic Prostitutes, and Why Suicide Bombers Should Buy Life Insurance

We spend most of our waking hours outside home wearing shoes. So it is important that the shoes be comfortable and promote good health of our feet and legs.

I’ve been regularly wearing a few different models of Vivo Barefoot shoes by Terra Plana for a few months now. They have become my footwear of choice.

I find them extremely comfortable and I feel that my feet and legs have become healthier as a result of wearing them. Compared to Vibram Five Fingers shoes, the Vivo Barefoot shoes have the look of regular shoes and can be worn in most social and formal occasions. The Vivo Barefoot shoes come in dozens of fashionable designs. My wife suggested the models I bought.

The Terra Plana Web page lists some benefits of barefoot shoes that I agree with:

1. Strengthens the muscles in your feet.
2. Realigns your natural posture.
3. Feeling the ground, stimulates sensory perception.
4. Flexes your feet as nature designed.

The casual shoes I wear on evenings and weekends are Vivo Barefoot (Oak Black Oil Suede and Oak Dk Brown Oil Suede) and so are the dress “formal” shoes I wear to work with a suit (Dharma Black Veg Tanned Leather). While Terra Plana does not currently make Vivo Barefoot formal dress shoes, I find that their black and brown leather shoes are fine to wear with a suit and tie to work. The comfort and health benefits provided by these shoes make up for the fact that they are not fully formal looking dress shoes. I find that they provide better grip on the ground than dress shoes and are thus great for my walking commute to work in Midtown Manhattan.

For outdoor sports like hiking and running, I wear Vibram Five Fingers shoes, but for social occasions and for work I wear Vivo Barefoot shoes. I find that the Vivo Barefoot shoes and the Vibram Five Fingers shoes complement each other well. Walking in Vivo Barefoot shoes most of the time keeps my feet in healthy and fit condition for when I hike, run, or otherwise exercise in Vibram Five Fingers.

For further reading, I suggest a recent article in the New York Times that discusses Vivo Barefoot and Vibram Five Fingers.

One drawback is that these shoes are expensive (about $150 for a pair) but I was fortunate to be able to get mine much cheaper. I had gone to the Manhattan store wearing my Vibram Five Fingers shoes. That led to a conversation about barefoot shoes with the store salespeople who were friendly and they realized I was seriously interested in such shoes. I politely and respectfully requested the store sales agent for a discount. She seemed agreeable. I then offered to buy two pairs if she’d give me a good price. She kindly agreed and suggested that if I buy the previous models which had been replaced by the new models, she could give me an even better discount. I agreed. By this time the salespeople and I had a good rapport. Before she started the paperwork for the sale, I further asked I bought three pairs if she could give me an even better deal. She agreed. So I purchased three pairs and got them pair much cheaper than the list price. In work and in personal life, negotiations should result in win/win for both sides and this was another negotiation that left both parties happy. (Note: I haven’t posted my purchase price here since that wouldn’t be fair. After negotiating a good deal it is not a good idea to publicize the terms. Market conditions and other circumstances vary and I don’t want to mislead anyone about what price they could get. )

I highly recommend these shoes.

Rating:

If you manage technology for a company that has a large Web presence, it is likely that a large percentage of your total technology costs is spent on the Web hosting environment, including the Content Delivery Network (CDN, e.g. Akamai). In this article, we discuss some ways to manage these costs.

Before we discuss how to optimize your architecture and applications to have economical and the optimally low hosting expenses, let us develop a model for comprehensively understanding a site’s Web hosting costs.

Step 1. Develop a model for allocating technology operations & infrastructure costs to each Web site/brand

Let us assume for this example that your company operates some medium to large Web sites and spends $100K/month on fully managed¹ origin² Web hosting and another $50K/month on CDN. That means your company spends $1.8MM/year on Web sites hosting.

It is important to add origin Web hosting and CDN costs to know your true Web hosting costs, especially if you operate multiple Web brands and need to allocate Web hosting costs back to each. For example, let us assume you have two Web sites: brandA.com, a dynamic ecommerce site costing $10K/month on origin hosting plus $2K/month on CDN; and brandB.com, serving a lot of videos and photos costing $5K/month on origin hosting plus $19K/month on CDN. In this example, brandA.com actually costs $12K/month, which is half the hosting cost of brandB.com, $24K/month. Without adding the CDN costs, you may mistakenly assume the opposite that brandA.com costs twice as much to host as brandB.com. Origin hosting and CDN are two sides of the same coin. I recommend that you manage them both together from both technology/architecture and budget perspectives.

Then you add the costs of third-party vendor provided parts of the site rented in the software-as-service model. Next, add licensed software costs used at your hosting location. Let us assume that brandA.com also has:

• some blogs hosted at wordpress.com for $400/month
• Google Analytics for $0/month
• Other licensed platform/application software running on your servers billed separately from the managed hosting. Let us assume brandA.com’s share of that is $1,000/month.

So your Web hosting and infrastructure costs for brandA.com would be $13,400/month. That’s $160,800/year.

Assuming that many of your Web sites share infrastructure and systems management & support staff at your Web hosting provider, you may not have a precise allocation of costs to each brand. That’s ok: It doesn’t need to be perfect nor a staff-time consuming calculation every month. Work with your hosting provider and implement a formula/algorithm that provides a reasonably good breakup and needs to be changed only when there is a major infrastructure change.

Side Note: In order to stay competitive, adapt to changes in the market and meet changing customer sites, brandA.com also needs to do product and software development on a regular basis. However, that’s beyond the scope of this discussion. Managing ongoing product and software development costs for brandA.com could be the subject of another article.

Step 2. Regularly review the tech operations costs for each brand and make changes to control costs

Every month, review your tech operations costs for your business as a whole and for each brand. Make changes in technology and process as needed to manage your expenses. If you don’t review the expenses on a monthly basis, you run the risk of small increases happening in various places every month that add up to a lot.

Without active management done on a monthly basis, brandA.com could creep up from $13,400 to $16,000 the next month and $20,000 the month after. That $1.8MM you were expecting to spend on hosting for the year could turn out to be $2.4MM.

So what does such active management include?

Monitor and manage your bandwidth charges. This is one to keen an eye on. If you bandwidth charges go over your fixed commit, your expenses can quickly blow over budget. If you find bandwidth use increasing, investigate the cause and make course corrections. In some cases, this may simply be due to expected increase in traffic, but in other cases it could be avoided. The related article on this site about taking advantage of browser caching to lower costs provides some tips.

Request your engineers to monitor and manage your servers resource usage (CPU, memory) so that the need for adding hardware can be avoided as much as possible. Enable and ensure regular communications between your technology operations team and your software development team so that software developers are alerted of any application behavior that is consuming more than expected server resources. Give the software developers time to resolve such issues when found.

Review the invoice details to make sure you understand and are in agreement with the invoice. A Web hosting bill can be very detailed and complex to understand. Do not hesitate to ask the hosting provider to explain and justify anything that you don’t understand. Don’t just assume the bills are always correct. They could (and occasionally will) be mistakes in the bills. Be sure to dispute these with the vendor in a respectful and friendly way.

These are just some examples. Please feel welcome to make more suggestions via comments on this post.

The time (and thus money invested) in controlling tech operations cost will be well worth the savings / avoidance of huge cost increases.

Keep abreast of evolving technologies and cost saving methods. Periodically review these with your vendor(s).

Cloud computing is exciting as a technology, and it is equally exciting as a pricing model.

If you find market conditions have changed drastically, request your vendor to consider lowering rates/prices even if you are locked into a contract. You don’t lose anything by asking and the vendor’s response will be an indicator of their customer service and long term business interest with you.

1. Fully managed Web hosting includes network & hardware infrastructure, 24×7 staff and real estate [↩]
2. The origin part of your Web hosting environments includes the network and server infrastructure at your hosting facility location(s) where your Web applications and installed and running. It could be in-house data centers or at providers such as RackSpace, IPSoft or Savvis [↩]

Companies that operate heavily trafficked Web sites can save thousands of dollars every month by maximizing their use of browser-side caching.

Large Web sites pay for bandwidth at their Web hosting data center and also at their content delivery network (CDN, e.g. Akamai, LimeLight, CDNetworks). Bandwidth costs add up to huge monthly bills. On small-business or personal Web sites where bandwidth costs don’t go over, this is not an issue, but on large Web sites, this is important to address and monitor.

Companies operating large Web sites often have complex situations like the following:

• An comprehensive and deep understanding of all technology cost drivers and their impacts on each other. For example, a programmer may think they are saving the company money by architecting an application in a way that it requires minimal hardware servers, but not realize that the same design actually results in even higher costs elsewhere like CDN bills.
• Busy development teams working on multiple projects on tight timelines. This results in compromises between product features/timelines and technical/architectural best practices/standards.
• Web content management and presentation platform(s) that have evolved over the years
• Staff churn over the years and an uneven distribution of technical knowledge and best practices about the Web site(s)
• The continued following of some obsolete “best practices” and standards that were established long ago when they were beneficial, but are now detrimental.

Tech teams at complex Web sites would likely find upon investigation that their Web sites suffer from problems that they either didn’t know about or didn’t know the extent of the damage they are causing.

One such problem is that certain static objects on the company’s Web pages that should be cached by the end users’ Web browsers are either not cached by the browsers at all or not cached enough. Some objects are at least cached by the CDN used by the company, but some perfectly cacheable objects are served all the way back form the origin servers for every request! An unnecessarily costly situation that can be avoided.

In addition to wasteful bandwidth charges resulting in high monthly bills, there are also other disadvantages caused by cacheable objects being unnecessarily served from origin servers:

• They slow down your Web pages. Instead of the browser being able to use local copies of these objects, it has to fetch them all the way from your origin servers.
• Unnecessary load on origin Web servers and network equipment at Web hosting facility. This can be an especially severe problem when a Web site experiences a sudden many-fold increase in traffic caused by a prominent incoming link on the home page of a high traffic like Yahoo, MSN or Google.
• Additional storage in logs at the origin Web hosting locations’ servers and other devices.
• Unneeded processing and work the origin servers, network equipment, CDN, the Internet in the middle all the way up to the client browsers have to do to transfer these objects from origin to the end user’s browser. Be environmentally friendly and avoid all this is costly waste.

The increase in bandwidth, load on servers and networking equipment and log file storage space increases caused by a few objects on Web pages being served by origin servers for every request may mistakenly seem like an insignificant problem, but little drops of water make the mighty oceans. Some calculations will show that for large Web sites, the cost of this can add up to tens of thousands of dollars a month in bandwidth costs alone.

How should companies operating large Web sites solve this problem?

For technology managers:

• Make it a best practice to maximize the use of browser-side caching on your Web pages. Discuss this topic with the entire Web technology team. Awareness among the information workers is important so that they can keep this in mind for future work and also address what’s already in place. Show the engineers some sample calculations to illustrate how much money is wasted in avoidable bandwidth costs: that will prove this is not an insignificant issue.
• If this problem is widespread in your Web site(s), make the initial cleanup a formal project. Analyze how much money you’d save and other problems you’d solve by fixing this and present it to the finance and business management. Once you show the cost savings, especially in this economy, this project will not be hard to justify.

For engineers:

• Read the article about optimizing caching at Google Code for technical details on how to leverage browser and proxy caching. It explains the use of HTTP headers like Cache-Control, Expires, Last-Modified, and Etag.
• Review any objects that are served by origin servers every time for legacy reasons that may now be obsolete.
• Combine some JavaScript files commonly used by your Web pages so that the one unified and shared file would have higher caching probability. Do the same with external CSS style sheets.
• Study a good book on Web site optimization like Even Faster Web Sites: Performance Best Practices for Web Developers. Share these recommendations and hold a discussion with your tech and production colleagues.

Sometimes simplicity and minimalism provide wonderful experiences. The Vibram Five Fingers shoes are excellent for outdoor activities and for working out at the gym. The natural feeling that comes with being outdoors with these shoes enables the wearer to feel one with nature. Walking while wearing these shoes is a sensory experience that allows you to notice and feel the Earth.

Time Magazine named Vibram Five Fingers shoes among the best inventions of 2007 along with other products like the Apple iPhone.

Besides Vibram, some other companies also market shoes that provide a barefoot-like experience. Nike sells the Nike Free line of shoes, though the Vibram Five Fingers experience is much closer to being barefoot. A company called Terra Plana makes their Vivo Barefoot shoes. These shoes do provide a good level of protection and safety compared to walking truly barefoot. Personally, I would not be barefoot while hiking in the mountains or walking the streets of New York City, so these shoes provide me with an excellent balance of safety, comfort, good exercise and the thrill of a natural feeling.

According to Vibram, Nike and Terra Plana’s separate marketing materials and others’ independent research, the barefoot-like experience provided by such shoes is good for the health of your feet and legs on the long term, since they allow your muscles and bones to get proper exercise that other shoes inhibit.

Since these shoes do not offer the cushioning via a thick sole of protection regular shoes do, you do need to be cautious while wearing them, especially while you are getting used to them. If you go hiking with these, consider taking along a pair of regular shoes as backup. I plan to go hiking wearing these and they will likely become my hiking shoes of choice, a position currently held by my Nike Free 3.0 shoes. I’ve hiked in the Georgia mountains wearing Nike Free 5.0 shoes and on a part of the Appalachian trail in New Jersey wearing the even thinner soled Nike Free 3.0 shoes. Those hikes included treading on sharp rocks and river crossings. Walking through water over the riverbed wearing barefoot-like shoes is such a soothing experience. I kept a pair of regular hiking shoes in my backpack as backup, but was able to complete the hikes without needing to resort to the backup shoes. Hiking wearing the Nike Free shoes that felt close to being barefoot was a wonderful zen like experience.

One drawback: Since the Vibram Five Fingers shoes look too unconventional, they are not suitable for wearing to many social events. A suggestion for Vibram: To some models, add a thin cloth or rubber film/layer on top that makes them look more like regular shoes from above while preserving the independent fingers movement on the bottom sole. Such a thin layer of film would be for looks only. Beneath it, the toes would still be independent. With that, people would be able to wear these shoes in many social situations.

I love Vibram Five Fingers shoes and highly recommend them. I wish that Vibram decides to make other versions of these shoes that will provide all the benefits these shoes do, but also look more like conventional shoes from top. If Vibram does so, they would become the shoes of my choice for all casual wear. It would be even more awesome if Vibram one day designs Five Fingers shoes that look like formal dress shoes from aboce. If they do that, various models of Five Fingers shoes would become my favorite shoes for almost all walks of life.

Rating:

For an executive, having a management team of people who are good at their jobs and work well with each other is one of the most important factors that lead to success together. Observing a number of successful projects, I realized that it is critical that your management team members care for each other, work well together and give to each other. Their sincere collaboration is far more important than their individual strengths.

I began to write this article impressed by how well the management team comprising of my direct reports functioned, collaborating with each other towards shared success. I was pleasantly surprised by how these directors shared responsibilities, how closely they worked with people in each other’s teams and how comfortably they gave credit to each other. When conflicts arose between them, they frankly, respectfully and nicely expressed them to each other, often one-on-one. Every time, they resolved them quickly and came out with a closer professional relationship. They actively and regularly talked to quell any turf battles between each other’s departments before they could form.

They had a wonderful professional relationship. They barely knew each other outside of work, having busy personal lives with their families on most evenings and weekends. I felt that my management team and I were like a work-family, sticking together through good and bad times, always believing that our success comes as a team.

When you manage and organize your company or your department, spend time multiple times a week with your direct reports together so that you all work well with each other towards shared success. In turn, they should ensure that their direct reports care about each other and collaborate. If you have, say five direct reports, make sure that just the six of you get together in a room to work openly and collaboratively at least twice a week (assuming you are in the same geographic location). The forum for this need not be always a staff meeting, it could be a working session on a project.

I was struggling to come up with suitable words to describe this and its importance. While reading the book The Four Obsessions of an Extraordinary Executive: A Leadership Fable by Patrick Lencioni, I found that the first discipline described in the story talks exactly of this and hence is the title of this article. The book is written as a fictional story that teaches leadership lessons. It is easy to read being under 200 pages in large typeface which you can read in one evening. I highly recommend it.

One of the differences between Amazon EC2 server instances and normal servers is that the server’s local disk storage state (i.e. changes to data) on EC2 instances does not persist over instance shutdowns and powering on. This was mentioned in my earlier post about hosting my Web site on Amazon EC2 and S3,

Therefore, it is a good idea to store your home directory, Web document root and databases on an Amazon EBS volume, where the data does persist like in a normal networked hard drive. Another benefit of using an Amazon EBS volume as a data disk is that it separates your operating system image from your data. This way, when you upgrade from a server instance with less computing power to one with more computing power, you can reattach your data drive to it for use there.

You can create an EBS volume and attach it to your EC2 server instance using a procedure similar to the following.

First, create an EBS volume.

You can use Elasticfox Firefox Extension for Amazon EC2 to:

• create a EBS volume
• attach it to your EC2 instance
• alias it to a device, In this example, we use /dev/sdh

Then attach the “disk” to your EC2 instance and move your folders to it using a procedure similar to the following commands issued from a bash shell.

# Initialize (format) the EBS drive to prepare it for use
# Note: replace /dev/sdh below with the device you used for this EBS drive
mkfs.ext3 /dev/sdh
#
# Create the mount point where the EBS drive will be mounted
sudo mkdir /mnt/rj-09031301
# Side note: I use a naming convention of rj-YYMMDDNN to assign unique names
# to my disk drives, where YYMMDD is the date the drive was put into service
# and NN is the serial number of the disk created that day.
#
# Mount the EBS drive
sudo mount -t ext3 /dev/sdh /mnt/rj-09031301
#
# Temporarily stop the Apache Web server
sudo /etc/init.d/apache2 stop
#
# Move the current /home folder to a temporary backup
# This temporary backup folder can be deleted later
sudo mv /home /home.backup
#
# Symbolic link the home folder on the EBS disk as the /home folder
sudo ln -s /mnt/rj-09031301/home /home
#
# Start the Apache Web server
sudo /etc/init.d/apache2 start


Limitations:

One current limitation of EBS volumes is that a particular EBS disk can only be attached to one server instance at a given time. Hopefully, in a near future version upgrade of EC2 and EBS, Amazon will enable an EBS volume to be attached to multiple concurrent server instances. That will enable EBS to be used similar to how SAN or NAS storage is used in a traditional (pre cloud computing era) server environment. That will enable scaling Web (and other) applications without having to copy and synchronize data across multiple EBS instances. Until Amazon adds that feature, you will need to maintain one EBS disk per server and keep their data in sync. One method of making the initial clones is to use the feature that creates a snapshot of an EBS volume onto S3.

Related article on Amazon’s site:

• Running MySQL on Amazon EC2 with Elastic Block Store

  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663