Web Site of Rajiv Pant

In 2010, Cloud Computing is likely to see increasing adoption. Migrating Web applications from one data center to another is a complex project. To assist you in migrating Web applications from your hosting facilities to cloud hosting solutions like Amazon EC2, Microsoft Azure or RackSpace’s Cloud offerings, I’ve published a set of checklists for migrating Web applications to the Cloud.

These are not meant to be comprehensive step-by-step, ordered project plans with task dependencies. These are checklists in the style of those used in other industries like Aviation and Surgery where complex projects need to be performed. Their goal is get the known tasks covered so that you can spend your energies on any unexpected ones. To learn more about the practice of using checklists in complex projects, I recommend the book Checklist Manifesto by Atul Gawande.

The checklists are published on the RevolutionCloud book Web site at:

www.revolutioncloud.com/2010/01/checklists-migration/

and on the Checklists Wiki Web site at:

www.listswiki.com/wiki/IT_Web_Application_Migration

If you manage technology for a company that has a large Web presence, it is likely that a large percentage of your total technology costs is spent on the Web hosting environment, including the Content Delivery Network (CDN, e.g. Akamai). In this article, we discuss some ways to manage these costs.

Before we discuss how to optimize your architecture and applications to have economical and the optimally low hosting expenses, let us develop a model for comprehensively understanding a site’s Web hosting costs.

Step 1. Develop a model for allocating technology operations & infrastructure costs to each Web site/brand

Let us assume for this example that your company operates some medium to large Web sites and spends $100K/month on fully managed¹ origin² Web hosting and another $50K/month on CDN. That means your company spends $1.8MM/year on Web sites hosting.

It is important to add origin Web hosting and CDN costs to know your true Web hosting costs, especially if you operate multiple Web brands and need to allocate Web hosting costs back to each. For example, let us assume you have two Web sites: brandA.com, a dynamic ecommerce site costing $10K/month on origin hosting plus $2K/month on CDN; and brandB.com, serving a lot of videos and photos costing $5K/month on origin hosting plus $19K/month on CDN. In this example, brandA.com actually costs $12K/month, which is half the hosting cost of brandB.com, $24K/month. Without adding the CDN costs, you may mistakenly assume the opposite that brandA.com costs twice as much to host as brandB.com. Origin hosting and CDN are two sides of the same coin. I recommend that you manage them both together from both technology/architecture and budget perspectives.

Then you add the costs of third-party vendor provided parts of the site rented in the software-as-service model. Next, add licensed software costs used at your hosting location. Let us assume that brandA.com also has:

• some blogs hosted at wordpress.com for $400/month
• Google Analytics for $0/month
• Other licensed platform/application software running on your servers billed separately from the managed hosting. Let us assume brandA.com’s share of that is $1,000/month.

So your Web hosting and infrastructure costs for brandA.com would be $13,400/month. That’s $160,800/year.

Assuming that many of your Web sites share infrastructure and systems management & support staff at your Web hosting provider, you may not have a precise allocation of costs to each brand. That’s ok: It doesn’t need to be perfect nor a staff-time consuming calculation every month. Work with your hosting provider and implement a formula/algorithm that provides a reasonably good breakup and needs to be changed only when there is a major infrastructure change.

Side Note: In order to stay competitive, adapt to changes in the market and meet changing customer sites, brandA.com also needs to do product and software development on a regular basis. However, that’s beyond the scope of this discussion. Managing ongoing product and software development costs for brandA.com could be the subject of another article.

Step 2. Regularly review the tech operations costs for each brand and make changes to control costs

Every month, review your tech operations costs for your business as a whole and for each brand. Make changes in technology and process as needed to manage your expenses. If you don’t review the expenses on a monthly basis, you run the risk of small increases happening in various places every month that add up to a lot.

Without active management done on a monthly basis, brandA.com could creep up from $13,400 to $16,000 the next month and $20,000 the month after. That $1.8MM you were expecting to spend on hosting for the year could turn out to be $2.4MM.

So what does such active management include?

Monitor and manage your bandwidth charges. This is one to keen an eye on. If you bandwidth charges go over your fixed commit, your expenses can quickly blow over budget. If you find bandwidth use increasing, investigate the cause and make course corrections. In some cases, this may simply be due to expected increase in traffic, but in other cases it could be avoided. The related article on this site about taking advantage of browser caching to lower costs provides some tips.

Request your engineers to monitor and manage your servers resource usage (CPU, memory) so that the need for adding hardware can be avoided as much as possible. Enable and ensure regular communications between your technology operations team and your software development team so that software developers are alerted of any application behavior that is consuming more than expected server resources. Give the software developers time to resolve such issues when found.

Review the invoice details to make sure you understand and are in agreement with the invoice. A Web hosting bill can be very detailed and complex to understand. Do not hesitate to ask the hosting provider to explain and justify anything that you don’t understand. Don’t just assume the bills are always correct. They could (and occasionally will) be mistakes in the bills. Be sure to dispute these with the vendor in a respectful and friendly way.

These are just some examples. Please feel welcome to make more suggestions via comments on this post.

The time (and thus money invested) in controlling tech operations cost will be well worth the savings / avoidance of huge cost increases.

Keep abreast of evolving technologies and cost saving methods. Periodically review these with your vendor(s).

Cloud computing is exciting as a technology, and it is equally exciting as a pricing model.

If you find market conditions have changed drastically, request your vendor to consider lowering rates/prices even if you are locked into a contract. You don’t lose anything by asking and the vendor’s response will be an indicator of their customer service and long term business interest with you.

1. Fully managed Web hosting includes network & hardware infrastructure, 24×7 staff and real estate [↩]
2. The origin part of your Web hosting environments includes the network and server infrastructure at your hosting facility location(s) where your Web applications and installed and running. It could be in-house data centers or at providers such as RackSpace, IPSoft or Savvis [↩]

Companies that operate heavily trafficked Web sites can save thousands of dollars every month by maximizing their use of browser-side caching.

Large Web sites pay for bandwidth at their Web hosting data center and also at their content delivery network (CDN, e.g. Akamai, LimeLight, CDNetworks). Bandwidth costs add up to huge monthly bills. On small-business or personal Web sites where bandwidth costs don’t go over, this is not an issue, but on large Web sites, this is important to address and monitor.

Companies operating large Web sites often have complex situations like the following:

• An comprehensive and deep understanding of all technology cost drivers and their impacts on each other. For example, a programmer may think they are saving the company money by architecting an application in a way that it requires minimal hardware servers, but not realize that the same design actually results in even higher costs elsewhere like CDN bills.
• Busy development teams working on multiple projects on tight timelines. This results in compromises between product features/timelines and technical/architectural best practices/standards.
• Web content management and presentation platform(s) that have evolved over the years
• Staff churn over the years and an uneven distribution of technical knowledge and best practices about the Web site(s)
• The continued following of some obsolete “best practices” and standards that were established long ago when they were beneficial, but are now detrimental.

Tech teams at complex Web sites would likely find upon investigation that their Web sites suffer from problems that they either didn’t know about or didn’t know the extent of the damage they are causing.

One such problem is that certain static objects on the company’s Web pages that should be cached by the end users’ Web browsers are either not cached by the browsers at all or not cached enough. Some objects are at least cached by the CDN used by the company, but some perfectly cacheable objects are served all the way back form the origin servers for every request! An unnecessarily costly situation that can be avoided.

In addition to wasteful bandwidth charges resulting in high monthly bills, there are also other disadvantages caused by cacheable objects being unnecessarily served from origin servers:

• They slow down your Web pages. Instead of the browser being able to use local copies of these objects, it has to fetch them all the way from your origin servers.
• Unnecessary load on origin Web servers and network equipment at Web hosting facility. This can be an especially severe problem when a Web site experiences a sudden many-fold increase in traffic caused by a prominent incoming link on the home page of a high traffic like Yahoo, MSN or Google.
• Additional storage in logs at the origin Web hosting locations’ servers and other devices.
• Unneeded processing and work the origin servers, network equipment, CDN, the Internet in the middle all the way up to the client browsers have to do to transfer these objects from origin to the end user’s browser. Be environmentally friendly and avoid all this is costly waste.

The increase in bandwidth, load on servers and networking equipment and log file storage space increases caused by a few objects on Web pages being served by origin servers for every request may mistakenly seem like an insignificant problem, but little drops of water make the mighty oceans. Some calculations will show that for large Web sites, the cost of this can add up to tens of thousands of dollars a month in bandwidth costs alone.

How should companies operating large Web sites solve this problem?

For technology managers:

• Make it a best practice to maximize the use of browser-side caching on your Web pages. Discuss this topic with the entire Web technology team. Awareness among the information workers is important so that they can keep this in mind for future work and also address what’s already in place. Show the engineers some sample calculations to illustrate how much money is wasted in avoidable bandwidth costs: that will prove this is not an insignificant issue.
• If this problem is widespread in your Web site(s), make the initial cleanup a formal project. Analyze how much money you’d save and other problems you’d solve by fixing this and present it to the finance and business management. Once you show the cost savings, especially in this economy, this project will not be hard to justify.

For engineers:

• Read the article about optimizing caching at Google Code for technical details on how to leverage browser and proxy caching. It explains the use of HTTP headers like Cache-Control, Expires, Last-Modified, and Etag.
• Review any objects that are served by origin servers every time for legacy reasons that may now be obsolete.
• Combine some JavaScript files commonly used by your Web pages so that the one unified and shared file would have higher caching probability. Do the same with external CSS style sheets.
• Study a good book on Web site optimization like Even Faster Web Sites: Performance Best Practices for Web Developers. Share these recommendations and hold a discussion with your tech and production colleagues.

For an executive, having a management team of people who are good at their jobs and work well with each other is one of the most important factors that lead to success together. Observing a number of successful projects, I realized that it is critical that your management team members care for each other, work well together and give to each other. Their sincere collaboration is far more important than their individual strengths.

I began to write this article impressed by how well the management team comprising of my direct reports functioned, collaborating with each other towards shared success. I was pleasantly surprised by how these directors shared responsibilities, how closely they worked with people in each other’s teams and how comfortably they gave credit to each other. When conflicts arose between them, they frankly, respectfully and nicely expressed them to each other, often one-on-one. Every time, they resolved them quickly and came out with a closer professional relationship. They actively and regularly talked to quell any turf battles between each other’s departments before they could form.

They had a wonderful professional relationship. They barely knew each other outside of work, having busy personal lives with their families on most evenings and weekends. I felt that my management team and I were like a work-family, sticking together through good and bad times, always believing that our success comes as a team.

When you manage and organize your company or your department, spend time multiple times a week with your direct reports together so that you all work well with each other towards shared success. In turn, they should ensure that their direct reports care about each other and collaborate. If you have, say five direct reports, make sure that just the six of you get together in a room to work openly and collaboratively at least twice a week (assuming you are in the same geographic location). The forum for this need not be always a staff meeting, it could be a working session on a project.

I was struggling to come up with suitable words to describe this and its importance. While reading the book The Four Obsessions of an Extraordinary Executive: A Leadership Fable by Patrick Lencioni, I found that the first discipline described in the story talks exactly of this and hence is the title of this article. The book is written as a fictional story that teaches leadership lessons. It is easy to read being under 200 pages in large typeface which you can read in one evening. I highly recommend it.

There are many good ways to organize your technology department. This article presents one of them. It is written for someone in a CTO, CIO, VP Technology or a consultant role looking for suggestions on organizing or reorganizing your Web technology department. It is best suited for you if your organization has the following characteristics:

• You manage software engineering, implementation and technology operations for 3 or more Web sites.
• Yours is a medium to large size organization with somewhere between 20 to 150 technology staff.
• Internal corporate IT functions such as desktop support, telecommunications services and internal business systems are beyond the scope of this article.

The Venn diagram below presents one model of organizing your department into 3 sub-departments.

Web Technology Department Organization Venn Diagram Illustrating Purposeful Overlap Among Sub-Departments

Many CTOs organize their technology departments as 2 sub-departments: Software Engineering and Technology Operations. Software engineering is the function that is responsible for developing and/or implementing Web application software. Technology Operations is responsible for running, maintaining and supporting the Web applications.

If you operate 1 or 2 Web sites, having these 2 sub-departments is a good approach. For 3 or more Web sites, organizing Software Engineering into Site Engineering and Platform Engineering has some benefits.

Site Engineering is focused on working on the Web sites’ direct projects. Its work includes

• Small and large projects for adding or changing functionality on the Web sites
• Bug fixes on the Web site applications

Platform Engineering is typically smaller than the other two organizations and typically includes functions like:

• Architecture across sites
• Shared applications across sites
• Common libraries across sites
• Research & Development (R&D)

Technology Operations includes functions such as:

• Systems & Applications Administration
• Infrastructure Management
• 24×7 Tech Support
• Builds & Configuration
• Release Management
• Testing & Quality Assurance (QA) *
• Technical Analysis
• Technical Project Management
• Budget Management

* Note: QA can also be a completely independent department or part of another department outside technology.

These three departments have purposeful overlap of responsibilities as illustrated in the Venn diagram above. That helps minimize the chances of the departments becoming silos with walls between them. For success, it is important that your entire department functions as one integrated unit. Some shared goals & responsibilities are required for mutual success.

To make this work, you need 3 directors who head up these departments who work well together, collaborate often and are not sensitive about their turf. They should know that a successful technology manager is not an individual-only contributor, but a great team player with peers. They should have strong goodwill among each other and welcome each other to work directly with their teams. Such a collaborative team is essential.

If you supervise employees or are in a senior position relative to some others, make it a habit to thank employees when they do a good job.

Here are some tips for effective appreciation of employees’ work:

Appreciation is good even when it is an expected part of their duties. If the work is good quality, thank the person.

Don’t trivialize your thanks, however. Don’t thank an employee for just sending a meeting agenda or sending a recap of a meeting unless the agenda or recap they sent is so good that it impresses you. (An exception to this rule would be if in the company most people forget to send meeting recaps and this person did and you are working to encourage this good habit.)

Don’t just send an email saying “thank you” and nothing else. Add at least a sentence or two explaining why you appreciate what the person did.

Don’t always send an email. Mix it up. Sometimes walk up to the employee and thank them in person. If they are in a remote office location, call them by phone to tell them.

When thanking someone by replying to an email they have sent, send a thank you note directly to the person. Don’t do a reply-all unless it is a major above & beyond accomplishment that deserves public congratulations. When you do send a public thank you to an employee with other people cc’d, do take the time to write at least a paragraph or a few bullet points mentioning the benefits of the accomplishment.

Mention by name the person(s) when you thank people. Don’t just thank “the team”, or “all who worked on this project” out of laziness instead of calling by name the people you are thanking, unless you have a good reason for not naming names.

To be effective at anything in life, balance is required Be it personal life, leadership, management, technical work: balance is essential. You need to be aware of balance at all times.

I plan to write a series of articles about this subject.

Some topics I discuss with friends and colleagues:

• standardization vs. innovation & creativity (technology, management)
• dedicated resources vs. shared services (management, leadership, technology)
• security & privacy vs. convenience (technology, product design)
• centralization vs. decentralization & empowerment (leadership, management)

I’d love to hear your topic suggestions and read your viewpoints and experiences. Please do so via the comments section here and in future articles.

My colleague Jon Stokes who is Senior Editor and co-founder at Ars Technica interviewed me on the topic of IT consumerization, and about the shifting boundary between professional and the personal profiles.

Jon writes:

For this third installment of my series on IT consumerization, I interviewed Rajiv Pant, vice president of technology at CondéNet (the digital arm of Condé Nast Publications). Rajiv is mainly responsible for the company’s online presence, but he’s watching many of the same trends and themes that I’ve outlined in the previous two installments play out across CondéNet.

The full article is at:
http://arstechnica.com/articles/paedia/it-consumerization-enterprise.ars

My journalist colleagues at Wired.com published some of my comments related to Amazon S3.¹ Wired also posted another article titled Customers Shrug Off S3 Service Failure. I agree with the views of many of the customers expressed in the article. Don MacAskill, CEO of the popular photo hosting site Smugmug, wrote an understanding post about it.

My entire career working for media companies, I’ve held firm the belief that the uptime, reliability, performance, scalability, performance and security of commercial Web sites is of paramount importance. When sites that I’ve been responsible for have had issues, my colleagues and I have given our personal time and energy to resolution. With my teams, I spend considerable time on proactive measures. I’ve had the honor of working closely with and learning from some who do an excellent job running technology operations.

Experience has taught that things can and sometimes do go wrong. Sometimes calculated risks don’t pan out. Sometimes mistakes cause problems. We are human. We should strive for perfection; we can get close to it, but not fully attain it. We should be prepared for such scenarios. When they happen, we should work diligently and expeditiously on resolution and have frequent and honest communications with stakeholders and customers. Such communications during the incident should include:

Update 2010-Jan-24: This checklist is now maintained on the Checklists Wiki Web site at:

www.listswiki.com/wiki/IT_Incident_Reporting

During-Incident Communication Checklist

• Current status
• What is the full impact?
• Estimated time to resolution
• Any recommended workarounds until resolution, if practical
• Assurance that it is being worked on
  • It often helps to mention who all are working on it and what they are doing

The post-incident communications to stakeholders and customers should include:

Post-Incident Communication Checklist

• Summary
• What happened, how and why it happened?
  • Including full description of all impact
  • Do not blame² third-parties or say things like “beyond our control”. A technology leader takes responsibility equally for both insourced and outsourced products and services.³
• How it was resolved
  • If the resolution is temporary or long-term
• Next steps
• Plan for eliminating or minimizing this and similar incidents from happening again
• Thank all those who helped resolve and the customers for their understanding
• Mention the monetary credits you plan to give as per the Service Level Agreement (SLA)
  • Specify any additional ‘make goods’ or returns you plan to make to the customers above and beyond the credits as per SLA, if appropriate.

Stakeholders and customers here refer to internal customers of the technology operations team (e.g. the concerned folks in editorial, marketing, sales, finance, legal and other departments). External communications to the public Internet should be handled in consultation with legal and public relations.

S3’s outage (or any outage) isn’t to be taken lightly, but I have faith Amazon and their customers will learn from it.

Disclaimers:

• As explained in the terms of use of this site, any opinions expressed on my personal Web site do not reflect those of any employer, past or present. My Web site and I in my personal life neither represent nor speak for any corporation.
• I have no affiliation, financial or otherwise with Amazon.com. I happen to be a user of their products and services, some of which I like and some that I don’t.
• Personal Web sites like this are exempt from the performance requirements of corporate Web sites :-) My personal Web site is for expressing, learning and R&D. It also happens to be hosted on Amazon EC2 and S3.

1. Silicon Alley Insider and ValleyWag have amusing spins on it. :-) [↩]
2. There may be extreme instances, especially when criminal activity or malicious wrongdoing was the cause where it would be appropriate to blame someone. [↩]
3. It is ok to mention service providers, or describing external events for explaining what happened, but don’t do it in a “it was their fault, not ours” tone. The technology leader should factually describe what happened and take responsibility. [↩]

Described here is one way to enable technologists to grow their careers in your organization while still allowing them to focus on the type of work they are best at and enjoy most.

The typical management career growth path does not suit some technical people. These information workers need to grow in their careers (gain greater compensation, responsibilities and influence) without having to become managers of other people. A good way to achieve that goal is to create a technical career growth track in your organization.

The following table shows management seniority positions alongside their equivalent technical seniority positions.

In this model, for example, an architect role is at the same compensation and influence level as a manager role, assuming that the particular manager and architect being compared add similar value to the company. If the organization requires more layers, say, a senior architect would be at the same level as a senior manager.

If the organization prefers consistent titles for levels, the system could name them like this: the fellow role as director & fellow, the senior architect role as senior manager & architect, etc. In the case of a fellow who is at a VP or SVP level, they should always be named VP & fellow or SVP & fellow.

Here is a definition of the fellow role from WikiPedia:¹

Large corporations in research and development-intensive industries ² appoint a small number of senior scientists and engineers as Fellows. Fellow is the most senior rank or title one can achieve on a technical career, though some fellows also hold business titles such as vice president or chief technology officer.

Such a technical career growth plan brings many benefits to your organization.

• It helps retain good technologists who want to grow in their careers, but want to do keep doing the type of work they are best at and enjoy doing: technical work.
• It avoids brilliant technical people from being “pushed” (by themselves or their supervisors trying to “reward” them) into people-management responsibilities.
• It reduces situations of having too many people-managers but not enough people-management positions over time as people get promoted.

Care should be taken to recognize that some technical people do enjoy making the transition to people-management roles and the presence such a technical track should not discourage them. Having an alternate career growth track option is about presenting employees with more than one choice.

A similar system could also be applied to other departments with individual contributors. For example, creative design.

Related Articles on Other Sites

• Fog Creek Professional Ladder by Joel Spolsky
• Construx’s professional development ladder

1. Definition of Fellow at WikiPedia http://en.wikipedia.org/wiki/Fellow and Wikitionary http://en.wiktionary.org/wiki/fellow [↩]
2. IBM or Sun Microsystems in information technology, and Boston Scientific in Medical Devices for example [↩]