How do you launch a suite of Mobile, Web, and Voice products from ideation to go live in 12 weeks?

Check out the blog post by Very, the product development partner I worked with in my previous job at Thrive Global to design, build and launch multiple Mobile, Web, and Voice-controlled products from ideation to go live in 12 weeks!

Thrive Global was created to end the stress epidemic. Founded by Arianna Huffington, the comprehensive platform tackles burnout — a widespread barrier to productivity and success. By delivering sustainable, science-based solutions, Thrive Global enhances wellbeing and performance, so that people can move from surviving to thriving.

9 Reasons Why News Media Web Sites Should Consider Moving to HTTPS in 2015

If you work in news media and are interested in technology, you may enjoy my article listing 9 Reasons Why News Media Web Sites Should Consider Moving to HTTPS in 2015. I co-authored it with Eitan Konigsburg and Elena Kvochko, two colleagues with expertise, deep knowledge and passion for cyber security, privacy and technology.

It is published on the Times Open Blog maintained by the Software Engineering Team at The New York Times.

My personal Web site, rajiv.com is served exclusively on HTTPS thanks to CloudFlare.

Maker’s Schedule (For Managers Too)

The following memo from a department head to staff is an example of how to implement a productive maker’s schedule at your workplace. This approach recommends starting with baby steps, evaluating results and making changes accordingly.

Comic strip from XKCD

Dear Colleagues in the Technology, Project Management and Product Teams,

Executive Summary:1

We are implementing a maker’s schedule starting this Friday May 31st which means developers will have from 12 noon onwards on Fridays to focus exclusively on writing code, with no meetings or other interruptions. (This also applies to other contributors besides developers. For more information, see details below.) The goal of this is to increase productivity, creativity and job satisfaction. This practice is based on science and employed by other successful organizations. We request your understanding, your support, and your help in making Friday afternoons meeting-free.

The Details:

We are implementing a maker’s schedule system starting Friday May 31, 2013. What is it? A maker’s schedule is calendar scheduling system that gives a group of people a continuous multi-hour block of time to focus on their work with minimal productivity diminishing things like distractions, context switching or frequent interruptions.

The word maker in this context often refers to people like software engineers, designers, testers, systems engineers, infrastructure engineers, documentation authors, editors or anyone else making something. What they are making could be software code, documentation or a configured server. It need not even be technical work. Managers also do the work of making: writing a memo, editing a budget spreadsheet or creating a slide presentation, for example.2

The human brain has evolved in a way that creative work, innovation and productivity are maximized when a person is able to focus and work on one task at a time for multiple hours. It takes several minutes, often half an hour or longer to get in the flow state of mind that results in peak performance at work. Hourlong or half-hour long meetings peppered throughout the day with breaks in between supposedly to do productive work result in low quality work, cause stress, and lead to unhappiness.

One way to do better quality work, get more done and be happier in your job is to divide your day into two halves. Get all your meetings, emails and administrative tasks done in the first half and spend the entire second half of the day doing enjoyable creative work that puts you in the flow state of mind. You will leave the office less stressed, more satisfied and happier each day.

If you are interested in learning more about the maker’s schedule concept, the article titled Maker’s Schedule, Manager’s Schedule by Paul Graham is a good introduction.

Below are the answers to some frequently asked questions.

Q: Who does this policy apply to?
A: This applies to all makers as listed above, especially all engineers and quality assurance staff, people who spend the majority of their time writing, designing or implementing software code, systems or designs. This also extends to those in management roles who’d like to use this time to do maker’s work.  At this time, we are not applying this policy to employees with special employment contracts like guild or unionized employees.

Q: When will we have maker’s hours?
A: Fridays after 12 noon, i.e. the latter half of all Fridays going forward until further notice.

Q: Does this mean I can go home early on Fridays?
A: No. This is not a summer hours policy. This is meant to be uninterrupted software engineering and development time. It does not change anything about when you are expected to be in the office. The prior agreed upon schedules you have with the company will continue.

Q: Why Fridays and why only Friday afternoons?
A: We analyzed our organization’s current meetings schedule and found that Friday afternoon is the period where there are least meetings and those meetings can be rescheduled with least impact. We are starting the pilot program with Friday afternoons. After some months of evaluating the results, we may extend it, keep it the same or cancel it. Until Further notice, this policy applies only to Friday afternoons.

Q: Does this mean I only get Friday afternoons to write code?
A: No :-) What this means is that we must all do our best to not organize, nor attend meetings on Friday afternoons so that time is exclusively reserved for writing code, building systems and doing other maker’s work. You are expected to do maker’s work every business day and to manage your own schedule to block off enough time to do that on others day the same way you already do.

Q: What about production emergencies? Can I get called into an emergency meeting to deal with a critical production emergency?
A: Yes. Production emergencies qualify among the rare exceptions.

Q: What about meetings between makers? For example, between two software engineers.
A: That is a slippery slope. We are strongly discouraging meetings on Friday afternoons in this policy, but we are not the meeting police and are not going to ban all meetings, especially if all the attendees have a strong desire to meet. We trust you to use your best judgement and lean towards not holding meetings on Friday afternoons unless you determine you have a good reason to make an exception to this policy. Our suggestion is this: Pair programming is encouraged. Working sessions are ok, assuming that in the entire working session multiple makers are making something together. However having staff meetings at this time is not a good idea. Nor is it a good time to have your weekly 1-on-1 with your manager. Remember your manager is likely to be using this time to do their own maker’s work. So on the question of can developers hold a meeting with just developers at this time, ask yourself why. What is the meeting for? Is it a working session where each of you will make something together? If yes, that’s likely fine. If not, schedule it for another time.

Q: What should I do when someone invites me to a meeting on Friday afternoon and I plan to observe the maker’s schedule and write code at that time?
A: Please always be respectful, courteous and friendly while declining meetings. Use your discretion and common sense. If the meeting request comes from someone it may not be wise to decline, consult with your boss. In many cases, you can politely, respectfully and nicely point the meeting requester to this policy at http://www.rajiv.com/blog/2013/05/24/makers-schedule-for-managers-too/ and suggest or request another time. Letting your collaborators know about this policy in advance will also help.

  1. Thanks to David Perpich for suggesting this executive summary. []
  2. Since processing email has become such an information overload problem, distraction and waste of time these days, we hesitate to classify doing email as productive maker’s work. If you don’t have the unproductive bad habit of checking your email every 15 minutes and instead you process your email during a few blocks of time a day, you may consider email productive work too. []

Product Maintenance vs. New Development on Web Sites, Mobile Apps and Other Digital Products

maintenance

Maintenance of a digital business product (e.g. a Web site, mobile app, or software) refers to the work that includes modifications made after delivery to production to fix bugs, address compliance/security issues, or improve scalability/performance These modifications can be to the product’s software code, configuration, documentation, hardware, or surrounding network.

Maintenance is often contrasted with the development of new features and functionality to enhance or increase the capabilities of the product.

The distinction between maintenance vs. new feature development comes up in both internal and external contexts. In the external context, it pertains to a maintenance contract with an outside vendor. A vendor is likely to define maintenance in strict and narrow terms to minimize costs the vendor has to incur. In the internal context it is helpful in analyzing staffing and resourcing capacity of an organization which is needed to understand the bandwidth available to work on new initiatives.

The purpose of this article is to help answer questions such as: What do you mean by maintenance work? What should we consider maintenance vs. new feature development? Is there an industry standard definition? What’s our selection criteria? Can you give some examples? Other questions that often accompany these are: What ratio of maintenance vs. new features development work is considered best practice for our industry? How does our organization’s situation compare with that? These latter two questions require analysis specific to your organization and a set of reference organizations to answer.

The following illustrates some typical attributes of maintenance vs. new features.

Maintenance

New Features

Consequences of not doing Often maintenance work is not optional, i.e. the consequences of not doing maintenance range from the gradual decline of the product to complete breakdown of the product. Not doing maintenance often causes legal, security and other compliance issues. Implementation of new features, functionality and services is often optional. The negative consequences of not implementing new features typically include loss of competitive advantages, loss of market share, and decrease of relevance.
Financial impact The financial value of an asset decreases in the absence of maintenance. Maintenance work helps maintain the value of the product, but typically does not substantially increase it. New features and functionality increase the financial value of an asset.

Types of Maintenance Done on Web Sites and Mobile Apps

Learning from the The International Standards Organization and International Electrotechnical Commission’s ISO/IEC 14764 specification, we define maintenance of a digital business product such as a Web site or mobile application into the following four categories:

  • Corrective maintenance: Reactive modification of the software, configuration or infrastructure powering a Web site or mobile app performed after delivery to correct discovered problems. E.g.
    • Bug fixes
    • Fixing design compatibility (HTML/JavaScript/CSS) issues with Web browsers that have been around for a while.
    • Install security patches to address existing security/privacy holes.
  • Adaptive maintenance: Modification of a software product performed after delivery to keep a software product usable in a changed or changing environment. E.g.:
    • Modifying a Web site to work with an newly emerging Web browser like a recently released version of Google Chrome.
    • Modifying a mobile app to comply with changes made by third parties, e.g. new terms of service from Facebook or Twitter, a new API released by Google+.
    • Increase capacity to handle increasing traffic on a Web site.
  • Perfective maintenance: Modification of a software product after delivery to improve performance or maintainability. E.g.:
    • Changes to servers or network to make the Web site serve quicker.
    • Refactoring the software code to make it less expensive to maintain and modify.
  • Preventive maintenance: Modification of a software product after delivery to detect and correct latent faults in the software product before they become effective faults. E.g.:
    • Y2K :-) fixes made on or before December 1999. Remember that? When we had to install patches and test software on the Web sites in the 90s.

The following are some examples of maintenance work contrasted with new features development.

Maintenance

New Features

Fixing bugs in existing software. Developing new Web applications or mobile apps. Adding major new features and functionality to existing apps.
Modifying software code on a Web application to comply with changes to Facebook’s application programming interface (API). Adding new software code to add support in a Web application for a new social network, e.g. Google+.
Rewriting parts of a Web application to handle increased usage (E.g. more users, users storing more data, growing traffic.) Developing a complete new Web application to replace an existing application to address the shortcomings the previous one has.
Routine work like generating, analyzing and publishing periodic reports. Designing, developing and automating a new report that has not existed before in the organization. Adding significant and major new features to an existing report.

In certain cases, the distinction between maintenance and new development can be blurry. For example, when rewriting (rebuilding) major parts of an application to overcome its shortcomings, at what point do we consider it a replacement (new development)? In such cases, it may be subjective based on how the output of the work is classified, branded and marketed. If it is being announced, launched and sold as a great new product to replace the old, it would be new development. This can cause issues when a customer has a maintenance contract with a vendor and doesn’t want to pay for the new version that doesn’t add substantial new functionality. That, however like all contracts, becomes matter of trust, relationships and reputation.

In the cases where an external vendor has a maintenance contract with a customer, the vendor is likely to not classify certain maintenance activities as maintenance since doing so would incur significant and unexpected costs the vendor would become responsible for. For example, a vendor may not include many adaptive types of maintenance in the contract. A vendor may want to charge the client more to handle increased usage.

Ratio of Maintenance vs. New Development

There is no “magic number” answer to this question. Ratios like 80/20, 70/30, 60/40 or 50/50 are specific to an organization in a specific phase. In today’s dynamic and hyper-competitive environment, seeking a best practice ratio for an industry may be misleading. A company or project in the startup phase typically has a lot more new development than maintenance happening than a organization with a mature product. Even within a particular company, the ratio changes as new projects start, are worked on and are completed. Decommissioning obsolete products also lowers the maintenance.

Related observations

Going down to maintenance mode for specific products does make good sense in certain circumstances. For example, when you are working on a major new version of the Web site or product, you can put the current (soon to be replaced) version in maintenance mode.

Sometimes, in times of financial hardship, companies try to hunker down to maintenance mode where they shed the new development work and do only the work required to “keep the lights on” to ride out the storm. This type of horizontal cutting of all products may initially seem like a good approach but in most cases, it is not a wise practice. In most circumstances of financial hardship, it is often wiser to cut products vertically. In other words, focus on doing a fewer number of products well rather than keep doing the same number of products poorly. Stopping new development and product innovation enables existing competitors to gain a huge lead over you. It creates opportunities for new competitors to your business and leads to the decline of your brand.

My article titled Trinity Method of Technology Management makes a case for decommissioning products the way Google does seasonal cleaning by ending products that no longer suit the company’s revised strategy.

References

 

Hosting Large-Scale Web Sites: Contract Review Guide for the CTO

If you host and operate large-scale Web sites, or negotiate contract agreements with vendors that provide such services, you need to understand what should be included in a Web hosting infrastructure. This knowledge will help you in three areas:

  1. Providing reliability, scalability & good performance
  2. Minimizing risks via security, privacy, regulatory compliance and reduction of vulnerability to potential lawsuits
  3. Reducing and controlling costs

This guide is meant to help you review upcoming contracts as well as existing services.

Likely audience for this article: Managers, directors and vice presidents of technology, operations or finance at organizations operating large-scale Web sites; Executives supervising technology: CTO, CIO, CFO, COO.

Seven Aspects of Large-Scale Web Hosting

Large-scale Web hosting infrastructure and services can be organized into the following seven areas:

  1. Servers & Environments
  2. Network & Other Appliances
  3. Managed Hosting Services
  4. Third-party Provided Services
  5. Program Management Office, PMO
  6. Account Management
  7. Infrastructure & Facilities

Checklist for Review

You can use the following checklist to review your hosting services or a vendor’s proposal.

What to look for

When you review each item below, consider:

  • Is this item included in the vendor’s proposal or in the services we are currently receiving? If it is not included, what are the good reasons it isn’t included?
  • Is this needed for my organization’s current business requirements? Can we do without it? Is it a must have or nice to have for present and reasonable future needs?
  • What are the alternatives?
  • What is the unit price of this item? How does the price scale up as needs grow? How does the price scale down when need for this item decreases?
  • What level of fault-tolerance does this item need? i.e. redundancy, standby backups, time to recover

Some of the above review questions may apply only to things and not apply to services and processes.

Servers

Servers may be physical hardware servers and/or virtual servers managed using software such as VMWare, Parallels Virtuozzo or Xen. The services listed below can each run on separate servers or multiple services can run on a server. It is generally better to have servers running only one (or minimum number) of the major services listed below. That reduces complexity and saves expensive staff time saved maintaining, troubleshooting and recovering. Virtualization makes it economical to have multiple virtual servers on the shared physical hardware economize costs.

The following is a list of commonly found services at large-scale Web sites that require servers.

  • Web
  • Application
    • Content Management software. This is the software that the Editorial and Production teams use to submit, edit, package and manage articles, photos and other Web site content
    • Dynamic Content Assembly. Typically done using Portal Server software, either third-party supplied or in-house developed
    • Data Processing. E.g. workflow engines, jobs/tasks processing servers
    • Middleware
    • Other applications. These are applications that happen to be separate from the main content management system. They could be separate for any number of reasons. E.g. blogs, forums
  • Database

Server Environments

An environment is a self-sufficient set of servers assigned to serve a purpose as described below. Large-scale Web sites typically utilize multiple environments.

  • Production
    • This serves the Web sites to the customers and public.
    • Typically has 99.9% or higher uptime guarantee in the Service Level Agreement
      Please refer to the accompanying table titled Understanding SLA Uptime Guarantee Percentages to compare different time windows when the SLA Uptime measurement gets reset. I recommend that you ensure that the reset window you get is the same duration as your billing cycle (usually monthly) or shorter. This will help avoid having long downtimes without penalty.
  • Staging
    • This is the environment where content packages are developed, integrated and previewed by Editorial, Design and Production teams before they are published to the end-users. For example, when working on a major site redesign or relaunch for several months. Since the tech teams are often making changes to the Development Integration and QA environments, they are not suitable for content integration work by the Editorial and Design teams. Staging is used in large-scale Web sites where mutiple Editors, Designers and Production staff are collaboratively creating content packages and new sections. In smaller Web sites or in cases where just one or two Editors are working on a piece of content like an individual article, previewing is done in the Production environment itself with access controls.
  • Quality Assurance (QA)
    • The QA engineers perform Functional Testing and Load Testing here. Doing functional testing while a load test is running is sometimes a good idea as it simulates usage closer to live production.
  • Development Integration
    • Software product code developed by different engineers is integrated here. There could be continuous integration or nightly builds.
    • This is where developers ensure that their code works with other developers’ code (does not break the build, and does not conflict resulting in undesired functionality)
    • Programmers should ensure that the product works here before handing it off to the QA engineers for testing

In a virtualized system the environments may not be physically separate and may regularly grow and shrink at different times. For example when hosted at a cloud computing provider, the QA environment may scale up during load testing and shut down completely during the hours the QA team is not working.

Network & Other Appliances

These are devices to which various servers are directly or indirectly connected.

Managed Hosting Services

  • Systems Administration
    • This typically includes all the management of the physical hardware up to and including the operating system and popular applications that complement the operating system.
  • Database Administration Services
  • Applications Management Services
    • This typically includes all the administration of the applications that run on top of the operating system.
  • Systems Monitoring, Alerting & Reporting
  • Web Support Help Desk, 24×7

Third-party Services

Program Management Office, PMO

  • Project Management
    • PM people, organization, processes
    • Collaborative project management tools, e.g. JIRA, RallyDev, Mingle
    • Shared documentation management tools, e.g. Wiki
  • Change Management Processes & Tools
    • Documentation system
    • Tools for source control, build & deployment
  • RASIC Matrix Describing Roles & Responsibilities
  • Escalation Flowcharts
  • Crisis Management & Emergency Procedures

Account Management

  • Customer service
  • Relationship management
  • Master Services Agreement, MSA
  • Statements of Work, SOW
  • Service Level Agreement, SLA
    • What to look for in the SLA is the subject of a separate article in this series.
  • Billing & Service Level Agreements
    • Monthly bills provided by telecommunications (telco) and hosting companies tend be extremely complex and lengthy. As a result, they are difficult and time-consuming to review.
    • Always factor in one-time setup fees and any implementation fees paid to the vendor and/or their partners in the total cost of the contract. Don’t look only at the recurring charges. A simple way to do this is this:
      contract cost = implementation fees + (estimated recurring fees x number of recurrences committed to)
      e.g. contract cost for 1 year = setup fees + (estimated monthly charges x 12)
      For most hosting / telco contracts I recommend this simple calculation over more sophisticated methods that factor in time value of money because the recurring fees are estimates anyway.
    • Make sure that 1-year contract is really a 1-year contract and not effectively a 13-month, 15-month or even longer contract by ensuring the following:
    • The contract’s start date is the first date for which the recurring billing begins. This is useful in determining the default end date of the contract. For example:
      If you agree to a 1-year contract with monthly billing when the first monthly bill will be for services provided April, 1, 2010 through April 20, 2010, then the default termination date for the contract is March 31, 2011. If the service provider estimates 3 months for implementation that ends on June 30, 2010 and they charge you the monthly services for April, May and June, don’t let the vendor tell you the contract start date is July 1. If you paid the monthly fees for services provided on April 1, then the start date is April 1.
    • If the vendor charges you fractional monthly fees for the implementation period and/or charges you one-time set up fees, then you should negotiate and agree on a contract end date that is fair to both parties. Use this guideline: The contract commitment should aim towards a certain money target (revenue for the vendor). If the implementation fees are equivalent to say, 3 months of recurring billing, you might agree that end date is after 9 months of the first recurring billing cycle.

Tips for Reviewing Technology Vendor Contracts and Service Level Agreements (SLA)

  • Don’t let the vendor use a lower monthly rate for calculating SLA credits.An example: The vendor’s contract section X.YZ1 states that the customer’s service credits will be calculated against a monthly rate of $6,000.00 per month. However, the vendor’s estimated total charges seem to be at least $10,000 per month. Don’t let the vendor calculate service credits based on a lower monthly bill than the actual monthly bill.
  • Don’t get locked into a deal where you could be stuck with overages every month.An example: The vendor’s contract section X.YZ2 locks the customer into the vendor’s service for two years for a total of between $80K/month to $100K/month if the customer remains at under 100 million page views per month. If the customer’s page views go over 100 million in any month, then there will be additional overage charges. There is no out clause nor a pre-determined next rate tier in the customer’s favor in the contract. If customer s traffic rises to regularly being over 100 million page views per month, the customer will be trapped in a contract with recurring overage charges. Make sure that if you have overages in the future, you can move into the next tier, preferably at a better rate.
  • Beware of vaguely defined scheduled maintenance and make sure scheduled maintenance needs customer’s prior approval.An example: The SLA section X.YZ3 states that the vendor can schedule maintenance downtime with 48 hours notice. They can give the customer notice by one of many means. There is no requirement for the customer to review or even acknowledge receipt. This is slanted too much in vendor s favor. The customer should have some ability to reschedule scheduled maintenance or ask for it to be shorter in duration if it interferes with the customer’s business.
  • Make sure that service credits can also be redeemed as cash.An example: The SLA section X.YZ4 states that service credits are not cash. Such credits will only be applied to future service billings. This is usually fine, except if it happens in the last month of the contract or if there is not enough future usage to use up the credits. In such instances, service credits should be payable as cash.
  • If the vendor will charge you for overages, the vendor needs to be responsible for service at the overage usage levels too.An example: The SLA section X.YZ5 states that response time service credits will not apply if monthly page-views exceed 120 million. This is not fair to the customer. The vendor is fine with charging the customer overage fees, but not being responsible for level of service at those levels. If the vendor charges overage fees, it should bind them to providing full service at the exceeded usage as well.

Infrastructure & Facilities

This item, infrastructure & facilities, is beyond the scope of this article. It includes the buildings, electric power, generators, climate control, physical security and related staffing.

This article is part of a series titled “Guide for the CTO: A compilation of articles on how to lead and manage technologies, projects and people”.

Understanding SLA Uptime Percentages

The table below helps illustrate why you should ensure that the “SLA Uptime Measurement Meter Reset Window” is the same duration as your billing cycle or shorter.

Availability % Downtime per year Downtime per month (30 days) Downtime per week Downtime per day
90 “one nine” 36.5 days 72 hours 16.8 hours 2.4000 hours
95 18.25 days 36 hours 8.4 hours 1.2000 hours
97 10.96 days 21.6 hours 5.04 hours 43.2000 minutes
98 7.3 days 14.4 hours 3.36 hours 28.8000 minutes
99 “two nines” 3.65 days 7.2 hours 1.68 hours 14.4000 minutes
99.5 1.83 days 3.6 hours 50.4 minutes 7.2000 minutes
99.8 17.52 hours 86.23 minutes 20.16 minutes 2.8800 minutes
99.9 “three nines” 8.76 hours 43.8 minutes 10.1 minutes 1.4400 minutes
99.95 4.38 hours 21.56 minutes 5.04 minutes 43.2000 seconds
99.99 “four nines” 52.56 minutes 4.32 minutes 1.01 minutes 8.6400 seconds
99.999 “five nines” 5.26 minutes 25.9 seconds 6.05 seconds 0.8640 seconds
99.9999 “six nines” 31.5 seconds 2.59 seconds 0.605 seconds 0.0864 seconds
99.99999 “seven nines” 3.15 seconds 0.259 seconds 0.0605 seconds 0.0086 seconds

Sources: Wikipedia, my calculations

Checklist for Migration of Web Application from Traditional Hosting to Cloud

In 2010, Cloud Computing is likely to see increasing adoption. Migrating Web applications from one data center to another is a complex project. To assist you in migrating Web applications from your hosting facilities to cloud hosting solutions like Amazon EC2, Microsoft Azure or RackSpace’s Cloud offerings, I’ve published a set of checklists for migrating Web applications to the Cloud.

These are not meant to be comprehensive step-by-step, ordered project plans with task dependencies. These are checklists in the style of those used in other industries like Aviation and Surgery where complex projects need to be performed. Their goal is get the known tasks covered so that you can spend your energies on any unexpected ones. To learn more about the practice of using checklists in complex projects, I recommend the book Checklist Manifesto by Atul Gawande.

Your project manager should adapt them for your project. If you are not familiar with some of the technical terms below, don’t worry: Your engineers will understand them.

Pre-Cutover Migration Checklist

The pre-cutover checklist should not contain any tasks that “set the ship on sail”, i.e. you should be able to complete the pre-cutover tasks, pausing and adjusting where needed without worry that there is no turning back.

  • Set up communications and collaboration
    • Introduce migration team members to each other by name and role
    • Set up email lists and/or blog for communications
    • Ensure that appropriate business stakeholders, customers and technical partners and vendors are in the communications. (E.g. CDN, third-party ASP)
  • Communicate via email and/or blog
    • Migration plan and schedule
    • Any special instructions, FYI, especially any disruptions like publishing freezes
    • Who to contact if they find issues
    • Why this migration is being done
  • Design maintenance message pages, if required
  • Setup transition DNS entries
  • Set up any redirects, if needed
  • Make CDN configuration changes, if needed
  • Check that monitoring is in place and update if needed
    • Internal systems monitoring
    • External (e.g. Keynote, Gomez)
  • Create data/content migration plan/checklist
    • Databases
    • Content in file systems
    • Multimedia (photos, videos)
    • Data that may not transfer over and needs to be rebuilt at new environment (e.g. Search-engine indexes, database indexes, database statistics)
  • Export and import initial content into new environment
  • Install base software and platforms at new environment
  • Install your Web applications at new environment
  • Compare configurations at old environments with configurations at new environments
  • Do QA testing of Web applications at new environment using transition DNS names
  • Review rollback plan to check that it will actually work if needed.
    • Test parts of it, where practical
  • Lower production DNS TTL for switchover

During-Cutover Migration Checklist

  • Communicate that migration cutover is starting
  • Data/content migration
    • Import/refresh delta content
    • Rebuild any data required at new environment (e.g. Search-engine indexes, database indexes, database statistics)
  • Activate Web applications at new environment
  • Do QA testing of Web applications at new environment
  • Communicate
    • Communicate any publishing freezes and other disruptions
    • Activate maintenance message pages if applicable
  • Switch DNS to point Web application to new hosting environment
  • Communicate
    • Disable maintenance message pages if applicable
    • When publishing freezes and any disruptions are over
    • Communicate that the Web application is ready for QA testing in production.
  • Flush CDN content cache, if needed
  • Do QA testing of the Web application in production
    • From the private network
    • From the public Internet
  • Communicate
    • The QA testing at the new hosting location’s production environment has passed
    • Any changes for accessing tools at the new hosting location
  • Confirm that DNS changes have propagated to the Internet

Post-Cutover Migration Checklist

  • Cleanup
    • Remove any temporary redirects that are no longer needed
    • Remove temporary DNS entries that are no longer needed
    • Revert any CDN configuration changes that are no longer needed
    • Flush CDN content cache, if needed
  • Check that incoming traffic to old hosting environment has faded away down to zero
  • Check that traffic numbers at new hosting location don’t show any significant change from old hosting location
    • Soon after launch
    • A few days after launch
  • Check monitoring
    • Internal systems monitoring
    • External (e.g. Keynote, Gomez)
  • Increase DNS TTL settings back to normal
  • Archive all required data from old environment into economical long-term storage (e.g. tape)
  • Decommission old hosting environment
  • Communicate
    • Project completion status
    • Any remaining items and next steps
    • Any changes to support at new hosting environment

The checklists are also published on the RevolutionCloud book Web site at www.revolutioncloud.com/2010/01/checklists-migration/ and on the Checklists Wiki Web site at www.checklistnow.org/wiki/IT_Web_Application_Migration

Save Money On Hosting & CDN By Optimizing Your Architecture & Applications

If you manage technology for a company that has a large Web presence, it is likely that a large percentage of your total technology costs is spent on the Web hosting environment, including the Content Delivery Network (CDN, e.g. Akamai, LimeLight, CDNetworks, Cotendo). In this article, we discuss some ways to manage these costs.

Before we discuss how to optimize your architecture and applications to have economical and the optimally low hosting expenses, let us develop a model for comprehensively understanding a site’s Web hosting costs.

Step 1. Develop a model for allocating technology operations & infrastructure costs to each Web site/brand

Let us assume for this example that your company operates some medium to large Web sites and spends $100K/month on fully managed1 origin2 Web hosting and another $50K/month on CDN. That means your company spends $1.8MM/year on Web sites hosting.

It is important to add origin Web hosting and CDN costs to know your true Web hosting costs, especially if you operate multiple Web brands and need to allocate Web hosting costs back to each. For example, let us assume you have two Web sites: brandA.com, a dynamic ecommerce site costing $10K/month on origin hosting plus $2K/month on CDN; and brandB.com, serving a lot of videos and photos costing $5K/month on origin hosting plus $19K/month on CDN. In this example, brandA.com actually costs $12K/month, which is half the hosting cost of brandB.com, $24K/month. Without adding the CDN costs, you may mistakenly assume the opposite that brandA.com costs twice as much to host as brandB.com. Origin hosting and CDN are two sides of the same coin. We recommend that you manage them both together from both technology/architecture and budget perspectives.

Then you add the costs of third-party vendor provided parts of the site rented in the software-as-service model. Next, add licensed software costs used at your hosting location. Let us assume that brandA.com also has:

  • some blogs hosted at wordpress.com for $400/month
  • Google Analytics for $0/month
  • Other licensed platform/application software running on your servers billed separately from the managed hosting. Let us assume brandA.com’s share of that is $1,000/month.

So your Web hosting and infrastructure costs for brandA.com would be $13,400/month. That’s $160,800/year.

Assuming that many of your Web sites share infrastructure and systems management & support staff at your Web hosting provider, you may not have a precise allocation of costs to each brand. That’s ok: It doesn’t need to be perfect nor a staff-time consuming calculation every month. Work with your hosting provider and implement a formula/algorithm that provides a reasonably good breakup and needs to be changed only when there is a major infrastructure change.

Side Note: In order to stay competitive, adapt to changes in the market and meet changing customer sites, brandA.com also needs to do product and software development on a regular basis. However, that’s beyond the scope of this discussion. Managing ongoing product and software development costs for brandA.com could be the subject of another article.

Step 2. Regularly review the tech operations costs for each brand and make changes to control costs

Every month, review your tech operations costs for your business as a whole and for each brand. Make changes in technology and process as needed to manage your expenses. If you don’t review the expenses on a monthly basis, you run the risk of small increases happening in various places every month that add up to a lot.

Without active management done on a monthly basis, brandA.com could creep up from $13,400 to $16,000 the next month and $20,000 the month after. That $1.8MM you were expecting to spend on hosting for the year could turn out to be $2.4MM.

So what does such active management include?

Monitor and manage your bandwidth charges. This is one to keen an eye on. If you bandwidth charges go over your fixed commit, your expenses can quickly blow over budget. If you find bandwidth use increasing, investigate the cause and make course corrections. In some cases, this may simply be due to expected increase in traffic, but in other cases it could be avoided. A related article about taking advantage of browser caching to lower costs provides some tips.

Request your engineers to monitor and manage your servers resource usage (CPU, memory) so that the need for adding hardware can be avoided as much as possible. Enable and ensure regular communications between your technology operations team and your software development team so that software developers are alerted of any application behavior that is consuming more than expected server resources. Give the software developers time to resolve such issues when found.

Review the invoice details to make sure you understand and are in agreement with the invoice. A Web hosting bill can be very detailed and complex to understand. Do not hesitate to ask the hosting provider to explain and justify anything that you don’t understand. Don’t just assume the bills are always correct. They could (and occasionally will) be mistakes in the bills. Be sure to dispute these with the vendor in a respectful and friendly way.

These are just some examples. Please feel welcome to make more suggestions via comments on this post.

The time (and thus money invested) in controlling tech operations cost will be well worth the savings / avoidance of huge cost increases.

Keep abreast of evolving technologies and cost saving methods. Periodically review these with your vendor(s).

Cloud computing is exciting as a technology, and it is equally exciting as a pricing model.

If you find market conditions have changed drastically, request your vendor to consider lowering rates/prices even if you are locked into a contract. You don’t lose anything by asking and the vendor’s response will be an indicator of their customer service and long term business interest with you.

  1. Fully managed Web hosting includes network & hardware infrastructure, 24×7 staff and real estate []
  2. The origin part of your Web hosting environments includes the network and server infrastructure at your hosting facility location(s) where your Web applications and installed and running. It could be in-house data centers or at providers such as RackSpace, IPSoft or Savvis []

Save Your Company Money In Monthly Bills Using Browser Caching

Bandwidth MoneyCompanies that operate heavily trafficked Web sites can save thousands of dollars every month by maximizing their use of browser-side caching.

Large Web sites pay for bandwidth at their Web hosting data center and also at their content delivery network (CDN, e.g. Akamai, LimeLight, CDNetworks). Bandwidth costs add up to huge monthly bills. On small-business or personal Web sites where bandwidth costs don’t go over, this is not an issue, but on large Web sites, this is important to address and monitor.

Companies operating large Web sites often have complex situations like the following:

  • An comprehensive and deep understanding of all technology cost drivers and their impacts on each other. For example, a programmer may think they are saving the company money by architecting an application in a way that it requires minimal hardware servers, but not realize that the same design actually results in even higher costs elsewhere like CDN bills.
  • Busy development teams working on multiple projects on tight timelines. This results in compromises between product features/timelines and technical/architectural best practices/standards.
  • Web content management and presentation platform(s) that have evolved over the years
  • Staff churn over the years and an uneven distribution of technical knowledge and best practices about the Web site(s)
  • The continued following of some obsolete “best practices” and standards that were established long ago when they were beneficial, but are now detrimental.

Tech teams at complex Web sites would likely find upon investigation that their Web sites suffer from problems that they either didn’t know about or didn’t know the extent of the damage they are causing.

One such problem is that certain static objects on the company’s Web pages that should be cached by the end users’ Web browsers are either not cached by the browsers at all or not cached enough. Some objects are at least cached by the CDN used by the company, but some perfectly cacheable objects are served all the way back form the origin servers for every request! An unnecessarily costly situation that can be avoided.

In addition to wasteful bandwidth charges resulting in high monthly bills, there are also other disadvantages caused by cacheable objects being unnecessarily served from origin servers:

  • They slow down your Web pages. Instead of the browser being able to use local copies of these objects, it has to fetch them all the way from your origin servers.
  • Unnecessary load on origin Web servers and network equipment at Web hosting facility. This can be an especially severe problem when a Web site experiences a sudden many-fold increase in traffic caused by a prominent incoming link on the home page of a high traffic like Yahoo, MSN or Google.
  • Additional storage in logs at the origin Web hosting locations’ servers and other devices.
  • Unneeded processing and work the origin servers, network equipment, CDN, the Internet in the middle all the way up to the client browsers have to do to transfer these objects from origin to the end user’s browser. Be environmentally friendly and avoid all this is costly waste.

The increase in bandwidth, load on servers and networking equipment and log file storage space increases caused by a few objects on Web pages being served by origin servers for every request may mistakenly seem like an insignificant problem, but little drops of water make the mighty oceans. Some calculations will show that for large Web sites, the cost of this can add up to tens of thousands of dollars a month in bandwidth costs alone.

How should companies operating large Web sites solve this problem?

For technology managers:

  • Make it a best practice to maximize the use of browser-side caching on your Web pages. Discuss this topic with the entire Web technology team. Awareness among the information workers is important so that they can keep this in mind for future work and also address what’s already in place. Show the engineers some sample calculations to illustrate how much money is wasted in avoidable bandwidth costs: that will prove this is not an insignificant issue.
  • If this problem is widespread in your Web site(s), make the initial cleanup a formal project. Analyze how much money you’d save and other problems you’d solve by fixing this and present it to the finance and business management. Once you show the cost savings, especially in this economy, this project will not be hard to justify.

For engineers:

  • Read the article about optimizing caching at Google Code for technical details on how to leverage browser and proxy caching. It explains the use of HTTP headers like Cache-Control, Expires, Last-Modified, and Etag.
  • Review any objects that are served by origin servers every time for legacy reasons that may now be obsolete.
  • Combine some JavaScript files commonly used by your Web pages so that the one unified and shared file would have higher caching probability. Do the same with external CSS style sheets.
  • Study a good book on Web site optimization like Even Faster Web Sites: Performance Best Practices for Web Developers. Share these recommendations and hold a discussion with your tech and production colleagues.

Future of Content Management for News Media for Web sites

Content on Web sites should be managed using systems that were designed from the ground up for the Web. Traditional content management systems with a legacy of features and workflows used for paper-based print products like newspapers and magazines are unsuitable for Web sites. The future of news media content management for Web sites is in:

  • simple & quick workflows
  • blogs & wikis as the main content types for text
  • social networking & community publishing

simple & quick workflows

Complex editorial workflows make sense for print products (on paper) , where once the edition is done, the content and presentation state is “locked” and sent to the presses. Working with Web content writers and editors over the past decade, I have learned that simple, quick workflows are preferable for Web sites. Many Web site producers who hail from print backgrounds now share the same conclusion that complex content management is a hindrance to successful Web site production.

The concept of an edition of the entire product is not necessary for a content Web site. The atomic unit that can be managed and published together can be a package of articles and multimedia or even just one article. A Web site is a living, dynamic, ever changing collection of content where individual items can be updated whenever required or desired or even automatically based on usage.

To be competitive, content needs to be updated and published quickly. Corrections can be made anytime. Thus for Web sites, the editing and approval process should be streamlined and quick all the way from authoring to posting on the site.

A new concept

The ability of online word processors like Google Docs or WriteWith to enable multiple people to edit a document simultaneously and collaboratively is different paradigm from traditional check-in/check-out access control.

blogs & wikis as the main content types for text

Content management system (CMS) which offers the simplicity of blogs and are extensible via plug-ins to add functionality like WordPress or MovableType, make good foundations of a CMS for a news media Web site.

For revisions, editing history and access control, wiki software works well. WikiPedia and WikiNews, which are powered by the MediaWiki software are two good examples.

The concept of content management systems that combine the agility of blogs and editorial control of wikis is interesting to follow. The term bliki seems to be the leading classification of such products.

In many newsrooms, writers are increasingly using blog posts to publish news articles instead of their enterprise-class content management systems. When asked why, they reply because it is simpler and quicker and they don’t need the overhead of things like complex approvals, advanced version tracking and access controls.

social networking & community publishing

Managing content using a blog or wiki is social networking and community publishing activity. On the readership side, successful social news sites like Digg and Reddit have accelerated the evolution of journalism and readership habits towards the social/community model. The distinction between authors and readers itself is blurring with wikis and comments on blogs.

Social networking features are being added to a variety of Web sites. Going forward, expect to see social networking and community features in content management systems.

Conclusion

Media companies should move to using CMS products that prefer simplicity over complex editorial workflows which were a legacy of writing and editing for print products. A news item, story or blog post should be the same content type. It is likely that blogging products that have proven so successful in empowering talented individuals in competing with large companies will evolve into content management systems with the addition of wiki functionality.