Web Site of Rajiv Pant

My journalist colleagues at Wired.com published some of my comments related to Amazon S3.¹ Wired also posted another article titled Customers Shrug Off S3 Service Failure. I agree with the views of many of the customers expressed in the article. Don MacAskill, CEO of the popular photo hosting site Smugmug, wrote an understanding post about it.

My entire career working for media companies, I’ve held firm the belief that the uptime, reliability, performance, scalability, performance and security of commercial Web sites is of paramount importance. When sites that I’ve been responsible for have had issues, my colleagues and I have given our personal time and energy to resolution. With my teams, I spend considerable time on proactive measures. In my jobs, I’ve had the honor of working closely with and learning from some who do an excellent job running technology operations.²

Experience has taught that things can and sometimes do go wrong. Sometimes calculated risks don’t pan out. Sometimes mistakes cause problems. We are human. We should strive for perfection; we can get close to it, but not fully attain it. We should be prepared for such scenarios. When they happen, we should work diligently and expeditiously on resolution and have frequent and honest communications with stakeholders and customers. Such communications during the incident should include:

• Current status
• What is the full impact?
• Estimated time to resolution
• Any recommended workarounds until resolution, if practical
• Assurance that it is being worked on
  • It often helps to mention who all are working on it and what they are doing

The post-incident communications to stakeholders and customers should include:

• Summary
• What happened, how and why it happened?
  • Including full description of all impact
  • Do not blame³ third-parties or say things like “beyond our control”. A technology leader takes responsibility equally for both insourced and outsourced products and services.⁴
• How it was resolved
  • If the resolution is temporary or long-term
• Next steps
• Plan for eliminating or minimizing this and similar incidents from happening again
• Thank all those who helped resolve and the customers for their understanding
  • Specify any ‘make goods’ or returns you plan to make to the customers, if appropriate

Stakeholders and customers here refer to internal customers of the technology operations team (e.g. the concerned folks in editorial, marketing, sales, finance, legal and other departments). External communications to the public Internet should be handled in consultation with legal and public relations.

S3’s outage (or any outage) isn’t to be taken lightly, but I have faith Amazon and their customers will learn from it.

Disclaimers:

• As explained in the terms of use of this site, any opinions expressed on my personal Web site do not reflect those of any employer, past or present. My Web site and I in my personal life neither represent nor speak for any corporation.
• I have no affiliation, financial or otherwise with Amazon.com. I happen to be a user of their products and services, some of which I like and some that I don’t.
• Personal Web sites like this are exempt from the performance requirements of corporate Web sites My personal Web site is for expressing, learning and R&D. It also happens to be hosted on Amazon EC2 and S3.

1. Silicon Alley Insider and ValleyWag have amusing spins on it. [↩]
2. E.g. Some very talented and accomplished technology operations leaders including Bob Pick, Jamie Gerdes (now at Google), Mark Cox, Chris Jennewein, Patrick Carter, Rocky Vienna and Sharon Mandell. [↩]
3. There may be extreme instances, especially when criminal activity or malicious wrongdoing was the cause where it would be appropriate to blame someone. [↩]
4. It is ok to name service providers  or mention external events for explaining what happened, but don’t do it in a “it was their fault, not ours” tone. The technology leader should factually describe what happened and take responsibility. [↩]

Described here is one way to enable technologists to grow their careers in your organization while still allowing them to focus on the type of work they are best at and enjoy most.

The typical management career growth path does not suit some technical people. These information workers need to grow in their careers (gain greater compensation, responsibilities and influence) without having to become managers of other people. A good way to achieve that goal is to create a technical career growth track in your organization.

The following table shows management seniority positions alongside their equivalent technical seniority positions.

In this model, for example, an architect role is at the same compensation and influence level as a manager role, assuming that the particular manager and architect being compared add similar value to the company. If the organization requires more layers, say, a senior architect would be at the same level as a senior manager.

If the organization prefers consistent titles for levels, the system could name them like this: the fellow role as director & fellow, the senior architect role as senior manager & architect, etc. In the case of a fellow who is at a VP or SVP level, they should always be named VP & fellow or SVP & fellow.

Here is a definition of the fellow role from WikiPedia:¹

Large corporations in research and development-intensive industries ² appoint a small number of senior scientists and engineers as Fellows. Fellow is the most senior rank or title one can achieve on a technical career, though some fellows also hold business titles such as vice president or chief technology officer.

Such a technical career growth plan brings many benefits to your organization.

• It helps retain good technologists who want to grow in their careers, but want to do keep doing the type of work they are best at and enjoy doing: technical work.
• It avoids brilliant technical people from being “pushed” (by themselves or their supervisors trying to “reward” them) into people-management responsibilities.
• It reduces situations of having too many people-managers but not enough people-management positions over time as people get promoted.

Care should be taken to recognize that some technical people do enjoy making the transition to people-management roles and the presence such a technical track should not discourage them. Having an alternate career growth track option is about presenting employees with more than one choice.

A similar system could also be applied to other departments with individual contributors. For example, creative design.

1. Definition of Fellow at WikiPedia http://en.wikipedia.org/wiki/Fellow and Wikitionary http://en.wiktionary.org/wiki/fellow [↩]
2. IBM or Sun Microsystems in information technology, and Boston Scientific in Medical Devices for example [↩]

I’ve found this to be an effective way of evaluating potential hires compared to just interviewing in a question/answer format: Put them to work for a few hours (or even days/weeks/months as contractors) and see how well they perform.

Having someone do the job as a short-term temporary contractor before hiring them is one of my favorite ways of testing them for the job. However, sometimes this isn’t practical. In those cases, having the candidate work on a short test assignment for a few hours can be very telling.

For example, when interviewing a software programmer, set them up on a computer and give them a programming assignment representative of the type of work they would do on the job. Give them reasonable freedom as you’d give an actual employee. If they need to research references / solutions on the Web, let them. If they want to call a friend for consulting help, that’s fine. If they ask for your or their “coworkers” for help within reason, that’s okay too.

Hiring a journalist? Give them a reporting, writing, and/or editing assignment. Hiring a web site producer? Have them build a mini-web site or edit a copy of an existing site. Hiring a designer, have them design a logo for a business or product. (This will help determine how well they can relate their design skills to the business aspects.)

While this is a simulation, the more realistic you can make it, the better.

For information worker type jobs, coming up with a realistic assignment is easier than one for a management or executive leadership position candidate. Another challenge with this is that a manager and leader’s job requires significantly more interaction with others. That means this simulation will consume a fair amount of your and your team’s time. However, remember that manager and executive leaders are very important hires, so the time spent in finding the best possible hire is totally worth it.

Some factors to consider in your evaluation. How resourceful is the person? Do they communicate with the customers? How do they interact with the customers? Do they get the job done? Do they seek help beyond their own resources? Do they become a drain on others’ productivity? How is the quality of their work? What’s their balance of being a team player and contributing individually? How is their planning? How is their presentation? Do they document their solution? How well do they explain their solution?

I admit this is too time consuming to do for everyone who applies for the job. You do need to filter the list down to between three to five candidates for this working interview. There are a couple of ways you can do this. For contract-to-potentially-hire positions, you can rely on a trusted vendor to do the filtering for you according to your guidelines. For others, you do need to review their resumes, see who has recommended them, do phone interviews and give them tests. Computerized tests are a good way to save you and your team some time during the initial screening, provided you present the test in a way that is not demeaning to the candidate. The time/money invested in setting up an automated online test is worth it. The test needs to be respectfully presented as a part of the standard process and should not feel like an entrance exam or impersonal screening.

So next time you have a position to fill, consider using a working-interview.

At interviews, technology executives are often asked about build vs. buy. The question would be better articulated as own vs. rent. See, building and buying in the true sense are often part of the same ideology: owning. For example, Google bought Blogger and YouTube. As a result, Blogger and YouTube programmers (builders) became Google employees. When people ask about companies buying software, they are almost always referring to leasing or renting software.

I consider using open source to be in the own camp, because you can withing reason do as you please with your copy of the open source software and no one can take it away from you. Open source makes owning even better. It enables owning and sharing at the same time, which benefits the community.

Often technology executives answer this question ambiguously. They claim they have no preference towards either owning or renting software. To me, when a person provides such a noncommittal answer, it means they might lack leadership, vision, a clear philosophy, or the courage to give an honest answer to a prospective employer.

I generally prefer owning over renting. This applies not just to software, but to other things in my life like owning a home and owning a car. There are other people who prefer to rent apartments and lease cars. Neither philosophy may be better in general than the other, but one of them is always better depending on the circumstances. Circumstances by themselves don’t make the own vs. rent decision. That’s where people come in. A leader does not manage their company or department at the whim of circumstances. A leader has a style, has preferences and applies them to the situation. Leaders who prefer either one of owning or renting can be equally successful in the same organization and circumstances.

Here I present my viewpoint on why sometimes a philosophy of renting software as much as possible runs companies into trouble. There are numerous examples of vendor solutions in search of problems. A vendor will often wow a team of executives with a product presentation and demo. The company will agree to rent the software at a high cost. Later, the company will find it wasn’t worth the investment, even though they may not admit it or even like to talk about it.

When a company owns a software product core to its business, it can use it towards a competetive advantage. When you own the software product, you also have control over your data. When a vendor hosts and manages your data, in most cases you have practically relinquished control of your data even though your contract may say on paper it is yours.

When you own the products you use, it is generally easier to integrate with other systems. Your timelines are less at the mercy of vendors.

However the future of software may be changing. With web applications becoming popular, even home users may be renting software on a pay-per-use basis. I’ll have more to write on that subject. Stay tuned.

When implementing a content management system or other product, customers often ask for workflows that require the least number of steps required to any given complete task. At first, this seems to make perfect sense; however consider this example of a car:

Before you can get inside your car and start driving you have to perform the following steps:

1. unlock the car door
2. pull the door handle
3. open the door
4. get inside the car
5. close the door

Steps 1, 2, 3, and 5 seem to add unnecessary actions to the workflow. The goal here is to be able to start driving to get to the destination. Over the course multiple car trips over a day, these steps seem to “waste” a lot time. An easier and “better” workflow may be for cars not to have a door at all. Then you’d save the steps of having to open and close the door.

However, with the current level of commonly available technologies, it makes sense for a car to have a door and require these steps before you can start driving.

Extra steps are often required to provide security, maintainability, reuse, reliability, scalability and performance.

Shortcuts aren’t always the best solution. You may save steps and thus cost now with shortcuts, but as a result you may pay much more later in other costs.

As technology advances, some necessary steps can be automated or eliminated. For example, some cars now have keyless entry that eliminates some of these steps. In the future, an advanced version of keyless entry may even open and close the door for you. However, expecting those in a car of today would be impractical.

Similarly in content management and other software extra steps aren’t always a bad thing. A good content management system isn’t one that allows web site producers to complete their tasks in the least number of steps. It is one that enables completion of the task in an optimal number of steps balanced with other factors like reuse, maintainability, flexibility and security.

Starting early, not driving recklessly fast

People who have worked with me are familiar with my trait of challenging the team to bring products and solutions to market as soon as possible. I’m a strong proponent for quickness to market and love to deliver sooner than the initially projected timeline. In this article, however, I present a different viewpoint for balance.

In product development, the question often comes up: How can we be quicker and faster to market with our products? We should ask instead: How can we be earlier to market with our products than our competitors? We should also ask: Is it more important to be early, or to deliver good quality and innovation?

For the medium and long term good of your organization and in the best interest of your customers, it is more important to deliver a high quality and innovative product than to deliver it quicker.

In most cases, successful companies are not the ones who are fast or early to deliver products, but those that deliver better products.

Take Google for example. They were a couple of years late to the Web search engine market and were reinventing a product that had already been established by others. Many thought the search engine market was already saturated. Remember some of the early ones like Infoseek, Lycos? Where are they now? Consider Microsoft and Apple: most of their products are not early, but they often succeed. The iPod came years after the early portable digital audio players. MySpace.com came up to dominate online social networking a couple of years after Friendster, Tribe and Orkut were already established.

Even when analyzing products whose success was due their being early to market, we find that early does not imply fast. These projects often started early and were executed at a comfortable, smooth pace.

As the saying goes, when you ask for quick and dirty, you get both. The benefits of speed to market are for the short term. In some cases, it does make sense to go for quick, short-term solutions. In all cases, however, one must give serious thought to whether that’s the correct path to choose considering the medium and long term goals.

People & Teamwork

In projects, working fast is often a recipe for failure, especially after starting late. The overwhelming majority of projects are not like 100 meter races, where speed results in victory. They are like football games, where factors like teamwork have much greater influence on winning.

The greatest factor affecting the success of projects is not speed, not technology, not even process or planning. It is people. Invest your time, energy and resources on your people and they will make your projects succeed more than anything else.

Whether you are a leader, manager or information worker, I recommend learning more about the people factor and practicing better people related activities at work. Here is a quote I like from a book: “People under time pressure don’t work better; they just work faster. In order to work faster, they may have to sacrifice the quality of the product and their own job satisfaction.” — Peopleware: Productive Projects and Teams, 2nd Edition, Tom DeMarco and Timothy Lister

Keep in mind this order of descending significance of factors in projects’ success:

1. People & teamwork
2. Priorities
3. Planning
4. Process & operations
5. Products & technologies
6. Pace & acceleration

Nowadays, it is becoming fashionable to belittle technology. I hear people say things like “technologies should not drive business decisions“, “define your requirements without worrying about technology and ask technology [people] to deliver them“. May sound logical at first, but is it? Consider this imaginary conversation between Bill, a technologist and Plato, a Platonist businessperson.

Bill: How do you commute to work?
Plato: I drive.
Bill: Why don’t you fly or teleport?
Plato: What do you mean?
Bill: Why do you use a car that drives on the road, why not a personal flying machine or a teleporter?
Plato: Because they aren’t invented yet.

Defining business needs without consideration of technology is impractical. It is being quixotic and ignoring the current reality. Leave the dreaming for the innovators since they and businesspeople are usually different people.

Think about ten important inventions that changed the world. How many of them were created with a business plan? How do you think the wheel was invented? How was fire discovered? Did someone create this world wide web with a business plan? Many important things were created with a business plan. The point here is not that business is unimportant, but that technology is important on its own merit.

Sound business practices have an important place in this world. Technology and innovation have a place in this world. One is not master to the other.

Victory is not defeating others. Victory is winning others.

The three skills that people in an organization have are leading, managing, and working. Every person has a combination of all three. I refer to a leader as someone who is strongest in leadership, a manager as one who is strongest in management and a worker as one who is strongest in workmanship.

To understand better what leadership is, let us first discuss it in the context of what leadership is not.

Leadership is not equivalent to managing others. A leader manages him/her self while teaching and guiding people to manage themselves and others.

Leadership is different from management in several aspects. While a person can be a good leader as well as a good manager, the two are two different talents. A President & CEO needs to be both a leader and a manager. The Chief Architect can just be a leader. A Project Manager can be just a manager. A programmer can just be a worker. A manager often needs to make others do things, but rarely needs to do those things her/himself. This does not imply that a manager’s role is less significant. The manager does the important work of making the workers successfully do their work.

A manager does not need to be an innovator, but a leader needs to be. A manager follows the rules and makes workers follow the rules. A manager is most effective when things are going according to plan. A leader on the other hand, is equally effective when things are going according to plan or not. A leader rewrites the rules when necessary. A leader guides the manager in coming up with a new plan when needed. A leader often realizes it even before a new plan is needed. A leader is a visionary. A leader imagines great things. The leader formulates a high level plan. The manager creates a detailed plan from that and manages workers to implement using that plan.

To be a leader does not mean being a high and mighty boss. A leader is a servant as much as a commander. A good leader cares about others. A leader must lead others to success. If a leader’s goal was to achieve success only for one’s own self, then I’d call him or her a climber, for one who climbs to success, but not a leader. A leader who does not benefit others serves no purpose in an organization or in society.

Being a leader sometimes requires sacrificing ones own interests for the good of others.

Contrary to some beliefs, a leader should be generally popular and liked. A leader does not do just what the leader thinks is best for others, but what the team decides is best for the team. A leader provides results and enlightenment (as in understanding of those results). If people consistently do not understand what or why the leader is doing, the leader needs to be replaced. A leader realizes that people are intelligent and gives them appropriate credit.

A leader is not merely a teacher. A leader is a student and an instructor.

Leadership is not easy. It is not for everyone. Can anyone become a good leader? Yes. Should everyone become a leader? No. (As an illustration consider the question: Should everyone become a carpenter? No.)

(Note: In his career, the author has been successful and won awards while working at various levels of Programmer, Manager, Regional Director, Vice President and External Consultant. The knowledge presented here is compiled from those experiences and by learning from others. The author does not claim these ideas as original.)