Web Site of Rajiv Pant

CAREER-CLEAR version 2.0 2010-Aug-26

CAREER-CLEAR is a system for doing fair, consistent and constructive employee performance evaluations and determining employee rank, title and compensation. It is meant to be used by supervisors to identify areas for improvement for their employees and to guide their career growth.

Employees are scored in a total of 5 categories. Upto 10 points can be earned in each category for a total of upto 50 points. The final score is then multiplied by a factor of 2 to give a standard scale of 0 to 100. Using a normalized 100 point scale allows it to remain consistent (by adjusting the factor) even as companies add/remove categories and items.

The resulting numerical score can be mapped to the employee’s level of seniority or rank for title and/or compensation. For example, a score of 81-100 could map to director/VP levels; 61-80 manager; 41-60 engineer/contributor; 21-40 junior level/apprentice. Since different departments — for example, software engineering and quality assurance testing — may have different pay scales, this score maps directly to rank/title, and those are mapped to salaries corresponding to the departments’ market rates.

The first four categories are described below. The fifth category is defined as discretionary/user-defined. CAREER-CLEAR is designed to be used in the real world, in a diversity of organizations and on a regular basis. The system won’t suceed if it is too rigid. On the other hand, the system must meet its goals of being fair, consistent and constructive for all employees. To accomodate and balance these goals, 20% of the criteria is meant to be user-defined at descretion of the manager within the fair, consistent and constructive guidelines.

It is inspired by systems described to be in use at Microsoft, Construx, FogCreek (Joel on Software) and Conde Nast Digital Technology. The latter was developed by Bobby Chowdhury, Brian Murphy, Janet Kasdan and Rajiv Pant.

The 5 categories are: Caliber, Leadership, Expertise, Role and Discretionary.

Caliber

This section measures the talent of the employee in general (non-technical) areas.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Ownership – Has identifiable long-term ownership of projects. This is a measure of the criticality, complexity and / or number of projects the employee has ownership in.
2. Responsibility – Is consistently reliable in terms of deliverables and time.
3. Communication – Communicates effectively with peers and other colleagues. Listens to and understands others’ viewpoints, challenges, needs and desires.
4. Consistency – Is approachable, predictable, receptive and consistently applies good judgment in all interpersonal interactions in the work place.
5. Innovative – Innovates and stays abreast of emerging technologies and finds ways to incorporate those technologies into systems.

Leadership

This section evaluates the positive influence the employee has on others.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Teacher, Coach & Motivator – Mentors others, makes great use of all information sharing tools available and is an active presenter. Rallies the troops and improves morale.
2. Enabler – Empowers and enables others to succeed.
3. Exemplary – Leads by example and goes above and beyond the ‘requirements’.
4. Maturity & Humility – Embraces others’ solutions, even when incompatible with one’s own. Incorporates feedback from others to find the best solutions.
5. Connector – Has familiarity with the ecosystem beyond one’s own projects. Functions as a hub which others are drawn to for a quick answer or a quick redirect towards an answer.

Expertise

This section quantifies the skills and experience of the employee related to the job function.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Fundamentals – Understands of the core technical concepts aligned with the given job function. This may include data structures & algorithms, testing, networking, etc.
2. Breadth of Expertise – Is a subject matter expert and go-to person for many areas of technology.
3. Pragmatic – Has a demonstrated ability to identify the best solution to balance what’s most theoretically ideal against what might be the most practical due to concerns about security, scalability, time to market pressures and cost.
4. Automator – Consistently works to drive improvement in processes and systems.
5. “Boy/Girl Scout Rule” – Leaves code and systems better off than they found them.

Role

This section enumerates the employee’s role and areas of contribution within the organization and beyond.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Strategic – Provides sound vision for broad, long-term goals.
2. Tactical – Oversees many projects or activities that move the organization towards strategic goals.
3. Operational – Steers day-to-day processes that achieve the tactical goals.
4. Executional – Implements repetitive tasks that make up the operational processes. A measure of quantity and more importantly, quality of work produced.
5. Industry Recognition – Is recognized externally as a leading technologist through contributions to open source projects, blogging, writing books, participating in technical committees, speaking at conferences, etc.

Discretionary

Please be sure to adhere to the goals of being fair, consistent and constructive for all employees in using this discretionary section. This category is not meant to be used to justify favoritism nor meant to be arbitrary. Good descretion comes from rational, reasonable and relevant criteria. Place items here that are not already covered in other categories and are important to your organization. A good rule of thumb is that you must be able to justify any criteria you apply here.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. discretionary / user-defined item 1
2. discretionary / user-defined item 2
3. discretionary / user-defined item 3
4. discretionary / user-defined item 4
5. discretionary / user-defined item 5

Tweet

Save

If you host and operate large-scale Web sites, or negotiate contract agreements with vendors that provide such services, you need to understand what should be included in a Web hosting infrastructure. This knowledge will help you in three areas:

1. Providing reliability, scalability & good performance
2. Minimizing risks via security, privacy, regulatory compliance and reduction of vulnerability to potential lawsuits
3. Reducing and controlling costs

This guide is meant to help you review upcoming contracts as well as existing services.

Likely audience for this article: Managers, directors and vice presidents of technology, operations or finance at organizations operating large-scale Web sites; Executives supervising technology: CTO, CIO, CFO, COO.

Seven Aspects of Large-Scale Web Hosting

Large-scale Web hosting infrastructure and services can be organized into the following seven areas:

1. Servers & Environments
2. Network & Other Appliances
3. Managed Hosting Services
4. Third-party Provided Services
5. Program Management Office, PMO
6. Account Management
7. Infrastructure & Facilities

Checklist for Review

You can use the following checklist to review your hosting services or a vendor’s proposal.

What to look for

When you review each item below, consider:

• Is this item included in the vendor’s proposal or in the services we are currently receiving? If it is not included, what are the good reasons it isn’t included?
• Is this needed for my organization’s current business requirements? Can we do without it? Is it a “must have” or “nice to have” for present and reasonable future needs?
• What are the alternatives?
• What is the unit price of this item? How does the price scale up as needs grow? How does the price scale down when need for this item decreases?
• What level of fault-tolerance does this item need? i.e. redundancy, standby backups, time to recover

Some of the above review questions may apply only to things and not apply to services and processes.

Servers

Servers may be physical hardware servers and/or virtual servers managed using software such as VMWare, Parallels Virtuozzo or Xen. The services listed below can each run on separate servers or multiple services can run on a server. It is generally better to have servers running only one (or minimum number) of the major services listed below. That reduces complexity and saves expensive staff time saved maintaining, troubleshooting and recovering. Virtualization makes it economical to have multiple virtual servers on the shared physical hardware economize costs.

The following is a list of commonly found services at large-scale Web sites that require servers.

• Web
  • HTTP(S) Content Delivery. E.g. Apache HTTP Server software
  • Streaming Content Delivery
  • Cache. E.g. to run products like Squid Cache, memcached
• Application
  • Content Management software. This is the software that the Editorial and Production teams use to submit, edit, package and manage articles, photos and other Web site content
  • Dynamic Content Assembly. Typically done using Portal Server software, either third-party supplied or in-house developed
  • Data Processing. E.g. workflow engines, jobs/tasks processing servers
  • Middleware
  • Other applications. These are applications that happen to be separate from the main content management system. They could be separate for any number of reasons. E.g. blogs, forums
• Database
  • Relational Databases. E.g. Oracle, MySQL, PostgreSQL
  • Non-Relational Data Stores. E.g. Key-value, “NoSQL” stores

Server Environments

An environment is a self-sufficient set of servers assigned to serve a purpose as described below. Large-scale Web sites typically utilize multiple environments.

• Production
  • This serves the Web sites to the customers and public.
  • Typically has 99.9% or higher uptime guarantee in the Service Level Agreement
• Staging
  • This is the environment where content packages are developed, integrated and previewed by Editorial, Design and Production teams before they are published to the end-users. For example, when working on a major site redesign or relaunch for several months. Since the tech teams are often making changes to the Development Integration and QA environments, they are not suitable for content integration work by the Editorial and Design teams. Staging is used in large-scale Web sites where mutiple Editors, Designers and Production staff are collaboratively creating content packages and new sections. In smaller Web sites or in cases where just one or two Editors are working on a piece of content like an individual article, previewing is done in the Production environment itself with access controls.

• Quality Assurance (QA)
  • The QA engineers perform Functional Testing and Load Testing here. Doing functional testing while a load test is running is sometimes a good idea as it simulates usage closer to live production.
• Development Integration
  • Software product code developed by different engineers is integrated here. There could be continuous integration or nightly builds.
  • This is where developers ensure that their code works with other developers’ code (does not break the build, and does not conflict resulting in undesired functionality)
  • Programmers should ensure that the product works here before handing it off to the QA engineers for testing

In a virtualized system the environments may not be physically separate and may regularly grow and shrink at different times. For example when hosted at a cloud computing provider, the QA environment may scale up during load testing and shut down completely during the hours the QA team is not working.

Network & Other Appliances

These are devices to which various servers are directly or indirectly connected.

• Routers
• Load Balancers
• Firewalls
• Shared Storage (Storage Area Network, SAN; Network Attached Storage, NAS)
• Backup & Restore systems
• Bandwidth (at origin hosting)
  • Bandwidth is not a physical thing, but since like electricity, fuel or cell-phone “minutes”, is something that is metered and paid for monthly, bandwidth usage and charges need to be carefully managed.

Managed Hosting Services

• Systems Administration
  • This typically includes all the management of the physical hardware up to and including the operating system and popular applications that complement the operating system.
• Database Administration Services
• Applications Management Services
  • This typically includes all the administration of the applications that run on top of the operating system.
• Systems Monitoring, Alerting & Reporting
• Web Support Help Desk, 24×7

Third-party Services

• Content Delivery Network, CDN (e.g. Akamai, Limelight, CDNetworks)
  • CDN Network Storage
  • CDN Bandwidth Rates for HTTP and Streaming
• External Monitoring, Alerting & Reporting (e.g. Gomez, Keynote)

Program Management Office, PMO

• Project Management
  • PM people, organization, processes
  • Collaborative project management tools, e.g. JIRA, RallyDev, Mingle
  • Shared documentation management tools, e.g. Wiki
• Change Management Processes & Tools
  • Documentation system
  • Tools for source control, build & deployment
• RASIC Matrix Describing Roles & Responsibilities
• Escalation Flowcharts
• Crisis Management & Emergency Procedures

Account Management

• Customer service
• Relationship management
• Master Services Agreement, MSA
• Statements of Work, SOW
• Service Level Agreement, SLA
  • What to look for in the SLA is the subject of a separate article in this series.
• Billing
  • Monthly bills provided by telecommunications (telco) and hosting companies tend be extremely complex and lengthy. As a result, they are difficult and time-consuming to review.

Infrastructure & Facilities

This item, infrastructure & facilities, is beyond the scope of this article. It includes the buildings, electric power, generators, climate control, physical security and related staffing.

This article is part of a series titled “Guide for the CTO: A compilation of articles on how to lead and manage technologies, projects and people”.

Tweet

1 save

In 2010, Cloud Computing is likely to see increasing adoption. Migrating Web applications from one data center to another is a complex project. To assist you in migrating Web applications from your hosting facilities to cloud hosting solutions like Amazon EC2, Microsoft Azure or RackSpace’s Cloud offerings, I’ve published a set of checklists for migrating Web applications to the Cloud.

These are not meant to be comprehensive step-by-step, ordered project plans with task dependencies. These are checklists in the style of those used in other industries like Aviation and Surgery where complex projects need to be performed. Their goal is get the known tasks covered so that you can spend your energies on any unexpected ones. To learn more about the practice of using checklists in complex projects, I recommend the book Checklist Manifesto by Atul Gawande.

The checklists are published on the RevolutionCloud book Web site at:

www.revolutioncloud.com/2010/01/checklists-migration/

and on the Checklists Wiki Web site at:

www.listswiki.com/wiki/IT_Web_Application_Migration

Tweet

Save

My journalist colleagues at Wired.com published some of my comments related to Amazon S3.¹ Wired also posted another article titled Customers Shrug Off S3 Service Failure. I agree with the views of many of the customers expressed in the article. Don MacAskill, CEO of the popular photo hosting site Smugmug, wrote an understanding post about it.

My entire career working for media companies, I’ve held firm the belief that the uptime, reliability, performance, scalability, performance and security of commercial Web sites is of paramount importance. When sites that I’ve been responsible for have had issues, my colleagues and I have given our personal time and energy to resolution. With my teams, I spend considerable time on proactive measures. I’ve had the honor of working closely with and learning from some who do an excellent job running technology operations.

Experience has taught that things can and sometimes do go wrong. Sometimes calculated risks don’t pan out. Sometimes mistakes cause problems. We are human. We should strive for perfection; we can get close to it, but not fully attain it. We should be prepared for such scenarios. When they happen, we should work diligently and expeditiously on resolution and have frequent and honest communications with stakeholders and customers. Such communications during the incident should include:

Update 2010-Jan-24: This checklist is now maintained on the Checklists Wiki Web site at:

www.listswiki.com/wiki/IT_Incident_Reporting

During-Incident Communication Checklist

• Current status
• What is the full impact?
• Estimated time to resolution
• Any recommended workarounds until resolution, if practical
• Assurance that it is being worked on
  • It often helps to mention who all are working on it and what they are doing

The post-incident communications to stakeholders and customers should include:

Post-Incident Communication Checklist

• Summary
• What happened, how and why it happened?
  • Including full description of all impact
  • Do not blame² third-parties or say things like “beyond our control”. A technology leader takes responsibility equally for both insourced and outsourced products and services.³
• How it was resolved
  • If the resolution is temporary or long-term
• Next steps
• Plan for eliminating or minimizing this and similar incidents from happening again
• Thank all those who helped resolve and the customers for their understanding
• Mention the monetary credits you plan to give as per the Service Level Agreement (SLA)
  • Specify any additional ‘make goods’ or returns you plan to make to the customers above and beyond the credits as per SLA, if appropriate.

Stakeholders and customers here refer to internal customers of the technology operations team (e.g. the concerned folks in editorial, marketing, sales, finance, legal and other departments). External communications to the public Internet should be handled in consultation with legal and public relations.

S3′s outage (or any outage) isn’t to be taken lightly, but I have faith Amazon and their customers will learn from it.

Disclaimers:

• As explained in the terms of use of this site, any opinions expressed on my personal Web site do not reflect those of any employer, past or present. My Web site and I in my personal life neither represent nor speak for any corporation.
• I have no affiliation, financial or otherwise with Amazon.com. I happen to be a user of their products and services, some of which I like and some that I don’t.
• Personal Web sites like this are exempt from the performance requirements of corporate Web sites :-) My personal Web site is for expressing, learning and R&D. It also happens to be hosted on Amazon EC2 and S3.

Tweet

2 saves

1. Silicon Alley Insider and ValleyWag have amusing spins on it. :-) [↩]
2. There may be extreme instances, especially when criminal activity or malicious wrongdoing was the cause where it would be appropriate to blame someone. [↩]
3. It is ok to mention service providers, or describing external events for explaining what happened, but don’t do it in a “it was their fault, not ours” tone. The technology leader should factually describe what happened and take responsibility. [↩]

Update 2010-Jan-24: This checklist is now maintained on the Checklists Wiki Web site at:

www.listswiki.com/wiki/Business_Travel_Checklist

Document Revision 2.2 2001/07/20

Tweet

Save