Category Archives: Management

Articles about leadership, management in a business environment.

3 Dimensions of a Technology Team

Organizing Software Engineering Teams to Balance Products, Partners & Professions

This organizational design for a technology department aims to optimally blend the need for the technology team to be an engine of innovation, a customer-service organization and technically excellent.

It views the staff, roles and responsibilities in three dimensions: products, partnerships, and professions.

img

  1. Products and projects the technology teams work on
  2. Partners, customers, and stakeholders of the technology department
  3. Professions of the various technology staff members

Organization by products focuses on providing great user experiences via small, agile and nimble teams that have ownership of their work.

Organization by partners focuses on being an effective client-service organization to other parts of the company.

Organization by professions provides for learning, best practices and career development of the tech employees.

This article first describes the three models and then suggests ways to bring them together into one integrated organization design.

Organizing by small product focused teams

Product

A product is defined as something that customers use and experience. A product consists of features and functionality.

It refers to a digital entity, a physical good or a combination of the physical and virtual. For example, it could be a communication app that people use via their smartphones and computers. It could be a health device that people wear and access via their smartphones and computers.

The goal of a product is to meet customer’s needs & desires, to solve problems for customers including ones they didn’t know of, and to delight them. Many products aim to help make the world a better place, be it in small or big ways.

Product Team

One way to think of a product team is as a mini-company within the larger organization. A product team has customers, which could be external, internal or both.

A product team develops the product, makes it available to customers (“ships” it), and maintains the product. Maintenance of the product includes new features, major upgrades and innovations.

Roles in a product team

A product team has several roles which are filled by professionals with certain skills, knowledge and interest.

The roles can be narrow in scope, like front-end JavaScript developer or broad, like software engineer. The roles include the professions of engineers, designers, subject matter experts, marketers, and managers. For a longer list of roles in a product team with examples, see the section titled “list of professions/roles” under organization by professions below.

People and Roles

These roles are not necessarily each held by different people. There is no prescribed restriction on who can hold which role or how many roles. Depending on the team’s needs, people’s skills & interests, and budget & resource situation, any permutation of the roles can be held by the people in the team.

A team should make the best decisions in allocation of the roles. The roles held by people are expected to change as the team scales up or down. The team should give importance to checks and balances via separation of duties where required for compliance of regulations.

Team Size

As a minimum requirement, at least three people are required for it to be a product team that encompasses a sufficient set of the roles required. In a few rare cases, the team might be able to start with just two people.

Proximity

For a product team to be successfully implemented, it needs to consider and address evidence-based management science related to human collaboration.

An important recommendation is that the team “resides” together, i.e. works together on a weekly if not daily basis. If the team works out of the same office, then they should work in close proximity. If the team is geographically separated, like the teams at Auttomatic/WordPress, then they should have “virtual proximity”, i.e. be in close contact using video conferencing and other online collaboration tools.

Organizing by Partners, Customers, and Stakeholders

Aligning the technology department with business departments

In this model, each line of business, department and product/service at the company is assigned a technology partner in the technology organization.

Examples of stakeholders in a media company include the heads of advertising, marketing, finance, customer service and editorial.

The technology partner is a senior, widely-respected and influential member of the technology management team who serves as the primary technology leader for that business, department or product/service. Excellent client-relationship skills are a requirement.

The technology partner may also serve as the solutions architect for the client, or may have solutions architects assigned to them.

Technology Partner Role Description

The technology partner is the go-to person for all technology matters for the client department, including for engineering, project, and technology, areas that the technology partner does not directly manage. For example, a large department like Advertising will have projects that could span multiple tech departments.

The responsibilities of the partner include:

  • Assess Current Capability: Evaluate the current technical capabilities of all the technology systems.
  • Define Future Capability: Understand strategic industry trends to plan future technology roadmaps.
  • Technical Lead for product development: Responsible for technology execution, bringing new products to market and optimization of existing offerings.
  • Plan Capacity: Translate roadmaps into future year capacity plans.
  • Collaborate with other tech teams: Work with other product teams to service the needs of the business needs / departments / products where other teams in the technology organization are the owners of systems.
  • Subject Matter Expert: As the primary contact for a business / department / product act as the SME for all technology.

Organizing by Professions

In this model, the technical people are organized by the professions they choose to belong to.

Professions and Roles

This is a sample list of professions and roles gleaned from companies that have products in the digital media space. This list can be easily adapted to other industries, for example, by including the relevant subject matter expert roles. This was compiled between the years of 2007 to 2014 and reflects some prominent roles from then.

Each profession/role is grouped under one of five major categories:

  • Technologist/Maker
  • Designer/Artist
  • Subject Matter Expert/Analyst
  • Marketer/Seller
  • Manager/Admin

List of professions/roles

Technologist/Maker
Software Development Engineer, Server-side “backend” Software Engineer, Client-side “frontend” Software Engineer, Mobile iPhone and Tablet Software Engineer, iOS Software Engineer, Android Software Engineer, Content Management (CMS) Engineer, Testing Engineer, App Release Engineer, App Operations Engineer, DevOps Engineer, Systems Administrator, Database Administrator, Security/Privacy & Compliance Analyst

Designer/Artist
User Experience Designer, Visual Designer

Subject Matter Expert/Analyst
Product Manager, Business Analyst, Journalist, Reporter/Content Author, Editor

Marketer/Seller
Salesperson, Marketer, Customer Service Representative, Internal Advocate, Documentation Writer

Manager/Admin
Project Manager, Finances & Budget Manager, Team & People Manager

Integrating the three dimensions

Now that we have described the three dimensions, let us discuss ways to make it all work together.

Reporting Relationships

Some people believe that “reporting relationships are irrelevant in today’s world”, but social science research has repeatedly proven this to be false.1  Reporting structures and hierarchies are an essential and integral part of any organization of humans.

So rather than pretend that reporting relationships in a team don’t matter, it is important to address them in setting up these teams.

Let us start by acknowledging that human relationships are complex, nuanced and changing, so there is no one formula to organize your teams. Every approach to organizing reporting relationships has its advantages and drawbacks. Here, we present a reasonable but imperfect model.

This model applies to medium to large size organizations, but not to small companies or very-large companies. In a small company of less than 10 people, there are unlikely to be departments by profession where people report into. In very large companies’ each business unit may operate as an independent company with its own departments by profession (e.g. their own finance, HR and engineering).

We also make some assumptions. For example, that it is preferable that each team member has exactly one primary boss who they formally report to.2

Departments by Profession

It is important to define what is expected of a primary boss in your organization so you can best determine how to structure reporting relationships. For example:

  • Does a boss’ role include the long-term development of the employee’s career, existing skills and new skills?
  • Does a boss need to be a subject matter expert in the chosen profession of the employee? Or, is the employee best served by being part of a team of others with the same or similar professions?
  • Does a boss need to objectively and independently look at the employee’s work as serve as a checks and balances relative to the product/project team the employee is presently doing day to day work in?

If your answer to the above questions is yes, then it is likely that setting up the primary reporting relationships by profession is a good solution for you.

What does reporting relationships by profession mean? It means that primary (and formal) reporting relationships are organized by departments of people in similar professions. For example, HR professionals report into the HR department, finance professionals report into the finance department and software engineering professionals report into the software engineering department.

In medium to large sized companies, having the primary reporting relationship by professional departments often has advantages over having direct reporting into the product/project team. In fact, most consulting firms use this model. Some of the benefits include:

  • As new products are created, old products decomissioned and projects start and end, employees don’t need to switch from boss to boss. By having a boss who has seen an employee’s work over multiple products, services and projects, the employee can receive better coaching and development support.
  • A boss who manages a team of employees of the same profession helps share best practices, standards and development opportunities across the organization.
  • Such a boss can also serve as a check and balance for a product/project team’s short-term needs vs. the best long-term interests of the company. The employee has a primary person higher up to consult with outside their current product/project team.

Most medium to large size companies have departments by profession. For example, they have departments for finance, HR, sales, marketing, software engineering, design, etc. This makes good sense for reasons like the above.

Such departments often assign (or even embed) their professionals within other departments and product/project teams. For example, finance (or HR) may assign a finance person or team (or HR person or team) to work with the technology department. This person would likely be assigned full time to the technology department and work in close proximity to a part of it.

The organizational model recommended here views technology as a collection of departments by technical professions. For example, software development, quality assurance and testing, infrastructure engineering, and project management. Each of these technology departments embeds their staff into product/project teams.

The overall technology department also assigns a technology partner to every other department by profession. That’s how these three dimensions come together.

Balancing Professional Departments with Product/Project Teams

In setting up direct reporting relationships by professional departments, it is critically important to not let the departments become siloed. Each employee should be focused on, deeply care about and feel accountable to the product/project they are working on.

When the person’s primary boss is a functional head outside the team they are currently doing day to day work in, both the employee and the boss must ensure the employee is highly committed to their product/project team whether it is in their department or outside it. One way to accomplish this: Each employee can also be accountable to a lead within the product/project team via a dotted line if needed. In some cases, the employee’s boss and their lead within the team can be the same person.

The goal of the functional department is to both support product/project teams and serve as checks and balances, but never to work against them.

Non-Technology Departments

While this article was written for organizing technology departments at companies where a substantial amount of work involves software engineering, the thinking here maight be adapatable to other types of departments. If you have thoughts on how to do that or have other approaches to organizing teams in organizations, please do share them

  1. Hierarchy is Good. Hierarchy is Essential. And Less Isn’t Always Better by Professor Bob Sutton at Stanford University https://www.linkedin.com/today/post/article/20140112221140-15893932-hierarchy-is-good-hierarchy-is-essential-and-less-isn-t-always-better  []
  2. If someone has scientific evidence supporting this, please let me know.  []

3 Roles of a CTO: Culture. Technology. Operations.

This is a guide for CTOs, VPs of Software Engineering and other technology managers responsible for a software engineering organization. The purpose of this checklist is to help the CTO cover the areas of culture, technology and operations in their teams. It is presented in the form of a memo to direct reports.

cto-culture-technology-operations


Dear Tech Management Team Colleagues,

For those of you who have weekly 1:1 meetings with me, I’d like to suggest this template as a guide for our regular discussions. I value your experience, so please feel free to suggest making this format even better. I’d like us to cover three major areas on a regular basis.

  1. Culture
  2. Technology
  3. Operations

Each of these three major areas is further divided into three sub-categories containing a list of items to consider reviewing.

The first time you see this list, it may seem too long to review in a 30 minute meeting. This is a guideline to structure our conversations. You are not expected to discuss each one of these at every meeting. This checklist will help us review things that are relevant at the time. Managers have successfully used this checklist to review pertinent items in less than 30 minutes.

Tip: Here is one way to effectively use this. Let us both spend 5 to 10 minutes to read this checklist in advance of each of our regular 1:1 meetings. We can even use the first 5 minutes of our meeting to read it. Then we will both have a good idea of which items are relevant for the next discussion from our perspectives.

Culture “people, behaviors & teamwork”

Relationships

  1. How well are your team members collaborating with each other?
  2. …with their colleagues in other tech teams?
  3. …with their stakeholders, customers and executives?
  4. Are there any tensions that I need to be aware of?
  5. Advocacy: What should we do to be better understood, respected, and appreciated by our stakeholders, customers and executives and vice versa?
  6. Anything in this area that you are waiting on or need from me?
  7. Is there anything non-work-related that you’d like to share?

Retention

  1. Are the people in your teams happy with their work? How is team morale?
  2. Is the work intellectually challenging?
  3. Are they learning new things and getting better at existing skills?
  4. Do they feel they are making a positive impact?
  5. Are we taking good care of them?
  6. Are we proactively providing feedback, coaching, and training?
  7. Is anyone considering leaving that you know of?
  8. Has anyone given notice?
  9. Is there anything related to retention that you are waiting on or need from me?

Recruiting

  1. Are you feeling a staffing shortage this week?
  2. Are we thinking ahead and planning for capacity, skills and having some slack for flexibility?
  3. How many open positions do you have in your team this week? How long have they been open?
  4. What are you doing for recruiting?
  5. Is there anything related to recruiting that you are waiting on or need from me?

Technology “engineering, infrastructure & innovation”

Architecture

  1. What new technologies, platforms, products and APIs are we evaluating?
  2. …implementing?
  3. …decommissioning?
  4. …consolidating?
  5. …releasing as open source or making public?

Integration

  1. What are we doing to support integrations across teams?
  2. Are you facing any challenges integrations across teams?

DevOps

  1. In what areas are we implementing temporary hacks?
  2. How are your apps and platforms doing with respect to their goals in Performance & Scalability?
  3. …Reliability?
  4. …Security?
  5. …Test Coverage?
  6. …Technical Debt?
  7. …On calls and P1s?
  8. How are the integrations, process and relationships between the development, infrastructure and security folks?

Operations “projects, sustainability & recycling”

Work

  1. Are there any projects at risk?
  2. Are there any changes to a) due dates, and/or b) delivery dates?
  3. What did we a) accomplish, and b) work on over the past week?
  4. What do we plan to do over the next week?
  5. What projects/work can we decommission?
  6. What was your budget forecast? How are we doing with respect to it? Any budget issues?
  7. Did we recently say “no” to a stakeholder or executive’s project request (or say something would be very hard to do) that I should know about?
  8. … any that we said “yes” to that I should know about? :-)
  9. Did we give any estimates that I should know about?
  10. What can I do to help? What do you need from me?

Learn

  1. What did we learn from the past week?
  2. Are we sharing these learnings with others who’d benefit from them?
  3. Did we do any retrospectives? What changes are we making based on retrospectives?
  4. Are there any process changes that you recommend? … for both outside and inside of your teams.
  5. How can I help?

Challenges

  1. What issues are we facing now or are likely to face in the future?
  2. What prioritization problems are we facing?
  3. What do you suggest are our countermeasures to address those issues?
  4. How can I and/or others help and support you or remove obstacles from your path?

Mixing it up

To prevent our weekly discussions from feeling too structured and getting stale, I suggest mixing it up a bit. Let us try this format for 3 out of every 4 of our regular 1:1s and keep 1 meeting free-form.

We can also break monotony by switching the locations of these meetings and having some of these discussions walking about.

Why discuss this in a meeting and not ask for this information in a weekly status report?

… because no one likes to write a status report, but everyone likes to talk :-)

Let us take a test and learn approach with this and adjust as we go along.

Thank you in advance for your help with this.

3-5-7 Meeting Format for Weekly Staff Meetings

If you are the manager of a team of people at your job, here is a format we suggest for running your staff meetings. We call it the 3-5-7 format because of its convention of giving 3 to 5 minutes per person to answer 7 questions. This system assumes that you have fewer than ten direct reports so that you can complete such a staff meeting in under one hour.

The purpose of a staff meeting need not be to get status reports. If you have excellent collaboration tools at work where statuses, issues and risks are already documented, that’s preferable. Some companies like Automattic (WordPress) make great use of internal blogs for communication. However, face-to-face meetings are continue to be useful because our brains have evolved being wired for being most effective in face-to-face conversations for several things.

An in-person (or via video conference) discussion structured around these questions is likely to be effective in finding solutions, building a more collaborative team and keeping everyone on the same page.

Here are the seven questions we suggest you request each attendee to come prepared to answer.

  1. What did we (you and the team reporting in to you) do over the past week?
  2. What did you learn over the past week?
  3. What do we (you and the team reporting in to you) plan to do over the next week?
  4. What issues are we (you and the team reporting in to you) facing now or are likely to face in the future?
  5. What do you suggest are our countermeasures to address those issues?
  6. What do you need help with from the rest of us in this meeting?
  7. Is there anything non-work-related that you’d like to share?

Each person may answer the seven questions the order of their choice and may also combine the answers to multiple questions. The only requirement is that all seven areas be answered in a focused, efficient, and effective narrative lasting between three to five minutes.

Some of this advice is based on management experiences shared by Don Kiefer in an operations management class he teaches at MIT’s Sloan School of Business.

Suggested Template For Requesting a Meeting

Every time someone calls a meeting, they should consider using this simple template.

[ meeting-invitation-template begins ]

The desired outcome of this meeting is:

  • e.g. Come to agreement on solution for issue X
  • e.g. Make a decision about Y.
  • e.g. Share announcements about topic Z.
  • e.g. Continue to grow a good working relationship with each other by socializing in person.

Note: Explain what this meeting is meant to accomplish, instead of providing a description of the meeting. Focus on the desired result of the meeting. A meeting should accomplish one or more of three things:

  1. Solve problem(s)
  2. Make decision(s)
  3. Share knowledgeand agree to act on it and/or make it a practice
    • Knowledge, as in: data –leads-to–> information –leads-to–> knowledge –leads-to–> practice

You should come to this meeting because:

  • e.g. You are likely to have input into potential solutions for issue X
  • e.g. You are one of the folks who has a viewpoint on what decision to make regarding Y.
  • e.g. It would benefit you from hearing the announcements in this meeting.
  • e.g. This is your opportunity to ask questions about topic Z.

Note: Give the attendees at least one good reason to attend. Sometimes attendees have no idea why they are invited to this meeting. Don’t be seen as a waster of others’ time.

The guidelines for participating in this meeting are:

  • e.g. Please come prepared having read the document about ChaosMonkey.
  • e.g. Laptops & mobile communication devices are considered contraband during this meeting. If it is critical for you to have a computer during this meeting, bring a desktop computer :-)

Note: Set the expectations of the participations.

[ meeting-invitation-template ends ]

Further Reading & Thoughts:

Templates for Replying to Meeting Requests & Polite Ways to Decline Meetings

As much as possible, we should only attend meetings where we are active participants, not mere attendees with nothing to contribute to the defined outcome of the meeting. There are some exceptions to this like training sessions, educational presentations or others where the purpose for attendees is to learn something.

When I receive a meeting request, I often reply with the following text. In fact, I have it saved as a TextExpander macro as a shortcut to typing it.

May I please request the following information in advance of this meeting? It will enable me to prepare, participate and be productive in the meeting.

  1. What decisions need to be made at this meeting (if any)?
  2. What problems need to be solved at this meeting (if any)?
  3. What knowledge is planned to be shared at this meeting? (topics or documents)

Thank you,

Time Management Tip: When you receive an invite for a meeting at work where you believe you may not add much value, reply to the invite with a polite message like:

Thank you for inviting me to this meeting. It seems from the subject, agenda and attendees list that I’m not a required participant for this meeting. If I’m mistaken and my presence is required in this meeting, please accept my apologies and let me know that I should attend.

This is preferable to ignoring the meeting invite or declining without comment that may come across as rude. I have this text also saved as a TextExpander macro.

Discussion about declining meetings: https://plus.google.com/107443707510532643538/posts/inUkYy1Ufg7

When to have and when not to schedule meetings

Companies should, by default, avoid scheduling meetings that start before 10am or end after 5pm. If an employee comes to the office at 8am on some days, it is often to use the two hours of the morning before meetings to catch up and/or get a head start on the day. Meetings that start before 10am are often harmful overall since they put the attendees in reactive catch up mode for the rest of the day. Similarly, meetings that go on beyond 5pm (or worse, start after 5pm) take away valuable time from employees that they use to absorb information and events of the day, catch up with replying to email and get ready for the next work day.

i.e. Companies should consider any time outside the 10am to 5pm window to be not available for meetings and definitely not any weekly recurring meetings.

Preferably, employees who are ‘makers’ should have one 4-hour continuous block of time each day when they are free from meetings. (‘Makers’ differentiated from ‘Managers’)

50/25 Meeting Format

If you manage a team, value your team members time and want to improve productivity at your workplace with a simple change, consider implementing the 50/25 Meeting Recommendation that some companies are embracing. You can communicate something like the following to your team:

Dear Colleagues,

We deeply value your time, your productivity and your comfort at the workplace. As a part of our initiative to make your workday more productive, less hectic and better manageable, we recommend a 50/25 meeting format. It is simple concept: As much as possible, let us take all our meetings that are 1-hour long and shorten them to 50 minutes. For our meetings that are half-hour long, let us limit them to 25 minutes.

You will find that a 50 minute meeting will accomplish no less than a 60 minute meeting did and a 25 minute meeting will be as productive as a 30 minute one was. In fact, by having clear 50 minute and 25 minute deadlines, our meetings are likely to be better focused, on topic and more attentive. (For example: Since you will have time after the meeting to check email, there is likely to be less temptation to check emails during the meeting itself.)

The extra 10 and 5 minutes will give you valuable time back that could be used for many useful activities: Getting in the frame of mind for the next meeting or task; checking your messages to see if there is something urgent that needs your attention; or simply taking a bio break.

Please note that this not a mandate, but a recommendation. We realize that you may not be able to do this for every meeting. What we ask is that you consider doing this for meetings that you organize or can influence. As a result, we will make our great work culture even better, less stressful and even fun.

Further Reading & Thoughts:

  • NYTimes article about Larry Page, Google’s founder and new CEO instituting the same 50/25 meeting recommendation at Google:
  • http://www.nytimes.com/2011/11/10/technology/googles-chief-works-to-trim-a-bloated-ship.html?pagewanted=all
  • If a meeting accomplishes all its goals in even less than the 50 or 25 minutes, please, by all means end the meeting even sooner.
  • We suggest that you do book the full hour or half hour in the calendar even as you implement the above so that others don’t schedule over the “10 minutes left over” in your calendar.

Thank you for considering this,

[Signature]

A discussion about this 50/25 Meeting Format: https://plus.google.com/107443707510532643538/posts/AtYgnmbhtqc

Using Laptops or Smartphones in Meetings

You might think that no one is noticing when you are using your phone if you try to hide it below the table surface, but you’d be mistaken. It is like picking your nose: Your being oblivious of others doesn’t make you invisible to them. Photo Credit: Brad Kagawa

Using smartphones — or worse, laptops — during in-person meetings diminishes productivity, is disrespectful to others and decreases your brainpower. Yes, scientific evidence indicates that multitasking makes people less intelligent.1

When you are  doing something unrelated on your phone or laptop in inappropriate situations (e.g. during business meetings), you lose out because you become oblivious to the environment, people, and subtleties around you.

However, there are a few situations where it makes good sense to use a laptop or smartphone during in-person meetings.

  • When you are the designated note-taker for this meeting.
    • Taking notes on a computer or smartphone saves time, and is more accurate than taking paper notes and digitizing them later.
    • Notes on paper can’t be searched easily, pile up as clutter and are less environmentally-friendly.
    • It is more secure than taking notes on paper that can be forgotten and read by others who should not have access to the information.
    • Meeting notes and action items can be automatically saved in real time and shared quickly after the meeting.
    • There should be only one person taking notes during a meeting. If it is a negotiation between two opposing sides, then there should be no more than one note-taker per side.
  • When you need to quickly look something up that is relevant to the discussion and is either necessary or helpful to the meeting in progress.
  • Entering action items that come up during the meeting into your to-do-list so that you can focus on the meeting. This is useful for people who use the GTD system with a tool like OmniFocus.
  • Quickly and discreetly asking a question, or sharing an opinion or information over instant message without disturbing others in the meeting.
  • The distractions on the device could be managed if the user is disciplined and remains focussed on the meeting, perhaps even using the laptop to participate more actively in the meeting. After all, even a person using pen and paper can be distracted doodling or daydreaming.
  • This is the digital age.

Tip: When you bring a laptop to a group meeting or one-on-one meeting, each time respectfully explain to the others beforehand that you will use the laptop for taking notes and recording action items in your to do list only. Inform them that you will be focusing attention on the discussion and that the laptop is simply your digital notepad.

There are also many reasons against using laptops or smartphones during meetings:

  • It comes across as disrespectful to some other meeting attendees, especially those with traditional styles of working.
  • The laptop screen creates a “wall” between you and the people sitting across you.
  • The laptop does make it easy to get distracted into reading your email or other online activities. (A tablet like the iPad that lies flat on the table like a writing pad does not have this problem.)

Tip: At the start of your meeting, announce that if anyone needs to use their phone or laptop, they should step out of the room, use their device outside and return when done. This way, attendees have the freedom and won’t feel constrained.

In most situations, the drawbacks of using a laptop or smartphone during an in-person meeting far outweigh the benefits.

Tip: Provide a mobile phone charging area in your meeting rooms to encourage attendees to put away their mobile phones and participate.

What do you think? Here is link to a related discussion about using laptops, smartphones and other communications devices in meetings.

(Updated: 2014 July 26)

  1. The High Cost of Multitasking: http://blog.fuze.com/the-high-cost-of-multitasking-infographic/  []

What I Learned During the Hacking Attacks of August 28, 2013

One of the most important lessons that was reinforced to me during the Aug 28th hacking attacks on Melbourne IT, the domain registrar for the Web sites of The New York Times, Twitter and many others was the importance of human relationships, personal networking and real-time communications during an emergency situation.

In this article, I’ll mention some of the human collaboration and technical aspects of the lessons that were reinforced. Minimizing the chances of having outages and getting hacked is beyond the scope of this article. This post is about resilience (i.e. dealing with and recovering) not prevention.

Note: To learn about the incident itself, see the section ‘Hacking Incident: Information & Misinformation‘ below.

The Power of Human Collaboration

I witnessed the power of multi-participant video conferencing that is commonplace these days thanks to Google Hangouts. Back in 2009, I wrote an article about the benefits of using real-time textual group chat during incident management and emergency situations. All the lessons mentioned in that article were yet again reinforced to me during this incident. I suggest reading that article to review the real-time communications recommendations in it.

Matthew Prince, CEO of CloudFlare summed it up well at the end of his blog post:

We spend our time building technical networks, but it’s comforting to know that the human network is effective as well.

My colleague Jon Oden Tweeted:

Well that was a fun day… lots of fantastic people in the tech community is a silver lining

I felt so honored, humbled, and happy when my friends and their friends at some of the best known Internet companies went above and beyond to help and spent most of their day in a Google Hangout video conference, helping restore public access to The New York Times’ Web site and other sites for the public on the Internet. They did this because they are good, helpful and tech experts. They took time out of their own jobs to help because they care about a common good cause and fighting against malevolence.

It was inspiring to watch so many brilliant tech people and leaders from multiple companies, many of who met on the video call for the first time, collaborated so well together and overcame the problems together. The combination of multi-participant video conferencing and text chat in Google Hangouts made it feel like we were all working in the same physical room together.

I’d like to thank John RobertsMatthew Prince and the engineers at CloudFlare; David Ulevitch and several engineers at OpenDNS; Mandar Gokhale, John W. McCarthy and others at GoDaddy;  The technical infrastructure engineering team at Google; Bob Lord at Twitter;  Sara Boddy and team at Demand Media; and many others who helped yesterday. You all showed amazing teamwork on the video conference yesterday.

Technical Lessons

If you run a high profile Web site, it is critically important that your Disaster Recovery & Business Continuity Plan includes dealing with an emergency when your primary systems  are unavailable, despite all the safeguards and backups you have in place. The domain registration hijack was one such example. A company can only have one registrar that holds their domain name registration information. There is no concept of a backup or failover registrar for a domain. To deal with this single point of failure, you need a backup Web presence on a separate domain.

Backup Web site on Separate Domain

You should maintain a backup Web site that:

  • Has a different domain name. For example, if your site is at example.com, your backup domain could be example.net.
  • Is registered with a different domain name registrar than your primary one. For example, if your primary registrar is MarkMonitor for example.com, then use Network Solutions for the backup domain example.net.
  • Uses DNS service hosted somewhere else. For example, if you run and host your own DNS servers for example.com, use an outsourced DNS hosting service like CloudFlare for example.net.
  • Uses a different Content Delivery Network (CDN). For example, if you use Akamai for example.com, use CloudFlare for example.net. You must have a CDN on your backup Web site so that it can handle your traffic.
  • Is hosted somewhere other than where your primary site is hosted and is implemented using a different (much simpler) technology platform that is highly likely to not have the same vulnerabilities.
  • Can feed your Mobile apps. Your mobile applications should be designed to be aware of this backup Web site and should be able to switch to it (automatically or via manual intervention) for retrieving content.
  • Does not share administrative access, logins and passwords with the primary site.
  • Preferably, this backup domain example.net should be managed by a separate team. This has two benefits: 1. In the situation of the primary team itself being compromised (sysadmin accounts hacked or a rogue employee) 2. The separate team can work on activating the backup site while the primary team focuses on restoring service of the primary site.

What you use your backup Web site for is up to you. If your primary Web site is a news and media Web site, you could use the backup Web site to publish content during an emergency impacting the primary Web site. If it is impractical for the backup Web site to provide similar (or a subset of) functionality, you could use it for providing status updates and communicating what is going on.

When the backup domain is not needed (which will hopefully be the case 99.9% or more of the time), it could simply be used for providing systems status, explaining it is in place for emergencies and linking to the primary Web site.

Access to a Reliable Public DNS

For end users (i.e. people on the public Internet visiting Web sites), I highly recommend considering using OpenDNS (instructions here) and/or Google Public DNS (instructions here) either as primary or as backup DNS providers.

Other

There are also other lessons, not specific to this incident, both process-related and technical that I’ll write about in a separate article.

Hacking Incident: Information & Misinformation

Wired Magazine’s article titled ‘Syrian Electronic Army’ Takes Down The New York Times correctly explained that:

There’s no evidence that the Times’ internal systems were compromised. Instead, the attackers got control of the NYTimes.com domain name this afternoon through the paper’s domain name registrar, Melbourne IT…

Melbourne IT is the company that manages the domain name registration for The New York Times, Twitter and many other well-known sites. On Aug 28, it was Melbourne IT’s computer systems that were hacked which enabled the perpetrators to hijack The NY Times, Twitter and other companies’ domain names.

Surprisingly, the article on Ars Technica (an otherwise well-respected technology publication) was inaccurate and misleading. It incorrectly stated (quote) “The Times DNS records have been altered, and now point to an Australian hosting company, Melbourne IT.” (end quote) That would lead readers to incorrectly believe that the hackers redirected nytimes.com to DNS or fake Web sites hosted at Melbourne IT.

I’m surprised that the Ars Technica staff did not do their research before writing that post. Melbourne IT is (and has been for years) the official domain name registrar of The New York Times. In addition to being a domain name registrar, Melbourne IT also happens to be a hosting company, but that had nothing to do with the incident. As far as I know, none of the sites impacted that day used Melbourne IT for anything other than domain registration.

What Ars Technica should have said (like their sister publication Wired did) is that the perpetrators hijacked the nytimes.com and other Web sites by hacking Melbourne IT, the company that holds their domain name registration.

The Ars article is also misleading in its claim that The nytimes.com DNS records were altered. It was the domain registration records at Melbourne IT that were altered that then pointed to a whole different set of DNS servers outside of The NY Times’ control.

The Ars Technica article also pointed readers to NY Times’ URL by one of its IP addresses, which was also a mistake. If the Ars Technica folks had tested it themselves, they’d have realized that pointing people to the URL by IP was not the recommended way to access nytimes.com during the incident. Clicking on links on the IP page leads back to www.nytimes.com by name. They should have instead pointed readers to the alternate news.nytco.com URL recommended by The New York Times’ staff, which is what the Wired article did. They could have also suggested other good solutions like switching to using OpenDNS. In fact, I’d have expected a technical publication of Ars Technica’s good reputation to have published a step-by-step guide on switching to OpenDNS, which they wrote about back in 2006.