Web Site of Rajiv Pant

CAREER-CLEAR version 2.0 2010-Aug-26

CAREER-CLEAR is a system for doing fair, consistent and constructive employee performance evaluations and determining employee rank, title and compensation. It is meant to be used by supervisors to identify areas for improvement for their employees and to guide their career growth.

Employees are scored in a total of 5 categories. Upto 10 points can be earned in each category for a total of upto 50 points. The final score is then multiplied by a factor of 2 to give a standard scale of 0 to 100. Using a normalized 100 point scale allows it to remain consistent (by adjusting the factor) even as companies add/remove categories and items.

The resulting numerical score can be mapped to the employee’s level of seniority or rank for title and/or compensation. For example, a score of 81-100 could map to director/VP levels; 61-80 manager; 41-60 engineer/contributor; 21-40 junior level/apprentice. Since different departments — for example, software engineering and quality assurance testing — may have different pay scales, this score maps directly to rank/title, and those are mapped to salaries corresponding to the departments’ market rates.

The first four categories are described below. The fifth category is defined as discretionary/user-defined. CAREER-CLEAR is designed to be used in the real world, in a diversity of organizations and on a regular basis. The system won’t suceed if it is too rigid. On the other hand, the system must meet its goals of being fair, consistent and constructive for all employees. To accomodate and balance these goals, 20% of the criteria is meant to be user-defined at descretion of the manager within the fair, consistent and constructive guidelines.

It is inspired by systems described to be in use at Microsoft, Construx, FogCreek (Joel on Software) and Conde Nast Digital Technology. The latter was developed by Bobby Chowdhury, Brian Murphy, Janet Kasdan and Rajiv Pant.

The 5 categories are: Caliber, Leadership, Expertise, Role and Discretionary.

Caliber

This section measures the talent of the employee in general (non-technical) areas.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Ownership – Has identifiable long-term ownership of projects. This is a measure of the criticality, complexity and / or number of projects the employee has ownership in.
2. Responsibility – Is consistently reliable in terms of deliverables and time.
3. Communication – Communicates effectively with peers and other colleagues. Listens to and understands others’ viewpoints, challenges, needs and desires.
4. Consistency – Is approachable, predictable, receptive and consistently applies good judgment in all interpersonal interactions in the work place.
5. Innovative – Innovates and stays abreast of emerging technologies and finds ways to incorporate those technologies into systems.

Leadership

This section evaluates the positive influence the employee has on others.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Teacher, Coach & Motivator – Mentors others, makes great use of all information sharing tools available and is an active presenter. Rallies the troops and improves morale.
2. Enabler – Empowers and enables others to succeed.
3. Exemplary – Leads by example and goes above and beyond the ‘requirements’.
4. Maturity & Humility – Embraces others’ solutions, even when incompatible with one’s own. Incorporates feedback from others to find the best solutions.
5. Connector – Has familiarity with the ecosystem beyond one’s own projects. Functions as a hub which others are drawn to for a quick answer or a quick redirect towards an answer.

Expertise

This section quantifies the skills and experience of the employee related to the job function.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Fundamentals – Understands of the core technical concepts aligned with the given job function. This may include data structures & algorithms, testing, networking, etc.
2. Breadth of Expertise – Is a subject matter expert and go-to person for many areas of technology.
3. Pragmatic – Has a demonstrated ability to identify the best solution to balance what’s most theoretically ideal against what might be the most practical due to concerns about security, scalability, time to market pressures and cost.
4. Automator – Consistently works to drive improvement in processes and systems.
5. “Boy/Girl Scout Rule” – Leaves code and systems better off than they found them.

Role

This section enumerates the employee’s role and areas of contribution within the organization and beyond.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. Strategic – Provides sound vision for broad, long-term goals.
2. Tactical – Oversees many projects or activities that move the organization towards strategic goals.
3. Operational – Steers day-to-day processes that achieve the tactical goals.
4. Executional – Implements repetitive tasks that make up the operational processes. A measure of quantity and more importantly, quality of work produced.
5. Industry Recognition – Is recognized externally as a leading technologist through contributions to open source projects, blogging, writing books, participating in technical committees, speaking at conferences, etc.

Discretionary

Please be sure to adhere to the goals of being fair, consistent and constructive for all employees in using this discretionary section. This category is not meant to be used to justify favoritism nor meant to be arbitrary. Good descretion comes from rational, reasonable and relevant criteria. Place items here that are not already covered in other categories and are important to your organization. A good rule of thumb is that you must be able to justify any criteria you apply here.

Scoring: Above Average=2, Average=1, Below Average=0. Add the score for each of the heuristics. Max Score=10 points.

1. discretionary / user-defined item 1
2. discretionary / user-defined item 2
3. discretionary / user-defined item 3
4. discretionary / user-defined item 4
5. discretionary / user-defined item 5

Tweet

Save

If you host and operate large-scale Web sites, or negotiate contract agreements with vendors that provide such services, you need to understand what should be included in a Web hosting infrastructure. This knowledge will help you in three areas:

1. Providing reliability, scalability & good performance
2. Minimizing risks via security, privacy, regulatory compliance and reduction of vulnerability to potential lawsuits
3. Reducing and controlling costs

This guide is meant to help you review upcoming contracts as well as existing services.

Likely audience for this article: Managers, directors and vice presidents of technology, operations or finance at organizations operating large-scale Web sites; Executives supervising technology: CTO, CIO, CFO, COO.

Seven Aspects of Large-Scale Web Hosting

Large-scale Web hosting infrastructure and services can be organized into the following seven areas:

1. Servers & Environments
2. Network & Other Appliances
3. Managed Hosting Services
4. Third-party Provided Services
5. Program Management Office, PMO
6. Account Management
7. Infrastructure & Facilities

Checklist for Review

You can use the following checklist to review your hosting services or a vendor’s proposal.

What to look for

When you review each item below, consider:

• Is this item included in the vendor’s proposal or in the services we are currently receiving? If it is not included, what are the good reasons it isn’t included?
• Is this needed for my organization’s current business requirements? Can we do without it? Is it a “must have” or “nice to have” for present and reasonable future needs?
• What are the alternatives?
• What is the unit price of this item? How does the price scale up as needs grow? How does the price scale down when need for this item decreases?
• What level of fault-tolerance does this item need? i.e. redundancy, standby backups, time to recover

Some of the above review questions may apply only to things and not apply to services and processes.

Servers

Servers may be physical hardware servers and/or virtual servers managed using software such as VMWare, Parallels Virtuozzo or Xen. The services listed below can each run on separate servers or multiple services can run on a server. It is generally better to have servers running only one (or minimum number) of the major services listed below. That reduces complexity and saves expensive staff time saved maintaining, troubleshooting and recovering. Virtualization makes it economical to have multiple virtual servers on the shared physical hardware economize costs.

The following is a list of commonly found services at large-scale Web sites that require servers.

• Web
  • HTTP(S) Content Delivery. E.g. Apache HTTP Server software
  • Streaming Content Delivery
  • Cache. E.g. to run products like Squid Cache, memcached
• Application
  • Content Management software. This is the software that the Editorial and Production teams use to submit, edit, package and manage articles, photos and other Web site content
  • Dynamic Content Assembly. Typically done using Portal Server software, either third-party supplied or in-house developed
  • Data Processing. E.g. workflow engines, jobs/tasks processing servers
  • Middleware
  • Other applications. These are applications that happen to be separate from the main content management system. They could be separate for any number of reasons. E.g. blogs, forums
• Database
  • Relational Databases. E.g. Oracle, MySQL, PostgreSQL
  • Non-Relational Data Stores. E.g. Key-value, “NoSQL” stores

Server Environments

An environment is a self-sufficient set of servers assigned to serve a purpose as described below. Large-scale Web sites typically utilize multiple environments.

• Production
  • This serves the Web sites to the customers and public.
  • Typically has 99.9% or higher uptime guarantee in the Service Level Agreement
• Staging
  • This is the environment where content packages are developed, integrated and previewed by Editorial, Design and Production teams before they are published to the end-users. For example, when working on a major site redesign or relaunch for several months. Since the tech teams are often making changes to the Development Integration and QA environments, they are not suitable for content integration work by the Editorial and Design teams. Staging is used in large-scale Web sites where mutiple Editors, Designers and Production staff are collaboratively creating content packages and new sections. In smaller Web sites or in cases where just one or two Editors are working on a piece of content like an individual article, previewing is done in the Production environment itself with access controls.

• Quality Assurance (QA)
  • The QA engineers perform Functional Testing and Load Testing here. Doing functional testing while a load test is running is sometimes a good idea as it simulates usage closer to live production.
• Development Integration
  • Software product code developed by different engineers is integrated here. There could be continuous integration or nightly builds.
  • This is where developers ensure that their code works with other developers’ code (does not break the build, and does not conflict resulting in undesired functionality)
  • Programmers should ensure that the product works here before handing it off to the QA engineers for testing

In a virtualized system the environments may not be physically separate and may regularly grow and shrink at different times. For example when hosted at a cloud computing provider, the QA environment may scale up during load testing and shut down completely during the hours the QA team is not working.

Network & Other Appliances

These are devices to which various servers are directly or indirectly connected.

• Routers
• Load Balancers
• Firewalls
• Shared Storage (Storage Area Network, SAN; Network Attached Storage, NAS)
• Backup & Restore systems
• Bandwidth (at origin hosting)
  • Bandwidth is not a physical thing, but since like electricity, fuel or cell-phone “minutes”, is something that is metered and paid for monthly, bandwidth usage and charges need to be carefully managed.

Managed Hosting Services

• Systems Administration
  • This typically includes all the management of the physical hardware up to and including the operating system and popular applications that complement the operating system.
• Database Administration Services
• Applications Management Services
  • This typically includes all the administration of the applications that run on top of the operating system.
• Systems Monitoring, Alerting & Reporting
• Web Support Help Desk, 24×7

Third-party Services

• Content Delivery Network, CDN (e.g. Akamai, Limelight, CDNetworks)
  • CDN Network Storage
  • CDN Bandwidth Rates for HTTP and Streaming
• External Monitoring, Alerting & Reporting (e.g. Gomez, Keynote)

Program Management Office, PMO

• Project Management
  • PM people, organization, processes
  • Collaborative project management tools, e.g. JIRA, RallyDev, Mingle
  • Shared documentation management tools, e.g. Wiki
• Change Management Processes & Tools
  • Documentation system
  • Tools for source control, build & deployment
• RASIC Matrix Describing Roles & Responsibilities
• Escalation Flowcharts
• Crisis Management & Emergency Procedures

Account Management

• Customer service
• Relationship management
• Master Services Agreement, MSA
• Statements of Work, SOW
• Service Level Agreement, SLA
  • What to look for in the SLA is the subject of a separate article in this series.
• Billing
  • Monthly bills provided by telecommunications (telco) and hosting companies tend be extremely complex and lengthy. As a result, they are difficult and time-consuming to review.

Infrastructure & Facilities

This item, infrastructure & facilities, is beyond the scope of this article. It includes the buildings, electric power, generators, climate control, physical security and related staffing.

This article is part of a series titled “Guide for the CTO: A compilation of articles on how to lead and manage technologies, projects and people”.

Tweet

1 save

In 2010, Cloud Computing is likely to see increasing adoption. Migrating Web applications from one data center to another is a complex project. To assist you in migrating Web applications from your hosting facilities to cloud hosting solutions like Amazon EC2, Microsoft Azure or RackSpace’s Cloud offerings, I’ve published a set of checklists for migrating Web applications to the Cloud.

These are not meant to be comprehensive step-by-step, ordered project plans with task dependencies. These are checklists in the style of those used in other industries like Aviation and Surgery where complex projects need to be performed. Their goal is get the known tasks covered so that you can spend your energies on any unexpected ones. To learn more about the practice of using checklists in complex projects, I recommend the book Checklist Manifesto by Atul Gawande.

The checklists are published on the RevolutionCloud book Web site at:

www.revolutioncloud.com/2010/01/checklists-migration/

and on the Checklists Wiki Web site at:

www.listswiki.com/wiki/IT_Web_Application_Migration

Tweet

Save

Problem to solve:

When a team of engineers is dealing with a real time incident, such as a system outage or troubleshooting a problem, having good communications is critically important. The appropriate communications tool can make a world of difference in dealing with the issue and learning from it afterwards. As important as engineering is, lack of good communications is what often gets tech teams in trouble.

Improvements that can be make:
Enable real-time communication in certain collaborative tasks (described below), reduce unnecessary email traffic and clutter, enable people to to focus better on their tasks, reduce time wasted in bringing each other up to speed

Description and solution:

When multiple people are working together in real-time on a near term collaborative task, such as:

• Troubleshooting
• Build and deployment
• Web application migration
• Upgrade or maintenance
• QA testing

Many companies use a phone conference and/or email to assist in real-time while the collaborative activity above is ongoing.

Since Email is not instantaneous and real-time the way a group chat application is, and since email is not a suitable medium for quick questions, and quick one-line  responses, some companies also use a real-time group chat tool like IRC to enable and facilitate real-time conversation.

Benefits of using IRC or a real-time group chat tool are:

• Tech managers, project managers, crisis managers and new tech people joining the effort can quickly catch up with what has been going on (in any level of detail they want) by reading the IRC history transcript so far. This is a much faster and efficient way than using email or pulling someone away to talk in person asking what has been going on. (If email were to be used instead of IRC, a new person joining in would have missed the previous emails on the topic.)
• When an engineer working on such a collaborative task steps away for a while and comes back, they can quickly catch up on what transpired while they were away by reading the IRC history transcript.
• Email is not cluttered by short back and forth messages with lots of text to read and filter
• The IRC transcript can be used for the incident retrospective (“post-mortem”).
• Unlike a phone-only conference, the IRC transcript can be read and analyzed to learn lessons from this incident. For example:
  • Analyze what problems the team ran into
  • Analyze what worked and what didn’t
  • Analyze how well people collaborated and communicated
  • Timelines of events

I can personally attest to the above benefits. Over the past 15+ years, my development and operations teams in different companies have regularly used IRC to great advantage.

Tools like Wikis and blogs are great for collaboration, documentation and sharing information on projects. An group chat like IRC is an indispensable tool for real-time collaboration.

Tweet

Save

If you manage technology for a company that has a large Web presence, it is likely that a large percentage of your total technology costs is spent on the Web hosting environment, including the Content Delivery Network (CDN, e.g. Akamai). In this article, we discuss some ways to manage these costs.

Before we discuss how to optimize your architecture and applications to have economical and the optimally low hosting expenses, let us develop a model for comprehensively understanding a site’s Web hosting costs.

Step 1. Develop a model for allocating technology operations & infrastructure costs to each Web site/brand

Let us assume for this example that your company operates some medium to large Web sites and spends $100K/month on fully managed¹ origin² Web hosting and another $50K/month on CDN. That means your company spends $1.8MM/year on Web sites hosting.

It is important to add origin Web hosting and CDN costs to know your true Web hosting costs, especially if you operate multiple Web brands and need to allocate Web hosting costs back to each. For example, let us assume you have two Web sites: brandA.com, a dynamic ecommerce site costing $10K/month on origin hosting plus $2K/month on CDN; and brandB.com, serving a lot of videos and photos costing $5K/month on origin hosting plus $19K/month on CDN. In this example, brandA.com actually costs $12K/month, which is half the hosting cost of brandB.com, $24K/month. Without adding the CDN costs, you may mistakenly assume the opposite that brandA.com costs twice as much to host as brandB.com. Origin hosting and CDN are two sides of the same coin. I recommend that you manage them both together from both technology/architecture and budget perspectives.

Then you add the costs of third-party vendor provided parts of the site rented in the software-as-service model. Next, add licensed software costs used at your hosting location. Let us assume that brandA.com also has:

• some blogs hosted at wordpress.com for $400/month
• Google Analytics for $0/month
• Other licensed platform/application software running on your servers billed separately from the managed hosting. Let us assume brandA.com’s share of that is $1,000/month.

So your Web hosting and infrastructure costs for brandA.com would be $13,400/month. That’s $160,800/year.

Assuming that many of your Web sites share infrastructure and systems management & support staff at your Web hosting provider, you may not have a precise allocation of costs to each brand. That’s ok: It doesn’t need to be perfect nor a staff-time consuming calculation every month. Work with your hosting provider and implement a formula/algorithm that provides a reasonably good breakup and needs to be changed only when there is a major infrastructure change.

Side Note: In order to stay competitive, adapt to changes in the market and meet changing customer sites, brandA.com also needs to do product and software development on a regular basis. However, that’s beyond the scope of this discussion. Managing ongoing product and software development costs for brandA.com could be the subject of another article.

Step 2. Regularly review the tech operations costs for each brand and make changes to control costs

Every month, review your tech operations costs for your business as a whole and for each brand. Make changes in technology and process as needed to manage your expenses. If you don’t review the expenses on a monthly basis, you run the risk of small increases happening in various places every month that add up to a lot.

Without active management done on a monthly basis, brandA.com could creep up from $13,400 to $16,000 the next month and $20,000 the month after. That $1.8MM you were expecting to spend on hosting for the year could turn out to be $2.4MM.

So what does such active management include?

Monitor and manage your bandwidth charges. This is one to keen an eye on. If you bandwidth charges go over your fixed commit, your expenses can quickly blow over budget. If you find bandwidth use increasing, investigate the cause and make course corrections. In some cases, this may simply be due to expected increase in traffic, but in other cases it could be avoided. The related article on this site about taking advantage of browser caching to lower costs provides some tips.

Request your engineers to monitor and manage your servers resource usage (CPU, memory) so that the need for adding hardware can be avoided as much as possible. Enable and ensure regular communications between your technology operations team and your software development team so that software developers are alerted of any application behavior that is consuming more than expected server resources. Give the software developers time to resolve such issues when found.

Review the invoice details to make sure you understand and are in agreement with the invoice. A Web hosting bill can be very detailed and complex to understand. Do not hesitate to ask the hosting provider to explain and justify anything that you don’t understand. Don’t just assume the bills are always correct. They could (and occasionally will) be mistakes in the bills. Be sure to dispute these with the vendor in a respectful and friendly way.

These are just some examples. Please feel welcome to make more suggestions via comments on this post.

The time (and thus money invested) in controlling tech operations cost will be well worth the savings / avoidance of huge cost increases.

Keep abreast of evolving technologies and cost saving methods. Periodically review these with your vendor(s).

Cloud computing is exciting as a technology, and it is equally exciting as a pricing model.

If you find market conditions have changed drastically, request your vendor to consider lowering rates/prices even if you are locked into a contract. You don’t lose anything by asking and the vendor’s response will be an indicator of their customer service and long term business interest with you.

Tweet

Save

1. Fully managed Web hosting includes network & hardware infrastructure, 24×7 staff and real estate [↩]
2. The origin part of your Web hosting environments includes the network and server infrastructure at your hosting facility location(s) where your Web applications and installed and running. It could be in-house data centers or at providers such as RackSpace, IPSoft or Savvis [↩]

Companies that operate heavily trafficked Web sites can save thousands of dollars every month by maximizing their use of browser-side caching.

Large Web sites pay for bandwidth at their Web hosting data center and also at their content delivery network (CDN, e.g. Akamai, LimeLight, CDNetworks). Bandwidth costs add up to huge monthly bills. On small-business or personal Web sites where bandwidth costs don’t go over, this is not an issue, but on large Web sites, this is important to address and monitor.

Companies operating large Web sites often have complex situations like the following:

• An comprehensive and deep understanding of all technology cost drivers and their impacts on each other. For example, a programmer may think they are saving the company money by architecting an application in a way that it requires minimal hardware servers, but not realize that the same design actually results in even higher costs elsewhere like CDN bills.
• Busy development teams working on multiple projects on tight timelines. This results in compromises between product features/timelines and technical/architectural best practices/standards.
• Web content management and presentation platform(s) that have evolved over the years
• Staff churn over the years and an uneven distribution of technical knowledge and best practices about the Web site(s)
• The continued following of some obsolete “best practices” and standards that were established long ago when they were beneficial, but are now detrimental.

Tech teams at complex Web sites would likely find upon investigation that their Web sites suffer from problems that they either didn’t know about or didn’t know the extent of the damage they are causing.

One such problem is that certain static objects on the company’s Web pages that should be cached by the end users’ Web browsers are either not cached by the browsers at all or not cached enough. Some objects are at least cached by the CDN used by the company, but some perfectly cacheable objects are served all the way back form the origin servers for every request! An unnecessarily costly situation that can be avoided.

In addition to wasteful bandwidth charges resulting in high monthly bills, there are also other disadvantages caused by cacheable objects being unnecessarily served from origin servers:

• They slow down your Web pages. Instead of the browser being able to use local copies of these objects, it has to fetch them all the way from your origin servers.
• Unnecessary load on origin Web servers and network equipment at Web hosting facility. This can be an especially severe problem when a Web site experiences a sudden many-fold increase in traffic caused by a prominent incoming link on the home page of a high traffic like Yahoo, MSN or Google.
• Additional storage in logs at the origin Web hosting locations’ servers and other devices.
• Unneeded processing and work the origin servers, network equipment, CDN, the Internet in the middle all the way up to the client browsers have to do to transfer these objects from origin to the end user’s browser. Be environmentally friendly and avoid all this is costly waste.

The increase in bandwidth, load on servers and networking equipment and log file storage space increases caused by a few objects on Web pages being served by origin servers for every request may mistakenly seem like an insignificant problem, but little drops of water make the mighty oceans. Some calculations will show that for large Web sites, the cost of this can add up to tens of thousands of dollars a month in bandwidth costs alone.

How should companies operating large Web sites solve this problem?

For technology managers:

• Make it a best practice to maximize the use of browser-side caching on your Web pages. Discuss this topic with the entire Web technology team. Awareness among the information workers is important so that they can keep this in mind for future work and also address what’s already in place. Show the engineers some sample calculations to illustrate how much money is wasted in avoidable bandwidth costs: that will prove this is not an insignificant issue.
• If this problem is widespread in your Web site(s), make the initial cleanup a formal project. Analyze how much money you’d save and other problems you’d solve by fixing this and present it to the finance and business management. Once you show the cost savings, especially in this economy, this project will not be hard to justify.

For engineers:

• Read the article about optimizing caching at Google Code for technical details on how to leverage browser and proxy caching. It explains the use of HTTP headers like Cache-Control, Expires, Last-Modified, and Etag.
• Review any objects that are served by origin servers every time for legacy reasons that may now be obsolete.
• Combine some JavaScript files commonly used by your Web pages so that the one unified and shared file would have higher caching probability. Do the same with external CSS style sheets.
• Study a good book on Web site optimization like Even Faster Web Sites: Performance Best Practices for Web Developers. Share these recommendations and hold a discussion with your tech and production colleagues.

Tweet

1 save

One of the differences between Amazon EC2 server instances and normal servers is that the server’s local disk storage state (i.e. changes to data) on EC2 instances does not persist over instance shutdowns and powering on. This was mentioned in my earlier post about hosting my Web site on Amazon EC2 and S3,

Therefore, it is a good idea to store your home directory, Web document root and databases on an Amazon EBS volume, where the data does persist like in a normal networked hard drive. Another benefit of using an Amazon EBS volume as a data disk is that it separates your operating system image from your data. This way, when you upgrade from a server instance with less computing power to one with more computing power, you can reattach your data drive to it for use there.

You can create an EBS volume and attach it to your EC2 server instance using a procedure similar to the following.

First, create an EBS volume.

You can use Elasticfox Firefox Extension for Amazon EC2 to:

• create a EBS volume
• attach it to your EC2 instance
• alias it to a device, In this example, we use /dev/sdh

Then attach the “disk” to your EC2 instance and move your folders to it using a procedure similar to the following commands issued from a bash shell.

# Initialize (format) the EBS drive to prepare it for use
# Note: replace /dev/sdh below with the device you used for this EBS drive
mkfs.ext3 /dev/sdh
#
# Create the mount point where the EBS drive will be mounted
sudo mkdir /mnt/rj-09031301
# Side note: I use a naming convention of rj-YYMMDDNN to assign unique names
# to my disk drives, where YYMMDD is the date the drive was put into service
# and NN is the serial number of the disk created that day.
#
# Mount the EBS drive
sudo mount -t ext3 /dev/sdh /mnt/rj-09031301
#
# Temporarily stop the Apache Web server
sudo /etc/init.d/apache2 stop
#
# Move the current /home folder to a temporary backup
# This temporary backup folder can be deleted later
sudo mv /home /home.backup
#
# Symbolic link the home folder on the EBS disk as the /home folder
sudo ln -s /mnt/rj-09031301/home /home
#
# Start the Apache Web server
sudo /etc/init.d/apache2 start


Limitations:

One current limitation of EBS volumes is that a particular EBS disk can only be attached to one server instance at a given time. Hopefully, in a near future version upgrade of EC2 and EBS, Amazon will enable an EBS volume to be attached to multiple concurrent server instances. That will enable EBS to be used similar to how SAN or NAS storage is used in a traditional (pre cloud computing era) server environment. That will enable scaling Web (and other) applications without having to copy and synchronize data across multiple EBS instances. Until Amazon adds that feature, you will need to maintain one EBS disk per server and keep their data in sync. One method of making the initial clones is to use the feature that creates a snapshot of an EBS volume onto S3.

Related article on Amazon’s site:

• Running MySQL on Amazon EC2 with Elastic Block Store

  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663

Tweet

22 saves

There are many good ways to organize your technology department. This article presents one of them. It is written for someone in a CTO, CIO, VP Technology or a consultant role looking for suggestions on organizing or reorganizing your Web technology department. It is best suited for you if your organization has the following characteristics:

• You manage software engineering, implementation and technology operations for 3 or more Web sites.
• Yours is a medium to large size organization with somewhere between 20 to 150 technology staff.
• Internal corporate IT functions such as desktop support, telecommunications services and internal business systems are beyond the scope of this article.

The Venn diagram below presents one model of organizing your department into 3 sub-departments.

Web Technology Department Organization Venn Diagram Illustrating Purposeful Overlap Among Sub-Departments

Many CTOs organize their technology departments as 2 sub-departments: Software Engineering and Technology Operations. Software engineering is the function that is responsible for developing and/or implementing Web application software. Technology Operations is responsible for running, maintaining and supporting the Web applications.

If you operate 1 or 2 Web sites, having these 2 sub-departments is a good approach. For 3 or more Web sites, organizing Software Engineering into Site Engineering and Platform Engineering has some benefits.

Site Engineering is focused on working on the Web sites’ direct projects. Its work includes

• Small and large projects for adding or changing functionality on the Web sites
• Bug fixes on the Web site applications

Platform Engineering is typically smaller than the other two organizations and typically includes functions like:

• Architecture across sites
• Shared applications across sites
• Common libraries across sites
• Research & Development (R&D)

Technology Operations includes functions such as:

• Systems & Applications Administration
• Infrastructure Management
• 24×7 Tech Support
• Builds & Configuration
• Release Management
• Testing & Quality Assurance (QA) *
• Technical Analysis
• Technical Project Management
• Budget Management

* Note: QA can also be a completely independent department or part of another department outside technology.

These three departments have purposeful overlap of responsibilities as illustrated in the Venn diagram above. That helps minimize the chances of the departments becoming silos with walls between them. For success, it is important that your entire department functions as one integrated unit. Some shared goals & responsibilities are required for mutual success.

To make this work, you need 3 directors who head up these departments who work well together, collaborate often and are not sensitive about their turf. They should know that a successful technology manager is not an individual-only contributor, but a great team player with peers. They should have strong goodwill among each other and welcome each other to work directly with their teams. Such a collaborative team is essential.

Tweet

1 save

I now use a device called Drobo (2nd generation), manufactured by Data Robotics, Inc. as the primary data storage and backup medium at my home location. I have attached it via a USB 2.0 cable to my Apple Airport Extreme wireless network router. The Airport Extreme enables me to share USB 2.0 based storage devices on my home network so they can be simultaneously used by multiple computers. This system of making the same hard disk(s) available to multiple computers in a network is called Network Attached Storage (NAS).

The Drobo replaced my USB 2.0 external hard disk drive manufactured by Western Digital (WD) that was earlier attached to my Airport Extreme. The Drobo has significant advantages over the previous WD drive:

• Data protection in the event of one hard drive failure as a result of wear and tear due to use over time
• Ability to increase the storage size of the device as the volume of my data grows (more photographs, music, videos, etc.)

A data protection strategy should include both local fault tolerance and remote storage in an offsite location. For off site storage, I keep copies of my data at online locations like Amazon S3, Smugmug, Google Docs, IMAP mail servers and Apple’s Mobile Me service. Since the volume of my data is in terabytes (~ 15 years of emails, photographs, music, videos), recovering large amounts of data from online locations is reserved for extreme situations when local storage is destroyed or corrupted. The Drobo uses technology to protect data in cases of failure (via normal wear and tear) of one of the hard disks inside the Drobo. The benefit I get is similar to the benefit provided by a set of technologies called RAID.

Unlike most RAID devices for home/small-business use, the Drobo allows me to mix and match hard drives of varying capacities. It has 4 bays to insert hard drives. For example, I can have two 1 TB drives today. Next month, I can add another 1.5 TB drive to the 3rd slot. A few months later, I can add a 2 TB drive to the 4th slot. Then when I need more space next year, I can replace one of the 1 TB drives with a 2 TB drive. As I make these changes, the Drobo will automatically recalculate the optimal distribution of my data across all these drives to maximize its storage space and provide data protection. Adding a new drive or replacing a drive with another is done without downtime. The Drobo stays up and running during disk changes and the data on it remains usable by my computers, even while I’m replacing a drive or when it is redistributing data on the new set of drives after a drive is inserted.

Benefits

• Save money by buying 1 TB drives today and 2 TB drives when they are cheaper in the future
• Save money by buying just hard disks for adding storage instead of buying a drive plus an enclosure and power supply adapter for each drive. This is also energy efficient (fewer power adapters) and good for the Environment (fewer drive enclosures made of plastic and power adapter units purchased)
• Save time by letting the Drobo take care of data protection at the local level. Also save time that would have been spent recovering data from remote locations in the event of a local drive failure.
• Peace of mind having good data protection at home.

Moving data from the WD external drive to Drobo

I transferred the files by having both drives directly connected to my Apple Macbook Pro: The Drobo to the Firewire 800 port and the WD to the USB 2.0 port.

My MacBook Air‘s Time Machine backups used to be stored on the USB external drive. Since it was attached to an Airport Extreme, the Apple Time Machine backups were stored in a special virtual storage location called a sparsebundle. I just copied the sparsebundle from the WD drive to the Drobo and now my Time Machine backups (including all the Time Machine history of my MacBook Air) are now transferred to my Drobo storage device. Thanks to my Time Machine bundles having been on a sparsebundle, it was easy to transfer them to my new Drobo storage device using a simple copy process using the Mac OS Finder.

Rating:

Tweet

Save

To be effective at anything in life, balance is required Be it personal life, leadership, management, technical work: balance is essential. You need to be aware of balance at all times.

I plan to write a series of articles about this subject.

Some topics I discuss with friends and colleagues:

• standardization vs. innovation & creativity (technology, management)
• dedicated resources vs. shared services (management, leadership, technology)
• security & privacy vs. convenience (technology, product design)
• centralization vs. decentralization & empowerment (leadership, management)

I’d love to hear your topic suggestions and read your viewpoints and experiences. Please do so via the comments section here and in future articles.

Tweet

Save