Defining High Availability

In todays business computing environments a wide variety of terms are used to describe systems management, systems performance and system availability. One commonly used term is High Availability (HA). This is a very broad term that can encompass many different levels of availability and the costs associated with the various levels of availability. This term is open to quite a bit of interpretation and this interpretation often leads to confusion about exactly what level of HA an application, device or service provides. Below are the items to factor in when assessing the actual availability of a given service to ensure that it meets your specific interpretation of HA.

Level-setting Expectations
High Availability can mean something different for each person that says or hears the term. It is important to level set expectations about HA and its meaning before having an in-depth discussion about how to meet the objectives laid out in an HA environment. Properly defining HA and calculating the costs associated with implementing HA has four components:

Time to recovery – It is important to understand how long a failure will take to recover from, this will allow you to properly choose solutions that can identify and recover from a failure within a given time frame. A failure can be a hardware problem, a software malfunction or a human error that causes the specific service to act in a way other then it was designed. There are many valid cases where time to recovery can take on the order of minutes or hours, there are other valid cases where recovery should be near-instantaneous.

Method of recovery – Method of recovery is an important component of planning and HA solution and it's associated cost. Many times recovering from failures is an automated fashion, but it is not uncommon to have an error that requires manual intervention to clear the problem. This is often done for categories of problems that are not critical to the operation of a business or customer impacting.

Data Loss and Corruption – Data loss and corruption is an important part of developing a strategy for HA. Data loss and corruption can occur during a failover of services between nodes, while the network works to get into a state of equilibrium after a change or during periods when a given service is down. All data has a value associated with it and when calculating the maximum allowable downtime for a service, data value should be calculated in as well.

Performance Impact– Often times a failure of a service component will cause a degradation in service, yet leave the service online for users. This degradation if often times acceptable assuming it is for a short, limited period of time. Understanding how users will use the service will enable you to understand what level of performance loss is acceptable.

A Perfect World
Before we continue into a discussion about how to achieve a given level of High Availability, I want to define my expectation when I hear the term High Availability. When I use the term HA I expect and application or service that can transparently handle failures from a user perspective. I expect an application that despite a failure on the back-end including a server, disk, network connection or otherwise will automatically failover in a way that the end user does not see a disruption in how they are used to interacting with the application. The user should see no degradation in service or loss of data because of the failure.

My definition of HA is assuming a perfect world and adequate funding to architect and implement such a solution. But as we know, IT is not always funded with the necessary money to make dreams into reality. In these cases we must refer back to the first list of components that make up HA to determine which items can be compromised on.

Defining HA for your Environment
Now that we have covered what items are used to define HA, and my definition of HA in a perfect IT world, lets discuss the process for defining a level of HA appropriate for your needs and balancing that with the associated costs with a given level of HA. First is to understand your user base and what their expectations are around application performance, response time and recovery. Things to consider are when your users use the application, how they enter data and what response time they are used to when interacting with the application.

Second is to define what the technical solution will look like for the above customer requirements. This stage is where you will evaluate various levels of redundancy and capability in any database servers, network components, data centers and application capabilities. This stage should include an evaluation of both vendor packaged solutions, and home grown solutions that will meet your needs. This assessment should also include a review of staff capabilities to determine if training will be needed for staff when implementing new technologies.

Third we will define the cost for each component of the above developed architecture. This cost is the cost for an optimal solution, broken down by each individual component. This cost should include all hardware, implementation and software licensing costs for a given period of time. A three year costing is standard within IT and is a good basis to compare several different solutions in an equal fashion.

Finally, we must evaluate the potential cost savings for each component of the solution if we were to cut back from an optimal solution to a more cost effective one. This evaluation should show the portions of the solution that can be implemented via multiple methods, and the associated costs for each method. This information is then used for comparison to balance the required level of HA with the budget available for the project. By properly understanding how much each component of the solution will cost, you can properly evaluate what the possible level of HA will be with each potential increase or decrease in project funding.

Methods for implementing HA
For most of this document I have avoided discussions actual technical solutions available on the market for implementing HAs. This omission was to ensure that HA was defined per your specific needs before defining possible hardware and software solutions. Now I am going to dive into several popular options on the market for assisting to make applications HA capable.

Linux-HA – Linux-HA is an open source solution for managing services across multiple nodes within a cluster for providing a basic high availability solution. Linux-HA is often used to provide automated failover for applications like Jboss, Apache, Lustre or FTP. While Linux-HA will not provide the sub-second failover that some environments need, it will allow administrators to easily setup a pair of servers to act as hot-standbys for one another.

Redundant Switch Fabrics – Modern ethernet switches have multiple levels of redundant capability including redundant controllers within a switch, redundant power supplies, and at the high end redundant switch fabrics that should one complete set of switches and routers fail, a second will seamlessly handle the failover and subsequent network traffic. Technologies like OSPF will ensure that routing of IP traffic continues uninterrupted and protocols like spanning tree will ensure that switches with multiple paths will utilize them in an optimal fashion during both regular and failover scenarios.

RAID – Redundant Arrays of Independent Disks (RAID) is a common method of ensuring that a single disk failure within a server does not cause data loss or corruption. RAID capability can be added through specialized hardware solutions or via low cost software solutions. Both provide a level of protection above standard disks, while keeping total solution costs low.

Oracle RAC – Oracle's Real Application Cluster (RAC) is a clustering solution, often associated with Oracle's database products for both providing high availability functionality, as well as a platform to scale a databases performance. While Oracle RAC is often more expensive then other clustering solutions from MySQL, it provides a very scalable and reliable platform for ensuring very high levels of availability for applications and their associated databases.

Fiber Channel – Fiber Channel solutions for attaching storage to servers often implement redundancy via dual, redundant fiber channel fabrics. These are often implemented utilizing completely separate switches, cables and power connections. This type of solution can ensure that common failures like cables and PCI cards will not cause a server to loose access to its storage or data corruption.

High Availability is often taken to mean something different for each person. Ultimately, HA is ensuring that customer and end user expectations are met for how an application performs and recovers in the event of a failure. When setting up an application, you must first define HA for your specific needs, you can then properly develop a solution that will meet those expectations. As with most projects within Information Technology, you will then have to assess each component of the solution and make possible tradeoffs to ensure the projects budget is met. Ensuring an application is available and properly recovers is a part of all major Information Technology projects, today there are many possible technical solutions to ensure your customers expectation of HA is met.

Hardware TCO – Predictable Planning with Refresh Cycles

Often, the most expensive investment any Information Technology (IT) organization will make is its base infrastructure; servers, storage and other various hardware. Yet, these hardware purchases are often given much less thought then software or services purchases and assumed to be routine, and just a cost of doing business. Hardware typically has several phases that should be evaluated as part of the purchase, these include the initial purchase price, the cost of maintaining it, and the ultimate cost of refreshing the hardware at the end of it's useful life. All should play an equal balance when evaluating new platforms, refresh cycles and testing new solutions for introduction to a company.

Often times when a company begins to assess the total cost of ownership (TCO) around its IT assets, it must involve teams not traditionally involved in IT planning. These teams can include facilities, engineering, building managers, application developers and data base administrators. Each of these groups can provide valuable input on how the servers and other infrastructure affect there environments and costs on a yearly basis.

There are three primary phases to all hardware purchases:

Initial Hardware Purchase
The initial hardware purchase is often thought to be the most expensive phase, but in reality after factoring in the support costs for a piece of hardware it turns out to be about one-third to one-fourth of the TCO. The initial price is often the easiest to evaluate, but should carefully be weighed against the long term costs of purchasing a specific brand or type of hardware.

Often vendors will allow for additional years of warranty coverage, or higher levels of support to be purchased when the system is first bought. These are often a wise investment if the hardware will be used longer then the initial warranty period. The increased level of support can often mean that your staff will spend less time supporting the system, and more time working on more beneficial tasks.

Support
The support costs are often the most expensive phases of hardware ownership. The support costs include patching the operating system, supplying power to the system, cooling the system and managing the applications hosted on the system. These costs are amortized over the life of the system, and over time can add up to be the most expensive part of the TCO formula. Often, these support costs are also where the most efficiencies can be gained to lower the TCO of the system.

There are many things that can be done to lower the support costs around hardware, most involve improved processes to cut back the amount of time staff have to spend manually managing each specific server. The most notable of these is automation of patch management. By utilizing tools to automate patch deployments and status monitoring, staff can cut significant manual administration time from each specific server. Proactive monitoring of system and application health can also play an important role in cutting down TCO for hardware and associated services. There are many packages available today to assist system administrators in proactively correcting both hardware and software faults before they cause a failure for the end users. These apps can ensure that staff isolate and correct problems as soon as possible to minimize the necessary time to correct faults.

Utilization is another space where the TCO for your servers can be lowered. By ensuring that servers do not run at idle for long periods of time, you can ensure that any power being used by the servers is being used efficiently. Often times, a single server can handle the load that multiple servers used to handle. It is much more efficient to power and cool a single server then multiple servers in this case, and scales very well as you begin looking at utilization rates across dozens or hundreds of servers.

Refresh
The final phase to evaluate for all hardware purchases is the refresh cycle. All hardware has a finite lifecycle in which at the end it will need to be replaced because it is either obsolete and not cost effective to maintain any longer. Obsolete in this context can be used in two ways, first to mean the hardware is so old is can no longer operate with the current operating systems, tools and patches available, or it no longer meets the business needs of your company.

There are often two methods that companies use to replace aging hardware, the first and most common is just purchasing new hardware when a system is no longer under warranty or has gotten too slow to use in the IT environment. More and more though, firms are implementing a rolling refresh cycle to add a level of predictability to all hardware purchases. A rolling refresh cycle allows a company to more clearly outlay capitol for IT investments, and better plan long term cycles for purchases, upgrades and replacements. Typically, a rolling refresh schedule is based on the standard warranty with newer server hardware, 3-5 years. A rolling refresh cycle also allows IT staff to better plan work loads by knowing ahead of time that new servers will need to be configured, tested and put into production.

Refresh cycle planning should also include an assessment of upcoming technologies and how that will affect purchases two and three years down the line. Every year hardware is faster and faster then before, and provides new possibilities for the amount of data that can be processed. In addition new technologies around virtualization are changing the dynamic of how system administrators provision new systems. No longer do system administrators add a single new server because of a single new application, today many different apps can be run on a single piece of hardware and kept separate from each other by using virtualization technologies.

As you assess your refresh cycle an important part of the TCO calculation is determining what applications can run within virtualized environments, and which will need separate hardware to run on. This will determine what level of consolidation can occur from year to year as the refresh occurs. As you look toward implementing a rolling refresh cycle, a first step is to understand how many existing servers are in place and how many existing applications. That data can then be used to develop a matrix of how things would look if virtualization were employed, and how a rolling refresh cycle could be utilized over time to ensure that all pieces of the infrastructure are upgraded in an expected period of time.

Minimizing the TCO of IT hardware is a key component of ensuring that the long term costs of owning the hardware are predictable and manageable. A rolling refresh cycle, paired with newer technologies like monitoring tools, automation tools and virtualization can allow IT staff to clearly plan how hardware will be used from purchase to end of life and how it will then be replaced. This cycle ensures staff can plan for future upgrades and migrations, as well as avoid last minute unexpected upgrades.

Platform Decisions - OS Choices

Following up on my previous discussion around platform decisions and solution architecture, I wanted to dive into Operating System (OS) choices. This is a difficult choice for many companies because of competing priorities, experience, training levels, costs and ultimately faith to one OS or another. Choosing an OS for production use is also a difficult choice because the options change so frequently, and the applications you will ultimately use may not work with your preferred choice of OS.

With most companies, this is rarely a discussion around which OS will be used exclusively. More often it is a discussion around which OS will be added to or eliminated from the infrastructure to either lower administration costs, lower maintenance costs or increase capabilities. Often, companies also break down an OS choice into groups, either server and desktop class systems, or by departmental needs. This separation can be very beneficial when discussing any changes; it allows the folks doing the assessment to clearly define needs and balance them based on focused groups, trying to balance needs across a large company can often prove to be difficult to impossible.

Primary Reasons for Change
First, lets explore the primary reasons a company would change the mix of OSs already in use within the IT organizations.

Lower Training Costs: Today Linux is the predominant OS within the Education communities. This creates an environment where new staff entering the work force are very experienced and knowledgeable on working with Linux based systems. This is important because by using an OS that potential staff are experienced with limits the training that is required to get and keep them proficient at system administration. Companies will often eliminate an OS from use because staff skills are not at a peak for it, and costs to keep them trained at appropriate levels continue to rise as an OSs dominance disappears. This was primarily seen with the large UNIX variants (Solaris, HP-UX, AIX), over time companies have limited the use of them because students were no longer coming with these skills from college, and existing staff were spending more and more time keeping up with training on these platforms.

Increase Performance: Performance is often a primary reason to evaluate utilizing a new OS. Most application vendors today support a very narrow sub-set of the available OSs on the market. Because of this, they must focus there resources on tuning and performance enhancements, at times companies can get a 20% to 30% improvement in application performance by moving the application to a better supported and tuned OS.

Lower Maintenance Costs: In todays world where Open Source is becoming more and more dominant in the business world, companies are reviewing there traditional support and licensing modes. There is a multitude of options available today from OSs that have no cost to use, to OSs that charge for all used instances. A company with legacy OSs in place has an opportunity to review how they negotiate support contracts with these new models so that they are paying for an appropriate level of support for all systems.

Increase Capabilities: Being able to provide a new capability that was previously not available is a large reason companies look to adding new OSs to there existing enterprise. Todays applications vendors rarely support all possible operating systems, more often then not, they choose a subset of OSs that they feel will best cover there potential market. Companies are constantly evaluating new applications for potential benefits to the companies bottom line, as part of this, often times a new OS must be brought in for the administration team to manage to provide new capabilities by adding new applications to the enterprise.

Assessment Questions
Second, lets look at some questions that can be asked when assessing possible OSs for use in your environment:

1. Why am I assessing my current install base of OSs? What is the goal of any changes?
2. What is the current cost, both in licensing and training, for all current OSs we have deployed?
3. What OSs are our staff skilled at administering? Both current utilized and non-utilized OSs.
4. For any new OSs we are assessing, what will be the training cost to get staff proficient at maintaining them? The yearly cost to keep our staffs' skills up to date?
5. What level of OS support can be provided by in-house resources and what will need to be included with any purchased support agreements?
6. What tools currently in place will need changes or license upgrades to support a new OS?
7. Will this OS introduce security vulnerabilities that will be unreasonably difficult to manage?
8. Is this system mission critical? Can the system utilize an OS with just community support and no formal SLAs?

Lets also look at some questions that can be used to asses if an existing OS should stay part of the enterprise or should be phased out:

1. What is the yearly cost in support for this OS by itself? In relation to other OSs within the company?
2. What percentage of staff in house are proficient on this versus other OSs?
3. What is the support cycle for this OS? How much longer will the vendor provide patches without additional support contract costs being incurred?
4. Does the vendor, both OS and application, provide a supported upgrade path to a newer version?

These questions can guide an informed decision about future OS choices. It is critical to understand the financial trade offs that each OS will introduce, including training costs, licensing and support costs, hardware costs and performance capabilities. Every OS has it's own specific details including cost of acquisition, cost of administration, cost of support, etc. These questions should be weighed against currently in house OSs, as well as new OSs that are potentially being considered for addition to your environment.

In addition to the financial questions for each OS, a company must consider the life cycle of the OS. Most OSs have formal release schedules for patches, upgrades and subsequent versions. It is important to evaluate any new OSs with these details in mind. It can end up being quite costly if an OS hits its end of life and you have to rapidly stop using it and migrate the work load to another platform, where as be carefully evaluating the roadmaps for the OS, you can make an informed decision that will successfully work with your in-house processes for support and upgrades.

Making a choice to add or eliminate an OS within a company can be a difficult one, both because of personal territorial issues, as well as complicated technical needs. It is important to focus on the true costs to the company related to the decision, this will ensure that training, implementation, licensing and support are factored in and staff fully understand the costs and ultimate decision.

Enterprise Architecture versus Solution Architecture

Recently I have been in several customer meetings where their newly hired Enterprise Architect(EA) joined to listen in and provide feedback. Most of these meetings were to discuss an individual cluster or system that is being implemented, and it seemed that most EAs these days are still too focused on systems, solutions and details; and too little focus is being paid to the true activities I see as relevant for an Enterprise Architect. I decided to throw my own comments out there about where an EA falls within an organization, and how that differs from what I call Solution Architects.

The way I see it, a Solution Architect is more closely associated with what I see as technical sales people, they focus on the individual system or application, focusing on the details of what software packages will work, what a good support model is, and how to implement it within the companies framework that is defined by the Enterprise Architect.

I then see the Enterprise Architect as a pathway between the companies Business Goals and the IT personnel that must delivery tools to meet and track those goals. The EAs goal is to define a set of policies at the company wide level that ensure things like legal compliance, consistent identity management and company wide reporting capabilities.

If an EA gets too involved in the Solution Architect level details, the company suffers because those higher level activities are not being managed appropriately. A successful EA has both the willingness and capability to work with the company executives and turn there business vision into a technology vision and push that down to the Solution Architects and IT staff.

Platform Decisions – Solution Architecture

Without a doubt, one of the oldest and most heated debates in the non-Windows world is emacs versus vi. Similarly the on going debate about the correct platforms on which to conduct business is rapidly approaching the same fervor. These platforms include a plethora of options including operating system choice (Windows, Linux, Solaris, HP-UX, etc) and systems architecture (cluster, grid, SMP), interconnects (Ethernet, Infiniband) and storage (NFS, GPFS, SAN, etc). We are going to avoid this altercation and focus on the questions that must be asked when objectively evaluating the systems architecture question. In this article we will focus strictly on the systems architecture question: do we use commodity parts to assemble into a cluster or grid, or do we purchase a more custom SMP type solution for running a company’s applications?

Before we start, it should be noted that there are a variety of definitions for cluster and grid in the industry today, for the purposes of this discussion we will define a cluster as an environment of identical commodity hardware, used to run a specific set of less than a dozen applications. A grid on the other hand is an interconnected set of systems used to run a variety of applications, usually numbering in the dozens or hundreds. These applications on the grid environmentmay cover a wide range of business needs and business units with little to no similarities in how they operate. Finally, with SMP we are talking about a system comprised of greater than or equal to eight processor sockets in a single domain. An SMP system is not built of commodity hardware like a cluster or grid, but is instead a purpose designed and built system. An SMP type platform could run any number of applications, both commercial and custom.

The discussion of which platform is best for an organization should encompass many things including implementation costs, staff skills, maintenance costs, platform capabilities, growth expectations and usage models. While some would assume that this is primarily a technical discussion, the majority of decisions are actually financial. Specifically, this discussion should revolve mostly around how the company is going to benefit over the long-term from the choices that are made. This decision is not an easy one for any company. It can involve legacy code and processes, a lack ofunderstanding or experience with currently unused platforms, and ultimately personal feelings about a given solution.

While it is important to seek the best technical solutions, the platform decision will ultimately be the one that primarily suits the company both from a business productivity as well as financial perspective. Clearly this decision should be made after consulting the various levels within the organization including users, system management personnel, line organization management and company executives. The users and system management representatives will be able to provide input on intended uses and technical capabilities to manage the new system. Line organization management will be able to provide input on how their departments will be able to use any new capabilities afforded them. The executives will add the strategic thinking to the mix.

Ultimately, this information should be presented to executive management for the decision making process. They will ensure that the input from the various stakeholders within the company is appropriately considered. They will also properly evaluate the financial benefits that the proposed solution offers. Finally, they will ensure that the company’s strategic business plans and goals are properly evaluated. There is no way that a team without corporate scope can adequately review the relevant questions that cover the entire organization. Further a higher-level team can ensure that a solution will meet more stakeholder needs than if the evaluation were done at any departmental level.

Correspondingly, I believe the evaluation team should ask the following questions when evaluating future system choices:

• What is the purchase price of this system?
• What is the implementation cost of this system? This should include not only costs to migrate applications and data, but also facility costs like additional power and cooling.
• Will this system integrate in with existing technologies used for storage, networking, authentication, and security?
• What is the yearly support cost of this system? This value should include the cost of staff to maintain the system, the cost to power and cool the system, the cost for regular preventive maintenance, and the cost of training for staff so they are up to date on managing and maintaining the system.
• What is the cost of adding new users to this system? If this system is successful and additional line-organizations within the company would like to utilize it, what will the costs to the organization and company be to migrate them to the new system?
• What will utilization of this system look like? Will this system provide a higher utilization rate because of its architecture then competing solutions? Clusters and Grids are primarily used in an organization to provide an environment that can adjust as the needs of various departments change.
• What capabilities can this system provide to assist with the company’s core competencies? Are there new business tools and methodologies that can be employed because of this added capability?
• What growth is expected on this system over the life of the system? What are the overall vendor and in-house lifecycles of the system? This is important so that the budgeting organizations can be prepared when it is time to upgrade or replace the system.
• What systems do the applications run most efficiently (e.g. SMP versus grid)? What type of interconnect (e.g. Ethernet, Infiniband, etc.) will provide the most efficient communications? How much memory is needed per core for the most efficient calculations? An analysis of all applications that will run on this platform should be done. This review of the applications should also include vendor communication about their recommended architectures with a key focus on ease of management and stabilit
  

By asking and answering these questions a company can get a complete accounting of which solution will provide the bestlong term benefit in cost, improved utilization-rates and more efficient growth. Ultimately the decision that is best for a company is the one that makes the most financial sense.

I believe that by going with a grid based solution, a growing and dynamic company will have a system in place that will easily change as the company’s needs and directions change. A grid based solution can provide most companies with aplatform that will provide capability for today’s needs, and change to accommodate tomorrows. Certainly I believe that enterprise grids will be one of the strongest candidate architectures for providing solutions for a company with diverse business processes.

In the coming weeks I intend to evaluate the other components including networking, operating systems and storage and look at what questions must be asked for each of these areas.