Merging Business and IT

Dell announces the open sourcing of multiple Hadoop Barclamps

2011-11-29T08:00:00.000-08:00

Today, my team at Dell is very excited to announce that we have released several barclamps into the opensource community.

These barclamps are designed and build to enable Hadoop environments to be rapidly deployed using Dell Crowbar.

The barclamps can be found on github, along with the project wiki including build information, FAQs and other important links.

Folks on my team also created a variety of videos to help you get started in building Crowbar and setting up Hadoop environments using Crowbar:

We encourage you to download these barclamps, test them in your environment, and join the crowbar@dell.com community.

InformationWeek Hadoop Webinar

2011-11-22T07:29:00.000-08:00

Last week I spoke at a webinar hosted by Information Week. The topic was deployment of Hadoop in your computing environments. I spoke about how Dell defines Big Data, what types of problems can be solved with a Hadoop solution and the components that Dell includes as part of the Hadoop solution.

The MP4 from the Webinar can be downloaded from here.

What is "Big Data"?

2011-09-05T20:43:00.000-07:00

One of the most commonly used terms today is Big Data, it is regularly used in blogs, product launches, architecture documents and speeches just to name a few. Big Data is being used to describe products, capabilities, features and new ideas about how to build and manage many of today’s new applications and the data that drives them. The struggle is that Big Data has different meanings to different people, and there in lies the problem. For any technology to survive the test of time in IT, it must be understood and accepted by a large enough segment of the user and administration population that the term and associated products evolve into a self-sustaining ecosystem.

I want to outline what I see as Big Data and the common definition I use for people that are struggling with the problems that Big Data often includes and helps to address. The most common emerging definition of Big Data is one that includes one or more of these three parameters – Volume, Velocity and Variety. There are many different ways to define Big Data, but I believe that by using these three parameters, you can clearly define what problems fall into Big Data versus traditional data management and analysis.

Volume – Volume is the measure of how much data a company has under their management, operation and analysis. Volume is typically measured in Gigabytes, Terabytes or Petabytes. Volume is not only an absolute number of current capacity, it can be expressed in data growth over time. Defining what Big Data is takes an evaluation of the companies total data-assets and their growth over time.
Velocity – Velocity is the time that elapses from the time a company receives a new data point to the time they must act on it and make a decision. This decision could be to make changes to a stock portfolio, change the pricing for a product or trigger the staff to make a change to the environment. Big Data typically contains customers that have a velocity requirement of at or near-real time decision making every time new data is received.
Variety – The third parameter that defines Big Data is Variety. Variety defines the types of data a company utilizes within its analysis tools, its customer applications and its business-driven workloads. Big Data customers are typically characterized by a multitude of data including user information, movies, pictures, GPS data, log files, and sales information or. While storing these types of data is not new in IT, Big Data has brought about users that make connections between data that previously was left in islands to analyze, manage and manipulate.

Now that we have defined Big Data – one or more parameters of Volume, Velocity or Variety, we can look at how people are using Big Data in their environments to drive better decision making for a company, faster responses to customer demands and more accurate forecasting of possible business trends. There are a variety of common themes used in Big Data environments:

Related but Unstructured – Many Big Data environments have lots of related data, but that data is unstructured. These types of data could include movies, images, log files or users. While all this data has an association with one another, that association is constantly changing based on how each of these items changes over time and what questions people are trying to ontain from the data sets.
Traditional Tools Don't Scale – The Big Data ecosystem is evolving rapidly with new tools for storing data, managing data, analyzing data and finding new uses of that data. These new tools have come about because typical tools for data storage do not scale to support the volume, velocity and variety that are common for Big Data.

Now that we have looked at what defines Big Data, as well as what commonality can be found with Big Data, who are some of the common consumers and operators of Big Data? How are they using their Big Data environments?

Facebook – Facebook is the first name that comes to mind for a lot of people when talking about Big Data. Facebook has an example of all three parameters to Big Data. Their Volume of data is well into the Petabyes and growing daily. The Velocity at which they must receive a piece of information and make suggestions to others based on that information is measured in seconds and the Variety of data that Facebook stores includes movies, pictures, places, users, usage information, log files and suggestions just to name a few.
Amazon – Amazon has a Big Data environment, and one of the most well known features of Amazon driven by Big Data is it's recommendation engine. Every time you purchase an item from Amazon, a list of suggestions is made at near-real time of other items that may interest you based on previous users recommendations and purchases. This Big Data need is driven by the immediate need for recommendations, Amazon can not reasonable run batch queries and recommend other items an hour after you finished your previous purchase.
Linkedin – Linked in uses Big Data to make recommendations on both contacts that you may know, as well as jobs that you may be interested in. Both of these problems are solved through the analysis of large sets of data based on constantly changing relationships and associations.

By using a companies data to it's full advantage, companies can use Big Data to be more efficient at business operations, more connected to users needs and more rapid to respond then competitions. But the concept of Big Data is only so useful, to really exploit these capabilities there must be tools that allow companies to quickly store, analyze and utilize their growing data sets. Some of the most common tools for exploiting Big Data are:

Hadoop – Hadoop is an Apache project and one of the more commonly used tools by customers that have Big Data. Hadoop provides a framework for storing and analyze large data sets with no restrictions on what types of data can be stored and analyzed.
R – R from Revolution Analytics provides an extremely powerful set of libraries and capabilities for analyzing large data sets, finding data associations and creating applications that exploit Big Data.
Accumulo – Accumulo is a recently released tool of the United States Government for utilizing data stored in a Big Table format.
HPCC – HPCC is an open source tool from LexisNexis to enable companies to store and process large, complex data sets that typically required proprietary technology to analyze.

Big Data is a powerful new concept within today’s IT environments. Implemented through a variety of tools. Big Data solutions enable companies to analyze data in new ways, enabling new levels of productivity and response to customers in new, rapid methods.

Dell | Cloudera solution for Apache Hadoop

2011-08-04T04:40:00.000-07:00

Public announcement is today!

http://dell.com/hadoop

http://content.dell.com/pr/en/corp/d/press-releases/2011-08-04-dell-hadoop-support.aspx

http://bartongeorge.net/2011/08/04/introducing-the-dell-cloudera-solution-for-hadoop-harnessing-the-power-of-big-data/

http://www.theregister.co.uk/2011/08/04/dell_cloudera_hadoop_stack/

Hadoop Whitepaper - Hadoop Business Cases

2011-07-05T21:21:00.001-07:00

I have posted the third in a series of Hadoop Whitepapers I have written for Dell. The third can be downloaded here. The topic is Business Use Cases for Hadoop.

A Blueprint for Cloud Security

2011-06-25T07:21:00.000-07:00

Time and time again, the most common question regarding cloud computing is about security. The question comes in many forms including how do I ensure my data is secure? or how do I guarantee unauthorized people do not access my data? or even my processes are not set to accommodate servers we do not own? These questions and many more, all come from the same fear, a reluctance to make a major change in architecture and strategy because of a lack of understanding, lack of tools or lack of knowledge.

Security is a broad topic, it can encompass many components including the network layer, server location, data center access, data storage devices, application architecture, logging, authentication, monitoring, business process or compliance with regulations, just to name a few. The focus of this article is best practices for ensuring that a plan for security in a cloud environment is complete and well planned for.

First, what is cloud computing in the context of security? In this context cloud computing is the use of computing resources that are provided from shared servers, data centers and environments. Cloud computing in the sense of security blurs the lines that traditionally separated the physical components of one application or company from another. This shared aspect of cloud computing is important to planning security, because historically, many security policies assumed the server and data was physically located in a data center the company controlled. This is no longer the case, companies may have thousands of IT resources they will never physically touch or have control over more then a remote login and the use of the resource.

As with all discussions around security, we must make some assumptions, these set the base of our understanding and focus our later best practices within the context of security for cloud computing:

Assume elasticity – Cloud environments can scale up and down quickly, from a security perspective this means things are constantly changing and the security policies, models, processes and tools must automatically support this dynamic environment.
Shared physical locality – Through the use of cloud computing, you will inevitably have your application on a server that also has applications from other companies, these can be partners, competitors, aggressors, hackers, or customers. Security policies within a cloud environment should accommodate this proximity to possible threats.
Data will take one of three forms – at rest, in transit or in process – Security policies for cloud computing should accommodate data state and ensure that all states are adequately protected, and that data is passed securely between states.
Physical security can not be guaranteed – Many cloud providers have instituted physical security well beyond what was possible in a corporate managed data center. This does not mean that all is without risk. Any time a resource within a shared facility is being used, the potential for the equipment to be physically accessed is a possibility. Data, process and applications should be architected to accommodate this without risk of data being compromised or availability being impacted.
Assume the server could disappear – Expanding on the above assumption, cloud security solutions should assume the server housing the data could disappear without warning. There have been documented cases of servers that house multiple customers being seized as well as servers failing and not being returned to service. Security plans should ensure that should a server disappear, the risk of data loss is as low as possible.
There is no edge – There is no longer a distinct line that can be drawn for where people will access a cloud based environment or where it will be managed from. This lack of a clear boundary that once would accommodate a firewall, must now be guarded by policies, monitoring, intrusion detection and application penetration testing.

Now that we have reviewed the assumptions behind all cloud environments, we can list some of the best practices (In no particular order of priority) for managing security relating to data and application access within a cloud environment:

Centralized Authentication and Authorization – Any cloud based environment should use a single, centralized method for authentication and authorization. This ensures that any rogue accounts that are created can be quickly identified and accounts can be rapidly disabled for those that no longer need them. This single mechanism for authentication and authorization should cover both the staff that manage the application and data, as well as the users that access and consume the application and associated data.
Centralized Key Management – The use of encryption for data storage and validation should be employees across all cloud environments. These should be implemented via a centralized key management solution so that data access can be revoked if necessary. A central key management solution will enable staff to provide access to data to those that are authorized, and ensure that access is removed when warranted.
Encrypt all at rest and in transit data – All data, not actively being processed, should be encrypted. This includes system log files, databases, unstructured data and data the application generates while running. While this has a high overhead in CPU cycles and time, the risk of missing data that should be encrypted is often too great to ignore.
Security handle in process data – Any data actively being used by the application should be handled in a way that minimizes the risk of exposure between processes, applications, users, and virtual machines as well as to stored in a location that is persistent, and encrypted. Logging should be done in a way to minimize the exposure of user data to those that are troubleshooting the environment. All in process data should be handled as short a time as possible to minimize the risk of exposure.
Use of host-based firewalls – As with all traditional security best practices, going back many years, host based firewalls should be utilized on all systems regardless of internal or external access.
Regular penetration testing by outsiders – Any company providing a publicly available site, hosted via a cloud computing solution, should employ the services of an external firm to periodically execute a vulnerability assessments and complete penetration testing of the environment. This outside perspective is important to review and test the design and implementation of the application and data security controls.
Staff Training – Training is critical for all team members that are expanding into roles including cloud computing. All staff need to be educated on the new process requirements, the new rules for deployment and the methods in which cloud computing is being employeed. This training ensures staff are comfortable with this new technology and working from a common base of knowledge and experience.
Accountability – Staff should be held accountable for what and how they expand into the cloud. Risk assessments should be done prior to large, complex changes to ensure staff have adequately assessed the risks, planned for mitigation strategies and implemented safeguards. Staff should be empowered to suggest changes and make improvements.
Change Management – Automated solutions for change and configuration management should be utilized to ensure that all software and servers deployed meet the same baseline standards for configuration. Change and configuration management systems simplify deployments and minimize the change that an oversight leads to a vulnerability.

It is a new world out there, IT managers have more options then ever before when considering how to deploy and utilize new services. Cloud computing is an entirely new way of thinking for many people and creates many new opportunities for scale, efficiency and improved operational models. Despite all that, there are three truths that we must account for when deploying solutions in cloud environments:

There is no edge any more, people consume and create resources from a multitude of places and from a variety of devices.
There is no stable state, applications are elastic and change regularly, we simply can not have a security checklist for new servers any more, we must use process and automation to ensure compliance.
Data is king, the amount being produced today is monumental, and it has huge corporate value. Data must be protected, regardless of state in a variety of circumstances that are no longer under the control of the companies data center manager.

Hadoop Whitepapers

2011-06-06T08:15:00.000-07:00

Below are links to two whitepapers I recently wrote and published as part of my role at Dell.

Enjoy!

Cloud Computing, what is it really?

2011-04-18T22:52:00.000-07:00

I have been in a lot of discussions recently where Cloud Computing is being discussed more then before. I wanted to take a break from EITRP and talk about this term and it's increasing use. Most of the definitions of Cloud Computing include the following words in some order and combination: elastic computing, remotely delivered, associated with SLAs, end user platform independent delivery method, scalable, and billed by usage.

Essentially, most people look at Cloud Computing as a change in the delivery of an application from a company run data center to a managed, remote facility that provides SLAs for the application and bills for actual usage of the application and the amount of data being stored. Most definitions of Cloud Computing also speak to underlying technologies like virtualization, elastic scalability, or hyper-scale data centers.

I believe that Cloud Computing is less about the technology and delivery method, and more about how people think about their information technology (IT) needs and operate what would traditionally be their IT departments. Cloud Computing is more about efficiency, essentially reviewing all current application and data needs, and instead of modifying existing processes to fit changing needs, it is about creating new methods for application delivery. I look at Cloud Computing as the point in time that IT leadership stops using old ways of deploying servers, installing applications and defining the method folks will use those applications. Cloud Computing is about IT leadership embracing and creating new ways to be more efficient with service delivery and how that delivery spans business, process, technology and finance.

Physical Infrastructure

A lot of interpretations of Cloud Computing involve eliminating a company of it's server infrastructure and utilizing shared billable resources from companies like Rackspace, ServerPronto, Amazon, or 1&1 just to name a few. This is commonly called Infrastructure as a Service (IaaS).

Cloud does not mean you ditch all your servers and use Amazon for your core business operations, it means you throw out all your processes when designing new solutions and design solutions around modern delivery and operational processes and newer proven technologies. This is a difficult change for many organizations, IT has developed many habits over the years and other teams have come to expect things from IT done in a certain way.

Staff Infrastructure

There is a common perception within IT departments that staff can be cut back or eliminated by the use of Cloud Computing. This perception largely builds out of the expectation that Cloud Computing is the 100% use of hosted infrastructure, and enables a company to function without the traditional roles of System Administrator, Storage Engineer or Operations Staff.

This belief is often misplaced. By utilizing IT environments that are more automated and/or hosted at alternate locations, the need for IT staff does not diminish, the skill sets that are needed and the required expertise changes. Many IT organizations that are utilizing new methods for operations that are born out of Cloud Computing are finding that traditional IT operations roles are become more architectural in nature and the skill sets are more closely aligned with Software Engineers then system administration staff.

Traditional IT roles involved in deployment of servers will be more focused on capacity planning, knowledge transfer with the application development teams and strategy development for the use of new and emerging tools and technologies. Modern IT departments that have embraced these new engagement models, look to the other organizations to drive tools based on their business needs, while the IT organization provides thought leadership around implementation and strategy. IT staff can not longer be just technical experts, they must understand the business, the financials of the company, the organizational goals and how that relates to technology.

Cloud Computing is not about the use of some new modern technology, although that is a by-product. Cloud Computing is not about throwing away your servers and buying cycles from other companies, but that is often done as part of it. Cloud Computing is a new method of thinking about IT. It is about starting with the business requirements and needs of an organization and enabling those need through the most automated, efficient way possible. This will be a combination of new skills for staff, new use of technology and different models for procuring, using and retiring resources.

Most importantly, Cloud Computing is not a product to be bought. It is a set of items, methods and best practices for deploying the multitude of products, tools and solutions that IT has at it's disposal and utilizing new ones for efficiency when they become available. Cloud Computing is a mindset for efficiency that enables dynamic businesses.

Now I want to touch on Enterprise Information Technology Resource Planning (EITRP) before I close out this post and show how resource planning across an organization enables a more dynamic IT environment that is sensitive to the business's needs. EITRP enables organizations to map the organizational goals, processes, rules and resources that a company has at it's core. This mapping enables clear modeling and simulation of changes to the environment. Cloud Computing provides the operational model for implementing the resources documented within EITRP.

Explanation and definition of EITRP Associations

2011-03-13T22:37:00.000-07:00

This is my fourth posting in the series to introduce EITRP. This posting focuses on the concept of 'Associations' within EITRP.

Enterprise Information Technology Resource Planning (EITRP) is a new method for documenting and modeling the resources that contribute to the creation, operation and retirement of assets, policies and systems in modern Information Technology (IT) environments. EITRP enables organizations to clearly document the resources that contribute to and guide how an IT department implements and manages the necessary tools for a company to conduct business. IT departments can be represented and described in a similar way, EITRP exploits these similarities to provide a consistent method for documenting the life-cycle of an IT environment.

The core of EITRP representations of an organizations is the idea of 'Tiers.' 'Tiers' are representations of the common building blocks of all organizations, infrastructure and resources supporting that infrastructure. 'Tiers' ensure that any EITRP model can be referenced, simulated and documented in a consistent manner. Each 'Tier' is unique in the data is contains and provides for standards of documentation.

A 'View' is a subset of an EITRP model formatted and limited to the components necessary for a given role within the organization. A 'View' is a representation of what staff levels within an organization will own, manipulate and utilize the data contained in one or more 'Tiers.'

Building on the concepts of 'Tier's and 'Views' we add 'Associations' to our glossary for EITRP. An 'Association' is the documented relationship of a piece of information(an object) contained in one 'Tier' to it's corresponding information in another 'Tier'. 'Associations' define how a change to one 'Tiers' parameters will affect another 'Tiers' parameters.

'Associations' allow a 'Model' to be built for an organization that shows not only the present state of the organization, but the affect of changes within that 'Model.' 'Associations' allow EITRP to be used to document current information, future state information and what changes will occur as a migration or change between states occurs. 'Associations' are an important part of EITRP for documenting information including dependencies, costs, implications and resource needs.

There are three types of 'Associations' that can occur between objects in individual 'Tiers'. When referencing an 'Association' we always start with the lower 'Tier' in the relationship, moving to the higher 'Tier' in the relationship.

one-to-one (1t1) – A one-to-one 'Association' is defined as a link between two objects, each in a separate 'Tier'. Each object if manipulated, will only affect the other single object within the 'Association.'
one-to-many (1tM) - A one-to-many 'Association' is defined as a link between greater then two objects, one of which is in a lower 'Tier' and the remaining are contained in a higher 'Tier'. The only object will affect many if manipulated, where as the multiple objects, if any are changed, will affect the single object in the lower 'Tier.'
many-to-many (MtM) - A many-to-many 'Association' is defined as a link between greater then four objects, two or more of which are in a lower 'Tier' and the remaining are contained in a higher 'Tier.' The lower 'Tier' objects will affect multiple if manipulated, as well as the multiple higher 'Tier' objects, if any are changed, will affect the lower 'Tier' objects.

Above are two simple examples of a set of 'Tiers' within an EITRP environment, and the associated objects in those 'Tiers' and their 'Associations'. These 'Associations' are simple and only show the basic type of 'Association' but could also contain information about what a change to one component will do the other associated component.

The primary purpose behind 'Associations' is to enable EITRP models to be used for automation within an IT environment. As IT environments continue to grow, it will be more and more vital to automate as many processes, changes and evolutions of the environment as possible. EITRP enables the necessary information about the environment, it's resources and it's parameters to be documented in a way to allow for automation of changes over time.

Future postings will cover additional topics including:

'Model' – The definition and uses of a 'Model', the overarching term for a complete set of data points across all tiers, including 'Associations' that provide for a representation of the organization and it's resources.
EITRP Data Standards - I will publish the initial version of the EITRP Data Standards, there are the methods for documenting information in a standard way within the EITRP 'Tiers' and 'Views' to allow that data to be referenced, manipulated and modeled for changes within the environment.

Explanation and definition of EITRP Views

2011-03-03T23:28:00.000-08:00

This is my third posting in the series to introduce EITRP. This posting focuses on the concept of 'Views' within EITRP.

Within Enterprise Information Technology Resource Planning (EITRP) 'Views' are a representation of the staff that will interact with each of the 'Tiers' and utilize the information they contain. A 'View' is a subset of an EITRP model formatted and limited to the components necessary for a given role within the organization. A 'View' is a representation of what staff levels within an organization will own, manipulate and utilize the data contained in one or more 'Tiers.'

The example 'Views' below are based on the most common job roles within an Information Technology (IT) department at a medium to large company. The actual defined 'Views' for a companies EITRP models will be specific to that organization and updated to reflect changes in the companies structure, management styles and organizational needs. The 'Views' represented below are meant as a guide for companies that are adopting EITRP and can be modified as needed over time.

The most common views that will be utilized in some form at most organizations are:

Chief Information Officer (CIO) – The CIO will most commonly be responsible for the process and associated data within an organization to execute on the companies core business requirements. This role will require the CIO to influence and drive components contained in the higher 'Tiers' related to company rules, process and policies.
Chief Technology Officer (CTO) – The CTO will expand on the CIO role, while influencing those policies and process, the CTO will also own and define the SLAs the IT organization will work towards at the need of the business and the services that will be delivered to the business for implementing the processes and policies above.
Architect – Architects within the context of IT are responsible for taking the defined SLAs and services from the CTO and turning them into a design that can be implemented across their organization or company wide. The Architect is responsible for managing the risk and reward tradeoff analysis of the possible solutions for a given need within the environment.
Administrator – The Administrator is the role most commonly associated with the implementation of the environment, based on the design from the Architects. The Administrators primary responsibility is to implement new designs and provide feedback to the Architects for use in future designs.
Operations – Operations staff are the first team to respond to abnormalities within the environment. Operations staff are responsible for monitoring for compliance with SLAs and taking prescribed action to remedy all non-compliance.

In larger organizations, it is possible that each 'View' will have multiple individuals that fill that role. Smaller companies could then have a situation where a single individual falls into multiple 'Views.' It is important that when a company is adopting and defining their EITRP 'Tiers' and 'Views' they account for the current structure and needs, but include provisions for the evolution of the company. This will allow the EITRP model to evolve over time and properly account for changes, growth or contraction within the organization and how that change affects the resources managed by the IT organization.

One core component of EITRP is to create a 'Model' that encompasses all resources within the IT environment and their associations. This concept of 'Views' is important for ensuring that as new 'Models' are created, the proper 'Associations' are in place for the evolution of the organization, allowing company leadership to better anticipate how changes will impact the organization.

Future postings will cover additional topics including:

'Associations' – The relationship of a data point in one tier and how it relates to the data point in another tier.
'Model' – The definition and uses of a 'Model', the overarching term for a complete set of data points across all tiers, including 'Associations' that provide for a representation of the organization and it's resources.
EITRP Data Standards - I will publish the initial version of the EITRP Data Standards, there are the methods for documenting information in a standard way within the EITRP 'Tiers' and 'Views' to allow that data to be referenced, manipulated and modeled for changes within the environment.

Explanation and definition of EITRP Tiers

2011-02-19T23:03:00.000-08:00

This is my second posting in the series to introduce EITRP. This posting focuses on the concept of 'Tiers' within EITRP.

Enterprise Information Technology Resource Planning (EITRP) is a new method for documenting and modeling the resources that contribute to the creation, operation and retirement of assets, policies and systems in modern Information Technology (IT) environments. EITRP enables organizations to clearly document the resources that contribute to and guide how an IT department implements and manages the necessary tools for a company to conduct business. The majority of the IT environments that are active today can be represented and described in a similar way, EITRP exploits those similarities to provide a consistent method for documenting the life cycle of an IT environment.

Each 'Tier' is unique in the data is contains and provides for standards of documentation. Every 'Tier' has relationships with one or more 'Tiers' for data dependency and references. These associations ensure that simulations can be done of environmental changes based on the data stored in an EITRP model and the necessary dependencies are properly referenced, updated and correlated. The following graphic and bullet points explain the tiers as they are contained in the EITRP model:

Company – The 'Company' tier is used to document details and information regarding the organization the EITRP model is being developed for. The syntax standards will include methods for documenting all legal entities of the company and subsidiaries. The company tier will also be used to document information about staff, and company locations.
Business Rules – 'Business Rules' are all policies for the company that influence how business is conducted. These rules will include hiring of staff, revenue recognition and financial reporting policies as well as industry specific policies regarding data retention, documentation, corporation governance as well as all associated business rules.
Business Process – The 'Business Process' tier is the location for documenting all work steps necessary to implement the controls and safeguards documented in the 'Business Rules' tier. This 'Business Process' tier is used to define all work flows within an environment for ensuring smooth operation of the business.
Policies – 'Policies' are derivatives of 'Business Rules'. 'Policies' are the rules that will be implemented within the technical environment to enforce the 'Business Rules'. These 'Policies' will be used within the EITRP model to validate that 'Business Process' are being followed and 'Business Rules' are properly implemented. 'Policies' will take the form of documenting who can do what, when and how they can do it.
Service Level Agreements (SLA) – 'Service Level Agreements' are the objectives the IT organization will use when defining all IT solutions. These will be used to determine technical measurements including response time, capacity, availability and scheduled downtime to name some of the measurements. This list will be used as the baseline for defining how the IT environment will be measured for success after each new evolution of the environment.
Services – The 'Services' tier is the documented list of applications and services that are exposed to the users within the 'Company.' 'Services' define how a specific application is accessed, what methods are available to access it, what that access provides and associates the 'Service' with the appropriate 'SLA' and 'Infrastructure.' Examples of 'Services' could be DNS, web sites, or APIs.
Infrastructure – The 'Infrastructure' tier is used to document the types, models, locations and configurations of all the supporting hardware and solutions for the 'Services' tier. This tier will include documentation of a companies servers, storage, data centers, network devices and the associated configuration parameters for those devices. All previous tiers have focused on definition of information and documentation of that information, this tier focuses on the design and documentation of physical components to support the higher tiers.
Data – The 'Data' tier is used to document the types of data a company creates, manages and requires and the associated parameters for that data. Parameters for the data could include how the data is used, access criteria, retention criteria and security policies. The 'Business Rules', 'Business Process', 'Policies' and 'SLA' tiers will influence the implementation and operation of the data represented by this tier.

Tiers are an important part of the developing EITRP framework. 'Tiers' allow a consistent representation of a corporation, documenting how the companies goals, policies and procedures are influenced by the IT environment and how the IT environment enables the companies goals and policies.

In my next post I will discuss 'Views.' 'Views' is a concept within EITRP that is for representing what staff have a primary and secondary role with each of the 'Tiers.' 'Views' define how an organization manages the IT environment and how policies go from being defined to being implemented. After discussing 'Views' we will continue with defining EITRP Associations and how each 'Tier' is influenced by each other and what dependencies are created and documented in each environment. These dependencies will then be used to implement the modeling capabilities for a company to understand how a change in one 'Tier' will affect change in a separate 'Tier.'

An introduction to EITRP

2011-02-13T23:42:00.000-08:00

Modern Information Technology (IT) departments create, consume and manage a variety of resources. These individually include data, people, processes and infrastructure; collectively this includes the connections between these individual components. IT today is primarily about enabling the core business of a company, be it manufacturing, software development, oil and gas exploration or car sales. Each IT organization at these various types of firms has the same primary focus of enabling the business and its leadership to be successful in how they lead, transform and delivery on their core business.

Enterprise Information Technology Resource Planning (EITRP) is a new paradigm within computing to enable all resources related to IT to be articulated, modeled and documented in a single, consistent syntax. This enables strategic planning within an IT department and its relationships with the business needs that it supports for the rest of the corporation. EITRP is a new set of standards being developed for the purpose of articulating the resources managed by an IT organization and how those relate to and affect the policies and processes critical to execution of a companies' core business function.

The potential for this consistent syntax is that IT organizations can standardize how they communicate information about all IT resources throughout the management chain. This standard method of communication will enable leaders at all levels to simulate changes to one portion of the environment or one resource and see how that change will affect the related components, resources and processes. This simulation and modeling capability will allow faster time to market and more rapid deployments within IT by fully understating how and change will ripple through and organization.

This is the first of multiple postings about EITRP. Future postings will outline the tiered structure of EITRP, the syntax for documenting a corporations' environments and the formulas for consistent calculation of costs, changes and modifications to the resources within an environment.

Considerations for always-on environments

2010-08-12T23:52:00.000-07:00

In today’s information technology (IT) departments, more and more applications are becoming 24x7 in nature, requiring availability at all times, with minimal or no downtime to accommodate upgrades, patches, troubleshooting or maintenance activities. This is a tall order with today’s integrated solutions. Ensuring that all pieces of an environment work regardless of maintenance, outages or upgrades is a complex challenge, often only addressed by companies with large resources at their disposal.

There is no magic bullet to solve all availability problems. But, if you are in a position to develop a solution interactively and own the entire infrastructure, there are considerations that can be planned for ahead of time to provide significantly better available of an application than could be obtained through traditional methods for supporting always-on type applications.

The most common challenges in today’s always-an environments are:

Hardware Failures - Hardware fails, there is nothing that we can do about that. The more hardware you have, the more components that will fail. All modern environments should be designed to ensure that critical applications and services are not taken offline by failed hardware.
Facility Failures - Modern data centers are a complex combination of traditional construction, complex power generation and distribution, cooling and environmental controls. These systems, while redundant, can still fail for a variety of reasons. Additionally, data centers in many locations are susceptible to environmental disasters. All modern application environments should span multiple locations in a way that ensures a failure of the application is not caused by the failure of a single facility.
Upgrades Must Occur - Upgrades are one of those activities within IT that must occur. There is a lot of flexibility around when and how they are completed, but eventually, all hardware and software will need to be updated to ensure new features are available, the solution is secure and stable.
Rapid Growth - Today’s IT environments are growing at a very rapid pace compared to even 5 years ago. Many organizations add new servers and applications on a weekly basis. This growth must be managed at the same time that legacy systems are upgrades, eliminated and supported through all this change.
Data - Both data size and the life cycle of that data is becoming more complex to manage. More and more data presents challenges to upgrades, backup, recovery, available bandwidth and a list of other pieces of the environment. As data grows, it also becomes more valuable to the organization and the life cycle of the data much be managed in a more automated way to ensure that data is deleted when required and and the flip side, is available when required.
External Dependencies - No matter how many resources and skills a company has internally, almost all firms have to use a third-party product that is developed by an outside firm for some portion of their IT environment. These third-party products present challenges in integration, management and support that consume time and resources from the IT department using them.
Shifting Loads - Very few solutions today have a consistent load on the environment every day, and every hour. This variation adds complexity by requiring a constant monitoring of capacity and adjustments based on user demand.
Variety of Delivery Platforms - Most applications today are delivered via a variety of platforms including web browsers, smart phones, exposed web services or appliances. Each of these requires a separate set of development and testing procedures, each has separate security policies and all have very different methods for storing, rendering and handling dynamic content. Modern applications must support a seamless experience across all of them for each individual user and their preferences.

While that is not an exhaustive list, it is a good intro to the kinds of challenges that are encountered in most IT departments today. In order to overcome those, many shops develop their own software to ensure they have features and capabilities to make the above challenges easier to manage. Here are some of the most common design considerations that can ultimately lead to design solutions when designing new software within your environment:

Mindset of Rolling Upgrades - In today’s environments, 100% up time is not an unreasonable expectation when serving an application supporting users in multiple timezones and countries. Modern software should be designed in a way that upgrades can be done in a non-disruptive way, allowing users to continue using the application during upgrades and other maintenance activities. While taking the entire application offline is not often an option, software can be written to allow some subset of servers or features to be down for periods of time, while preserving other portions of the application.
Inflight Transaction State Tracking - Modern applications often times will communicate with many different systems across a single or multiple data centers. All this communication must be properly tracked so that an application can recover and a user continue with their transactions, even if hardware and software within the environment fails. It is critical that modern software implement mechanism for tracking the state of communications between servers, this state can be used to recover should a failure of a server or facility occur. This tracking and ability to recover automatically will allow the application to function regardless of the underlying hardware state.
Consistent and Automated QA Processes - Quality assurance within the software development realm has become more and more critical as applications have become more complex and upgrades more often. Having a fully automated regression testing environment will allow new builds of an application to quickly be tested and allow for new tests to be developed as bugs are found to ensure that they do not make it into future releases.

Today’s 24x7 IT environments are extremely complex and very difficult to manage. As they grow and user demands grow, they will only become more complex. Automation is one primary way to stay ahead of this challenge - any process, from QA to deployment that can be automated, should be automated, this increases quality, decreases variation and limits the chance for human mistakes. That paired with distributed software and hardware environments will ensure that as the environment scales, it will not become more susceptible to a failure of any single component. Finally, design all applications assuming that things will fail. By ensuring that applications can recover from failures, either hardware or software, you can ensure that the end user experience is as positive as possible.

The Trend of consolidation in IT

2010-08-02T03:06:00.000-07:00

Consolidation is a common term with Information Technology (IT) departments, it is often used by CIOs and set as a goal for IT departments. While consolidation is a valuable goal, IT departments need to focus less on the concept of Consolidation, and more on setting good, long term habits for the department around hardware purchases, reuse and elimination of assets. Consolidation makes for a good short-term project with defined end dates and targets. What is more critical for IT departments, is to embrace the concept of Return on Investment (ROI) and ensure that any solution deployed has a solid ROI for the business and properly factors in all costs for the life-cycle of the purchase. The more focus on long term planning and ROI calculations, the fewer times an organization will find itself in a place that it fells it must start a Consolidation project. By using the ROI calculations during all projects it ensures the IT portion of the business is as efficient as possible.

I have used this term consolidation multiple times already, but what does it really mean? Within the context of IT, consolidation is most often the process of reviewing all applications and the servers they are hosted on and getting rid of components that are not necessary any more with the goal of utilizing the most up to date technology to better match capacity needs with actual capacity.

In my mind, more important then working on consolidation projects, IT can better utilize their project time and resources by looking at Return on Investment (ROI) for the entire enterprise while working the many projects that make up an IT environment. By reviewing ROI as part of projects a company can ensure that the solutions being implemented will be the cost-effective over the life of the project and not require large consolidation projects that consume time, money and human resources. ROI can be influenced both positively and negatively by a regular update cycle being defined for both hardware and software. This regular cycle ensures that all projects are reviewed for changes on a regular basis to ensure a chance is made to go with modern hardware, less capacity or different solutions to minimize cost.

What type of things should be reviewed by all projects to ensure that capacity does not grow so large that a consolidation project is needed?

Reducing hardware - As part of all new projects, updates, changes or capacity requests the hardware needs should be closely reviewed to ensure that the amount of hardware is not excessive, but rather just enough to handle the availability needs, performance and geographic distribution. A capacity plan should be developed to ensure that additional capacity can be added as needed, but before performance is negatively impacted.
Reducing the number of vendors - ROI for all projects should include a review of complexity and the associated cost of managing that complexity. Consolidation often targets reducing the number of vendors with an IT environment. Carefully reviewing vendors at the beginning of projects can ensure that all vendors have a strong reason for being added and adding costs to projects related to vendor management, support escalation and compatibility matrices.
Reducing the number of instances - Often times companies will have multiple copies of applications or data sets, often referred too as instances, running within the environment. These are created for a variety of reasons, but often contribute to high administration costs and difficulty when auditing the environment. Projects should carefully review the use of multiple instances and ensure that there is a valid business need prior to deploying a solution that could require consolidation or high administration costs after initial deployment.
Rotation of hardware - All hardware will eventually hit the end of it's useful life and have to be upgraded or replaced. To minimize the need for consolidation, new projects should plan during implementation what the minimum and maximum life-cycle of the hardware for the project will be. This will ensure that as the capacity needs grow, hardware can be rotated in for the project to ensure capacity and needs match.
Upgrades of software - Software requires regular patches and upgrades to ensure stability and the ability to easily apply required patches later in the lifecycle of the solution. Projects should include adequate time for applying incremental patches as needed to eliminate the potential of costly downtime periods and staffing resources should a critical patch become available, but not be able to be applied due to missing prerequisites. All project plans should include adequate time for upgrades, maintenance and testing to ensure maximum stability and manageability.
Placement of new applications - Often times the assumption is made that a new application requires a new server, or some variation on that theme. All projects should include a careful review to ensure that all capacity being added will be utilized at it's maximum levels and ensure that duplication of services, hardware and capabilities are not being added into the environment.
Proactive addition of capacity - One method of ensuring that consolidation is not needed down the road, and your IT environment is running efficiently is through the use of capacity planning, and proactively adding the necessary capacity to handle demand and by using centralized management of capacity. This centralized management of capacity will ensure the environment has capacity to handle growth, old hardware is adequately retired and that capacity being added is based on adequate planning and not guesses or last minute panicked needs by end users.

I have used the term return-on-investment (ROI) several times. What components make up the ROI for an IT focused project:

Current administration costs - These administration costs include the physical power and cooling necessary to run the servers for a given set of applications and the associated data, the labor necessary to upgrade, patch and keep the environment stable and the maintenance costs for any hardware or software being used in the environment.
Delta in new solution administration costs - The delta, or difference, in the cost of managing the new solution being proposed. This delta could be represented in weeks, months or even years, depending on the size and complexity of the project being assessed.

As you can probably tell, I am not a fan of consolidation projects. I prefer to manage an enterprise proactively and ensure that capacity added, is capacity needed. But, that is not always an option. in the event you are beginning to look at your environment for potential areas of efficiency, here are some items to review for possible consolidations savings:

Application review - The quickest way to consolidate is to eliminate duplicate applications and functionality from the enterprise. Review all applications and ensure they have a valid business need, executive sponsor and are not a duplicate for other functionality found within the enterprise.
Hardware review - A thorough review of all hardware in the data center will allow you to catalog it's age, power consumption, speed, memory capacity and use in the environment. This information can be compared to the latest information on systems available and determine if power or space savings can be gained by moving to newer hardware platforms.
Data center costs - Paired with the above data, a review of the costs for space and power for your data center can be completed to see if savings can be found through the use of less space within the data center.

Consolidation is a commonly used buzz word in IT today. Consolidation is most often the use of visualization to cut back the number of servers utilized in an environment, and reaping cost savings from the lower power, cooling and space costs. Consolidation can also include work like eliminating applications with duplicate functionality and eliminating pockets of information that must be managed separately from the rest of the corporate enterprise. While consolidation is a worthwhile goal in all IT departments, it is equally important that IT leadership ensure that as new solutions are deployed, they are done in a cost efficient manner and with the necessary amount of capacity. This will ensure capacity is not sitting idle and will need to be consolidated down the road. IT departments can save both money and time by ensuring that a solid strategy is in place to add capacity, tools, servers and applications in a way that is the most efficient for the enterprise.

Developing "your network" at a new job

2010-07-22T20:09:00.000-07:00

Today, one of the most important tools for any job is "your network." This network is not the one that your computers connect too, nor is it a physical item in your office. "Your network" is the group of people that you have established relationships with that can assist you in doing your job more efficiently. These relationships allow you to quickly gather information, answer questions and complete tasks by quickly knowing who within "your network" to call on and who is best suited to assist with a task.

"Your network" is critical to ensuring you can be a success in any role. As people become more distributed in the jobs they complete and jobs become more complicated, the number of other staff you must work with will quickly grow. By having a network of established peers, you can:

Quickly locate those folks that are necessary to collaborate with on projects to be successful
Ensure that you focus on your tasks and not tasks that another organization is better suited to complete
Have a group of staff to ask questions of, get advise from and look too for mentoring or leadership guidance
Look good in the eyes of your peers and customers by efficiently completing tasks and projects
Have a group of folks to speak to when things at work get stressful
Avoid unnecessary mistakes by learning from your peer's experiences

The most critical time to have a strong network to rely on is when you start a new job, or position within the same company. A strong network will allow you to quickly learn the position and be successful at your new tasks. The most common ways to build "your network" in a new role are:

Call your peers or sit down over a cup of coffee with them. You can use this time to discuss team dynamics, their roles, their capabilities and understand what they do to be successful on the team.
Make a note of important names mentioned during conference calls, meetings and in passing. Reach out to these individual offering a short introduction, and ask about their role. This will help understand when they can be engaged on projects.
The most important item in my mind when building your network is asking questions. As a new member of any team, it is not expected that you know everything on day one. Asking questions and following up with the proper subject matter experts shows engagement, passion and a desire to grow with the team.

Until now, most of what I have written is geared towards developing your network within your primary job, at work and with your coworkers. Another important aspect of "your network" is developing relationships with those that are in other industries and similar roles at other companies. These types of networking opportunities allow you to grow not only in your role, but within your industry and develop as an authority within your field.

Long term, the larger your network, the more opportunities you will have for exciting projects, new roles and assistance on your projects. With the mobility in the modern job market, and the uncertainties around long term, consistent employment, having a large network ensures you always have methods for locating new opportunities and sharing your capabilities with others.

These networks do not develop over night, they often take months or years to develop at any company or within a new role. Patience pays off, when starting in a new role, reach out to as many people as possible to introduce yourself and ask about their roles. Some folks will be more receptive than others, but the key is to share your capabilities, understand other peoples and in time, you will develop a strong network to assist in completing your duties.

Ultimately, some of these contacts you develop may turn into longer term relationships. Some of my best friends are people that I first met as peers or members of my network while at a job or working on a project.

Enabling Users

2010-04-06T23:45:00.000-07:00

The primary goal of any organization responsible for the deployment and support of desktops, laptops and telephones should be to enable the end users to do their job as efficiently as possible. These users often have a primary job that does not include supporting the platform (desktop, laptop, application or mobile device), only using the platform for their primary job. The first step to providing a sufficient level of service is to clearly define your user base, this will help your organization group users based on common skills, common usage patterns and common support requirements. After the user base has been categorized, a support model and list of required tools for each category can be assembled.

By properly grouping your users, IT leadership can more clearly understand how their needs evolve, what their training requirements are and what tools are necessary to enable them to perform their primary job function(s). Grouping users is an evolutionary process that should be reviewed and updated yearly at a minimum, but more often if a companies' structure is regularly changing.

The first step is to create categories to put the various users into, four to five groups is often enough to include all users, but not too many that managing the documentation for them becomes cumbersome. The most common user groups I have seen and worked with are classified as:

Company Administration - This group includes people like executive assistance, accountants, human resources and managers. This group's primary use of Information Technology (IT) resources is to complete their job, and traditionally these staff have little to no IT support or troubleshooting experience. This group will most often have little to no opinion about what applications they use for their job, just that the applications allow them to complete their job. They are completely at the mercy of the IT department for system setup, configuration, upgrades and repair.
Executive Staff - This group is characterized as often having the same IT skills as Company Administration, but needing a higher level of access to corporate resources and mobility because of travel and meeting schedules. This group's primary characteristic is the higher need for the ability to safely use, transport and protect highly confidential company information, as well as access it in a timely manner.
Information Technology Staff - IT staff are characterized as needing high levels of access to a majority of the systems and data within the environment. This group will also commonly need remote access to a majority of the environment to facilitate after hours work, upgrades and outage management. This group is often capable of supporting their own systems for operations like new software install, troubleshooting and upgrades.
Power User - A Power Use is someone that would traditionally fall under the Company Administration category, but is distinguished by their opinions of what applications they prefer to use based on previous experience, as well as their ability to provide basic support for them-self by installing new applications, upgrading applications and troubleshooting the most common support problems they may encounter.
Developer - Software Developers often fall into their own category as a user type because of their need for testing and development environments, in addition to their need for common business tools like documents, email and web browsing. Developers often need places to compile code, the ability to change administrative settings and install differing versions of libraries that common users to not need access too. Developers often require multiple systems that can be quickly built and rebuild as their testing needs change.

After defining your categories of users, you can define a support model for each group. This support model will include what types of systems they are assigned, what level of support they get during business hours and off business hours and what software licences will be acquired for all staff in each respective group. These definitions can be used to understand the cost for supporting each group of users, as well as the cost for upgrades when new software becomes available or is requested by large groups of staff. The most common support models, that can then be paired with user categories, are:

Full-Support - This model is typical of providing a help desk that is capable and properly staffed to respond to support requests 7 days a week, 24 hours a day. This model is the most expensive, but is critical if staff are expected to work off hours like weekends and holidays. This model is most commonly used for users in the Executive Staff category and Company Administration category.
Business-Hours Full-Support - This model is typical of providing a help desk that is capable and properly staffed to respond to support requests 5 days a week, 12 hours a day. This support model is most often used for Company Administration staff that require IT support for all issues they may encounter, but do not commonly work after regular business hours. This model of support is less expensive then supporting users 24 hours a day, but creates the potential that staff will not be able to work after hours if they experience severe IT problems after hours.
Partial Support - This model is typical of providing a help desk that is capable and properly staffed to respond to support requests for staff that have completed the first level of support and require an escalation to the IT department for support or resolution. The Power Users and Developers will most often use this model for requesting additional information or an exception to company policies after doing the necessary research to support the request.
Self-Support - This model is typical of providing on-line help for users with no telephone or live component for the most common support requests. This model is typically used for highly capable users that prefer to manage their own IT environment. This model typically provides a ticket or automated system for requesting work to be completed by the IT department.

We have now defined our users, based on their common support requirements and usage. We have paired that with a support model to ensure both cost efficiency, as well as a balance to ensure users can work when necessary and get support quickly when needed. Now we must define what tools each user category gets based on their business and productivity needs:

Desktop Computer - Desktop computers are still the least expensive method to provide access to email, web browsing and network based applications. They can be very powerful for demanding users, but lack the portability to allow users to work from home, or while traveling. While desktop computers are still viable for Company Administration staff, Power Users and Executives tend to avoid them.
Laptop Computer - Laptops are growing as a percentage of the systems in use in most companies because of their flexibility, portability and smaller power consumption. Laptops, while often slower then desktop systems, have advanced in the past few years and provide for even the most demanding users. Laptops are most common for Company Executives and Power Users.
Smart-phone - Smart phones cover a wide range of products including the iPhone, Blackberry devices, and Android powered devices. These are designed to allow for calendar, email and basic document access while on the run or out of the office. These are becoming more common across all categories of workers to enable them to stay in touch no matter where they are during the day. The use of smart-phone like devices will only increase and more and more applications are optimized for mobile access.
Cloud-hosted Virtual Machine - With the popularity of cloud computing, it is becoming very inexpensive to dedicate multiple hosted Virtual Machines to every developer or staff member for development and testing purposes. These are most commonly assigned to Developers and Information Technology Staff. While these do provide a lot of benefits for productivity and testing, they must be implemented to ensure corporate security is maintained.
Administrative Access - Administrative Access, regardless of Operating System, refers to the ability of a single user to install software locally and manipulate local configuration parameters. While there are varying levels of Administrative Access with today's fine-grained access tools, this category most often applies to Power Users, Developers and Information Technology Staff.
System Performance - System performance is a tool defined by the speed of the equipment assigned to a given user, as well as the amount of storage space it contains and physical memory the device has. Commonly speaking, the faster a system an employee has, the more they can multitask and the faster they will be able to complete actions. All users, regardless of category, should be assigned systems that are fast enough to ensure they are not waiting for operations to complete. Human capitol is much more expensive then faster computers.
Bring Your Own PC (BYOPC) - Bring Your Own PC is a new method many companies are looking towards to enable their Power Users by allowing the user to choose the type of system they use for work. This allows the employee to choose the system that they are most comfortable and productive with. It will take some time if this does catch on because of the increased complexity involved for IT departments to ensure various platforms can access common company resources like time cards, email and file sharing. BYOPC also creates security issues that must be managed by the company.

Today, companies have a variety of user categories, each with their own, specific needs for the IT systems they need to complete their job. By carefully grouping users, they can be better understood and provided with the tools and a support model that enables them to be as efficient as possible. This common support model can then drive company planning for costs, upgrades, and other changes. Understanding that not all users are the same is the first step to ensuring that IT provides all users the tools they need to complete their job, regardless of their individual IT skills or opinions.

Capacity planning for business IT Systems

2010-03-24T13:30:00.001-07:00

Capacity planning is a critical part of all Information Technology (IT) environments. It ensures that the number of servers, licenses, physical memory, bandwidth, disk space, room space, etc is sized properly for the maximum return on investment. This ensures that the user experience is positive, while ensuring that unused capacity is not being paid for and managed without a valid return on investment. Capacity planning is ensuring that the workload for a given environment is properly understood and mapped to create a link between a specific number of users and the amount of infrastructure needed to handle those users’ applications. That information is then laid out on a calendar to ensure that the load over time is understood so that capacity can be added and removed as necessary, without a negative impact on user experience.

This document is meant to serve as a guide to what information to consider when beginning the process of capacity planning within your environment. The purpose is to list the most common considerations and strategies for ensuring your capacity plan is an adequate model for infrastructure growth. First, let’s cover some terms and concepts that are important to understanding capacity planning:

Types of Capacity – Capacity is a very broad term, and within the realm of IT there are a variety of places that need to be considered when evaluating current and future capacity needs. While these are separate areas within the same context, each of them directly affects each other. The most common areas of focus for capacity planning are compute power (CPU speed and quantity), memory capacity (both RAM and hard disk), bandwidth (both within a single server and between devices), space (data center, office and storage facilities), data center (power and cooling), and human capitol (staff and contractors for operating the environment, and the relevant skills they posses).

365-day Calendar – Every business has highs and lows in terms of capacity needs. These can vary from month to month, and as often as hourly within the same day. As part of any long term capacity plan, a long-term calendar is needed to show the highs and lows in capacity needs. This should incorporate in holidays that affect the load on the environment, the needs of the business for reporting and trending and any audit needs based on industry specific rules.

User Load relative to Capacity – This is a formal mapping of a specific user load, to a defined amount of capacity with constraints around user response time, availability and user experience. This is the building block for a corporate wide capacity plan and enables staff to understand how much capacity must be added based on user growth. Most organizations will use their Service Level Agreements (SLAs) for setting this relationship.

The process to developing a capacity plan can be long and involve many steps depending on the complexity of the environment, dynamic nature of the work load, and the type of software being using. These are the most common steps (in no particular order) for information gathering as part of developing this capacity plan:

Load Testing – This is the process of testing specific user loads on a known capacity of hardware. These, when done multiple times on varying configurations can develop a capacity model for how many users a set of hardware can support at maximum without performance degradation.

Review of similar environments/workloads – This step is to ensure that knowledge gained within the industry, in similar environments is applied to your capacity plan and your specific environment. This step is not meant to assume some other workload is identical to yours, it is probably not, but there probably are similar workloads that can provide guidance on what to test in the Load Testing phase and what models need to be developed to properly plan capacity.

Trial and Error – A large part of load testing is trial and error. Many environments are simply too large to fully test in a development or test environment. This trial and error can be done in a strategic manner, testing types of capacity needs that are the most likely to be impacted by large or abnormal loads.

SLAs – This is the process of documenting what contractual requirements are in place for ensuring the users get the level of availability, uptime, and performance they expect and have paid for with the service.

The above steps are part of the technical process to developing a capacity plan that is unique to your environment and its needs. These, in addition to carefully documenting when and how to increase capacity can ensure that when the environment hits pre-set triggers, capacity can be added easily, ensuring a consistent user experience.

Capacity planning is ensuring that a clearly defined user experience can be mapped to a specific amount of infrastructure to support that experience. This plan should include not only what increments capacity can be added in, but also what triggers cause that capacity increase to occur. This plan, when associated with a calendar of business needs, trends and holidays can ensure the proactive growth of the environment and a consistent user experience, without having excess capacity that is costing the firm money and not being fully utilized.

Remote Team Dynamics

2010-03-08T08:51:00.001-08:00

In recent years companies have increased the speed at which they downsized offices and subsequently hired more staff "working remote." "Working remote" can include a variety of alternative working arrangements, but is most commonly characterized by staff that work primarily from their home, or the customer location. This has created many teams that the staff are distributed across the country and the world. These remote employees typically have the freedom to work the hours they are most productive, as well as at the location they are most comfortable at, this could include coworking spaces, coffee shops or parts.

One significant change as a team becomes more distributed and remote is that communication channels and patterns must evolve to ensure staff feel the same level of connection that they would if they worked in a traditional office setting. Communication models must adapt to ensure that staff not only feel connected to their team and manager, but that they have effective methods to reach out to their team for discussions, advise and coordination.

I have worked in a variety of roles where my manager and I were in different states, as well as managed teams spread out as far as Australia, while I was based in Texas. This presented a unique challenge in ensuring that all team members had the same information and capabilities to do their job, regardless of their specific locations or timezones. Below are a few of the most successful methods I have found for managing a team that is distributed:

Weekly Team Meetings

The primary method for team communication, pass down and discussion should be a weekly call. This provides a known, consistent forum for the team to discuss changes within the team, within the company and pass down information from management to the team. The focus of the call should be kept on items and topics that are relevant to the majority of the team, sideline discussions should be scheduled at a different time to discuss topics in detail that do not interest or affect the entire group. Regular team calls are a great opportunity to foster team trust. These calls provide a place for team members to share their knowledge and experience as well as allow for open communication on issues that need a second opinion or escalation.

The time of the day that these calls are held is critical to ensuring maximum participation and limiting the impact of the call on the regular work of the team. For teams that are spread across timezones it is beneficial to hold calls at alternating times, either presenting the information twice, or to ensure that if one timezone must be up for a very early call, they do no have to make that sacrifice every week, but other regions have calls at off times periodically as well. Another option is to record the calls so that folks can listen to them at a more convenient time.

An agenda for the call should always be sent ahead of time, this will allow people to prepare for the call. An agenda can also be used to set time limits for various topics to ensure that one topic does not unexpectedly consume the entire schedule time for the call.

Finally, meeting notes should be provided after each call. These reinforce any policies stated to the team and allow the staff a reference to refer too later should they forget what was said or decided on the call. These meeting notes can also serve as the official record for any decisions that require review and approval by the team or management.

Roundtable

Every call should contain a roundtable, this provides all team members a brief period to share lessons learned that impact the rest of the team, and allow folks to understand what their peers are working on and may be able to collaborate on.

Each person's time should be limited so that they may mention one highlight and one lowlight of the week. The purpose is to share lessons learns with the team so that best practices can be shared across the organization.

Alternative Team Communication

In addition to a regular team call, there are several other methods that can be used for communicating with the team and building strong bonds between members, regardless of location.

Team Discussion List - An email distribution list should be available for the members of the remote team to communicate on topics that would normally warrant a hallway conversation. This could be technical discussions, product discussions or questions posed to the team about a customer or product. This forum provides the team a known path for team communication and input.

Team Watercooler List - One item that gets missed a lot with remote teams is the loss of hallway discussions on personal issues or announcements. A separate distribution list should be available to the team for topic discussion that is not immediately applicable to the company, but allows employees on the team to get to know each other better and share good news from their personal lives. This allows a name and personality to be shared for each member of the team.

Remote teams are a new challenge that companies are beginning to experience as more and more staff work from home or other alternative working arrangements. By having regular communication with the team, it allows these staff that are separated to keep in close contact, develop a trust for one another and ensure all team members have quick paths for discussion with the team. Ensuing communication flows regularly ensures that these remote employees feel connected to the team and have the information they need to be successful in their roles at the company.

Defining Seniority in IT

2010-01-14T01:33:00.000-08:00

Seniority within Information Technology (IT) departments has always been a hard thing to define. The technologies in use shift so rapidly and people move jobs quite often; that determining who is the most Senior person on a team is often not as simple as who has been there the longest or who has worked in the field the longest.

Seniority in the sense of this posting is having the position and experience that the fellow members of your team go to you when they have questions, need advice or otherwise need a second opinion on what they are working on. Seniority is also defining the person that your managers are most likely to go to when they need direction on a project, or need to delegate important work.

Traditionally, companies have often set seniority within a team based primary on time within a given job. This is a less adequate measure in IT because the technology changes so rapidly, a person must keep up with both the technology and what are often called soft skills. These soft skills enable an employee to be more flexible in what they work on, and more dynamic in who they interact with within a company, based on the project and needs of the day.

So, ultimately, what makes you as an employee more senior within your team and subsequently more valuable to the organization?

When positioning the more senior staff up with the lower experienced team members, it is a balance of multiple skills and experience types. One does not necessarily replace another, and a truly Senior member of any team must posses all the following skills at a minimum, with an emphasis on some more then others based on the job role:

Contributions (Time, Company Goals, Knowledge) - Contributions are the most important part to establishing a position as a Senior member of an IT team. Contributions can be technical in nature, time or knowledge, but all show other members of the team both your dedication, capabilities, and commitment to the future success of the company.
Experience (Current Technologies, Past Technologies) - Experience with a wide range of technologies and hardware will enable you to make informed decisions and suggestions on future direction and architectures. A wealth of knowledge and hands-on experience will ensure that no matter the problem, you will have experience in how to approach it, even if the specific technology is a new one.
Understanding of the existing IT Infrastructure - A solid understanding of any existing IT infrastructure will enable you to fully understand legacy burdens when making future planning decisions. It also enables you to understand where the company has been and what has been tried so that if something worked or did not, that can properly be taken into account on future solutions.
Understanding of the business problem to be solved by IT - Information Technology (IT) is not the primary, driving factor for the majority of businesses. The majority of the companies out there only use IT as a way to meet their primary market more effectively. The most senior staff in IT must understand not only the IT aspect, but the technology and business behind the companies primary market. This ensures that IT is properly aligned and working towards the larger company-wide goals.
Ability to interact with varying levels of staff and management - Interacting with various levels of staff and management within a company is an important skill. It shows that you not only understand the challenges of each level, but you understand what types and details of communication need to be understood at each level. The proper level of detail and big-picture at each level of communication can ensure quick decision making and solid support from executive management when escalations are necessary.
Understanding of the company direction - Having a full understanding of a companies long term goals and direction allow IT staff to ensure that suggestions, plans and comments will not become obsolete early in the project life cycle. By showing your managers that you understand the direction, allows you to put the companies best interests first and work towards hitting those goals.
Time Management - Time management is your ability to prioritize projects based on deliverable dates and ensure that appropriate forward progress is made on all projects to meet the appropriate targets for delivery. Time management shows senior leaders at the company that you understand the complexities of juggling many projects and can compensate as unexpected items come your way.
Project Management - Project Management is your ability to manage not only your tasks and deliverables, but the dependencies between them and the work of other staff. This type of leadership enables you to work with larger, more complex teams, as well as provide status updates to management on project progression.
Knowledge Transfer (Mentoring) - This is your ability to assist other staff in developing, both company specific knowledge, as well as industry knowledge. The goal in this category should be to develop into a staff member that others are comfortable speaking with for advise and input, knowing that you can provide a unique, relevant insight for them.

Now - the big question - "What about my salary, how does that relate?" - Salary is a difficult subject for some folks. Some people prefer to discuss salary as a very private matter, others feel it is a public topic for discussion. Regardless of a person's choice, their salary is a reflection of the value a company sees in them. If a company is willing to provide a higher salary, they expect a higher level of return. The more traits you posses from the above list and the higher level of development of those traits will translate into your ability to provide more value to your employer.

I hope this has provided some insight into how staff are defined as Senior within IT. Ultimately, it should be your goal to develop the proper balance of the skills listed above, based on your job role. The more experience you can gain in each area and the more expertise, the wider a range of jobs you can hold and staff you can interact with. That flexibility will create value for the company you work for and put yourself at the top of your peers.

Risk Workshops

2009-10-28T01:07:00.000-07:00

Risk workshops are an important part of managing risk for all projects. They are typically done at the beginning of a project and any time a major change is made to the requirements or acceptance documents for the project. Risk workshops are brainstorming sessions to develop a list of risks for a project and determine the factors associated with those risks so that they can be mitigated.

Throughout this posting I use the word project a lot. In this context, project is a defined set of activities with a given beginning and end. Projects are common in all companies to take a unit of work and properly manage that unit of work to completion.

A Risk Workshop is a sit-down, face to face, thorough review of all aspects of a project. This commonly includes delivery schedules, acceptance plans, project financials and contracts associated with project delivery. The primary purpose of the risk workshop is to allow both project participants and outside observers to brainstorm all possible risks that could come up and come to agreement on risk level and mitigation strategies.

There are 2 priorities for all risk workshops. All participants should enter with these at the top of their mind as they look for mitigations strategies and project plan changes:

Protecting the company from financial loss. This can include penalties for delays or missed features, or having to redo work because of low quality.
Delivering the project on-time. All risk mitigation strategies should be worked together with developing the project plan so that estimations are realistic.

The primary deliverable for all risk workshops will be a risk workbook. The most common categories to track for each risk in this workbook are:

Risk Number – A unique identifier for the project so that team members can clearly communicate about each individual risk.
Raised By – The name of the individual who first brought up the risk. This should be tracked in the event clarification is needed about the risk or impact.
Date raised – The date that the risk was first discussed. All notes associated with the risk should have associated dates as well to track the progression of the risk discussions.
E/B/C – Engagement/Business/Customer – This category will define the risk type. Engagement is a risk associated with contractual details or the relationship between customer and vendor. Business risks are a risk that a product lifecycle may change or priories shift for a team. Customer risks are associated with delays on the customer side, either because of pre-requisites not being met or changes to the customer's requirements.
Description – This is the detailed description of the risk, what it affects and any supporting details to what could trigger it.
Risk Cost – This is the monetary cost of correcting the risk should it become a problem during the project. This includes time, facilities, and all associated resources needed to resolve the risk should it become a problem.
Risk % - This is the chance that the risk will occur during the project. This is used in pair with the risk cost to determine a risk budget for the project.
Mitigation Strategy – This category defines the solution for mitigating the risk. This should define what steps will be taken to lessen the chance of the risk turning into a problem. This could include additional staff on the project, earlier testing, or a chance in architecture.
Mitigation Cost – Mitigation cost documents the cost to minimize the chance of a risk occurring. This cost is then compared to the chance of the risk occurring and the cost of the risk occurring to determine if the mitigation cost should be spent, or continuing on the project and managing the risk if it does occur.
Risk Owner – The risk owner is the individual that best understands the risk and associated mitigation strategies. This is most commonly the person responsible for monitoring for the risk occurring and documenting the risk mitigation strategies.
Risk Trigger – Not all risks will become problems and impact a project. The risk trigger is what defines when the risk does become a problem so that staff can take steps to address the problem.

The first part of any risk workshop is to discuss the objective and purpose with the participants. All risk workshops should begin with a discussion of why the team has come together and what deliverable is expected at the end of the meeting. This deliverable will most often be a risk workbook containing all risks and their associated risk level, potential cost and mitigation strategies. All risk workshops should set time limits to ensure that if a discussion occurs on one risk, the meeting time is not overwhelmed. This time limit is too ensure everyone has a chance to speak on the topic. If consensus is not reached in that time frame, someone delegated by management should be responsible for getting input from all parties and making a decision on the risk level and other details.

I can not remember a project that I have worked on that had zero risk, or a list of zero risks. All projects have some level of risk, and the purpose of a risk workshop is to clearly define them and the plan for avoiding delays because of them. A long list of risks coming out of the risk workshop shows that the team was successful in thinking of possible pitfalls and mitigating them. The purpose of the meeting should not be to have a list of zero risks, that is not the same thing as a zero risk project.

As part of the risk mitigation portion of the risk workshop, there are two primary strategies for handling high risk components of a project:

Redesign – Often times a design can be redone to limit, or minimize the risk of a project. The redesign may have other impacts including cost of delivery or schedule impact that must be weighed against the potential risk.
Risk Mitigation – Mitigation is the most common strategy for managing risk. This is the early planning of how to handle a risk, should it become a problem. Mitigation often involves having clearly defined paths for escalation to other teams or additional resources available.

Ultimately the risk workbook will be used to develop a risk budget. This risk budget will be built into the project financials to ensure adequate resources are available to respond to risks if they do become problems, as well as providing funding to cover risk mitigation as necessary.

Risk workshops are a critical component to all successful projects. A risk workshop allows for all interested parties to express any risks they foresee and how to properly plan for and mitigate these risks. Risk workshops should not consume an unlimited amount of time, but should allow everyone to express an opinion to risk levels and allow that to be documented in the risk workbook for the project. Risk workbooks are living documents for the duration of a project and provide a single reference for developing the risk budget and showing mitigation strategies for a project.

Time scheduling for IT Staff

2009-10-03T16:22:00.000-07:00

Information Technology (IT) staff often must juggle both daily demands of user requests and daily repair activities, with long term projects like upgrade testing, capacity planning and new feature evaluation. These two distinct types of work are difficult to juggle, in addition to a never ending array of meetings, office interruptions and service outages. Many IT jobs today are high stress, both because of the level of work to be completed, but as well as the chronic mis-management of time, creating both higher stress levels and lower productivity levels.

As with all professions, the goal with time management, by both staff and management should be to minimize context switching. A context switch is each time a person must change from one task to another; this can include changing project focus, phone calls, office interruptions or stopping a task to goto a meeting. By limiting context switching IT management can allow more time for staff to focus, and provide them clearer blocks of time to complete their work, in a more efficient way.

It is quite common within the IT space to schedule meetings mid-day as well as pull staff into meetings during the day. This is quite disruptive and often not necessary. It is important that managers within IT organizations clearly define what constitutes an emergency and how to properly justify pulling staff away from their daily work load versus planning for a meeting in the future.

Suggestions for minimizing interruptions and increasing time utilization:

Meeting Free Days – Blocking out days specifically for meetings will allow the remaining days to be used by staff to focus, free of interruptions on long term projects, research and other work that is more efficiently completed during a focused period of time.

Set Aside Time for Ticket Based Work – It is very common for IT organizations to have a ticket tracking system to handle incoming requests and common tasks. This should be monitored by a dedicated person; if that is not possible time should be dedicated for other staff for monitoring. Tracking and managing many small requests in the middle of project based work is very disruptive and negatively affects productivity on the long term projects.

Clearly Defined Office Hours – Clearly defining staff's office hours can set a stage for limiting interruptions to minimal times within the day and giving staff dedicated time for focusing on ticket based work and project based work. This will ensure that staff are available for drop in discussions, but that these do not dominate their available time.

Staff Privacy – One method to ensure IT staff can focus and ensure time is used properly is giving IT staff a private office and workspace. All IT jobs require some level of collaboration, but they also require time to focus on projects and work as an individual. This focus requires a place free of interruptions like ringing phones, conference calls, others talking in the hall way and side discussions.

Within IT, time management is important to ensure staff can properly focus on both daily needs as well as long term projects and goals. By minimizing context switching by the use for dedicated blocks of time, staff can have better focus and concentration on their projects, and ensuring completion on time and minimal delay and interruptions.

Importance of Code Reviews

2009-09-19T01:32:00.000-07:00

Code reviews are an important part of the software development process. They are the period during development where a more senior team member reviews the code written by another team member, prior to submission into a companies version control system. Code reviews are a formal process to both improve the quality of submitted code, as well as to allow for mentoring of all developers on the team.

Any time a piece of code is being submitted for eventual inclusion in an application, a code review should be part of the process prior to formal inclusion. This ensures that a minimum of two people review all changes to the software to check for defects. This code review process also ensures that knowledge is duplicated within the enterprise to better manage project transition and long term support responsibilities for all applications.

There are several primary areas that should be of focus for all code reviews:

Company Coding Standards
All companies should have standards for software development. These should include the libraries used during development, the documentation of the code base and the languages used for development. This is the first item that should be reviewed during all code reviews. By reviewing all code for adherence to company standards, it ensures all team members not only follow the standards, but have a chance to learn any standards that they may not be aware of or that may have changed.

Company Enterprise Architecture Standards
In addition to company coding standards, all firms should have a formal set of Enterprise Architecture (EA) standards. These often include how data is stored, managed, tagged, backed up and secured during transport and manipulation. All code reviews should ensure that new code being submitted follows existing company EA standards for ease of interoperability, as well as long term software life cycle management.

Mentoring
Mentoring is a key component of all code reviews. Code reviews allow senior staff to review code of their teammates, and provide them suggestions for improvement based on experience. This mentoring is key to ensuring better long term quality from all produced code, as well as for providing staff a path for development. Each staff member that is having their code reviewed could potentially be reviewing code in the future, so it is key that this mentoring process be official, and an important part of the software development teams culture.

Security
In today's IT environments, security is a critical component of software development. All code reviews should include a portion of time for reviewing security to ensure that input and variables are handled securely, that temp data is cleaned up properly and that host to host communication is handled in a secure fashion, just to name a few.

Security is a complex topic, especially in the software development arena because of the wide range of attacks, challenges and threats. Code reviews allow for a formal process to ensure common mistakes are not made, previous mistakes are not made again and that staff have a forum for discussion of implementation details.

Scalability
Today, many applications are scaling to levels of usage never first envisioned when the application was written. This causes many problems for both the administrators of these applications, as well as the users. Code reviews should ensure that applications are properly handling resources like CPU time, system memory and disk bandwidth as to allow the application to properly scale over time. Scalability is a combination of many components, both the responsibility of the developer and other IT administrators; code reviews should ensure that all code written is properly prepared to scale over time and handle even the most extreme loads on the system.

Coding Quality
Ultimately, the final key of all code reviews is ensuring quality. Quality can come from many aspects of the code base including documentation, ease of understanding of the code and the maintainability of the code. These are all key aspects that if properly addressed and corrected during code reviews can ensure not only better developers, but more manageable code over time.

Code reviews are an important process component for all companies developing software, either for internal use or external sale. Code reviews ensure that staff are formally mentored on the code they contribute, allowing them to increase their skills and experience as developers and become more valuable to the organization over time. A side effect of this mentoring is higher quality code submissions, with fewer defects and better long term manageability of the code base.

Migrating Applications between OS Platforms

2009-09-06T01:36:00.000-07:00

At some point in time most Information Technology (IT) departments have had to migrate an application or service from one platform to another, in this case I mean a different operating system as the platform. This is most often driven by a cost savings that can be obtained on the new platform, either through lower hardware maintenance costs, or lower support costs for the software on this new platform. The challenge with these migrations is that often times, the application is stable on the existing platform, and any migration introduces the risk of introducing instability.

The points of review documented below are not specific to any operating system (OS) on the market, but rather are a guide for migrating from any single OS to a different OS. Currently the IT world is seeing the largest percentage of these types of application migrations from UNIX-based platforms to a Linux based platform. But, just because this is occurring now does not mean this will always be the most common migration path, in time a new OS could come on the market providing advantages not currently available.

Many modern programming languages are portable in the sense that they can very easily be migrated from one host OS to another. This is not true for some legacy programming languages; this framework is meat to cover both these cases. Even with modern programming languages, some underlying libraries can vary from OS to OS and will require detailed migration planning.

Below is a framework for the process for reviewing the application being migrated and developing a plan for the migration. This framework is structured to ensure that the same steps can be used, regardless of the original and future OS.

Application Source Code
When initially reviewing an application to migrate from one OS platform to another, the source code must be checked from a process, availability and legal standpoint. This is the first phase to determine if the application can even be migrated to a new platform.

Is the source code available?
This is often an overlooked component of legacy applications. Often times the source code is not available either because it was lost, or because the intellectual property for the application has been transferred to another party. This is an important part of porting an application, and can cause alternate applications to be looked at or developing the application from scratch.

Legal obligations?
As part of reviewing the availability of the source code, it is also important to review legal obligations around that source code. Specifically open source applications often have requirements for submitting changes to the community, depending on the usage model of the application. These legal obligations are also important regarding trademarks, copyrights, and their implications on staff that previously worked on the application being reviewed.

Review of Application Source Code
After determining if the application source code is available, and determining what changes can be made and how to communicate that to external parties that may be required, it is time to review the source code technically to develop a plan for the migration and porting activities later.

What language?
Looking at what language the application is developed in is a first critical step. This will enable the planning team to determine if the company has the necessary skills to port the application, or if external resources will be needed for the migration. Knowing the language can also assist with planing supportability on the new OS based on how the well the language is supported and used in the community.

What libraries?
As part of reviewing the source code, a review of the libraries used should be done. This review should be done to ensure that the libraries will properly work on the new OS, that they are still available, and that they are compatible with other libraries that will need to be installed. This is the time to ensure no dependency problems are found later in the migration.

Deprecated calls?
The source code review should also include an assessment of what calls and functions are now deprecated; this can include external libraries, kernel functions and other external resources. Any section of the application code that references deprecated functions should be reviewed to determine the best supportable path forward to ensure that functionality is not compromised.

Define Testing and Roll out Strategy
Now that the source code has been reviewed, it is time to define success for the migration. This component of the process is to ensure that relevant metrics are clearly defined for the time period of the migration, and after the migration so that staff using the application are not negatively impacted by the migration.

Data Integrity
Defining data integrity standards should be the first metric for all migrations from OS to OS. This is critical to ensure that data is consistent both during the migration, and handled in the proper way after the migration. A migration of an application from one OS to another should not ever require the compromise of data integrity standards.

Functionality
Second to data integrity is functionality. Staff become used to the tools they use on daily basis, and any change in the capability or functionality of those tools can cause a significant drop in performance. All migrations should include reviews to ensure that all utilized features will continue to be available for staff to utilize.

Performance
Performance is an important metric to define prior to migrating an application from one OS to another. Performance can change dramatically between OS platforms and should be planned for both testing and proper application tuning during the migration process. Performance can include many metrics including response time, reporting generation time and response time under heavy loads.

Security
While one OS is not necessarily more or less secure then another, each have their own methods for setting permissions, logging system activity and patching against known vulnerabilities. The migration plan should include a proper review of these differences to ensure that staff are properly trained to handle securing the application once it is running on the new OS.

Stability
Stability is commonly defined as uptime or availability of an application. Introducing a new OS to an environment can change the availability characteristics, either because of new, unfamiliar processes, or because of a misplaced expectation about an OS's capability. A plan should be developed to define what availability is required of the application, and documentation on monitoring those metrics.

Porting of Code
After defining the above metrics, we can begin the longest portion of any application migration. The actual porting and testing of the application to the new OS platform. This phase will include both making modifications to the code base to ensure it works on the new OS platform, as well as testing the application on the new OS platform to ensure it properly meets the metrics defined above for success.

Maintenance Cycle Definition
During the porting of the application data can be gathered about necessary maintenance that will need to be done regularly on the new platform. This maintenance cycle will need to include time to update patches to both the underlying OS, as well as do maintenance on the data supporting the application. This maintenance cycle should be defined prior to roll out so that staff can be properly trained on this maintenance cycle and end users can be prepared for a possible change in availability policies for the application.

Update DR Processes and Tools
Disaster Recovery (DR) is an important component of all application migrations, ensuring that a properly plan is in place to recover from catastrophic failures and ensure the data and application are available for use. As part of the application migration, the DR processes should be reviewed to adequately reflect the changes in how the application is hosted and what precautions should be taken for backup, replication and training for recovery.

Training
Training is a two part activity, both the administrators for the application as well as the end users will need to be trained on the changes in administering and using the application. Training should be provided to the appropriate staff prior to migrating the application, this will ensure that staff are ready for all change that come as part of the migration. Training should additionally be made available for staff to reference back after the migration to answer questions that could come up about the migration.

Application Roll out
After the above metrics for success are defined, the code is ported and tested and staff are trained, the application migration can be completed. This migration will include the migration of any necessary data for the application, as well as the application delivery infrastructure. This migration can be done is phases if the architecture of the application will support it, or may require an extended outage to properly migrate and test all components.

Migrating an application from one hosting OS to another is a common practice, yet, very often it is done with very little planning. As IT continues to evolve, it is inevitable that new OSs with innovative features will become available, necessitating the need to migration applications between them. Keeping a solid process that is followed each and every time will ensure stability in the migration, integrity of the data and maintaining productivity of the end users.

Importance of IT Audits

2009-08-10T05:47:00.000-07:00

In todays business world, many companies think that audits are constrained to company financial reports only. Companies often overlook external audits as a way to increase security, productivity and reduce costs for other company operations, most notably information technology (IT). Financial audits are required by law for many types of companies, particularly publicly traded firms. This formal audit done on a regular basis ensures that end consumers of this financial information can be confident that the results are consistently reported according to industry standards.

IT is often overlooked in audits, most commonly the only IT components reviewed are the specific applications for housing financial data and the software for reporting that data. All companies can benefit from a change in this mindset and begin to utilize regular external IT audits as a way to provide a neutral, third-party opinion about the controls and safeguards in place for the IT systems that a company relies on to conduct business

Audits can provide a variety of useful information to an organization, but most importantly they remove the risk associated with unknowns within an IT environment. Audits allow a company to say in a very certain way that their controls and safeguards meet industry standards. Regular audits ensure that each year controls and safeguards are updated to accommodate changes in the industry for standards for IT operations.

Thorough audits cover a variety of components of an IT environment, both technical and procedural. The most important part of an IT audit is not the validation of those processes, but the thorough testing of the environment to determine that everything is configured per the policies, as well as everything is configured per industry standards. The second component ensures an IT environment that can be compliant with legal requirements, and safe from the most common and expected threats.

When looking for an audit firm to complete an IT audit, here are the most common items you should ask them about how they handle, review and report on:

Staff Competencies

The technical skills and experience are the most important part of an external IT audit. The external firm should be reviewed to ensure they provide the highest quality of staff, with a diverse and relevant background to your organizations needs.

Audit Firm's Reputation

Ultimately, your company is going to rely on the reputation of the audit firm if any part of the audit ever comes into question by partner companies or other organizations. It is important to choose a firm with a solid reputation of quality work, quality reports and the willingness to follow up on questions after the audit.

Security

Security has several angles that must be considered when choosing an external audit firm. The first is the security they will provide for your company confidential data, both data they collect while conducting an audit, while also providing confidentiality as part of the audit.

Second, the firm must provide a solid review of security within your organization as part of the audit. This audit should include reviewing physical security, security policies, off site storage, data in transit and penetration testing of the network from an internal and external perspective. All audits should cover these aspects of security at a minimum, and use them as a basis for reviewing the rest of the enterprise for compliance with industry standards around encryption, authentication, logging, monitoring, alerting and incident response.

Current Controls

A complete audit will include a thorough review of all controls around access of data, change management, upgrades and staff responsibilities.

Controls include all aspects of change management. Ensuring that a proper plan is in place to approve, and track changes will ensure that consequences are fully planned and recovery plans are in place prior to upgrades, changes or migrations. Outside audit firms can provide experienced third-party recommendations about the level of process and it's adequacy within your organization.

Controls also include staff responsibilities and how responsibilities are delegated and enforced through both process and technical safeguards. An experienced audit firm will review these for accuracy as well as implementation details to ensure controls work as designed and are implemented where necessary.

Suggested Controls

As part of the controls review, an experienced audit firm will document controls that are needed, but not currently in place. The recommendations come from experience in the industry, as well as solid knowledge of compliance regulations.

Staff Training

An important part of all technical audits is a review of staff skills sets. Most external audit firms will do a review of current staff and their skill sets, this information will then be used when reviewing recommendations for additional technologies or controls within the organization. It is important that all suggested changes include a required list of skills so that your organization can properly train and equip your staff to implement an outside firms suggestions.

Company Culture

Often times, a companies culture is the reason for non-compliance with accepted industry standards, particularly in IT. External IT audits provide your company an opportunity to have external, experienced professionals observe how your staff operate. The external perspective is often very useful in isolating unanticipated challenges that may come because of a specific culture within your company.

Data Protection

External audits should include a detailed review of how data within your organization is categorized, and subsequently protected from loss and disclosure. This review will be both technical and procedural to ensure that gaps are not present in the current solutions. This portion of the audit should include not only how data is managed on a daily basis within your company, but should also include how data is backed up, replicated and protected from loss in the event of a serious facility failure or loss.

Legal and Compliance

This is often the most difficult portion of an audit because of the highly specialized skills needed to complete a compliance review. Reputable audit firms will be able to provide the necessary legal knowledge as part of the audit to ensure that policies are in accordance with legal requirements. These regulatory requirements are most common in financial services and health care industries.

Cost Analysis

All findings from an IT audit will have specific costs associated with them. These costs could include both the cost to fix the problem with additional training, hardware or software; as well as the potential cost to the company if the problem is not corrected. Audit firms should be able to work with your organization to determine and document these costs for use in determining a remediation plan and prioritizing the findings from the audit.

Penetration Testing

Most IT audits will include penetration testing of your organizations network, applications, servers and data storage facilities. This is an important part of all audits because it tests the active controls in place, as well as allows for the locating of additional controls that are needed. It is important to find an audit firm with experience with these types of audits; this experience will both increase the potential for findings, as well as limit the chances for adverse consequences during the testing process.

In a perfect world, an external IT audit will cover an entire company, not just specific departments. This provides the most thorough results because an external entity is reviewing all departments and organizations in a consistent manner and providing documentation to senior management of how the various organizations interact and affect one another. Often times companies will do focused audits, only looking at a specific department or subset of the IT infrastructure. While these can yield important information, they should be used with caution because they will potentially miss other important areas for improvement.

Finally, be open minded at the end of any audit when reviewing the results from the external firm. It is possible that you will be shocked after the first audit at the shear number of findings. This is not necessarily bad. A long list of recommendations could show that the firm doing your audit was very thorough and provide you with a solid basis for improvement. The most important part to reviewing the audit results is repetition – you want to make sure that a long list of recommendations is no repeated on subsequent years. Use the list as a chance to improve so that the audit firm is not continually finding the same problems year after year.

Lustre 1.8 and Pools

2009-08-03T04:34:00.000-07:00

Beginning with Lustre 1.8, the concept of pools was introduced. Pools are a method for isolating groups of OSTs based on common characteristics. This is most commonly used to group OSTs based on similar hardware type or RAID configuration. An example would be to have a pool of very high performance SAS disks, and a lower performance set of SATA disks, within the same filesystem. Pools will allow users to specify which pool their files are read from and written too.

Next to each section of commands is the system they must be run from.

For these commands, 'lusfs01' is the name of the lustre file system. pool1 and pool2 are the names of the example pools, and we have a total of 10 OSTs within this file system.

Creating a new pool (MGS)
# lctl pool_new lusfs01.pool1
# lctl pool_new lusfs01.pool2

Assigning OSTs to a pool (MGS)
# lctl pool_add lusfs01.pool1 lustre-OST000[0-3]_UUID
# lctl pool_add lusfs01.pool2 lustre-OST000[4-7]_UUID

Listing Available pools (MGS)
# lfs pool_list lusfs01

List OSTs in a given pool (MGS)
# lfs pool_list lusfs01.pool1
# lfs pool_list lusfs01.pool2

Setting a file/directory strip to use a specific pool (Client)
# lfs setstripe -p pool1 /lusfs01/dir1
# lfs setstripe -p pool1 /lusfs01/dir1/file1
# lfs setstripe -p pool2 /lusfs01/dir2
# lfs setstripe -p pool2 /lusfs01/dir2/file1

Interviewing in IT – Finding Solid Candidates

2009-07-07T05:11:00.000-07:00

One challenge within all IT organizations is finding and interviewing candidates to ensure that those hired into an organization will not only bring necessary skills, but also allow the organization to grow and evolve. Interviewing methods vary from company to company based on a variety of factors including culture, past experiences and human resource department influences. I hope to explore some factors I believe lead to successfully interviewing of candidates.

In my mind, there are four primary purposes of interviews:

Determine if the candidate has the proper technical skills to be successful. This includes a combination of technical knowledge and past experiences.
Determine if the candidates' personality is compatible with both existing team members as well as the company culture.
Determine if the candidate has the necessary willingness to learn new skills and learn from their team to evolve and grow in the position.
(Sometimes) Determine if the candidate has specific domain knowledge or industry connections that can only be obtained by hiring from outside the organization.

Here are the key items I keep in mind when interviewing, as well as encourage others to think about when interviewing candidates for my team and other teams I work with.

How Long?
Don't focus on how long the individual will be at the company; assigning a time frame will defer focus from the more critical aspects of the candidates' place within the organization. Focus instead on what the individuals career goals are. Can those be met at this company? What types of technology do they want to work on and learn? If these are technologies your company uses and will be using, the chances of a long term candidate increase greatly.

Technology is not the only factor, focus on other aspects of the individuals career goals. Are they expecting promotions into management or technical lead positions? These types of advancement opportunities will determine how long the candidate will be at the company.

2-way Street
I have been to many interviews that the individual conducting the interview forgot that interviews are 2-way streets. While it is important for a company to determine if an individual will perform well within the company structure and culture, it is equally important that the individual get a realistic feel for the company and see the company in a good light. You don't want to risk loosing a strong candidate because the focus was too much on asking the individual questions, and not enough on allowing them to see the possibilities your company has to offer.

Brain teasers are fine, but.....
Many organizations today use brain-teaser type questions to understand how a candidate solves problems and their methods for thinking and responding to stressful situations. These tests have a valid place within interviews, but need to be used properly to ensure you are getting valid results from the tests. It is important to only ask brain teasers that the individual will have the domain knowledge to approach answering. I have been in many Network Architect interviews and asked questions about algorithms around managing large datasets. This was not a good use of time for me or the interviewer. It is important to ensure that even if the candidate can not answer the brain-teasers, they have enough base knowledge to share their through process for solving the puzzle.

Personality Tests
Personality Tests tend to fade in and out of popularity as an interviewing tool. Many companies argue that they provide a glimpse into the candidates tendencies and habits and can allow the human resources department to determine their compatibility with existing personality types at the company. The problem is that many of these tests are quite easy to “study” for and this can heavily skew the results. My feeling is that taking the candidate to lunch or coffee with a small group of team members will show much more then a standard test ever will. The key with an interview is to make sure both parties are comfortable enough that they communicate as they would in any standard work situation.

By conducting the interview in places other then your standard conference room, you have a better chance of understanding what the individuals work habits are and how they converse with others. It is easy to watch how a candidate talks with folks at the office; it is an entirely different view to see how they interact with the wait staff at a restaurant and a very useful data point on their personality.

Work Location
“Remote” working is all the rage these days, especially in IT. You know my thoughts on it from previous postings, I think that it has it's place, but not all positions can function effectively while being isolated at home. Deciding if a position is going to be “remote” should be done before interviewing potential candidates, this decision should be based on the type of work this position will be doing, as well as that of the team and how well that work can be done in a distributed fashion.

Not only does work location include being “remote” or not, it also includes possible relocation of the candidate. It is important to set expectations up front on the companies' policies for relocation. If there is no relocation budget and the candidate is three-states away, it is probably not worth pursuing.

Finally, it is important to understand an individuals' constraints around relocation, both time frames and potential locations. If a candidate does not like cold, yet your company is based in northern Alaska, there is probably little need to continue the interview process.

Technical Questions
I do not suggest jumping immediately into complex technical questions, no matter what job you are interviewing a candidate for. I suggest rather working up to the point they appear to be at from their resume. This means checking for the base knowledge and experience, this is a good method to see how the candidate responds. Do they seem passionate about the work? Do they speak more about knowledge or experiences? Do they speak about where they go to stay current on the industry?

One of my favorite interview tactics is to ask a technical question about a problem we recently encountered. This is a great opportunity to see not only the candidates thought process, but also what level of detail they go into for solving the problem and what tools they would use to approach the problem. Continuing to ask questions about the problem will also show a lot about how they respond under pressure and how they communicate to other team members in the face of a stressful situation.

Technical questions also provide a good basis for seeing what knowledge the candidate has about the industry as a whole. Do they understand the benefits of one vendor or another? Or do they focus on using what they know and are comfortable with?

Domain/Vertical Knowledge
Most information technology professionals are specialized, that is, they work in a specific industry; these can include Financial Services, Oil & Gas, High Performance Computing, etc. Each of these domains has specific tools and applications, as well as industry accepted methods of accomplishing tasks.

When interviewing it is important to decide up front if you are looking for a candidate that is experienced in your specific domain, or if you are looking for a strong IT candidate that could bring an outside perspective to your organization. This will drive what questions you ask, as well as where you go to look for potential candidates to fill the position.

Interviewing is an art, it takes time to develop a process that works for you and your organization. Starting with hard technical questions will not help you determine if a candidate is qualified, there is much more to how a candidate will succeed or fail in your organization. You must treat all interviews as 2-way streets; this allows both parties to get an accurate picture all all aspects of the position including location, expectations, team dynamics and technical knowledge as well as desire to learn and evolve.

In my mind, technical skills are secondary for today's information technology positions. More important then all the technical knowledge in the world is how well the candidate communicates with team members, responds under pressure, understands trade offs and benefits analysis and ultimately the candidates desire to learn and grow and IT evolves. No matter how technical someone is, if they can not get along with them team they will be a detriment to the organization.

Talent Management in Information Technology

2009-06-01T00:54:00.000-07:00

The Information Technology (IT) sector is evolving very rapidly, over time it has developed a reputation as a high stress career field, with low personal rewards, little chance of advancement and ultimately an area of low morale. While this is not the case within all IT departments, it has become a common perception of the industry as a whole. This perception has been driven by a variety of factors including fewer staff available to complete tasks and the use of outsourcing for lower cost resources. We as an industry need to take more ownership of our staff and skills and work to develop them internally in a way that provides companies maximum value from their IT organization, while ensuring staff are successful, stable and ultimately happy with their work environments.

First, we should define the “standard” IT employee, and I am not talking about the system administrator that works all hours of the night while drinking Red Bull. I am talking about those traits that make IT staff want to work in the technology field. While this description will not cover all IT employees, there are certain traits that stand out more in the IT space then other professions:

Curious by nature
Detail focused
Quest for understanding
Prefer the technology to the business
Like to build things
Opinionated
Seeking Recognition

We now have a better understanding of what IT staff do and expect by nature. Lets explore the primary drivers that go into their ability to focus on their job and deliver successfully to the companies bottom line. This is by no means an exhaustive list, but in my experience these are the top items within an IT department that contribute to employees truly enjoying their job; employees just staying for the paycheck versus employees that are looking to move on to other opportunities.

Pay – There is a common misconception in IT that folks will work at a given company because it is interesting and exciting work, but this is only true to a point. We all have bills to pay for rent, utilities, food, entertainment, and student loans. An employee paid market average for a given area will often stay at a company if the work is exciting. But no level of excitement and interest is enough to make up for below average pay when an individual is struggling to pay their bills. Companies should strive to ensure their salaries are consistent with the local market for the level of skills an employee is utilizing at work each day.

There are also other costs that go directly with pay. It is very expensive to loose an employee, they take lots of company experience and knowledge with them. Transferring that knowledge to a new employee is costly in the time they are learning the business, they are contributing less to the bottom line. Companies should always evaluate the cost of raises for staff to ensure they stay at the area average to the cost of loosing one employee and having to hire another.

Opportunities – Staff within IT organizations are curious by nature, with that trait they constantly want to expand their knowledge and experience. Some of the most successful staff I have worked with in IT move on a regular basis, not always up in the corporate ladder, but most often laterally to other jobs that are of interest to them. This provides them a benefit of increasing their skill sets, and provides a benefit to the company because corporate knowledge is not lost when an individual moves within the company. IT staff should be provided opportunities to move both within the IT department and within the company. This movement and change of jobs is often what IT staff need to ensure they do not get burned out while allowing them to stay engaged with their jobs.

Interesting Work – Very few people are content doing the same activity every day. Because of the general trait for IT staff being on a constant quest for understanding, most IT staff are always looking for new and exciting projects. It allows them to be creative and develop new solutions to the problem. Sadly, there is always going to be some tasks that are more interesting then others within IT departments. Effort should be used to ensure that any less then desirable tasks be evenly spread across available team members, and that team members understand that while they may have been assigned a less then desirable project, so were their teammates.

Staff should be encouraged to not only take on projects assigned to them, but to come forward with ideas they have for improvement within the organization. This encourages all team members to have a stake in the organization and feel ownership of not only their projects, but other tasks that they may see a need for completing.

Flexibility – Employees appreciate when their management allows them the flexibility to work when they are most productive. I am not necessarily talking about allowing staff to work in their pajamas from home, but more referring to ensuring employees do not feel tied down with a specific schedule that causes them less productivity.

Working from home has become very common in many organizations, especially IT. I believe that a lot of organizations have taken it a step too far and staff are beginning to feel the isolation of working by themselves each and every day. I believe that staff should be provided the tools and flexibility that if they choose to work remote for an afternoon, that should be allowed. I believe that the majority of a 40-hour work week should be spent in the office, it encourages staff to communicate with their coworkers, take a vested ownership in the daily operations of the business and ensures the company develops a culture of it's own.

No two people are alike when it comes to sleeping schedules. This has a very negative effect when employees are asked to begin their day at a time that is not natural for them. I believe that staff should be given the flexibility to arrive and begin their day when they will be most productive. This does not mean everyone should sleep in until noon and begin work at 1PM. It does mean that should a staff member prefer to work later in the evening because that is when they are most productive, the company should encourage this behavior.

Ultimately an organization is only as strong as the communication between it's team members. Above I mentioned that working from home all the time is a suboptimal choice; the primary reason for that belief is that communication can be challenging when everyone is so spread out. The ability to quickly gather team members in the office and discuss a topic can ensure minimal time is wasted when a decision must be made. I believe that office hours, also called core hours, is an optimal method to ensure staff have the flexibility to work remotely, while encouraging team communication. By having all staff in the office for certain set periods, often 10-12AM and 2-4PM, it ensures that if a staff member is needed for a discussion, they can be found. This use of core hours, I believe, provides a good balance of allowing flexibility for staff and ensuring a solid team dynamic takes hold for the team.

Work Space – One common perception within IT over the years has been that putting staff in shared space will allow better collaboration. The challenge is that very little IT work is collaborative in nature, the bulk of the work that must be accomplished is individuals working on their pieces of a project, and this type of activity requires that staff be able to focus. Shared spaces have a lot of benefits in terms of quick access to others, but at the cost of decreased focus due to noise and other distractions. I believe that companies need two primary types of space available for their IT staff, private offices to allow for focus and concentration, and shared collaboration areas to allow for quick meetings and discussions.

Private office space allows staff to have an area that is their own to focus on their work and not be distracted by outside noise, phone calls, hall way discussions or projects. Each staff member should have an office that allows them to close the door and focus free of distractions.

Common areas should be available to encourage team discussions and impromptu meetings. Very rarely can a decision be made faster then by pulling the team together for a quick discussion in the hallway. These common areas within an office space should have enough white board space that notes can be kept on any design ideas or other notes from the discussion. These common areas will also encourage inclusion of all project members, and not just a subset that may discuss the matter in a private office or on a conference call.

Loyalty – Companies often expect a certain level of loyalty from all staff, but do not necessarily show that level of loyalty back to their staff. Having the CEO walk around at the company holiday party and shake hands only goes so far to telling employees that they are valued. I encourage all managers within IT to regularly call out their accomplishments of their staff to the rest of the organization. IT staff strive for recognition, it is what encourages them to do their best every day. When managers publicly acknowledge a job well done it tells the employee and all their coworkers that the effort and work are appreciated.

Now that we have those out of the way, lets explore the deep dark truth of IT. Even if a company does each of those perfectly, some staff are going to leave. This is just the nature of the business. No matter how hard a company and it's managers try, there will always be staff that are looking for something that the company can not provide. When this case occurs, and it will, it is important that the employee and company both act as professional as possible. There is an old saying in HR, “don't burn your bridges.” That applies to both the company and the individual. IT is such a rapidly evolving industry that even if the match between a company and employee is not correct now, it is very possible that a match will be made down the road after either the company or the individual evolve. “Don't burn your bridges” applies to both the employee and the employer, even if a staff member leaves, they still have institutional knowledge that could be of value down the road.

Someone I have worked with many times over the years has a very clear way to sum up the relationship between employee and employer, “Pay me well, Treat me well, Wish me well.” Meaning that if you pay your employees fairly and treat them wonderfully they will do quality work, if you pay them exceptionally well and have higher stress levels they will do quality work, but should an employee not be treated fairly and is not being paid well enough to compensate, the company should “Wish them well” in new opportunities.

Understanding Lustre Internals

2009-05-21T18:19:00.000-07:00

Lustre can be a complex package to manage and understand. The folks at ORNL, with assistance from the Lustre Center of Excellence have put out a wonderful paper on Understanding Lustre Internals.

I recommend that all Lustre administrators read it, it is very useful information for understanding how all the Lustre pieces plug together.

"Cloud" and HPC?, Huh?

2009-05-05T18:33:00.000-07:00

I have tried for the most part to not post on this phenomenon known as "cloud computing." "Cloud" is still evolving and as such has many different meanings. The reason this whitepaper caught my attention is it's attempt at connecting high performance computing (HPC) with "cloud computing." The way I see it, "cloud" is still more of an evolving idea then a true product. True, many companies are offering "cloud" products, but the standards are still evolving, as is the true meaning of "cloud computing."

In my mind "cloud" is the next logical evolution of computing - better resource management through enabling applications to better communicate with their supporting infrastructures (servers, storage, network, cpu and memory resources) to allow applications to have the intelligence to scale up and down based on demand. "Cloud Computing" also has a valid connection to outsourcing in the sense that shared infrastructures will at some point over take privately managed information technology (IT) infrastrucures that are common today.

There are several points about the above listed whitepaper from UnivaUD that caught my attention:

MPI was only mentioned once. The Message Passing Interface (MPI) is the standard on which most HPC applications and platforms are built. For a paper to truly look at the potential of outsourcing HPC to a "cloud" environment, an indepth review of MPI will need to be done to ensure the proper updates are made to handle the additional physical layer errors that could occur in a shared environment, as well as the added challenges of communication in an unknown environment.
There was very little mention of the actual applications that are common in HPC. Applications like Fluent, NAMD, NWChem, Gaussian, and FFTW are commonly used on clusters built in house to meet the specific needs of a given community. Moving those applications from these small, in-house envirronments will take time and review to ensure they are able to scale in shared environments, as well as properly handle the increased variation possible in hardware and configurations.
There was no mention of parallel file systems. This is a fundamental requirement of modern HPC environments. To truly move common HPC environments into the "cloud" a solution will be needed for data management and transfer at the high speeds required of todays applications.

In short, the above linked whitepaper is common of what I am seeing in the "cloud" space; lots of talk of the possible benefits around the use of shared environemnts. What we need to stop doing as a community is trying to associate all things IT with "cloud." I have no doubt that in time we will evolve to more use of shared resources - this has been occuring for quite a while with the migration to larger clusters within universities and national laboratories, as well as the ongoing outsourcing of email and specific applications - but as a community we need to ensure that each time we change how we do things for a given area of IT it is with specific goals in mind. Without those clearly defined goals we will not know if we were successful.

As time allows I hope to explore the above issues, particularly looking at alternatives for parallel file systems in environments that may have varying latency, and are distributed over various data centers.

Balancing Security and Productivity – Part 4 of 4

2009-05-04T04:47:00.000-07:00

Proxy Internet Connections

Companies often look to proxy servers as a method to monitor and block harmful traffic from their networks. Proxy servers provide a gateway between company networks and outside networks to ensure that all connections are logged, filtered and denied per company policies. Proxy servers can present a challenge because they can often slow access for staff, and inadvertently limit access to sites that are authorized, but may initially appear unauthorized to the automated tools limiting access.

Open Internet Access – Open internet access is allowing staff unrestricted connections from a corporate network to the outside world; these connections are free from any proxy servers, bandwidth restrictions or other traffic filters. While this can allow for maximum ability for the staff to conduct their jobs, the question must be asked, is this too much access? When a network allows that level of connectivity going out, there is inevitable risk that confidential information could be transmitted out of the company with little or no record of the event.
Limited Internet Access – Outside access can be limited by a variety of methods including blocking specific ports, utilizing proxy servers or utilizing other network traffic monitoring solutions. When used correctly, these tools can not only prevent company confidential information from being inappropriately transmitted outside the company, but they can also provide a solid audit trail in the event an investigation is needed. The trade off is that staff's performance will be affected by possible slowdowns due to the overhead of the tools as well as the potential that the traffic being blocked or targeted does have a requirement for conducting business and an employees productivity will be affected adversely.

In part 1 of this discussion we asked the question; how balance allowing employees to access company data with a personal device that connections to proprietary company information? The answer will ultimately be different for every company. But there are some common criteria that will be consistent across all solutions:

Consistency of security policies - It is critical that just when a staff member is using a personal laptop, the security policies are not being compromised for this benefit. This means that personal systems must adhere to the same policies for storage of company data, use of virus scanning applications and use and storage of company passwords.
Centralization of storage – By utilizing central, company controlled storage, it allows the information technology (IT) department to ensure all company data is regularly backed up, archived and available in the event of laptop or mobile device loss. There are many tools on the market that can automatically replicate data from remote devices to a company managed data center. This ensures data is always available, regardless of the type of device connecting or ownership of the device.

Finding the proper balance of security and productivity is a complicated, dynamic process for both the end users and those forming company policies. Any company today must ensure that they have the proper IT resources at their disposal to do their job and that those tools are open enough for staff to utilize in the most efficient way, but closed enough that propriety or otherwise confidential data is not put at unnecessary risk. All risks have a potential downside and all functionality has a potential benefit, both of which can be expressed in dollars. It is important to ensure that the balance of that risk and benefit is on the side of benefits, and that the risk is not so great as to cause harm to your company.

Balancing Security and Productivity – Part 3 of 4

2009-05-01T04:13:00.000-07:00

Database Encryption

Often companies will encrypt data stored within a database. This ensures that data is secure from simple eavesdropping by requiring a key to manipulate or view the data.

Encrypted Databases – Encrypted databases are becoming more common, either encrypted in their entirety, or portions of the database that are particularly sensitive. While encrypted databases to provide a lot of protection to unauthorized users, they do potentially provide slower access because of the additional CPU time needed to decrypt the data for use. Encrypted databases also pose a hazard for data loss in the event the keys necessary for data encryption and decryption are lost or otherwise must be regenerated.
Non-Encrypted Database – Standard databases are most common today, essentially databases that store the data in traditional ways without encryption. The risk they pose is that if the clients of the database are compromised, or backups of the database are compromised it is quite trivial to read the data contained in that database, which could contain personal information like user names, passwords and addresses. While traditional, non-encrypted databases can scale much larger because of the lower CPU usage, they do have significant risk to data compromise.

Device Ownership

Device ownership is often a big topic of discussion, especially within companies hiring younger workers right out of college. Individuals will often get very comfortable with a platform while in school and expect to be using that same platform when they enter the workforce. When they later find out that their employer has a different OS or brand of laptop, employees will often use their personal devices for company business.

Company Devices – From a security standpoint, company owned devices are the most secure option, but at a cost. Employees will be less productive if they are forced to use a platform they are uncomfortable with or new too using. Company owned devices ensure that the company can recover the device should an employee leave and ensures that all software being used is licensed, virus free and properly monitored by corporate IT staff.
Personal Devices – While personal devices can allow workers to be more productive and comfortable with their operating environment, it comes at the cost of very decentralized IT management. Personal devices may not necessarily be covered by corporate software licensing agreements, and may not be kept up to date for security patches per company policy.
Combination – Most firms have settled on a combination of allowing personal hardware, but putting policies and tools in place to ensure it is managed by a centralized IT organization. This ensures that staff can have the tools they a are most familiar with, but data integrity, security and virus scanning is updated as company policies evolve.

File Transfer Policies

All companies have the need to transfer files, both internally and externally for review, collaboration and company communication. These documents present a risk to the company because confidential information could inadvertently be sent to unauthorized parties.

File Attachments to Email – Attaching files to email has several risks including a large need for capacity in the mail servers to handle the volume of traffic, as well as the potential that files could be inadvertently sent outside the company. While some modern email systems have the ability to scan out going email for specific content, this is often time consuming and can slow down the flow of communication.
Collaboration Tools – Limiting employee's ability to send files via email attachments is becoming much more common; as a solution to the need to share files, many companies are beginning to use collaboration tools like Trac, Twiki or Sharepoint. These solutions allow files to be stored internally, access to be restricted back and to ensure proper versions of files are available for those that need them, with out the risk of outsiders having email and attachments inadvertently forward to them.

Balancing Security and Productivity – Part 2 of 4

2009-04-22T19:16:00.000-07:00

Chat Applications and Boundaries

Many companies are looking to real-time communication tools like instant messenger and other chat applications to enable staff to communicate real time, either internally or with external customers or partners. These tools can enable staff to be very efficient at communication and issue escalations, but the risks of information being shared incorrectly, or not properly archived present a risk that should be evaluated.

Internal-only – Internal only chat solutions provide staff the ability to quickly communicate internally, while limiting the change of accidental exposure of customer data outside the company. What internal-only chat solutions lack is the ability to communicate real time with customers or partners. By eliminating this capability, staff could have to use other, more time consuming solutions for external communication.
Internal and external – By providing staff with the ability to chat real time both internally, and externally they are enabled to communicate real time with customers, partners and other outside groups that contribute to the bottom line. The potential risk is a staff member could send an incorrect file, or cut/paste incorrect text into a chat window and reveal company proprietary data to an external entity.
No-chat – On one end of the extreme is to block all real-time chat communication, limiting staff to communication using standard email or phone conversations. While this can ensure no company sponsored tools are used for external communication, todays tech-savvy employees will often attempt to circumvent this limitation and use their own tools, potentially creating larger security implications because of non-centralized management. While eliminating chat applications can contribute to a more secure environment, the potential effect on employee productivity can be negative.
Compliance – Compliance is the other large factor for chat and other instant messenger type applications. Compliance can include a variety of items include detailed record keeping, legal documentation of discussions and industry-standard policies for data storage and handling. Most chat applications offer the option of storing an archive of all discussions, this feature should be evaluated against compliance requirements to ensure that necessary records are kept and unnecessary information is purged.

File Storage Locations

Storing of company files, including email archives, customer communications and other company documents must be done in a way that files can be recovered if lost, but also to ensure that access to those files is only grated to those requiring access to complete their assigned job. Few companies have a consistent method for file storage and sharing; most companies have differing policies for each department. It is important that a company have a defined policy that becomes part of the corporate culture to ensure collaboration and exchange of ideas, as well as compliance for document storage.

Local – Local file storage is individual employees storing company documents on the computers and other devices they use for conducting company business. Local file storage presents a challenge in all facets of security because of a lack of an audit trail for file access, a lack of recovery capabilities if an employee accidentally deletes a file, a lack of a recovery mechanism for lost laptops and ultimately a lack of recoverability if an employee were to leave and take their laptop with them. While local only storage provides an individual employee with the easiest access to the files they work with regularly, the company as a whole has very limited visibility into that employees archive of company data.
Network Shares – Network shares provide a loosely controlled environment for storing files that individual staff members have worked on or created. Network shares provide minimal levels of recoverability because they can be backed up more easily then individual laptops and desktops, and they can also do minimal revision control. They do lack real audit capabilities for file access and updates and do not provide staff a formal method for communicating who is working on any given document at any given time. Because of the lack of real auditing paired with the lack of real capability around access controls, network shares are not a good long term strategy for a company that could have many documents to manage.
Shared Collaboration Sites – Shared collaboration sites are the most common method in companies today to share files and documents internally. They provide a very robust method for storing documents, managing multiple revisions and managing access controls for documents based on a variety of factors including need-to-know, manager approval, project participation and department ownership.

Operating System Usage

Many companies will evaluate a given operating system (OS) as part of a security review, when the actual OS in use is a very minor component of the equation. At some point in time a security vulnerability has been found in all major operating systems. The risk posed by these various vulnerabilities has much more to do with how the vulnerability is responded too then the actual OS with the vulnerability.

Staff Skill Level – Probably the most important topic when addressing what operating systems (OS) to use in any environment is skill set of the system administration team, yet it is often not looked at in depth. Staff are most efficient at administering operating systems that they are familiar with and have experience with. If new operating systems are introduced, the initial ramp up time to be proficient for staff can be on the order of months. During this time there is risk that best practices will not be followed and work could potentially have to be redone. When evaluating operating systems for a given environment, the time consideration for training staff with the necessary skills must be considered.
Patch Process – The process to install performance, security and feature upgrade packages differs very widely from OS to OS. This has significant implications to the security of a system, the longer it takes the administration team to install patches, the longer a vulnerability could be exploited. When reviewing new operating systems, the tools they offer for installing and managing patches should be reviewed to ensure that patches can be installed and tested in a timely manner.
Vendor Relationship and commitment – A vendor's commitment to a particular OS and application stack is critical to ensuring a secure environment. When reviewing operating systems for use in your environment, it is important to understand the vendors commitment to the platform; this has implications for the speed of patches being released, as well as the capabilities a vendor has for developing patches in a timely manner.

Lustre Users Group 2009

2009-04-21T06:30:00.000-07:00

Last week we held the 2009 Lustre Users Group. It was a success; we had the largest user turn out ever.

All slides can be found here.

I did a presentation on Best Practices for the Sun Lustre Storage System, those slides can be found here.

Balancing Security and Productivity – Part 1 of 4

2009-04-17T03:15:00.000-07:00

This is the first part of an ongoing discussion. The additional parts will be posted in the coming weeks.

An often challenging debate in any IT organization is the proper balance of security and productivity. Most organizations struggle to balance a loss in productivity for staff due to tighter security restrictions around passwords, data access, allowed applications, automated monitoring and threat detection. People at various levels within an organization will have differing solutions for balancing risk and ease of completing work for various staff. Every risk that must be understood for security changes has an associated cost, either in the cost of lost data, lost capability or bad publicity. On the flip side, every change made in the name of security and lowering risk could potentially lower employee productivity which can both affect output and have a cost, as well as affect morale if tasks become more difficult to complete.

In addition to evaluating risk for security policies and it's impact on staff and their productivity is assessing that impact across different staff with different duties at the company. Often times staff with more tightly controlled tasks are easier to limit impact for then staff that have a larger range of duties that may require off hours work, remote work or constantly changing duties and tasks.

With any activity within an enterprise, be it adding an application, adding a new mobile device or adding a new network connection poses a level of risk. That risk must be weighed against the benefits gained by adding that network connection. Take one of the most common tasks for an IT department; adding a new active network connection to someones office within a company facility. This activity has little risk associated with it because most often only staff will be in the area and able to physically use the connection. The benefit of this can be great by allowing an additional productive staff member, an additional printer for staff use or allowing faster network access then existing connections would allow. In this case this risk to reward balance is reasonable. Now take an activity that is just as common; installing VPN software on a laptop so that a staff member can connect to the company network remotely. What if this laptop is then lost and has company data on it? What if this laptop is infected with a virus that could infect other corporate machines? I intend to explore various trade offs that must often be reviewed by IT departments and the associated risks and rewards that go with each.

Passwords versus Tokens

One of the most common methods for increasing security within a computing environment is by eliminating one-time passwords and replacing them with a token based approach for non-reusable passwords. In this forum I call any authentication solution that provides a challenge response or requires an external token to be the alternative to standard passwords. There are several trade offs that must be considered for this approach to provide a high-level of assurance that accounts are only used by the designated owners:

Login Speed – Using tokens or other 2-factor methods for logins has the potential to slow down staffs' ability to login. If a staff member can not find their token for login that will slow down their ability to complete tasks. Additionally, the time needed to use a token is often longer then the time required to enter a traditional password from memory and be authenticated.

Seamless Integration – Integration company wide can pose a challenge for tokens and 2-factor authentication solutions. While much improvement has been made on this level with modern identity management tools, most firms still have a diverse range of applications and integration with all of them is often not possible. This leaves companies in a situation where they must decide which applications and tools make sense for token based authentication and which should remain password based.
Ease of Memory – Tokens often use a pin number that is shorter then common passwords. This shorted pin paired with a specific token that is time specific creates a combination of information that is easier to remember, and thus less likely to be written down by staff. This ease of memory of necessary login information can ensure a situation where staff passwords are

VPN versus Public Secure Web Sites

There are two primary methods for ensuring that company data is secure when being accessed by employees and authorized personnel; the primary method is to use web based applications that run over encrypted channels, the https protocol is the most common. Often times companies will implement a virtual private network (VPN) solution to further ensure that all data transmitted is secure.

The primary issue being discussed here is providing access to company applications to staff that are located in remote locations, this could be working from home, while on travel or via remote devices.

VPN Assurances – VPNs, when properly used can ensure compliance with a variety of company security policies around virus protection, password length and expiration and a systems patch status. These policies can ensure all hosts connected to the companies network are secure. The trade off is that VPNs are often difficult for users to utilize because of the time necessary to connect and the technical challenge in ensuring users can always connect to the VPN when necessary.
VPN Restrictions – While VPNs ensure that systems connected to the network meet compliance, they restrict an employees ability to login quickly and complete a task. If an employee needs access but does not have a company computer, a VPN only approach may limit their ability to use nearby computers to complete the task.
Availability of Web Based Applications – Web based applications that are encrypted and outside of company VPN infrastructure allow staff to connect in a secure fashion, regardless of who's computer they are using. While this does enable productive work to be done in more locations, it increases the potential that data or passwords could be compromised by keystroke loggers on non-company controlled machines.

Security considerations in a virtualized environment

2009-04-01T12:30:00.000-07:00

Virtualization is becoming the standard method for consolidating large information technology (IT) environments down to less hardware then was once required. Because of the rapid increase in both processor performance, and memory density, paired with increased disk capacities, a single server can handle the load that it used to take many to accomplish.

This consolidation effort has presented multiple challenges, including:

Increased complexity of IT environments
Increased requirements for System Administrator's skills sets
Unknown quantities around security within virtualized environments
Increased need for processes to ensure compliance with applicable industry regulations
Increased need for executives to understand resource utilization and allocation across the environment(s)
Increased need for disaster recovery planning so that single hardware outages do not cripple an environment

I am going to talk primarily about the security aspect, and some mitigation techniques used with virtualization. Security is a difficult subject within virtualization because the topic is in it's infancy and because of that we are still learning the proper processes that are needed to secure virtual environments at the same level our traditional physical infrastructures are secured at. The introduction of hypervisors within an IT environment add a level of complexity to the environment, and create an entirely new tier where data access, user authorization and monitoring must be implemented to ensure security.

Lets also talk about the boundaries for our discussion and the definition of security I will use for the remainder of this posting. Security can mean many things to many different people. The boundaries for what falls within the realm of a security team within a company will also vary greatly from firm to firm. Security as I describe it is the actions and processes that ensure an individual can only access and modify data that management has approved them access too. This includes ensuring permissions and other configuration settings are only changed by those authorized, and private information is only accessed by those that management feel have a valid reason to access it.

Definitions

Physical Host – A physical server running a hypervisor and having one or more virtual machines active on it
Virtual Machine – A single running instance of an operating system (OS) sharing physical resources with other running OS instance
Hypervisor – The software layer that resides on a physical host and allows multiple concurrent virtual machines to effectively share the same physical resources
System Administrator – An individual with root or administrative level rights on one or more physical or virtual hosts
SAN Administrator – An individual with the ability to manipulate shared storage devices or switch configuration between shared storage and servers using that storage
VLANs – Virtual Local Area Networks, a method to logically partition a single physical network into multiple logical networks
LUNSs – Logical Units, a unit of storage exported from a shared storage device to one or more hosts

Now, lets discuss some scenarios that are specific to virtualization, and some techniques to mitigate these threats.

Administrators with full access to hypervisors
Probably the best known and most thought about security vulnerability within virtualized environments is the hypervisor and it's inherent access to the virtual machines above it. Most current virtualization solutions have a single root user at the hypervisor level with access to power virtual machines up and down, modify virtual machines (VM) boot parameters and gain console access to those VMs.

This type of model requires both a high level of trust for system administrators, as well good processes in place to ensure all changes are approved, properly tested and periodically reviewed by staff other then those responsible for making them. All administrators within a virtual environment should only have access privileges on systems required to complete their job, and systems that contain data they are authorized to see and handle. Management should implement audit policies to periodically review logs and ensure that all changes were approved, properly tested and meet all IT policies.

Console access to VMs
Most hypervisors by default will allow anyone with administrative rights on the host system to access the console for all VMs hosted on that system. This creates a situation where an unauthorized party could access the console of a system and perform password recovery activities, or see system output to the console.

Ensuring that administrators have the least amount of access to successfully complete their job is key to ensuring that console access is limited to those that need it. Often times, administrators will rarely need to access the console of a system because of technologies like remote desktop and remote shells for managing a virtual system. Modern hypervisors will allow permissions to be set so that console access is only given to those that are authorized. It is suggested this be enabled so that an administrator can only access the console for systems they are immediately responsible for.

Patches at the hypervisor level
The hypervisor within a virtual environment creates a single tier with essentially administrator level access to many more systems then the administrator would have before virtualization. This hypervisor layer has access too all VM data, the ability to power VMs up and down and the ability to see the console for all VMs on a single physical server. This hypervisor layer adds a single tier of access, that if compromised could create a path to easy compromise of many additional systems.

Ensuring security now requires additional levels of testing during the phase that was traditionally penetration testing. New applications must also include load testing from a security standpoint to ensure that new applications, if compromised would not affect the performance or response time of remaining applications. This all means that a security patch at the hypervisor level has much more sever implications then patches on individual VMs because of the increased threat.

Ultimately, the most important aspect with hypervisor security is ensuring that only those that require access to it, can connect to management tools. This means using host based and network based firewalls to explicitly allow traffic that is allowed and deny all other connections to the hypervisor for VM management. In addition to restricting access, companies should have an efficient process to test patches when they are released from the vendor to ensure they are implemented, particularly at the hypervisor as quick as possible to limit any windows of opportunity.

Complexity
Any addition of new technology, tools or features has the potential to add more complexity to an already complex IT environment. Complexity creates a variety of long term problems including making upgrades harder to manage, creating the potential for mistakes and configuration errors, creating the potential for one change adversely affecting other aspects of the environment, and most notable putting a higher workload on IT staff.

As virtual environments grow, testing and validating all processes becomes only more critical. The best defense to complexity is careful documentation that has been tested, and is continually updated to reflect changes in the environment or methods of management around that environment or the company as a whole. The more carefully things are documented, the more efficiently actions can then be automated, ensuring that the potential for human error is further removed. By automating processes around auditing, patch testing, software deployment and VM creation, IT staff can be left to focus on operational efficiencies, while ensuring that all systems will operate within the boundaries of company policy with minimal intervention.

LUNs Zoned to Hypervisor
It is common to utilize a SAN in todays virtualized environment to simplify management of data growth, movement of virtual machines and increase performance of backups. This use of a SAN creates a level within the hypervisor, that anyone with administrative access to the hypervisor can manipulate the LUNs destined for virtual machines. This creates the potential for not only having people access data they do not have the need to access, but the potential that data is manipulated without proper authorization.

Properly encrypting data at the file system level will ensure that data is only accessed by authorized applications and users. Encrypting data ensures that only the authorized application and administrators can manipulate production data, this level of assurance also ensures that if any physical disks were to become unaccounted for, management can be assured the data will not be read by unauthorized parties.

Ability to power VMs up and down
Virtual machines share an underlying management infrastructure and physical machine infrastructure. This creates the potential that a rouge system administrator or staff member can cause harm to one segment of the infrastructure, simply because they have access to another. Having a shared hypervisor creates the potential that if the administrator account is abused, systems can be stopped, started and rebooted at unexpected time.

Critical services should not be hosted in virtual environments. This will ensure an added layer of protection for things like LDAP, Kerberos, Active Directory, DNS and critical web servers. By hosting these critical services on dedicated virtual machines, you ensure that security problems within the hypervisor environment, or rogue staff do not cause harm to the services that are most critical to the stability of your enterprise.

Staff accounts with permissions to power up and down VMs should be closely monitored and restricted to only allow access to the systems an administrator needs to access to complete their job. This limiting of access will ensure that if an account is abused, the damage it can incur is limited in scope.

Shared networks on physical machines
Companies often times will use VLANs as a way to separate systems based on usage, security risk, data type and physical site. This reliance on VLANs often times extends as far out as the firewalls at the edge of a corporate network. When using virtual machines, there is the added risk of having multiple virtual machines on a single physical machine that require separate VLANs to function and adhere to existing network policies. Mistakes with initial virtual machine setup, as well as system compromises can create a situation where VMs add unexpected paths between networks.

When initially planning the use of virtual machines, it is vital to include the staff responsible for both security, as well as network routing and switching implementation. They can provide valuable insight into the reasons for using VLANs or other network separation techniques. By including them, you can review what physical systems will house what virtual machines, and if network changes will be required to ensure security is not compromised and unexpected paths are not created between separate networks.

Implementing a new VM
Implementing new virtual machines has an inherent risk in both the threats posed by any new applications, but additionally the necessity to manage and patch an additional host within the environment. Every new virtual machine is a full OS that could potentially compromised, or otherwise used to launch attacks on your network, or others' networks.

A toolkit should be implemented before any virtual machines are activated that is used for two primary purposes:

Penetration Testing on new systems – All new hosts should be properly tested to ensure they meet company security policies. This testing process should include a review of running services, a review of host level firewall policies, a review of active system accounts and passwords and finally, ensure the system is integrated in with corporate monitoring and patch management tools
Patch management and monitoring on all systems – A corporate wide patch management suite should be used and inclusive off all virtual machines. This centralization will ensure staff are aware of all virtual machines that are active, and aware of systems that are not up to date on security patches. More advanced tools can also provide staff with the ability to quickly audit systems for other security policies like password length, password expiration and firewall policies.

All virtual machines should be retired as soon as they are no longer needed. This removes the overhead on staff of managing the system, and removes the risk of having the system sit potentially unmonitored and used. Virtual machines should be considered the same as the sprawl of old, unused physical servers, and removed as soon as practically possible.

Application layer vulnerabilities
Ultimately a server is only as strong as it's weakest active service, and most often servers are compromises not because of a lack of OS patches, but because of failed application implementations or configuration errors. VMs are vulnerable to this same risk around application level security problems. Virtual machines have the added risk of being compromised that if their load increases, they put other virtual machines on the same physical infrastructure at risk

Boundaries should be enforced across all tiers of an infrastructure; storage, physical systems, network connections, management tools and applications. An application is an extension of the OS from a security perspective, and an applications residing on a physical system via virtual machines should have similar security characteristics including risk, data classification and company policies.

Externally facing VMs
The location and use of VMs must be closely tracked. If a physical host has VMs with both internal access and access from external users, the threat of outside attacks affecting internal resources increases dramatically. Any VM on a single physical host is vulnerable to a host of threats because of the other VMs it shares physical resources with.

By working with the networking and security teams before implementing virtual machines, system administrators can ensure that physical hosts only host common virtual machines, grouped by access levels, data classification and risk. Most companies do not cross network boundaries with virtual machines. Separate physical machines will be places in each separate security environment to host virtual machines for that security and access level.

Audits and Tools
Auditing is a critical function in all IT environments. By properly auditing an environment, administrators can be notified to problems before they become serious or data is potentially compromised. A solid audit trail is often required by outside firms that may certify a companies ability to house or process certain types of data. Auditing is an entire topic on its own, but some common items to monitor and alert in a consistent fashion are:
System level logs from all hosts, both physical and virtual
Monitoring network traffic for unexpected changes to typical traffic pasterns
Logging of all manipulation of VMs including console usage, powering on and off of systems, installation of patches and changes to configuration files
Changes to storage configuration that could include LUNs, zoning or encryption characteristics

Security within a virtual environment has the same underlying principals as the traditional physical environments we are accustomed too. Least access must be ensured so that compromised accounts or rogue staff have a limited amount of damage that can be caused. Process is the most important way to ensure access is limited in a way that staff can successful complete their job, yet not access resources they do not have an immediate need to work with. Clear process can ensure new systems are thoroughly tested, reviewed and put into service, and then managed for the life of the application or host. Staff are more effective at overall administration if consistency is ensured across an environment.

IPv6 backward-compatibility

2009-03-26T06:55:00.000-07:00

Network World has an article this week about the challenges adopting Ipv6, specifically the lack of backward compatibility. The article does a good job describing the challenges in IPv6 deployments around how to migrate without impacting services and the problems presented because some devices have long life cycles and may never be IPv6 capable.

LUG 2009

2009-03-14T10:45:00.000-07:00

For those attending the Lustre Users Group 2009, I will be presenting on Best Practices for the Sun Storage Cluster. A full agenda is at http://blogs.sun.com/HPC/resource/agenda.pdf

More information on the LUG can be found at http://www.regonline.com/LUG09

What a business system should look like

2009-03-08T13:21:00.000-07:00

One ongoing debate within the desktop support community is what makes for a successful desktop platform within an enterprise? From my perspective, there are several key features a desktop must have to make it a viable platform within a company environment:

Integrated capability for centralized password management, virus definitions, and security policies.
Centralized capability to deploy software and monitor license usage
System that is priced correctly to replace about every 2.5-3 years. This ensures staff have fast machines with up to date software to complete their roles.
Simplicity of security auditing to ensure that IT staff know where data is, when and that it is secure
Abundance of tools from third parties to manage the platform remotely to minimize support costs

Why do I rattle off this list? There was a very good article in CIO recently that hits on the topic and covers the common discussion topics well.

Ethernet Improvements

2009-03-01T03:14:00.000-08:00

I wanted to call attention to a couple interesting projects in the ethernet space. They both have the goal of lowering the overhead that is commonly associated with using ethernet within an HPC environment.

Open-MX
GAMMA

There is a very good write up at http://www.linux-mag.com/id/7253, giving a little more background on each.

Managing in IT

2009-02-10T18:00:00.001-08:00

A very good posting on managing technical/IT folks and the quirks that go along with the common personalities you will find in IT.

http://www.wikihow.com/Manage-Geeks

Lustre Monitoring

2009-02-10T08:50:00.000-08:00

I have had several Lustre deployments recently that included performance monitoring, here are the common tools folks use for monitoring their Lustre environments. As time allows I will get a HOWTO posted for each on installing and configuring them.

Lustre 1.6.6 with MX 1.2.7

2009-01-25T07:46:00.001-08:00

Below is the process for installing Lustre 1.6.6 while using MX (Myricom) as the transport.

1) Compile and install Lustre Kernel
- yum install rpm-build redhat-rpm-config
- mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
- echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
- rpm -ivh kernel-lustre-source-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm (can be obtained from http://www.sun.com/software/products/lustre/get.jsp)
- make distclean
- make oldconfig dep bzImage modules
- cp /boot/config-`uname -r` .config
- make oldconfig || make menuconfig
- make include/asm
- make include/linux/version.h
- make SUBDIRS=scripts
- make rpm
- rpm -ivh ~/rpmbuild/kernel-lustre-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm
- mkinitrd /boot/2.6.18-92.1.10.el5_lustre.1.6.6
- Update /etc/grub.conf with new kernel boot information
- /sbin/shutdown 0 -r

2) Compile and install MX Stack
- cd /usr/src/
- gunzip mx_1.2.7.tar.gz (can be obtained from www.myri.com/scs/)
- tar -xvf mx_1.2.7.tar
- cd mx-1.2.7
- ln -s common include
- ./configure --with-kernel-lib
- make
- make install

3) Compile and install Lustre
- cd /usr/src/
- gunzip lustre-1.6.6.tar.gz (can be obtained from http://www.sun.com/software/products/lustre/get.jsp)
- tar -xvf lustre-1.6.6.tar
- cd lustre-1.6.6
- ./configure --with-linux=/usr/src/linux --with-mx=/usr/src/mx-1.2.7
- make
- make rpms (at the bottom of the output it will show location of the generated RPMs)
- rpm -ivh lustre-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm
lustre-modules-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm
lustre-ldiskfs-3.0.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm

4) Add the following lines to /etc/modprobe.conf
options kmxlnd hosts=/etc/hosts.mxlnd
options lnet networks=mx0(myri0),tcp0(eth0)

5) Populate myri0 Configuration with proper IP addresses
- vim /etc/sysconfig/network-scripts/myri0

6) Populate /etc/hosts.mxlnd with the following information
# IP HOST BOARD EP_ID

7) Start Lustre by mounting the disks that contain the MGS, MDT and OSS data stores

Automated OS Provisioning

2009-01-19T13:40:00.001-08:00

Automated operating system (OS) provisioning is an important part of any IT department. It allows staff to rapidly build new servers or virtual machines that are identical to all others currently in use. This lack of variation allows for easy tracking of necessary upgrades, and minimizes the number of combinations of software and hardware that must be tested before deploying updates and patches. This commonality across systems also allows system administrators to work on new systems with very little ramp-up time to understand how one system may be different from all the others.

I have been working in several large, heterogeneous environments recently that required the ability to build common OS images, across a variety of OSs and platforms. Here are the common utilities for the different OSs to use when deploying OS images, in one nice clean list:

Solaris
Jumpstart

HP-UX
Ignite-UX

Linux (SLES)
AutoYAST

Linux (Fedora/RHEL/CentOS)
Kickstart
Cobbler

Linux (Ubuntu)
Kickstart
preseed

AIX
Network Installation Manager (NIM)

Windows (Server 2003)
Unattended

Windows (Server 2008)
Automated Installation Kit (AIK)

Windows (XP)
Unattended

Windows (Vista)
Automated Installation Kit (AIK)

Multi-OS Support
Altiris

Virtualizing your Quality Assurance Environment

2009-01-05T08:59:00.001-08:00

Today virtualization is used in many Information Technology (IT) shops for a variety of purposes including:

Disaster Recovery
Enable better utilization of hardware assets
Create demonstration environments for new applications
Compartmentalize applications for security
Legacy application support

Virtualization is a two-edged sword, it can allow much better utilization of resources and allow for much more efficient separation of services and applications. Virtualization can also add an additional level of complexity to often already complicated environments. Because of this complexity, virtualization must be properly planned and evaluated before implementing it within an environment to ensure that the benefits it offers outweigh the additional costs and time to manage a more complex environment.

How can virtualization improve QA?
I am going to talk about utilizing virtualization in an information technology (IT) environment for a specific goal; creating an automated environment to allow test driven development and automated quality assurance of software builds to facilitate automated movement of software into a production environment.

This targeted goal is specifically meant to speed up the testing process for in house developed applications, traditionally web based. While speeding up the testing process, we want to ensure that we do not raise the need for more quality assurance (QA) staff. A goal is to ensure that all tools implemented maintain a strong balance of automation and human interaction, without increasing the time commitment for individual staff or their teams. By properly maintaining a balance of automated development and manual testing, we can ensure development teams can scale to support a growing number of deployed versions in the field, and a constantly growing list of features.

In addition to ensuring a scalable model for growing a QA environment, we should ensure that any QA process enables testing of the entire software stack, including libraries, OS patches, third-party software, application servers and associated databases. This testing should be isolated from any underlying hardware to ensure applications are portable between various hardware platforms. This isolation and integrated testing will be completed by creating a new VM as part of each software build. Each of these VMs will contain all necessary applications, services, libraries, third-party applications and data to complete the testing process. This generation of a single VM for each software build will ensure that testing is completed on the entire stack, and all results are easily reproducible.

Tools necessary for success
The key to ensuring a successful QA environment that can dynamically test new software builds and automatically move them between environments is ensuring that the correct tools are in place to provide staff the appropriate level of visibility into the automated process, without causing an excessive work load to be added to the developers and software testing teams.

Here are the most common tools I see the need for in a QA environment:

Bug/defect/feature tracking.
Reporting for patches applied to the source tree and the source of the patches.
Association of patches and the bug/defect corrected.
Automated testing framework.
Association of testing results with specific bug/defect reports.
Reporting capabilities for number of defects/bugs per line of code.
Reporting capabilities for number of bugs/defects per developer.
Tool to show what features and bug fixes will be available in each given release, and the progress towards version completion.
Tool to show time necessary to complete each new feature, and time necessary to correct reported bugs/defects.
Tagging of builds with unique identifiers that associate a build with a list of included features, corrected bugs and patches.

While these requirements are listed as separate capabilities, the fewer actual tools and the more integrated the data is from these tools, the more efficiently decisions can be made and staff can input the required data. The more efficiently the tools supporting both the development and the testing process can be integrated, the more efficiently the developers can see possible areas for improvement with development. The more efficiently this data can be gathered and reported in an automated fashion, the more efficiently the development teams will be able to use it for improvement.

Environments
In my opinion, most development teams can complete their testing activities by using a system of five separate environments, each with specific purposes and goals. Larger development teams may very well have many more then this, but smaller teams should seldom have fewer as it creates the potential for different stages of testing to overlap in unpredictable ways. I envision the following environments and associated usage patterns, in the order they would be used to complete a final application build:

Sandbox – New library testing, manual builds, developing unit tests.
Development – Ensure error free builds, ensure library version matches and test database structure is correct.
Quality – Test application against unit tests, test application response time and test data level integrity.
User Acceptance Testing (UAT) – Test user input; both correct and invalid data handling, test application response time, test interactions with outside applications and tools. Limited testing by knowledgeable end users of the application.
Production – Customer facing application implemented in a way to meet all required Service Level Agreements (SLAs).

Development to Production
Now that we have defined our tools for a consistent, reproducibility build process, lets compartmentalize that within a virtual machine (VM) for complete control and reproducibility. The goal of compartmentalizing it is to completely remove external influences from testing results, these external influences can include varying hardware platforms, inconsistent library versions and updated data models.

This process to safely compartmentalize an application and test it within a VM is three steps:

First – Our first step is to define a clear process for moving a build from one environment to the next in the testing process. This process should include testing for errors and warnings as part of the build process, defining what are acceptable pass and fail ranges for all unit tests and defining what performance benchmarks must be met by the application for each stage of testing. This step will also include defining any manual or management approvals required for moving software builds from one testing environment to another.

Second – After defining the process for properly testing each component, we must clearly define what is part of a build, and what components are external. This will assist us is developing our testing matrix for versions of our application, any outside data models and applications and all associated libraries. This step will include that we properly define the boundaries for the automated testing process, versus what testing will require manual review and intervention.

Third – The final step is developing and testing the process for moving builds from one environment to another, while ensuring that no changes are made between testing phases, and all builds are archived in a way they can be referenced at a later date if necessary. This step is to ensure that both testing from one stage is valid during another, but also ensuring that any testing failures and findings are archived for future analysis and review.

Final Thoughts
By utilizing VMs to compartmentalize applications during the development and testing cycle you can provide not only isolated, reproducible environments, you can easily roll back to archived versions of the production application if a deployment fails. VMs provide a way to test an entire application stack, including outside software, libraries and data models in an integrated way to ensure stability when deployed in a production environment. As the technology around VMs continues to evolve developers will only get more capabilities around snapshots of VMs, the ability to roll back in time within a VM and better performance modeling and characterization capabilities.

I did not discuss any specific hypervisors because this is meant to be a discussion around the business objectives of a testing environment. Most hypervisors on the market can be automated in a way to deploy this automated testing and movement between environments. Most hypervisors on the market also have associated tools for quickly cloning test environments for analysis, performance monitoring or simple archiving. When reviewing your environment and choosing a possible hypervisor for your development environment, these tools can provide invaluable capabilities to both development and testing staff, as well as your system administrators responsible for the production environments.

Networking in IT

2008-12-10T12:50:00.000-08:00

There is no doubt, we are seeing a challenging time in the economy, and it is trickling down to the Information Technology (IT) sector. One of the most important parts of weathering a storm like this in the IT industry is focusing on ensuring your network of coworkers, alumni and friends is strong. By developing a strong network, you have a team to folks you can turn to for advise, recommendations, job postings and the inside track on potential job leads.

The other benefit to networking, beyond looking for a new position, is to develop professionally in your current position. By interacting with others in your field and similar fields you can build your personal toolbox by learning from other peoples' experiences and skills. By regularly working with others you can see what methodologies they use to be successful and what tools they have developed and found to ensure they are efficient in their roles.

Here are some common methods myself and others have used to build a community within the IT space:

Users Groups – Most cities today have multiple users groups including Linux, Oracle, MySQL, Dell and DB2 just to name a few. These organizations are always looking for speakers and folks to hold lab sessions. Volunteer to present, volunteer to organize meetings and volunteer to recruit other speakers. It is a wonderful way to meet folks in similar roles as well as share your knowledge and experience with others.

Brown Bag Events – Host a brown bag at your office, invite your coworkers and do a short talk about a topic that interests you or you think would be of relevance in your environment. This gives you publicity within the company, and allows members of other teams to see the skill and experience they have available when new projects come up.

Operate your companies test bed – Often times companies will have a test and quality assurance environment to use for testing new software deployments, to complete software builds and to test new vendor hardware. Often times this environment does not fall on corporate IT, or the quality departments to manage, but is somewhere in the middle. Volunteer to manage this environment and take real ownership of it. This will give you a great forum to meet people in other departments, as well as share your ideas in a way that will allow them to be utilized in production for the company.

Blogging – Blogging is a simple, effective method to put your ideas out in the public for comments, development and to show your level of expertise in a field. Blogging allows you to share ideas and findings as you write them. Blogging provides communication in a forum that while not peer-reviewed, others can comment on your postings and post additional follow up information.

Conferences – Presenting at conferences is a wonderful way to show both your level of experience, as well as new ideas and methodologies you can bring to your field. Conferences provide a peer-reviewed environment that you can submit papers and do talks. These type of environments show not only your level of expertise, but that others in your field value your contributions and capabilities.

Defining High Availability

2008-12-04T15:31:00.000-08:00

In todays business computing environments a wide variety of terms are used to describe systems management, systems performance and system availability. One commonly used term is High Availability (HA). This is a very broad term that can encompass many different levels of availability and the costs associated with the various levels of availability. This term is open to quite a bit of interpretation and this interpretation often leads to confusion about exactly what level of HA an application, device or service provides. Below are the items to factor in when assessing the actual availability of a given service to ensure that it meets your specific interpretation of HA.

Level-setting Expectations
High Availability can mean something different for each person that says or hears the term. It is important to level set expectations about HA and its meaning before having an in-depth discussion about how to meet the objectives laid out in an HA environment. Properly defining HA and calculating the costs associated with implementing HA has four components:

Time to recovery – It is important to understand how long a failure will take to recover from, this will allow you to properly choose solutions that can identify and recover from a failure within a given time frame. A failure can be a hardware problem, a software malfunction or a human error that causes the specific service to act in a way other then it was designed. There are many valid cases where time to recovery can take on the order of minutes or hours, there are other valid cases where recovery should be near-instantaneous.

Method of recovery – Method of recovery is an important component of planning and HA solution and it's associated cost. Many times recovering from failures is an automated fashion, but it is not uncommon to have an error that requires manual intervention to clear the problem. This is often done for categories of problems that are not critical to the operation of a business or customer impacting.

Data Loss and Corruption – Data loss and corruption is an important part of developing a strategy for HA. Data loss and corruption can occur during a failover of services between nodes, while the network works to get into a state of equilibrium after a change or during periods when a given service is down. All data has a value associated with it and when calculating the maximum allowable downtime for a service, data value should be calculated in as well.

Performance Impact– Often times a failure of a service component will cause a degradation in service, yet leave the service online for users. This degradation if often times acceptable assuming it is for a short, limited period of time. Understanding how users will use the service will enable you to understand what level of performance loss is acceptable.

A Perfect World
Before we continue into a discussion about how to achieve a given level of High Availability, I want to define my expectation when I hear the term High Availability. When I use the term HA I expect and application or service that can transparently handle failures from a user perspective. I expect an application that despite a failure on the back-end including a server, disk, network connection or otherwise will automatically failover in a way that the end user does not see a disruption in how they are used to interacting with the application. The user should see no degradation in service or loss of data because of the failure.

My definition of HA is assuming a perfect world and adequate funding to architect and implement such a solution. But as we know, IT is not always funded with the necessary money to make dreams into reality. In these cases we must refer back to the first list of components that make up HA to determine which items can be compromised on.

Defining HA for your Environment
Now that we have covered what items are used to define HA, and my definition of HA in a perfect IT world, lets discuss the process for defining a level of HA appropriate for your needs and balancing that with the associated costs with a given level of HA. First is to understand your user base and what their expectations are around application performance, response time and recovery. Things to consider are when your users use the application, how they enter data and what response time they are used to when interacting with the application.

Second is to define what the technical solution will look like for the above customer requirements. This stage is where you will evaluate various levels of redundancy and capability in any database servers, network components, data centers and application capabilities. This stage should include an evaluation of both vendor packaged solutions, and home grown solutions that will meet your needs. This assessment should also include a review of staff capabilities to determine if training will be needed for staff when implementing new technologies.

Third we will define the cost for each component of the above developed architecture. This cost is the cost for an optimal solution, broken down by each individual component. This cost should include all hardware, implementation and software licensing costs for a given period of time. A three year costing is standard within IT and is a good basis to compare several different solutions in an equal fashion.

Finally, we must evaluate the potential cost savings for each component of the solution if we were to cut back from an optimal solution to a more cost effective one. This evaluation should show the portions of the solution that can be implemented via multiple methods, and the associated costs for each method. This information is then used for comparison to balance the required level of HA with the budget available for the project. By properly understanding how much each component of the solution will cost, you can properly evaluate what the possible level of HA will be with each potential increase or decrease in project funding.

Methods for implementing HA
For most of this document I have avoided discussions actual technical solutions available on the market for implementing HAs. This omission was to ensure that HA was defined per your specific needs before defining possible hardware and software solutions. Now I am going to dive into several popular options on the market for assisting to make applications HA capable.

Linux-HA – Linux-HA is an open source solution for managing services across multiple nodes within a cluster for providing a basic high availability solution. Linux-HA is often used to provide automated failover for applications like Jboss, Apache, Lustre or FTP. While Linux-HA will not provide the sub-second failover that some environments need, it will allow administrators to easily setup a pair of servers to act as hot-standbys for one another.

Redundant Switch Fabrics – Modern ethernet switches have multiple levels of redundant capability including redundant controllers within a switch, redundant power supplies, and at the high end redundant switch fabrics that should one complete set of switches and routers fail, a second will seamlessly handle the failover and subsequent network traffic. Technologies like OSPF will ensure that routing of IP traffic continues uninterrupted and protocols like spanning tree will ensure that switches with multiple paths will utilize them in an optimal fashion during both regular and failover scenarios.

RAID – Redundant Arrays of Independent Disks (RAID) is a common method of ensuring that a single disk failure within a server does not cause data loss or corruption. RAID capability can be added through specialized hardware solutions or via low cost software solutions. Both provide a level of protection above standard disks, while keeping total solution costs low.

Oracle RAC – Oracle's Real Application Cluster (RAC) is a clustering solution, often associated with Oracle's database products for both providing high availability functionality, as well as a platform to scale a databases performance. While Oracle RAC is often more expensive then other clustering solutions from MySQL, it provides a very scalable and reliable platform for ensuring very high levels of availability for applications and their associated databases.

Fiber Channel – Fiber Channel solutions for attaching storage to servers often implement redundancy via dual, redundant fiber channel fabrics. These are often implemented utilizing completely separate switches, cables and power connections. This type of solution can ensure that common failures like cables and PCI cards will not cause a server to loose access to its storage or data corruption.

High Availability is often taken to mean something different for each person. Ultimately, HA is ensuring that customer and end user expectations are met for how an application performs and recovers in the event of a failure. When setting up an application, you must first define HA for your specific needs, you can then properly develop a solution that will meet those expectations. As with most projects within Information Technology, you will then have to assess each component of the solution and make possible tradeoffs to ensure the projects budget is met. Ensuring an application is available and properly recovers is a part of all major Information Technology projects, today there are many possible technical solutions to ensure your customers expectation of HA is met.

Building a Lustre Patchless Client

2008-12-04T08:05:00.001-08:00

One common need within Lustre environments is the requirement to build Lustre clients using standard Linux kernels. Lustre servers commonly have a custom kernel with specific patches to optimize performance, but clients do not always require these kernel patches.

These directions will enable you to build the RPMs necessary to install the Lustre client bits on a system with a standard Redhat kernel.

1) Umount all Lustre clients and comment entries from /etc/fstab

2) Reboot a node into the standard redhat kernel you would like to build the client for. Assumption for these directions is RHEL 2.6.18-92.1.13 x86_64.

3) Install the full kernel source tree for the running kernel
- cd ~
- yum install rpm-build redhat-rpm-config unifdef
- mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
- rpm -i http://mirror.centos.org/centos/5/updates/SRPMS/kernel-2.6.18-92.1.13.el5.src.rpm

4) Unzip the lustre bits
- Download from http://www.sun.com/software/products/lustre/get.jsp
- mv lustre-1.6.6.tar.gz /usr/src
- gunzip lustre-1.6.6.tar.gz
- tar -xvf lustre-1.6.6.tar

5) Prep the kernel tree for building Lustre
- cd /usr/src/linux
- cp /boot/config-'uname -r' .config
- make oldconfig || make menuconfig
- make include/asm
- make include/linux/version.h
- make SUBDIRS=scripts

6) Configure the build - configure will detect an unpatched kernel and only build the client
- cd lustre
- ./configure --with-linux=/usr/src/linux

7) Create RPMs
- make rpms

8) You should get a set of Lustre RPMs in the build directory.
- ls ~/rpmbuild/RPMS

9) rpm -e lustre*

10) Install new client bits
- rpm -ivh lustre-client-1.6.6-2.6.18_92.1.1.13.el5.x86_64.rpm
- rpm -ivh lustre-modules-1.6.6-2.6.18_92.1.1.13.el5.x86_64.rpm

11) Remount all Lustre mounts
- vi /etc/fstab
uncomment lustre lines
- mount -a

Implementing Lustre Failover

2008-12-01T07:39:00.000-08:00

Linux-HA, also referred too as Heartbeat is an OpenSource tool for managing services across multiple nodes within a cluster. Linux-HA ensures that a given service or disk is only running or mounted on a single server within the cluster at a given time. Linux-HA ensures that if a server within the cluster was to fail, the other server was become active for the service automatically, minimizing downtime for the users.

A default install, as I will document today, only catches problems with a server in the cluster not responding to Linux-HA communication. If a node was to have other problems like failed disks, failed network auxiliary connections or errors in I/O access, Heartbeat would not catch and respond to those failures without additional instrumentation.

These below directions are how to implement Lustre-HA to provide for more automated failover of Lustre services. These directions were developed and tested with Lustre version 1.6.5.1 and Linux-HA version 2.1.4.

Assumptions

4 total nodes (2 node-pairs)

1 MGS (Lustre Management Servers)

1 MDS (Lustre Metadata Server)

1 MDT (Metadata Target) on the MDS

2 OSSs (Lustre Object Storage Servers) (OSS01 and OSS02)

2 OSTs (Object Storage Targets) per OSS (OST00-OST03)

The MGS and MDS will be on a pair of clustered servers
Nodes MGS and MDS have access to the same shared physical disks
Nodes OSS01 and OSS02 have access to the same shared physical disks
The name of the filesystem is 'lustre'
STONITH method is IPMI and the IPMI interface is configured for remote access
No software RAID, all RAID is implemented via hardware solutions

Configuring Linux-HA
1) Install Linux-HA
# yum -y install heartbeat

2) Comment out all Lustre mounts from /etc/fstab and umount existing Lustre server and client filesystems. This will ensure no data corruption or contention issues when starting Heartbeat.
MGS/MDS Pair
mgs # cat /etc/fstab | grep lus
#/dev/MGTDISK /mnt/lustre/mgs lustre defaults,_netdev 0 0
mds # cat /etc/fstab | grep lus
#/dev/MDTDISK /mnt/lustre/mds lustre defaults,_netdev 0 0

OSS Pair
oss01 # cat /etc/fstab | grep lus
#/dev/OST00DISK /mnt/lustre/oss00 lustre defaults,_netdev 0 0
#/dev/OST02DISK /mnt/lustre/oss02 lustre defaults,_netdev 0 0
oss02 # cat /etc/fstab | grep lus
#/dev/OST01DISK /mnt/lustre/oss01 lustre defaults,_netdev 0 0
#/dev/OST03DISK /mnt/lustre/oss03 lustre defaults,_netdev 0 0

3) Create all mount points on both nodes in each node-pair
MGS/MDS Pair
# mkdir /mnt/lustre/mgt
# mkdir /mnt/lustre/mdt
OSS Pair
# mkdir /mnt/lustre/ost00
# mkdir /mnt/lustre/ost01
# mkdir /mnt/lustre/ost02
# mkdir /mnt/lustre/ost03

4) Execute '/sbin/chkconfig –level 345 heartbeat on' on all 4 nodes

5) /etc/ha.d/ha.cf changes
MGS/MDS Pair
# cat ha.cf | grep -v '#'
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
udpport 10100
auto_failback off
stonith_host mgs external/ipmi mds 10.0.1.100 admin adminpassword
stonith_host mds external/ipmi mgs 10.0.1.101 admin adminpassword
node mgs
node mds

OSS Pair
# cat ha.cf | grep -v '#'
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
# different from MGS/MDS node-pair
udpport 10101
auto_failback off
stonith_host oss01 external/ipmi oss02 10.0.1.102 admin adminpassword
stonith_host oss02 external/ipmi oss01 10.0.1.103 admin adminpassword
node oss01
node oss02

6) /etc/ha.d/authkeys changes
MGS/MDS Pair
# cat authkeys | grep -v '#'
auth 2
2 sha1 SetYourMGSMDSPhasphraseHere
OSS Pair
# cat authkeys | grep -v '#'
auth 2
2 sha1 SetYourOSSPhasphraseHere

7) /etc/ha.d/haresources changes
MGS/MDS Pair
# cat haresources | grep -v '#'
mgs Filesystem::/dev/MGTDISK::/mnt/lustre/mgt::lustre
mds Filesystem::/dev/MDTDISK::/mnt/lustre/mdt::lustre
OSS Pair
# cat haresource | grep -v '#'
oss01 Filesystem::/dev/OST00DISK::/mnt/lustre/ost00::lustre
oss02 Filesystem::/dev/OST01DISK::/mnt/lustre/ost01::lustre
oss01 Filesystem::/dev/OST02DISK::/mnt/lustre/ost02::lustre
oss02 Filesystem::/dev/OST03DISK::/mnt/lustre/ost03::lustre

8) Specify the address of the failover MGS node for all Lustre filesystem components
mds # tunefs.lustre --writeconf --erase-params --fsname=lustre --failnode=10.0.0.101 /dev/MDTDISK
oss01 # tunefs.lustre --writeconf --erase-params --fsname=lustre --failnode=10.0.0.101 /dev/OST00DISK
oss01 # tunefs.lustre --writeconf --erase-params --fsname=lustre --failnode=10.0.0.101 /dev/OST01DISK
oss02 # tunefs.lustre --writeconf --erase-params --fsname=lustre --failnode=10.0.0.101 /dev/OST01
oss01 # tunefs.lustre --writeconf --erase-params --fsname=lustre --failnode=10.0.0.101 /dev/OST02DISK

9) Execute 'service heartbeat start' on MGS/MDS pair

10) Execute 'service heartbeat start' on OSS pair

11) Mount the Lustre filesystem on all clients
client # mount -t lustre 10.0.0.101@tcp0,10.0.0.100@tcp0:/lustre /mnt/lustre
client # cat /etc/fstab | grep lustre
10.0.0.101@tcp0,10.0.0.100@tcp0:/lustre /mnt/lustre lustre defaults 0 0

With the above setup, if a single node within each pair (MGS/MDS and OSS01/OSS02) were to fail, after the specified timeout period the clients would be able successfully recover and continue their I/O operations. Linux-HA is not designed for immediate failover, and a recovery can often take on the order of minutes when resources need to move from one node in a pair to another. While this solution will not provide immediate failover, it will allow administrators to setup an inexpensive system that will automatically recovery from hardware failures without lengthy downtimes and impacts to users.

Security Planning in HPC

2008-11-17T06:25:00.000-08:00

Todays high performance compute (HPC) solutions have many components including compute nodes, shared storage systems, high capacity tape archiving systems and shared interconnects including ethernet and Infiniband. One primary reason companies are turning to HPC solutions is the cost benefits of shared infrastructure that can be leveraged across many different projects and teams. While this shared usage model can allow for managed, cost effective growth, it also introduces new security risks and requirements for policies and tools to ensure previously separate data is managed properly in a shared environment.

This shared infrastructure model that is often used in HPC has several areas around data security that should be addressed prior to deploying shared solutions. Often times companies will have departments working on sensitive work while others work on very public projects, other firms could be working with their customers proprietary data and most companies have a threat from outside competitors trying to gain access to confidential work. All of these issues must be addressed in shared HPC solutions to ensure data is always secure, a reliable audit platform is in place and that security policies can be changed in a rapid fashion as company needs and policies change.

When evaluating an HPC solution to ensure data access is managed within company policy, there are several components within the cluster that should be reviewed separately:

Shared file systems – Todays HPC solutions have become rapidly successfully because of the availably of massively parallel file systems. These are scalable solutions for doing very high speed I/O and are often times available on all nodes within a cluster.

Databases – More often then ever companies are utilizing databases as a way to organize massive amounts of both transactional and reporting data. Often times these databases are paired with HPC solutions to evaluate the data in a very scalable and reliable method. These databases often contain a variety of data including sales, forecasting, payroll, procurement and scheduling just to name a few.

Local disk – More often then not, compute nodes have local disk in them to provide a local operating system and swap space. This swap space and possibly temporary file systems can provide a space for users to store data while jobs are running, but is also a location that must be assessed to ensure access is provided to those that need it.

Compute node memory – Compute nodes also have local physical memory that could be exploited by software flaws to allow unexpected access.

Interconnects – Todays HPC systems often use a high speed interconnect like Infiniband or 10Gbit Ethernet, these, like any other type of network connections present the opportunity for sniffing or otherwise monitoring traffic.

Policies
Todays companies often work for a variety of customers, as well as work on internal projects. It can be a complicated balancing act ensuring that data access policies are in place to properly handle those cases. Some data will require very restrictive policies, while others will require a very open policy around usage and access. Often time separate filesystems can be utilized to ensure data is stored in manageable locations and access granted pursuant to company policies.

There are two primary components to developing these security policies, first is to assess the risk associated with each component of the system, this risk assessment can include costs in dollars, costs in time and public perception if data was to be handled incorrectly per industry best practices or legal guidelines. Policies can then be developed to mitigate that risk to acceptable levels.

Some common methods to mitigate risk across the above components are:

Data Isolation – Within a shared computing environment data can be isolated in a variety of ways including physical isolation using different storage arrays, logical isolation using technology like VLANs and access restrictions like file permissions.

Audit Trails – Considering audit trails and how to implement them is important. This ensures that there is both a path to isolating and resolving problems, but also that legal compliance regulations are met. Audit trails can include system log files, authentication log files,resource manager logs and many others to provide end to end documentation of a user and their activities.

Consistent Identity Management – To properly ensure that data is accessed by the correct individuals and audit trails are consistent it is important to ensure identity management solutions are in place that handle HPC environments, as well as other enterprise type computing resources in a consistent method. Identity Management can be provided by tools like LDAP and Kerberos, as well as more advanced authentication and authorization systems.

Notifications – Notifications are an important part of the change management process within an HPC environment. Notifications can include alerts to security staff, administrators or management that portions of the cluster are out or company compliance, or attempts to access restricted resources have occurred. Notifications can come from a variety of tools within an HPC environment, but should be uniform in format and information so that staff can respond rapidly to unexpected cluster issues.

Data Cleanup – Often jobs within an HPC environment will create temporary files on individual nodes, as well as on shared filesystems. These files have an impact to a systems risk assessment and should be properly cleaned up after they are no longer needed. By removing all data that is not needed, it limits that data that needs to be accounted for, as well as the potential exposure if a single system is compromised.

We have just finished reviewing risk assessments within an HPC environment. These allow management and administrators of HPC systems to understand the costs (political, financial, time) of any failure in security plans or processes. In addition to understanding risk, there is the added complication of enforcing these policies in a way that is consistent across the cluster, consistent across the company and provides a proper audit trail. The most common methods of software implementation for these security policies are:

File System Permissions – File system permissions are the most common place to implement security controls, as well as one of the easiest items to complete and ensure compliance with. These permissions allow administrators at the lowest level to grant and deny access to data based on need. These do not assist with restricting back access to unauthorized individuals, but do contribute to ensuring that day to day operation of the system is done reliably and security.

Centralized Monitoring – Centralized monitoring and policy management are key to ensuring consistent security and minimizing human error. By using a central repository for all log entries, it allows staff to implement tools to rapidly catch any activity that is unauthorized or unexpected and respond with the proper speed. Centralized policy management through the use of tools like Identity management allow staff to quickly add or remove access based on business needs. By centralizing this policy management a company can ensure that the often manual process of removing access is removed and proper checks are in place to ensure access changes are updated accordingly.

Resource Manager – Most modern clusters make use of a job scheduler, or resource manager to allocate nodes and other resources to individual users to complete jobs. Most schedulers allow the allocation of resource groups and restrictions on those groups to an individual user or users. By extending this functionality it is possible to restrict users jobs to run on systems that have data they are allowed to see, and ensure they can not access nodes with filesystems they do not have permissions to utilize. The resource manager is a centralized tool that provides great flexibility in ensuring users have access to the resources they need, but no other resources or data.

Mounted File Systems – Often times HPC environments will utilize a variety of automated tools to unmount and remount filesystems based on user access needs. By un-mounting a filesystem that is not required for a given user, it adds an additional level of access protection above file permissions to ensure only authorized users access the data contained on a given filesystem.

Shared infrastructure is a challenge in all environments when assessing security solutions. A shared infrastructure means that additional precaution must be taken in implementation and security policies to ensure that data and resources are used when expected and by only authorized individuals. When planning a shared environment the initial process should begin with a risk assessment to understand what components of the solutions could be exploited and what the costs in time and money would be if that were to occur. That risk assessment can then be used to ensure the proper safeguards are implemented with available technologies to reduce the risk to a manageable and acceptable level for the company. Ultimately all safeguards should be implemented in a way that limits the potential for accidental failures in safeguards and reduces the need for manual administration and intervention. Shared resources are a challenge, but when properly managed, can ensure better overall utilization for a company without sacrificing on security.

Security Threats and Response Plans

2008-11-03T04:44:00.000-08:00

Risk management is an important component of a complete security plan for any company. In the area of cyber security this often has two fronts; assessing security threats and documenting responses. Both are equally valuable, and if planned correctly can ensure that no matter the threat a company faces, there is processes in place to properly manage, communicate and eliminate the threat. In todays security environment, a threat can mean a variety of things including viruses, data compromises, lost laptops, network intrusion attempts, insider threats and physical compromises.

This risk planning also has other purposes outside of planning for and responding too threats. This information, once gathered, can also be used as a basis for understanding risk around different types of threats. Often lower level threats have such a low level of risk that responding too all of them would be a waste of company resources, yet more complex attacks require faster, more urgent response. These risk assessments can also help staff plan the appropriate solutions around patch management, firewalls, network controls and other tools meant to stop intrusion. By properly understanding threats and there potential impact to production services, and staffs' time, a proper mitigation plan can also be developed.

Threat Matrix
Planning for threats can often be a daunting task, even for the most seasoned of security professionals. The challenge comes from the inability to know what exact threats are in the wild every day, and the new threats that are constantly emerging. There are many details that will need to be documented and considered when planning for the various known threats that are in the wild, these include who is causing the threat and who is the target, how is the attack being carried out, what safeguards are being affected as part of the threat, what changes will be needed to eliminate the threat, what is the cost of responding to the threat, both in prevention and if the threat is successful.

By planning and carefully documenting the process for responding for known threats, we develop experience that can then be used for responding to unknown threats. The following questions and information gathering can assist in developing this threat matrix. This matrix serves as a starting point when responding to known threats, and will be added to as new threats are encountered.

Who is causing the threat?
This question can have multiple components, it looks at whether the threat is being caused by someone internally or someone externally, as well as is the threat caused by a person, or rogue software. This is an important point to assess all threats to ensure that the safeguards added are in the correct place to mitigate the threat, and that resources are in the appropriate places to respond to the threat.

Who is the target?
A proper understanding of the target is important so that the impact of the threat is understood. From the target we can ascertain if sensitive customer information is at risk, if availability of public services is at risk or if we are at risk for a legal compliance issue. By who, I specifically mean what server, host, application, database, router, firewall or any other device that could be attacked.

This information can also be used to track developing patterns of attack. As these threats are rolled into response plans, a plan of documentation can ensure that patterns of threats are tracked and managed properly.

Avenue of attack?
This question evaluates how the threat is affecting a companies infrastructure. This could be technical avenues like via outside network connections or email, but can also be physical level like a person in your building or outside the building. Understanding the avenue of attack is critical to responding so that an attach response does not cause undue outages to other portions of the infrastructure or unnecessary outages to customer facing services.

Safeguards affected?
It is important to understand what safeguards are potentially compromised by a given threat. This could include firewalls, application validation checks, database encryption or filesystem encryption. Understanding the affected safeguards will later allow processes to be developed that mitigate the threat as quickly and efficiently as possible by understanding how best to stop the threat.

Changes to stop the threat?
This is a detailed list of what configuration changes or otherwise will need to be implemented to stop the threat. These are used to develop the process in the Response Matrix to slow and eliminate the threat. Understanding these responses is also important so that a risk versus reward analysis can be done. Often times, the change to eliminate the threat is so drastic that other problems with services result. By understanding what changes are required, management can make informed decisions about ignoring the threat, or what various response to use against the threat.

Cost of responding?
Understanding the implications of a threat are important when developing an appropriate level of response to the threat. The cost of responding can be communicated in multiple ways including cost in dollars or cost in time. All are important factors to use when deciding on response plans to threats, and the level of risk associated with various response plans.

Cost if threat is successful?
The other important cost associated with responding to a threat is the cost if the threat is successful. This could mean many things depending on the type of threat; it could be an outage in customer facing services, a loss of customer data or the financial impact of not providing the services to customers.

Current mitigation plan in place?
This item assess the safeguards that are in place to mitigate a given threat. This can include firewalls, security patches, passwords, identity management solutions or a host of application layer safeguards related to data scrubbing and input validation.

By no means are these all the items that should be assessed in a threat matrix. These are the most common ones that most companies will have documented for all threats. Additional information can be included in the threat matrix for specific applications, operating systems, network types and levels of data that a company processes. When developing a threat matrix, a company should evaluate all applications, hosts, networks, network connections and associated tools. By evaluating these items a list can be developed that includes the items that could be attacked, and what methods could be used in an attack.

Response Matrix
After defining the known threats, a response matrix can be developed specific to the companies needs and risks. These needs and risks can be calculated from the cost portion of the above developed matrix of threats. These needs and risks can be used as a basis for planning resources around responses, legal obligations when threats are encountered and documentation policies around responding to threats. This response matrix should contain detailed procedures for responding to several categories of threats.

The initial component of all response matrices is a list of types of incidents, this can usually be broken down into the follow categories, known as threat types:

Known incidents that have previously been experienced and have a documented response plan
Known incidents that have not been experienced, but have a response plan in place
Unknown incidents that do not have an associated response plan

Responding to threats and attacks must be a methodical process that encompasses many challenges including speed, communication, documentation and follow up. All of these must be managed while ensuring that customers and staff are impacted as little as necessary. Responding must be a coordinated effort between the various teams within a company that are responsible for system administration, data security, compliance, network administration and enterprise architecture.

Threat Type 1
Threat type one is the category that a company will know the most about and be able to plan in a detailed way how to respond. This category will be threats that have previously been responded too and resolved. Each time they have previously occurred a follow up meeting should have been done to revise and improve the process for the specific threat.

Threat Type 2
The threats that will be listed in the second category will include well known exploits and attack avenues as well as threats that other companies have actively faced. These threats will also be listed in the threat matrix, although with a side note that the company has not previously had to respond to them, but does anticipate the threat.

Threat Type 3
This category is often the most complex to respond too because the actual threats are unknown. The process for responding to unknown threats must be dynamic enough to handle a wide enough range that all threats are properly responded too, but rigid enough that legal implications are handled and communication channels do not break down in the face on unknown threats.

Internal versus External
One important component to all response plans is understanding if the threat you are planning for is internal or external. Internal threats come from staff that are either intentionally out to cause the company harm, or systems that are setup in a way they allow staff access that was not intended and subsequently has negative consequences.

Often times the threat, be it internal or external, plays an important role in how the company responds. If the threat is internal, it is often important to bring in outside resources to assess the problems and develop a mitigation plan, ensuing that the company is not vulnerable from future insider threats.

Response Teams
Another key component of all response matrices is a carefully planned list of individuals and teams to be included in response activities. As a company, you should evaluate if you have the appropriate level of technical capability in house to respond to known and unknown threats, as well as what additional capabilities need to be brought in when responding to threats. Outside resources could be technical staff specialized in security, or marketing staff focused on public relations issues, it could even be law enforcement to track down the source of threats. Today a lot of companies also ensure that when responding to major threats, legal council is brought in to ensure compliance with data handling and reporting requirements.

Response Training
Each type of threat that might be responded too should have associated required training for staff. This could include computer forensics, data analysis, legal implications or technical skills. This training, done on regular intervals, ensures that staff have both a process and the proper training to effectively respond to threats.

While these are not all the possible categories for each response within the response matrix, they begin to provide a basis for response planning. Additional items can be included in the response plan based on company specifics, industry legalities and management preferences. All response plans should be detailed enough that staff have clear directions to follow in possibly chaotic situations. Response plans should also have regular reviews of the process to ensure they are updated to reflect changes in company management structure, changes in technology and changes in industry trends.

Response Methods
The majority of this document has been focused at defining a matrix of threats that a companies information systems face. These threats can then used as a basis when defining responses in a coordinated fashion, while most of the writing was about defining manual processes, this is only the first step to automating the responses and processes. After defining the manual processes for responding to threats, a system of automation can be put in place for the responses that make sense. Automated solutions work very well for defined threats that have clear responses; a good example is a service on the company network being attacked by an outside system, an automated system that blocks the source of the attack and notifies staff ensures that the threat is immediately contained and staff notified in a timely manner.

Automating the response plan can also aid in the communication of threats and coordination of activities. There are a variety of tools available for tracking incidents, most include the ability to automatically notify the correct staff about status changes of an incident, and provide automated methods to escalate issues between groups and individuals. These tools ensure not only smooth communication during a normally chaotic time, but also a good audit trail after the fact to review an incident to plan for better responses the next time.

Responding to security threats within a company can often be a chaotic time. The more time that is spent up front identifying threats, and developing response processes, the more effectively a company can both understand and respond to threats. A clearly defined process for threat response can ensure that no steps are missed, lessons are documented for future use and communication between teams is effective and efficient. The constantly changing security threats in todays environments means that process is critical to ensure staff are prepared and respond accordingly to all threats, known and unknown.

Building a new Lustre Filesystem

2008-10-17T07:33:00.000-07:00

Here are the quick and dirty steps to create a new Lustre filesystem for testing purposes. I use this at times to test out commands and test benchmarking tools, not to test performance, but to ensure they operate correctly.

This is a simple test environment on a single system with a single physical disk. Lustre is designed for scalability so these commands can be run on multiple machines and across many disks to ensure that a bottleneck does not occur in larger environments. The purposes of this is to generate a working Lustre filesystem for testing and sandbox work.

This set of directions assumes you have compiled and installed both the Lustre kernel and the Lustre userspace bits. Check my previous blog posting for how to complete those items if necessary. This also assumes that you have a spare physical disk that can be partitioned to create the various components of the filesystem. In the example case below I created the filesystem within a Xen virtual machine.

1) Create a script to partition the disk that will be used for testing (using /dev/xvdb for example purposes)
#!/bin/sh
sfdisk /dev/xvdb << EOF
,1ooo,L
,1000,L
,2000,L
,2000,L
EOF

2) Format the MGS Partition
- mkfs.lustre --mgs --reformat /dev/xvdb1
- mkdir -p /mnt/mgs
- mount -t lustre /dev/xvdb1 /mnt/mgs

3) Format the MDT Partition
- mkfs.lustre --mdt --reformat --mgsnid=127.0.0.1 --fsname=lusfs01 /dev/xvdb2
- mkdir -p /mnt/lusfs01/mdt
- mount -t lustre /dev/xvdb2 /mnt/lusfs01/mdt

4) Format the First OST Partition
- mkfs.lustre --ost --reformat --mgsnid=127.0.0.1 --fsname=lusfs01 /dev/xvdb3
- mkdir -p /mnt/lusfs01/ost00
- mount -t lustre /dev/xvdb3 /mnt/lusfs01/ost00

5) Format the Second OST Partition
- mkfs.lustre --ost --reformat --mgsnid=127.0.0.1 --fsname=lusfs01 /dev/xvdb4
- mkdir -p /mnt/lusfs01/ost01
- mount -t lustre /dev/xvdb4 /mnt/lusfs01/ost01

6) Mount the client view of the filesystem
- mkdir -p /mnt/lusfs01/client
- mount -t lustre 127.0.0.1@tcp0:/lusfs01 /mnt/lusfs01/client

At this point you should be able to do an ls, touch, rm or any other standard file manipulation command on files in /mnt/lusfs01/client.

Building Lustre 1.6.5.1 against the latest Redhat Kernel

2008-10-16T18:12:00.000-07:00

I was at a customer site this week and had the need to build Lustre 1.6.5.1 against the latest kernel from Redhat, 2.6.18-92.1.13. Being this process has multiple steps, I thought I would document it so that others do not have to reinvent the wheel.

1) Prep a build environment
- cd ~
- yum install rpm-build redhat-rpm-config unifdef
- mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
- echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
- rpm -i http://mirror.centos.org/centos/5/updates/SRPMS/kernel-2.6.18-92.1.13.el5.src.rpm
- cd ~/rpmbuild/SPECS
- rpmbuild -bp --target=`uname -m` kernel-2.6.spec 2> prep-err.log | tee prep-out.log

2) Download and install quilt (quilt is used for applying kernel patches from a series file)
- cd ~
- wget http://download.savannah.gnu.org/releases/quilt/quilt-0.47.tar.gz
- gunzip quilt-0.47.tar.gz
- tar -xvf quilt-0.47.tar
- cd quilt-0.47
- ./configure
- make
- make install

3) Prepare the Lustre source code
- Download from http://www.sun.com/software/products/lustre/get.jsp
- mv lustre-1.6.5.1.tar.gz /usr/src
- gunzip lustre-1.6.5.1.tar.gz
- tar -xvf lustre-1.6.5.1.tar

4) Apply the Lustre kernel-space patches to the kernel source tree
- cd /root/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/
- ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel5.series series (there are several diffrent series files in the series dir, choose the one closest to your environment)
- ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/patches patches
- quilt push -av

5) Compile a new kernel from source
- make distclean
- make oldconfig dep bzImage modules
- cp /boot/config-`uname -r` .config
- make oldconfig || make menuconfig
- make include/asm
- make include/linux/version.h
- make SUBDIRS=scripts
- make rpm
- rpm -ivh ~/rpmbuild/RPMS/kernel-2.6.18prep-1.x86_64.rpm
- mkinitrd /boot/initrd-2.6.18-prep.img 2.6.18-prep
- Update /etc/grub.conf with new kernel boot information

6) Reboot system with new, patched kernel

7) Compile Lustre with the new kernel running
- cd /usr/src/lustre-1.6.5.1
- ./configure --with-linux=/root/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
- make rpms (Build RPMs will be in ~/rpmbuild/RPMS)

8) Install the appropriate RPMs for your environment

Getting Ahead in IT

2008-10-08T15:06:00.001-07:00

I speak with folks in Information Technology (IT) regularly that tell me how hard it is to get ahead in IT and that there are too few opportunities for promotion in IT. I hear this from folks at all levels including architect, developer, administrator and team management. While getting ahead in IT can be difficult, it is not impossible. IT is often the last organization within a company that Human Resources (HR) considers when developing career paths, career training and mentoring plans. There are a variety of things that people in IT can do to ensure they get noticed by management and advance as a result.

Have a career path
First and foremost in IT it is important to have a career path, know what you want to get out of your career in the immediate future, in 5 years, and beyond. This will allow you to strategically pick projects that increase your ability to meet these goals. This plan will also allow you to speak with your manager, HR and other company leaders about training, mentoring, and other activities to increase your skill set.

A career path is not necessarily a goal to be promoted or to obtain a title that you would like. While it can be those things, it is more often targets for growth in technical capability, a goal to become management, or a goal to develop skills in one role that will enable you to deliver more efficiently in another role. Each of these goals requires a different focus on skills development, but all require the same open communication with your current management about your goals.

Focus on Business Needs
Second, being in IT does not preclude you from participating in the business. IT is an integral part of any business, and the IT folks that excel are the ones that understand how IT can help the business grow and be more efficient. Working to establish an understanding of the companies core principals will allow you to suggest improvements in IT, as well as establish a value within the company.

Very seldom do IT folks participate in business focused meetings. This is a shame. This is a very good opportunity for IT folks to not only learn about a companies' operations, but it is also a good chance to suggest new and better methods that IT can provide for the core business of a company. The IT staff that take the initiative and participate in the business discussions will be the ones most noticed as new opportunities within the company open up.

Bring Forward Ideas
Being noticed is important in all careers, especially in IT where management often forgets how critical IT is to the success of a company. The easiest way to be noticed is to speak up, if you have an idea for improvements, be it a new tool, an improved process or a new piece of hardware, suggest it to management. If your immediate manger does not see the value, mention it next time you are talking to other managers. Management at all companies appreciate individuals that are proactive enough to suggest ways for improvement before being asked.

Set Boundaries
At high stress times when projects are due and deadlines are tight it may not seem like it, but managers will notice and respect you for holding to your principals. Setting boundaries is important in all jobs, especially in IT where often long hours are the norm and expected. Work with your management to let them know about outside obligations, either organizations or family. This will ensure your management is aware of other things you are involved in. Most importantly hold to those boundaries. It is normal to have a long evening when a project is due, or the systems are down. The important thing is to stick to your boundaries and do not allow one late evening to morph into constantly working excessively long days.

Setting boundaries will also help keep stress within a reasonable level. If you can keep your stress lower, you will not only be a more efficient employee, but you will be able to more effectively assist others and complete your own projects.

Foster an environment of personal development
Often times folks say that a work environment is created by management, this is only partially true. A dedicated staff member can also create an environment where others share, feel comfortable asking questions and learn. By letting others know you are available for questions or conversation, you are letting others know that you are willing to help others develop their skills and experience. This attitude can be used to influence management at all levels to formalize personal development plans.

Staying positive is an important component of developing an environment people want to work in. No one wants to be known as the angry employee. It can be tough to keep your cool at all times, but being the level headed employee goes a long way to making yourself approachable and creating a comfortable working environment.

Be a Team Player
Everyone is told from childhood to be a team player, but what does that mean in todays business environment? Being a team player is a combination of sharing the work load, accepting projects that benefit the team as a whole and ensuring information is shared for the benefit of the team.

Most importantly, especially for folks that manage teams and projects, do not pass along a task because you do not want to do it. The quickest way to get noticed is step up and complete the work that needs to be done, but would otherwise fall by the way side because folks do not find it interesting. You certainly do not want to make a career out of working on uninteresting projects, but picking up one now and again will not be a career killer and will get you some recognition.

The most efficient way to show you are a team player is to not work in a vacuum. As you and your team work on projects, solicit the input from other teams. Use there input to carefully evaluate your teams' assumptions and project goals, and make adjustments as necessary. This shows that you value their input and experience, and will ultimately enable you to create a better product for the company.

Document, document, document. In todays world where people regularly move roles and companies, it is critical to ensure that all tasks, no matter how trivial, are documented so that others can complete them. If your company has a wiki, use it. If your company does not have a wiki, put one online for others to use. The fastest way to develop mindshare within a company is to be the person who has contributed the most to internal repositories. By ensuring your tasks are carefully documented, you are telling company management you are not trying to force them into keeping you, you are telling them you care about the companies long term success and letting them know you can be moved to other roles and new staff can take over your tasks.

Moving On
Ultimately, some combinations of employees and companies will not mesh well. In times like that it is appropriate to look at opportunities elsewhere. There are a lot of companies in IT today, and often times a different one will provide you the opportunities you are looking for. But make sure that any time you are looking to move companies you closely assess why your current company is not providing what you need to meet your goals, and work to find a place that will assist in meeting those goals.

When looking for a new role or starting a new role, remember that these things take time. It can often take months or years to feel at home at a new company and really feel like you are a highly contributing member of the team. When looking for a new role, discuss with your potential manager how long staff have been at the company, what types of development opportunities they offer and how the team dynamics work. This will ensure that any job change is meaningful and a path to newer and better things.

IT is not the dead end that it is often made out to be. It is also not a simple process to make yourself known in a large pool of people and advance in IT. To succeed in IT you must have a clear set of goals for your career, and use those to develop a clear list of activities to meet those goals. Moving up is not an immediate process, but by committing the time to development and communication, you can let management know you are willing and capable to take on new challenges and meet your career goals in the process.

Succeeding in todays services driven IT market

2008-10-05T12:10:00.001-07:00

The Information Technology (IT) space is undergoing a dramatic shift. This shift from hardware and software based sales processes will have a dramatic impact for those executives that manage IT vendors, those who sell IT solutions and those who market them. As hardware prices have fallen and hardware has become more commodity in nature, companies are focusing less time on purchasing hardware, and more on ensuring that their business needs are being met by their IT systems. Companies are beginning to realize that IT can be an enabler to ensure that their primary business is run as efficiently as possible. This realization is opening new markets around services, primarily highly complex consulting and integration focused services.

Service Definitions
To enable organizations to successfully meet these new services driven customer needs, there is a separation occurring within a lot of services organizations. This separation is usually along operational lines to enable each type of services delivery organization to effectively delivery value to their customer, in a scalable manner. Organizations can be broken into four distinct services delivery teams, Professional Services, Consulting Services, Managed Services and Support Services.

There are a variety of interpretations today about what types of offerings are available from a services organization, and how they are branded publicly. When speaking about Professional Services, I envision an organization that is focused on product delivery and integration. Professional Services are often the organization a vendor uses for deploying their hardware and software in customer environments. Professional Services personnel are often experts in a companies portfolio, as well as the products integration with other offerings on the market.

Consulting services are a higher caliber of Professional Services in my experience. Consulting Services are offerings around custom integration or custom development, either system or software. Consulting Services tend to be more complex deals that run longer, and do not necessarily have a hardware or commercial software component as Professional Services would.

Managed services are often used as a way to ensure a vendor has a long term presence at a customer site. Managed Services offerings are often provided to the customer to manage the on-site hardware and software that a customer has purchased, but does not have the staff to operate day-to-day. Managed Services are often long term agreements for a company to ensure a customers' IT operations are stable and managed per industry best practices.

Finally, Support Services. Support Services are typically the contracts that are purchased with hardware and software to entitle the owner to a clear path for product assistance. This is most often phone support and access to patches for the product for bugs and security vulnerabilities.

Each of these four offerings is distinct, they each have their own lifecycle, associated costs and required skill levels for delivery. It is important to distinguish the various service offerings when developing sales strategies, as well as delivery methodologies. Each one is a different type of purchase for the customer, and has different implications to the cost/benefit trade off analysis work that customers do when purchasing services.

Hardware is cheap
One large influence on the drive towards services and specifically Consulting Services is the drive towards cheaper hardware. Hardware today is based on standards and commodity parts that enable a larger number of vendors to sell the same capabilities and components. Because of this commonality around features, customers look mostly to price when comparing two similar pieces of hardware. The companies that strive are the ones that realize hardware is only a platform for running a business; the real value to companies in todays fast paced market is putting highly capable solutions on this common hardware to enable a customer to be more successful.

Many companies today rely on a predictable, regular refresh cycle for all hardware. This enables companies to position themselves to deliver solutions around managing services on top of this regular refresh. This refresh requires companies to ensure that data and applications are implemented in a way that when the next hardware refresh comes, the data and applications can be easily migrated. Often times customers do not posses the necessary staff in house to implement these types of software provisioning, they will turn to consulting organizations to implement integrated solutions around these refresh cycles.

Fixing Business Problems
Todays customers are looking more and more to IT as a way to enable their core business in a more efficient way. Customers are using data warehouses as a way to process vast amounts of data to ensure the business is being managed correctly, they are using customer resource management systems to ensure customer requests are handled efficiently and correctly the first time, and they are using mobile devices to connect remote workers to the office and get them information as soon as possible.

Often times companies do not have the necessary staff in house to both implement and manage todays complex solutions. Customers often must balance having too many versus too few staff, and often will lean towards fewer staff, and contracting the complex implementations and projects that require more time then staff immediately have.

Customers today will also look to outside services for guidance in business inefficiencies. Often customers see a value to outside input when reviewing legacy processes, this outside input can ensure that new processes are developed with an understanding of currently available technology and tools that can assist with driving productivity. These business efficiencies have a multitude of possible solutions including business intelligence tools with an associated data warehouse, a formal enterprise architecture program, automated provisioning of new services or automated software development assistance.

Business Intelligence
Todays businesses rely on information for making decisions, as well as reviewing previous decisions. in a systematic way. This information must be organized and have associated tools for reporting. Often times companies will look to outside firms to assist them with managing their sales information, forecasting, process assessment, manufacturing data and purchasing. Todays business intelligence solutions rely on expertise in these areas in data management, data mining, data cleansing and ultimately reporting using accurate and proven methods.

Enterprise Architecture
Enterprise Architecture is growing in popularity as companies look to formalize how business processes and company visions become IT systems. Todays TOGAF and the Zachman Frameworks are used by countless organizations to formally document the architecture that IT will follow for implementing systems, tools, software and support services. Few companies today have the expertise in house to develop a formal Enterprise Architecture program, and because of that will look to outside companies that have expert level knowledge and experience with the above frameworks.

Automated Provisioning
Speed is an important factor when doing business today. Companies that can rapidly adapt to change are more successful in meeting customer demands and needs. A companies IT systems are a critical component to all adjustments to market environments. By being able to more rapidly provision new services, or capacity for existing services, companies can ensure they are ready for this change. Automated provisioning ensures that minimal staff intervention is needed when bringing online new services; this both lowers the time to market as well as the costs associated with bringing new capabilities and capacity to market. Companies will often look for experienced outside assistance when developing automated provisioning systems, this outside experience can ensure that new services are brought online both efficiently and correctly.

Software Development
Software development can be a complicated orchestra including requirements gathering, architecture, development, internal testing, and finally customer testing. Companies will often look for external assistance with developing unit tests and automated regression testing environments. Outside resources can provide a unique perspective on the development and testing process because of there disconnect from the entire development process, they are able to focus all efforts on testing for defects and usability.

Selling to the decision makers
This shift in IT purchases from hardware to services has a dramatic impact on the sales process for vendors, particularly those that sell both hardware and services. As more and more IT solutions are purchased that are directly tied to company objectives, fewer purchases will be made by the managers and staff implementing the solutions. More and more large IT purchases are being made by a companies executives including the CIO, CFO, CTO and COO. These individuals are no longer focusing on the technology behind the products, they are looking to vendor solutions and offerings as a way to increase productivity, increase output, and to better understand and manage their business.

When selling services and solutions today, sales teams must articulate to potential customers the immediate and long term costs of solutions, and how those costs will directly affect the bottom line of the business. The cost of individual servers, licenses and data centers are no longer looked at with the level of scrutiny they once were. Today purchasers are looking to the total cost of a solution in implementation costs, reoccurring costs, and upgrade costs; then comparing those costs to the measurable benefits once the solution is in place.

Considerations for solution based purchases
Now that we know who is making the decisions around big IT purchases and that hardware is a small piece of the equation, we need to understand what items influence these decisions. These will enable solution developers and sales teams to properly positions services within a customer organization to enable effective business solutions.

Life-cycle Support
Customers look at vendors to provide them solutions that will meet the immediate needs as well as provide paths as the company grows aver time. The paths can be upgrades, accommodation of new legal requirements, growth and manageability. As part of the sales process it is critical to communicate to the customer a vendors capabilities around not only solution implementation, but also upgrades, changes and solution support.

Solution Ownership
Todays companies look to do business with vendors that will own solutions from end to end. This ownership requires the vendors to have solid methodologies around product development, delivery, support and upgrades. This does not mean a vendor needs to develop all products in house, or have a software package for every customer, but vendors should be able to provide their customers a single point of contact for all phases of complex projects.

Solution Flexibility
Companies today want to ensure they are not locked in to any specific solution, be it hardware, software or a specific consultant. For a consultant to put in the extra effort to ensure a solution is properly documented and communicated to the customer, shows the customer the dedication to their business and helping them succeed.

Cost Management
When purchasing services related IT solutions, companies today are looking at more then the initial cost of the contract, or the options. They are looking at the benefits the company will see because of the solutions, these benefits can be more efficient operations, more customers, or simplified growth paths. It is critical that as part of the sales process the costs and benefits are both understood and communicated to the customer.

To be successful in todays IT market, vendors must focus on correcting customers business problems, and work to become a trusted adviser in their business operations. Customers today are looking for long term solutions to their IT needs that will ensure they are competitive and able to not only grow, but change as the market demands. This has caused a dramatic shift away from purchases focused strictly on servers and storage, but to purchases of solutions. These solutions must have definitive cost returns over time that allow management to clearly understand how their business will be positively impacted.

Preparing for an IPv6 Deployment

2008-09-15T07:36:00.001-07:00

IPv6 is the talk of the Internet, there are varying degrees of urgency stating that we will run out of existing IPv4 space within 2 years, with some saying there is enough IPv4 space left for 10 years. No matter which prediction is correct, eventually, IPv4 space will be exhausted and companies will have to begin migrating to IPv6 to ensure the availability of publicly routable IP space. Few companies have begun to evaluate the problem in detail; the sooner companies begin to evaluate their infrastructure, the more smoothly they can plan a migration from IPv4 to IPv6 and the longer period of time they can amortize the costs over.

IPv6 is an upgrade to the most basic elements of the internet and the networks that connect companies, individuals and the devices we have become so accustomed to using like our Blackberrys, iPhones and laptops. Making changes to the basis of all connectivity is not an easy thing to accomplish, or even begin planning for. The dependencies are unique and well established over many years of additions, improvements and research around IP connectivity. In this paper I intend to break down the process for companies to begin evaluating this upgrade to there infrastructure so that a strategic plan for IPv6 deployment can be developed.

Deployment Options
When deploying IPv6 there are a variety of options to ensure that no services will be interrupted during the time that both IPv4 and IPv6 are operational; both at your company, and across the global internet. Both options are important to consider because they each can work for specific cases to provide a bridge to IPv6 enable a system or application.

Parallel Stacks
When evaluating options for implementing IPv6 without impacting existing IPv4 traffic, most companies are looking to vendors to provide parallel stack capability, also called dual stack in some cases. By utilizing a parallel stack solution, companies can bring up IPv6 capability in parallel to their existing IPv4 deployments. This ensures that services can be migrated as they are fully tested and validated on IPv6. This parallel stack solution does come at a cost because of the overhead of administering two separate logical networks within a single physical network.

IPv6 NAT-PT
Todays modern routers are also offering capabilities to do NAT for traffic between IPv4 and IPv6, and vice-versa. IPv6 NAT-PT is a capability to have devices in your network with both IPv4 and IPv6 addresses assigned to them, these devices can then used as gateways for devices to use as a connection point to newer IPv6 devices. IPv6 NAT-PT was designed to provide a step from IPv4 to IPv6. IPv6 NAT-PT is a very specific use of the above mentioned parallel stack; ensuring that devices that only speak one protocol, can access devices speaking the alternate protocol.

Assessment

Overall Questions
The first step to evaluating the impact of an IPv6 update is by reviewing the high level components of your Information Technology (IT) systems. This evaluation is to begin looking at vendor commitments, capability and simplicity of system upgrades and regulatory impacts:
1)Inventory all vendors you currently use, document what there current and future support plans are for IPv6? What assistance can they provide either through documentation or consulting services to assist with a migration?
2)Review all applications, which ones are developed in house and which are commercial software? Are all the commercial software vendors still in business?
3)Do you have any legacy systems that no longer are covered by support agreements?
4)Do you have any systems that are covered by federal laws for data consistency? What legal rules are in place governing how these systems are tested and maintained?

Infrastructure Tiers
The next step is to systematically evaluate each component that contributes to the operational capability of your IT systems:

Network – Todays networks are complex sets of routers, switches, Intrusion Detection Systems (IDS) and physical links between sites. To properly assess this portion for an IPv6 upgrade, an audit must be done for each device. It should assess what types of Deployment Options the device supports, how the vendor plans to support IPv6 on this platform and, what upgrades, either software or hardware will be required for IPv6 support.

VPN Infrastructure – Enterprises are increasingly reliant on VPNs to secure traffic in todays mobile workforce. The software and hardware supporting these VPN sessions needs to be tested and evaluated to ensure it will support future IPv6 connections and traffic as well as a mix of traffic during any transition periods.

Applications – Applications will be the most complex and time consuming component of the evaluation process. Most companies have many dozens of applications in place, if not more, that must be evaluated to ensure that they will properly migrate to IPv6. This assessment for each application will need to include outside dependencies like license servers, database servers or client software on users individual machines.

Monitoring Tools – Todays enterprises have a diverse collection of tools used for monitoring network usage, network performance, application usage, application availability, users connection. All this information is critical to both developing and ensuring compliance with SLAs. As part of a complete IPv6 assessment all monitoring tools, both performance and availably, should be evaluated to ensure they can provide the same level of detail in monitoring, as well as properly store and report data that could be IPv6 or IPv4 specific.

Core services – Core services, including DNS, DHCP, and file sharing are some of the most critical components to an IPv6 migration. These services form the basis for all user experiences and if implemented correctly, will ensure that a transition to IPv6 is seamless to the users.

Mobile Devices – Mobile devices are becoming a standard for doing business in todays mobile workforce. As you begin to develop your IPv6 transition plan, it is important to include these as part of the assessment to ensure they will continue to operate through the transition and when the transition is complete. You should begin by speaking with your mobile device vendors to understand if the carriers network will support your IPv6 plans, as well as your employees handheld devices. This assessment will allow you to develop a cost associated with upgrading or replacing units in the field.

End Users Systems – Todays mobile workforce means that many staff have a laptop and a desktop system at a minimum, with more then one laptop per person in a lot of cases. All these devices need to be evaluated for IPv6 support to see if they will support the proposed changes, and what, if any upgrades will be needed for full support. This will have an impact to both the schedule and cost of an IPv6 deployment.

Migration Timeline
Now that we have developed a list of what will need to be upgraded, and paired that with a list of what upgrades our vendors will support, we can use that to develop a process and timeline to test all necessary changes, upgrade appropriate systems and eventually move an an environment where IPv6 is fully operational across all IT systems. This planning stage, taking what we know will need to be updated and planning how to update and test it, is the most important part of an IPv6 migration. This stage is our best opportunity to ensure we understand the time commitments for this project, the costs this project will incur and the potential challenges we will run into.

As we develop a complete migration process, there are many angles that must be included to ensure all services rolled out are ready for prime time and allow your staff to be as proficient as they were in an all IPv4 world. We must ensure that we understand what software will need to be upgraded, what software re-written, and how to test those changes so that we do not introduce complications.

After we have a plan for making the appropriate software updates and testing them, you can develop a detailed plan for how to implement IPv6. This plan should include which services will be upgraded first, second and so on. This plan should also include what groups of users will be the first to migrate so that they can be made aware of the plans and provide input during the migration process. This input can then be used to make each subsequent step smoother then the previous one.

In addition to the migration plans and testing plans, plans for disaster recovery and maintenance will need to be updated. Because IPv6 is such a radical change from current technologies, most maintenance plans and disaster recovery plans will need to be updated to handle the varied techniques that will need to be used once IPv6 is in place and operational.

Industry Commitments
The last big question lingering after a through assessment of your infrastructure is, what about the rest of the industry and our vendors? That is currently a point of contention in the vendors space, vendors are hesitant to implement IPv6 capability until customers demand it, and customers are hesitant to implement IPv6 until vendors provide a fully support capability in there products. The most visible of this contention is with todays modern firewall products, today very few fully support the RFCs around IPv6, but as most companies look to IPv6, this is an early capability that must be in place to continue rolling out IPv6.

As time continues, more and more vendors will adopt and support IPv6 in the same ways they support IPv4. IPv4 has taken decades to grow to the point of adoption it is at now, along that path many, many enhancements have been made to the routers, switches and servers that power our enterprise environments. As time continues to progress, more and more customers will push vendors to add complete IPv6 capabilities, as they do it will enable a larger portion of companies to begin to fully embrace IPv6.

Hardware TCO – Predictable Planning with Refresh Cycles

2008-09-04T19:42:00.000-07:00

Often, the most expensive investment any Information Technology (IT) organization will make is its base infrastructure; servers, storage and other various hardware. Yet, these hardware purchases are often given much less thought then software or services purchases and assumed to be routine, and just a cost of doing business. Hardware typically has several phases that should be evaluated as part of the purchase, these include the initial purchase price, the cost of maintaining it, and the ultimate cost of refreshing the hardware at the end of it's useful life. All should play an equal balance when evaluating new platforms, refresh cycles and testing new solutions for introduction to a company.

Often times when a company begins to assess the total cost of ownership (TCO) around its IT assets, it must involve teams not traditionally involved in IT planning. These teams can include facilities, engineering, building managers, application developers and data base administrators. Each of these groups can provide valuable input on how the servers and other infrastructure affect there environments and costs on a yearly basis.

There are three primary phases to all hardware purchases:

Initial Hardware Purchase
The initial hardware purchase is often thought to be the most expensive phase, but in reality after factoring in the support costs for a piece of hardware it turns out to be about one-third to one-fourth of the TCO. The initial price is often the easiest to evaluate, but should carefully be weighed against the long term costs of purchasing a specific brand or type of hardware.

Often vendors will allow for additional years of warranty coverage, or higher levels of support to be purchased when the system is first bought. These are often a wise investment if the hardware will be used longer then the initial warranty period. The increased level of support can often mean that your staff will spend less time supporting the system, and more time working on more beneficial tasks.

Support
The support costs are often the most expensive phases of hardware ownership. The support costs include patching the operating system, supplying power to the system, cooling the system and managing the applications hosted on the system. These costs are amortized over the life of the system, and over time can add up to be the most expensive part of the TCO formula. Often, these support costs are also where the most efficiencies can be gained to lower the TCO of the system.

There are many things that can be done to lower the support costs around hardware, most involve improved processes to cut back the amount of time staff have to spend manually managing each specific server. The most notable of these is automation of patch management. By utilizing tools to automate patch deployments and status monitoring, staff can cut significant manual administration time from each specific server. Proactive monitoring of system and application health can also play an important role in cutting down TCO for hardware and associated services. There are many packages available today to assist system administrators in proactively correcting both hardware and software faults before they cause a failure for the end users. These apps can ensure that staff isolate and correct problems as soon as possible to minimize the necessary time to correct faults.

Utilization is another space where the TCO for your servers can be lowered. By ensuring that servers do not run at idle for long periods of time, you can ensure that any power being used by the servers is being used efficiently. Often times, a single server can handle the load that multiple servers used to handle. It is much more efficient to power and cool a single server then multiple servers in this case, and scales very well as you begin looking at utilization rates across dozens or hundreds of servers.

Refresh
The final phase to evaluate for all hardware purchases is the refresh cycle. All hardware has a finite lifecycle in which at the end it will need to be replaced because it is either obsolete and not cost effective to maintain any longer. Obsolete in this context can be used in two ways, first to mean the hardware is so old is can no longer operate with the current operating systems, tools and patches available, or it no longer meets the business needs of your company.

There are often two methods that companies use to replace aging hardware, the first and most common is just purchasing new hardware when a system is no longer under warranty or has gotten too slow to use in the IT environment. More and more though, firms are implementing a rolling refresh cycle to add a level of predictability to all hardware purchases. A rolling refresh cycle allows a company to more clearly outlay capitol for IT investments, and better plan long term cycles for purchases, upgrades and replacements. Typically, a rolling refresh schedule is based on the standard warranty with newer server hardware, 3-5 years. A rolling refresh cycle also allows IT staff to better plan work loads by knowing ahead of time that new servers will need to be configured, tested and put into production.

Refresh cycle planning should also include an assessment of upcoming technologies and how that will affect purchases two and three years down the line. Every year hardware is faster and faster then before, and provides new possibilities for the amount of data that can be processed. In addition new technologies around virtualization are changing the dynamic of how system administrators provision new systems. No longer do system administrators add a single new server because of a single new application, today many different apps can be run on a single piece of hardware and kept separate from each other by using virtualization technologies.

As you assess your refresh cycle an important part of the TCO calculation is determining what applications can run within virtualized environments, and which will need separate hardware to run on. This will determine what level of consolidation can occur from year to year as the refresh occurs. As you look toward implementing a rolling refresh cycle, a first step is to understand how many existing servers are in place and how many existing applications. That data can then be used to develop a matrix of how things would look if virtualization were employed, and how a rolling refresh cycle could be utilized over time to ensure that all pieces of the infrastructure are upgraded in an expected period of time.

Minimizing the TCO of IT hardware is a key component of ensuring that the long term costs of owning the hardware are predictable and manageable. A rolling refresh cycle, paired with newer technologies like monitoring tools, automation tools and virtualization can allow IT staff to clearly plan how hardware will be used from purchase to end of life and how it will then be replaced. This cycle ensures staff can plan for future upgrades and migrations, as well as avoid last minute unexpected upgrades.

Defining "the edge"

2008-08-28T10:42:00.000-07:00

I was in a planning meeting with a customer recently and we were assessing the customers security plan. We had two major topics to discuss, the first was in regards to data management and compliance. The second, and the one we discussed at length was in regards to previous policies they had around what they called “the edge”, the previous end of their network and the beginning of folks and systems they could not trust. The discussion went on for a while with us working towards consensus on how to define “the edge.” I believe we made the right decisions for there needs, but wanted to continue the discussion, I imagine most companies have this discussion at some point and will continue as new technologies evolve.

At one time "the edge" of any given network was easy to find; the last router between you and the upstream access provider. But today, "the edge" is getting increasingly difficult to find, and this has implications for the fundamentals of Information Technology (IT) including patching and password policies, and the most complicated of questions including privacy, monitoring and forensics. Today we have to evaluate many different details in regards to where “the edge” truly is, these include PDAs, company laptops with VPN access, employees home systems, thumb drives, and outside vendors/contractors.

The most important implications around defining what constitutes “the edge” is defining how customers, and staff will be able to access servers, services and storage. By clearly defining “the edge” we can then work to define what services will be publicly accessible, and which ones will be restricted by VPN access, firewalls, or other mechanisms. By defining “the edge” we also have a baseline to use when defining policies for information management, information tracking and information retention. These are critical areas in todays world of compliance, being able to precisely say who accessed and stored what day and when, is almost necessity.

When defining “the edge”, I start by listing all possible devices (laptop, desktop, thumb drive, PDA, cell phone, etc) that an employee or partner could use to access data that is not publicly available. This should be a list of devices currently allowed and possible technologies to employ. This data could include sales presentations, engineering documents, support forums, or any other data that is intentionally kept private to provide a competitive edge in your industry.

Second, I work to list where those devices could possible be used (office, Starbucks, employees home, airport, restaurant, etc). This is important to understand what implications those devices have including being lost, stolen, or a staff member having a conversation listened too by an outside party. This list should include the associate risks and possibility of it occurring at each location. The chance of a desktop system being stolen from the office is relatively low compared to a laptop being stolen while at the coffee shop. This does not imply that less security should be utilized to protect data on office systems, but that different techniques should be employed to do so.

The final component of defining “the edge” is defining appropriate policies for each device based on risk to the device and associated data, and a cost benefit tradeoff analysis for which devices should be allowed and which should not because of the level of risk they pose. These policies should take into account technologies like full disk encryption, passwords and non-reusable password generators, Virtual Private Network (VPN) technologies, and physical security like cable locks for laptops. Each potential technology is a tool to lower the risk and increase the reward for offering various tools and capabilities to employees.

Ultimately, this is a discussion around what risks can be outweighed by there benefits in a business setting. Often times staff can gain a significant level of productivity by having access too laptops, PDAs and other mobile devices, the company must weigh that additional productivity against the risk of a company device becoming compromised.

The concept of “the edge” is always going to be present for a companies IT infrastructure. As Web 2.0 and associated architectures grow, the ability to present more and more tools and capability to staff is only going to increase. By properly laying the ground work for how staff securely access these systems, a company can ensure that new tools can increase productivity without negatively impacting the risk to the company.

Risk Management in HPC

2008-08-27T09:40:00.000-07:00

Risk management is a very broad topic within the project management space. It covers planning for and understanding the most unimaginable of possibilities within a project so that a plan is in place to respond to these situations and mitigate risk across the project. I will focus on risk management specifically in High Performance Compute (HPC) deployments. HPC, like any other specialty area has it's own specific risks and possibilities. Within HPC these risks are both procedural and technical in nature, but have equal implications to the overall delivery of a successful solution.

Risk management in any project begins with a risk assessment, this includes both identifying risks and possible risk mitigation techniques. This can be done through a variety of methods including brainstorming, the Delphi technique, or by referencing internal documentation about similar, previous projects. This initial assessment phase is critical to ensure that both risks and responses are captured. By capturing both of these upfront, it allows for better communication around the known risks, and better preparation for managing unknown risks. This risk assessment will produce a risk matrix, this is the documented list of possible risks to a project and there mitigation, or response plans. The risk matrix will become part of the overall project delivery plan.

Risk Matrix
When beginning any HPC project, either an initial deployment or an upgrade, it is important to develop a risk matrix. This can include both known risks (late delivery, poor performance, failed hardware) as well as unknown risks. The unknown risks category is much more difficult to define for just that reason, but a common approach is to define levels of severity and responses. These responses can include procedural details, escalation details, communication information and documentation about the problem to prevent a reoccurrence.

This matrix should include a variety of information including:

Risk name – Should be unique within the company to facilitate communication between groups and departments
Risk Type – Minimal, Moderate, Severe, Extreme, etc
Cost if this risk occurs – This can be in time, money or loss of reputation, or all of the above.
Process to recovery – It is important to document early on how to respond to the risk and correct any problems that have developed because of the risk
Risk Owner – Often times a specific individual has additional experience with dealing with a specific risk and can work as a Subject Matter Expert (SME) for the project team
Outcome documentation – Clearing defining what should be documented should the risk occur so that it can be responded too
Communication Channels - different risks require that different staff and management become engaged, it is important to document who should be involved should a risk occur
Time Component – Every risk has a response, every response has a time component associated with it. It is important to understand these time components up front, it will allow project management staff to adjust schedules accordingly should a risk occur

Known Risks
Often times, known risks are the easiest for people to plan for, but very difficult to handle. This understanding up front and anticipation of the risk or problem can often fool us into believing we know the best response to the problem, when often the only way to truly understand how to respond to a problem is to do it incorrectly one or more times.

Lets explore some common risks that are specific to HPC deployments, and the most common mitigation strategies to combat them:

Application Scaling
A fundamental premise of HPC is that applications should scale in a way that makes more hardware produce more accurate results and/or more efficient production of data. Because of this an application is often expected to perform with the same scalability on 64 nodes, as it does on 128 and often many more. This type of scalability must be architected into the application as it is written and improved on as hardware performance evolves over time. Every time a newer, faster or bigger cluster is installed, there is an inherent risk that the applications previously used will not properly scale on the new platform.

Often times the best mitigation strategy for this risk is proper planning, testing and benchmarking; before system deployment. The most difficult time to manage an application scaling problem is after a customer's hardware has been delivered and installed. By benchmarking and testing the application prior to shipment, the expectations with the customer can be properly set. It also allows proper time for working with any development teams to troubleshoot scaling problems and correct them before presenting results and completing acceptance testing with the customer.

Facility Limitations
HPC solutions often use large amounts of power, cooling and space within a data center compared to a companies business support systems or database centric systems. Because of the large facility needs of HPC it is very common for customers to underestimate the facility needs, or the numbers to be poorly communicated from a vendor to a customer. The power and cooling requirements can also vary widely based upon the customers final use and intended application of the cluster.

All facility design issues should be managed and planed for before hardware is shipped or systems are assembled. To ensure a smooth cluster delivery, it is critical that site planning and assessment be done as part of the system design. This site planning should ensure there is enough power, cooling and space to accommodate the cluster. It should additionally work to ensure the power and cooling are in the proper places and can be directed to the cluster in the recommended fashion.

Mean Time between Failure (MTBF)
MTBF is a calculation used to understand how often components across a single node or cluster will fail. It averages the known and designed life cycle of all individual components to provide a time between each individual component failure. These component failures can either be severe enough to impact the whole system, or just portions of a cluster based on the cluster's design. Often times a completed cluster will fail in unexpected ways because of the MTBF characteristics of putting large numbers of compute nodes in a single fabric. If proper redundancy is not built into critical systems of the cluster, a customer satisfaction issue can develop because of prolonged and unplanned for outages.

By properly assessing all uptime requirements from the customer, a system can be designed that will provide the uptime necessary to conduct business regardless of the MTBF that is collective across all components. Each individual service and capability of the cluster should be assessed to ensure that the proper level of redundancy including clustered nodes, redundant power, and redundant disks is included with the complete solution.

Performance Targets, I/O and Compute
Performance guarantees are often included in HPC proposals to customers to provide a level of comfort when planning times for job completion and capacity planning for an organization. These numbers can often be sources of concern as a system is brought online if compute capacity is not as promised or I/O is not operating as fast as expected or promised.

There is often misunderstandings with complete cluster deployments about a clusters capability for sustained versus peak performance. Sustained is most often the number used for a representative test of how the system will perform over its life cycle. Where as peak is the level of performance often stated for bragging rights because it is the theoretical maximum potential of a given cluster.

There is very little that can be done after delivery of a system if this type of risk comes up, other then giving the customer the additional hardware to pull the sustained performance number up to the peak performance number. This can be a very expensive response. This is the reason that the staff doing HPC architecture must fully understand application benchmarking and performance when designing new clusters. All numbers should also be reviewed by multiple people, this will insure errors in math or testing methodologies do not go unnoticed.

Unknown Risks
Often times planning for unknown risks can be the most stressful, but can yield the most gains when actually responding. This is because of a lack of prior perceptions and the ability to be very creative with responses and future mitigation strategies. Risk planning for unknown risks is often an exercise in understanding the levels of severities that could occur with a problem, and associating it with the appropriate level of response and future prevention.

When defining response strategies for unknown risks, often the first step is to define levels of severity that could develop from any given problem. A common list is:

Most severe level of risk, requires executive management level response to the customer and has a high percentage cost to the project (greater then 50% of project revenue is at risk).
Severe level of risk, requires executive level of response and carries a medium level of financial risk (less then 50% of project revenue is at risk).
Medium level project risk, requires senior management response, could or could not have a financial impact on the project, but does have a deliverable and schedule component.
Lower level risk, has an impact on project schedule, but no negative impact on project financials.
The lowest level of project risk, often just a communication issue with a customer or potential misunderstanding. Often no schedule impact or financial impact to the project.

The next step, after defining a list of possible problem levels, is to define mitigation strategies for each. Each mitigation strategy should include the following:

Steps to research and understand the problem.
Communication channels, who needs to be communicated with for a problem of this magnitude and how are they communicated with. This needs to include both customer and company contacts that will be necessary to correct the problem.
Flow chart for responding, this is the path to determining the appropriate response and deciding if more resources, either financial or staffing, are needed to correct the risk.
Documentation to prevent future occurrences is important. It is important to ensure that any information about the project is gathered and documented to be used in house to prevent future occurrences of the same risk.
Risk closure document. A checklist to document that all protocol was followed and the risk was corrected. This should include components that the risk will not return on the same project because mitigation techniques have been implemented.

The mitigation strategies for the various levels of unknown risks can all be the same, or each level can have its own mitigation strategy. Often a different strategy is used for each of the levels because different executives or financial analysts will need to be involved in the response because of the problems that can be different from level to level. The mitigation strategies are the companies last line of defense within a project to ensure that all problems, no matter the level can be resolved to ensure a smooth project delivery,

Path Forward
The most important component of risk management is skill and experience development. It is important to ensure that as a company, you have processes to document all experience that is gained as part of risk management within managing your projects. This knowledge must be documented so that other teams, new teams and new staff can learn from previous experience of the company.

The more efficient a job that is done with documenting risk response and lessons learned, the more efficiently companies can scope future projects. This allows companies to much more accurately assess costs for future projects, as well as risk versus reward tradeoffs for large, complex projects. Ultimately the best way to manage risk is to understand it before beginning actual deployment and implementation on a project. This comes from a combination of utilizing all data collected on previous projects as well as techniques like brain storming and the Delphi technique to ensure as many possible risks are documented with appropriate response plans.

Platform Decisions - OS Choices

2008-08-11T07:09:00.000-07:00

Following up on my previous discussion around platform decisions and solution architecture, I wanted to dive into Operating System (OS) choices. This is a difficult choice for many companies because of competing priorities, experience, training levels, costs and ultimately faith to one OS or another. Choosing an OS for production use is also a difficult choice because the options change so frequently, and the applications you will ultimately use may not work with your preferred choice of OS.

With most companies, this is rarely a discussion around which OS will be used exclusively. More often it is a discussion around which OS will be added to or eliminated from the infrastructure to either lower administration costs, lower maintenance costs or increase capabilities. Often, companies also break down an OS choice into groups, either server and desktop class systems, or by departmental needs. This separation can be very beneficial when discussing any changes; it allows the folks doing the assessment to clearly define needs and balance them based on focused groups, trying to balance needs across a large company can often prove to be difficult to impossible.

Primary Reasons for Change
First, lets explore the primary reasons a company would change the mix of OSs already in use within the IT organizations.

Lower Training Costs: Today Linux is the predominant OS within the Education communities. This creates an environment where new staff entering the work force are very experienced and knowledgeable on working with Linux based systems. This is important because by using an OS that potential staff are experienced with limits the training that is required to get and keep them proficient at system administration. Companies will often eliminate an OS from use because staff skills are not at a peak for it, and costs to keep them trained at appropriate levels continue to rise as an OSs dominance disappears. This was primarily seen with the large UNIX variants (Solaris, HP-UX, AIX), over time companies have limited the use of them because students were no longer coming with these skills from college, and existing staff were spending more and more time keeping up with training on these platforms.

Increase Performance: Performance is often a primary reason to evaluate utilizing a new OS. Most application vendors today support a very narrow sub-set of the available OSs on the market. Because of this, they must focus there resources on tuning and performance enhancements, at times companies can get a 20% to 30% improvement in application performance by moving the application to a better supported and tuned OS.

Lower Maintenance Costs: In todays world where Open Source is becoming more and more dominant in the business world, companies are reviewing there traditional support and licensing modes. There is a multitude of options available today from OSs that have no cost to use, to OSs that charge for all used instances. A company with legacy OSs in place has an opportunity to review how they negotiate support contracts with these new models so that they are paying for an appropriate level of support for all systems.

Increase Capabilities: Being able to provide a new capability that was previously not available is a large reason companies look to adding new OSs to there existing enterprise. Todays applications vendors rarely support all possible operating systems, more often then not, they choose a subset of OSs that they feel will best cover there potential market. Companies are constantly evaluating new applications for potential benefits to the companies bottom line, as part of this, often times a new OS must be brought in for the administration team to manage to provide new capabilities by adding new applications to the enterprise.

Assessment Questions
Second, lets look at some questions that can be asked when assessing possible OSs for use in your environment:

Why am I assessing my current install base of OSs? What is the goal of any changes?
What is the current cost, both in licensing and training, for all current OSs we have deployed?
What OSs are our staff skilled at administering? Both current utilized and non-utilized OSs.
For any new OSs we are assessing, what will be the training cost to get staff proficient at maintaining them? The yearly cost to keep our staffs' skills up to date?
What level of OS support can be provided by in-house resources and what will need to be included with any purchased support agreements?
What tools currently in place will need changes or license upgrades to support a new OS?
Will this OS introduce security vulnerabilities that will be unreasonably difficult to manage?
Is this system mission critical? Can the system utilize an OS with just community support and no formal SLAs?

Lets also look at some questions that can be used to asses if an existing OS should stay part of the enterprise or should be phased out:

What is the yearly cost in support for this OS by itself? In relation to other OSs within the company?
What percentage of staff in house are proficient on this versus other OSs?
What is the support cycle for this OS? How much longer will the vendor provide patches without additional support contract costs being incurred?
Does the vendor, both OS and application, provide a supported upgrade path to a newer version?

These questions can guide an informed decision about future OS choices. It is critical to understand the financial trade offs that each OS will introduce, including training costs, licensing and support costs, hardware costs and performance capabilities. Every OS has it's own specific details including cost of acquisition, cost of administration, cost of support, etc. These questions should be weighed against currently in house OSs, as well as new OSs that are potentially being considered for addition to your environment.

In addition to the financial questions for each OS, a company must consider the life cycle of the OS. Most OSs have formal release schedules for patches, upgrades and subsequent versions. It is important to evaluate any new OSs with these details in mind. It can end up being quite costly if an OS hits its end of life and you have to rapidly stop using it and migrate the work load to another platform, where as be carefully evaluating the roadmaps for the OS, you can make an informed decision that will successfully work with your in-house processes for support and upgrades.

Making a choice to add or eliminate an OS within a company can be a difficult one, both because of personal territorial issues, as well as complicated technical needs. It is important to focus on the true costs to the company related to the decision, this will ensure that training, implementation, licensing and support are factored in and staff fully understand the costs and ultimate decision.

Tools for Effective Cluster Management

2008-08-05T07:06:00.000-07:00

To continue my previous post on cluster management, I wanted to focus on the tools that are available for implementing and monitoring cluster health including process, hardware and configuration management.

There are two primary ways that one can go about building a change management and cluster management system. The first is going with a complete Linux stack solution that is integrated with a scheduler, monitoring utilities and OS deployment Tools. The second is to build a suite of tools using commercially or open source available tools in the field. Both have there benefits and tradeoffs, ultimately most firms use a combination of the two.

Types of Tools
There are several types of tools that are necessary to manage any cluster, large or small. The tools are categorized by the need they fill in the overall management of a cluster, including request tracking, change management, availability monitoring, performance monitoring and operating system deployment.

It is important when evaluating an HPC software stack, either complete or built from individual pieces, to ensure that each of these components is included, and evaluated for the capability they will provide versus similar, competing products.

Complete Stacks
Complete HPC stacks are becoming more common because of there ease of integration, and integrated support models. Complete stacks usually consist of all the base software that is needed to deploy and manage a cluster, as well as the libraries needed for parallel job execution. These stacks significantly cut the time needed to deploy new clusters, as well as ensure that all initial software on the system is compatible and fully tested.

The difficulty with stacks is there set versions of libraries and smaller compatibility matrices. These stacks are very tightly integrated solutions that ensure they are compatible and stable. They can present a challenge for sites that have outside requirements for different versions of libraries and compilers then the complete stack provides. While this is a challenge for some complex installations, this standard set of tested and integrated libraries provides a much easier solution for companies just using mainstream ISV applications. The developers of the primary stacks on the market work to ensure there kernel and library versions are within the framework that the primary ISVs support and expect.

Individual Tools
Even in environments where a complete HPC stack solution has been deployed, there could be the need for additional tools to meet all operational requirements. The individual tools mentioned below can be used to fill some of these needs, as well as be used as a starting point for companies that decide to not use an integrated stack solution, but instead roll there own.

The primary benefit to rolling your own stack based on these and other tools is that it will much more clearly meet your companies needs. The integrated stacks are meant as a solution to meet very broad HPC needs within a given customer base, but by developing a custom stack, a company can ensure all there specific needs are met and integrate in with existing company platforms. This integration can include management APIs that are similar to existing platforms, as well as data integration to ensure reporting, authentication and logging meets company standards.

Specific Tools

Sun HPC Software, Linux Edition (http://www.sun.com/software/products/hpcsoftware/index.xml) – The Sun Linux HPC Stack is an integrated solution of open source software for deploying and managing the compute resources within an HPC environment. It includes a variety of tools for performance and availability monitoring, OS deployment and management, troubleshooting and necessary libraries to support the primary interconnects on the market.

Rocks (http://www.rocksclusters.org/wordpress/) - Rocks is an open source, community driven integrated solution for deploying and managing clusters. It is based on a concept of rolls, each roll is specific to an application or set of tools that could be needed in an HPC environment. This modularity allows users to add the components they need as there needs evolve.

Trac (http://trac.edgewall.org/wiki/TracDownload) – Trac is a toolkit originally designed to be used in software development organizations. It has integrated capabilities for tracking bugs, release cycles, source code and a wiki for documenting notes and process information. These may all seem like software development specific capabilities, but they can all be used in very effective ways to better manage and document the associated processes for a cluster.

Request Tracker (http://bestpractical.com/rt/) - Request Tracker is an integrated tool for tracking, responding too and reporting on support requests. It is heavily used in call center environments, and works very well for HPC environments to track customer requests for support, requests for upgrades and other system changes.

RASilience (http://sourceforge.net/projects/rasilience/) - RASilience is built around Request Tracker with the Asset Tracker and Event Tracker add-ons. It is an interface and general-purpose engine for gathering, filtering, and dispatching system events. It can be used to provide event correlation across all nodes and other components within a cluster.

Nagios (http://www.nagios.org/) – Nagios is an open source monitoring solution built on the idea of plugins, plugins can be developed to monitor a wide variety of platforms and applications, while reporting back to a central interface for notification management, escalation and reporting capabilities.

Ganglia (http://ganglia.info/) - Ganglia is a highly scalable, distributed monitoring tool for Clusters. It is capable of providing historical information on node utilization rates and performance information via XML feeds from individual nodes, that can subsequently be aggregated for centralized viewing and reporting.

OneSIS (http://www.onesis.org/) - OneSIS is a tool to managing system images, both diskless and diskfull. OneSIS is an effective tool to ensuring that all images within a cluster are stored from a central repository, and integrated in with the appropriate tools to utilize kickstart for installing new operating system images, as well as booting nodes in a diskless environment.

Sun Grid Engine (http://gridengine.sunsource.net/) - SGE is a distributed resource manager which has proven scalability to 38,000 cores within a Grid environment. SGE is rapidly being updated by Sun to more efficiently handle multi-threading and too improve launch times for jobs, as well as tty output for non-interactive jobs.

Cluster Administration Package (http://www.capforge.org/cgi-bin/trac.cgi) – CAP is a set of tools for integrating clusters. It is designed and tested to accomplish three main objectives; Information Management, Control and Installation. CAP is a proven tool for deploying and managing a centralized set of configuration files within a cluster, and ensuring that any changes to master configuration files are correctly propagated to all nodes within the cluster.

Cbench (http://cbench.sourceforge.net/) – Cbench is a set of tools for benchmarking and characterizing performance on clusters. Cbench can be used for both initial bring up of new systems, as well as testing of hardware that has been upgraded, modified or repaired.

ConMan (http://home.gna.org/conman/) - ConMan is a console management utility. It is most often used as an aggregator for a large number of serial console outputs within clusters. It can be used to both take console output and redirect it to a file for later reference, as well as allow administrators to redirect output to a console in ReadWrite mode.

Netdump (http://www.redhat.com/support/wpapers/redhat/netdump/) - Netdump is a crash dump logging utility from Redhat. The purpose of Netdump is to ensure that if a node with no console attached crashes, administrators have a reference point within logs to catch the crash and debug output.

Logsurfer (http://www.crypt.gen.nz/logsurfer/) - Logsurfer is a regular expression driven utility for matching incoming log entries and taking action based up matches. Logsurfer can do a variety of actions based upon a match including running an external script, or counting the number of entries until a threshold is met.

Specific Tool Integration Techniques
These are some specific methods myself and some colleges have used to integrate these tools into larger frameworks used for change management and monitoring within Enterprise Environments. These are meant as a way to show how the different tools, used in combination, can simplify cluster management and lower administration costs. All of these methods have also been tested at scales well beyond typical HPC systems today, including OneSIS and Cbench which have been tested up to scales of 4500 nodes.

OneSIS
OneSIS can be used in two primary methods within a cluster, each can be used independently or in combination. The first and most common is to assemble an image that is then deployed to all compute nodes and installed locally. OneSIS can also be used to distribute that image to all compute nodes so they can run in a diskless fashion, using the image from a central management server.

These methods can also be used in combination when preparing to upgrade a cluster. A new image can be developed and booted into a diskless mode on a subset of a clusters nodes. Those nodes can then be used to test all applications and cluster uses to ensure the image is correct. Once that testing is complete, OneSIS can be used to ensure an exact copy of the tested image in installed on all compute nodes. This method ensure that no bad images are installed on the cluster, and that the majority of the cluster nodes can be left in place for production users while the new image is tested.

Nagios
Nagios is a very dynamic tool because of its ability to use plugins for monitoring and response. Plugins can be written for any variety of hardware within a cluster to ensure they are online, are not showing excessive physical errors and do not need proactive attention. Nagios's dynamic nature also allows plugins that allow it to communicate with centralized databases of node information and report are hardware or node problems to RT for proper tracking and attention

Nagios plugins can easily be used to remotely execute health check scripts on compute nodes. These health check scripts can check to ensure nodes are operating and responding correctly, there are no hung processes that might affect future jobs, and that the nodes configuration files and libraries are the expected versions. If Nagios does detect an error on a given node, it can easily be configured to automatically open an RT ticket for staff to repair the node, and mark the node offline in the job scheduler until such time as the node is repaired.

Cbench
Cbench is a wonderful tool for automating the process of both bringing up new clusters as well as testing hardware that has been repaired or replaced to ensure it meets the same benchmarks as other hardware in the cluster. Cbench has a collection of benchmarks that can be used to benchmark a new cluster to ensure that the system, storage, memory and attached file systems perform as designed. This can be a valuable tool in locating issues that were introduced during deployment and will ultimately cause performances decreases for users.

Cbench can also be used to ensure that all hardware that was repaired was done so correctly before being reintroduced into the cluster. By properly benchmarking a cluster at installation time, it allows support staff to run identical benchmarks on nodes that have been subsequently repaired. These new results can be compared to the initial results from the cluster and ensure that the node is now operating as peak, expected performance.

Logsurfer
Logsurfer is best used as an aggregator and automated response mechanism within a cluster. By having all nodes send their respective logs to a central log host, it enables the cluster administrators to configure a single Logsurfer daemon to monitor and respond to appropriate log entries.

Many sites will subsequently configure Logsurfer to proactively mark nodes in the scheduler offline if an error is found in the logs relating to that node. This ensures that no future jobs are run on the node until repair staff are able to verify the node is operating correctly and repair the reason for the initial error.

Final Thoughts
Clusters are complex mixes of hardware and software, the more effectively the tools are picked and integrated early in system design, the more efficiently the system can be managed. There are many tools available, both commercial and open source, that can be used in cluster environments. It is critical that each ones benefits, tradeoffs and scalability be weighed when picking the tools for for environment.

As a final thought, clusters are complex solutions that often require customization at every level. This can also be extended to the applications used to manage the cluster, but was not mentioned previously in this document. It is always an option to develop a tool in house for your needs, chances are, if you have a need, so does someone else. The majority of the tools above were developed because a single company had a need, developed a tool to meet that need and put the tool back into the community for everyone else to use. This is a wonderful way to not only continue improving the capabilities we as a community have around clusters, but is a great way to get company recognition in a rapidly growing field.

Enterprise Architecture versus Solution Architecture

2008-08-01T07:46:00.000-07:00

Recently I have been in several customer meetings where their newly hired Enterprise Architect(EA) joined to listen in and provide feedback. Most of these meetings were to discuss an individual cluster or system that is being implemented, and it seemed that most EAs these days are still too focused on systems, solutions and details; and too little focus is being paid to the true activities I see as relevant for an Enterprise Architect. I decided to throw my own comments out there about where an EA falls within an organization, and how that differs from what I call Solution Architects.

The way I see it, a Solution Architect is more closely associated with what I see as technical sales people, they focus on the individual system or application, focusing on the details of what software packages will work, what a good support model is, and how to implement it within the companies framework that is defined by the Enterprise Architect.

I then see the Enterprise Architect as a pathway between the companies Business Goals and the IT personnel that must delivery tools to meet and track those goals. The EAs goal is to define a set of policies at the company wide level that ensure things like legal compliance, consistent identity management and company wide reporting capabilities.

If an EA gets too involved in the Solution Architect level details, the company suffers because those higher level activities are not being managed appropriately. A successful EA has both the willingness and capability to work with the company executives and turn there business vision into a technology vision and push that down to the Solution Architects and IT staff.

Defining Effective Cluster Management Processes

2008-07-28T11:24:00.000-07:00

Abstract

Todays high performance compute clusters are more complex then ever; they are an intricate set of hardware, middleware and processes that ensure a robust, reliable platform form companies to conduct up to the most critical business processing. When designing the processes to manage these systems companies must ensure they factor in todays needs, as well as tomorrows possibilities. This paper works towards addressing the complex issue of defining the process that will ultimately be used to manage these clusters as the are implemented and grow over time.

Definitions

I want to begin by defining some terms that I use through out the document, this will ensure the same understanding of common terms as I use them:

system – A host that contains a single operating system system image for multiple processor sockets, with all memory addressed from the single operating system image.

cluster – More then one system interconnected and managed through a common fabric.

enterprise – A class of system that supports operations of a business, this could include systems running Oracle DB, Application servers, SAP, etc.

scheduler – A resource manager to use within a cluster to ensure maximum effective use of all resources.

jobs – Submissions by individual users to the scheduler to accomplish a task on the cluster.

resources – Capabilities of the cluster to include processors, memory, storage and interconnects.

interconnect – The fabric in which a cluster uses for communication. Can be Gigabit Ethernet, Infiniband, Quadrics or others.

Intro

Todays clusters are much larger then they have been at any time in the past. As grid and cloud computing continue to increase in popularity, the number of systems that must be managed in a single environment will only continue to grow. As the number of systems continues to grow, administrators are going to continue to struggle with keeping consistent software across all systems, recovering failed systems and managing the location of applications and data and associated versions that are installed.

As the number of systems grows, companies and administrators will need more refined tools and processes to ensure systems are configured as expected, properly report problems that can be tracked in a meaningful way for upgrades, rotations and maintenance. Tools must respond properly with where problems are located and how to correct them, this will ensure companies are not relying on senior staff for on call duties and general troubleshooting.

When designing and managing these complex clusters, process must be the number one item considered. The more clearly the process for managing the cluster; including upgrades, changes, failures and testing, the more reliable the system will be over time, and the fewer unexpected problems that will result from failed processes, a lack of process or unexpected consequences of changes.

Second only to process is metrics. It must be clearly defined how these complex clusters will be monitored and measured for success. These metrics can encompass many things including uptime, job completion time, jobs completed, staff metrics and scalability metrics. The process of defining the metrics to gauge success must begin with an evaluation of the business goals that are to met by utilizing an cluster for company workloads. These metrics must accurately gauge what factors show success by migrating existing applications to an cluster, as well as implementing new tools now that the capabilities are in place.

Another key of these processes and tools is that they must be designed to scale as the customers' cluster grows. Any designed solution must factor in not only todays systems to manage, but also the expected growth in the coming years. This will ensure that all processes and tools are scalable and do not need to be replaced to upgraded as the cluster grows.

Overall, this problem is two-fold. The proper tools must be in place to support clearly defined and tested processes. There is a plethora of tools available today that provide change management, process tracking and cluster monitoring. It is important that companies understand the benefits and tradeoffs of each available tool when deciding on how to implement these processes for there environment. Some companies will find the current available tools are more then sufficient to meet there business needs, while others will find that developing new tools in house will better suite there needs.

The realm of high performance computing is no longer the island it once was. Today many staff, processes and tools can and are being shared between departments. As time continues, high performance computing and clusters will just be another set of systems that must be maintained within a company, and no longer a separate department as they are now.

Defining Business Goals
Now we will explore defining our change management processes and associated support processes for a large clustered environment This begins with defining the business goals, some questions to ask when defining these goals are:

What is the maximum allowable downtime that can be afforded this cluster?
How much time per month will the support staff need to handle routine maintenance?
What recurring events might impact performance on the cluster? This could include end of quarter financial processing, data warehousing activities and compliance reporting activities that are given priority over standard users.
How will users be grouped on the system in relation to job function, priorities and load types?

Our business goals are a key component of the information that will later be used when documenting all processes in details. These will serve as the pseudo-goals that must be hit, and will serve to define the metrics we will define in the next section. These business goals should be aligned closely with the mission and vision of the company, as well as the specific teams that will be utilizing this cluster.

This step should be mostly a business discussion, while avoiding technical architecture discussions, I believe a better set of business aligned goals can be achieved, without having to discuss tradeoffs yet for technology and cost. The cost and technology tradeoffs can be discussed and factored in after metrics for success and goals have been defined. This will ensure that tradeoffs are fully understand in there own context, and not part of this business goals discussion.

When defining these business goals, an honest assessment of both minimal and optimal goals must be done. By defining both optimal goals and minimal, we will be able to have a proper discussion later in the process about tradeoffs, while really understanding what is acceptable levels of compromise, and what is not. The optimal goals will be what management would like to see accomplished given that time and money were of no object. The minimal goals should reflect the minimum outcome from the project over it's life cycle such that the company receives a financial benefit from the system, but not necessarily all features and possible results.

Scoping and Metrics
Second, we must ask a variety of questions to define the scope of system management and the metrics used to gauge success for each component. This can include a variety of components including:

File systems
Hardware
Interconnect
Facilities – This can include the data center that houses the cluster, the offices that the users reside in and any facilities where data is stored and managed in relation to this cluster
User training and support

Some examples of metrics that can be gathered and tracked are:

Average job completion time
Maximum job completion time
Users accessing the system over time
Support requests logged by users
Number of supported applications on the cluster
User data volume and churn per month
Measured MTBF versus vendor expected MTBF
Any company specific information that will later be used to gauge success

Metrics are key to ensuring that the business goals are objectively measured and monitored over time. These metrics must be defined before implementing an new cluster management processes so that the proper usage and user information is tracked and kept for future analysis. Metrics can be defined in a variety of ways including along company management lines, along functional organization lines, or across lines of business or customers. By accurately assessing all the possible structures, and providing metrics for each, the company will have a usable set of metrics that can also evolve as the companies structure does.

Metrics are also a constantly evolving item that should be kept up to date to match evolving company goals and structures. As the companies management structure changes, or goals and missions change, the metrics should be updated to reflect these changes to ensure that costs are accurately tracked, and benefits can be accurately tracked.

Scoping is the process of defining the boundaries for these metrics and associated processes. Scoping is important to ensure that we do not try and touch too much at the same time, while still ensuring that all relevant information and teams are included in discussions. Scoping is part of the metrics section because the metrics are directly related to scoping and vice versa. To properly understand how we are going to asses progress, we must fully understand what we are assessing. When defining scope it is important to have readily available charts of staff alignment, project alignment and budgeting information so that lines can be drawn at what will be included and what will be handled as a separate project.

Defining Process
Third, we will define these processes around the answers to the above questions. Some items to consider when taking these into account are:

How often does the site anticipate new software to be installed?
How often does the site anticipate upgrading the host operating systems of the cluster?
Understanding interactions between libraries and applications?
How will we document and track installed applications, libraries and versions?
What shared file systems will be utilized on this cluster?
What dependencies are in place for cluster monitoring?

After defining the business goals and the metrics for tracking those goals we can begin to define the process that will be followed to meet those objectives. The process must be flexible enough to evolve with the cluster and applications, but also must be rigid enough to ensure that all business goals are met and metrics are correctly tracked and reported.

Defining process involves two major components, managing the cluster when things are optimal, and proper response when things fail to work as designed and expected. To properly address each of these categories it is critical to define the process as three separate components.

The three components of the final process will be:

Change management – How to effectively assess changes and plan for maintenance to have a minimum and understandable impact and risk.
Failure Response – having a detailed process to handle all types of failures, including technical response, escalation process, documentation process and user notification process. This process is only to handle known types of failures, and will be updated by the next process in the event an unknown or new type of failure occurs.
Unknown situation – Finally, it is important to have a general process for handling unknown situations. This should include how to contact and include the appropriate staff for resolution, how to escalation problems that span multiple teams and how to document the correct process to respond should the problem occur again.

Tradeoffs
Ultimately, most organizations are also going to have to discuss tradeoffs. It would be optimal to track all possible service related details, report on them as well as provide 24x7 levels of support, but this is often cost prohibitive. After a firm has completed defining their business goals, define there metrics for success and defining the policies to meet those goals, a tradeoffs discussion is next. This is to evaluate the cost of meeting not only the best case scenarios for business goals, but also the minimum acceptable for success.

Tradeoffs are a difficult component because each team that is involved in discussions will have goals they can not change, or processes that must be kept in tact. By ensuring that all teams affected by these decisions are also at the table for the tradeoffs discussion, a company can ensure that all relevant voices are heard and tradeoffs are fully understood when decisions are made. The tradeoff discussion does not necessarily need to be a decision of what must be given up, it can also be a decision of what can be put off until a later date, or done by a different team for better efficiency.

Tradeoffs must also balance both costs and benefits, an increased cost must be fully justified by increased benefits. Just as a decrease in benefits must be balanced by an appropriate cut in costs. By assessing each business requirement against the cost to implement versus the long term benefits, a company will be able to accurately assess if the benefit is worth the cost.

Todays clusters are complex solutions containing many staff, hardware components, software components and requirements. These must all be assembled in a way that they accomplish the goals for a given project, but are fluid enough to rapidly adjust to encountered problems, as well as a changing business landscape. By taking a systematic approach to defining the management processes for a given cluster, a company can ensure that all business objectives are being met and staff are keenly aware of the goals of the cluster.

Staffing Complications

2008-07-25T15:04:00.000-07:00

One on-going problem within IT that I see is the inability for management and HR to work effectively as a team and develop a competent strategy for hiring, maintaining appropriate staffing levels, training and retaining employees. This comes from a belief that folks in IT do not need training, that a company can hire staff that are ready at the starting gate. This is insane considering how this is not expected in any other field. Take an accounting firm; they factor in training costs for all CPAs they hire, regardless of skill and experience level, this is because they understand the costs of having and retaining educated, competent employees.

This posting by James McGovern hits the nail on the head.