Hadoop Whitepapers

Below are links to two whitepapers I recently wrote and published as part of my role at Dell.

• Introduction to Hadoop
• Hadoop in the Enterprise

Enjoy!

Cloud Computing, what is it really?

I have been in a lot of discussions recently where Cloud Computing is being discussed more then before. I wanted to take a break from EITRP and talk about this term and it's increasing use. Most of the definitions of Cloud Computing include the following words in some order and combination: elastic computing, remotely delivered, associated with SLAs, end user platform independent delivery method, scalable, and billed by usage.

Essentially, most people look at Cloud Computing as a change in the delivery of an application from a company run data center to a managed, remote facility that provides SLAs for the application and bills for actual usage of the application and the amount of data being stored. Most definitions of Cloud Computing also speak to underlying technologies like virtualization, elastic scalability, or hyper-scale data centers.

I believe that Cloud Computing is less about the technology and delivery method, and more about how people think about their information technology (IT) needs and operate what would traditionally be their IT departments. Cloud Computing is more about efficiency, essentially reviewing all current application and data needs, and instead of modifying existing processes to fit changing needs, it is about creating new methods for application delivery. I look at Cloud Computing as the point in time that IT leadership stops using old ways of deploying servers, installing applications and defining the method folks will use those applications. Cloud Computing is about IT leadership embracing and creating new ways to be more efficient with service delivery and how that delivery spans business, process, technology and finance.

Physical Infrastructure

A lot of interpretations of Cloud Computing involve eliminating a company of it's server infrastructure and utilizing shared billable resources from companies like Rackspace, ServerPronto, Amazon, or 1&1 just to name a few. This is commonly called Infrastructure as a Service (IaaS).

Cloud does not mean you ditch all your servers and use Amazon for your core business operations, it means you throw out all your processes when designing new solutions and design solutions around modern delivery and operational processes and newer proven technologies. This is a difficult change for many organizations, IT has developed many habits over the years and other teams have come to expect things from IT done in a certain way.

Staff Infrastructure

There is a common perception within IT departments that staff can be cut back or eliminated by the use of Cloud Computing. This perception largely builds out of the expectation that Cloud Computing is the 100% use of hosted infrastructure, and enables a company to function without the traditional roles of System Administrator, Storage Engineer or Operations Staff.

This belief is often misplaced. By utilizing IT environments that are more automated and/or hosted at alternate locations, the need for IT staff does not diminish, the skill sets that are needed and the required expertise changes. Many IT organizations that are utilizing new methods for operations that are born out of Cloud Computing are finding that traditional IT operations roles are become more architectural in nature and the skill sets are more closely aligned with Software Engineers then system administration staff.

Traditional IT roles involved in deployment of servers will be more focused on capacity planning, knowledge transfer with the application development teams and strategy development for the use of new and emerging tools and technologies. Modern IT departments that have embraced these new engagement models, look to the other organizations to drive tools based on their business needs, while the IT organization provides thought leadership around implementation and strategy. IT staff can not longer be just technical experts, they must understand the business, the financials of the company, the organizational goals and how that relates to technology.

Cloud Computing is not about the use of some new modern technology, although that is a by-product. Cloud Computing is not about throwing away your servers and buying cycles from other companies, but that is often done as part of it. Cloud Computing is a new method of thinking about IT. It is about starting with the business requirements and needs of an organization and enabling those need through the most automated, efficient way possible. This will be a combination of new skills for staff, new use of technology and different models for procuring, using and retiring resources.

Most importantly, Cloud Computing is not a product to be bought. It is a set of items, methods and best practices for deploying the multitude of products, tools and solutions that IT has at it's disposal and utilizing new ones for efficiency when they become available. Cloud Computing is a mindset for efficiency that enables dynamic businesses.

Now I want to touch on Enterprise Information Technology Resource Planning (EITRP) before I close out this post and show how resource planning across an organization enables a more dynamic IT environment that is sensitive to the business's needs. EITRP enables organizations to map the organizational goals, processes, rules and resources that a company has at it's core. This mapping enables clear modeling and simulation of changes to the environment. Cloud Computing provides the operational model for implementing the resources documented within EITRP.

Explanation and definition of EITRP Associations

This is my fourth posting in the series to introduce EITRP. This posting focuses on the concept of 'Associations' within EITRP.

Enterprise Information Technology Resource Planning (EITRP) is a new method for documenting and modeling the resources that contribute to the creation, operation and retirement of assets, policies and systems in modern Information Technology (IT) environments. EITRP enables organizations to clearly document the resources that contribute to and guide how an IT department implements and manages the necessary tools for a company to conduct business. IT departments can be represented and described in a similar way, EITRP exploits these similarities to provide a consistent method for documenting the life-cycle of an IT environment.

The core of EITRP representations of an organizations is the idea of 'Tiers.' 'Tiers' are representations of the common building blocks of all organizations, infrastructure and resources supporting that infrastructure. 'Tiers' ensure that any EITRP model can be referenced, simulated and documented in a consistent manner. Each 'Tier' is unique in the data is contains and provides for standards of documentation.

A 'View' is a subset of an EITRP model formatted and limited to the components necessary for a given role within the organization. A 'View' is a representation of what staff levels within an organization will own, manipulate and utilize the data contained in one or more 'Tiers.'

Building on the concepts of 'Tier's and 'Views' we add 'Associations' to our glossary for EITRP. An 'Association' is the documented relationship of a piece of information(an object) contained in one 'Tier' to it's corresponding information in another 'Tier'. 'Associations' define how a change to one 'Tiers' parameters will affect another 'Tiers' parameters.

'Associations' allow a 'Model' to be built for an organization that shows not only the present state of the organization, but the affect of changes within that 'Model.' 'Associations' allow EITRP to be used to document current information, future state information and what changes will occur as a migration or change between states occurs. 'Associations' are an important part of EITRP for documenting information including dependencies, costs, implications and resource needs.

There are three types of 'Associations' that can occur between objects in individual 'Tiers'. When referencing an 'Association' we always start with the lower 'Tier' in the relationship, moving to the higher 'Tier' in the relationship.

• one-to-one (1t1) – A one-to-one 'Association' is defined as a link between two objects, each in a separate 'Tier'. Each object if manipulated, will only affect the other single object within the 'Association.'

• one-to-many (1tM) - A one-to-many 'Association' is defined as a link between greater then two objects, one of which is in a lower 'Tier' and the remaining are contained in a higher 'Tier'. The only object will affect many if manipulated, where as the multiple objects, if any are changed, will affect the single object in the lower 'Tier.'

• many-to-many (MtM) - A many-to-many 'Association' is defined as a link between greater then four objects, two or more of which are in a lower 'Tier' and the remaining are contained in a higher 'Tier.' The lower 'Tier' objects will affect multiple if manipulated, as well as the multiple higher 'Tier' objects, if any are changed, will affect the lower 'Tier' objects.

Above are two simple examples of a set of 'Tiers' within an EITRP environment, and the associated objects in those 'Tiers' and their 'Associations'. These 'Associations' are simple and only show the basic type of 'Association' but could also contain information about what a change to one component will do the other associated component.

The primary purpose behind 'Associations' is to enable EITRP models to be used for automation within an IT environment. As IT environments continue to grow, it will be more and more vital to automate as many processes, changes and evolutions of the environment as possible. EITRP enables the necessary information about the environment, it's resources and it's parameters to be documented in a way to allow for automation of changes over time.

Future postings will cover additional topics including:

• 'Model' – The definition and uses of a 'Model', the overarching term for a complete set of data points across all tiers, including 'Associations' that provide for a representation of the organization and it's resources.

• EITRP Data Standards - I will publish the initial version of the EITRP Data Standards, there are the methods for documenting information in a standard way within the EITRP 'Tiers' and 'Views' to allow that data to be referenced, manipulated and modeled for changes within the environment.

Explanation and definition of EITRP Views

This is my third posting in the series to introduce EITRP. This posting focuses on the concept of 'Views' within EITRP.

Within Enterprise Information Technology Resource Planning (EITRP) 'Views' are a representation of the staff that will interact with each of the 'Tiers' and utilize the information they contain. A 'View' is a subset of an EITRP model formatted and limited to the components necessary for a given role within the organization. A 'View' is a representation of what staff levels within an organization will own, manipulate and utilize the data contained in one or more 'Tiers.'

The example 'Views' below are based on the most common job roles within an Information Technology (IT) department at a medium to large company. The actual defined 'Views' for a companies EITRP models will be specific to that organization and updated to reflect changes in the companies structure, management styles and organizational needs. The 'Views' represented below are meant as a guide for companies that are adopting EITRP and can be modified as needed over time.

The most common views that will be utilized in some form at most organizations are:

• Chief Information Officer (CIO) – The CIO will most commonly be responsible for the process and associated data within an organization to execute on the companies core business requirements. This role will require the CIO to influence and drive components contained in the higher 'Tiers' related to company rules, process and policies.

• Chief Technology Officer (CTO) – The CTO will expand on the CIO role, while influencing those policies and process, the CTO will also own and define the SLAs the IT organization will work towards at the need of the business and the services that will be delivered to the business for implementing the processes and policies above.

• Architect – Architects within the context of IT are responsible for taking the defined SLAs and services from the CTO and turning them into a design that can be implemented across their organization or company wide. The Architect is responsible for managing the risk and reward tradeoff analysis of the possible solutions for a given need within the environment.

• Administrator – The Administrator is the role most commonly associated with the implementation of the environment, based on the design from the Architects. The Administrators primary responsibility is to implement new designs and provide feedback to the Architects for use in future designs.

• Operations – Operations staff are the first team to respond to abnormalities within the environment. Operations staff are responsible for monitoring for compliance with SLAs and taking prescribed action to remedy all non-compliance.

In larger organizations, it is possible that each 'View' will have multiple individuals that fill that role. Smaller companies could then have a situation where a single individual falls into multiple 'Views.' It is important that when a company is adopting and defining their EITRP 'Tiers' and 'Views' they account for the current structure and needs, but include provisions for the evolution of the company. This will allow the EITRP model to evolve over time and properly account for changes, growth or contraction within the organization and how that change affects the resources managed by the IT organization.

One core component of EITRP is to create a 'Model' that encompasses all resources within the IT environment and their associations. This concept of 'Views' is important for ensuring that as new 'Models' are created, the proper 'Associations' are in place for the evolution of the organization, allowing company leadership to better anticipate how changes will impact the organization.

Future postings will cover additional topics including:

• 'Associations' – The relationship of a data point in one tier and how it relates to the data point in another tier.

• 'Model' – The definition and uses of a 'Model', the overarching term for a complete set of data points across all tiers, including 'Associations' that provide for a representation of the organization and it's resources.

• EITRP Data Standards - I will publish the initial version of the EITRP Data Standards, there are the methods for documenting information in a standard way within the EITRP 'Tiers' and 'Views' to allow that data to be referenced, manipulated and modeled for changes within the environment.