Unlocking the value of Big data in the Cloud

Successful businesses today are data driven and focus on fast iteration.  The ability to quickly test new products, features and user experiences; while measuring the impact and adjusting user experiences in an iterative fashion.  Cloud based Big data solutions enable organizations to quickly deploy new technologies, integrate with existing business systems and iterate the solution as business needs change.

While most organizations have a cloud-first policy, many also still stick to traditional architectures for new systems because of experience and comfort by staff with on-premise based solutions. On-premise based solutions provide a level of comfort through experience with previous implementations, but can also insert unnecessary delays into delivery of capabilities to the business.  Struggles with current on-premise technologies can include:

• Delays – The time necessary to deploy on-premise solutions is often measured in weeks and months.  This time is a combination of working with vendors, waiting for equipment to ship and finally installing and configuring new systems.
• Risk – In todays environment of complex IT systems and changing business requirements, all new application deployments have risk associated with project failure, cost over runs or changes to business requirements.  On-premise solutions have a longer design cycle, because the cost of a failure project is much higher in resources, capital costs and recovery time.
• Capital Costs – On-premise solutions have higher capital costs because of the initial hardware and data center space required to begin.  These capital costs are often difficult to absorb in organizations with tight budgets and limited cash flow.
• Scalability – Scaling with on-premise solutions means keeping spare capacity around with the expectation that it will be needed.  Often this means over provisioning environments to ensure proper response time and hedge against delays in purchasing additional capacity.

There is a lot of comment in the technology community that Big Data in the Cloud has limited adoption, the reasons vary, but often include cost, security and compliance concerns, and performance.  While there were periods of time, that technology maturity did create these challenges, the speed of evolution with cloud based solutions has enabled Big data platforms to be efficient and effectively deployed today, speeding time to value for the business and new capability adoption.

With advances in technology, the ability to build Big data platforms in the cloud can speed adoption, lower risk and increase security through consistency in deployment methods.

• Agility – Cloud providers like Amazon and Google have a variety of different tools for building Big data environments.  These tools span NoSQL capabilities, unstructured text processing and relational environments for supporting transaction processing.  Modern Big data environments require multiple tools for creating integrated pipelines for data ingest, analysis and presentation.  These cloud solutions enable users to quickly spin up new capabilities, one piece at a time, test them and either put them in production or turn them off.
• Elasticity – The primary value of any public cloud environment is the ability to almost-immediately scale capacity up and down based on your specific user and workload demands.  This ability ensures prompt response on all workloads and minimizes expenses related to unused capacity.
• Security – A key component to security is repeatability and ensuring that operations staff do not create security threats through misconfigurations.  Cloud environments create simple, easy to reproduce methods for deployment of systems, connectivity and access controls. 
• Data Mashup – Many public cloud providers provide access to local, public data sets for combining with in-house data.  This data is locally accessible, eliminating transit costs, and often low cost to access for testing model creation or other analysis.
• Optimization –Cloud based applications gain the performance advantages of optimization across thousands of users and varying workloads.  Each cloud provider works to ensure that queries on large data sets are optimized and provide rapid response to users, without specific tuning by the users.
•  Risk –Cloud based solutions enable organizations to quickly change priorities and operational requirements.  Because cloud resources have no up front commitments or long term contracts, organizations can adjust or eliminate resources that are unneeded temporarily while business needs adjust and clarify.
• Capital Costs – Cloud based solutions eliminate the large capital costs traditionally associated with data center builds outs and server purchases.  Organizations can begin projects small, with minimal budget impact until project success is proven.

With the continued rise of both capability and agility with cloud-based offerings, Big Data platforms can be successfully deployed, with minimal risk.  Cloud based Big data solutions give organizations the ability to quickly test new capabilities, minimize capital costs and scale the environment as needs change and grow.  Big data solutions enable organizations to quickly analyze complex data, make informed decisions and measure the impact of changes to their business model.  Cloud based solutions ensure that the features and capabilities needed to build these environments can iterate just as quickly.