Tuesday Big Data Series

Need for Scalability: Data Explosion

Today, the world contains vast amount of digital data that is increasing rapidly. This endless pool of data, if managed and used effectively can lead to wider business opportunities.


Image Source

But, what is the cause of this data explosion?

  • The number of people using the Internet by either providing information or interacting with the already present information.
  • Availability of high-end digital devices like smart phones, laptops, tablets, and anything that begins with the word “Smart” (Smart Watch, Smart TV, Smart Homes etc.).
  • Organizations that use the online space to grow their businesses; like the e-commerce websites, banking websites etc. that provide services on the click of a button.
  • Logging or tracking user activity. For instance, a user is active on a website for a week after which he has been inactive for 3 days. This information showing his inactivity is logged. Consider another scenario where a user is looking for a product on Amazon and adds it to his cart but does not buy it yet. This activity is again logged. Such logs can provide meaningful insights for running a marketing campaign.
  • Social Media websites like Facebook, Twitter, and LinkedIn etc. that allow users to share information in different formats like text, images, videos etc.

Businesses should be prepared to handle this explosion of volume on the web seamlessly. And, with the speed at which this data is increasing, it becomes a challenge already.

Say, you have a marketing campaign running to drive customers to your newly launched website. This website initially gets 100 hits but all of a sudden the traffic increases from 100 hits per day to 10,000 hits a day. If you are not ready to handle the increase in traffic, your website may go down.

As another example, let’s consider an e-commerce website. The traffic during holiday season or during sale increases drastically. The website should be able to handle the increase in load otherwise it could incur heavy losses.

The challenge here is to build a scalable system that is able to accommodate the existing data as well as the data that is expected to come in future without affecting the performance or causing any failures because of the sudden increase in volume/traffic. Also, the scalability solution should be cost-effective. These are the reasons why organizations are moving away from traditional databases to big data solutions.

One of the high scalable frameworks used to meet this goal is Apache Hadoop. A Hadoop cluster is built up of nodes or machines. These machines are inexpensive commodity servers. It uses hundreds of commodity servers operating in parallel to store and distribute large amounts of data. Hadoop provides an easy expansion from a few nodes to a thousand nodes. Also, since it uses commodity hardware, it is cost-effective.

Big Data is going to get bigger. And, the key to make the most of this data is by managing its scalability.

Like it? Tweet it! Tweet: Need for Scalability: Data Explosion http://ctt.ec/NLXr2 @themarketeng

2 thoughts on “Need for Scalability: Data Explosion

  1. i loved the last point about how big data will get bigger and the key is to manage the scalability. How do you suggest large healthcare companies manage the data?


  2. @Jean, good question! For large data sets used by healthcare companies, Hadoop can definitely be the primary choice to manage these data sets. However, in addition to that, SAP-HANA is a very powerful in-memory platform that can be used to analyze these data sets in a much faster way (e.g. by pulling chunks of data in-memory). Check out this informative article about what I just mentioned – http://www.forbes.com/sites/danwoods/2012/05/16/how-hadoop-and-sap-hana-can-accelerate-big-data-startups/#621fc3e06341


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s