Big Data can be defined as high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. Big data is basically about a new set of algorithms, topology and use of resources that enable us to gather, analyze and show more data.

Every day, we create thousand of bytes of data, the number grows in an exponential way.

In the past all data was stored in a relational database, so to handle it, it was a question of simple rules to select data from a database. Today 80% of data is unstructured, and this percentage is growing: videos, pictures, comments, GPS, transactions, locations…

Under this situation, individuals and companies face scenarios where the data is too big or it moves too fast or it exceeds current processing capacity. This limit to handle the data comes from:

• Volume: to overcome the excessive size of data requires scalable technologies and distributed approaches to querying or finding a given data.
• Velocity: we need to organize the resources (CPUs, memories, networks…) to enable real-time data that enables us to make decisions at the right time.
• Variety : the unstructured data makes that the known paradigms of data search are not valid anymore. Since this moment you can differentiate between “Hadoop people” and “RDBMS people”, they are a different thing.

Think about Facebook, in the terms mentioned above:

• Volume: the amount of data introduced by all people, companies distributed around the world. Imagine how they should organize all information.
• Velocity: As user I want all as fast as possible, imagine a facebook that takes 2 seconds to refresh a screen: nobody would use it.
• Variety: Facebook is tracking data from different nature, and the number of natures is growing up.

Well, Facebook, Yahoo, Twitter, EBay… they all use Hadoop.

I’m reading about Content Delivery Network (CDN) and I found the Apache Hadoop project. I have been shocked about the nature of the project, where this comes from and all the toolkit generated around it. It’s massive amount of information, but fascinating for me.

Hadoop project comes from the need of requiring more resources for a give goal. The solution has been to distribute the data and the processing of data. You need to process a huge amount of data with a simple computer that offers limited processing cycles, then you use combined group of computers to run these processes in less time.

The major resources considered while distributed processing system are: Processor time, memory, hard drive space, network bandwidth. For instance virtual servers is a  sophisticated software that detects idle CPU capacity on a rack of physical server and parcels out the virtual environments to utilize it.

There are so many challenges on distributed processing when it’s applied at large scale, and Hadoop faces them. It’s important to mention these challenges to understand (or admire) what the Apache Hadoop project does.

• One individual compute node may overheat, crash, experience hard drive failures, or run out of memory or disk space.
• The networks can experience partial or total failure if switches and routers break down. The network congestion which causes data transfer.
• Multiple implementations or versions of client software may speak slightly different protocols from one another.
• If the input data set is several terabytes, then this would require a thousand or more machines to hold it in RAM.
• Intermediate data sets generated while performing a large-scale computation can take several times more space than what the original input data.
• Synchronization between multiple machines.

In each of the mentioned cases, the distributed system should be able to recover from the component failure or transient error condition and continue to make progress.