Grid Computing

The high-performance computing (HPC) and grid computing communities have bee doing large-scale data processing for years, using such application program interface (APIs) as the Message Passing Interface (MPI). Broadly, the approach in HPC is t distribute the work across a cluster of machines, which access a shared filesystem, hoste by a storage area network (SAN). This works well for predominantly compute-intensiv jobs, but it becomes a problem when nodes need to access larger data volumes (hundred of gigabytes, the point at which Hadoop really starts to shine), since the networ bandwidth is the bottleneck and compute nodes become idle

Hadoop tries to co-locate the data with the compute nodes, so data access is fast because i is local.[8] This feature, known as data locality, is at the heart of data processing i Hadoop and is the reason for its good performance. Recognizing that network bandwidt is the most precious resource in a data center environment (it is easy to saturate networ links by copying data around), Hadoop goes to great lengths to conserve it by explicitl modeling network topology. Notice that this arrangement does not preclude high-CP analyses in Hadoop

MPI gives great control to programmers, but it requires that they explicitly handle th mechanics of the data flow, exposed via low-level C routines and constructs such a sockets, as well as the higher-level algorithms for the analyses. Processing in Hadoo operates only at the higher level: the programmer thinks in terms of the data model (suc as key-value pairs for MapReduce), while the data flow remains implicit

Coordinating the processes in a large-scale distributed computation is a challenge. Th hardest aspect is gracefully handling partial failure — when you don’t know whether o not a remote process has failed — and still making progress with the overall computation Distributed processing frameworks like MapReduce spare the programmer from having t think about failure, since the implementation detects failed tasks and reschedule replacements on machines that are healthy. MapReduce is able to do this because it is shared-nothing architecture, meaning that tasks have no dependence on one other. (This i a slight oversimplification, since the output from mappers is fed to the reducers, but this i under the control of the MapReduce system; in this case, it needs to take more car rerunning a failed reducer than rerunning a failed map, because it has to make sure it ca retrieve the necessary map outputs and, if not, regenerate them by running the relevan maps again.) So from the programmer’s point of view, the order in which the tasks ru doesn’t matter. By contrast, MPI programs have to explicitly manage their ow checkpointing and recovery, which gives more control to the programmer but makes the more difficult to write.