Abstract

There is a growing need for ad-hoc analysis of extremely large data sets, especially at internet companies\nwhere invocation critically depends on being able to analyze terabytes of data collected everyday Parallel database\nproducts prohibitively expensive at the scale. The success of map-reduce programming model and implementations on\nhardware is the evidence for all. But the map-reduce is too low-level and rigid and the maintenance of user code is very\ndifficult and it is hard to reuse. Here it describes a new language called Pig Latin that spots a difference between\ndeclarative style of SQL (Structure Query Language) and low-level procedural style of map-reduce. Thus, Pig is fully\nimplemented and complies Pig Latin into physical plans that are executed over Hadoop, an open-source, map-reduce\nimplemented.