Friday "Term of the week" Series

Term of the Week : Apache Tez

The Apache Tez project is an extensible framework built on top of Apache Hadoop YARN. It is used to process data, that earlier took multiple MR jobs, now in a single Tez job which uses Directed Acyclic Graph (DAG) for data processing. It is used for building high performance batch and interactive data processing applications.… Continue reading Term of the Week : Apache Tez

Monday Technology Series

What is Apache Tez?

The Apache Tez project is an extensible framework built on top of Apache Hadoop YARN. It is used to process data, that earlier took multiple MR jobs, now in a single Tez job which uses Directed Acyclic Graph (DAG) for data processing. It is used for building high performance batch and interactive data processing applications. It drastically improves… Continue reading What is Apache Tez?

Friday "Term of the week" Series

Term of the Week : Apache Pig

The term Apache Pig  refers to an open source scripting platform by Apache used for analyzing and processing large data sets. It allows users to write complex map reduce problems using a simple scripting language called Pig Latin. Pig translates the Pig Latin script into MapReduce so that it can be executed within YARN for access to a… Continue reading Term of the Week : Apache Pig

Tuesday Big Data Series

Hive or Pig?

Which one of them is your favorite – Hive or Pig? What do you prefer to work with? People often confuse as to when to use Hive and when to use Pig. And, while in most of the cases, either of it can be used, the question that arises is why both of them exist in… Continue reading Hive or Pig?

Monday Technology Series

What is Apache Hive?

Apache Hive is a data warehouse infrastructure built on top of Hadoop which allows querying and managing large datasets residing in distributed storage. It provides an SQL-like language called as HiveQL with schema on read and transparently converts queries to map reduce, tez or spark jobs. All these execution engines run on Hadoop YARN. The HiveQL language also… Continue reading What is Apache Hive?

Friday "Term of the week" Series

Term of the Week : HDFS

Image Source HDFS or the Hadoop Distributed File System, is a Java-based filesystem for storing large volumes of data in the Hadoop framework. It solves the problem of storing and managing enormous amounts of data with it’s high scalability, fault-tolerance, high availability and cost efficiency that make it so popular. Related Posts: HDFS and it’s features that make… Continue reading Term of the Week : HDFS

Monday Technology Series

Apache Pig and it’s awesome features!

Pig is an open source scripting platform by Apache used for analyzing and processing large data sets. It allows users to write complex map reduce problems using a simple scripting language called Pig Latin. Pig translates the Pig Latin script into MapReduce so that it can be executed within YARN for access to a single dataset stored… Continue reading Apache Pig and it’s awesome features!

Tuesday Big Data Series

Understanding HDFS quotas

Every Hadoop system has an Hadoop Administrator and Hadoop users/developers. The Administrator is responsible for deployment and maintenance of the entire infrastructure. He is responsible for cluster availability, file system management, security, installation of latest updates, and all other things that need to keep the system up and running. The administrator is also responsible for… Continue reading Understanding HDFS quotas

Monday Technology Series

What is Oozie?

Why do we need Oozie? The Hadoop stack consists of a variety of tools like Pig, Map Reduce, Hive, HBase, Sqoop etc. At times, when dealing with large data sets, we might have to use a combination of either of these technologies along with plain old Java, Python, Perl or shell scripts to get work done… Continue reading What is Oozie?

Tuesday Big Data Series

Understanding NameNode and DataNode in HDFS

HDFS has a master/slave architecture and is built-up of basically two kinds of nodes: NameNode (which acts as a Master) and DataNodes (which acts as slaves). NameNode and DataNode are pieces of software that are designed to run on commodity machines which typically run on a GNU/Linux operating system (OS). HDFS is built using Java language,… Continue reading Understanding NameNode and DataNode in HDFS