Tuesday Big Data Series

How does Apache Storm work?

In the previous post, we saw an introduction to Apache Storm, it’s characteristics and different use cases. Now, let us take a look at the Apache Storm architecture below –

Screen Shot 2016-07-08 at 8.26.58 PM.png


A storm cluster is similar to a Hadoop cluster. In a Hadoop cluster, we run what is called as MapReduce jobs, but in Storm we run Topologies. The key difference between a MapReduce job and a Topology is that a MapReduce job eventually finishes, whereas, a topology processes messages forever (until it’s killed).

A storm has three sets of nodes –

  1. Nimbus Node
  2. Zookeeper
  3. Supervisor

Nimbus Node

Nimbus node is the master node and is similar to the Hadoop JobTracker. It’s main tasks include –

  • Uploading the computations for execution.
  • Distribution of code in the cluster.
  • Launching workers across the cluster.
  • Assigning tasks to machines.
  • Monitoring computation and relocating resources or workers as and when needed.
  • Monitoring failures.

Zookeeper Node

  • Zookeeper node coordinates the storm cluster. All the coordination between Nimbus and Supervisors is done through Zookeeper.
  • In addition to this, since the Nimbus daemon and Supervisor daemons are fail-fast and stateless; all state is kept in Zookeeper or on local disk. This means you can kill -9 Nimbus or the Supervisors and they’ll start back up like nothing happened. This design leads to Storm clusters being incredibly stable.

Supervisor Node

Each worker node runs a daemon called the Supervisor.

  • Supervisor node communicates with the Nimbus node through zookeeper.
  • It listens for work assigned to its machine and starts and stops worker processes as necessary based on what Nimbus has assigned to it.
  • It starts or stops workers according to the signals from Nimbus node.
  • Each worker process executes a subset of a topology; a running topology consists of many worker processes spread across many machines.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s