In the previous post, we saw an introduction to Apache Storm, it’s characteristics and different use cases. Now, let us take a look at the Apache Storm architecture below –

Reference

A storm cluster is similar to a Hadoop cluster. In a Hadoop cluster, we run what is called as MapReduce jobs, but in Storm we run Topologies. The key difference between a MapReduce job and a Topology is that a MapReduce job eventually finishes, whereas, a topology processes messages forever (until it’s killed).

A storm has three sets of nodes –

1. Nimbus Node
2. Zookeeper
3. Supervisor

Nimbus Node

Nimbus node is the master node and is similar to the Hadoop JobTracker. It’s main tasks include –

• Uploading the computations for execution.
• Distribution of code in the cluster.
• Launching workers across the cluster.
• Assigning tasks to machines.
• Monitoring computation and relocating resources or workers as and when needed.
• Monitoring failures.

Zookeeper Node

• Zookeeper node coordinates the storm cluster. All the coordination between Nimbus and Supervisors is done through Zookeeper.
• In addition to this, since the Nimbus daemon and Supervisor daemons are fail-fast and stateless; all state is kept in Zookeeper or on local disk. This means you can kill -9 Nimbus or the Supervisors and they’ll start back up like nothing happened. This design leads to Storm clusters being incredibly stable.

Supervisor Node

Each worker node runs a daemon called the Supervisor.

• Supervisor node communicates with the Nimbus node through zookeeper.
• It listens for work assigned to its machine and starts and stops worker processes as necessary based on what Nimbus has assigned to it.
• It starts or stops workers according to the signals from Nimbus node.
• Each worker process executes a subset of a topology; a running topology consists of many worker processes spread across many machines.