Collector Type: Agent
Category: Application Monitors
Application Name: Spark
Global Template Name: Linux - Apache Spark Monitors
Introduction
Apache Spark is an open-source distributed general-purpose cluster-computing framework. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.
Pre-requisite: Spark REST API
Parameters
| Name | Default Value | Description |
|---|---|---|
| Protocol | http | The protocol to be used. |
| Host IP Address | 127.0.0.1 | The host on which Spark is running. |
| Spark Web UI Port | 8080 | The port on which Spark Web UI is running. |
Note: All field attributes are mandatory, use default values wherever applicable.
Collected Metrics
| Metric Name | Display Name | Description |
|---|---|---|
| spark.workers | Spark Workers | The number of workers connected to the master. |
| spark.cores | Spark Cores | The number of CPUs available for all workers. |
| spark.cores.used | Spark Cores Used | The number of CPUs used for all applications. |
| spark.applications.active | Spark Applications Active | The number of applications waiting or running. |
| spark.applications.completed | Spark Applications Completed | The number of application completed. |
| spark.drivers.active | Spark Drivers Active | The number of drivers available. |
| spark.status | Spark Status | Available status of spark master ex:alive. |
| spark.memory | Spark Memory | Calculates the total memory available on spark master. |
| spark.memory.used | Spark Memory Used | Calculates the memory used by the applications on spark master. |