Spark - Statistics

Description

It monitors the spark job,stage,RDD and executor parameters.Job parameters : Spark Job Num Tasks, Spark Job Num Active Tasks, Spark Job Num Skipped Tasks, Spark Job Num Failed Tasks, Spark Job Num Completed Tasks, Spark Job Num Active Stages, Spark Job Num Completed Stages, Spark Job Num Skipped Stages, Spark Job Num Failed Stages.Stage parameters : Spark Stage Num Active Tasks, Spark Stage Num Complete Tasks, Spark Stage Num Failed Tasks, Spark Stage Executor Run Time, Spark Stage Input Bytes, Spark Stage Input Records, Spark Stage Output Bytes, Spark Stage Output Records, Spark Stage Shuffle Read Bytes, Spark Stage Shuffle Read Records, Spark Stage Shuffle Write Bytes, Spark Stage Shuffle Write Records, Spark Stage Memory Bytes Spilled.RDD parameters : Spark Rdd Memory Used, Spark Rdd Disk Used.Executor parameters : Spark Executor Memory Used, Spark Executor Disk Used, Spark Executor Active Tasks, Spark Executor Failed Tasks, Spark Executor Completed Tasks, Spark Executor Total Tasks, Spark Executor Total Duration, Spark Executor Total Input Bytes, Spark Executor Total Shuffle Read, Spark Executor Total Shuffle Write. Validated on Spark v2.3.1

Prerequisites

No prerequisite

Supported Metric

Monitor NameMetric NameMetric Display NameUnitDescription
Spark - Executor Statisticsspark.executor.active_tasksSpark Executor Active TasksNULLNumber of active tasks in the application's executors
spark.executor.total_shuffle_writeSpark Executor Total Shuffle WriteNULLTotal number of shuffled bytes in the application's executors
spark.executor.failed_tasksSpark Executor Failed TasksNULLNumber of failed tasks in the application's executors
spark.executor.total_shuffle_readSpark Executor Total Shuffle ReadNULLTotal number of bytes read during a shuffle in the application's executors
spark.executor.completed_tasksSpark Executor Completed TasksNULLNumber of completed tasks in the application's executors
spark.executor.memory_usedSpark Executor Memory UsedNULLAmount of memory used for cached RDDs in the application's executors
spark.executor.total_tasksSpark Executor Total TasksNULLTotal number of tasks in the application's executors
spark.executor.total_durationSpark Executor Total DurationNULLFraction of time (ms/s) spent by the application's executors executing tasks
spark.executor.disk_usedSpark Executor Disk UsedNULLAmount of disk space used by persisted RDDs in the application's executors
spark.executor.total_input_bytesSpark Executor Total Input BytesNULLTotal number of input bytes in the application's executors
Spark - Job Statisticsspark.job.num_skipped_stagesSpark Job Num Skipped StagesNULLNumber of skipped stages in the application
spark.job.num_failed_tasksSpark Job Num Failed TasksNULLNumber of failed tasks in the application
spark.job.num_completed_stagesSpark Job Num Completed StagesNULLNumber of completed stages in the application
spark.job.num_active_stagesSpark Job Num Active StagesNULLNumber of active stages in the application
spark.job.num_failed_stagesSpark Job Num Failed StagesNULLNumber of failed stages in the application
spark.job.num_skipped_tasksSpark Job Num Skipped TasksNULLNumber of skipped tasks in the application
spark.job.num_tasksSpark Job Num TasksNULLNumber of tasks in the application
spark.job.num_active_tasksSpark Job Num Active TasksNULLNumber of active tasks in the application
spark.job.num_completed_tasksSpark Job Num Completed TasksNULLNumber of completed tasks in the application
Spark - RDD Statisticsspark.rdd.memory_usedSpark Rdd Memory UsedNULLAmount of memory used in the application's persisted RDDs
spark.rdd.disk_usedSpark Rdd Disk UsedNULLAmount of disk space used by persisted RDDs in the application
Spark - Stage Statisticsspark.stage.shuffle_read_recordsSpark Stage Shuffle Read RecordsNULLNumber of records read during a shuffle in the application's stages
spark.stage.shuffle_write_recordsSpark Stage Shuffle Write RecordsNULLNumber of shuffled records in the application's stages
spark.stage.input_bytesSpark Stage Input BytesNULLInput bytes in the application's stages
spark.stage.shuffle_read_bytesSpark Stage Shuffle Read BytesNULLNumber of bytes read during a shuffle in the application's stages
spark.stage.shuffle_write_bytesSpark Stage Shuffle Write BytesNULLNumber of shuffled bytes in the application's stages
spark.stage.num_active_tasksSpark Stage Num Active TasksNULLNumber of active tasks in the application's stages
spark.stage.output_recordsSpark Stage Output RecordsNULLOutput records in the application's stages
spark.stage.memory_bytes_spilledSpark Stage Memory Bytes SpilledNULLNumber of bytes spilled to disk in the application's stages
spark.stage.num_complete_tasksSpark Stage Num Complete TasksNULLNumber of complete tasks in the application's stages
spark.stage.input_recordsSpark Stage Input RecordsNULLInput records in the application's stages
spark.stage.output_bytesSpark Stage Output BytesNULLOutput bytes in the application's stages
spark.stage.num_failed_tasksSpark Stage Num Failed TasksNULLNumber of failed tasks in the application's stages
spark.stage.executor_run_timeSpark Stage Executor Run TimeNULLFraction of time (ms/s) spent by the executor in the application's stages