Application Version and Upgrade Details

Application VersionBug fixes / Enhancements
5.0.4Fix provided related to component level threshold alerting.
5.0.3Fixed component level threshold alert enabling and disabling.
5.0.2Resource Display Order changes and Sub-Category changes.
Click here to view the earlier version updates
Application VersionBug fixes / Enhancements
5.0.1Curated DashBoard, cache flush changes.
5.0.0Added new Native Type NVIDIA BCM Linux Server and Metrics.
4.0.0Added support for nfs-server metrics on NVIDIA BCM Head Node.
3.0.0Added support for nfs-mount metrics on NVIDIA BCM Physical Node
2.0.0Supported new metric "nvidia_bcm_cluster_smartHdaTemp".
1.0.0Initial Support.

Introduction

NVIDIA Bright Cluster Manager offers fast deployment, and end-to-end management for heterogeneous high-performance computing (HPC) and AI server clusters at the edge in the data center, and in multi/hybrid-cloud environments. It automates provisioning and administration for clusters ranging in size from a couple of nodes to hundreds of thousands. It also supports CPU-based and NVIDIA GPU-accelerated systems, and enables orchestration with Kubernetes.

Monitoring Use cases

  • In case of any threshold breach or unexpected metric behavior, based on configurations the device monitoring helps to collect the metric values with respect to time and sends alerts to the intended customer team to act up immediately.
  • It helps the customer with smooth functioning of business with minimal or zero downtime in case of any infrastructure related issues occurring.

Resource Hierarchy

NVIDIA BrightCluster Manager
            → NVIDIA BCM Head Node
                → NVIDIA BCM Virtual Node
                → NVIDIA BCM Physical Node
                → NVIDIA BCM Linux Server

Prerequisites

  • OpsRamp Nextgen Gateway supported version 14.0.0 and above.
  • OpsRamp Classic Gateway supported version 14.0.0 and above.
    Note: OpsRamp recommends using the latest Gateway version for full coverage of recent bug fixes, enhancements, etc.

Default Monitoring Configurations

Bright Cluster Manager has default Global Device Management Policies, Global Templates, Global Monitors and Global Metrics in OpsRamp. You can customize these default monitoring configurations as per your business use cases by cloning respective Global Templates, and Global Device Management Policies. OpsRamp recommends doing this activity before installing the application to avoid noise alerts and data.

  1. Default Global Device Management Policies

    OpsRamp has a Global Device Management Policy for each Native Type of NVIDIA Bright Cluster Manager. You can find those Device Management Policies at Setup > Resources > Device Management Policies, search with suggested names in global scope. Each Device Management Policy follows below naming convention:

    {appName nativeType - version}

    Ex: NVIDIA BrightCluster Manager NVIDIA BCM Head Node (i.e, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node, version = 1)

  2. Default Global Templates

    OpsRamp has a Global Template for each Native Type of NVIDIA Bright Cluster Manager. You can find those templates at Setup > Monitoring > Templates, search with suggested names in global scope. Each template follows below naming convention:

    {appName nativeType 'Template' - version}

    Ex : nvidia-bright-cluster-manager NVIDIA BCM Head Node Template - 1 (i.e, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node Template, version = 1)

  3. Default Global Monitors

    OpsRamp has a Global Monitors for each Native Type which has monitoring support. You can find those monitors at Setup > Monitoring > Monitors, search with suggested names in global scope. Each Monitors follows below naming convention:

    {monitorKey appName nativeType - version}

    Ex: NVIDIA BCM Head Node Monitor nvidia-bright-cluster-manager NVIDIA BCM Head Node Container 1(i.e, monitorKey = NVIDIA BCM Head Node Monitor, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node, version = 1)

Supported Metrics

Click here to view the supported metrics
Native TypeMetric NameDisplay NameUnitsDescriptionMetric Available from Application Version
NVIDIA BrightCluster Managernvidia_bcm_cluster_nodeAvailabilityStatusnvidia bcm cluster nodeAvailabilityStatusNAIt monitors the availability status of each node. Below are the possible status values: UP-0, Down-1, Closed-2, installing-3, installer_failed-4, installer_rebooting-5, installer_callinginit-6, installer_unreachable-7, installer_burning-8, burning-9, unknown-10, opening-11, going_down-12, pending-13, no data-141.0.0
nvidia_bcm_cluster_slurmDaemon_statusnvidia bcm cluster slurmDaemon statusNAIt monitors the running status of slurmDaemon & allocation status of the each node, Below are the possible states: allocated-0, idle-1, down-21.0.0
nvidia_bcm_cluster_gpuUnits_closednvidia bcm cluster gpuUnits closedcountIt monitors the count of how many GPU units in closed state1.0.0
nvidia_bcm_cluster_gpuUnits_downnvidia bcm cluster gpuUnits downcountIt monitors the count of how many GPU units in Down state1.0.0
nvidia_bcm_cluster_gpuUnits_totalnvidia bcm cluster gpuUnits totalcountIt monitors the count of total GPU units1.0.0
nvidia_bcm_cluster_gpuUnits_upnvidia bcm cluster gpuUnits upcountIt monitors the count of how many GPU units are UP.1.0.0
nvidia_bcm_cluster_cpuIdlenvidia bcm cluster cpuIdle%It monitors the % of CPU Idle Time.1.0.0
nvidia_bcm_cluster_nodesTotalnvidia bcm cluster nodesTotalcountIt monitors the count of total number of nodes.1.0.0
nvidia_bcm_cluster_nodesUpnvidia bcm cluster nodesUpcountIt monitors the count of how many nodes are in UP state.1.0.0
nvidia_bcm_cluster_nodesDownnvidia bcm cluster nodesDowncountIt monitors the count of how many nodes are in DOWN state.1.0.0
nvidia_bcm_cluster_nodesClosednvidia bcm cluster nodesClosedcountIt monitors the count of how many nodes are in closed state.1.0.0
nvidia_bcm_cluster_devicesTotalnvidia bcm cluster devicesTotalcountIt monitors the count of total devices.1.0.0
nvidia_bcm_cluster_devicesUpnvidia bcm cluster devicesUpcountIt monitors the count of how many devices are in UP state.1.0.0
nvidia_bcm_cluster_devicesDownnvidia bcm cluster devicesDowncountIt monitors the count of how many devices are in DOWN state.1.0.0
nvidia_bcm_cluster_coresUpnvidia bcm cluster coresUpcountIt monitors the count of Cores which are in UP state.1.0.0
nvidia_bcm_cluster_coresTotalnvidia bcm cluster coresTotalcountIt monitors the total count of cores1.0.0
nvidia_bcm_cluster_totalUsersLoginnvidia bcm cluster totalUsersLogincountIt monitors the count of total users login.1.0.0
nvidia_bcm_cluster_totalKnownUsersnvidia bcm cluster totalKnownUserscountIt monitors the count of total known users.1.0.0
nvidia_bcm_cluster_occupationRatenvidia bcm cluster occupationRate%It monitors the Occupation rate in %.1.0.0
nvidia_bcm_cluster_cpuUtilizationnvidia bcm cluster cpuUtilization%It monitors the % of CPU Utilization1.0.0
nvidia_bcm_cluster_failedJobsnvidia bcm cluster failedJobscountReturns number of failed jobs on the cluster.1.0.0
nvidia_bcm_cluster_queuedJobsnvidia bcm cluster queuedJobscountReturns number jobs in queue on the cluster.1.0.0
nvidia_bcm_cluster_runningJobsnvidia bcm cluster runningJobscountReturns number of jobs running on the cluster.1.0.0
nvidia_bcm_cluster_nodesInQueuenvidia bcm cluster nodes InQueuecountReturns number of nodes queued on the cluster1.0.0
nvidia_bcm_cluster_uniqueUserLogincountnvidia bcm cluster uniqueUserLogincountcountReturns number of unique users logged in to the cluster.1.0.0
nvidia_bcm_cluster_smartHdaTempnvidia bcm cluster smartHdaTempcelsiusIt monitors the temperature of spindle disks2.0.0
NVIDIA BCM Head Nodenvidia_bcm_headNode_healthStatusnvidia bcm headNode healthStatusNAHealth status(PASS or FAIL) of each component.1.0.0
nvidia_bcm_headNode_blockedProcessesnvidia bcm headNode blockedProcessescountBlocked processes waiting for I/O.1.0.0
nvidia_bcm_headNode_systemCpuTimenvidia bcm headNode systemCpuTimejifflesCPU time spent in system mode.1.0.0
nvidia_bcm_headNode_cpuWaitTimenvidia bcm headNode cpuWaitTimejifflesCPU time spent in I/O wait mode.1.0.0
nvidia_bcm_headNode_errorsRecvnvidia bcm headNode errorsRecvErrors per SecPackets/s received with error.1.0.0
nvidia_bcm_headNode_errorsSentnvidia bcm headNode errorsSentErrors per SecPackets/s sent with error.1.0.0
nvidia_bcm_headNode_hardwareCorruptedMemorynvidia bcm headNode hardwareCorruptedMemoryBytesHardware corrupted memory detected by ECC.1.0.0
nvidia_bcm_headNode_memoryFreenvidia bcm headNode memoryFreeGBFree system memory.1.0.0
nvidia_bcm_headNode_gpu_utilizationnvidia bcm headNode gpu utilization%Average GPU utilization percentage.1.0.0
nvidia_bcm_headNode_gpu_temperaturenvidia bcm headNode gpu temperatureCelsiusGPU temperature.1.0.0
nvidia_bcm_headNode_nfs_client_rpcRetransnvidia_bcm_headNode_nfs_client_rpcRetranscountReturns NFS Client RPC Retrans4.0.0
nvidia_bcm_headNode_nfs_server_fileStalenvidia_bcm_headNode_nfs_server_fileStalecountReturns NFS Server File Stales4.0.0
nvidia_bcm_headNode_nfs_server_packetsnvidia_bcm_headNode_nfs_server_packetscountReturns NFS Server Packets4.0.0
nvidia_bcm_headNode_nfs_server_replyHitsnvidia_bcm_headNode_nfs_server_replyHitscountReturns NFS Server Reply Hits on HeadNode4.0.0
nvidia_bcm_headNode_nfs_server_replyMissesnvidia_bcm_headNode_nfs_server_replyMissescountReturns NFS Server Reply Miss on HeadNode4.0.0
nvidia_bcm_headNode_nfs_server_rpcBadAuthnvidia_bcm_headNode_nfs_server_rpcBadAuthReturns NFS Server Reply RPC BadAuth4.0.0
nvidia_bcm_headNode_nfs_server_rpcBadCallsnvidia_bcm_headNode_nfs_server_rpcBadCallscountReturns NFS Server Reply RPC BadCalls4.0.0
nvidia_bcm_headNode_nfs_server_rpcBadClntnvidia_bcm_headNode_nfs_server_rpcBadClntReturns NFS Server Reply RPC BadClnt4.0.0
nvidia_bcm_headNode_nfs_server_rpcXdrCallnvidia_bcm_headNode_nfs_server_rpcXdrCallReturns NFS Server Reply RPC XDR Call4.0.0
NVIDIA BCM Physical Nodenvidia_bcm_physicalNode_healthStatusnvidia bcm physicalNode healthStatusNAHealth status(PASS or FAIL) of each component.1.0.0
nvidia_bcm_physicalNode_blockedProcessesnvidia bcm physicalNode blockedProcessescountBlocked processes waiting for I/O.1.0.0
nvidia_bcm_physicalNode_systemCpuTimenvidia bcm physicalNode systemCpuTimejifflesCPU time spent in system mode.1.0.0
nvidia_bcm_physicalNode_cpuWaitTimenvidia bcm physicalNode cpuWaitTimejifflesCPU time spent in I/O wait mode.1.0.0
nvidia_bcm_physicalNode_errorsRecvnvidia bcm physicalNode errorsRecvErrors per SecPackets/s received with error.1.0.0
nvidia_bcm_physicalNode_errorsSentnvidia bcm physicalNode errorsSentErrors per SecPackets/s sent with error.1.0.0
nvidia_bcm_physicalNode_hardwareCorruptedMemorynvidia bcm physicalNode hardwareCorruptedMemoryBytesHardware corrupted memory detected by ECC.1.0.0
nvidia_bcm_physicalNode_memoryFreenvidia bcm physicalNode memoryFreeGBFree system memory.1.0.0
nvidia_bcm_physicalNode_gpu_utilizationnvidia bcm physicalNode gpu utilization%GPU utilization percentage1.0.0
nvidia_bcm_physicalNode_gpu_temperaturenvidia bcm physicalNode gpu temperatureCelsiusGPU temperature.1.0.0
nvidia_bcm_physicalNode_nfsmount_totalSizenvidia bcm physicalNode nfsmount totalSizeGBIt monitors the total size of the nfs mount file on the node.3.0.0
nvidia_bcm_physicalNode_nfsmount_usedSizenvidia bcm physicalNode nfsmount usedSizeGBIt monitors the used size of the nfs mount file on the node.3.0.0
nvidia_bcm_physicalNode_nfsmount_utilizationnvidia bcm physicalNode nfsmount utilization%It monitors the percentage utilization of the nfs mount file on the node.3.0.0
NVIDIA BCM Linux Servernvidia_bcm_linuxServer_nfs_server_packetsnvidia_bcm_linuxServer_nfs_server_packetscountReturns NFS Server Packets5.0.0
nvidia_bcm_linuxServer_nfs_server_replyHitsnvidia_bcm_linuxServer_nfs_server_replyHitscountReturns NFS Server Reply Hits on LinuxServer5.0.0
nvidia_bcm_linuxServer_nfs_server_replyMissesnvidia_bcm_linuxServer_nfs_server_replyMissescountReturns NFS Server Reply Miss on LinuxServer5.0.0
nvidia_bcm_linuxServer_nfs_server_rpcBadCallsnvidia_bcm_linuxServer_nfs_server_rpcBadCallscountReturns NFS Server Reply RPC BadCalls5.0.0
nvidia_bcm_linuxServer_nfs_server_rpcBadAuthnvidia_bcm_linuxServer_nfs_server_rpcBadAuthcountReturns NFS Server Reply RPC BadAuth5.0.0
nvidia_bcm_linuxServer_nfs_server_rpcBadClntnvidia_bcm_linuxServer_nfs_server_rpcBadClntcountReturns NFS Server Reply RPC BadClnt5.0.0
nvidia_bcm_linuxServer_nfs_server_fileStalenvidia_bcm_linuxServer_nfs_server_fileStalecountReturns NFS Server File Stales5.0.0
nvidia_bcm_linuxServer_memory_utilizationnvidia_bcm_linuxServer_memory_utilization%It monitors the percentage memory utilization of Linux server5.0.0
nvidia_bcm_linuxServer_disk_utilizationnvidia_bcm_linuxServer_disk_utilization%It monitors the percentage disk utilization of Linux server5.0.0
nvidia_bcm_linuxServer_cpu_utilizationnvidia_bcm_linuxServer_cpu_utilization%It monitors the percentage cpu utilization of Linux server5.0.0

Configure and Install the NVIDIA Bright Cluster Manager Integration

  1. From All Clients, select a client.
  2. Go to Setup > Account.
  3. Select the Integrations and Apps tab.
  4. The Installed Integrations page, where all the installed applications are displayed.
    Note: If there are no installed applications, it will navigate to the Available Integrations and Apps page.
  5. Click + ADD on the Installed Integrations page. The Available Integrations and Apps page displays all the available applications along with the newly created application with the version.
    Note: Search for the application using the search option available. Also you can use the All Categories option to search.
  1. Click ADD in the Nvidia Bright Cluster Manager tile.
  2. In the Configurations page, click + ADD. The Add Configuration page appears.
  3. Enter the below mentioned BASIC INFORMATION:
FunctionalityDescription
NameEnter the name for the configuration.
BCM Manager IP AddressProvide the BCM Manager IP Address.
BCM Manager CredentialsSelect the credentials from the drop-down list.

Note: Click + Add to create a credential.
CMSH Source Path in BCM ManagerEnter the source path for the BCM Manager.

Note: By default /cm/local/apps/cmd/bin is provided.

Notes:

  • BCM Manager IP Address should be accessible from Gateway.
  • Select App Failure Notifications; if turned on, you will be notified in case of an application failure that is, Connectivity Exception, Authentication Exception.
  1. Select the below mentioned Custom Attribute:
FunctionalityDescription
Custom AttributeSelect the custom attribute from the drop down list box.
ValueSelect the value from the drop down list box.

Note: The custom attribute that you add here will be assigned to all the resources that are created by the integration. You can add a maximum of five custom attributes (key and value pair).

  1. In the RESOURCE TYPE section, select:

    • ALL: All the existing and future resources will be discovered.
    • SELECT: You can select one or multiple resources to be discovered.

  2. In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:

    • Minutes
    • Hourly
    • Daily
    • Weekly
    • Monthly

  3. Click ADD.

Now the configuration is saved and displayed on the configurations page after you save it.
Note: From the same page, you may Edit and Remove the created configuration.

  1. Click NEXT.

  2. Below are the optional steps you can perform on the Installing page.

  • Under the ADVANCED SETTINGS, Select the Bypass Resource Reconciliation option, if you wish to bypass resource reconciliation when encountering the same resources discovered by multiple applications.

    Note: If two different applications provide identical discovery attributes, two separate resources will be generated with those respective attributes from the individual discoveries.

Cisco FirePower
  • Click +ADD to create a new collector by providing a name or use the pre-populated name.
Aruba Airwave Integrations
  1. Select an existing registered profile.
Aruba Airwave Integrations
  1. Click FINISH.

The integration is now installed and displayed on the Installed Integration page. Use the search field to find the installed integration.

Modify the Configuration

View the NVIDIA Bright Cluster Manager Details

The NVIDIA Bright Cluster Manager integration is displayed in the Infrastructure > Resources > Cluster. You can navigate to the Attributes tab to view the discovery details, and the Metrics tab to view the metric details for NVIDIA Bright Cluster Manager.

NVIDIA Bright Cluster Manager
NVIDIA Bright Cluster Manager

Resource Filter Input keys

Click here to view the Supported Input Keys
Resource TypeKeys
All TypesresourceName
hostName
aliasName
dnsName
ipAddress
macAddress
os
make
model
serialNumber
NVIDIA BrightCluster Manager
NVIDIA BCM Head NodeType
Network
Category
NVIDIA BCM Physical NodeType
Category
NVIDIA BCM Virtual NodeType
Network
Category

Supported Alert Custom Macros

Customize the alert subject and description with below macros then it will generate alert based on customisation.
Supported macros keys:

Click here to view the alert subject and description with macros

                                ${resource.name}

                                ${resource.ip}

                                ${resource.mac}

                                ${resource.aliasname}

                                ${resource.os}

                                ${resource.type}

                                ${resource.dnsname}

                                ${resource.alternateip}

                                ${resource.make}

                                ${resource.model}

                                ${resource.serialnumber}

                                ${resource.systemId}

                                ${parent.resource.name}

                                ${Custom attributes on the resource}

Risks, Limitations & Assumptions

  • Application can handle Critical/Recovery failure notifications for below two cases when user enables App Failure Notifications in configuration:
    • Connectivity Exception (ConnectTimeoutException, HttpHostConnectException, UnknownHostException).
    • Authentication Exception (UnauthorizedException).
  • Application will not send any duplicate/repeat failure alert notification until the already existed critical alert is recovered.
  • Application cannot control monitoring pause/resume actions based on above alerts.
  • Metrics can be used to monitor NVIDIA Bright Cluster Manager resources and can generate alerts based on the threshold values.
  • No support of showing activity logs.
  • The Template Applied Time will only be displayed if the collector profile (Classic and NextGen Gateway) is version 18.1.0 or higher.
  • This application supports both Classic Gateway and NextGen Gateway.
  • Application is not returning any data if any SSH connectivity issues (based on monitoring and discovery frequency our App will try to establish SSH connections to the target device).
  • Application will work only with SSH credentials, with ssh port 22 in the open state.
  • Virtual Nodes discovery and Monitoring will not be supported.
  • To discover and monitor the NVIDIA BCM Linux Server and its metrics, sshpass should be installed on the BCM Cluster. If the sshpass package is not available, ensure that the credentials of both the BCM cluster and the Linux Server are the same.