OpsRamp allows you to ship the Prometheus (OpenMetricFormat) metrics to the SaaS platform with the use of Prometheus Remote Write integration.

Prerequisites

  • An environment instrumented with Prometheus that has access to the OpsRamp portal.

Add a Prometheus Remote Write integration

To install Prometheus Remote Write integration:

  1. Click All Clients and from the drop-down list, select a client.

  2. Navigate to Setup > Accounts.

  3. Click on the Integrations and Apps tile.

  4. The Installed Integrations page, where all the installed applications are displayed. Note: If there are no installed applications, it will navigate to the Available Integrations and Apps page.

  5. Click + ADD on the Installed Integrations page. The Available Integrations and Apps page displays all the available applications along with the newly created application with the version.
    Note: You can even search for the application using the search option available. Also you can use the All Categories option to search.

  6. Click +ADD in the Prometheus Remote Write application.

  7. Enter a Name for the Prometheus Remote Write integration.

  8. Select the Type: Normal or Containerized.
    Normal: Select this type when Prometheus is installed on a virtual machine or bare metal machine.
    Containerized: Select this type when Prometheus is installed in a container orchestration environment like Kubernetes.

  9. Click Next.

    For Normal deployment:

    1. Follow the below given on-screen instructions displayed after installing the integration and click Finish:
    For Containerized deployment:

    ConfigMap:

    1. Follow the below given on-screen instructions displayed after installing the integration:

    For Prometheus Operator:

    1. Follow the below given on-screen instructions displayed after installing the integration:

    Sample configurations

      Normal:

        global:
          scrape_interval: 15s
          evaluation_interval: 30s
          body_size_limit: 15MB
          sample_limit: 1500
          target_limit: 30
          label_limit: 30
          label_name_length_limit: 200
          label_value_length_limit: 200
      
          external_labels:
            OpsRampIntegrationName : DSF
      
        rule_files:
          - "first.rules"
          - "my/*.rules"
      
        remote_write:
          - url: "https://portal.opsramp.net/prom/push"
            authorization:
              type: Bearer
              credentials: ahsd78fyasdifhasuidfhasdifuh88ha
            headers:
              tenantId: jgowioer-b492-732d-a851-0d603769be80
              integrationUUID: INTG-137e9ab3-cccc-bbbb-dddd-2723d7aaaaaa
      
        scrape_configs:
          - job_name: docker
            docker_sd_configs:
              - host: unix:///var/run/docker.sock
      
        alerting:
          alertmanagers:
            - scheme: https
              static_configs:
                - targets:
                    - "1.2.3.4:9093"
                    - "1.2.3.5:9093"
                    - "1.2.3.6:9093"

      Containerized:

        ConfigMap:
          global:
            scrape_interval: 15s
            evaluation_interval: 30s
            body_size_limit: 15MB
            sample_limit: 1500
            target_limit: 30
            label_limit: 30
            label_name_length_limit: 200
            label_value_length_limit: 200
          
            external_labels:
              OpsRampIntegrationName : DSF
          
          rule_files:
            - "first.rules"
            - "my/*.rules"
          
          remote_write:
            - url: "https://portal.opsramp.net/prom/push"
              authorization:
                type: Bearer
                credentials: ahsd78fyasdifhasuidfhasdifuh88ha
              headers:
                tenantId: jgowioer-b492-732d-a851-0d603769be80
                integrationUUID: INTG-137e9ab3-cccc-bbbb-dddd-2723d7aaaaaa
          
          scrape_configs:
            - job_name: docker
              docker_sd_configs:
                - host: unix:///var/run/docker.sock
          
          alerting:
            alertmanagers:
              - scheme: https
                static_configs:
                  - targets:
                      - "1.2.3.4:9093"
                      - "1.2.3.5:9093"
                      - "1.2.3.6:9093"
          Operator:
            Prometheus configuration YAML:
              apiVersion: monitoring.coreos.com/v1
              kind: Prometheus
              metadata:
                labels:
                  app.kubernetes.io/component: prometheus
                  app.kubernetes.io/instance: k8s
                  app.kubernetes.io/name: prometheus
                  app.kubernetes.io/part-of: kube-prometheus
                  app.kubernetes.io/version: 2.44.0
                name: k8s
                namespace: monitoring
              spec:
                alerting:
                  alertmanagers:
                  - apiVersion: v2
                    name: alertmanager-main
                    namespace: monitoring
                    port: web
                enableFeatures: []
                externalLabels:
                  OpsRampIntegrationName : gke-cluster
                image: quay.io/prometheus/prometheus:v2.44.0
                nodeSelector:
                  kubernetes.io/os: linux
                podMetadata:
                  labels:
                    app.kubernetes.io/component: prometheus
                    app.kubernetes.io/instance: k8s
                    app.kubernetes.io/name: prometheus
                    app.kubernetes.io/part-of: kube-prometheus
                    app.kubernetes.io/version: 2.44.0
                podMonitorNamespaceSelector: {}
                podMonitorSelector: {}
                probeNamespaceSelector: {}
                probeSelector: {}
                replicas: 2
                resources:
                  requests:
                    memory: 400Mi
                ruleNamespaceSelector: {}
                ruleSelector: {}
                securityContext:
                  fsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1000
                serviceAccountName: prometheus-k8s
                serviceMonitorNamespaceSelector: {}
                serviceMonitorSelector: {}
                version: 2.44.0
                remoteWrite:
                  - url: "https://portal.opsramp.com/prom/push"
                    authorization:
                      type: Bearer
                      credentials:
                        name: remote-write-opsramp # give name of the k8s secret 
                        key: token # give the name of key from which the token needs to be taken
                    headers:
                      tenantId: jgowioer-b492-732d-a851-0d603769be80
                      integrationUUID: INTG-137e9ab3-cccc-bbbb-dddd-2723d7aaaaaa
              
            Prometheus Kubernetes Secret:
              apiVersion: v1
              kind: Secret
              metadata:
                name: remote-write-opsramp
              type: Opaque
              data:
                token: ahsd78fyasdifhasuidfhasdifuh88ha
              

          Constraints

          The number of active series per metric per client is 50000. You can avoid the limit by configuring Prometheus to filter metrics. For example, use the following configuration to limit apiserver_request_duration_seconds_bucket and etcd_request_duration_seconds_bucket metrics:

          remoteWrite:
            - url: http://portal.opsramp.net/prom/push
              authorization:
                type: Bearer
                credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
              headers:
                tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
              writeRelabelConfigs:
              - action: drop
                regex: apiserver_request_duration_seconds_bucket
                sourceLabels:
                - __name__
              - action: drop
                regex: etcd_request_duration_seconds_bucket
                sourceLabels:
                - __name__

          Configuration

          Remote Write Tuning

          Below are the recommended Prometheus Remote-Write Configuration tuning:

          Configuration ParameterDescription
          capacityHow many samples are queued in memory per shard before blocking reading from the WAL. When the WAL is blocked, samples cannot be appended to any shards and all throughput ceases.
          max_shardsThe maximum number of shards, or parallelism, Prometheus uses for each remote write queue.
          min_shardsThe minimum number of shards used by Prometheus, which is the number of shards used when remote write starts.
          max_samples_per_sendThe maximum number of samples sent per batch, which can be adjusted depending on the back end in use.
          batch_send_deadlineThe maximum time interval between sends for a single shard, in seconds.
          min_backoffThe minimum time to wait before retrying a failed request, in seconds.
          max_backoffThe maximum time to wait before retrying a failed request, in seconds.

          See: https://prometheus.io/docs/practices/remote_write/ for more details.

          Configuration examples to filter metrics pushed to remote write

          Prometheus can be configured to filter the metrics to be pushed to a remote endpoint like the agent. See the Prometheus configuration remote_write property documentation for more information.

          Drop samples example

          The following writeRelabelConfigs configuration drops samples with a metric name that starts with go:

          remote_write:
            - url: http://portal.opsramp.net/prom/push
              authorization:
                type: Bearer
                credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
              headers:
                tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
              writeRelabelConfigs:
              - sourceLabels: [__name__]
                regex: 'go_.*'
                action: drop

          Keep samples example

          The following writeRelabelConfigs configuration keeps samples with a metric name that starts with go:

          remote_write:
            - url: http://portal.opsramp.net/prom/push
              authorization:
                type: Bearer
                credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
              headers:
                tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
              writeRelabelConfigs:
                - sourceLabels: [__name__]
                  regex: 'go_.*'
                  action: keep

          Replace samples example

          You can also change a label using the replace action before pushing it to a remote endpoint. The following example replaces samples with a metric name that starts with go with the name sample_go_label:

          remote_write:
            - url: http://portal.opsramp.net/prom/push
              authorization:
                type: Bearer
                credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
              headers:
                tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
              writeRelabelConfigs:
              - sourceLabels: [__name__]
                regex: 'go_.*'
                action: replace
                targetLabels: sample_go_label