OpsRamp allows you to ship the Prometheus (OpenMetricFormat) metrics to the SaaS platform with the use of Prometheus Remote Write integration.
Prerequisites
- An environment instrumented with Prometheus that has access to the OpsRamp portal.
Add a Prometheus Remote Write integration
To install Prometheus Remote Write integration:
Click All Clients and from the drop-down list, select a client.
Navigate to Setup > Accounts.
Click on the Integrations and Apps tile.
The Installed Integrations page, where all the installed applications are displayed. Note: If there are no installed applications, it will navigate to the Available Integrations and Apps page.
Click + ADD on the Installed Integrations page. The Available Integrations and Apps page displays all the available applications along with the newly created application with the version.
Note: You can even search for the application using the search option available. Also you can use the All Categories option to search.Click +ADD in the Prometheus Remote Write application.
Enter a Name for the Prometheus Remote Write integration.
Select the Type: Normal or Containerized.
Normal: Select this type when Prometheus is installed on a virtual machine or bare metal machine.
Containerized: Select this type when Prometheus is installed in a container orchestration environment like Kubernetes.Click Next.
For Normal deployment:
- Follow the below given on-screen instructions displayed after installing the integration and click Finish:
- For Containerized deployment:
- Follow the below given on-screen instructions displayed after installing the integration:
ConfigMap:
For Prometheus Operator:
- Follow the below given on-screen instructions displayed after installing the integration:
Sample configurations
Normal:
global:
scrape_interval: 15s
evaluation_interval: 30s
body_size_limit: 15MB
sample_limit: 1500
target_limit: 30
label_limit: 30
label_name_length_limit: 200
label_value_length_limit: 200
external_labels:
OpsRampIntegrationName : DSF
rule_files:
- "first.rules"
- "my/*.rules"
remote_write:
- url: "https://portal.opsramp.net/prom/push"
authorization:
type: Bearer
credentials: ahsd78fyasdifhasuidfhasdifuh88ha
headers:
tenantId: jgowioer-b492-732d-a851-0d603769be80
integrationUUID: INTG-137e9ab3-cccc-bbbb-dddd-2723d7aaaaaa
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
alerting:
alertmanagers:
- scheme: https
static_configs:
- targets:
- "1.2.3.4:9093"
- "1.2.3.5:9093"
- "1.2.3.6:9093"
Containerized:
ConfigMap:
global:
scrape_interval: 15s
evaluation_interval: 30s
body_size_limit: 15MB
sample_limit: 1500
target_limit: 30
label_limit: 30
label_name_length_limit: 200
label_value_length_limit: 200
external_labels:
OpsRampIntegrationName : DSF
rule_files:
- "first.rules"
- "my/*.rules"
remote_write:
- url: "https://portal.opsramp.net/prom/push"
authorization:
type: Bearer
credentials: ahsd78fyasdifhasuidfhasdifuh88ha
headers:
tenantId: jgowioer-b492-732d-a851-0d603769be80
integrationUUID: INTG-137e9ab3-cccc-bbbb-dddd-2723d7aaaaaa
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
alerting:
alertmanagers:
- scheme: https
static_configs:
- targets:
- "1.2.3.4:9093"
- "1.2.3.5:9093"
- "1.2.3.6:9093"
Operator:
Prometheus configuration YAML:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.44.0
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- apiVersion: v2
name: alertmanager-main
namespace: monitoring
port: web
enableFeatures: []
externalLabels:
OpsRampIntegrationName : gke-cluster
image: quay.io/prometheus/prometheus:v2.44.0
nodeSelector:
kubernetes.io/os: linux
podMetadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.44.0
podMonitorNamespaceSelector: {}
podMonitorSelector: {}
probeNamespaceSelector: {}
probeSelector: {}
replicas: 2
resources:
requests:
memory: 400Mi
ruleNamespaceSelector: {}
ruleSelector: {}
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: 2.44.0
remoteWrite:
- url: "https://portal.opsramp.com/prom/push"
authorization:
type: Bearer
credentials:
name: remote-write-opsramp # give name of the k8s secret
key: token # give the name of key from which the token needs to be taken
headers:
tenantId: jgowioer-b492-732d-a851-0d603769be80
integrationUUID: INTG-137e9ab3-cccc-bbbb-dddd-2723d7aaaaaa
Prometheus Kubernetes Secret:
apiVersion: v1
kind: Secret
metadata:
name: remote-write-opsramp
type: Opaque
data:
token: ahsd78fyasdifhasuidfhasdifuh88ha
Constraints
The number of active series per metric per client is 50000. You can avoid the limit by configuring Prometheus to filter metrics. For example, use the following configuration to limit apiserver_request_duration_seconds_bucket
and etcd_request_duration_seconds_bucket
metrics:
remoteWrite:
- url: http://portal.opsramp.net/prom/push
authorization:
type: Bearer
credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
headers:
tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
writeRelabelConfigs:
- action: drop
regex: apiserver_request_duration_seconds_bucket
sourceLabels:
- __name__
- action: drop
regex: etcd_request_duration_seconds_bucket
sourceLabels:
- __name__
Configuration
Remote Write Tuning
Below are the recommended Prometheus Remote-Write Configuration tuning:
Configuration Parameter | Description |
---|---|
capacity | How many samples are queued in memory per shard before blocking reading from the WAL. When the WAL is blocked, samples cannot be appended to any shards and all throughput ceases. |
max_shards | The maximum number of shards, or parallelism, Prometheus uses for each remote write queue. |
min_shards | The minimum number of shards used by Prometheus, which is the number of shards used when remote write starts. |
max_samples_per_send | The maximum number of samples sent per batch, which can be adjusted depending on the back end in use. |
batch_send_deadline | The maximum time interval between sends for a single shard, in seconds. |
min_backoff | The minimum time to wait before retrying a failed request, in seconds. |
max_backoff | The maximum time to wait before retrying a failed request, in seconds. |
See: https://prometheus.io/docs/practices/remote_write/ for more details.
Configuration examples to filter metrics pushed to remote write
Prometheus can be configured to filter the metrics to be pushed to a remote endpoint like the agent. See the Prometheus configuration remote_write property documentation for more information.
Drop samples example
The following writeRelabelConfigs
configuration drops samples with a metric name that starts with go
:
remote_write:
- url: http://portal.opsramp.net/prom/push
authorization:
type: Bearer
credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
headers:
tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
writeRelabelConfigs:
- sourceLabels: [__name__]
regex: 'go_.*'
action: drop
Keep samples example
The following writeRelabelConfigs
configuration keeps samples with a metric name that starts with go
:
remote_write:
- url: http://portal.opsramp.net/prom/push
authorization:
type: Bearer
credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
headers:
tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
writeRelabelConfigs:
- sourceLabels: [__name__]
regex: 'go_.*'
action: keep
Replace samples example
You can also change a label using the replace action before pushing it to a remote endpoint. The following example replaces samples with a metric name that starts with go
with the name sample_go_label
:
remote_write:
- url: http://portal.opsramp.net/prom/push
authorization:
type: Bearer
credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
headers:
tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
writeRelabelConfigs:
- sourceLabels: [__name__]
regex: 'go_.*'
action: replace
targetLabels: sample_go_label