OpenSearch Operator: Monitoring

Prometheus metrics

Every OpenSearch node exposes Prometheus metrics at /_prometheus/metrics, through the Prometheus exporter plugin. Scraping requires the metrics user, whose password the operator stores in the opensearch-metrics-basic-auth Secret in its own namespace. It's the same for all clusters, so a single ServiceMonitor can scrape every cluster.

Index level metrics are disabled by default to keep their high cardinality labels out of Prometheus. Set prometheus.indices: true in a cluster's config to enable them.

On Reclaim the Stack

Enabling the operator component also installs a ServiceMonitor for all OpenSearch clusters and the Grafana dashboard, see Installation. There's nothing else to set up.

On any Kubernetes cluster

Add a ServiceMonitor in the operator's namespace, where it can use the operator's opensearch-metrics-basic-auth Secret directly:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: opensearch
  namespace: opensearch-operator
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: opensearch-operator
      app.kubernetes.io/name: opensearch
  endpoints:
    - port: http
      scheme: http
      path: /_prometheus/metrics
      interval: 30s
      scrapeTimeout: 10s
      basicAuth:
        username:
          name: opensearch-metrics-basic-auth
          key: username
        password:
          name: opensearch-metrics-basic-auth
          key: password

Prometheus has to select ServiceMonitors outside its own namespace and without its release label. With kube-prometheus-stack, set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues: false, as Reclaim the Stack does.

The selector only matches the headless service of each cluster, which includes pods that aren't ready, so every pod is scraped exactly once. The client service deliberately lacks the app.kubernetes.io/name label: a selector which also matched it would scrape every pod twice.

Grafana dashboard

An OpenSearch dashboard shows the health, nodes, shards and thread pools of each cluster, its indexing and search rates and latencies, caches, JVM heap and garbage collection, and the CPU, memory and disk usage of its nodes.

On Reclaim the Stack it comes with the operator component. Elsewhere, import examples/opensearch-grafana-dashboard.json from the operator repository in Grafana.

Alerts

A set of Prometheus alerts covers all OpenSearch clusters:

On Reclaim the Stack, enable the alerts component in platform/opensearch-operator/kustomization.yaml, see Installation. The alerts show in Prometheus, and notify you through Alertmanager once it's enabled in platform/kube-prometheus-stack/kustomization.yaml. Elsewhere, take the PrometheusRule from platform/opensearch-operator/alerts/prometheus-rules.yaml in the get-started repository.

Cluster status

The status of each OpenSearch resource shows its health, nodes and version, what the operator is doing, and Reconciled and Ready conditions, see Status. The operator also emits events for notable steps and problems, eg. ReconcileFailed, RollingRestartBlocked or SnapshotRepositoriesFailed Warnings:

kubectl get opensearch -A
kubectl describe opensearch <cluster-name>
kubectl get events --field-selector involvedObject.kind=OpenSearch

Argo CD

With the Argo CD health check installed, an application whose OpenSearch cluster failed to apply its latest change, or is red, unreachable or rejects the operator's credentials, shows as Degraded in Argo CD, with the reason in the health message.

The operator itself

The operator logs to stdout, eg. kubectl logs -n opensearch-operator deployment/opensearch-operator. With SENTRY_DSN set, errors in the operator itself are reported to Sentry, see Configuration.