Rules API

Observatorium is featured with a multi-tenant API that enables tenants to write and read Prometheus recording and alerting rules.

The API is defined using the OpenAPI specification format.

The goal is to enable tenants to create, modify and access their own rules.

Usage#

Note: All requests to the endpoints below are tenant scoped; this means that each request requires a valid authentication token in the Authorization header.

Create rules#

PUT /api/v1/rules/raw

Set rules for a tenant.

Example request#

curl -X PUT --data-binary @alerting-rule.yaml --header "Content-Type: application/yaml" http://<observatorium-api-url>/api/metrics/v1/<tenant>/api/v1/rules/raw

Where:

• alerting-rule.yaml is a YAML file containing the desired rule definition. It can contain recording rules, alerting rules or both.
  • Note: Each time a PUT request is made to the /api/v1/rules/raw endpoint, the rules contained in the request will overwrite all other rules for that tenant.
• <observatorium-api-url> is the URL where the Observatorium API is hosted.
• <tenant> is the tenant name

Example of a rule.yaml file#

The rule.yaml should be defined following the Observatorium OpenAPI specification. The syntax is based on the Prometheus recording and alerting rules syntax.

Example of a rule.yaml file containing an alerting rule:

groups:
  - name: test-alerting-rule
    interval: 30s
    rules:
    - alert: HighRequestLatency
      expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
      for: 10m
      labels:
        severity: page
      annotations:
        summary: High request latency

Example response#

successfully updated rules file

List rules#

GET /api/v1/rules/raw

List rules for a tenant.

Example request#

curl http://<observatorium-api-url>/api/metrics/v1/<tenant>/api/v1/rules/raw

Where:

• <observatorium-api-url> is the URL where Observatorium API is hosted.
• <tenant> is the tenant name

Example response#

groups:
- interval: 30s
  name: test-alerting-rule
  rules:
  - alert: HighRequestLatency
    annotations:
      summary: High request latency
    expr: job:request_latency_seconds:mean5m{job="myjob",tenant_id="1234"}
      > 0.5
    for: 10m
    labels:
      severity: page
      tenant_id: 1234

The response format is in application/yaml.

Note: In the current implementation from the Rules OpenAPI specification in the Observatorium API, to better validate tenant rules, the label tenant_id is enforced in the read path:

• In the labels field and
• In the metrics that are present in the expression defined in the expr field.

Difference between /api/v1/rules/raw and /api/v1/rules endpoints#

Note that the /api/v1/rules/raw endpoint differs from /api/v1/rules endpoint:

• /api/v1/rules/raw supports GET and PUT requests, as described above. It refers only to the rules that were defined by the tenant that was authorized to use the endpoint.
• /api/v1/rules supports GET requests and is the endpoint that is proxied by the Observatorium API to the read endpoint (in this case, Thanos Querier). It lists rules that are currently being evaluated for the tenant.