Rules API
Observatorium is featured with a multi-tenant API that enables tenants to write and read Prometheus recording and alerting rules.
The API is defined using the OpenAPI specification format.
The goal is to enable tenants to create, modify and access their own rules.
Usage
Note: All requests to the endpoints below are tenant scoped; this means that each request requires a valid authentication token in the Authorization
header.
Create rules
PUT /api/v1/rules/raw
Set rules for a tenant.
Example request
curl -X PUT --data-binary @alerting-rule.yaml --header "Content-Type: application/yaml" http://<observatorium-api-url>/api/metrics/v1/<tenant>/api/v1/rules/raw
Where:
alerting-rule.yaml
is a YAML file containing the desired rule definition. It can contain recording rules, alerting rules or both.- Note: Each time a PUT request is made to the
/api/v1/rules/raw
endpoint, the rules contained in the request will overwrite all other rules for that tenant.
- Note: Each time a PUT request is made to the
<observatorium-api-url>
is the URL where the Observatorium API is hosted.<tenant>
is the tenant name
Example of a rule.yaml file
The rule.yaml
should be defined following the Observatorium OpenAPI specification. The syntax is based on the Prometheus recording and alerting rules syntax.
Example of a rule.yaml
file containing an alerting rule:
groups:
- name: test-alerting-rule
interval: 30s
rules:
- alert: HighRequestLatency
expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
for: 10m
labels:
severity: page
annotations:
summary: High request latency
Example response
successfully updated rules file
Status Code | Description |
---|---|
200 | Successfully updated rules file. |
401 | Error finding tenant/tenant ID. |
500 | A server side error happened while trying to create rules or while trying to write the response body. |
List rules
GET /api/v1/rules/raw
List rules for a tenant.
Example request
curl http://<observatorium-api-url>/api/metrics/v1/<tenant>/api/v1/rules/raw
Where:
<observatorium-api-url>
is the URL where Observatorium API is hosted.<tenant>
is the tenant name
Example response
groups:
- interval: 30s
name: test-alerting-rule
rules:
- alert: HighRequestLatency
annotations:
summary: High request latency
expr: job:request_latency_seconds:mean5m{job="myjob",tenant_id="1234"}
> 0.5
for: 10m
labels:
severity: page
tenant_id: 1234
The response format is in application/yaml
.
Note: In the current implementation from the Rules OpenAPI specification in the Observatorium API, to better validate tenant rules, the label tenant_id
is enforced in the read path:
- In the
labels
field and - In the metrics that are present in the expression defined in the
expr
field.
Status Code | Description |
---|---|
200 | Successfully listed rules. |
401 | Error finding tenant/tenant ID. |
500 | A server side error happened while trying to list rules. |
Difference between /api/v1/rules/raw and /api/v1/rules endpoints
Note that the /api/v1/rules/raw
endpoint differs from /api/v1/rules
endpoint:
/api/v1/rules/raw
supportsGET
andPUT
requests, as described above. It refers only to the rules that were defined by the tenant that was authorized to use the endpoint./api/v1/rules
supportsGET
requests and is the endpoint that is proxied by the Observatorium API to the read endpoint (in this case, Thanos Querier). It lists rules that are currently being evaluated for the tenant.