pint

pint is a Prometheus rule linter/validator.

Requirements

pint will run checks on Prometheus alerting & recording rules to detect potential problems with those rules. Some checks rely only on the rule itself and can be run “offline” - without talking to any Prometheus server. You can run pint in “offline” if you:

Don’t pass any configuration file to pint.
You pass configuration file to pint that doesn’t contain any prometheus definition.
You pass --offline flag to pint command.

Most checks included in pint will require sending queries to a running Prometheus server where those rules are, or would be, deployed. Those checks are enabled if you pass a configuration file to pint that includes at least one prometheus block. Checks might use various Prometheus HTTP API endpoints to retrieve extra information, for example Prometheus configuration or metrics metadata. If you run pint against a different service, like Thanos some checks might return problems due to API call errors, since not all Prometheus HTTP APIs are supported by it. In that case, you might want to disable failing checks in the pint configuration file.

IMPORTANT pint is a tool we wrote to work with our Prometheus deployment. It’s not intended to be used with other services that offer partial compatibility with Prometheus, there are NO PLANS to add support for any other services. The only reason we would add support for other systems is if we started to use them ourselves.

Usage

There are three modes it works in:

CI PR linting
Ad-hoc linting of a selected files or directories
A daemon that continuously checks selected files or directories and expose metrics describing all discovered problems.

Pull Requests

Run it with pint ci. Git is currently the only supported VCS.

When pint ci runs it will find all files in the current working directory and try to parse them as Prometheus rules. Then it will look for all commits on the current branch that are not present in the parent branch and to decide which rules were modified. Checks are run only on modified rules but they require the full list of all rules to find any cross-rule dependencies.

Running pint ci doesn’t require any configuration but it’s recommended to add a pint config file with ci section containing at least the include option. This will ensure that pint validates only Prometheus rules and ignores other files.

Results can optionally be reported as comments on a pull request when using one of supported platforms:

Exit code will be one (1) if any issues were detected with severity Bug or higher. This permits running pint in your CI system whilst at the same you will get detailed reports on your source control system.

If any commit on the PR contains [skip ci] or [no ci] somewhere in the commit message then pint will skip running all checks.

GitHub Actions

The easiest way of using pint with GitHub Actions is by using prymitive/pint-action. Here’s an example workflow:

name: pint

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  pint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run pint
        uses: prymitive/pint-action@v1
        with:
          token: ${{ github.token }}
          # directory containing Prometheus rules
          workdir: 'rules'

To customise pint checks create a .pint.hcl file in the root of your repository. See Configuration for a description of all options.

When pint runs checks after a push to a branch (for example after a merge), then it will pass workdir option to pint lint, which means that all files inside rules directory will be checked.

Ad-hoc

Check specified files and report any found issue. You can pass directory paths and use glob patterns as arguments to select files for checking.

You can lint selected files:

pint lint rules.yml

or directories:

pint lint path/to/dir

or both:

pint lint path/to/dir file.yml path/file.yml path/dir

Using glob patterns:

pint lint path/*.yml path/*.yaml

Watch mode

Run pint as a daemon in watch mode where it continuously checks all rules found in selected files and exposes metrics about found problems.

By default it will start a HTTP server on port 8080 and run all checks every 10 minutes. This can be customised by passing extra flags to the watch command. Run pint watch -h to see all available flags.

Currently supported HTTP paths:

/health - static endpoint for liveness probes.
/metrics - returns Prometheus metrics, see below.

Manually selecting files and directories

You can tell it to continuously test specific files or directories:

pint watch glob $GLOB_1 $GLOB_2 ... $GLOB_N

Example:

pint watch glob /etc/prometheus/rules-*.yml /etc/prometheus/rules.d

If provide a config file for pint with some Prometheus server definitions then pint will also run “online” checks for it to, for example, ensure all time series used inside your alerting rules are still present. Example config:

prometheus "local" {
  uri = "http://localhost:9090"
}

Getting list of files to check from Prometheus

You can also point pint directly at a Prometheus server from the config file. On every iteration, before starting any checks, pint will query Prometheus API to get the current value of rule_files Prometheus config option and then run checks on all matching files. This way if you test your rules against a running Prometheus instance then you don’t need to manually specify any paths or directories.

Usage:

pint watch rule_files $prometheus

Where $prometheus is the name of prometheus configuration block from pint config file.

Example:

pint watch rule_files local

Accessing watch mode metrics

Query /metrics HTTP endpoint to see all expose metrics, example with default flags:

curl -s http://localhost:8080/metrics

Or setup Prometheus scrape job:

scrape_configs:
  - job_name: pint
    static_configs:
      - targets: ['localhost:8080']

Available metrics:

pint_problem - exported for every problem detected by pint. To avoid exposing too many metrics at once pass --max-problems flag to watch command. When this flag is set, pint will expose only up to --max-problems value number of pint_problem metrics.
pint_problems - this metric is the total number of all problems detected by pint, including those not exported due to the --max-problems flag.

The pint problem metric can include the owner label for each rule. This is useful to route alerts based on metrics to the right team. To set a rule owner add a # pint file/owner $owner comment in a file, to set an owner for all rules in that file. You can also set an owner per rule, by adding # pint rule/owner $owner comment around given rule.

Example:

# pint file/owner bob

- alert: ...
  expr: ...

# pint rule/owner alice
- alert: ...
  expr: ...

Here’s an example alert you can use for problems detected by pint:

- alert: Pint Problem Detected
  # pint_problem is only present if pint detects any problems
  # pint disable promql/series(pint_problem)
  expr: |
    sum without(instance, problem) (pint_problem) > 0
  for: 1h
  annotations:
    summary: |
      {{ with printf "pint_problem{filename='%s', name='%s', reporter='%s'}" .Labels.filename .Labels.name .Labels.reporter | query }}
        {{ . | first | label "problem" }}
      {{ end }}
    docs: "https://cloudflare.github.io/pint/checks/{{ $labels.reporter }}.html"

YAML parser

By default pint will expect all Prometheus rule files to be following the exact syntax Prometheus expects for YAML files containing recording and alerting rules. If you have Prometheus rules stored in YAML files with different YAML tree, but still retain the same set of fields, for example:

# Flat rule list
- alert: AlertName
  expr: up == 0
- record: sum:up
  expr: count(up == 1)

# Rules nested under custom tree
service:
  prometheus:
    rules:
      - alert: AlertName
        expr: up == 0
      - record: sum:up
        expr: count(up == 1)

You can still check these rules using pint, but you need to switch pint YAML parser into “relaxed” mode by adding this section to pint config file:

parser {
  relaxed = [ "my/files/*.yml" ]
}

See parser documentation for more details. “Relaxed” parser mode will load anything that can be parsed as Prometheus rule, while “strict” parser mode will fail if it reads a file that wouldn’t load cleanly as Prometheus config file.

If your repository contains other files, not only Prometheus rules, then tell pint to only check selected paths when running pint:

parser {
  include    = [ "rules/dev/.*.yml", "rules/prod/.*" ]
}

Control comments

There is a number of comments you can add to your rule files in order to change pint behaviour, some of them allow you to exclude selected files or line, see docs here for details.

There are a few requirements for any comment to be recognized by pint:

All comments must have a pint prefix.
All comments must have at least one space between # symbol and pint prefix.

Good comment examples:

# pint file/owner bob
#   pint file/owner bob

Bad comment examples:

# file/owner bob
#pint file/owner bob

Release Notes

See changelog for history of changes.

Quick start

Requirements:

Git
Go - current stable release

Steps:

Download a binary from Releases page or build from source:

git clone https://github.com/cloudflare/pint.git
cd pint
make

Run a simple syntax check on Prometheus alerting or recording rules file(s).
```
./pint lint /etc/prometheus/*.rules.yml
```
Configuration file is optional, but without it, pint will only run very basic syntax checks. See configuration for details on config syntax. By default pint will try to load configuration from .pint.hcl, you can specify a different path using --config flag:
```
./pint --config /etc/pint.hcl lint /etc/prometheus/rules/*.yml
```

There are docker images available on GitHub. Example usage:

docker run --mount=type=bind,source="$(pwd)",target=/rules,readonly ghcr.io/cloudflare/pint pint lint /rules

License

Copyright (c) 2021-2023 Cloudflare, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
    
    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

pint

Requirements

Usage

Pull Requests

GitHub Actions

Ad-hoc

Watch mode

Manually selecting files and directories

Getting list of files to check from Prometheus

Accessing watch mode metrics

YAML parser

Control comments

Release Notes

Quick start

License

Table of contents