Rules

cpu-node

9.68s ago

1.817ms

Rule State Error Last Evaluation Evaluation Time
record: job_instance_mode:node_cpu_seconds:avg_rate5m expr: avg by(job, instance, mode) (rate(node_cpu_seconds_total[5m])) ok 9.681s ago 1.813ms

sloth-slo-alerts-myservice-requests-availability

5.285s ago

2.392ms

Rule State Error Last Evaluation Evaluation Time
alert: MyServiceHighErrorRate expr: (max without(sloth_window) (slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate1h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432))) or (max without(sloth_window) (slo:sli_error:ratio_rate30m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (6 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (6 * 0.0009999999999999432))) labels: category: availability routing_key: myteam severity: pageteam sloth_severity: page annotations: summary: High error rate on 'myservice' requests responses title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. ok 5.285s ago 1.322ms
alert: MyServiceHighErrorRate expr: (max without(sloth_window) (slo:sli_error:ratio_rate2h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (3 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate1d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (3 * 0.0009999999999999432))) or (max without(sloth_window) (slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate3d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (1 * 0.0009999999999999432))) labels: category: availability severity: slack slack_channel: '#alerts-myteam' sloth_severity: ticket annotations: summary: High error rate on 'myservice' requests responses title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. ok 5.285s ago 1.051ms

sloth-slo-meta-recordings-myservice-requests-availability

8.376s ago

2.214ms

Rule State Error Last Evaluation Evaluation Time
record: slo:objective:ratio expr: vector(0.9990000000000001) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 8.376s ago 332.2us
record: slo:error_budget:ratio expr: vector(1 - 0.9990000000000001) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 8.376s ago 218.1us
record: slo:time_period:days expr: vector(30) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 8.376s ago 182.5us
record: slo:current_burn_rate:ratio expr: slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} / on(sloth_id, sloth_slo, sloth_service) group_left() slo:error_budget:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 8.376s ago 476.2us
record: slo:period_burn_rate:ratio expr: slo:sli_error:ratio_rate30d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} / on(sloth_id, sloth_slo, sloth_service) group_left() slo:error_budget:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 8.376s ago 486us
record: slo:period_error_budget_remaining:ratio expr: 1 - slo:period_burn_rate:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 8.376s ago 278.1us
record: sloth_slo_info expr: vector(1) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_mode: cli-gen-prom sloth_objective: "99.9" sloth_service: myservice sloth_slo: requests-availability sloth_spec: prometheus/v1 sloth_version: v0.11.0 tier: "2" ok 8.376s ago 209.2us

sloth-slo-sli-recordings-myservice-requests-availability

6.62s ago

5.781ms

Rule State Error Last Evaluation Evaluation Time
record: slo:sli_error:ratio_rate5m expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[5m]))) / (sum(rate(flask_http_request_total{job="Python test application"}[5m]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 5m tier: "2" ok 6.62s ago 772.1us
record: slo:sli_error:ratio_rate30m expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[30m]))) / (sum(rate(flask_http_request_total{job="Python test application"}[30m]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 30m tier: "2" ok 6.619s ago 736us
record: slo:sli_error:ratio_rate1h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[1h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[1h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 1h tier: "2" ok 6.619s ago 646.9us
record: slo:sli_error:ratio_rate2h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[2h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[2h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 2h tier: "2" ok 6.618s ago 670.6us
record: slo:sli_error:ratio_rate6h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[6h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[6h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 6h tier: "2" ok 6.617s ago 698.9us
record: slo:sli_error:ratio_rate1d expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[1d]))) / (sum(rate(flask_http_request_total{job="Python test application"}[1d]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 1d tier: "2" ok 6.617s ago 590.5us
record: slo:sli_error:ratio_rate3d expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[3d]))) / (sum(rate(flask_http_request_total{job="Python test application"}[3d]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 3d tier: "2" ok 6.616s ago 569.2us
record: slo:sli_error:ratio_rate30d expr: sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"}[30d]) / ignoring(sloth_window) count_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"}[30d]) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 30d tier: "2" ok 6.616s ago 1.054ms