Rules

cpu-node

2.128s ago

924.7us

Rule State Error Last Evaluation Evaluation Time
record: job_instance_mode:node_cpu_seconds:avg_rate5m expr: avg by(job, instance, mode) (rate(node_cpu_seconds_total[5m])) ok 2.128s ago 920.7us

sloth-slo-alerts-myservice-requests-availability

12.732s ago

612.7us

Rule State Error Last Evaluation Evaluation Time
alert: MyServiceHighErrorRate expr: (max without(sloth_window) (slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate1h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432))) or (max without(sloth_window) (slo:sli_error:ratio_rate30m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (6 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (6 * 0.0009999999999999432))) labels: category: availability routing_key: myteam severity: pageteam sloth_severity: page annotations: summary: High error rate on 'myservice' requests responses title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. ok 12.732s ago 351.9us
alert: MyServiceHighErrorRate expr: (max without(sloth_window) (slo:sli_error:ratio_rate2h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (3 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate1d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (3 * 0.0009999999999999432))) or (max without(sloth_window) (slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate3d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (1 * 0.0009999999999999432))) labels: category: availability severity: slack slack_channel: '#alerts-myteam' sloth_severity: ticket annotations: summary: High error rate on 'myservice' requests responses title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. ok 12.732s ago 256.5us

sloth-slo-meta-recordings-myservice-requests-availability

824ms ago

483.1us

Rule State Error Last Evaluation Evaluation Time
record: slo:objective:ratio expr: vector(0.9990000000000001) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 824ms ago 96.43us
record: slo:error_budget:ratio expr: vector(1 - 0.9990000000000001) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 824ms ago 41.52us
record: slo:time_period:days expr: vector(30) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 824ms ago 26.01us
record: slo:current_burn_rate:ratio expr: slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} / on(sloth_id, sloth_slo, sloth_service) group_left() slo:error_budget:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 824ms ago 92.91us
record: slo:period_burn_rate:ratio expr: slo:sli_error:ratio_rate30d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} / on(sloth_id, sloth_slo, sloth_service) group_left() slo:error_budget:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 824ms ago 131us
record: slo:period_error_budget_remaining:ratio expr: 1 - slo:period_burn_rate:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 823ms ago 54.16us
record: sloth_slo_info expr: vector(1) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_mode: cli-gen-prom sloth_objective: "99.9" sloth_service: myservice sloth_slo: requests-availability sloth_spec: prometheus/v1 sloth_version: v0.11.0 tier: "2" ok 823ms ago 34.41us

sloth-slo-sli-recordings-myservice-requests-availability

14.064s ago

5.76ms

Rule State Error Last Evaluation Evaluation Time
record: slo:sli_error:ratio_rate5m expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[5m]))) / (sum(rate(flask_http_request_total{job="Python test application"}[5m]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 5m tier: "2" ok 14.064s ago 739.6us
record: slo:sli_error:ratio_rate30m expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[30m]))) / (sum(rate(flask_http_request_total{job="Python test application"}[30m]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 30m tier: "2" ok 14.064s ago 557.9us
record: slo:sli_error:ratio_rate1h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[1h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[1h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 1h tier: "2" ok 14.063s ago 639.2us
record: slo:sli_error:ratio_rate2h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[2h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[2h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 2h tier: "2" ok 14.063s ago 604.9us
record: slo:sli_error:ratio_rate6h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[6h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[6h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 6h tier: "2" ok 14.062s ago 757.9us
record: slo:sli_error:ratio_rate1d expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[1d]))) / (sum(rate(flask_http_request_total{job="Python test application"}[1d]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 1d tier: "2" ok 14.061s ago 633.7us
record: slo:sli_error:ratio_rate3d expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[3d]))) / (sum(rate(flask_http_request_total{job="Python test application"}[3d]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 3d tier: "2" ok 14.061s ago 672.7us
record: slo:sli_error:ratio_rate30d expr: sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"}[30d]) / ignoring(sloth_window) count_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"}[30d]) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 30d tier: "2" ok 14.06s ago 1.115ms