Rules

cpu-node

13.667s ago

2.253ms

Rule State Error Last Evaluation Evaluation Time
record: job_instance_mode:node_cpu_seconds:avg_rate5m expr: avg by(job, instance, mode) (rate(node_cpu_seconds_total[5m])) ok 13.667s ago 2.248ms

sloth-slo-alerts-myservice-requests-availability

9.272s ago

2.294ms

Rule State Error Last Evaluation Evaluation Time
alert: MyServiceHighErrorRate expr: (max without(sloth_window) (slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate1h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432))) or (max without(sloth_window) (slo:sli_error:ratio_rate30m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (6 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (6 * 0.0009999999999999432))) labels: category: availability routing_key: myteam severity: pageteam sloth_severity: page annotations: summary: High error rate on 'myservice' requests responses title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. ok 9.273s ago 1.237ms
alert: MyServiceHighErrorRate expr: (max without(sloth_window) (slo:sli_error:ratio_rate2h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (3 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate1d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (3 * 0.0009999999999999432))) or (max without(sloth_window) (slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) and max without(sloth_window) (slo:sli_error:ratio_rate3d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} > (1 * 0.0009999999999999432))) labels: category: availability severity: slack slack_channel: '#alerts-myteam' sloth_severity: ticket annotations: summary: High error rate on 'myservice' requests responses title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. ok 9.272s ago 1.041ms

sloth-slo-meta-recordings-myservice-requests-availability

12.364s ago

379.2us

Rule State Error Last Evaluation Evaluation Time
record: slo:objective:ratio expr: vector(0.9990000000000001) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 12.364s ago 79.41us
record: slo:error_budget:ratio expr: vector(1 - 0.9990000000000001) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 12.364s ago 39.54us
record: slo:time_period:days expr: vector(30) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 12.365s ago 28.5us
record: slo:current_burn_rate:ratio expr: slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} / on(sloth_id, sloth_slo, sloth_service) group_left() slo:error_budget:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 12.365s ago 90.62us
record: slo:period_burn_rate:ratio expr: slo:sli_error:ratio_rate30d{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} / on(sloth_id, sloth_slo, sloth_service) group_left() slo:error_budget:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 12.365s ago 61.62us
record: slo:period_error_budget_remaining:ratio expr: 1 - slo:period_burn_rate:ratio{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"} labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability tier: "2" ok 12.365s ago 42.87us
record: sloth_slo_info expr: vector(1) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_mode: cli-gen-prom sloth_objective: "99.9" sloth_service: myservice sloth_slo: requests-availability sloth_spec: prometheus/v1 sloth_version: v0.11.0 tier: "2" ok 12.366s ago 29.86us

sloth-slo-sli-recordings-myservice-requests-availability

10.608s ago

5.548ms

Rule State Error Last Evaluation Evaluation Time
record: slo:sli_error:ratio_rate5m expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[5m]))) / (sum(rate(flask_http_request_total{job="Python test application"}[5m]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 5m tier: "2" ok 10.608s ago 883us
record: slo:sli_error:ratio_rate30m expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[30m]))) / (sum(rate(flask_http_request_total{job="Python test application"}[30m]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 30m tier: "2" ok 10.608s ago 549.8us
record: slo:sli_error:ratio_rate1h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[1h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[1h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 1h tier: "2" ok 10.607s ago 504.8us
record: slo:sli_error:ratio_rate2h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[2h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[2h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 2h tier: "2" ok 10.607s ago 684.4us
record: slo:sli_error:ratio_rate6h expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[6h]))) / (sum(rate(flask_http_request_total{job="Python test application"}[6h]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 6h tier: "2" ok 10.607s ago 595.9us
record: slo:sli_error:ratio_rate1d expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[1d]))) / (sum(rate(flask_http_request_total{job="Python test application"}[1d]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 1d tier: "2" ok 10.607s ago 622.8us
record: slo:sli_error:ratio_rate3d expr: (sum(rate(flask_http_request_total{job="Python test application",status=~"(5..|429)"}[3d]))) / (sum(rate(flask_http_request_total{job="Python test application"}[3d]))) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 3d tier: "2" ok 10.606s ago 623.1us
record: slo:sli_error:ratio_rate30d expr: sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"}[30d]) / ignoring(sloth_window) count_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability",sloth_service="myservice",sloth_slo="requests-availability"}[30d]) labels: owner: myteam repo: myorg/myservice sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 30d tier: "2" ok 10.606s ago 1.048ms