Skip to main content

Sum rate

test:axiom_http_requests_total
| where path == /.*(elastic\/_bulk|ingest|(?:v1\/(traces|logs|metrics))).*/
| where code < 400
| align to 5m using rate
| group by method, path, code using sum

Average over time

test-with-minus.com:axiomdb_transport_ingest_pressure as snot
| where time_window == "1m"
| where service == /axiomdb-[a-f]/
| group using max
| as cookie
| map < 0.4
| align to 7d using avg
| as cake

Histogram

test:axiom_http_request_duration_seconds_bucket
| where code == /[123]../
| bucket by method, path to 5m using interpolate_histogram(0.90, max, 0.99)

Filtered histogram

axiom-dev.metrics:axiomdb_transport_request_duration_seconds_bucket
| filter status == /[123..]/
| filter handler == /^ingest$/
| filter le == /^0\.5/
| map rate
| align to 30s using avg
| group by handler, le using sum
| bucket by handler to 30s using interpolate_histogram(min, 0.5, max)

Compute

Compute error rate

{ 
    test:axiom_http_request_duration_seconds_count
    | where container == /axiom-ingest-pod|axiom-api-pod|axiom-login-pod|axiom-integrations-pod/
    | where path == /.*(elastic\/_bulk|ingest|(?:v1\/(traces|logs|metrics))).*/
    | where code >= 500 // note since things go through prom we currently store it as string but this is how we'd want this query to IO_WORKER
    | map interpolate::linear
    | map rate
    | align to 5m using avg
    | group using sum;
    test:axiom_http_request_duration_seconds_count
    | where container == /axiom-ingest-pod|axiom-api-pod|axiom-login-pod|axiom-integrations-pod/
    | where path == ".*(elastic/_bulk|ingest|(?:v1/(traces|logs|metrics))).*"
    | map interpolate::linear
    | map rate
    | align to 5m using avg
    | group using sum
}
| compute error_rate using /

Compute running average

{
    test:metric_a
    | align to 1m over 5m using avg; // emit every 1m running values over 5m
    test:metric_a
    | align to 1m over 1h using avg // emit every 1m running values over 5m
}
| compute avg_over_time using /

Service Level Objectives (SLO)

test-with-minus.com:axiomdb_transport_ingest_pressure
| where time_window == "1m"
| where service == /axiomdb-[a-f]/
| group using max
| map < 0.4
| align to 7d using avg

SLO histogram

test:http_server_request_duration_seconds_bucket
| bucket to 1m using interpolate_histogram(0.99)
| map < 0.4
| align to 7d using avg
I