A static alert — p99_latency > 500ms — has two failure modes, and you’ve almost certainly hit both. It’s too noisy during your Tuesday 9am traffic peak, when 500ms is normal load-driven latency, not a problem. And it’s too slow during a Sunday 3am degradation, when 500ms would be a genuine 5x regression against the near-idle baseline that hour actually has — but the same fixed threshold either fires constantly or never fires meaningfully, because it has no concept of what’s normal for that specific hour.
This is a seasonality problem before it’s a machine learning problem: normal latency has a daily and weekly shape, and a threshold that ignores that shape is comparing today’s 3am to Tuesday’s 9am. The fix isn’t inherently exotic — it’s decomposing the signal into trend/seasonal/residual components and scoring anomalies against the residual, optionally combined with a multivariate model (Isolation Forest) that catches correlated degradations a single-metric threshold can’t see at all.
Why single-metric thresholds miss correlated degradations
A service can have normal p99 latency, normal error rate, and normal request volume individually, while the combination — slightly elevated latency plus slightly elevated error rate plus a slight request rate dip, all simultaneously — is actually the earliest signal of a degrading downstream dependency. Isolation Forest, an unsupervised anomaly detection algorithm, is built for exactly this: it scores how easily a data point can be “isolated” from the rest of the dataset via random feature splits, and points that are unusual across a combination of features isolate faster than points that are unusual on just one.
Stage 1: feature engineering from raw telemetry
Pull rolling-window aggregates per service, not raw request-level events — the model should reason about system state over a window (e.g., 1 minute), not individual requests:
import pandas as pd
import numpy as np
def build_feature_windows(raw_metrics: pd.DataFrame, window: str = "1min") -> pd.DataFrame:
"""raw_metrics has columns: timestamp, service, latency_ms, status_code"""
raw_metrics["is_error"] = raw_metrics["status_code"] >= 500
grouped = raw_metrics.set_index("timestamp").groupby("service").resample(window)
features = grouped.agg(
p50_latency=("latency_ms", lambda x: x.quantile(0.5)),
p99_latency=("latency_ms", lambda x: x.quantile(0.99)),
error_rate=("is_error", "mean"),
request_count=("latency_ms", "count"),
).reset_index()
return features.dropna()
Aggregating to p50 and p99 separately (rather than just mean latency) matters — a mean can hide a bimodal distribution where most requests are fast and a growing minority are very slow, exactly the pattern that precedes a cascading failure.
Stage 2: seasonal decomposition to remove the daily/weekly shape
Before scoring anomalies, strip out the predictable seasonal pattern so the model scores deviation from expected-for-this-hour, not deviation from the dataset’s overall average:
from statsmodels.tsa.seasonal import STL
def detrend_series(series: pd.Series, period: int) -> pd.Series:
"""period = number of samples per seasonal cycle (e.g. 1440 for daily cycle at 1min resolution)"""
stl_result = STL(series, period=period, robust=True).fit()
return stl_result.resid # what's left after removing trend + seasonality
def add_residual_features(features: pd.DataFrame, period_minutes: int = 1440) -> pd.DataFrame:
for service, group in features.groupby("service"):
idx = group.index
for col in ["p99_latency", "error_rate", "request_count"]:
if len(group) >= 2 * period_minutes: # STL needs at least 2 full cycles
features.loc[idx, f"{col}_resid"] = detrend_series(group[col], period_minutes).values
else:
features.loc[idx, f"{col}_resid"] = group[col] - group[col].rolling(60, min_periods=1).mean()
return features
robust=True on the STL call matters specifically for observability data — a robust decomposition down-weights the influence of past outliers (previous incidents) on the fitted seasonal shape, so an old incident doesn’t get baked into what the model considers “normal” going forward. The fallback path (simple rolling-mean subtraction) matters too: STL needs at least two full seasonal cycles of history, so a newly onboarded service without 2+ days of data yet needs a simpler baseline until it accumulates enough history.
Stage 3: multivariate anomaly scoring with Isolation Forest
With seasonality removed, feed the residuals — not the raw values — into Isolation Forest, so the model is scoring “how unusual is this deviation from expected,” not “how unusual is this raw number”:
from sklearn.ensemble import IsolationForest
FEATURE_COLS = ["p99_latency_resid", "error_rate_resid", "request_count_resid"]
def fit_anomaly_model(features: pd.DataFrame, contamination: float = 0.01) -> IsolationForest:
model = IsolationForest(
contamination=contamination, # expected fraction of anomalous windows
n_estimators=200,
random_state=42,
)
model.fit(features[FEATURE_COLS])
return model
def score_windows(model: IsolationForest, features: pd.DataFrame) -> pd.DataFrame:
features = features.copy()
features["anomaly_score"] = model.decision_function(features[FEATURE_COLS])
features["is_anomaly"] = model.predict(features[FEATURE_COLS]) == -1
return features
contamination=0.01 is a deliberate, tunable assumption — “1% of time windows are genuinely anomalous” — and it directly controls alert volume. Set it too high and you’re back to noisy alerting; set it too low and genuine degradations get scored as normal. Tune this against your actual historical incident rate, not a default.
Stage 4: requiring persistence before alerting
A single anomalous 1-minute window is often noise — a brief GC pause, a deploy in progress, one slow request pulling up the p99. Require the anomaly to persist across consecutive windows before it becomes a page:
def find_persistent_anomalies(scored: pd.DataFrame, min_consecutive: int = 3) -> pd.DataFrame:
scored = scored.sort_values("timestamp")
scored["anomaly_streak"] = (
scored["is_anomaly"]
.groupby((~scored["is_anomaly"]).cumsum())
.cumsum()
)
return scored[scored["anomaly_streak"] >= min_consecutive]
This single change is usually what determines whether the on-call team keeps the alert enabled — three consecutive anomalous 1-minute windows (a 3-minute sustained degradation) is a fundamentally different signal than one noisy minute, and treating them the same is how ML-based alerting earns a reputation for being unreliable.
Stage 5: explainability — which metric actually drove the flag
An anomaly score alone tells an on-call engineer that something is wrong, not what. Add a simple per-feature deviation ranking so the alert itself points at the likely cause:
def explain_anomaly(row: pd.Series, feature_stats: dict) -> list[str]:
contributions = []
for col in FEATURE_COLS:
z_score = abs(row[col] - feature_stats[col]["mean"]) / feature_stats[col]["std"]
contributions.append((col, z_score))
contributions.sort(key=lambda x: x[1], reverse=True)
return [f"{col} (z={z:.1f})" for col, z in contributions[:2]]
An alert that reads “anomaly detected in checkout-service: driven by error_rate_resid (z=4.2), p99_latency_resid (z=2.1)” gets investigated in seconds. One that just says “anomaly score: -0.31” gets a shrug and, eventually, a muted channel.
Backtesting against known incidents before trusting it in production
Before this replaces or supplements existing alerting, validate it against your actual incident history — did the model flag the windows around past real incidents, and how far in advance:
def backtest_against_incidents(scored: pd.DataFrame, known_incidents: list[dict]) -> dict:
results = {}
for incident in known_incidents:
window = scored[
(scored["service"] == incident["service"])
& (scored["timestamp"] >= incident["start"] - pd.Timedelta("30min"))
& (scored["timestamp"] <= incident["start"])
]
first_flag = window[window["is_anomaly"]]["timestamp"].min()
lead_time = (incident["start"] - first_flag).total_seconds() / 60 if pd.notna(first_flag) else None
results[incident["id"]] = {"lead_time_minutes": lead_time, "detected": lead_time is not None}
return results
If this backtest shows the model catches most known incidents with meaningful lead time (even 5-10 minutes ahead of the static-threshold alert firing), that’s the concrete evidence to justify running it in production alongside — not instead of — your existing alerts, at least initially.
The takeaway
The AI in this pipeline isn’t doing anything mysterious — it’s scoring multivariate deviation after the predictable daily/weekly pattern has been explicitly removed, which a naive single-metric threshold structurally can’t do. The unglamorous parts (seasonal decomposition, persistence filtering, explainability) are what determine whether this becomes a trusted signal your on-call team keeps enabled, or a source of unexplained pages that gets muted within a month like every noisy alerting system before it.