From 20-second endpoints to a 0.5 to 0.7 s p95
I took the slowest endpoints from 20 s to under 2 s, removed every latency spike for ~270 users, and cut slow requests by more than 80%, with no new errors.
Outcome
- 20 s to <2 s slowest endpoints
- −80%+ requests over 3 s
- 0 p95 spikes over 2 s
Stack
Ask me about this
The problem
Some endpoints took 20 seconds. Business-hours p95 sat between 1.0 and 1.7 seconds, the API had up to 13 windows a day where p95 went over 2 seconds, and health checks failed with up to 30 503s a day, which made the autoscaler churn instances.
What I did
- Took the slowest endpoints from 20 s to under 2 s by eliminating N+1 queries and tuning database queries and async I/O.
- Tuned the API and added a second uvicorn worker: windows with p95 over 2 s fell from up to 13 a day to 0, and business-hour p95 went from 1.0 to 1.7 s down to 0.5 to 0.7 s.
- Guarded every change with a pytest suite of 1,400+ tests, so the speed did not cost correctness.
Illustration of the pattern. In production: the slowest endpoints went from 20 s to under 2 s by removing N+1 queries and tuning the SQL and async I/O.
The result
- Requests over 3 s down by more than 80%: from up to 3,900 a day to 469.
- Health-check 503s from up to 30 a day to 0, and autoscale instance churn cut in half.
- No new errors.
13windows a day over 2 s, at most
1.0 to 1.7 s0.5 to 0.7 sbusiness-hours p95
Illustration of the shape. The counts are real: windows with p95 over 2 s fell from up to 13 a day to 0, for ~270 users.
When things break: a post-incident review
I wrote the post-incident review for a 65-minute production outage caused by Key Vault references resolving empty after a slot swap: root cause, a recovery runbook, and 12 corrective actions.
Measured
Slowest API endpoints
KinlyRemoved N+1 queries, tuned the database queries and async I/O, behind a 1,400+ test pytest suite.
Business-hours p95 latency
KinlyTuned the API and added a second uvicorn worker, for ~270 users.
Requests slower than 3 s
KinlyCut by more than 80%, with no new errors.
Health-check 503s
KinlyAnd autoscale instance churn cut by half.
Windows with p95 over 2 s
KinlyLatency spikes eliminated for ~270 users.