How I Approach Debugging a Production API Issue

This is a starter draft. The process below is how I generally work; edit it freely to match how you want to present it, and replace the examples with ones from your own work.

A production API failure is a different problem from a bug on your laptop. You usually cannot attach a debugger, the data is real, other people are waiting, and the temptation is to start changing things. Changing things is the last step, not the first.

Here is the sequence I work through.

Establish what is actually failing#

Before touching anything, get specific answers to four questions:

• What is the exact symptom? "The API is down" and "POST /orders returns 500 for about one request in ten" are different problems.
• When did it start? Line that up against deployments, configuration changes, scheduled jobs and traffic patterns.
• Who is affected? All users, one tenant, one region, one client version.
• Is it getting worse? A steady error rate and a climbing one call for different urgency.

A surprising number of "mysterious" incidents resolve at this stage, because the answers point straight at a change that went out an hour ago.

Read the logs before forming a theory#

It is easy to arrive with a theory and then read the logs looking for confirmation. Read them first.

What I look for, in order:

1. The first occurrence, not the loudest one. Cascading failures bury their own cause.
2. The full stack trace, including inner exceptions — the useful line is often three frames down.
3. Correlation identifiers, so one request can be followed across services.
4. What was happening around the error: slow queries, connection pool warnings, timeouts, retries.

2026-01-12 09:14:22 WARN  [req:7f3a9c] Connection pool at 95% (19/20)
2026-01-12 09:14:24 ERROR [req:7f3a9c] Timeout expired. The timeout period elapsed
                          prior to obtaining a connection from the pool.

Those two lines together say something quite different from the error alone: this is not a broken endpoint, it is connection exhaustion. The endpoint that threw may be a victim, not the culprit.

Narrow the blast radius#

Ask what the failing requests have in common and what separates them from the ones that succeed:

• One endpoint or many?
• One HTTP method?
• Requests carrying a particular parameter, payload size or tenant?
• Only authenticated requests? Only the first request after an idle period?

Each answer removes a large region of the search space. This is the step people skip when they are in a hurry, and it is the step that saves the most time.

Reproduce it#

A defect you cannot reproduce is a defect you cannot verify as fixed. In order of preference:

1. Reproduce locally against a copy of production-shaped data.
2. Reproduce in a staging environment.
3. Reproduce in production with a read-only request that is safe to repeat.

The words production-shaped matter. Plenty of bugs only exist at real data volumes, with real null values, with the one customer whose name contains an apostrophe.

Fix at the right layer#

Once the cause is clear, there is usually a choice of where to fix it, and the tempting option is the shallow one.

Shallow fixes are sometimes the right call under incident pressure — but then they are a mitigation with a follow-up ticket attached, not a resolution.

Verify, then leave the system easier to debug#

Confirm the original reproduction now passes, and that the error rate returns to its baseline rather than merely looking better.

Then spend ten more minutes on the thing that would have made this faster: the log line that was missing, the metric nobody was watching, the error message that said Object reference not set to an instance of an object instead of which object and which request. That is the part of incident work that compounds.

A short checklist#

• [ ] Symptom stated precisely, with a time window
• [ ] First occurrence located in the logs
• [ ] Recent deployments and configuration changes checked
• [ ] Failing vs. succeeding requests compared
• [ ] Defect reproduced
• [ ] Cause understood — not just a change that makes the symptom stop
• [ ] Fix applied at the appropriate layer
• [ ] Original reproduction verified
• [ ] Diagnostics improved for next time