"I looked into why APIs actually go down. It's rarely a hack — it's an unhandled exception."

I looked into why APIs actually go down. It's rarely a hack — it's an unhandled exception.

I kept reading API outage post-mortems expecting the dramatic stuff — breaches, data leaks.
What I found was much more boring, and much more common: a handler throws, nobody caught it,
the caller gets a 500.

A few things I read that stuck with me:

• "Many 500 responses come from unhandled exceptions in controller logic, middleware, or
  background tasks. A null reference, an unexpected type, or a missing config value can throw
  an error that bubbles up past your normal checks." (thetoolstrunk, 500 Error API guide [1])
• "Many situations that currently produce 500 errors actually reflect client input or
  predictable conditions. Mapping those to 4xx cuts noise and gives callers clearer feedback." [1]
• The advice everyone repeats: deliberately send invalid data, malformed JSON, extreme values,
  and check your API returns a clean 4xx instead of a 5xx. (apidog [2])
• And the clean dividing line: "400-series for client problems, 500-series for server faults."
  (APIsec [3])

That last line matters for what I'm building, so let me be upfront about scope.

WHAT I'M BUILDING

preship — a pre-launch reliability check for FastAPI. You point it at a staging URL, it pokes
every documented endpoint with unexpected inputs, and tells you what breaks the contract:
500 crashes, undocumented status codes, response-schema mismatches. Then it hands you a
copy-paste fix prompt per pattern.

It lives almost entirely in that 5xx / contract-conformance world above. Which means I have to
be honest about the other half:

WHAT IT DOES NOT DO (the part most tools stay quiet about)

• It is NOT a security scanner. No auth-bypass, no injection, no IDOR, no data-leak detection.
  Those are mostly a 401/403 authorization problem — a different tool's job. If your handler
  leaks an extra undeclared field in a response, preship won't catch it, because that field is
  legal under the JSON schema. That's a security concern, not a reliability one.
• It misses crashes that only fire on one magic value (e.g. a bug that triggers only when
  code == 1337). Black-box fuzzing can't guess arbitrary constants.
• It can't reach endpoints behind a login.
• It needs an app that actually boots and serves /openapi.json.

I'd rather say that out loud than let someone think it covers more than it does.

WHY I'M TESTING IT THIS WAY

Before telling anyone "this works," I wanted to know exactly where it works and where it
doesn't. So I spent a few days trying to break my own tool, one question per day:

• Day 1 — how many DIFFERENT kinds of 500 does it catch? I built 6 handlers that each crash a
  different way (missing validation, empty-list math, nested-body bug, async handler,
  None-deref, and a conditional magic-value crash). It caught 5/6. The one it missed was the
  magic-value one — exactly the black-box limit above. Good: the limit is real and now I know it.
• Day 2 — does it false-flag GOOD code? I fed it well-built handlers (guarded 400s, exact
  response models, documented codes). Zero false positives — except one: it was flagging
  documented 5xx responses (like a 503 you intentionally return for maintenance) as if they
  were crashes. That was a real bug in my tool.
• Day 3 — I mapped the exact boundary of all three checks, then fixed that 5xx false-positive:
  a documented 5xx is now shown as low-severity "intended", while an undocumented 500 stays HIGH.

Then I checked the fix survived in the actually-published package (not just my local source),
and walked through the first-run experience as if I'd never seen the tool. Both held up.

WHERE IT IS NOW

It's on PyPI (pip install preship) and the code is public. But I want to be clear: this is
still me validating it through self-testing. Real-world surface is narrow — the hard part
turned out to be getting apps to even boot, not finding bugs once they do.

If you build FastAPI APIs and want to poke at it, I'd genuinely like to hear where it breaks
or where the scope feels wrong. That feedback is what I'm after right now, more than installs.

[1] thetoolstrunk.com/500-error-api
[2] apidog.com/blog/status-code-500-internal-server-error
[3] apisec.ai/blog/api-failure-7-causes-and-how-to-fix-them

Check out the repo here:https://github.com/ghkfuddl1327-wq/preship