I shipped background removal this week — and almost let a "nice to have" model eat 1/3 of my server

Solo founder, week 17 of building Convertify (a free image converter) in public. This week I added background removal. The build was fine. The interesting part was a decision I almost got wrong.The setup. Every "add background removal" guide says pip install rembg and you're done. But my whole backend is Rust, and rembg is Python a second runtime to deploy, a second thing to crash at 3am. As a solo dev, every extra moving part is a tax I pay forever. So I spent the week running the same ML models natively in Rust instead. One binary, nothing to babysit. That part worked.The decision that almost bit me. The model I picked (ISNet) gives sharp cutouts. But when I finally benchmarked it on my actual VPS instead of trusting vibes, the numbers were sobering:
228 requests, real images
p50 2.4s, p95 9.6s speed was fine
peak RAM: 2.9 GB on a 7.7 GB box that also runs Postgres, Redis, and the web app
That model was quietly claiming a third of my server, permanently, for a feature that has low traffic yet. There's a lighter model (43 MB vs 171 MB) that would've dropped RAM massively. The easy "responsible" move was to swap to it.I kept the heavy one on purpose. Here's the reasoning I made myself write down: the box has headroom today, my target pages (signatures, product shots) are exactly where the heavy model's quality edge matters, and swapping is a 3-line change I can make in 20 minutes the day RAM actually gets tight. So I logged it as a conscious tradeoff with a documented exit, not a thing I'm ignoring. Deciding on real numbers instead of swapping out of vague guilt felt like the actual senior move.The unglamorous win. Doing the backlink pass for the launch, I found that 3 of the 4 directories on my to-do list were already done my own tracking file had them marked wrong. I'd have wasted an evening re-submitting things that already existed. Cleaning up the lie in my own notes was more valuable than any new task that day. Build-in-public lesson: your tracker rots too, audit it.Shipped this week: background removal live on prod (free, no signup, no watermark, image never touches disk), a benchmark widget on the page showing the real numbers above, and a technical write-up as a linkable asset.Next week: turning one tool into use-case pages (passport photos, product shots) without drowning in duplicate content. More SEO grind than code.Question for other solo founders: how do you decide when "it runs fine on my box today" is good enough vs premature optimization you'll regret? I keep landing on "measure, decide, document the exit" but curious how others draw that line.