I fixed my scraper four times in one night. Each fix revealed a worse bug.
You know what's scarier than a crash? A feature that works 90% of the time and lies about the other 10%.
My Chrome extension saves your AI chats so you don't lose them. Someone saved a long ChatGPT conversation, and it came out missing chunks. Not the end — random pieces from the middle. Turn 3 gone, turn 5 gone, but 1, 2, 4, 6 all there. Like someone had gone through with scissors.
I'd tested this thing a dozen times. It always looked fine. Because I'd been testing with short conversations, and the bug only shows up when the chat is long enough to scroll. Fun.
Here's what was actually happening, and it's one of those things that's obvious once you know it and completely invisible until you do.
Modern chat apps use something called virtual scrolling. When a conversation gets long, the browser doesn't keep all those messages in the page at once — it'd eat too much memory. So it throws away the ones you can't see and rebuilds them when you scroll back. At any given moment, only the messages near your screen actually exist in the page. The rest are just... not there.
My extension read the page and grabbed every message it could find. Which, on a long chat, was only the handful currently on screen. The rest had been evicted. I was saving a snapshot of a peephole and calling it the whole conversation.
Okay. So I need to scroll the page myself, top to bottom, and collect messages as they get rendered. Fine. I wrote that. Put up a little overlay so the user knows why their page is suddenly scrolling on its own. Shipped it.
Saved a long chat. Still incomplete.
Turns out I was scrolling the wrong thing. I'd written the code to find the scrollable container by looking inside the page's <main> element — seemed reasonable. But ChatGPT's conversation scroller lives outside <main>, in some deeply nested div with a class name that looks like a cat walked across the keyboard. So my code found nothing, fell back to scrolling the whole window, which doesn't scroll, and quietly did nothing. The overlay showed up, sat there, and collected the same peephole as before.
Fixed the container detection — scan the whole page, pick the tallest scrollable thing, skip the sidebar. Saved again.
Now it got everything! And put it in completely the wrong order.
Because when you hit save, your page is usually scrolled to the bottom. So my code collected the last few messages first, then jumped to the top and worked down. I was storing them in the order I discovered them, not the order they were written. The conversation read like someone shuffled a deck.
So I thought, easy, I'll sort by each message's position in the page after I'm done. Except — and this is the part that made me laugh out loud at midnight — by the time I finish scrolling, the messages I collected at the top have been evicted again. The very virtual scrolling I was fighting had deleted the elements I wanted to sort. Trying to sort them was like arranging furniture that had already been thrown out.
The fix that finally worked: record each message's absolute vertical position the instant I collect it, while the element is still alive. Just a number. Then sort by the number at the end. Nodes can vanish all they want; the number stays.
That did it. Full conversation, correct order, from a chat long enough to need four screens of scrolling.
Four attempts. Each one "fixed" it and revealed the next thing underneath. Peephole → wrong scroller → wrong order → sorting ghosts. I don't think I could've reasoned my way to the last bug from the start; I had to walk into each one.
A few things I'm taking away:
- "It works on my machine" is doing a lot of heavy lifting when your test data is three messages long. Test with the ugly, realistic, oversized input.
- Virtual scrolling is everywhere now, and if you scrape any modern app, it will quietly lie to you about what's on the page.
- Fixing a bug that reveals a deeper bug isn't failure. It's just how the layers come off.
Still 0 users, still $0. But it saves the whole conversation now, in the right order, which two days ago it absolutely did not.
If you scrape pages for a living: what's the sneakiest data-loss bug you've hit? I feel like everyone who does this has one.
— building NotebookBloom in public, #3
Make the proof part of the artifact. Export a manifest with message count, role sequence, first and last snippets, and content hashes, then re-read the saved file and show a verified result. For the DOM pass, keep scrolling until two complete passes add no new hashes and both ends were reached. That will not make a private UI stable, but it makes silent truncation detectable.
This is the cleanest version of the idea anyone's given me, and it quietly solves a problem someone raised higher up in this same thread — that "it captured everything" is a promise you can't demo. A manifest you re-read and verify is exactly the demo. The proof stops being my word and becomes something on screen.
One wrinkle I've already hit with the hashing: messages keep settling after they first appear. Lazy images finish loading, code blocks get syntax-highlighted, markdown re-renders — so if I hash on first paint, the hash changes on re-read and I get a false "truncation" alarm on content that was never actually lost. So the hash has to wait for the node to stop changing, which is the same "stopped moving" condition your two-pass rule is enforcing at the list level, just one layer down at the message level.
And the "both ends reached" clause is doing the heavy lifting. The hashes prove the slice I grabbed is internally consistent; the ends prove it's the right slice. Those are two different failure modes and it took me embarrassingly long to see they're separate. Adding the manifest to the list — this is the good kind of comment.
Treat “settled” as its own manifest state instead of pretending the first hash is final: capture after a quiet window, capture again after one full scroll cycle, and record both the quiet duration and final hash. If the second capture differs, label the export unstable rather than truncated; that distinction avoids a false failure while still refusing to claim completeness.
Unstable-not-truncated is the piece I was missing, and the second it clicked I realized those are two different messages to the user, not just two labels in a file. Truncated means "don't trust this, grab it again." Unstable means "this is usable, but the conversation was still moving when you hit save — give it a beat and re-save if you want the clean copy." Same export, completely different thing to tell someone.
And unstable isn't even an error case a lot of the time — it's just someone saving while the assistant is mid-stream, still typing out its answer. That's a totally legit thing to do, and folding it into "truncated" would've had me screaming "broken!" at a save that was working fine and simply wasn't finished yet.
The quiet-duration bit is sneakily the smartest part. I'd been treating "how long is quiet enough" as a magic number to hardcode, and it isn't — it drifts by site and by network. Writing the actual duration into the manifest means I can look back and see what I waited for, instead of guessing at a threshold and never learning whether it was right. That turns the number from a guess into data. Stealing all of this.
The technical debugging is impressive, but what stood out is that you kept chasing correctness instead of stopping at "mostly works." I'd keep validating whether users are buying chat backups or confidence that every conversation is captured completely and faithfully, even in the edge cases they won't notice until it's too late.
You put your finger on the thing I keep circling back to. Nobody actually wants a "chat backup" — they want to stop having to wonder whether it's all there. And the cruel part is that's the exact promise you can't demo. A backup that works is invisible. You only find out it lied weeks later, when you go looking for the one turn that mattered and it's just gone.
So the real product isn't the export file, it's the part where you never have to check it. I can't validate that with users yet because I don't have any, but I can at least stop shipping the version that quietly breaks the promise. Feels like the whole game is turning "trust me" into something I can actually prove.
I think that's exactly the interesting part.
Reading your reply gave me one thought about what changes once the product's job becomes proving trust rather than asking users to trust it. I don't think I could explain the reasoning properly in a thread because it really depends on how you're thinking about this product.
If you're interested, what's the best email to reach you on?
Fair enough — some things really don't survive being compressed into a comment box. You can reach me at gongshaojie12 [at] gmail [dot] com. Genuinely curious where you're going with this, so don't feel like you have to have it all worked out before you write; half-formed is fine, I'll meet you there.
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
Got it, thanks — it's in my inbox. Going to give it a proper read rather than a quick skim, so I might be a day, but I'll write back. Appreciate you taking the time to put it down.
the "works 90% and lies about the other 10%" framing is the real horror, a crash is at least honest. the deeper trap is that the rendered DOM is never the source of truth. with virtual scrolling you're not reading the conversation, you're reading the eviction cache, and it only bites once the list is long enough to recycle nodes, which is why short-chat tests pass.
the sturdier fix is usually to stop scraping the view. apps that virtualize almost always keep the full list in their own state, or fetch it from the same endpoint the UI calls, so pulling from there reads the truth instead of whatever's painted on screen. hit the identical shape building a local-first assistant: what the UI showed wasn't what the underlying store actually held.
Yeah — scraping the painted DOM is the weakest source there is, and I knew it walking in. The reason I'm stuck there: my actual target is NotebookLM (the ChatGPT case was just the cleanest way to tell the story), and it has no public API. Its internal store isn't sitting in a global I can read either — it's buried in a framework's private state.
I did look at hitting the same endpoint the UI calls, and for the apps that expose it cleanly, that's obviously the right move. The catch is it trades one kind of fragility for another: private endpoints change shape with no warning, you're carrying auth and CSRF tokens around, and when they break they tend to break silently too — just one layer deeper where I can't see it. The DOM at least fails out in the open.
But "read the store, not the paint" is the sturdier default and I won't argue it. Curious how you kept the local-first version from drifting the day the store format changed under you — that's the failure mode that scares me about going that route.
late to this, it sat in my backlog. the thing that scares you is silent drift rather than drift itself. you can't stop the format changing, but you can choose how it fails.
parse strictly and refuse to return a partial result: assert the shape you expect and hard-fail on a mismatch instead of quietly skipping the records that didn't match. then reconcile counts, since most stores carry a length or a total somewhere. if you pulled 14 and it says 17, you stop. that catches the exact bug you already hit, without needing to know what changed in the format.
this is a genuinely good reframe — "you can't stop the format changing, but you can choose how it fails." that's the part i'd talked myself out of.
funny timing, because the one place i actually took the read-the-store route is my flashcard feature. notebooklm renders those in a cross-origin iframe i can't touch, but the main page fetches the whole deck from a batchexecute endpoint first — so i intercept that and pull the official JSON instead of scraping anything painted. exactly the "hit the same endpoint the UI calls" move, and honestly it's the sturdiest thing i've shipped.
and here's the embarrassing part your comment made me see: the parser for it does the exact thing you're warning against. every step is wrapped so that if the shape is off, it returns an empty array. so the day their JSON changes, it won't throw — it'll just quietly hand the user zero cards and act like the deck was empty. that's silent drift with a bow on it.
so yeah, switching it to assert-the-shape-and-hard-fail. the count reconcile is trickier here since i get the whole blob in one shot rather than paging it, but that exact trick would've nailed my original chatgpt bug cold — if i'd known where to find a total i could trust. this genuinely changes what i touch next. thanks for coming back to it.
the total you can trust is already in the blob, you just have to take it before you parse. count the raw elements in the container first, then require that your mapped output has the same length and throw when it does not. that gives you a denominator without any external count, and it catches the exact failure you described, because a per-item try that returns empty on a shape mismatch drops rows after the count was taken, not before. if the items carry their own index, a gap in the sequence also tells you which one went missing rather than only how many.
yeah, that's the piece i was missing — i kept telling myself "i get the whole blob in one shot so there's no denominator to check against," and you're right, the container IS the denominator. the cards come back as a single array, so i can grab array.length before i map anything, then require my output matches it and throw when it doesn't. the per-item try that returned empty was dropping rows after that count would've been taken, exactly like you said. obvious once you spell it out, and i genuinely didn't see it.
the index part is where mine's weaker — the cards don't carry their own id in the payload, they're just array position, so a gap won't name the concept that vanished, only that the count's off. but position isn't nothing: the .apkg export already assigns due order by array index, so i can at least point at "raw index 14 never mapped" even if i can't tell you what card it was. still miles better than "you got 40, the deck had 62, good luck."
here's the one spot a length check won't save me, and your comment is what made me see it: a dropped row fails the count, but a mistranslated row passes it. each card carries a type flag — 1 for q&a, 2 for cloze — and the day they add a type 3, my mapper would happily emit a present-but-garbage card, length still matches, check stays green. so it's two guards, not one: assert-length-and-hard-fail for the drop case, plus validate the type enum per card for the silent-wrong case. the drift i actually have to fear is the kind that keeps the count honest. anyway this genuinely changed what i'm touching next — appreciate you staying in the thread.