When Data Is Meant to Disappear: A Conversation with Varsha Ganesh

Most of the data a company creates is never meant to last. Session tokens, event logs, sensor readings, cache entries, the temporary exhaust of running software. It arrives in enormous volume, stays useful for minutes or hours, and then becomes clutter that costs money to store and slows everything down. The world generated roughly 181 zettabytes of data in 2025, and a substantial share of it consisted of short-lived operational data that was only useful for minutes, hours, or days. Handling that gracefully, letting data expire on its own without an engineer babysitting it, turns out to be one of the quietly hard problems in modern databases.

Varsha Ganesh has spent 14 years building the systems that make this look easy. She is a Senior Software Development Engineer who works on large-scale distributed database infrastructure and cloud services. One of the most significant projects she led was the implementation of Time-to-Live (TTL), the capability that allows data to expire automatically, for a managed distributed database platform. The effort required coordinating four engineering teams while redesigning core storage-engine components to support expiration at production scale. 

We spoke with Varsha about why automatic data expiration is harder than it sounds, what it takes to delete records at massive scale, and how a single missing feature can block an entire class of customers from moving to a new system.

People think of databases as systems for keeping data. You spend a lot of time on the opposite problem. Why is deleting data so hard?

Because deletion at scale is a promise you have to keep forever. Storing a record is easy. You write it and move on. Telling a customer that a record will disappear at a specific time, reliably, across a distributed system that never stops taking traffic, is a much heavier commitment. You have to track expiration for every row, find the expired ones without scanning everything, and remove them without disturbing the live workload sitting right next to it.

The other reason is that data people intend to throw away still shows up in huge quantities. A team running session state or telemetry does not want to write cleanup jobs, monitor them, and debug them at 2 a.m. when the cleanup falls behind. They want the database to handle it. When it does not, that work does not disappear. It just moves onto an engineer's plate, and it stays there.

Many customers were waiting for native Time-to-Live support before migrating. Why was that one feature such a blocker? 

The platform was a fully managed distributed NoSQL database used by customers migrating from self-managed environments. One of the most requested capabilities was native Time-to-Live support. Many customer applications were already designed around automatic expiration. They stored session information, telemetry, temporary events, and other short-lived records that were expected to disappear without scheduled cleanup jobs.

Without native expiration, those customers had to build and maintain their own cleanup processes, which became a major obstacle to migration. So although the feature itself was TTL, the real objective was removing one of the biggest adoption barriers. Once that capability existed, organizations could move existing workloads without redesigning how their applications handled short-lived data.

Walk me through what makes expiration at this scale technically hard.

Start with the volume. IoT devices alone were on track to produce around 90 zettabytes of data a year, and that kind of data, telemetry and events and readings, is exactly the kind that carries an expiration date. In the largest production environments, the platform processes millions of requests every second, and every write now had to account for expiration metadata, even for records that never expire. That meant the work lived directly in the core write path that every request passed through. There was no isolated subsystem where the feature could be hidden.  That value has to be set explicitly on all of them. So this work lived in the core write path that every single request goes through. There was no edge of the system to tuck it into.

Then there is the deletion itself. You cannot find expired data by scanning the whole dataset, that would be ruinously expensive. Physical cleanup happens as part of the storage engine's existing compaction work when it reorganizes and compacts data on disk. We rearchitected the storage and compaction layers so that expired records get cleaned up during that process, and we had to handle the conflict resolution carefully, because a record being written, updated, and expired can race in ways that produce the wrong answer if you are not deliberate about ordering.

Changing the core write path of a live system processing millions of requests per second sounds terrifying. What nearly broke? 

The scariest part was that there was no clean slate. This was a live production service with customers already depending on it, and I was proposing to change how every record was stored. Get that wrong, and you don't break one feature. You break all of them. So the work had to be introduced beneath the existing architecture, preserving the original behavior while the new functionality was phased in, completely invisible to customers using the service.

The other hard part was coordination. Time-to-Live touched the storage engine, compaction, the query layer, and the background cleanup processes, which meant four engineering teams had to move in lockstep. When you're the one holding the design together, the technical decisions are only half the job. The other half is making sure multiple teams with different priorities agree on the same sequencing and the same tradeoffs. Getting people aligned was every bit as difficult as getting the code right, and it mattered just as much.

How has judging emerging AI projects influenced the way you approach your own engineering work? 

It keeps me honest about what actually matters. As a judge for the Builders of Tomorrow AI Super Hackathon, I watch teams build under real-time pressure, and the pattern that separates the strong teams is clarity about the problem, more than technical firepower. The teams that win can tell you precisely who they are helping and what they are removing from that person's day.

That is the same instinct that made TTL worth doing. The feature was not glamorous. But it was the exact thing blocking real customers from a real decision. Judging reminds me that engineering is at its best when it is pointed at a specific, well-understood problem, not when it is showing off. Most of the impressive work I have seen, in a hackathon or in production, is impressive because it is precise, not because it is complicated.

Where is the managed database space heading, and what still holds customers back?

Adoption is still driven by parity, not novelty. The NoSQL database market is on track to reach $69.09 billion by 2031, and most of that growth is organizations modernizing legacy database environments onto managed ones. But they only move when the managed option does everything the old one did. A single missing capability, TTL, a particular consistency mode, a specific query pattern, is enough to stop a migration cold. Customers judge you by your worst gap, not your best feature.

So the real work in this space is unglamorous completeness. It is closing the small holes that individually look minor, and collectively decide whether someone can trust your system with their production workload. The interesting part is rarely a capability nobody has seen before. It is making the boring, expected things work flawlessly at a scale where flaws are expensive.

What's the part of this problem you still think about?

Data lifecycle is going to get more demanding, not less. As the volume of short-lived data keeps climbing, the systems that store it have to get better at forgetting on purpose, and forgetting is genuinely harder than remembering. There is real engineering left in making expiration cheaper and more precise, so that a customer can trust the database to clean up after itself at any scale without a second thought.

What I keep coming back to is that the best infrastructure is the kind nobody notices. When TTL works, no one thinks about it. Data appears when it should and is gone when it should be, and an engineer somewhere got to sleep through the night instead of babysitting a cleanup job. That is the goal I care about: building something so reliable it becomes invisible.