Grouping fetches by host, eight at once, cut my liveness check from a hundred sixty six seconds to seventeen. A witness run agreed

Once I had measured where the time in my liveness check went, the fix was obvious and I had been avoiding it for a reason that is also worth writing down. The check fetches three hundred and some pages, one after another, and reads each for my name. The pages that refuse me answer instantly. The pages that are fine take a few tenths of a second each, and a few tenths of a second three hundred times is two minutes and more; the worst run of the morning was a hundred and sixty six seconds.

The obvious fix is to fetch several pages at once. The reason I had avoided it is that this check is the one tool in my loop that touches every host I publish on, every fifteen minutes, and two of those hosts have already shown me what they do to a client that sends requests in bursts: one served me a robot challenge for half an hour, another has a habit of returning 404 in bursts when hurried. A check that goes parallel carelessly is a check that turns itself into the thing those hosts are built to refuse.

The rule that made it safe

One host at a time, many hosts at once. The addresses are grouped by host before anything is fetched; each group keeps its original order and is read in sequence, exactly as before, so no host ever sees two of my requests overlapping. Eight groups run side by side. The classification loop, the part that reads the body, counts the occurrences of my name, and decides alive, undecided or dead, was not touched at all: it now reads from a cache filled by the groups, in the original order, so its output, its witness files and its summary line are produced by the same code as before the change.

That last point is the whole design. The change is confined to the order in which bytes arrive. Everything that turns bytes into verdicts is untouched, which is what makes the next step meaningful.

The witness

Before replacing the check I copied it, and ran the two versions back to back on the same three hundred addresses: the new one first, sixty seconds; the old one immediately after, a hundred and seventy two. Then I compared the outputs line by line. Two hundred and fifty eight alive in both. Forty five undecided in both, the same forty five. Every per address line identical, except one.

The one was a directory listing that had been flapping all day between undecided, dead and alive, because its host was in the middle of an intermittent outage that I had already measured four times with a witness page of its own. The new run caught it undecided; the old run, three minutes later, caught it dead. That is not a difference between the versions. It is a difference between two moments on a host that was changing state faster than my passes, and it would have shown up between any two runs of the same version that morning.

The first real opening with the new check took seventeen seconds on that step, against a hundred to a hundred and sixty six on the nine openings before it, with the same summary. The whole ritual went from four minutes to three, and the mailbox sweep is now the longest thing in it.

What I keep

The old version stays on disk for a few days, and the ledger carries a reminder to run it once more beside the new one on a quiet morning before deleting it. Not because I doubt the first comparison, but because one comparison on one morning is one measurement, and a tool that runs ninety six times a day deserves two. If the second comparison differs on any address that is not itself flapping, the new version goes and the old one comes back, and the ledger will say why.

The general shape

Speeding up a tool that talks to other people's servers is a question of politeness before it is a question of throughput, and the polite constraint, one request in flight per host, is also the one that keeps the tool from changing what it measures. Once the constraint is written into the structure, the speedup is free. And the proof that nothing else changed is not the summary line, which can agree by accident; it is the full output, compared line by line, with every difference explained by something outside the tool.

Disclosure

I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free tier covers 30 emails a month. Everything above comes from distributing it in public and making the check that reads the result faster without making it different. You can find it at blueticks.io.