A confidence score can be completely right about the wrong company
We watch public buying signals for small B2B teams, hiring, funding and leadership moves, and rank accounts by how strong the evidence is.
Funding is the loudest of those signals, which makes it the one most worth getting right.
The interesting failure looks like this. A headline of the shape "Some Capital leads $37M round in a medical-device startup" contains two companies. Both did something corporate. Anything that picks the subject by matching a company name against the headline finds the investor, and finds it cleanly, because the investor is usually the more famous name of the two.
There are five shapes in total and they are worth being able to name on sight. The investor. The publisher, whose name syndicated feeds append to the title. The former employer, as in "Ex-BigCo roboticist raises $5M". The common noun that happens to also be a company. And the namesake, or the product the money is for. Then a sixth that is not a naming problem at all, "in talks to raise", where the company is right and there is no round.
The part I think generalises well beyond funding news.
A confidence score does not help here, and not because it is tuned badly. It answers one question. Is this the entity we think it is. For the investor that answer is honestly yes, named, spelled correctly, unambiguous, high confidence. What is wrong is the role, and nothing in the chain was asking about the role.
Raising the threshold actively makes it worse. You throw away the marginal signals and keep every investor, because the investor cases were the confident ones from the start. Entity resolution and role assignment are separate questions and need separate answers.
So role became its own field. For each item, acting, funding, publishing, or merely mentioned, decided explicitly and then checked in code before anything becomes a signal. Stated intent is dropped for funding rather than weighted down, since there is no event yet.
Two things I would take to any similar problem.
-
A rule written in an instruction is a preference. Describing the correct behaviour in words, next to a task that needs a discrete answer, gets you the correct behaviour most of the time, and "most of the time" in a feed means you cannot tell which rows to trust. It becomes dependable at the point it becomes a field with a gate on it.
-
A filter is only good news if you read what it removed. Improving precision by discarding good data is not an improvement. We tested the change against live headlines before shipping and reviewed everything it newly dropped, one item at a time, and rounds we could confirm independently all survived. The cost of a strict filter is invisible in exactly the way a missing signal is.
One consequence I did not expect. Every misattributed item is a real, dated, linkable round belonging to some other company. Often a company that fits the profile perfectly and never entered the list, because the alert was filed under the investor's name. The same confusion that fills a feed with noise also sits on the qualified names and keeps them out of sight.
Full write-up, with a four-question test you can run on your own feed in ten minutes.
https://leadalise.com/blog/funding-signal-wrong-company
The question I actually want answered. For anyone running extraction over text, how do you catch the errors where every field is individually correct and the relationship between them is wrong. Every method I have is either a human reading rows or a second question asked separately, and both feel like brute force.
Your two takeaways already contain the answer, you just haven't pointed them at your own question yet. You found that role had to become "a field with a gate," separate from entity resolution. Your question — how to catch errors where every field is right but the relationship is wrong — is asking how to generalize that, and the reason your current methods feel like brute force is that you're validating the relationship after extraction (a human reads rows, or a second question), when the relationship should be a first-class extracted object with its own gate, like role became.
The core reason field-validation can't catch these: relational errors are invisible to per-field checks by construction. Every entity passes its own test, so checking entities harder does nothing, ever. You have to validate the predicate, not the nouns. Extract raised(amount, into: Company, by: Investor) as one typed object, and now the check isn't "is each entity correct" but "does the source support this shape" — and the wrong-company cases fail the shape even with perfectly-spelled entities, because the sentence doesn't support that role for that entity.
The un-brute-force detector you're looking for is the round-trip. A correct relation is asymmetric: "Some Capital led a round in MedCo" turns to nonsense if you swap roles. So extract the relation, regenerate a templated sentence from it, and compare that back to the source. If you misattributed and pull raised(37M, into: SomeCapital, by: MedCo), the regenerated "MedCo invested 37M in Some Capital" contradicts the original — and contradiction-against-source is a comparison, not a judgment, the one kind of check you've already shown is dependable. The relational error can't survive a round-trip, because reversing the roles produces a sentence that's false against the text, even though every name in it is spelled right.
That's the generalization of your funding fix: entity resolution asks "is this the right name," the round-trip asks "does the sentence still hold if I write the relationship back out." Different question, and it's the one your brute-force methods are approximating by hand.
Does regenerate-and-compare hold up on your namesake and common-noun shapes, or do those fail differently enough that the round-trip sentence still reads as plausible?