Identity Resolution

Identity resolution is the unglamorous data-quality layer everything else stands on: making sure one footballer is one record, even though Oraca draws him from multiple independent data providers that don't share a common player ID. Get this wrong and a player's appearances split across two records, his rating halves, and a recommendation points at a ghost. This is the system that recognises "Ben Knight" from one source and "Benjamin Knight" from another as the same human, and keeps them as one.

In one line: our sources don't agree on a player's identity, so we infer it — group candidates by date of birth, match on a nickname-aware name signature with a nationality check, pick the richest record, re-point every reference onto it, and remove the duplicate — then guard the result with nightly checks that fail loudly if a single avoidable duplicate or a broken reference survives.

The problem: several sources, no shared key

Players arrive from more than one provider. Each provider's own ID is unique on its own terms, but nothing inherently links a record of a player from one source to the record of the same player from another — the only thing connecting them is a fuzzy date-of-birth-and-name guess. Left alone, that produces cross-source twins: two records for one person. They aren't a one-off bug to fix once; they're a recurring class that has to be found and collapsed continuously, which is why this runs every night rather than once.

Recognising the same player

This is a classic record-linkage problem — deciding when two records refer to the same real-world person — and we treat it with the standard, precision-first toolkit:

• Date of birth as the blocking key. Comparisons only happen within an exact-DOB group. This makes the matching both cheap and high-precision — two players with different birthdays are never confused — though it also means a wrong or missing birth date in one source can stop a real match (a deliberate trade we'll come back to).
• A nickname-aware name signature. Names are normalised (lower-cased, accents stripped, split into tokens) and one name must be a sensible prefix-subset of the other — so B. Knight, Ben Knight and Benjamin Knight link, because "ben" is the start of "benjamin", while Iñigo Martínez and Ángel Martínez stay apart.
• A nationality check. If both records state a nationality and they disagree, they are not the same player.
• A brothers guard. A bare-initial first name ("L. Zouma") has to share that initial with the other record, so a surname-only coincidence can't merge Lionel and Yoan Zouma. This guard is exactly why some plausible-looking pairs are held back for a human to check rather than merged automatically.

A match is collapsed automatically only when it is unambiguous — two records, one per source, names that line up, nationalities consistent. Anything else — a three-way cluster, a bare-initial ambiguity, a birth-date disagreement — is routed to a human review queue, never merged blind.

Merging — keep, re-point, remove

When a match is safe to merge, we keep the richest record (the one with the most match history and context), move every reference from the duplicate onto the keeper, and remove the duplicate. Because the duplicate is genuinely removed rather than left lingering, the whole player table stays canonical: every screen and every chat answer can read it directly, with no after-the-fact alias step.

Clubs have the identical problem — the same club spelled two ways across sources — and it's resolved the same way, and first, so that when players are matched, their clubs already line up.

How we keep it honest

Because merging the wrong two records is as damaging as missing a real duplicate, the system defends in depth rather than trusting any single step, and ends every night with fail-closed checks: one that asserts zero avoidable duplicates remain after the merge, and one that asserts no surviving record points at a removed player. If either is violated, the run fails loudly rather than quietly shipping broken data — the same principle that runs through how the data stays correct. A coverage classifier also tracks where each player sits in the unified pool, so a gap is visible rather than silent.

What it can't do

• Precision over recall, on purpose. The automatic-merge bar is deliberately strict, so genuinely ambiguous pairs wait in the review queue rather than risk a wrong merge — which means the live duplicate count has a small floor until a human clears them. We would far rather leave two records briefly separate than fuse two different players, which is a one-way door.
• Date of birth is load-bearing. A wrong or missing birth date in one source means two records are never even compared; players who share a birthday rely entirely on the name and nationality checks.
• Removal is unforgiving. Because a duplicate is genuinely removed, any reference that isn't re-pointed would dangle — which is exactly what the nightly check exists to catch.
• Coverage, not model. Players who appear in none of our sources, or only thinly, are harder to resolve confidently.

The research behind it

The system is applied record linkage / entity resolution — deciding when two records refer to the same real-world entity.

• Fellegi, I. P. & Sunter, A. B. (1969). A theory for record linkage. JASA. — The foundational framework for deciding match / non-match / possible-match from agreement on fields; our deterministic rules are a precision-first cousin, with "possible match" realised as the human review queue.
• Christen, P. (2012). Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer. — The standard reference; the pipeline shape — blocking, comparison, classification, clustering — is exactly this, with exact date-of-birth as the blocking key.
• Cohen, W., Ravikumar, P. & Fienberg, S. (2003). A comparison of string distance metrics for name-matching tasks. — Name matching is its own hard problem — nicknames, initials, accents, hyphenation — and the basis for our name signature.