Cleaning up duplicate contacts without losing data
Duplicates are not a tidiness problem but a trust problem: as soon as the same contact exists twice with different states, you have to check the list before believing it. And that is exactly when people stop using it.
In short
- Every clean-up starts with a complete backup — it is the only way back.
- Merging follows fixed rules per kind of field, not instinct: latest date, longest text, most active history.
- What most easily gets lost in a merge are histories and consents — both need a rule of their own.
- Without prevention the duplicates are back in six months: email as a required field, a check on creation, one import route.
How they arise
| Cause | Typical pattern |
|---|---|
| Different spellings | "Muster Ltd", "Muster", "muster ltd" |
| Several forms with no reconciliation | the same contact from two occasions |
| An import with no duplicate check | a trade fair list created entirely afresh |
| A second address for the same person | work and private |
| A change of employer | the same person, new company and address |
| Typos on manual creation | one digit, one letter |
The second-to-last row is a special case: the same person at a new company is not a duplicate but a new case with the same person. Merge the two and you lose the link to the old company — and with it the history.
Worth knowing
The heaviest loss in a merge is not names or addresses but consents and histories. If record A has a documented newsletter consent and B does not, the result has to carry that consent over — but with the original date and the original evidence.
A merged record where the consent date and evidence get lost is not merely incomplete — the consent can no longer be demonstrated. That is why the rule "which field wins" needs a line of its own for consents.
The rules for merging
| Kind of field | Rule | Reason |
|---|---|---|
| Name, company | the more complete version | "Muster Ltd" beats "Muster" |
| the most recently used | keep the old one as a secondary address | |
| Phone | keep both | second numbers are rarely wrong |
| Address | the most recently confirmed | not the most recently entered |
| Notes | merge both | never overwrite |
| Cases and histories | keep all | the actual value of the record |
| Consents | carry over with date and evidence | otherwise no longer demonstrable |
| Objections and unsubscribes | always win | an objection must never be lost |
The process in six steps
- A complete backup. An export of every record with every field, in a place where it can be found again. Do not start without this step.
- Find candidates. The same email address is certain; the same name plus the same company is likely; a similar name alone is only an indication.
- Merge the certain cases automatically. Identical email address — the error probability there is low.
- Look at the uncertain cases one by one. Above all the same names at different companies. That takes time and cannot be shortened.
- Check a sample. Open twenty merged records: are histories, notes and consents complete?
- Set up prevention. Otherwise the state is restored within six months.
Step four is the part everyone wants to shorten, and the part where the expensive mistakes happen. Two people with the same name at different companies are two people — merged automatically, you get a record that belongs to nobody and whose history mixes two different stories.
Those cases are rare, but they are irreversible once the history is mixed. So the rule is: anything not unambiguous via the email address gets decided by hand — even if that costs half a day.
Prevention
Email as a required field and key
A contact with no email address cannot be reconciled reliably. Where possible, it is the unambiguous identifier.
A check on creation
Before a new record comes into being, check whether the address already exists — and if so, open the existing one rather than creating a new one.
One import route
Trade fair lists, forms and manual entries all run through the same route with a duplicate check. A second import route is the most common source of new duplicates.
Standardise spellings
Company name with legal form, phone number in the same format, address to one pattern. That does not prevent every duplicate, but it makes them findable.
Help me clean duplicates out of our contact data. Our situation: - Number of contacts: [number] - System: [name or "spreadsheet"] - Fields we maintain: [list] - How contacts come in: [forms, import, manual] - Do we record consents with date and evidence? [yes/no] - Is there a field for objection or unsubscribe? [yes/no] Tasks: 1. Name the characteristics I should search on for duplicate candidates – separated into certain, likely, and only an indication. 2. For each of our kinds of field, phrase a rule for which value wins in a merge. Treat consents and objections separately and justify it. 3. Name the cases I must under no circumstances merge automatically. 4. Phrase a checklist for the sample review after merging. 5. Propose three measures that stop the duplicates being back in six months – matched to our intake routes. 6. Name what I have to observe on data protection in this clean-up. Do not invent field names I have not stated.
In closing
Cleaning up is manual work with clear rules: backup, certain cases automatically, uncertain ones by hand, a sample, prevention. The laborious part is the uncertain cases, and it cannot be shortened, because a wrongly merged history can no longer be separated.
Two rules stand above all the others: objections and unsubscribes always win, and consents get carried over with date and evidence. Without those two, a clean list is legally worth less than a messy one.
Common questions
How do you find duplicate contacts?
In three levels of certainty: an identical email address is a certain match, the same name plus the same company is likely, a similar name alone is only an indication. Only the first group should be merged automatically.
Which rules govern merging?
One fixed rule per kind of field: the more complete version for name and company, the most recently used email address, keep both phone numbers, the most recently confirmed address, merge notes rather than overwriting them, keep all cases, carry consents over with date and evidence — and objections always win.
What most easily gets lost in a merge?
Histories and consents. A merged record missing the consent date and evidence is not merely incomplete — the consent can no longer be demonstrated. That is why this point needs a rule of its own and a sample review.
Which cases must not be merged automatically?
The same names at different companies — those are usually two different people. And the same person after a change of employer: that is not a duplicate but a new case. Merge either automatically and you mix histories irreversibly.
How do you stop duplicates coming back?
Through four measures: the email address as a required field and unambiguous key, a check when creating new contacts, exactly one import route with a duplicate check, and consistent spellings for company name, phone number and address.
Marketing that sets itself up
The Studio Engine beta is live. Claim your spot and help shape it from the start.
Join the beta →