How to find duplicates in a long list

When you actually need this

Lists cobbled together from several sources are where duplicates come from: two spreadsheets stitched together, several people each sending their own version, or an export that ran twice. It also matters when values must be unique — order IDs, invite codes, prize draw entries.

Another use is log triage. If the same error line repeats within a short window, something is retrying in a loop rather than failing once.

• Merging several lists and checking overlap
• Verifying order IDs or codes are unique
• Counting how often a log line repeats

How to avoid getting it wrong

Keep one record per line and strip blank lines first. Blank lines themselves are ignored, but a trailing space or a full-width character makes two visually identical records compare as different.

Matching is exact and case-sensitive. Apple and apple are two entries, and so are a padded line and a clean one. Lowercase the text first if you want a case-insensitive check.

• One record per line, no leading or trailing spaces
• Lowercase first for case-insensitive checks
• Each duplicate is reported with its count

Cleaning up afterwards

This tool only reports — it never edits your text, so you can sanity-check the counts first. Once you're happy, run the text dedupe tool to drop repeated lines while keeping the first occurrence.

If you actually need to know which rows are new when merging two lists, compare the texts line by line with the text diff tool instead of deduping.