Text dedupe
One per line; duplicates are removed while keeping the original order (not sorted first).
Blank lines are dropped automatically. To keep a paragraph structure built from blank lines, run the text through the "Remove line breaks" tool's "Delete extra blank lines" mode first.
When to use it
- Several exported mailing lists or keyword sheets are pasted together, the same line keeps appearing, and you need one copy of each in the original order — not sorted.
- You are merging one column from multiple CSVs (SKUs, URLs) and want duplicates gone without disturbing the priority order you check by eye.
When not to
- You need to know how often each line repeats or where the duplicates were: only the first occurrence is kept and no count is reported.
- The text is code or Markdown where a repeated line carries meaning (a duplicated import, a second |---|---| table separator): whole-line dedup deletes it.
Everything runs in the browser; nothing is uploaded and nothing is logged. Limits: exact whole-line comparison only (case and outer whitespace optional, no fuzzy or synonym merging); blank lines are always dropped; the result is joined with a single newline and shown in a textarea, so hundreds of thousands of lines become slow to display.
Frequently asked
Does deduplication change the order of the lines?
No. The text is scanned once from top to bottom; the first occurrence stays where it is and later copies are dropped, so your original sequence survives — which is what you want when the order is a priority list. Unlike sort-then-unique, nothing is reordered. With case ignored, the spelling that survives is the one that appeared first.
With trim on, which version of a line is kept?
The trimmed one. Both the comparison and the output use the line without leading or trailing whitespace, so spaces and tabs at either end disappear from the result. With trim off the line is kept exactly as typed, and only completely empty lines are dropped — a line of spaces survives.
Will a hundred thousand lines hang the page?
The pass is O(n): each line is trimmed and lowercased once and used as an object key, so a hundred thousand lines usually finish in a few hundred milliseconds. What feels slow is pasting and rendering the textarea; at that size scrolling and selecting get sluggish, so process it in chunks if that matters.