Remove duplicate lines and keep the original order
Paste text with one item per line and click deduplicate. The first occurrence of each line is kept in place and later duplicates are dropped, so the original order survives. Case-insensitive and whitespace-trimmed comparison are optional, and blank lines are always removed.
Open the tool：Remove Duplicates
When line-based deduplication is the right move
Merge a few channel exports — email lists, keyword sheets, URL inventories — and the same line shows up repeatedly. What you usually need is first-occurrence order, not a sorted unique list, because the order encodes priority.
Log lines, support ticket titles and product SKU lists behave the same way. Comparing hundreds of lines by eye is not realistic, and sorting destroys the ordering you were trying to preserve.
- Combining one column from several CSV files, dropping repeated SKUs or URLs
- Cleaning up ad-channel, keyword or email lists
- Removing repeated lines from logs or scraped output
- Summarising open-ended survey answers to see the distinct responses
What the options actually do
Comparison is per line and exact. There is no fuzzy matching and no synonym merging. With case-insensitive comparison, the first spelling encountered is the one kept. With whitespace trimming, both comparison and output use the trimmed string, so leading and trailing spaces and tabs disappear from the result.
Blank lines are always dropped; a line containing only spaces survives unless trimming is enabled. The result tells you how many lines were kept and how many duplicates were removed, but not how many times each line occurred or where.
- Leave trimming off if leading indentation matters
- Decide which casing you want to keep before enabling case-insensitive mode
- Need counts or positions? Use sort | uniq -c or a script
- For hundreds of thousands of lines, work in chunks — paste and textarea rendering are the bottleneck
Alternatives and limits
If you need occurrence counts, use sort | uniq -c or a pivot table. If you need to collapse near-duplicates — the same URL with different utm parameters, for example — normalise first with a regex or a script, then deduplicate.
Do not run whole-line deduplication on code or Markdown. Duplicate imports, table separator rows and blank lines carry meaning, and deleting them breaks the file. Very large pastes also make the textarea sluggish to scroll and select; that is rendering, not the deduplication pass.
- Counts: sort | uniq -c or a spreadsheet pivot table
- Near-duplicates: normalise URLs, case and whitespace first
- Code and Markdown: use a linter or formatter, not line deduplication
FAQ
Does deduplication change the line order?
No. The tool walks from the first line to the last, keeps the first occurrence where it is and drops later duplicates, so the output follows your original order. That differs from sorting and then deduplicating, which reorders the list and suits cases where you only care which values exist.
How do I remove duplicates but keep the counts?
This tool outputs deduplicated text only, with no counts. For counts, use sort | uniq -c on the command line, or a pivot table over the column in Excel or Google Sheets. Another option is to sort so duplicates sit next to each other, then use conditional formatting to review them.
Why did deduplication remove far more lines than expected?
The usual cause is blank lines: every empty line is skipped, and lines containing only spaces survive only when trimming is off. The second cause is case-insensitive mode, which treats lines differing only in case as duplicates. Uncheck the options and run it again to compare the kept-line counts.
Related tool：Remove Duplicates