Remove Duplicate Words: Keeping First-Appearance Order in a Word List
Paste a paragraph and every repeated word is dropped after its first appearance. The result is one word per line, ordered by where each word first occurred, which is faster than scanning your own draft for repetitions by eye.
Open the tool：Remove Duplicate Words (Keep First Order)
When you need word-level deduplication
English drafts repeat the same adjective without the writer noticing, and pulling keywords out of copy for tags or a search-term list always starts with removing repeats. Word-level dedupe suits this better than line-level dedupe, which only removes lines that are identical from start to finish.
It also works as an editing check: after deduplication the words are listed one per line, so you can see which words your draft leans on and which appear only once. For non-native writers it is a quick way to catch vocabulary that repeats more than expected.
- Checking repetition in an English draft
- Deduplicating keywords and tag lists
- Turning a paragraph into a word list
How to get a clean result
Splitting uses whitespace only: spaces, tabs, and line breaks. Punctuation travels with the word, so word, and word count as two different tokens. To get a clean list, replace commas, full stops, and quotes with spaces before pasting, otherwise the same word can survive twice in different punctuation costumes.
Comparison ignores case, so The and the are treated as one word and the spelling of the first occurrence is kept. If you need everything lowercase, deduplicate first and convert case afterwards. The output is one word per line; the original punctuation and line structure are not preserved.
- Strip punctuation first, or one word can survive twice
- Case-insensitive matching, first spelling wins
- Chinese and Japanese are not split, so this suits space-delimited text
Where it stops, and what to use instead
There is no stemming: run, runs, and running count as three words. There is no stop-word filtering either, so the, a, and of stay in the output. For frequency counts or lemmatisation you need a proper tokeniser or a short script, not this.
Languages that do not put spaces between words, Chinese and Japanese in particular, arrive as a single token, so deduplication has almost no effect. Use a segmenter or split on punctuation for those languages, and be deliberate about which parts of mixed-language text you want to keep as they are.
- No stemming, so run, runs, running are separate
- No stop-word filtering
- Not suitable for Chinese or Japanese deduplication
FAQ
How is this different from a normal text dedupe?
A normal dedupe works on lines and removes a line only when the whole line repeats. This works on words, so repeats inside one line are removed too, and the output follows first-appearance order rather than being sorted alphabetically.
Why was my Chinese text not split into words?
Because splitting recognises only spaces, tabs, and line breaks, not Chinese word boundaries. A run of Chinese without spaces is treated as one token and comes back unchanged, so the tool is really for English and other space-delimited text.
Are The and the counted as duplicates, and which spelling is kept?
They are duplicates. Comparison ignores case and the first occurrence wins, so if The appears first the output says The, and if the appears first it says the. Run a case conversion after deduplicating if you need one consistent form.
Related tool：Remove Duplicate Words (Keep First Order)