How to List the Unique Words in a Text (and What Counts as a Duplicate)
Which words does a piece of writing actually use? Remove the repeats and what remains is the vocabulary. That list works as a glossary draft, a word-range check, or a tidy set of tags.
When a deduplicated list is what you need
Building a glossary, preparing a vocabulary list for a language class, or turning a pile of tags into a set: none of these care how often a word appears, only whether it appears. Picking duplicates out of dozens of lines by eye gets tiring fast, and a repeat five lines down is easy to miss.
A second use is comparing two drafts. Deduplicate each version, sort both, and the words that appeared or disappeared become visible. That beats reading both versions twice when you want to know which direction the edit went.
- First drafts of glossaries, term lists and flashcards
- Turning tags or keywords into a set without repeats
- Comparing the vocabulary of two drafts
- Checking whether a word occurs in a passage at all
What counts as the same word
The text is split on whitespace and compared case-insensitively: Word, word and WORD are one word, and the first spelling encountered is the one kept. So an early Word survives while a later word is dropped, and the list never carries two rows that differ only in case.
Punctuation is not stripped, so word and word, are different entries. Convert to lowercase or remove punctuation first if you want them merged. Chinese and Japanese sentences have no spaces, so an entire sentence counts as one word and the output is rarely useful without a tokeniser.
- Case-insensitive, keeping the first spelling seen
- Punctuation stays: word and word, are separate
- One word per line, in first-appearance order
- CJK text needs a tokeniser first
Where it stops, and what to pair it with
Once words are deduplicated you cannot tell how often each one occurred. To find overused words or check keyword density, use the word frequency counter, which lists counts from high to low. This tool answers whether a word appears, not how often.
It also does no lemmatisation: run, runs and running stay three separate entries, which gets in the way if you only care about the stem. For alphabetical order, pass the result to the word sorter; if the line structure has to survive, use the line deduplicator instead, which keeps the original lines.
- Need counts: use the word frequency counter
- Need alphabetical order: sort after deduplicating
- Need to keep lines: use line deduplication
- Inflected forms need manual merging or stemming
FAQ
Are words that differ only in case duplicates?
Yes. Comparison ignores case, so Word and word count as one word and only the first spelling is kept. If you need both spellings preserved, unify the case by hand first or process the two passages separately.
Can the result be sorted alphabetically?
Not directly. Output follows first-appearance order so you can check it against the source. Copy the result into the word sorter for alphabetical order, and lowercase the text first if you want the sort to ignore case.
Does it work for Chinese or Japanese?
Poorly. Splitting happens on spaces and line breaks, and those languages do not use spaces, so a whole sentence becomes one word and the deduplicated output is basically the original. Run a tokeniser first, then deduplicate its output.
Related tool：Unique Words