Everyone has typed “the the” at some point. It usually happens at a line break or after an edit, when your eye slides over the repeat because you already know what the sentence is meant to say. Spell checkers do not flag it, since both words are spelled correctly, and so it goes out in the email.
This tool finds repeated words and keeps one copy of each. Paste your text and the cleaned version appears on the right, with a count of how many words were dropped.
Two ways to count a repeat
“Only repeats right after each other” is the proofreading setting. It removes the second “is” in “this is is a test” and nothing else. A word that legitimately appears many times in a paragraph is left alone, and so is a sentence like “that that is true”, which is grammatical though it looks wrong. Check those by eye.
“Every repeat anywhere” is a different job altogether. It keeps only the first appearance of each word in the whole text and deletes every later one, so “the cat sat on the mat” becomes “the cat sat on mat”. The sentences no longer read as sentences. This mode is for building a list of the distinct words in a text, for instance when you want the vocabulary of a document or a set of tags from a pile of descriptions.
How words are compared
Capitalisation is ignored unless you untick that option, so “The” and “the” count as the same word. Punctuation stuck to a word is ignored when comparing. “Test,” and “test” match.
That has one side effect to be aware of. If the repeated word carries punctuation, the punctuation goes with it. “It was a test test.” becomes “It was a test”, with the full stop lost, because the second “test.” was the word that got removed. Glance at the end of sentences after a run.
The tool also respects line breaks. When it removes a word at the start of a line it does not pull the next line up into the previous one.
Related cleaning
For whole lines rather than words, use remove duplicate lines, which keeps the first or last occurrence and can ignore case and spacing. If you want a ranked list rather than a deduplicated one, the word frequency counter shows how many times each word appears. And to drop filler words like “the” and “and” altogether, see remove stop words; our guide to removing duplicate lines from a list covers the same ideas for lists.