Lists get messy. You merge two mailing lists and half the addresses appear twice. You scrape product names and the same item shows up five times. You collect keywords from three sources and they overlap. This tool takes the list and strips out the repeats, leaving each line once.
How to use it
Paste your list into the box, one item per line. The cleaned list appears underneath straight away, together with three numbers: how many lines went in, how many came out and how many duplicates were removed. Copy the result, or download it as a text file.
The options
Ignore upper and lower case. On by default. With it on, “Apple”, “APPLE” and “apple” are treated as the same line, and the first one you wrote is kept. Turn it off when capitalisation matters, for example in case-sensitive codes or file names.
Trim spaces around each line. On by default. Stray spaces at the start or end of a line are invisible but make two otherwise identical lines look different to a computer. Trimming means “apple” and “apple “ match. It also cleans the kept lines.
Remove empty lines. On by default. Blank lines are deleted from the output. Turn it off if you want to preserve spacing, though repeated blank lines will still collapse into one.
Sort the result A to Z. Off by default so your original order is kept. Turn it on to get a tidy alphabetical list as well. If you need other orderings, such as by length or in reverse, use sort lines.
Keep which copy. Choose whether the first or the last occurrence of each repeated line survives. In a list that’s updated over time, keeping the last copy preserves the most recent version.
Typical jobs
- Mailing lists and contact exports. Merge lists and remove double entries before importing. Pair this with ignoring case so differently capitalised emails don’t slip through.
- SEO and keyword lists. Combine keyword exports from several tools and keep each phrase once.
- Log files and data cleaning. Reduce a noisy list of messages to its distinct values.
- Reading lists, to-do lists and notes. Tidy up lists you’ve added to over months.
- Programming. Produce a unique set of identifiers, URLs or file paths.
For a step-by-step example, see our guide to removing duplicate lines from a list.
What it cannot do
It compares whole lines. If your list has two columns separated by a comma or a tab, a row counts as a duplicate only if the whole row matches. To de-duplicate by one column, copy just that column. Lines that differ by a single character, such as a typo, are different lines, so near-duplicates stay. If you suspect those, sort the list first with sort lines so similar entries sit together and are easy to spot by eye.
Also remember that cleaning is irreversible once you’ve copied over your original. Keep a copy of the source list until you’re sure the result is right.
Related tools
After de-duplicating, you may want to put items in order with sort lines, wrap each one in quotes with add text to each line, or fix a misspelling across the whole list with find and replace. To go the other way and create repeated lines on purpose, use the line repeater.
Comparing lists and tidying before you dedupe
If you have two lists and want to know what is in one and not the other, rather than merging them, compare two lists gives you the differences and overlap directly. Cleaning first makes deduplication more reliable: remove extra spaces and remove empty lines stop invisible differences hiding duplicates. If the repeats are words inside sentences rather than whole lines, use remove duplicate words, and to keep only lines containing a particular word, filter lines.