Txtrepeater.com

How to Remove Duplicate Lines From a List

October 14, 2026 · 4 min read · By Ash

listscleaninghow-to

Duplicates sneak into lists in the most ordinary ways. You merge two mailing lists. You export keywords from three tools. You copy rows from a report that someone else already tidied once. Before long you’ve got a list where a third of the entries are repeats, and every count, mailing and import is wrong.

Removing them is quick once you know what makes two lines “the same”. That turns out to be the part that trips people up.

What counts as a duplicate?

To you, “Apple” and “apple” are the same item. To a computer they’re different, because the capital A is a different character. In the same way, “apple” and “apple “ (with a trailing space) look identical on screen but are different strings of characters. Most “the tool missed some duplicates” complaints come down to one of three invisible differences:

  • Capitalisation. Apple, apple and APPLE.
  • Spaces. Leading or trailing spaces, or double spaces in the middle.
  • Line endings and hidden characters. Lists copied from different systems sometimes carry non-breaking spaces or stray symbols.

A good duplicate remover lets you decide how strict to be. In our remove duplicate lines tool, ignoring case and trimming spaces are both on by default, because that’s what most people mean by “the same”.

Method 1: A text tool (fastest for plain lists)

  1. Paste your list into the remove duplicate lines tool, one item per line.
  2. Decide on the options. Keep “Ignore upper and lower case” and “Trim spaces” on for ordinary lists. Turn them off for case-sensitive items such as codes.
  3. Check the numbers shown: lines in, lines out and duplicates removed. If the count of removed lines surprises you, look at the result.
  4. Copy the cleaned list.

Choosing which copy to keep matters when your list has an order. Keep the first occurrence when the earliest entry is the one you trust. Keep the last when the list is a log in which newer entries replace older ones.

Method 2: Excel

Excel has a built-in command. Select the column, go to the Data tab and choose Remove Duplicates. Excel asks which columns to compare and deletes the repeated rows, keeping the first one it finds. It compares text without regard to case, which surprises some people, so it’s more forgiving than it looks.

In recent versions of Excel you can also use the UNIQUE function, which leaves your original data alone and returns a new list:

=UNIQUE(A2:A500)

Leaving the original alone is safer, because Remove Duplicates changes your data in place.

Be careful with hidden spaces. Excel considers “apple” and “apple “ different. Clean first with TRIM:

=UNIQUE(TRIM(A2:A500))

Method 3: Google Sheets

Sheets works much the same way. Select the column, open the Data menu, choose Data cleanup, then Remove duplicates. You can also use UNIQUE:

=UNIQUE(A2:A500)

Wrap the range in TRIM if spaces might be a problem, and in LOWER if you want to ignore case.

Method 4: Sort first, then look

Sometimes you don’t want to delete anything automatically; you want to see the near-duplicates. Sort the list so similar items sit next to each other, then scan. Sort lines puts them in order, with natural ordering that handles numbers sensibly. Typos and variants such as “colour” and “color” end up adjacent, which makes them easy to spot and decide on by hand. Automatic removal will never catch those, because they’re genuinely different strings.

Checking your work

Always check the result before you replace the original.

  • Compare the counts. If you started with 1,000 lines and ended with 12, something’s odd, such as a list that was all one line because of how it was copied.
  • Spot-check. Pick a couple of entries you know were repeated and confirm one remains.
  • Keep the original. Don’t overwrite your source until you’re satisfied. A duplicate remover can’t bring a line back.
  • Watch for lookalikes. Names such as “Smith, J.” and “J. Smith” are the same person but different strings, and no tool will merge them for you.

After cleaning, you’ll often want the next step. Add text to each line wraps items in quotes or joins them into a comma-separated list for pasting into a query. Find and replace fixes a spelling across the whole list. If your list is full of unwanted line breaks, remove line breaks sorts that out first. And if you’ve overdone it and need the repeats back for testing, the text repeater will make as many as you like.

The short version

Decide what “the same” means, clean spaces and case, remove the repeats, and check the numbers. Use a text tool for plain lists and the built-in commands for spreadsheets. Keep a copy of the original, and never trust a cleaned list you haven’t looked at.

All articles