How to use this tool
- Paste one entry per line.
- Choose whether line edges, letter case, and empty lines should affect matching.
- Run Remove duplicates and review the unique list before copying it.
Understanding the output
The first matching line is retained and later matches are discarded. Duplicate totals count repeated entries, while skipped empty lines are omitted separately. Trimming changes the retained output as well as the comparison key; ignoring case changes matching but preserves the first line’s capitalization.
To arrange the unique entries afterward, use Sort Lines with the sorting order that matches your list.
Three lines become two entries
The list apple, pear, apple on separate lines becomes apple followed by pear. The output keeps the first apple rather than sorting the items alphabetically.
Decide what makes two entries equal
A repeated label can look identical while containing a leading space or different capitalization. Turning on trimming or case-insensitive matching can resolve these inconsistencies, but those choices are not always appropriate for identifiers. A case-sensitive code, file path, or password may distinguish values that ordinary prose would treat as equal.
Start with a small sample containing the edge cases you care about. Compare the retained entries with the source before replacing a larger list. This is especially important when whitespace is meaningful, such as indented data or fixed-width text.
Deduplication is different from validation
Removing repeated email addresses, product labels, or URLs does not prove that any remaining entry is valid or active. The tool compares lines, not the underlying people or resources they might represent. Different spellings of the same person remain different entries, and one shared address used by several people becomes a single line when it matches exactly.
Method and supported input
Line endings are normalized, optional trimming is applied, and a set tracks comparison keys. Unique entries are appended in their first-seen order.
- Each logical record occupies one line.
- The selected case and whitespace policy applies to every record.
Limitations
- No fuzzy matching, spelling correction, or identity matching is performed.
- Unicode characters that look alike can remain distinct.
Common questions
Will this sort the remaining list?
No. It keeps original first-occurrence order. Use the line sorter for alphabetical or natural order.
Does ignoring case change the visible capitalization?
No. The first matching line keeps its capitalization, although optional edge trimming still applies.