Remove Duplicate Lines
Remove repeated lines from text lists while keeping a clean set of unique entries for data cleanup and content processing.
Remove Duplicate Lines online
Remove duplicate lines from a list, keeping the first occurrence of each line and the order it appeared in — for lists that grew by accretion and copy-paste.
How to use this tool
- Paste your lines into the Lines box, one entry per line.
- Click Remove Duplicates. Each line that has appeared before is dropped; first occurrences stay in place.
- Copy the cleaned list from the result panel.
Common uses
Cleaning mailing and keyword lists merged from several sources, deduplicating exported tags, URLs or SKUs, tidying a .gitignore or config fragment, and shrinking a test dataset to its distinct cases.
How it works
The tool compares lines as exact text: a line is a duplicate only if it matches an earlier line character for character, including case and surrounding spaces. The first occurrence survives in its original position, so the output reads like your list with the repeats lifted out — order is preserved rather than sorted. Lines that look identical but differ invisibly (a trailing space, a capital letter, a tab instead of spaces) count as different lines, and the tool counts them as such.
Worked example
The list
apple banana apple Cherry banana
comes back as
apple banana Cherry banana
The second apple and second banana are gone — but look at the last line: "banana " with a trailing space survived as its own line, because to an exact comparison it is a different line. If a stubborn "duplicate" will not die, the difference is invisible — trailing whitespace is the usual suspect, and the Whitespace Remover on this site trims line edges, so run the list through it first and then dedupe.
Privacy and browser processing
Your text is processed in your browser with JavaScript — it is never uploaded or stored.
Frequently asked questions
Why is a line I can see twice still there?
Because the two copies differ invisibly — trailing spaces or tabs, different case, or different Unicode forms of the same character. Trim line edges with the Whitespace Remover first, and normalise case with the Text Case tool if case should not matter.
Does it keep the first or the last occurrence?
The first. Position in the output equals position of first appearance in the input, which makes the tool safe for ordered lists like priority queues and ranked keywords.
Can it ignore case when comparing?
Not directly — comparison is exact. Lowercase everything first with the Text Case converter if you want case-insensitive deduplication, accepting that the surviving copy will be the lowercase form.
Does it handle very long lists?
Browser memory is the only limit; lists of hundreds of thousands of short lines are fine. For million-line files a command-line sort | uniq is the better tool.
Is my list uploaded?
No. Deduplication runs in your browser; nothing is sent anywhere.