Remove Duplicate Lines
Clean repeated lines.
Loading tool…
What counts as a duplicate
Removing duplicates means keeping the first occurrence of each distinct line and discarding the rest. The interesting question is what makes two lines identical. Trailing whitespace is invisible but makes lines differ. So does case, so Admin and admin are two entries unless you say otherwise. So do different line endings when a file has passed between Windows and Unix systems.
Those invisible differences are why a deduplication that appears to have failed usually has not — the lines really are different, just not in ways you can see. Trimming whitespace and normalising case before comparing catches almost all of it. It is also worth deciding whether you want deduplication to preserve the original order, which is usually what people expect, or to sort as it goes.
Why we built this tool
Merged mailing lists, exported spreadsheets, concatenated log files, and combined keyword lists all arrive with repetition. So does any list assembled by hand over time.
Those lists are typically email addresses, customer records, or URLs — personal data in most jurisdictions, and precisely what you should not paste into an unfamiliar website. Deduplication is a hash-set operation that takes microseconds, so there is no technical reason for it to involve a server. Ours runs entirely in your browser: nothing transmitted, nothing retained, no line limit, and it keeps working with the network off.
Tips and common mistakes
- Trim whitespace and normalise case before deduplicating, or near-identical entries survive.
- Preserve the original order unless you specifically want the list sorted.
- Check the removal count against your expectation — a suspiciously low number usually means invisible differences.
- Deduplicating email lists before a send avoids messaging the same person twice.
Open Remove Duplicate Lines with a link
Remove Duplicate Lines can be handed its input by the link that opens it, so the page arrives with the work already done. This is meant for assistants and scripts: if a chatbot, an editor extension, or a command-line agent already holds your content, it can build a link instead of asking you to copy and paste into a box.
The payload rides in the fragment — the part of a URL after the #. Browsers never send the fragment to a server, so anything handed over this way stays on your device exactly as a dropped file would. A link that uses the query string instead is rewritten into the fragment by the page before any analytics or ad script can read the address.
Prefix any field with gz — #gztext= — to pass it as gzip-compressed base64url, which fits roughly five to ten times more into the same link. Links stay reliable up to about 8,000 characters; past that, chat apps and address bars start truncating them, and the page says so rather than loading half a document.
One exception to the privacy rule: ?url= tells the page to download the input from an address. That request goes from your browser straight to that server — not through ours — but it does leave your device, and it only works if the server allows cross-origin requests.
#text=— the text to work on
Example: https://mygadgets.ink/tools/remove-duplicate-lines/#text=...
Quick start: using Remove Duplicate Lines
- Paste your list into the input above.
- Decide whether comparison should ignore case and surrounding whitespace.
- Choose whether to preserve the original order or sort the output.
- Run the deduplication and check how many lines were removed.
- Copy the cleaned list.
Order the result with Sort Lines, or tidy stray whitespace first using Remove Extra Spaces.