Remove Duplicates from a List

What this does

This tool takes a list of lines and removes the repeated ones, or compares two lists using set operations. It was built for the moment you have two exports that should agree and don’t: a CRM list against a billing list, last month’s SKUs against this month’s, a list of user IDs against the ones that actually appear in a report.

Spreadsheets can do this with UNIQUE, COUNTIF or MATCH, but it takes a helper column and a formula you have to get right. When you just need the answer once, pasting into a box is faster.

How to use it

1. Choose what you want — removing duplicates from one list, or comparing two.
2. Paste your list. If you copied a range out of a spreadsheet, a column picker appears automatically.
3. Adjust matching if you need to: ignore capitalisation, fold accents, or treat values as emails, URLs or numbers.
4. Copy the result, or download it as a CSV that Excel opens cleanly.

A worked example

Say you export a customer list from two systems and want to know which customers exist in your CRM but never appear in billing. Copy the email column from each spreadsheet, paste one into each box, choose Only in list A, and set Treat values as → Email addresses.

That last setting matters. Exports frequently disagree about capitalisation, and one system may write Bob@Example.com where the other writes bob@example.com. Compared as plain text those are two different customers. Compared as email addresses they are correctly the same person, and your result stops being full of false positives.

Assumptions and edge cases

Anything that decides whether two values “match” is a judgement call, so here is every one this tool makes.

Blank lines

Skipped by default, and counted separately so you can see how many there were. Turn off 'Ignore blank lines' to keep them.

Surrounding spaces

Trimmed before comparing, because a trailing space from a spreadsheet export is almost never a real difference. The result shows your original text, spaces included.

Capitalisation

Ignored by default. Turn on 'Match capitalisation exactly' for case-sensitive data like IDs or tokens.

Accents

Compared as written unless you fold them. Folding uses Unicode decomposition, so it handles any script with combining marks, not just Latin.

Numbers

In number mode, 1,000.50 and 1.000,50 are the same value — the rightmost separator is read as the decimal point. Currency symbols and thousands separators, including the Swiss apostrophe, are ignored. Values that aren't numbers fall back to text comparison rather than silently collapsing together.

Email addresses

The domain is normalised fully and the local part is lowercased. Strictly, the part before the @ is case-sensitive; in practice no mail provider treats it that way, so this is the useful behaviour rather than the pedantic one.

URLs and domains

Scheme, credentials, www., port number, path, query and any trailing dot are all stripped, so https://www.example.com/pricing and example.com match.

Order

Preserved by default, in the order values first appeared, so results still line up against your source. Untick it for an alphabetical, numeric-aware sort.

Which duplicate survives

The first occurrence, with its original text intact.

Values starting with = + - or @

In the downloaded CSV only, these get a leading apostrophe. Excel reads such a cell as a formula rather than as text, which is a real hazard if your list came from a source you don't control — so the apostrophe marks it as text. It is not part of the value, and the on-screen result and the Copy button are unaffected.

Questions

Is my data uploaded anywhere?

No. Everything runs inside your browser tab using JavaScript. There is no server to send it to — this site is static files only, with no backend and no database. You can confirm it by opening your browser's network tab: nothing is sent while you work. That also means the tool keeps working if you go offline after the page loads.

How many rows can it handle?

Comfortably into the hundreds of thousands. Deduplicating 200,000 rows down to 120,000 unique values takes under 100 milliseconds in testing. The practical limit is your browser's memory for the text itself, not the comparison — if you are working with millions of rows, a spreadsheet or database is the better tool.

I pasted a column from Google Sheets and it detected several columns. Why?

When you copy a range out of Google Sheets or Excel, the clipboard contains tab-separated values, one row per line. If most rows contain tabs, this tool recognises a spreadsheet paste and offers a column picker rather than treating each whole row as one value. Pick the column you want, and tick 'First row is a header' if your paste included the header. Most similar tools skip this and compare entire rows, which is why nothing appears to match.

Why are café and cafe treated as different values?

By default, accented characters are compared as written, because in most lists an accent is meaningful. If you are reconciling data from systems that disagree about accents — a common problem when one export strips them — turn on 'Treat café and cafe as the same'. That folds accents before comparing, while still showing you the original spelling in the result.

Can I compare two lists instead of deduplicating one?

Yes. Switch the operation at the top. 'In both lists' shows shared values, 'Only in list A' shows what is missing from B, and 'Not in both' shows everything that appears on one side only. These are the operations you want when reconciling two exports that should agree but don't.

Why does the downloaded CSV wrap every value in quotes?

Because any value containing a comma would otherwise split into two columns when Excel opens it. Quoting every field and doubling internal quotes follows RFC 4180, the CSV standard, so the file opens correctly everywhere. The file also starts with a UTF-8 byte order mark, which is what makes Excel read accented and non-Latin characters correctly instead of turning them into symbols. One related safeguard: a value beginning with =, +, - or @ gets a leading apostrophe, because Excel would otherwise run it as a formula — which matters if your list came from a source you don't control.

Which copy of a duplicate is kept?

The first one, exactly as you typed or pasted it. Matching may ignore case, spacing or accents, but the value shown in the result is always the original text of the first occurrence — so you can paste results back into a spreadsheet without losing the original formatting.