CSV De-duplicator
This tool finds and removes duplicate rows from CSV data directly in your browser. Paste your CSV, select which columns define a duplicate, and get back a clean file with duplicates removed or flagged. No data is uploaded to a server. Your customer lists, transaction logs, and database exports stay on your device. The tool is free, requires no account, and works offline after the page loads.
Deduplication columns
Leave all checked for exact row duplicates, or pick specific columns.
No duplicates found.
5 rows in, 5 rows out
Preview (first 10 rows)
Before
| name | orders | |
|---|---|---|
| [email protected] | John | 3 |
| [email protected] | Jane | 1 |
| [email protected] | Johnny | 5 |
| [email protected] | Bob | 2 |
| [email protected] | Jane Doe | 2 |
After
| name | orders | |
|---|---|---|
| [email protected] | John | 3 |
| [email protected] | Jane | 1 |
| [email protected] | Johnny | 5 |
| [email protected] | Bob | 2 |
| [email protected] | Jane Doe | 2 |
CSV deduplication runs in your browser. Files are read locally and are not uploaded to a server.
How this tool works
The CSV deduplicator removes duplicate rows from a CSV file based on one or more columns you designate as the deduplication key. Only those key columns are compared when deciding whether two rows are duplicates -- other columns are ignored in the comparison. The tool offers three matching modes for the key comparison: exact (values must match character for character), case-insensitive (both values are lowercased before comparison), and trimmed (leading and trailing whitespace is stripped before comparison). When duplicates are found, the retention policy determines which copy to keep: first occurrence (in file order), last occurrence, or flag all occurrences for manual review in a separate output file. The results summary shows total input rows, duplicate groups identified, rows removed, and rows retained. Key assumption: if no key columns are selected, the tool compares all columns and only removes rows that are completely identical across every field, including fields that may contain timestamps or auto-increment values that would prevent any match. Edge case: near-duplicate records with slight spelling variations are not detected by exact or case-insensitive matching. A fuzzy matching mode is available that flags rows where key column values have an edit distance (Levenshtein distance) below a configurable threshold, but fuzzy matching is slower and produces false positives that must be manually reviewed.
Worked example
You merge two customer exports and need one row per email. Upload the CSV, select only the email column, keep the first row, and download the deduplicated file.
Frequently asked questions
Is my CSV data sent to a server?
No. The deduplication runs entirely in your browser using JavaScript. There is no upload and no network request. You can verify this by checking your browser's DevTools Network tab while using the tool. This is why you can safely paste customer data, email lists, or any file containing personal information.
How does it determine which row to keep when duplicates are found?
By default, it keeps the first row encountered and removes later duplicates. The \\\\\\\"first\\\\\\\" row is determined by the order in your CSV file, top to bottom. There is no option to keep the last occurrence or merge fields from duplicate rows in the current version. Test with your actual input data before deploying; edge cases often behave differently than expected with real-world content.
Can I deduplicate by multiple columns at once?
Yes. Select two or more columns as the duplicate key. A row is only considered a duplicate if all selected key columns match a previous row. For example, selecting both \\\\\\\"first_name\\\\\\\" and \\\\\\\"last_name\\\\\\\" means \\\\\\\"John Smith\\\\\\\" and \\\\\\\"John Adams\\\\\\\" are not duplicates even though the first name matches.
Does it handle large CSV files?
The tool processes files in memory in your browser. Files up to 10 MB (roughly 50,000 to 100,000 rows depending on column count) deduplicate in under two seconds in modern browsers. For files over 50 MB, consider using a command-line tool like awk, sort -u, or a Python script with pandas.
Is the comparison case-sensitive?
By default, yes. \\\\\\\"[email protected]\\\\\\\" and \\\\\\\"[email protected]\\\\\\\" are treated as different values. Toggle on \\\\\\\"Case-insensitive matching\\\\\\\" to treat them as the same. This is especially important for email deduplication, since email addresses are case-insensitive per RFC 5321.
Can I download the duplicates separately?
Yes. The tool produces two outputs: the deduplicated CSV and a separate CSV containing only the rows that were removed. Both are available for download. This lets you audit what was removed before using the clean file.