OnSumoTools

CSV De-duplicator

This tool finds and removes duplicate rows from CSV data directly in your browser. Paste your CSV, select which columns define a duplicate, and get back a clean file with duplicates removed or flagged. No data is uploaded to a server. Your customer lists, transaction logs, and database exports stay on your device. The tool is free, requires no account, and works offline after the page loads.

Deduplication columns

Leave all checked for exact row duplicates, or pick specific columns.

No duplicates found.

5 rows in, 5 rows out

Preview (first 10 rows)

CSV deduplication runs in your browser. Files are read locally and are not uploaded to a server.

How does the CSV De-duplicator work step by step?

The CSV deduplicator removes duplicate rows from a CSV file based on one or more columns you designate as the deduplication key. Only those key columns are compared when deciding whether two rows are duplicates -- other columns are ignored in the comparison. The tool offers three matching modes for the key comparison: exact (values must match character for character), case-insensitive (both values are lowercased before comparison), and trimmed (leading and trailing whitespace is stripped before comparison). When duplicates are found, the retention policy determines which copy to keep: first occurrence (in file order), last occurrence, or flag all occurrences for manual review in a separate output file. The results summary shows total input rows, duplicate groups identified, rows removed, and rows retained. Key assumption: if no key columns are selected, the tool compares all columns and only removes rows that are completely identical across every field, including fields that may contain timestamps or auto-increment values that would prevent any match. Edge case: near-duplicate records with slight spelling variations are not detected by exact or case-insensitive matching. A fuzzy matching mode is available that flags rows where key column values have an edit distance (Levenshtein distance) below a configurable threshold, but fuzzy matching is slower and produces false positives that must be manually reviewed.

What does a typical CSV De-duplicator result look like?

You merge two customer exports and need one row per email. Upload the CSV, select only the email column, keep the first row, and download the deduplicated file.

Frequently asked questions

  • Is my CSV data sent to a server?

    No. The deduplication runs entirely in your browser using JavaScript. There is no upload and no network request. You can verify this by checking your browser's DevTools Network tab while using the tool. This is why you can safely paste customer data, email lists, or any file containing personal information.

  • How does it determine which row to keep when duplicates are found?

    By default, it keeps the first row encountered and removes later duplicates. The \\\\\\\"first\\\\\\\" row is determined by the order in your CSV file, top to bottom. There is no option to keep the last occurrence or merge fields from duplicate rows in the current version. Test with your actual input data before deploying; edge cases often behave differently than expected with real-world content.

  • Can I deduplicate by multiple columns at once?

    Yes. Select two or more columns as the duplicate key. A row is only considered a duplicate if all selected key columns match a previous row. For example, selecting both \\\\\\\"first_name\\\\\\\" and \\\\\\\"last_name\\\\\\\" means \\\\\\\"John Smith\\\\\\\" and \\\\\\\"John Adams\\\\\\\" are not duplicates even though the first name matches.

  • Does it handle large CSV files?

    The tool processes files in memory in your browser. Files up to 10 MB (roughly 50,000 to 100,000 rows depending on column count) deduplicate in under two seconds in modern browsers. For files over 50 MB, consider using a command-line tool like awk, sort -u, or a Python script with pandas.

  • Is the comparison case-sensitive?

    By default, yes. \\\\\\\"[email protected]\\\\\\\" and \\\\\\\"[email protected]\\\\\\\" are treated as different values. Toggle on \\\\\\\"Case-insensitive matching\\\\\\\" to treat them as the same. This is especially important for email deduplication, since email addresses are case-insensitive per RFC 5321.

  • Can I download the duplicates separately?

    Yes. The tool produces two outputs: the deduplicated CSV and a separate CSV containing only the rows that were removed. Both are available for download. This lets you audit what was removed before using the clean file.

Related tools

Why duplicate rows cause trouble

Duplicates sneak into spreadsheets everywhere. You merge two exports, someone pastes a list twice, a form gets submitted again because the page was slow. Then a mailing goes out twice to the same person, or a report counts one customer as three.

Cleaning that by hand is slow and you'll miss rows. This tool does it in a few clicks, and it shows you what it removed.

Pick the columns that define a duplicate

Two rows aren't always identical in every cell. Think of a contact list where the same email shows up with a different signup date. You don't want to compare the whole row. You want to compare the email column only.

Select one column or several. A row counts as a duplicate when all the selected columns match. Choose email and country together and only people sharing both get flagged.

Then decide which copy survives. Keep first holds on to the earliest row in the file. Keep last takes the newest one. If your file is sorted by date, that choice matters a lot.

Case and whitespace options

Real data is messy. "[email protected]" and "[email protected] " are the same person, but a strict comparison treats them as different. Turn on the case option and trim option to catch those.

Leave them off when case matters, like with product codes or case-sensitive IDs. It's your call, and the preview tells you right away what changed.

Check the result before you download

A before and after preview shows the row counts and a sample of what's left. If the numbers look wrong, change the columns and run it again. Nothing is final until you download.

You can also download the removed rows as their own file. That's handy when you need to audit what got dropped, or when the duplicates themselves tell you something, like a customer who signed up three times.

Your file stays on your device

The CSV is processed inside your browser. It isn't uploaded to any server, so customer lists and internal exports stay private. Very large files depend on your device memory, so if a file with a few hundred thousand rows feels slow, split it first.

Tips for cleaner data

Back up the original before you start. Always.

Dedupe on the most specific column you trust, usually an ID or an email. Names alone are risky, since two real people can share one.

If you plan to compare the cleaned file against another version, the CSV diff viewer is a good next step.