Excel has a hard limit of 1,048,576 rows and slows to a crawl long before that. Open a 2 GB CSV and you get a frozen window, a truncated file or a crash. The answer is to stop loading the whole file into a grid and process it in a tool built for volume.
Plumb is built on a simple idea: the data your business runs on should be trustworthy before anyone draws conclusions from it. Cleaning, checking and reshaping data should not need macros, scripts or a weekend. Plumb keeps your files private, shows every change it proposes, and lets you approve fixes instead of hunting for problems by hand.
How Plumb helps: Data processing
Plumb's data processing streams large files instead of loading them all at once. It profiles the full dataset, applies cleaning steps (trimming, type fixes, de-duplication, filtering) across every row, and lets you preview results on a sample before running the job on everything.
How It Works
Pick a tool that processes data without rendering every row on screen.
- Check the size first: on macOS or Linux, wc -l file.csv counts rows without opening the file.
- In Excel: Data > Get Data > From Text/CSV > Transform Data. Power Query can clean files larger than the sheet limit; load the result to the Data Model or as a summary.
- In Python: for chunk in pd.read_csv('big.csv', chunksize=100_000) lets you clean and write out piece by piece.
- With DuckDB: SELECT DISTINCT * FROM 'big.csv' runs SQL directly on the file and handles millions of rows quickly.
- Write the cleaned result to a new file and keep the original untouched.
- In Plumb: upload the large file, preview fixes on a sample, then run them across the full dataset and download the cleaned output.
Where This Helps
Transaction logs
Clean months of payment or event data that would never open in Excel.
Web analytics exports
De-duplicate and filter large raw exports before analysis.
Data migrations
Prepare big legacy tables for import into a new system.
Try Plumb in early access
Plumb is in early access. Join the waitlist on tryplumb.pro to get an invite. Early access covers Excel (.xlsx), CSV and JSON files, and we are adding more sources as people ask for them. Your files are processed privately and are never used to train models.
Big files, no crashes
Now you have a repeatable way to do it, by hand or with Plumb. Join the waitlist and tell us what your data looks like; we read every reply.
Join the waitlist