Menu

Post image 1
Post image 2
Post image 3
Post image 4
1 / 4
0

Processing a 2GB CSV in Node Without Running Out of Memory

DEV Community: javascript·Coded Parts·3 days ago
#ecqCKJqS
#dev#file#const#line#memory#fullscreen
Reading 0:00
15s threshold

Why the obvious approach crashes, and how a few generator functions keep memory flat no matter how big the file gets. Here's a task that looks trivial on paper: Read a CSV export, filter the rows you care about, sum one column, write a small report. The kind of thing you bang out in ten minutes. Now say the file is around 2GB. The first version is four lines. It works great on a 5MB sample. Then you point it at the real export and Node falls over with JavaScript heap out of memory. The reflex is to do what most of us do first, bump --max-old-space-size, give it more heap, run it again. It gets further and dies again. That's the moment to stop fighting the symptom and look at what the code is actually asking the machine to do. Here is the thing worth internalizing: the size of your data does not have to dictate the size of your memory footprint. You can process a file bigger than your RAM.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More