Menu

Making Compression a Habit with zstd
📰
0

Making Compression a Habit with zstd

DEV Community·Wayne·about 1 month ago
#QTVi5UYs
Reading 0:00
15s threshold

With zstd being added to Python 3.14 , I've been using compressed files more often in my workflow. Here's what I've learned about making compression a habit. Python Data Processing with Compression Python 3.14 adds native zstd.open() support, which is a big step forward. Here's the comparison: Before 3.14 (with zstandard package): import zstandard as zstd import json import io # Writing compressed JSONL with Zstandard data = [ { " id " : 1 , " name " : " Alice " , " score " : 95 }, { " id " : 2 , " name " : " Bob " , " score " : 87 }, { " id " : 3 , " name " : " Charlie " , " score " : 92 } ] # Write with open ( ' data.jsonl.zst ' , ' wb ' ) as f : cctx = zstd . ZstdCompressor ( level = 3 ) with cctx . stream_writer ( f ) as writer : for record in data : line = ( json . dumps ( record ) + ' \n ' ). encode ( ' utf-8 ' ) writer . write ( line ) # Read with open ( ' data.jsonl.zst ' , ' rb ' ) as f : dctx = zstd . ZstdDecompressor () with dctx . stream_reader ( f ) as reader : text_stream = io .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More