Menu

Post image 1
Post image 2
1 / 2
0

I Built My Own Entropy Coder Because Deflate Doesn't Know What GN Knows

DEV Community·Buffer Overflow·29 days ago
#xTpOH2Px
Reading 0:00
15s threshold

I shipped gni-compression to npm two days ago. One of the first questions I got (from myself, running benchmarks at midnight): does it work on anything other than chat data? Short answer: not yet. Long answer: I found out exactly why, and it led me somewhere more interesting than I expected. The Benchmark That Told the Truth After the npm launch I ran GN against Silesia — the standard general text compression benchmark suite. Dickens, Webster, XML logs, binaries. Here's what came back: GN loses. Not slightly — brotli-6 is 10–30% better on general text depending on the corpus. Gzip-6 beats it too. The obvious question is why. GN beats brotli on chat data by ~2% consistently across 12 measurements. Same algorithm, different corpus, completely different result.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More