Tufazip explained: The transformer that squeezes 1GB of Wikipedia into 105MB

Who can forget Pied Piper from Silicon Valley? It was the company with a middle-out algorithm that compressed data to insane levels. Tufa Labs has done the same, creating something akin to a practical version of Pied Piper’s magic. Using an open-source program named tufazip, the company compresses 1GB of data of English Wikipedia into 105.39MB without missing out on a single byte. The system uses a 163 million parameter transformer; this is similar to the architecture behind chatbots. However, there is a twist. It takes seven and a half days to decompress the data using two Nvidia B200 GPUs. Here’s more on tufazip.

What is tufazip?

Tufazip is a lossless compression algorithm developed by Tufa Labs, which is an independent research lab that is doing some fundamental AI research. In the experiment, they used enwik9 as a test file, which is the first billion bytes of the English Wikipedia XML dump. The total size of the result is 105,388,353 bytes and it includes the decoder archive with weights and vocabulary, not just the data. The compressed text takes about 101MB and the decoder archive takes 4.4MB. Thus, it achieved roughly a 9.5-fold compression rate.

It should be noted that Tufazip cannot claim the top position in the Large Text Compression Benchmark. However, according to Tufa Labs, it is the best of those algorithms that use a neural network as the only predictor. All those that are higher in the ranking list combine the transformer architecture with hand-engineered models.

How does a transformer compress text?

Compression is prediction. A transformer predicts the next token and an arithmetic coder converts the predictions into bits. The more accurate the prediction, the less expensive it becomes. Common expressions are practically free. Surprises are expensive. Think about it, your friend who completes your sentences requires very little from you.

Decompression is the exact opposite of compression. The decoder performs the exact process as the encoder, makes the predictions and utilizes the saved bits to determine the correct token each time. Deterministic process is important because a single prediction error will ruin all predictions that come after.

Why does the model keep learning during compression?

Tufazip utilizes an online transformer that starts out trained from a quantized offline prior. It learns through compression and relearns everything to reconstruct the text during decompression. It doesn't reinvent itself from scratch. The training of that prior is not included in the compression time. This adaptation of the model on the fly allows for the model to learn the peculiarity of this particular file on the fly.

Everything should be synchronized, so the decoder is deterministic. The requirement for this project is that the repository needs precisely two visible NVIDIA B200 GPUs as well as CUDA 13.0 and certain library versions.

What’s the catch?

Speed. Compression takes around 631,548 seconds, and decompression takes about 645,748 seconds using two B200 GPUs. That is about 7.5 days to decompress, or about 1.5KB per second. And, the file is optimized specifically for one file. In addition, enwik9 is the 2006 Wikipedia benchmark sample, not the whole encyclopedia.

Can it replace Wikipedia on your phone?

Not really. A 105 MB file requiring two GPUs from a data center and one week to be decompressed is a research outcome and not a product. Offline Wikipedia can do better with just normal compression coupled with search capability.

But this is where the real value of the research comes in. Compression acts as a benchmark for language models because a good text predictor will compress the text well too. Tufazip has demonstrated that a 163 million parameter model, which is relatively small for chatbots, can get to that point with continuous training. The MIT-licensed decoder code is on GitHub, so anyone with the hardware and a spare week can check the claim.

Vyom Ramani

Vyom Ramani A journalist with a soft spot for tech, games, and things that go beep. While waiting for a delayed metro or rebooting his brain, you’ll find him solving Rubik’s Cubes, bingeing F1, or hunting for the next great snack.

View full profile