I wanted to bring down what it costs to hand big files to a model. It sounded easy: text repeats a lot, so it should be compressible.

I spent time trying ideas and measuring them. There were six, and none worked.

First, what a token is

The model doesn't read letters or words.

It reads tokens, which are chunks of text it keeps in a list of 200 thousand. That list is public and you can go through it in the tiktoken repository.

And you get charged by the token.

Take this sentence:

la esperanza es lo ultimo que se pierde

That's 8 tokens. One per word.

So the question was: how do I make that cost less than 8?

The six that didn't work

There was one that did bring the tokens down: compressing the file with gzip, the same thing .zip files have always used. It cut them by 3 times.

But the model can't open a zip.

The only thing it knows how to do with what you send it is read it.

And that's the underlying problem: however cheap it comes out, if the model doesn't understand it, it's no use.

Why language repeats so much

Read this:

El g_to se sub__ al te__do y no se qu_so ba__r.

You understood it anyway. Letters are missing and it didn't matter, because your head fills them in on its own.

That's the redundancy of language, and it's the reason we understand each other.

It's what lets you read through typos, follow a thick accent, or hear someone in a bar with music playing.

Claude Shannon, the mathematician who laid the groundwork for how information travels, measured it in 1951 and put a number on it: around 75%.

And there's the catch, because that 75% is exactly what I wanted to remove:

That filler I saw as waste turns out to be what makes understanding possible.

My six ideas were all trying to remove that very thing, each in its own way, which is why they all hit the same wall.

What does work is deleting

If you can't write it better, all that's left is saying less.

I tried LLMLingua-2, from Microsoft, which runs on your laptop and is free.

What it does is go word by word deciding whether to keep it or drop it, and it hands back the ones that survived exactly as they were, without summarizing or rewriting anything.

Here's how it looks. This goes in:

Of course, by now you and I know that the customer who has accepted and used the BUG products has been trapped into facing the influence of the reciprocity rule.

And this comes out:

customer accepted BUG products reciprocity rule

From 33 tokens to 7.

Gone are the of course, the you and I know, the who has, the thes. What's left are the words that say what the sentence is about, and with those the model puts the rest together.

To find out whether it actually worked, I took two pages of a book and wrote down every concrete fact they carried. There were 12, things like:

Then I compressed the text at different levels and counted how many of those 12 facts still showed up:

The 5-times one is what surprised me, and there are two different things being measured there.

Compressing 5 times means 80% of the words are gone. Out of every five words in the original, one is left.

But out of the 12 facts only one disappeared.

So nearly all the text goes and almost none of the information.

And what it deletes is roughly that filler Shannon measured. It's removing what's predictable and betting the model will put it back, the same way you did with the cat sentence.

How I tested it

In case you want to reproduce it, it's very little code.

pip install tiktoken llmlingua

To count tokens I use tiktoken, which is the tokenizer behind OpenAI's models. You give it text and it gives you back the list of tokens:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

enc.encode("la esperanza es lo ultimo que se pierde")
# 8 tokens

[enc.decode([t]) for t in enc.encode("la esperanza es lo ultimo que se pierde")]
# ['la', ' esperanza', ' es', ' lo', ' ultimo', ' que', ' se', ' pierde']

The second one shows where each word gets split. That's where you can see the space is attached to the token.

And to compress, LLMLingua-2. The rate is how much you want to keep, so 0.25 means compressing 4 times:

from llmlingua import PromptCompressor

compresor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True,
    device_map="cpu",
)

resultado = compresor.compress_prompt(texto, rate=0.25)
print(resultado["compressed_prompt"])

The first time it downloads the model, a few hundred megabytes. After that, the two pages I used as an example went through in under a second.

Two things I did find

Where there is room left

Everything above I tested on written text, meaning sentences in a row: a book, an email, a contract.

With a spreadsheet, a CSV or a database export it's a different story.

In a 3,000-row table the same column carries the same value over and over, exactly the same. That one you can write once and then reference, without losing anything.

In written text that barely ever happens. Sentences resemble each other, but it's rare for two to be identical.

So the dictionary idea, the one that failed me with la and que, does work. Just in a spreadsheet, not in a book.

What to use today

I didn't build anything new, but these three already exist and do the job:

What I take from it

The best part was measuring before programming. Each idea fell apart right away.

I didn't build what I wanted, but I came out understanding the problem a little better.

I didn't find a valid way to do it, but if something occurs to you, the idea is welcome.