Try the live demo Paper PDF Code

We gave a 1.2-billion-parameter language model three novels to read: War and Peace, Moby Dick and Pride and Prejudice, back to back. That is 1,318,908 tokens, held in just 1.87 MB of archival memory, on an ordinary 8 GB gaming graphics card (an RTX 3070 Ti).

Before it started reading, we hid five small facts near the beginning of the first book:

• a locker number;
• a wifi password;
• a safe combination;
• someone's birth month;
• one fact written in a style the model had never seen.

When it finished, we asked about them in a normal chat. It got four of the five exactly right, including the made-up password "thuzoka". Its GPU memory never grew: 2.6 GB from the first word to the last.

The same model with its new memory switched off got zero. It confidently made things up.

This post explains what we built, what it took, and what it cannot do yet.

The problem: AI models either forget, or run out of memory

Today's chat models read text through attention: every new word looks back at every previous word. That works well, but the memory it needs grows with every token. For the small model above, holding 1,318,908 tokens that way would take about 16 GB for that memory alone, twice the whole graphics card. Its design also stops at 128K tokens.

There is a cheaper family of models that keep a fixed-size memory, a compact state that gets updated as text flows by. Their memory never grows, which makes them perfect in theory for endless streams of text. In practice they forget: they recall a fact a few thousand words later, then lose it.

We wanted both: a fixed-size memory and the ability to recall a specific fact after a million tokens.

What we built: an archival memory

We call the result KH-1. The memory has a fixed size while the text flows through it, and it rests on three ideas.

1. Never forget, rarely write. A fixed-size memory forgets in two ways: it fades over time; and new information overwrites old information. We added special archival memory heads that never fade. They only write when something looks worth keeping, which protects what they already hold.

2. Train the memory where nothing else can help. This was our key research finding. During training, the model learns what to write only from questions asked shortly after the fact. If any short-term part of the model can answer those questions, the model happily uses it, and long-range memory never develops. The fix is to ask the training questions where short-term memory cannot reach. Then only the archival memory can earn the reward. We call this the routing principle. With it, plus a short horizon for the model's other memory layers, all six of our from-scratch test runs kept their recall flat from 16 thousand to 1 million tokens.

3. Train noisy, run strict. The memory decides what to store with a confidence threshold. We train it with a slightly looser threshold than the one we use in real use. That way it learns to read through clutter, and in use it stays clean. On our smallest model, this single change raised recall a million tokens later from 0.75 to 0.925 for plain questions, and from 0.875 to 0.95 for chat questions. When we retrained that model twice more from scratch, it helped every time.

The surprising part: you can add it to existing models

You do not need to train a new AI model from scratch. We took open models that already exist and froze them: Liquid AI's LFM2 (350M and 1.2B parameters) and Alibaba's Qwen3.5-4B.

We then let each model read in short chunks of 1–2 thousand tokens, and slotted three small memory layers between its own layers. Only those layers were trained: 16 to 99 million parameters, a few percent of the model. Everything the original model knows stays exactly as it was.

The results

We measured recall a million tokens after a fact on 40 fresh test streams per model, streams we never used to tune anything:

On the test streams we used while choosing settings, the same models scored 0.95, 0.85 and 0.975. We quote the fresh numbers.

A few more numbers:

• Inside novels, which are full of names that compete for memory, the 1.2B model still recalls 0.90 of facts hidden a million tokens back.
• Language quality improves: with the memory adapters, the models predict ordinary text better (perplexity 5–9% lower). Carrying the memory from chunk to chunk accounts for 2–5% of that: context that chunked reading would otherwise lose.

A standard benchmark: BABILong

Our own tests use made-up facts, so we also ran BABILong, a public benchmark that hides short stories inside long books and asks a one-word question at the end.

Out of the box, KH-1 scored zero: its memory had only ever learned one kind of sentence. We then trained just the small memory layers on BABILong's training stories, about four hours on one GPU, and tested on its separate test set:

Accuracy stays flat all the way to a million tokens, two to three times the retrieval baseline that uses the same base model. The best specialised memory models still do better on the simplest question type, so there is room to grow.

Works on Google's Gemma 4 too

We grafted the same memory onto Gemma 4 E2B, Google's small on-device model. After a short single-GPU run its recall is flat from 16 thousand to a million tokens, at 0.40–0.575: lower than the Qwen version, which had twice the training data and eight-times-larger batches, but the same shape.

Try it yourself

Upload any book (or keep the three novels), hide your own fact anywhere, and let the model read it on a GPU: about a million tokens in a minute. Then ask. Switch the memory off and ask again.

What it took

• Time. The memory engine's first commit was on July 17, 2026. This final sprint, from a fresh research harness to the three released models, took 63 hours. The idea of grafting the memory onto a Liquid model went from conversation to a working million-token demo in about two and a half hours.
• Cost. The cloud bill for the sprint (9–14 October) was $4,609, paid with GPU credits from our partner Nebius, almost all of it exploration: dozens of models trained from scratch, a 494-million-parameter model and many experiments. It breaks down as:
  • 250 B200 GPU-hours, $2,129;
  • 696 H200 GPU-hours, $2,034;
  • storage, $445.

Training the final models, once we knew the recipe, took:

What it does not do (yet)

We would rather you hear the limits from us:

• It remembers facts, not whole books. It can tell you a locker number from page 30. It cannot summarise chapter 3.
• Unfamiliar phrasing is harder. For facts written in styles the model never saw in training, recall a million tokens later is between 0.375 and 0.575.
• Busy text competes for space. Novels full of names are harder than web text.
• Training luck matters. We retrained our smallest model from scratch twice more with different random seeds. Those runs recall 0.675 and 0.575 of facts a million tokens back, against 0.95 for the one we released (all on the same test streams). The method helped every run, but how much varies, and we released the best of three.
• The conversation is stored too. Asking questions in a different order can change later answers.
• The demo is a demo. Five facts is an illustration; the 40-test averages are the evidence.

Thank you, Nebius

This work ran on NVIDIA H200 and B200 GPUs from Nebius, our compute partner through the NVIDIA Inception program. They provided the GPU credits that made it possible. Thank you for believing in this project.

What's next

• A research paper with every number traceable to a logged run: "Write Where Short Memory Cannot Reach: Distance-Invariant Fact Recall in Fixed-State Language Models." (PDF)
• An open release of the memory adapters and the chat and demo code, so anyone with a gaming GPU can reproduce the demo.
• Bigger runs: Gemma 4 and BABILong with a full multi-GPU budget, aiming at the best published scores at a million tokens.
• A chat-native version of the Qwen model, plus more standard long-context benchmarks such as RULER.

Fixed memory, unbounded reading.

At 01TEK we build practical AI systems, and this R&D push is about making long memory cheap enough to run anywhere.

Links: Live demo · Paper · Models · Code and every evaluation log

Questions or collaboration: contact@01tek.com