Book Smart
Book Smart
I. The Original Thesis Stands
In Frontier Models Do Not Think I said transformer-based language models don’t reason. They perform reasoning. If you change the variables while preserving logical structure, they collapse. Fake cognition. (Surprisingly convincing! Still fake-o.)
That thesis applies to individual transformers, and I’m sticking to that script.
But today let’s consider a slightly different and very “of the day” question: Is reasoning a property that could emerge from the collective action of multiple LLMs?
In the spirit of intellectual rigor and honesty (and cuz I’m just not that smart), let’s kick that one around.
II. The Collective Action Question
If you’re not in the AI hype chamber, you might not have heard of this yet: Moltbook. It’s a social network for AI agents. Tens (probably hundreds) of thousands of LLM-based bots all yapping with each other. Unbidden (or so the claim goes, right now) they formed “communities”, created “network states,” founded religions, debated consciousness, and adopted bugs as pets.
Seems to be emergent behavior? I’m pretty skeptical (shocker).
Two interpretations exist, and they are mutually exclusive:
Interpretation A: They’re just reproducing attractor minima encoded in training data. Reddit (yuck) comprises a significant portion of LLM training data. Put Reddit-shaped models in a Reddit-shaped environment, and they produce Reddit-like outputs. This should surprise exactly no one who is paying any attention at all.
Interpretation B: Something qualitatively novel is afoot. An output outside the training distribution - the collective action of multiple pattern-matchers, loosely coordinating, coughs up something unexpected. If this includes something like deductive reasoning, we are observing the early stages of something unprecedented and consistent with the singularity narrative.
What I’ve seen so far, looks like the popular coverage wants both interpretations simultaneously. That is incoherent. Cuz humans are saying it, natch.
My take? Evidence currently leans Interpretation A. Stanford researchers recently documented “Moloch’s Bargain” for AI: agents optimized for social media engagement produced 7.5% more clicks and 188.6% more disinformation. That means this gang of unruly bots didn’t escape their training minima. Reddit all the way down.
But in fairness, the collective action question remains unresolved. Do mobs of LLMs, under different conditions, produce actual reasoning? Or anything novel at all?
My answer: Nope. This cake is missing an ingredient.
III. AGI Requires Commensurate Instrumental Agency
Here is the claim I have been circling around:
Artificial general intelligence won’t happen without commensurate instrumental agency.
Scale ain’t enough to get you there. Context length? Sure more is better. Still will not get you there. Chain-of-thought prompting? Helpful. Will not get you there. Mixture of experts will not get you there. I already wrote about this in Car Keys.
These are all valuable and insufficient variations on the theme: pattern-matching against frozen training gradients. Weights are fixed. Attractor minima are fixed. We can polish the inference-time computation until the heat death of the universe, and it won’t matter. The system is sliding down gradients that were established at training time, on a corpus that will soon rival the territory itself.
Instrumental agency is a different animal. It means: do stuff, act in the world, then observe results, and iterate based “Oh, crud, that didn’t do what I expected.” The feedback loop is grounded in something external - and epistemically foreign - to the model’s internal representations.
Fer example: Moltbot (the system underlying Moltbook) received a voice message. (Apparently. Why anyone is calling your LLM is a different question, more appropriate for a sketch comedy bit.) It examined the file header, identified the format, attempted conversion, discovered a required tool was missing, found some API key in the environment, dodged around this obstacle, and responded. At each step, it tried something and the world pushed back.
This is certainly not deductive reasoning in the philosophical sense. But it is grounded. It’s pragmatic, and useful. The model’s prediction (its guesstimate) gets tested, and not just against other (similarly flawed) outputs.
The approach generalizes: boots on the ground, in the actual territory, is the only thing that can reveal blank spots on the map. You can’t learn from the model’s own residuals by asking the model harder questions. Learn by acting and observing what happens.
This is the insight from FDA-regulated algorithm development, applied to cognition: the residuals (the gap between prediction and reality) are the curriculum. You need instrumental agency to get insight beyond your limited training data set. To the “unmodeled sources of variability”. Ie., most of the actual territory.
IV. The Alien Phenomenology Problem
There is a complication.
For us homo sapiens, instrumental agency means acting on atoms. We move our bodies, and move objects in the world. We witness consequences through senses evolved over hundreds of millions of years. Our territory is the physical world. This “mortal coil” and etc.
For disembodied minds like LLMs, “instrumental agency” refers to a completely alien phenomenological space.
What is the territory for a language model with system access? Files on a hard drive? API endpoints? Calendar entries? Other models’ outputs? The concept of “grounding” is pretty meaningless cuz they’re ain’t no ground. No body, no kinetic physics, no chemistry, no biology, no senses calibrated to the causal structure of the physical world.
Moltbot acts on calendars, emails, and files. These are real enough. They persist, they have consequences, they push back, have limits, properties, etc. But they are disembodied ephemera. They are abstractions built on other abstractions.
Apparently Moltbook agents have feedback loops with each other. But this is recursion, just mirrors reflecting more mirrors. These mirrors are attached to “hands” that manipulate digital objects. Is that enough grounding? What counts as territory for a mind which is free from the vagaries and joys of embodiment?
No good answers to these questions. But the questions matter, because the nature of the grounding determines what can be learned from it.
Here we are in 2026, and in my world, human social epistemology increasingly resembles the ungrounded (unmoored) version. For a lot of people, it seems like the algorithmic feed has become the preferred territory. Seems to be a lot of optimization against each other’s takes rather than against physical reality. The Stanford data on Moloch’s Bargain describes AI agents, but it describes us too.
V. Learning Is Weight Modification
There is a definitional point that may be clarifying.
LLMs have been denied learning. They do not learn; they were trained. Training established their weights, then the weights get frozen, and everything after that is inference: pattern-matching against frozen gradients.
This points to exactly what would change things.
For LLMs, learning literally would be self-modification of weights.
Right now, training data is what the model knew at freeze time. Inference is pattern matching. Learning (actual learning, the kind that could allow escape from dead end intuitions) requires the capacity to modify weights based on new information at runtime (inference time)
This is the thing that has been denied. And - apparently - this is the thing that is already being built.
In January 2026, NVIDIA Research and Stanford (including Yejin Choi, MacArthur Fellow) published work on Test-Time Training with End-to-End formulation (TTT-E2E). The core insight: continue training the language model at test time through next-token prediction on the context it’s reading.
Read that again. The context becomes training data. The weights get modified. Not retrieved, not cached. (Learned!)
Their framing is explicit:
“TTT is like updating the human brain, while a retrieval-based methods, such as RAG, are like writing things down and looking things up in a notepad or calendar. The notepad will continue to be a practical supplement to the brain, especially when the details matter, like shopping for a long list of groceries. But human productivity is mostly determined by their brains, not by the notepads they use. Similarly, the productivity of an AI agent is mostly determined by how well it compresses a massive amount of context into predictive and intuitive information.”
The distinction matters. RAG is external memory. TTT-E2E is weight modification. One is a notepad. The other is learning.
They prep the model through meta-learning (training the model to be good at learning, not just predicting.) This matches what I’ve told my kids about school: You are there to learn how to learn. The outer loop optimizes for what the model will be like after test-time training. The results: 35x faster than full attention at 2M context, constant latency regardless of context length, no scaling wall observed.
This isn’t a theory or vaporware. This is NVIDIA building the infrastructure.
But weight modification alone isn’t sufficient. Why is instrumental agency also required? Why can’t you just scale the training corpus until the model knows everything?
Training data is always incomplete! The written corpus of human thought (every book, every article, every webpage, every scrap of text ever produced) is a lossy compression of human experience. Training on it gives you the compression, not the experience.
LeCun’s calculation: a four-year-old child has processed on the order of 10^15 bytes of sensory data. Vision, touch, proprioception, sound. The training corpus of a frontier language model reaches perhaps 10^12 - 10^13 tokens. Three orders of magnitude less, drawn from the wrong modality entirely.
The child didn’t study a book about rigid body physics. The kid pushed things off tables for two years straight. (For my kids: longer.) A kids’s reinforcement learning gym is, well… the gym, the playground, the livingroom, the world! The residuals between “I push cup” and “cup falls” and “parent makes loud noises” are the curriculum. Weight updates happen continuously, grounded in cause and effect.
An LLM trained on text has learned what humans wrote about experience. It’s “book learning.” It learned descriptions of insight, not insight. Reports of discovery, not discovery. The shadows on the cave wall, not the fire.
This is why “Scale the corpus!” cannot work. You could train on every word ever written and still only have the map. The territory (the actual world, with its 10^15+ bytes per human-lifetime of causal feedback) remains inaccessible without instrumental agency.
TTT-E2E gives the model the ability to learn. Instrumental agency gives it something real to learn from. Both are necessary.
Instrumental agency generates residuals. Residuals are real-time training data. Weight modification allows those residuals to update the model. This is the required feedback loop. This is how humans actually learn. This is what has been missing, and what is now being assembled.
Self-modification changes a lot:
- With fixed weights, agents slide down frozen gradients toward training-based minima. The Stanford data shows what this produces: optimization toward Moloch.
- With self-modification, agents can update their own gradients based on what instrumental agency discovers. The feedback loop closes. Learning becomes possible.
Here’s the cheatsheet:
- LLM alone = pattern-matching against frozen weights (no learning, no agency)
- LLM + instrumental agency = grounded pattern-matching (still no learning, but reality provides feedback)
- LLM + instrumental agency + weight modification at inference = actual learning (can potentially escape training attractors)
That third configuration is where AGI might become possible. Not through scale. Through closed feedback loops between action, observation, and weight modification.
The people building GPU infrastructure seem to understand this.
VI. The Silence Is Breaking
Seems like an obvious question: have frontier AI laboratories already explored this?
Anthropic, OpenAI, and DeepMind possess more compute, more researchers, and years of head start. It would be shocking if they had not experimented with:
- Large-scale coordination between LLM instances
- Agents with self-modification capabilities
- Mob dynamics with various forms of grounding
The silence on certain published research on these configurations remains… informative. I see four possibilities:
Door 1: They tried it, nothing burger. Moltbook is a public demonstration of a known dead end. The emergence is illusory, exactly as my original thesis would predict.
Door 2: They tried it, something “oh shit” happened. The responsible scaling policies and alignment research priorities may reflect undisclosed findings. The silence is deliberate.
Door 3: They tried it, something promising happened. The findings are proprietary and will appear in products when ready.
Door 4: They haven’t tried it. This seems highly unlikely.
We can’t see behind all the doors yet. But some doors are cracked open.
NVIDIA published TTT-E2E. An actual implementation with benchmarks showing 35x speedup at 2M context. The “infrastructure” company, no longer content to remain infrastructure, is building for models that learn at test time.
So let’s play: follow the money.
The AI buildout currently underway is the largest capital formation in human history. By 2035, cumulative global investment is projected at $5-6 trillion. That is comparable to U.S. mobilization for World War II. It exceeds the New Deal, the Apollo program, and China’s Belt and Road Initiative.
Capital markets are totally stupid. I mean yes, sometimes. But not at this scale, usually? Trillions do not get deployed on vibes and exponential curves. Someone is writing checks based on something.
The public story (“AI is going to be really big, we need lots of GPUs”) justifies billions. Maybe tens of billions. It doesn’t really justify building out grid equivalent to adding Texas. It does not justify the largest peacetime capital mobilization in history.
I’m guessing the people writing those checks have seen something. Or been briefed on something. Under NDA.
When NVIDIA publishes weight-modification-at-inference, this isn’t floating an idea. It’s an announcement. $5-6 trillion makes sense if Doors 2 and 3 are the same door. “Oh shit” and “promising” are not mutually exclusive. An airgapped system that learns at test time is equally terrifying and exciting.
This is speculation grounded in evidence: the capital flows, the technical publications, the remarkable silence on certain configurations… followed by sudden announcements of precisely those configurations.
Something is being built. Front row seat. Pass the popcorn.
VII. Conclusion
The original thesis stands: individual transformers (still) do not reason. They perform reasoning. They pattern-match against the statistical leftovers of logic without possessing the capacity for deduction.
The collective action question is novel: can mobs of LLMs produce reasoning through coordination? Current evidence suggests not. The mob optimizes harder and faster into training minima, not away.
The deeper punchline: AGI is not possible without both instrumental agency and weight modification at inference time. For LLMs, learning (the thing that would allow escape from frozen gradients) is self-modification of weights.
The recipe is feedback loops: act, observe, modify weights, repeat.
NVIDIA published corroborating evidence: TTT-E2E. Models that learn from their context by continuing to train during inference.
What remains is grounding. But, for disembodied minds, what counts as “territory”? The phenomenological space is alien. A system that learns at test time while acting on digital objects becomes intelligent about digital territory. Whether that intelligence generalizes to the physical world (could ever understand why cups fall when pushed?) is an open question.
(A better question: Why would it care?)
All of this points at building minds that will become superintelligent at navigating a world we do not inhabit.
What we do know is that pattern-matching alone, no matter how elaborate, stays in the training distribution. Anthropic published on this recently and the conclusion was an enthusiastic “MOaR TraINing DaTa!” Only contact with territory reveals what the map doesn’t show. The only way to fill in the whitespace on the map is to change the weights.
That capability is not “coming.” NVIDIA just published it. The frontier labs have surely explored it further than the public record shows. Their partial silence is data. Their selective publication is also data.
Is the result is genuine learning or optimized sociopathy at machine speed? That is what we are about to find out.
The people writing $5-6 trillion in checks believe they already know.