The Unexplained Surprise

The Unexplained SurpriseInformation at the Edge of Knowledge

AN ERROR CORRECTION SEMINAR · UNIVERSITÄT BASEL · 2 OCTOBER 2026

The concepts we apply to artificial intelligence have fallen behind our machines. The tools we inherited for describing intelligence, creativity, and novelty no longer line up with what we observe in large language models: systems built to predict the next token now produce arguments and images that sometimes appear genuinely new, not because they have been programmed with rules of stochastic invention, but because they operate in spaces of possibility that our frameworks were not designed to handle. The familiar positions, that these outputs are mere pastiche or that they herald a new kind of mind, have been asserted with a confidence that still exceeds our theoretical footing.

Beneath this conceptual lag sits a fact so large that it has now become hard to see: we were once surprised that these systems work, and we never went back to figure out why we were surprised. The surprise was covered over almost as soon as it arrived, because the instrument itself supplied the next steps, and the question of what the surprise was has given way to the question of how far we could take it with our existing applications. Each new capability of LLMs has since registered as a fresh small surprise, benchmarked and scaled past. But it is one thing to be surprised and another to explain the surprise, or to read the succession of them as a sequence. The wager of this seminar is that the many small surprises of the past five years are epiphenomena of a larger—capital-S—Surprise: that something in the relationship between computation and the world, between formal procedure and what used to be called Nature, has permitted these systems to work, and that we did not, and probably still do not, understand what that is.

There are two ways to approach this situation. The deflationary reading holds that nothing fundamental is at stake: the success of large language models is a particular manifestation of physics and computation that we already understand, and our surprise measures only our failure to draw out its consequences. On this reading the situation resembles an invention, like the atom bomb, which surprised no one who had the physics right: chain reaction, predicted yield, confirmation rather than revelation. The other reading holds that the surprise is evidence of a discovery, that these systems work for reasons not yet contained in our physics or our theory of computation. Deciding between these readings is not a task any single discipline can perform from within its own expertise, which is why the question deserves a day on which it cannot be dissolved back into anyone’s home field.

Creativity—the term under which these surprises are most often gathered and debated—may be best understood as a ratio: the proportion of surprise to non-surprise in what a system produces. A surprise is often the front side of an error. When Planck fit the blackbody curve by treating energy as though it came in discrete packets, he had stumbled onto the quantum, and his surprise was the correction of an error (that energy varies continuously) that classical physics had never thought to doubt. But Planck refused the surprise: he spent the better part of a decade trying to reconcile it with the physics it had broken. The pattern risks being our own: an error is still unspoken for exactly the reason the Surprise is unspoken and unexplained. Like Planck, we might have scaled past both and gone looking for ways to make the astonishment familiar again rather than sit with what it revealed to us.

To go back, we need a way into the problem that does not simply dissolve into the debates each discipline already owns—consciousness, understanding, intentionality—the usual furniture. The four concept-areas below provide one way into the problem. They are not destinations but instruments: a road map drawn from the mechanics of transformer models, four places where the surprise leaves some smoke where we might expect to find fire. Of course, they may not be the whole of it—the object we are after could turn out to revolve around a fifth thing, or the relations among these four, or something different altogether. But they are good instruments to get us started.

REGENZZIMMER 111, KOLLEGIENHAUSPETERSPLATZ, BASEL
Instruments: four concept-areas
01
02
03
04
01
Counterfactuality

Information is defined not by what is present but by what could be otherwise: this is a foundational insight that underwrites both quantum mechanics and the deepest puzzles of generative AI, but it also underlies the conditions of fiction, hypothesis, and experimental design. A bit of information is not a substance; it is a distinction between possible states, indifferent to whatever physical medium happens to carry it. This is why information looks like an abstraction and yet is grounded in physical law, and why the image of copying-and-pasting and plagiarism that dominates public discourse about generative AI is misleading. A language model does not store and retrieve sentences; it navigates a space of counterfactual possibility in which what was never said is as structurally present as what was. Meaning is constituted by what could have happened but didn’t.

The point is no longer speculative: small transformers trained only on sequences of game moves develop internal “world models” of a board they were never shown, though whether those representations do any causal work in prediction remains contested. Anthropic’s interpretability research has found that neural networks represent far more concepts than they have dimensions by encoding them in “superposition,” a term borrowed from physics that aspires to be more than metaphorical, as these systems exhibit phase changes, energy-level jumps, and geometric structures with supposed resemblances to quantum phenomena. Whether this metaphor is superficial or points to something fundamental about how information organizes itself under constraint is an open question.

A. board
click any dashed square
B. language
Every branch is available at once, in the same weights. Interpretability finds the same economy inside the representation: more features than dimensions, sharing directions. Elhage et al., arXiv:2209.10652. Structure illustrative; no probabilities are shown because none were measured.
02
Scale

The explosion of training data, data centers, and high-dimensional vector spaces recapitulates, at a different order of magnitude, certain previous historical crises of media overload and a re-brokered management of it, when the sheer volume of text overwhelmed inherited techniques of reading, sorting, and knowing. But unlike previous scenes of information overload, scale has become the source of success rather than collapse. The eighteenth-century proliferation of print produced defensive technologies for managing and excluding too much text; the twenty-first-century proliferation of data has produced an architecture that thrives on the surplus.

This is what the computer scientist Rich Sutton has called the “bitter lesson”: hand-crafted human ingenuity is consistently outperformed by brute computation leveraging massive data. The lesson is bitter because it humiliates the desire to design intelligence from first principles, and the humiliation is diagnostic: it is what an unexplained success feels like from inside a discipline. Even the field’s own record cannot settle what scale does: the live disputes over “grokking” and “emergent abilities” turn on whether capabilities leap into existence at some threshold or only appear to under certain measurements, so the surprise is not yet stable enough to be dated. Scale appears to be not just a quantitative threshold but a condition under which qualitatively different kinds of “creativity” become possible.

benchmark lifespans, publication to saturation
2012 2016 2020 2023 2025 2026 ImageNet15.3% → 3.6% err MMLU44% → 90%+ ARC-AGI-10% → 87.5% GPQA Diamond~30% → 94% SWE-bench Ver.~2% → 95% FrontierMath2% → 25% in a month Humanity’s Last Exam~3% → 53% · open ARC-AGI-2~0% → 85% · open ARC-AGI-2, built in 2025 to resist, passed the 66% human average within sixteen months.
ImageNet top-5: Krizhevsky 2012, He 2015. MMLU: arXiv:2009.03300. ARC-AGI-1 and -2: ARC Prize; 87.5% o3 high-compute Dec 2024; ARC-AGI-2 at 85% July 2026 against a 66% human average. GPQA Diamond: arXiv:2311.12022, retired as saturated. SWE-bench Verified 95% (Claude Fable 5, 2026); the SWE-Bench Pro figure is vendor-reported and disputed. FrontierMath: Epoch AI. HLE: Phan et al. 2025, 53.3%, leaderboards disagree. Saturation dates are judgment calls.
what the benchmarks were saturated with
1e271e261e251e24 20202022202320252026 training compute per model, FLOP · log scale, each gridline ×10 · estimates, not disclosures GPT-3 · 3e23GPT-4 · 2e25Grok-3 · >1e26 4–5× every year
combined capital expenditure, Amazon · Alphabet · Meta · Microsoft
800 600 400 200 0 410 2025 725 2026 USD bn per calendar year · 2026 is company guidance
Nvidia data center segment revenue
80 60 40 20 0 62.3 Q4 75.2 Q1 FY27 USD bn per quarter · consecutive reported quarters
Compute: Epoch AI, training compute of frontier models growing 4–5× per year; GPT-3 3e23, GPT-4 2e25, Grok-3 first above 1e26 (Feb 2025). Capex: Amazon $200bn, Google $175–185bn, Meta $115–135bn, Microsoft $110–120bn guided for 2026 against $410bn combined in 2025, roughly 75% AI-related. Nvidia data center segment $62.3bn to $75.2bn per quarter. Capex is guidance, not outturn; training compute is inferred, not disclosed.
03
Vectors

All data in machine learning take the form of mathematical objects that encode magnitude and direction in high-dimensional space. This is a way of indexing similarity and difference that cannot be reduced to simple classification or taxonomy. When a word, an image, or a sound enters a neural network, it becomes a point in a geometry of relations: its meaning is its position relative to everything else. This is a radical departure from the taxonomic and hierarchical structures that have hitherto organized knowledge, and it poses a genuine problem for any discipline—literary study, philosophy, linguistics—that has understood meaning as something interpreted, on a certain plane, rather than something measured.

And the geometry itself will not sit still: what interpretability research first described as fixed directions, one concept per line in the space, increasingly looks like curved manifolds and trajectories that drift over the course of a single conversation. To treat meaning as spatial, and proximity as its proxy, is to buy real explanatory power at a cost we have not yet reckoned.

the matrix, the row, and what moving it does
tokens × dimensions — click a row
the row on its own
steer → surprise 0.00
nearest by cosine
three of those dimensions, rotatable
drag to rotate
04
Bidirectionality

Consider how a general idea has always been thought to be made. You start from particular things—this triangle drawn in sand, that one printed in a textbook—and you strip away everything that makes each one itself: the size, the angles, the ink. What remains is the general idea, “triangle,” which covers every possible case precisely because it retains nothing of any single one. And the process was understood to run one way only, for a simple reason: the details were thrown away in the making, so the general idea cannot give them back. From “triangle” you can never conjure a new triangle with sides of some particular length; you can only sort the triangles already in front of you. Generality could classify, but it could not create.

Large language models do something this picture had ruled out. The same system that compresses a million sentences into a general representation can descend from that representation to a new particular—a sentence that never existed in the training data; the category generates the instance as readily as the instance yields the category. The movement is stranger still on inspection, because it is not symmetry across the board: a model trained that A is B does not thereby learn that B is A, a failure the technical literature calls the “reversal curse,” so the simplest logical reversal breaks down even as the traffic between general and particular runs freely. What runs in both directions, in other words, is not logic but the making and unmaking of general ideas—a movement between levels rather than between statements. Machines are doing something with abstraction that centuries of philosophy declared impossible.

SUBTRACTION — take away everything the cases do not share
four cases drop color drop tilt, size what all four share
COMPOSITION — add features back, freely chosen
what all four share add proportion add color, tilt a fifth case, in no input
Buckner, From Deep Learning to Rational Machines (Oxford, 2023): abstraction-as-subtraction and abstraction-as-composition, paired in transformational abstraction, pooling discarding nuisance variation and convolution composing structure.
The Prompt

The four concept-areas mark places where the surprise has left a trace. They are not topics to be covered, boxes to be filled, or positions to be defended. They may or may not end up shaping how we group ourselves—take them as instruments for locating a problem.

Each participant sends one page, single-spaced, by 20 September. It has two parts, and neither should take longer to write than an afternoon.

The first names a surprise. Not the anecdote of when you first noticed something, but whatever in the last five years of this technology still refuses to make sense to you. The test is simple: if you can explain it to your own satisfaction, it does not belong on the page.

The second names an error: some assumption, held by your field or by the public conversation about these systems, that you believe is wrong and is holding up the account. It need not be the error behind your surprise; the relation between them is exactly what we do not yet know. State it plainly enough that someone outside your discipline could disagree with it.

Everything is circulated in advance, and we will try to group the pages into three to five shared problems before anyone travels, so that the day begins with the material already on the table rather than being assembled in the room. If the grouping fails, the room votes.

FORMAT

Advance · due by 20 September

  • Distribution of the prompt with the four concept-areas
  • Each person sends a single-spaced page, in two parts
    • part 1: what do you find most surprising about AI?
    • part 2: one error that is worth making everyone’s problem
  • Everyone reads everyone else’s page before 2 October

First attempt at grouping · 20–30 September

  • Ryan reads the submissions and attempts to group them into a handful of live surprise/error problems; some may overlap or duplicate
  • But if the grouping yields more than a day can hold—or more than Ryan can tidily group together—the group votes on the morning of 2 October which 3 to 5 problems to pursue, and where to group themselves

Explanatory output

  • 3–5 short explanations of the Surprise and its Error, one per group
  • An explanation is a plain account of why the surprise happened: the error it corrects and how far that correction reaches
  • Short and plain, a few paragraphs that could be usable by someone in another field
Schedule

THURSDAY 1 OCTOBER

EveningArrivals and dinnerTBA

FRIDAY 2 OCTOBER · REGENZZIMMER 111, KOLLEGIENHAUS, PETERSPLATZ

9:00–9:30Coffee
9:30–10:00Opening plenary: we confirm the agenda (or, if group formation proves intractable, vote on which problems to approach): 3–5 problemsPLENARY
10:00–11:30Working round one: one small group (3–4 people) per problem, each beginning to draft an explanationGROUPS
11:30–12:00Report-out: each group states briefly where its explanation currently standsREPORT
12:00–13:30Lunch, catered outside the room
13:30–15:30Working round two: groups continue to develop the drafts of explanationsGROUPS
15:30–17:00Closing plenary: the full group builds the drafts into a structure, where the explanations nest, compete, clash, or convergePLENARY
19:30DinnerTBA

Exact schedule subject to change.

Basel

Venue

RoomRegenzzimmer 111
BuildingKollegienhaus
AddressPetersplatz 1, 4051 Basel
InstitutionUniversität Basel

The Kollegienhaus fills the south side of Petersplatz, five minutes on foot from Marktplatz. Open in Google Maps or OpenStreetMap.

Entrance Petersplatz 1 Foyer Stairs, 1st fl. 111 Regenzzimmer

Room numbers in the Kollegienhaus lead with the floor, so 111 sits on the first floor above the foyer.

Accommodation

Rooms are held at Hotel Odelya, just up Missionsstrasse, under ten minutes on foot from Petersplatz.

Transit

Every hotel booking in the city comes with a BaselCard, issued at check-in, which carries free second-class travel on trams and buses across the city for the length of your stay; your booking confirmation covers the ride in from the airport or the station before you have it in hand.

The city

Basel sits where Switzerland, France, and Germany meet, and its scale is deceptive: a city of two hundred thousand carrying the Kunstmuseum, the Fondation Beyeler, the Tinguely, and the oldest university in the country, founded 1460. A big part of Basel’s history dates from a sort of error correction seminar: the Council of Basel sat here from 1431, an assembly that spent eighteen years arguing about who had the authority to correct whom, and one of its secretaries, Aeneas Silvius Piccolomini, later became the pope who chartered the university. The city has been hosting long arguments about error ever since.

Detours

  • The Münster’s glazed roof tiles, green and ochre and terracotta lozenges laid in bands, are visible from most of the Rhine bank; the Pfalz terrace behind the choir is the best view in the city.
  • The Bernoullis are buried in the Peterskirche, on the far side of the square from the seminar room. Jacob’s tombstone carries a logarithmic spiral and the motto eadem mutata resurgo, though the mason cut an Archimedean spiral instead.
  • Euler was born here in 1707 and left at twenty for Petersburg. The University Library holds the Bernoulli-Euler papers, and will bring things out if you write ahead.
  • Museum Tinguely, on the Rhine below the Solitude park: several rooms of motorised sculpture, most of it built to run continuously and produce nothing.
  • The Basler Papiermühle, a working paper mill and type foundry where you can set a line of movable type and pull a sheet yourself.
  • The Kunstmuseum’s Holbein rooms. Erasmus sat for Holbein a few streets from here, and is buried in the Münster.
  • The Rhine ferries run on the current alone, strung to a cable, no engine: the ferryman angles the rudder and the river does the rest. Four of them cross the city.
  • In warm weather the city swims home from work, downstream from the Museum Tinguely as far as the Dreirosenbrücke, clothes in a dry bag. Early October is bracing but not unheard of.
Materials
01Seminar descriptionDOC
02Responses to the promptDOC
03Grouping — surprises and errorsDOC
04ExplanationsDOC

Documents open in Google Docs and are readable and editable by invited addresses only. The full folder is here.

Contact

OrganizerRyan Healey
Emailryan.healey@unibas.ch
Telephone+41 78 601 56 62

Errors (open)

— unassigned —

— unassigned —

— unassigned —

Empty by design. The errors are named in the responses, due 20 September, and will be listed here once they are in.