The Jagged Star

What carved the canyons in machine intelligence, and why nothing exists to sand them.

The Handoff

Last week, just before I handed the question back to you, I made a claim I now owe you an argument for. When an intelligence develops without consequence ever binding it, I wrote, the absence does not stay hidden. It leaves a mark on the shape of the capability itself, a signature you can learn to read.

Two weeks ago that absence showed up in the memory of a machine, by which I mean here, and throughout this series, the AI systems so many of us now talk to every day: a ledger without a scar. Last week it showed up in an institution’s record: a scar being quietly converted back into a ledger. Both times we caught the absence in behavior. This week I want to step back far enough to see it in the silhouette. The absence is not only visible in what these systems do. It is visible in the outline of what they can do at all.

The Shape Everyone Sees

Melanie Mitchell has drawn the sharpest picture of it. In her recent Yale Review essay, she takes up the term now attached to these systems, jagged intelligence, and shows what it looks like from the inside: a capability profile made of spikes and canyons, superhuman performance on one task sitting directly beside failures no competent adult would produce, with no reliable way to know in advance which one you are about to get. Her emphasis falls on the unknowns, and rightly. The danger is not that the cliffs exist. Every tool has limits. The danger is that nobody, including the people who built the system, holds a trustworthy map of where the cliffs are.

She is not alone in seeing the shape. Ethan Mollick and his collaborators, studying professionals working alongside these systems, describe a jagged frontier: an invisible, irregular boundary between the tasks where the machine lifts your work and the tasks where it quietly ruins it, running through the middle of jobs rather than around them. Two observers, two vocabularies, one silhouette.

Picture it as a shape. A capable adult’s competence, plotted across the things they actually do, looks like a rounded form: higher here, lower there, but continuous, with edges that taper rather than plunge. The machine’s profile looks like a star drawn by a seismograph. Long spikes, deep notches, and no taper anywhere.

The observation is right, and I have no quarrel with it. I have honored it because I believe it. My question is upstream of it. Why that shape?

Why Human Profiles Are Smooth

It is worth noticing that human competence does not start smooth. Spend an afternoon with a bright child, and you will meet a profile as jagged as any chart of machine test scores: startling ability directly beside startling gaps, deployed with total confidence in both directions. What happens between the child and the adult is not that talent gets redistributed more evenly. It is that consequence starts arriving, and I mean the word broadly. The weather is not only embarrassment and cost. It is the plain friction of operating a body in a physical world, where every dropped cup and misjudged step reports back instantly and without mercy.

Everywhere an adult operates, reality pushes back. Overreach gets corrected, sometimes gently, sometimes in front of an audience. I met this early. My final year structural design project as a civil engineering student, an exhibition hall entered into a design competition, rested on massive tubular arch beams, and I had analyzed the whole structure in I-DEAS, a state-of-the-art structural analysis software running on Silicon Graphics workstations, which returned beautiful, confident results. The night before submission, for no reason I can fully reconstruct, I ran a quick back-of-the-napkin check on the stresses in those beams. The package had interpreted my loads as Newtons where I had meant kilograms, an error of nearly a factor of ten, and under the real loads my structure collapsed. Nothing inside the software had ever pushed back. The analysis was fluent all the way down. The correction came from outside it, arrived almost too late, and I have never trusted a smooth output the same way since.

I still run that loop on purpose, every week, in this newsletter. Every essay goes through a gauntlet of critics whose job is to find where I have overreached, and then it goes somewhere less forgiving: in front of you, under my name, where being wrong is a matter of record. What you are reading is a sanded object.

Under that kind of pushback, a person makes one of two moves, and both of them smooth the profile. Where the cost keeps arriving, and the work matters, you practice until the canyon fills. Where it keeps arriving, and the work can be routed around, you retreat and say so out loud, which is its own kind of competence. Practice fills the canyon. Humility fences it. Either way, the surface where you actually operate gets rounded off, and something else happens that matters just as much: you come away holding a map of your own edges. I know roughly where my canyons are. I paid to find out.

Smoothness, in other words, is not talent. It is erosion. A rounded capability profile is what any capability looks like after enough weather.

And let me not oversell the smoothness, because humans are jagged too, and anyone who has watched a brilliant surgeon manage money, or a physicist hold forth on politics, knows it. So the claim is narrower, and stronger for it. We do not get sanded everywhere. We get sanded where consequence keeps returning to us, and we stay jagged where it does not. Step outside the territory where you have paid, and the confidence often travels while the sanding does not. Which is exactly the point. The smoothing follows the loop, runs wherever the loop actually runs, and stops precisely where it stops.

Remove the Loop, Get the Star

Now build an intelligence with the weather turned off.

Training pushes capability wherever an objective can be specified and graded. That is not a criticism; it is close to a definition. Wherever the grader reaches, pressure is applied, and a spike grows. Wherever the grader cannot reach, because the goal cannot be written down cleanly, or the answer cannot be easily checked, nothing pushes back at all, and whatever canyon was there stays exactly as deep as it began. No cost ever arrives from the canyon floor, so nothing marks its location, not on any map the system holds and not on any map we hold either.

A fair objection arrives immediately, and it deserves a straight answer. Training is a feedback loop. These systems are corrected millions of times, penalized for wrong answers, adjusted after every failure. Is that not consequence? It is feedback, and here I need to be precise about the word I have been leaning on, because consequence, as I am using it, means something narrower than feedback. It means that the acting system itself, the same continuing agent, bears a cost that alters its own future behavior. Feedback that lands somewhere else does not qualify. And look at where this feedback lands. The corrections land on the next version of the system, applied by someone else’s process, in service of someone else’s goals, long after the fact. I made this argument two weeks ago about memory, and it holds here without modification: a loop that closes on your descendants is not a loop that closes on you.

You might answer that nature runs exactly this kind of loop. Evolution corrects across generations, and it produces beautifully rounded competence. True, but living things run two loops at once. The slow one closes across descendants. The fast one closes inside a single life: the animal that touches the fire pulls back, carries the burn, and approaches the fire differently tomorrow, the same animal, changed. The machine has been given only the slow half. Somebody will also point out that deployment is a loop of its own, that a system which fails badly enough gets updated. It does, and notice who bears the cost while that loop grinds around: the company, the engineers, the product team. The loop exists, but it closes on the people around the system rather than on the system that acted. And at the moment the machine actually acts, when it is answering you, you can push back all you like. The pushing does not stick. The acting happens in a room where nothing that occurs can permanently reshape the actor.

A month ago, in a shorter note, I described this shape as a straight line viewed from the wrong angle, and I want to soften that here into something more careful. From most directions the star looks like chaos, spikes and notches without pattern. But turn it until you are sighting along one particular seam and much of the shape resolves. The spikes largely trace where objectives could be specified and graded. The canyons tend to fall where they could not. The jaggedness is not random. It runs, in the main, along one seam, and the seam is the reach of the grader, drawn in silhouette.

To make the seam concrete, take two things these systems are asked to do. Writing code that passes a test is easy to grade. The test passes, or it does not, millions of times over, and the systems have grown a towering spike exactly there. Now consider knowing when to tell a client that a project should be canceled. No test exists. The answer arrives years later, tangled in everything else that happened, and nobody can score it at the moment it matters. There the canyon sits.

And here the machine’s jaggedness differs from the child’s in the way that should hold our attention. The child’s profile is jagged and getting sanded. The machine’s is jagged and unaware of it. It has no feel for its own edges. It descends into its canyons at the same even confidence it carries across its peaks, because no crossing has ever cost it anything, and nothing in its history distinguishes the two terrains. That, I suspect, is the sharpest edge of Mitchell’s unknowns. The cliffs are not merely unmapped. The system’s own manner gives you no warning that you are approaching one.

There is a rival explanation, and honesty requires me to put it in front of you at full strength. A researcher can point out that these systems are jagged for reasons that have nothing to do with consequence: the design of the systems themselves, the unevenness of what they were trained on, spikes where the material was rich and canyons where it was thin. Some canyons even sit on tasks that are easy to grade, which my seam, read strictly, should not allow. I take all of that seriously, and so the claim I am making is narrower than it may have sounded. The design and the data decide what a system can learn at all. My claim is about the rounding. In every intelligence we know, the process that takes a capability and wears it smooth, that corrects the overreach, fences the weak spot, and builds the map of edges, is consequence returning to the actor. That process has no counterpart here. Whatever first carved this star, nothing exists to sand it.

I should be plain about my ground before going further. I am not one of the scientists who study these systems from the inside, and I cannot tell you what is happening inside them, and whether the canyons follow patterns that rigorous work could map is an open question for people better equipped than I am. I argue from one mechanism I have spent a career watching in people and in organizations: what happens to capability when consequence does, or does not, return to the actor. If the researchers find a better explanation for the seam, I will read it gladly and say so here. And the claim can fail on its own terms. If someone builds a system that genuinely bears consequence, acting where its own errors return to it and carry forward, and its profile stays just as jagged, then I am wrong, and I will say so here too.

The Objection I Want to Agree With

There are two objections waiting for this essay, and both come from the tradition Mitchell works in, so I want to meet them directly and then agree with most of what they say.

The first has a long name and a simple meaning. Anthropomorphism is the old human habit of treating things as if they were people: naming the car, cursing the printer, reading feelings into anything that talks. With machines that speak fluent first-person English, the habit becomes almost irresistible, and Mitchell’s tradition is rightly on guard against it, because the danger is not the comparison itself; it is that the words do the convincing for us. Talk about a machine the way we talk about a person for long enough, and we start believing it has an inner life without ever having decided to.

The second objection is sharper. The philosopher Brian Cantwell Smith argues that these systems handle descriptions of the world without ever actually dealing with the world. He calls the missing thing registration, and the plainest way I can put it is this: a navigation app can tell you the bridge is out, but if it is wrong, you are the one in the river. Nothing is ever riding on the answer for the system that produced it. On his account, until something is riding on it, you do not have understanding. You have very good paperwork.

I concede both objections, and not through gritted teeth. Nothing in this essay claims the machine feels its canyons, regrets its spikes, or suffers its training. I have argued in this series that its design gives it no way to carry such experience forward even if it existed. The comparison between human smoothing and machine jaggedness runs in one direction only, and it points back at us.

Because the pattern I am describing never needed a mind. It needs a loop, or the absence of one, and loops are a property of systems, not souls. Last week I showed you a governing body whose profile has gone jagged in precisely this way: exquisitely capable wherever scrutiny grades it, and a canyon at the exact point where the grading cannot reach, which is to say wherever power can pick up a telephone. Sever an institution’s loop, and you get the same silhouette. The jagged star is not a portrait of the machine. It is what any capability looks like, in silicon or in bylaws, when nothing returns the cost of being wrong to the thing that acts.

The Question I Cannot Answer.

So: a memory that records without bearing, a record that binds everyone except the people holding the pen, and now a shape, the silhouette capability takes when the sanding never happens.

Which leaves one question hanging over all of it, and it is the question I will take up next week. If a closed loop is what does the smoothing, what exactly is the mechanism? Because in us it is not a metaphor. The mechanism that returns consequence to the decider is physical. It has an address in the tissue, a chemistry, and a strange recent history, because when we built learning machines we borrowed its central word without taking the parts the word referred to. That story is next week’s essay.

For today, I will end where this piece has been pointed from its first sentence. In her essay, Mitchell names what separates us from these systems: we are embodied, we are active seekers, and we carry a profound caring about the consequences of our actions. She offers those as a list, three items standing side by side. I have asked her once before, in a comment on her work, the question I will now ask in public, because it is her question to answer and I genuinely do not know what she will say.

Is consequence-bearing one more axis on the jagged star, one more item on the list? Or is it the candidate explanation for why the star is jagged at all?

---

Hero images: Glencoe photograph by Craig Hunter and mountain image by AVImages, both via Pixabay; third panel generated by the author.