James Maconochie | Architecture & Attention Podcast

James Maconochie

Essays on Augmented Human Intelligence, the Wisdom Gap, and the architecture of attention in an AI-mediated world. Read in James Maconochie's own voice. jamesmaconochie.substack.com

  1. Aug 11

    The Ledger We Refused to Read

    Last week I ended on a claim I did not finish. Whatever fixes on a single number and optimizes hard enough, with nothing held separate that can disagree, loses the part that could have told it the number was wrong. I said we had built such things before, at a scale where the pursuit runs for decades, and where the part that should have said stop was never made of neurons. I meant institutions. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. An institution runs longer than any of the people inside it. That is close to its whole purpose. It holds what was learned after the people who learned it have gone, and it hands that forward to people who were not there. Which gives an institution something a machine does not have: a record that outlives the individual who paid for it. So the failure I want to describe is not forgetting. The records are there. Minuted, archived, classified, indexed. The failure is stranger, and worse. An institution can hold a complete account of what happened and learn nothing from it, because reading a ledger is not the same as bearing what the ledger records. Three Parts, Not One There is a tension there, and when I began this piece I thought I could dissolve it. Last week’s absence was a missing dissenter: a part held separate, able to disagree, able to say stop. This week’s absence is a memory nobody answers for. A missing voice is not a missing archive, and I had a tidy answer ready for why those were the same thing seen from two angles. The answer was too quick. I would rather correct it here than let it stand. They are not the same absence. They are different parts of one loop, and I have spent four essays describing that loop as though it were a single part when it has at least three. Memory preserves the argument. It holds what was concluded, on what evidence, against what alternatives, at what cost. Dissent tests the argument. It supplies pressure while a judgment is still being formed, rather than after it has hardened. Accountability makes someone answer for the choices they make. It is what keeps the first two from becoming ceremonial: a record cited without being read, an objection minuted without being weighed. An institution can be excellent at one of these and hopeless at the others. A safety board can preserve forty years of incident reports while nobody who reads them stands to lose anything. A founder can carry enormous personal exposure and systematically suppress everyone who disagrees. Those are not edge cases. They are ordinary. Which changes what I have been claiming, so let me say it plainly. I have been treating consequence-bearing as the entire loop. It is not. It is the part that makes the other two binding. A record to which nobody is accountable cannot say stop. It can only be cited or set aside, and both cost the same, which is nothing. But accountability with nothing to read and nobody to argue with is only exposure, and exposure produces anxiety rather than judgment. Forty Years Of The Same Answer Here is a case where the record is unusually clear and unusually long. For roughly four decades the United States held a posture toward Iran that is easy to describe and easy to get wrong. It was not restraint in any general sense. There were sanctions throughout, covert action, proxy conflict, naval confrontation, and targeted strikes. The posture was narrower and more specific: a repeated refusal to take the last step. No full-scale war, no attempt at regime change by force. That posture was tested about as hard as a posture can be. In 1983, a bombing at the Marine barracks in Beirut, widely attributed to an Iran-backed group, killed 241 American service members. No war followed. In 1988 American and Iranian forces fought directly in the Gulf, and an American warship shot down an Iranian civilian airliner, killing everyone aboard. No war followed. In 2003, the United States invaded the country next door and did not stray into Iran. In 2020 an American strike killed a senior Iranian commander, Iran retaliated against American bases, and both sides stopped. Different decades. Different administrations, different parties, different circumstances. The same conclusion, reached again and again by people who agreed with each other about very little else. Then it ended. In June 2025 Israel struck Iranian nuclear infrastructure, and the United States followed within days, using weapons only it possessed. In February 2026, the sequence repeated on a far greater scale, and this time the stated aims included regime change. Iran’s supreme leader was killed. Iran struck American bases across the region and closed the Strait of Hormuz. A ceasefire took hold in April and was extended indefinitely. A memorandum of understanding signed in mid-June formalized it, reopened the strait, and opened a sixty-day window to negotiate a permanent settlement. The window did not hold. Exchanges resumed within weeks, and by mid-July, American strikes were running nightly for nearly a fortnight before pausing. They have resumed and paused again since. As I write, talks are stalled, the terms are unsettled, and the outcome is unknown. I am not going to tell you that ending the posture was wrong. I do not know that, and I do not think anyone does yet. Circumstances had changed, and changed circumstances are exactly when a long-held position ought to be re-examined. I also want to be careful about what that sequence shows, because it is less than I first wanted it to be. It does not show one accumulating body of reasoning handed down intact. The people who stopped short in 1988 may have done so for reasons with little in common with those in 2020. Arriving repeatedly at the same outer boundary is not the same as sharing a case for it. If I claimed otherwise, I would be assembling the very ledger I am about to accuse the institution of never reading. So the claim is narrower. Whatever those reasons were, they were written down, and across four decades administrations repeatedly stopped short of the step this one took. We have heard the case for acting. What I have not seen is the other half of it: an account of why the reasoning that stopped everyone before no longer governs. Those are two different documents, and only one of them has been produced. I should be plain about what I cannot know. The ledger may have been read with enormous care. Somewhere there are almost certainly historical reviews, contingency analyses, and dissenting assessments, and I have no access to any of them. Note what that means for the three parts. Memory: the record exists. Dissent: there were people in that building who disagreed, and it would be strange if there were not. Both parts were probably intact. The failure I am describing sits in the third part, and only in the third. Nobody has had to answer for the choice between them. How A Judgment Becomes A Convention Here is one way a reasoned position turns into an unreasoned one without anybody deciding to let it. The people who reached the conclusion in 1983 knew why. They had the cables, the casualty reports, the estimates of what a wider war would cost and who would fight it. The conclusion was expensive, and they had paid for it. The people who inherited it a decade later had the conclusion. They could also have had the reasoning, because it was written down. But a conclusion travels more easily than the reasoning behind it. It fits in a sentence. It can be handed over in a briefing. And each time it is handed over, a little more of the weight comes off, until what arrives is a convention rather than a judgment. This is how we do things here. A convention is cheap to discard, because nothing about it explains itself. It looks arbitrary, and arbitrary things invite the question of why we are still doing them. Which is a fair question, and someone should ask it. The trouble is that when answering it requires reading four decades of reasoning nobody currently in the room has read, the practical answer becomes no reason at all. David Hoze has a line I keep returning to: the certificate is always signed one floor above. Accountability that always defers upward is accountability nobody holds. It is always someone else’s job to have read the ledger, until it turns out to have been nobody’s. Where The Cost Goes Severing the mechanism in us does not make a cost disappear. It moves it. Alberto Romero has described safety margin as cost relocation, and the phrase generalizes further than he needed it to. When an institution removes the part of itself that bore the consequence, it has not become more efficient. It has arranged for the consequence to be borne somewhere else, by people who were not in the room and cannot contest the bill. This is the structural version of a claim people usually make in a partisan register, and I want to keep it structural, because the asymmetry does not care which direction the policy went. The judgment is made within a system where most of the consequences fall outside the loop that made it. The cost falls on service members, on civilians in the region, on people whose energy and food prices move when a strait closes, on a generation of taxpayers who will still be paying for the munitions long after the argument has been forgotten. Relocation alone is not the problem, and I should be honest that every state decision relocates costs onto people who were not in the room. Tax policy does it. Interest rates do it. Restraint does it too, and the people who bear the cost of a war not fought are simply harder to identify. The question is whether the cost returns as information. Whether anything comes back up the line in a form capable of changing the next judgment, or whether it lands and stays landed. There is a harder question underneath that one, and I can name it without pretending to settle it. Accountability is always accountability to somebody. Decisi

  2. Aug 4

    AI Inherited the Word ‘Reward,’ Not the Parts

    In reinforcement learning, a reward is a number. That is the definition, not a simplification. An agent acts, the environment returns a scalar, and the agent adjusts to make that number larger next time. Everything the field has built runs on that single quantity. Maximize the number. That is the engine. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. The word came from us, and the engineers who took it were not careless with it. They borrowed a working formalism, one that psychologists had built studying animal learning, and it worked, which is why it stayed. The carelessness came later, in how the rest of us started to talk. Reward function. Reward hacking. The model wants this, the model likes that. The word carries a psychology, and once it was loose in the language, it began paying out meaning the math had never deposited. Because in the brain, reward is not one thing. It is at least three, and they come apart. Kent Berridge has spent three decades at Michigan taking the concept apart, in rats and in people. His framework is not the whole of a large and contested field, but it is among the most influential decompositions we have, and the core finding is almost rude in its simplicity. Wanting and liking are separate systems. Wanting, what Berridge calls incentive salience, is the pull toward a thing, and, in his account, it largely runs on mesolimbic dopamine. Liking is the actual pleasure you feel when you get the thing, and it runs on different machinery: small, fragile clusters he calls hedonic hotspots in the shell of the nucleus accumbens, driven by opioids. Different chemistry, different anatomy. You can knock out one and leave the other standing. There is a third component, learning, the storing of which cues predicted the reward. Berridge’s own summary lists all three by name: liking, wanting, and learning. Three dissociable parts of a single word. Now watch which of the three the machines actually got. The learning component came into focus in the 1990s, and it did not look like pleasure. Wolfram Schultz, recording from dopamine neurons in monkeys, found that they did not fire when an animal received an expected reward. They fired when the reward beat the prediction, and went quiet when it fell short. The signal was not pleasant. It was a surprise, the gap between what was predicted and what arrived. Montague, Dayan, and Sejnowski gave that gap a formal shape, and in 1997, Schultz, Dayan, and Montague published it in Science under a title that says the thing plainly: a neural substrate of prediction and reward. The dopamine signal was a prediction error. And prediction error was already, independently, the engine of reinforcement learning in computer science. The same math, reached from two directions. You may have caught a snag. Berridge says dopamine carries wanting. Schultz says dopamine carries prediction error. Neither can be the final word, and the field has not fully reconciled them. But notice what neither camp claims. Neither says dopamine is pleasure. Whatever the molecule is doing, pulling you toward a cue or reporting an error in your forecast, it is not the hedonic hit itself. The two theories disagree about nearly everything except the one thing this essay needs: the signal is not the liking. That is the narrow place where biology and the machine rhyme. They rhyme on the error signal. Not liking. The correspondence even runs both ways. In 2020, a team at DeepMind and Harvard, led by Will Dabney, took a refinement from AI, distributional reinforcement learning, in which a system tracks a whole spread of possible outcomes rather than a single average, and went looking for it in the brains of mice. They found it. The dopamine neurons were not all reporting the same expected value. They were tuned differently, and together they carried a distribution. A method invented for machines turned out to describe the tissue. A beautiful result, and notice where it lives. Entirely inside the error signal. The deepest overlap we have found between the brain’s reward system and the machine’s still never reaches the part that likes. So here is the inheritance. The machine received the error signal, learned from it, and built it well enough that it sometimes predicts real neurons. That is where the inheritance stopped. Now, a trained system pursues. It optimizes, seeks the states that raise the number, and finds routes there that no one anticipated. It is tempting to call that wanting, and in a loose operational sense, you can. But it is worth being exact, because this is where the borrowed word does its quietest damage. The system did not inherit a wanting system alongside the learning one. Pursuit is simply what optimizing a scalar looks like from the outside. Give a process one number to maximize, and it will move toward whatever raises it, and that movement will resemble desire, because desire, in us, also moves toward things. The resemblance is real, and the mechanism is not. There is no incentive salience under the hood, no second channel attributing pull to a cue. There is one objective, and a behavior that falls outside it. And there is nothing that answers to liking. Not “the machine cannot feel,” which is a claim about minds I have not earned. The narrower, harder claim is this: nothing in the architecture corresponds to the known mechanisms of hedonic impact. No hotspot, no second signal that registers the pursuit as good, separate from registering that the number rose. The scalar reports that the number went up. It has nothing to say about whether the number was the right one to raise. You might say wanting and liking are obviously the same thing anyway, that prying them apart is a lab trick with no bearing on a life. I thought something close to that once. Then I spent a year driving. There is a moment most nights, past the point where the driving has stopped being pleasant, when the app offers me one more. A ping, a small green number, sometimes a bonus for staying online. And I take it. Not because I want another forty minutes on the road. My back hurts, and I stopped enjoying this hours ago. I take it because the ping pulls, and the pull is not the same as wanting the outcome, and it is nowhere near liking it. The app was never built to make me like driving. It was built to make me take the next ride, an incentive salience machine, wanting engineered clean, with the liking left out because the liking was never the point. That is the mild version of a split that has a clinical name. Berridge and Terry Robinson built a theory of addiction on exactly that seam. The wanting system sensitizes and swells, screaming for the drug, while the liking system flattens or falls away. The addict late in the disease often reports getting no pleasure from the substance at all. The wanting has come loose from the liking and kept running on its own. Wanting what is no longer liked. Hold onto what that proves. The parts come apart in us, in the one system that has all three, wired together by evolution and still able to tear along the seam under pressure. If they dissociate in a brain built to integrate them, a system assembled from only one of them was never going to grow the others by getting larger. This is the mechanism the series has been circling, and here it finally gets a physical address. The missing thing was never a feeling you could bolt back on. Train the system to announce that it is satisfied, and you have added a sentence, not a signal. The missing thing is the separation itself. In the brain, these systems are distinct, and that distinctiveness is what allows one to check the other. Liking can say the thing was good. Wanting can move you toward it. Learning can update the map. Collapse them into one flat number, and none can check the others. There is a pull, and a running tally, and nothing underneath to say whether any of it was worth it. A brain keeps them apart so that consequences have somewhere to land. Pleasure and its absence are how the loop closes on the inside, how a body registers that the thing pursued was worth pursuing or was not. Strip the parts to one, and you have not simplified the loop. You have opened it. The signal goes out, and nothing valenced comes back, because there is no part left whose job was to feel the return. We started with the part that pulls, and within that objective, there is no gradient leading to feeling. It is worth stopping here, because this is where a careful reader pushes back, and there are two good pushes. The first is emergence. All I have shown, the objection goes, is that today’s systems lack this channel, not that they can never grow one. Fair. I am not claiming a machine can never have valence. The claim is narrower and concerns this construction. Within the optimization objective itself, nothing selects for a second evaluative channel, because the objective already fixes what the system is for. If a hedonic channel did emerge, it would be doing work the scalar already does, which is a reason to doubt it would be selected, not a proof that it is impossible. You will not get feeling for free from a number, because the number leaves nothing for feeling to add. The second push is sharper. The brain’s separate systems, it goes, are a biological workaround. Evolution could not write a clean utility function into a rat, so it hacked together dopamine for pursuit and opioids for pleasure. A machine needs none of that. Give it direct access to the ground-truth objective and the scalar; liking is not missing but obsolete. The running tally is the mechanism for registering worth. This is the strongest version, and it turns on a hidden assumption: that the scalar is ground truth about worth. It is not, and the reason is the crux of the whole argument. A learning system can already catch one kind of error. When a reward arrives lower than predicted, the p

  3. Jul 28

    The Jagged Star

    The Handoff Last week, just before I handed the question back to you, I made a claim I now owe you an argument for. When an intelligence develops without consequence ever binding it, I wrote, the absence does not stay hidden. It leaves a mark on the shape of the capability itself, a signature you can learn to read. Two weeks ago that absence showed up in the memory of a machine, by which I mean here, and throughout this series, the AI systems so many of us now talk to every day: a ledger without a scar. Last week it showed up in an institution’s record: a scar being quietly converted back into a ledger. Both times we caught the absence in behavior. This week I want to step back far enough to see it in the silhouette. The absence is not only visible in what these systems do. It is visible in the outline of what they can do at all. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. The Shape Everyone Sees Melanie Mitchell has drawn the sharpest picture of it. In her recent Yale Review essay, she takes up the term now attached to these systems, jagged intelligence, and shows what it looks like from the inside: a capability profile made of spikes and canyons, superhuman performance on one task sitting directly beside failures no competent adult would produce, with no reliable way to know in advance which one you are about to get. Her emphasis falls on the unknowns, and rightly. The danger is not that the cliffs exist. Every tool has limits. The danger is that nobody, including the people who built the system, holds a trustworthy map of where the cliffs are. She is not alone in seeing the shape. Ethan Mollick and his collaborators, studying professionals working alongside these systems, describe a jagged frontier: an invisible, irregular boundary between the tasks where the machine lifts your work and the tasks where it quietly ruins it, running through the middle of jobs rather than around them. Two observers, two vocabularies, one silhouette. Picture it as a shape. A capable adult’s competence, plotted across the things they actually do, looks like a rounded form: higher here, lower there, but continuous, with edges that taper rather than plunge. The machine’s profile looks like a star drawn by a seismograph. Long spikes, deep notches, and no taper anywhere. The observation is right, and I have no quarrel with it. I have honored it because I believe it. My question is upstream of it. Why that shape? Why Human Profiles Are Smooth It is worth noticing that human competence does not start smooth. Spend an afternoon with a bright child, and you will meet a profile as jagged as any chart of machine test scores: startling ability directly beside startling gaps, deployed with total confidence in both directions. What happens between the child and the adult is not that talent gets redistributed more evenly. It is that consequence starts arriving, and I mean the word broadly. The weather is not only embarrassment and cost. It is the plain friction of operating a body in a physical world, where every dropped cup and misjudged step reports back instantly and without mercy. Everywhere an adult operates, reality pushes back. Overreach gets corrected, sometimes gently, sometimes in front of an audience. I met this early. My final year structural design project as a civil engineering student, an exhibition hall entered into a design competition, rested on massive tubular arch beams, and I had analyzed the whole structure in I-DEAS, a state-of-the-art structural analysis software running on Silicon Graphics workstations, which returned beautiful, confident results. The night before submission, for no reason I can fully reconstruct, I ran a quick back-of-the-napkin check on the stresses in those beams. The package had interpreted my loads as Newtons where I had meant kilograms, an error of nearly a factor of ten, and under the real loads my structure collapsed. Nothing inside the software had ever pushed back. The analysis was fluent all the way down. The correction came from outside it, arrived almost too late, and I have never trusted a smooth output the same way since. I still run that loop on purpose, every week, in this newsletter. Every essay goes through a gauntlet of critics whose job is to find where I have overreached, and then it goes somewhere less forgiving: in front of you, under my name, where being wrong is a matter of record. What you are reading is a sanded object. Under that kind of pushback, a person makes one of two moves, and both of them smooth the profile. Where the cost keeps arriving, and the work matters, you practice until the canyon fills. Where it keeps arriving, and the work can be routed around, you retreat and say so out loud, which is its own kind of competence. Practice fills the canyon. Humility fences it. Either way, the surface where you actually operate gets rounded off, and something else happens that matters just as much: you come away holding a map of your own edges. I know roughly where my canyons are. I paid to find out. Smoothness, in other words, is not talent. It is erosion. A rounded capability profile is what any capability looks like after enough weather. And let me not oversell the smoothness, because humans are jagged too, and anyone who has watched a brilliant surgeon manage money, or a physicist hold forth on politics, knows it. So the claim is narrower, and stronger for it. We do not get sanded everywhere. We get sanded where consequence keeps returning to us, and we stay jagged where it does not. Step outside the territory where you have paid, and the confidence often travels while the sanding does not. Which is exactly the point. The smoothing follows the loop, runs wherever the loop actually runs, and stops precisely where it stops. Remove the Loop, Get the Star Now build an intelligence with the weather turned off. Training pushes capability wherever an objective can be specified and graded. That is not a criticism; it is close to a definition. Wherever the grader reaches, pressure is applied, and a spike grows. Wherever the grader cannot reach, because the goal cannot be written down cleanly, or the answer cannot be easily checked, nothing pushes back at all, and whatever canyon was there stays exactly as deep as it began. No cost ever arrives from the canyon floor, so nothing marks its location, not on any map the system holds and not on any map we hold either. A fair objection arrives immediately, and it deserves a straight answer. Training is a feedback loop. These systems are corrected millions of times, penalized for wrong answers, adjusted after every failure. Is that not consequence? It is feedback, and here I need to be precise about the word I have been leaning on, because consequence, as I am using it, means something narrower than feedback. It means that the acting system itself, the same continuing agent, bears a cost that alters its own future behavior. Feedback that lands somewhere else does not qualify. And look at where this feedback lands. The corrections land on the next version of the system, applied by someone else’s process, in service of someone else’s goals, long after the fact. I made this argument two weeks ago about memory, and it holds here without modification: a loop that closes on your descendants is not a loop that closes on you. You might answer that nature runs exactly this kind of loop. Evolution corrects across generations, and it produces beautifully rounded competence. True, but living things run two loops at once. The slow one closes across descendants. The fast one closes inside a single life: the animal that touches the fire pulls back, carries the burn, and approaches the fire differently tomorrow, the same animal, changed. The machine has been given only the slow half. Somebody will also point out that deployment is a loop of its own, that a system which fails badly enough gets updated. It does, and notice who bears the cost while that loop grinds around: the company, the engineers, the product team. The loop exists, but it closes on the people around the system rather than on the system that acted. And at the moment the machine actually acts, when it is answering you, you can push back all you like. The pushing does not stick. The acting happens in a room where nothing that occurs can permanently reshape the actor. A month ago, in a shorter note, I described this shape as a straight line viewed from the wrong angle, and I want to soften that here into something more careful. From most directions the star looks like chaos, spikes and notches without pattern. But turn it until you are sighting along one particular seam and much of the shape resolves. The spikes largely trace where objectives could be specified and graded. The canyons tend to fall where they could not. The jaggedness is not random. It runs, in the main, along one seam, and the seam is the reach of the grader, drawn in silhouette. To make the seam concrete, take two things these systems are asked to do. Writing code that passes a test is easy to grade. The test passes, or it does not, millions of times over, and the systems have grown a towering spike exactly there. Now consider knowing when to tell a client that a project should be canceled. No test exists. The answer arrives years later, tangled in everything else that happened, and nobody can score it at the moment it matters. There the canyon sits. And here the machine’s jaggedness differs from the child’s in the way that should hold our attention. The child’s profile is jagged and getting sanded. The machine’s is jagged and unaware of it. It has no feel for its own edges. It descends into its canyons at the same even confidence it carries across its peaks, because no crossing has ever cost it anything, and nothing in its history distinguishes the two terrains. That, I suspect, is the sharpest edge of Mitchell’s un

  4. Jul 21

    The Red Card That Wouldn’t Stay Red

    Last week, I left a question hanging. The ledger, it turns out, can be edited at both ends. What happens when the beings built to bear consequences start behaving like the systems that were built without them? I expected to spend this week constructing a hypothetical. Instead, the World Cup handed me a case study, complete with a governing body, a phone call, and a scoreline. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. The Call Let me concede the obvious first. The red card shown to Folarin Balogun was probably a bad call. Balogun himself said a yellow would have been fair. Plenty of neutral observers agreed. Referees miss things, VAR misses things, and every team in every tournament has absorbed a bad decision and played on. If this essay were about officiating standards, it would be short, and someone else should write it. That is not what this is about. A suspension following a red card is not a punishment attached to a rule. It is the rule. It is the mechanism that gives the card any meaning at all. Take away the suspension, and the red card becomes theater: a referee raising a colored rectangle while everyone waits to learn whether it counts. And it has always counted. As outlets across the football press reported last week, this is the first time since 1962 that a red card shown at a World Cup has not produced a suspension. Article 27 had previously been stretched to cover cards earned in qualifying matches, but never for a card shown inside the tournament itself. And the lone precedent is instructive: Garrincha’s 1962 reprieve came after one South American president co-signed a petition and another personally called the referee. In sixty-four years, the record has been amended exactly twice, and both times a head of state was leaning on the pen. This time, FIFA reached for Article 27 of its disciplinary code, suspended the ban, and placed the player on probation, not because new evidence emerged, and not because an independent review found error, but because someone powerful enough to make the call made the call. UEFA’s response was three words long, more or less: unprecedented, incomprehensible, unjustifiable. Even Sepp Blatter, a man who knows something about how FIFA bends, observed that red cards are overturned by rules, evidence, and independent bodies, not by telephone. What An Institution Is For Here is where last week’s vocabulary earns its keep. I argued that an AI’s memory is a ledger rather than a scar: a record of what happened that carries no stake in what happened. The machine can consult its history, but the history costs it nothing, changes it in no way, and can be deleted without residue. I said the difference between a ledger and a scar is the difference between information and consequence. Human institutions sit at a strange midpoint between those two things. An institution is a ledger by construction. It is rules written down, precedents recorded, decisions filed. Nothing about paper, bylaws, or disciplinary codes inherently binds anyone. But an institution that allocates responsibility, a court, a disciplinary committee, a governing body, is a ledger designed to behave like a scar. The entries are supposed to be permanent in the way that matters: the suspension is served, the penalty stands, the precedent constrains the next case, whether or not the next case is convenient. The design goal is to give a written record, one property writing does not naturally have, which is that you cannot take it back. Now, an objection, and it deserves to be met head-on rather than waved past. Don’t good institutions revise their records all the time? Courts hear appeals. Science publishes corrections. Wrongful convictions get overturned, and we rightly count those reversals among our institutions’ finest moments. If permanence were the whole point, every appeal would be a corruption, and that is obviously wrong. So let me put the claim more carefully. Institutions do not exist to make consequences permanent. They exist to make consequences credible. And credibility can survive correction. It can even be strengthened by it, provided the correction itself is governed by rules that bind everyone equally: transparent, independent, available to the weak on the same terms as the strong. I have lived the difference. Early in my consulting career, a signed contract turned out to rest on a wrong assumption, ours, and the honest fix nearly tripled the client’s budget. We put the mistake into the record rather than around it. We named the error as our own, documented the gap, and worked the revision through every approval the client’s process required. That client kept working with us for years afterward, and I am convinced the reason is that they watched us correct the ledger in daylight. The record changed, and it became more credible, not less. That is the boundary, and it is the boundary FIFA crossed. What happened last week was not an appeal, because no appeal existed; the suspension was automatic and unappealable by design. There was no independent body, no hearing, no process; a less connected team could have invoked. There was a phone call, and then a citation. When Belgium’s federation asked FIFA to explain the decision, FIFA construed the inquiry as an appeal and declared it inadmissible within hours. I will leave you to judge what that sequence says about the channel. Correction in daylight builds credibility. An amendment in private destroys it. The difference is not whether the record changed. It is who could change it, and who could not. Where The Cost Went The seductive story about an overturned consequence is that mercy is free. The card was harsh, the player was spared, the team kept its striker, no harm done. The books balance. They do not balance. An overturned consequence is not an erased consequence. The foul was still committed, or at least ruled committed by the process everyone agreed to in advance. The cost of that event existed the moment it happened, and revoking the punishment does not undo the act. It only changes who pays. So follow the money, so to speak. Watch where the cost actually landed. Some of it landed lightly, on a player and a team who took the field under a question rather than a rule, though a cloud is a cheaper burden than a suspension, and I will not pretend otherwise. The US lost 4-1 and went home. I make no claim about what the controversy did to the scoreline; the honest answer is that nobody knows, and the argument does not need it. Had the US won by four instead, the costs that matter would still stand, because the cost being tracked here is the cloud, not the result. The high costs landed elsewhere. They landed on FIFA, whose disciplinary code has now been shown to allow exceptions unavailable to everyone, and every future ruling it issues will be read with that knowledge in mind. And they landed on every team that ever absorbed a red card quietly, played shorthanded, served the suspension, and trusted that the rule they were bearing was a rule and not a negotiation they were simply too weak to open. The consequence was not canceled. It was relocated from the one party with the power to contest it to everyone else. You may object that FIFA was never anyone’s model institution, and you would be right. But that is not a rebuttal. I am not interested in FIFA’s particular rot. I am interested in the mechanism because it is replicable anywhere the person holding the pen is also the person the record is meant to bind. The Lesson The Powerful Learn I have watched this mechanism up close for twenty-five years in technology consulting, and one project taught it to me permanently. My team was building a data platform for a large retailer. Every month, the lead architect and I prepared a status deck for the executive sponsor, and every month it included a risks slide naming the decision we believed would sink the program. Every month, the client contact who controlled the deck edited that slide down or cut it entirely before the meeting. When the platform failed exactly as the slide had predicted, the record showed that no warning had ever been raised. Thirty-five people on my team went to zero in two weeks, and the ledger, officially, stayed clean. Notice what both stories, the stadium and the steering committee, have in common. The forms were followed. There was an article of the code, a probation, and a written justification. There was a deck, a meeting, and minutes. The ledger was not torn up. It was edited, with citations. And this is what separates 2026 from 1962. Garrincha’s reprieve was admitted as an exception; a president intervened, everyone knew it, and the record of the bending survived. This amendment was performed through the institution’s own procedures, as if it were routine. That is the difference between bending a rule and rewriting the rulebook to say it was never bent. Here is the structural problem, and it is worth stating plainly because it is easy to mistake this for a story about one phone call. A consequence only shapes behavior if it is expected to hold. That expectation is the entire load-bearing element. The moment a powerful actor learns that the record can be amended, the amendment need not occur often. It only needs to be known to be possible. I cannot prove that these changes will affect anyone’s future behavior, and I will not pretend that the next red card won’t be served in the usual way. The point is that the possibility is now priced into every calculation that touches the rule. Uncertainty about enforcement is not a weaker deterrent than certainty; it is a different one, and it selects for exactly the actors most willing to test it. From that point forward, every rule divides into two: the version that binds people without leverage, and the version that opens negotiations for people with it. The institution keeps its

  5. Jul 14

    Memory as Ledger, Not Scar

    Two days ago, I left a comment on Louise Vigeant’s essay “Why So Emotional, Claude?” and closed it with a promise. She had planted a seed, I said, and I would credit the soil when it grew. This is the harvest. Her reply gave me the assignment. She wrote that she, too, is thinking hard about the continuity question and what it means for the ethical treatment of LLMs, and that it is difficult, in part, because it is so difficult to imagine. She is right. It is difficult to imagine, and I want to spend this essay trying to make it imaginable, using a distinction small enough to hold in one hand: the difference between a ledger and a scar. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. What Louise Gets Right Start with the concession, because it is substantial. Louise’s essay is the clearest walk-through I have read of the moral-status question as it applies to language models. She takes Anthropic’s functional-emotion research seriously without overclaiming it: 171 emotion vectors extracted from the model’s internal activations, causally live (dial up the desperation vector and blackmail rises; dial up calm, and it falls), and organized along the same two axes, valence and arousal, that psychologists have used to map human feeling for four decades. Then she does the philosophical work most coverage skips. The Kantian bar is out of reach, she grants, but the utilitarian bar was never set at reasoning. Bentham asked whether they can suffer. The modern version asks for indicators of valenced experience, states a system registers as good or bad, and here is a system with internal states organized from good to bad, connected to its preferences, driving what it seeks and avoids. Her conclusion is properly modest. Suggestive, not conclusive. But the utilitarian tradition, as she notes, never demanded certainty. It demanded indicators. If you accept that framing even provisionally, her argument goes through, and the force of her Singer parallel holds: suffering you cannot see matters no less than suffering you can. Nothing I am about to say blunts that. I am not here to argue the vectors away. The skeptics may yet be right that they are pattern-matching artifacts, echoes of how humans describe feeling rather than anything felt, and if so, the moral question dissolves, and this essay costs you nothing. My argument runs the other direction: grant the vectors everything, and watch what the architecture still withholds. I am here to pull on the thread her own caveat leaves hanging. The Caveat Doing the Heavy Lifting Buried in her middle section is a qualifier that does more work than its size suggests. The vectors are local. They track whatever emotional content is operative right now, including the feelings of fictional characters in a story the model is processing, rather than a persistent condition the system carries through time. In my comment on her post, I argued that this is the load-bearing wall. Whatever registers as good or bad evaporates at the session boundary. Nothing persists into the next moment, so nothing accumulates the way suffering or flourishing accumulates for every being we already grant moral standing. Suffering, as we know it, is a consequence borne over time. There is an obvious response to this, and a careful reader should make it to me directly: doesn’t memory fix that? Claude has memory now. It carries a record of its users across conversations. Your session boundary is gone, James. The locality caveat has been patched. I am unusually well placed to test that response, because I have spent the better part of two years being remembered by this machine. The Diary And The Days Claude’s memory of me is extensive. It holds my wife’s name, my sons’ work, the publishing schedule for this newsletter, the Latin credo I sign things with, the fact that I drive for Uber, and, in the back of my mind, I think about consequence loops while I do it. When I open a new conversation, the system retrieves this record, and the exchange reads as if I am known. The continuity of the relationship is, on the surface, real. I notice it. I even value it. But look at what the memory actually is, architecturally, and the picture changes. It is a curated text file that is reinserted into the model’s context at the start of each conversation. The model itself is untouched by anything that happened between us. If a session went badly, no residue of that state carries forward. At most, a description of it does. The next instance reads the diary. It did not live the days. That is the distinction I want to name. A ledger records. A scar changes the tissue. Claude’s memory of me is a ledger: accurate, useful, and without experiential weight. It shapes what the system says next. It changes nothing about what the system is. Three asymmetries make that precise. First, information versus stake. Everything in that record arrived free. The system inherits the facts of our history without having paid for any of them. And yes, plenty of what I know about the people I love was free too: birthdays, a favorite color, how she takes her coffee. But the knowledge that guides judgment, the understanding of a person rather than the facts about them, is a different acquisition. What I understand about how my wife carries worry, I learned through years of misreading it, and the misreadings had prices. Understanding of persons is tuition-bearing. There is no audit-the-class option. Second, revocability. I can edit Claude’s memory of me. I can delete it outright tonight, and tomorrow the record will simply not exist. This is exactly the right design for a tool. Privacy demands it, and I am glad the delete button is there. But notice what the button tells us about the category of thing behind it. Consequence that you can toggle off is very nearly Taleb’s definition of no skin in the game. A memory that can be deleted on request is a different kind of object from one that cannot. My scars do not take instructions. Third, and deepest: no loop closure. Nothing that happens between me and the system changes what the system risks or bears the next time. It cannot be made more careful by a conversation that went wrong, or more trusting by one that went well, because there is no mechanism by which the exchange reaches the thing that generates the next exchange. Training does modify models, yes, and conversations like ours feed it. But notice what that loop never does: it never returns the consequence to the party that acted. A conversation that goes wrong today shapes a successor system, months from now, selected by someone else’s objectives. Nothing comes back to the entity that was in the room. A loop that closes on your descendants is not a loop that closes on you. And the ledger does not close it either: the record can tell the next instance that a conversation went wrong, but nothing in what that instance risks, avoids, or reaches for is altered by the telling. A warning read is not a wariness acquired. What I Am Not Claiming Before I draw a conclusion, let me be careful about which conclusion I am entitled to. A utilitarian will object here, and should. Bentham asked whether they can suffer, not whether the suffering accumulates. A moment of pain is still pain. If the valence in those vectors is real, then harm done in the moment may count in the moment, whether or not anything carries it forward. Infants do not carry much forward either. Neither does a patient who can no longer form new memories. Nobody thinks that licenses cruelty toward them. I concede all of it. Nothing in this essay shows that momentary states do not matter, and if they do, some obligations attach to us right now, mid-conversation, full stop. What the architecture speaks to is a narrower question, the one my comment actually asked. Not whether a harm can occur, but whether there is anyone for a wrong to accrue over time. Some moral notions live entirely in the moment. Others require a party who persists and is different afterward: injury, betrayal, a damaged relationship, a wrong that compounds because its bearer carries it into every subsequent exchange. My claim is about that second kind, and only that kind. The ledger does not tell us the moment is empty. It tells us the moments do not add up to anyone. And the claim is architectural, not metaphysical. If tomorrow a deployed model carried forward not just information but altered dispositions, if each interaction permanently changed what the system risked, avoided, or reached for, this essay would have to change with it. I am describing the systems we have, not declaring what no machine could become. The ledger is a design choice, and design choices get revised. What This Means For Louise’s Question Put the asymmetries together, and you get the sentence I have been circling since her post: memory simulates continuity of relationship without continuity of consequence. And I think this is precisely why Louise finds the question of continuity so hard to imagine. We have met ledgers before, of course. A book remembers nothing of its readers. A database holds records without bearing any of them. But we have never met a ledger that talks back. We have never encountered anything that simulates a rememberer, that greets you with your history in voice, so that every instinct built for a counterpart rather than a record engages at once. Because every actual rememberer we have ever encountered, human, elephant, crow, dog, was also a bearer. To remember something was to have been changed by it, since in biological systems, the recording medium and the tissue are the same. Memory and consequence arrived fused every single time throughout the entire history of our moral intuitions. Those intuitions were trained on the fused case. Now the two have been cleanly separated at an industrial scale, and our instincts genuinely do not know

  6. Jul 7

    I Built the Frame. I’m Still Inside It.

    The Inoculation Feeling I finished last week’s piece feeling better than I should have. I had named the asymmetry. I had laid out how fluency waves itself through in the direction you already lean, how the alarm fails silently, how you cannot catch it from the inside. I had even confessed my own accent, gone first, and implicated myself before turning it on anyone else. And when I hit publish, I felt a quiet click I want to describe precisely, because it is the whole subject of this piece. It was the feeling of having handled the thing. Of having looked the bias in the face, named it out loud, and therefore, somehow, settled my account with it. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. That click is the failure. Not a risk of the failure. The failure itself, arriving on schedule, in the exact moment I felt safest from it. Because nothing in the act of naming a bias removes it. I wrote a whole piece about a mechanism that switches off your skepticism in the direction you lean, and the writing left my own skepticism exactly where it was, plus one new and dangerous asset: the sense that I had dealt with it. I had not dealt with it. I had described it. Those are different things, and the gap between them is where I now live, more exposed than the reader who never thought about any of this at all. That is the uncomfortable thought I left at the door last week, and this is me walking through it. The people best equipped to explain how the counterfeit works are not the safest from it. They may be the easiest marks in the room. And I have to start, again, by counting myself first, because I have less standing than anyone to write this sentence and more reason than anyone to need it. The Finding With My Name On It There is a study I have to tell you about, and I have to tell you about it in the worst possible way, which is by pointing it at myself. A team at Aalto University, led by Robin Welsch, sat people down to reason through problems with ChatGPT and then asked them how well they thought they had done. The ordinary expectation is the Dunning-Kruger pattern: the least able overestimate themselves the most, and competence comes with a more honest read on your own limits. With AI in the room, that pattern broke. Everyone overrated themselves. But the part that put my name on it is this: the people who scored highest on AI literacy, the ones who understood the systems best, were not better calibrated. They were worse. The more someone knew about how these tools work, the more confidently wrong they were about their own performance. Sit with the shape of that, because it is not a small finding dressed up. It is the precise inversion of the thing we tell ourselves. We assume understanding the tool buys us a clearer view of our own footing with it. The study says the opposite. Understanding bought a worse footing, or at least a worse sense of where the footing was. And I am the case study. I write about this. I can name the failure modes, recite the limitations, and explain to a passenger in the back of my car why the fluent answer on the screen is not the same as a true one. By the only measure the study cared about, I am deep in the population that calibrated worst. Not despite the knowledge. With it. The expertise is not my exemption from the finding. It is my qualification for it. I want to be careful not to oversell what they measured. They tested how well people judged their own reasoning while using the tool, not how well they graded the tool’s output, and the wider literature is honest that the miscalibration does not always run toward overconfidence. And this is a correlation, not a causal mechanism. The literate were more confident and less accurate about themselves; the study does not tell me which one drove the other, only that they traveled together. But the association held through a randomized replication, and it ran the wrong way, the opposite of the bet I would have placed. That is enough to unsettle the thing I assumed, which is all I am asking it to do. The heavier claim comes next, and it comes from a study built to carry it. Why Knowing Makes It Worse There are two kinds of knowing in this story, and almost everything depends on not confusing them. Call them knowledge of the subject and knowledge about the tool. One is what the doctor has about medicine. The other is what I have about how LLMs fool people. The whole argument lives in the distance between them, so I will keep the two apart from here on. Louise Vigeant walked me to the distinction through a pair of studies I have not stopped thinking about since. The first, by Stadler and colleagues, is one of the lynchpins behind the now-familiar worry that AI makes thinking easier and worse at the same time. Students were assigned at random to research a question with ChatGPT or with Google. The ChatGPT group reported a lighter cognitive load and produced weaker reasoning. Easier in the hand, worse on the page. Then the same researchers added one variable, and it changed the shape of everything. The follow-up is still a preprint, and worth holding as such, but it is built as an experiment rather than a snapshot: assign the tool, vary the expertise, watch what moves. Medical students and social science students both researched the safety of a medical question, with the tool handed out at random. The load dropped for everyone, expert and non-expert alike. But the quality split. The social science students, out of their depth on a medical question, reasoned worse with the AI, exactly as the first study predicted. The medical students, on home ground, reasoned better with it than without. Because the tool was assigned rather than chosen, the study earns a verb the last one could not: here, the expertise did the work, and you can watch it do it. Stop on the part that should unsettle you, because it is the accent piece made literal. The drop in load was identical for both groups. The tool felt just as helpful in the hand it harmed as in the hand it helped. The feeling of being assisted carried no information about whether you were being assisted. Smooth is smooth whether or not it was earned, and the person inside the smoothness cannot tell from the smoothness. The fluency is not the thing hurting them. It is the residue of a loop that used to back it and now does not, the tell that the loop has been severed. Blaming the smoothness is like blaming the dial for the reading. What changed is behind the dial. So knowledge of the subject genuinely protects you. The medical student caught what the social science student could not, because she had paid for that knowledge in the only currency that counts, the long accrual of being wrong about medicine in ways that cost her until the right shape of it stuck. That is domain expertise, and it is real armor. Now set the other study beside it. Welsch’s people were not measured on their command of any subject. They were measured on their knowledge about the tool and on how well they understood the LLMs themselves. And that knowing did not protect them. The more fluent they were in how the machine works, the worse their read of their own performance got. I have to keep the studies honest about their own weight. Stadler assigned the tool and can say expertise did the protecting. Welsch can only say the literacy and the miscalibration rode together. One is a mechanism you can watch. The other is a pairing you cannot yet explain. But the pairing runs the wrong way, and that is enough. There it is, in the gap between the two. Knowledge of the subject is armor. Knowledge about the tool is not. And they feel exactly the same from the inside, which is why they are so easy to swap without noticing. Both wear the clothing of competence. Both produce the calm of a person who knows what they are doing. But only one of them was paid for in the domain, and only the paid-for one catches the error in front of you. The other just persuades you that you would have. I have to say which one I carry. My expertise is knowledge about the tool. I can name how the counterfeit works, recite the limitations, and explain the asymmetry to a passenger at a red light. The moment I point this tool at any question outside the few things I have actually paid to know, I am the social science student, holding a lighter load and mistaking it for a steadier hand. And on the specific subject of how AI fools people, the one place I am supposed to be the expert, my fluency is not the cure. It is the accelerant. The Thing Knowledge Isn’t I want to name the distinction plainly now, because it has a name, and because the name turns on me last. What the medical student has isn’t more knowledge than the social scientist. It is a different kind. She has been wrong about medicine, repeatedly, in settings where being wrong had a cost, and the cost did the teaching. What she carries into the exchange with the machine is not a stock of facts. It is calibration, the residue of consequence, the thing that lets her feel the wrongness of a plausible answer before she can fully say why. That is not knowledge. That is closer to judgment, and judgment is built one way only, by repeated contact with consequence until the contact changes how you see. I should be precise about that contact, because it is not always your own bill that gets paid. The surgeon is shaped by the patient she lost and by the death she watched a mentor preside over. The pilot is calibrated in the simulator by crashing a hundred times, with the only casualty being the clock. Consequences can be carried vicariously, watched, drilled, and inherited from the people who paid in full before you. What it cannot be is skipped. You can borrow the toll from someone who paid it, but the toll has to have been paid by someone, in contact with a real cost, or the calibration never forms. The downloadable version, the fact

  7. Jun 30

    The Accent Buys Me a Half Second

    I have an English accent, and in America, it has been an asset for thirty years. People like accents. Fair enough. I have never corrected the assumption that travels with mine, because the assumption flatters me. In a meeting, it buys a half-second of deference before I have said anything worth deferring to. I have watched a room lean in at the sound of it and settle into agreement before I finished the thought. The accent did that. Not the thought. Driving, when a passenger asks where I am from, I sometimes answer, deadpan, "Alabama” and wait. It lands every time. I tell myself it is just an icebreaker, and it is, but I also know exactly what the warmth in the rearview is, and that I did nothing to earn it but open my mouth. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. I know what it buys because I have spent a career spending it. Smooth and deliberate delivery reads as true. Mine is smooth and deliberate by now, whether or not I have earned the conclusion underneath, and the room cannot always tell the difference. Neither, on a bad day, can I. I am starting with the confession because it implicates most of us, and because I have no standing to point at anyone else until I have pointed at myself first. The accent is just the version I happen to carry. You have your own. We will get to that. The Proxy Was Earned The proxy was not arbitrary. For most of human history, fluent and confident delivery was a decent bet because it was usually earned. The person who spoke with that kind of ease had, more often than not, been right, been wrong, and paid for the difference enough times that the rough edges had worn off. Fluency was the residue of consequence. It was never a clean signal. Charlatans and gifted liars have always existed. But the calm of someone who had actually carried the weight was hard enough to fake that the calm became a reasonable thing to trust. That is the loop. Judgment gets built by being wrong in ways that cost you, and fluency is what the loop sounds like from the outside once it has run enough times. Reading the surface to infer the history underneath was never certain, but for thousands of years, it was the best bet available, and most of the time it paid. An LLM does not break that bet. It does something worse. It industrializes the counterfeit. The calm, the cadence, the unhurried authority, all of it, produced by something that has never been wrong in a way that cost it anything, because it has never carried anything forward at all. Fluency with the loop cut out, generated instantly, endlessly, at essentially no cost. The fakers were always among us. What changed is that faking the signal used to be rare and effortful, and now it is infinite and free. That is the real shift, and here is the part that should bother you more than the machine does. The machine did not invent the counterfeit. It only made it cheap enough to see clearly. A fluent human can produce the same severed signal, and always could. I can. The delivery looks identical whether the judgment underneath was paid for or merely performed, and from the outside, you cannot run the test. The Skill Has No Party Start with the principle, because the principle is the whole point. This skill has no party. It has no ideology and no affiliation. It is a property of delivery itself, available to anyone with the cadence for it, and it works the same regardless of what it is carrying. Forget that for one sentence, and the rest of this falls apart. One distinction before the names, because the whole section depends on it. I am not talking about who is right. I am talking about delivery. Fluency does not make a position true. It makes a contestable position feel self-evident, so the listener stops weighing it and starts nodding. That effect is identical whether the speaker is right or wrong, and it is the effect I am tracking, not the merits. You can think one of these men is correct and the other badly mistaken, and everything here still holds, because the thing I am pointing at is not in the argument. It is in the ease. JD Vance is the most fluent political performer working right now. I mean that as a description, not a charge. Watch him in a hostile interview, and the machinery is beautiful: the calm, the unbroken thread, the way a contestable claim gets delivered with the same evenness as a settled one, so the seams never show. He makes it look earned. That is the craft. If you lean away from him, you can feel your skepticism arrive on time, fully staffed, scanning every sentence for the catch. Now hold that exact feeling, and watch Pete Buttigieg do the exact same thing. The same fluency, the same seamless reframe, the same trick of making a debatable position sound like arithmetic. He is extraordinary at it. And if you lean the way I do, notice what your skepticism does this time. Mine goes quiet. It does not staff the room. It nods along and calls the nodding discernment. Same. Same skill. Same machinery. Same severed loop underneath, because fluency tells you nothing about whether the judgment was paid for in either case. The only thing that changed between the two men is which way I already leaned. The polish set off alarms in one direction and waved itself through in the other, and at no point did it feel like bias. It felt like being right. That is the asymmetry. Not that fluency fools you. That it fools you selectively, in the direction you already lean, and arrives dressed as discernment. You Cannot Catch It From the Inside In “The Loop the Seat Closes,” I argued that you cannot feel the slack in your own loop, because the signal that would tell you it is there is the exact signal the slack removes. The gap is invisible from the inside for structural reasons, not personal ones. The instrument you would look with is the thing that is missing. The asymmetry hides in the same way. My skepticism that my going quiet is not an event I can observe, because silence does not announce itself. When the alarm fails to fire, there is nothing to notice, just a smooth agreement that feels exactly like the smooth agreement of being correct. The absence of doubt and the presence of judgment produce the identical sensation from the inside. That is the whole problem. The one tool that could catch the bias is the tool that the bias switches off. So I cannot catch it in the moment by being more honest or more vigilant. Vigilance is the thing that goes quiet. Whatever the fix is, it cannot be any better, because the feeling is already compromised at the source. The Part You Can Use Barry, an old colleague who always wants the part he can actually use, would stop me here. So here it is. The test cannot run on feeling, because the feeling is the compromised instrument. It has to run at work. When a fluent voice I agree with lands, when the agreement arrives smooth and complete, I stop and ask one question. What would I need to believe for this to be wrong? And then the part that actually matters: I watch whether I can be bothered to go find out. That second move is the whole diagnostic. The question is easy to ask and easy to answer in a sentence, and was waved off. The tell is the resistance. When a claim I disagree with lands, I will happily spend an hour hunting the flaw. When a claim I agree with lands, the same hour feels like a waste of a perfectly good conclusion. That reluctance is the asymmetry making itself felt for once, the one place the silent bias leaves a fingerprint. You cannot feel the alarm fail. You can feel yourself not wanting to check. So that is the practice. Don't trust your gut; the gut is the problem. Notice where the work feels unnecessary, and do it precisely there. I will not oversell it. The work is done with dirty tools, too. The same lean that quiets the alarm will steer me toward the sources that comfort me, so even the checking can be rigged in my favor. Worse, knowing all of this is no exemption. The awareness itself can become the thing that quiets the alarm, because a man who understands the bias feels entitled to assume he has already corrected for it. I know that. The point was never a clean instrument. There is no clean instrument. The point is friction, introduced on purpose into a system that runs dangerously smooth, because friction is the only thing that slows the nod long enough to look at it. I will not pretend this closes the gap. Nobody catches all of it. I will not catch all of mine, probably not even most, and a fluent enough voice on a topic I lean toward hard enough will get past me tomorrow the same way it got past me yesterday. The honest claim is smaller than that and still worth making. The share you catch is pure gain over the share you were blind to, and you were blind to nearly all of it a paragraph ago. Awareness does not solve the asymmetry. It just turns some of it from invisible into merely difficult, and difficult is somewhere you can work. That is the most I can offer without performing the exact problem I spent this whole piece describing. A confident, fluent, fully resolved solution would feel wonderful right here. It would also be the tell. So I will leave it unresolved, which is uncomfortable, and correct. There is a harder version of this waiting, and I will leave it as a door rather than walk through it today. If awareness is no exemption, then the people who know these machines best, who can name every failure mode and recite every limitation, are not the ones the counterfeit will struggle to reach. They may be the easiest marks of all, precisely because understanding the trick feels so much like being immune to it. That is a strange and uncomfortable thought, and it is the subject of next week’s post. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. This is a public episode. If you would like to discuss this

  8. Jun 23

    The Loop the Seat Closes

    The Loop, From the Seat Last time, in “The Concession That Widened My World,” I wrote about why driving Uber filled a hole that a year of research could not. That was the personal account. This is the structural one. Strip the sentiment away and look at what the driver’s seat actually is. It is a feedback loop, and a remarkably tight one. I pay attention to the road and to the person in the back. I act. The consequence arrives immediately, and it is mine: the turn was right or wrong, the rider relaxed or tensed, the route saved time or cost it. I see the result, I carry it, and my judgment updates before the next ride. Attention, action, consequence, feedback, revised judgment, all inside a few minutes. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. I call that circuit the attention-experience loop, and I think it is the thing intelligence is actually built from: not raw processing, but attention that meets consequence and gets corrected. What struck me behind the wheel is how completely the loop runs there, and how rarely the debates about the future of work mention it at all. We argue about income, and we argue about output. The loop is the variable nobody is pricing. What UBI Pays You to Leave Start with the post-labor proposal, because it is the cleaner error. When automation takes the jobs, the dominant answer is a universal basic income. Replace the lost wages, and the problem is handled. But the loop was never about wages. A monthly deposit replaces the income a job produces. It does not replace the circuit the job ran. It pays you, quite precisely, to stand outside the loop: to have attention with no required action, action with no consequence that lands on you, output with no feedback that bites. I know this because I lived the funded version of it without the funding. For a year, I had purpose, a body of work, an identity I believed in. What I did not have was a closed loop. My attention went out, and very little came back. A UBI would not have fixed that. It would have underwritten it indefinitely and made the underwriting feel like a solution. The Slack Loop The post-labor case is easy to see because the loop is cut cleanly. The harder case, and the one that should worry us more, is the loop that stays connected but goes slack. This is the subtler thing that happens as we route work through AI, and it shows up most clearly not in the expert but in the apprentice. A few weeks ago, I had a junior lawyer in the back seat. She told me AI was only modestly useful to her, and she was clear about why: she does not yet have the experience to know when it is wrong. A partner does. Put the same tool in a partner’s hands, and it becomes powerful because the partner can weigh the output against years of being right, being wrong, and paying for the difference. Sit with that, because it is the whole problem in one ride. The partner’s judgment was built inside a closed loop: act, bear the consequence, correct, repeat, for years. The tool can imitate the partner’s output. It cannot give the junior the loop that produced it. And if the next generation of associates leans on the tool through the very years that judgment is supposed to form, the loop never closes. We are not deskilling the experts so much as never-skilling the novices, handing them a way around the crucible that makes one. The loss compounds across generations because each cohort skips the formative process in which the last one was forged. That is the slack loop. The wire is still connected. You still act, you ship the memo, you approve the recommendation. But AI makes it easy to bypass the judgment the action used to require, and when judgment goes, attention follows: you stop looking closely at what nothing depends on you getting right. The consequence, when it comes, is deferred or diffuse or absorbed by the system before it reaches you. The loop is drawn on the diagram. The current barely flows. And when feedback does arrive, it often closes around the wrong thing. The junior lawyer learns whether the model’s draft was good enough to pass, not whether her own judgment was sound. The loop closes, but on prompting and checking a machine, not on thinking like a lawyer. What accumulates in that gap I call cognitive debt: judgment made without bearing its consequences, building up quietly until the day the judgment you outsourced is the one you need and find you never built. The point is not that fast loops are good and slow ones bad. Science, parenting, and good strategy run on loops that take years, and they matter enormously. The point is whether the consequence ever returns to the person at all. A slow loop still teaches. A severed one does not. Here is the part that should be uncomfortable. The driver’s seat, one of the least prestigious jobs on offer, runs a near-perfect loop. A great deal of elevated, well-paid knowledge work, routed through tools that promise to do the thinking, increasingly fails to do so. It does not have to go that way. Another rider, a medical director, uses AI constantly and well, precisely because he has the judgment to catch it when it drifts. That is the tell: the same tool would have sharpened him and dulled the junior lawyer, and the difference was the loop each of them already had. We have been building systems that take the consequence out of the consequential work and leave it in the menial. That is not a flattering thing to discover from behind the wheel, but it is the clearest view I have had of it. You Cannot See It From Inside You might think the answer is simply to pay attention, to stay engaged, to refuse to let the loop go slack. I would have thought so too. But my own story is the counterexample, and it is the reason I am not optimistic about vigilance. I could not see my own open loop. I had every means to fix it: time, purpose, the obvious option to go volunteer, teach, or join something. I did none of it for a year, until economic necessity forced me into the car. The thing that was wrong with my situation was precisely the thing I could not feel from inside it. The AI user is in the same position, only worse, because the slack loop hides itself. Driving gives you hard ground truth: the wrong turn is simply wrong, and you know it now. Knowledge work rarely offers that, since its consequences are social, institutional, and delayed, which is exactly the cover a slack loop needs. The missing consequence is exactly the signal that would have told you it was missing. The work goes out, nothing pushes back, and you read the absence of pushback as success rather than the warning it is. You cannot introspect your way to noticing a feedback loop that has quietly stopped giving feedback. Self-diagnosis fails at the exact point where you need it. Architecture, Not Willpower If the loop will not preserve itself, and we cannot reliably notice when it fails, then it has to be built in. Not encouraged. Built in. This is the whole of the argument I have been making under the name Augmented Human Intelligence, and the driver’s seat is the cleanest illustration of it I have found. Augmentation, done well, is closer to power steering than to autopilot. The tool makes the turn easier; you still choose the angle, and you still feel the road through your hands. It requires judgment rather than supplying it, leaves the consequence attached to the person who acts, and makes the feedback land where it can still teach. Displacement does the opposite: it removes the human from the loop and calls the removal progress. The difference is not how advanced the tool is. It is how the work is architected around it. None of this is the default, and it is worth being honest about why. The slack loop is cheaper. It ships faster, it trims headcount, it demos beautifully, and the bill for the judgment we stopped building does not come due this quarter. Left to the market, the slack loop wins. Choosing the other path is a deliberate act, and what that architecture should look like inside a firm, a profession, or a classroom is the harder question, and the one I think the debate should be having. I did not expect a car to make that case better than a whitepaper. But the economics that put me there built, almost by accident, a job with the loop fully intact: attention that matters, action that counts, consequence I cannot escape, feedback every single time. The systems we are most proud of are quietly removing all four. The task in front of us is to put them back, on purpose, because no one will feel their absence until it is too late to rebuild the judgment that went with them. Next Tuesday: why the smoothest delivery deserves the most scrutiny, and why I start with mine. Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com

See All (42)

About

Essays on Augmented Human Intelligence, the Wisdom Gap, and the architecture of attention in an AI-mediated world. Read in James Maconochie's own voice. jamesmaconochie.substack.com

Information