Forty-seven minutes on the two words that follow every objective function ever written, and on what the machines have and have not learned to write. Two voices this time: the narrator is Lily, the synthetic voice that carried Single Player and The Third Fraction; my own cloned voice comes in five short passages about the loop in my automation rack, a submarine, a battery, and a number I wrote wrong. The question I get most often, from people who run our optimizer in their own houses, is the same question every week. Why is it doing this, now? The battery is charging when it feels like it should be selling. Or it sits still at eleven at night at a price that looks like a gift. The feeling in your body, that you would charge it now, is not the same thing as the sum. Most weeks the sum is right and the feeling is wrong. I have not stopped getting the question, and I have not stopped having the feeling. This episode is about why those two things come apart, and it starts with the grammar every optimization problem in the world shares. First a line that says what you want less of: the cost, the fuel, the waiting. Then two words. Then a list of things that must hold no matter what: the fuse must not blow, the car must be full by seven, supply must equal demand. The first line is the wish. The list is the world. A solver’s whole job is to find the best version of the wish that breaks nothing on the list. And the solver takes you literally. That is its whole power and its whole danger, and they are the same property. The price is a proof Every day at noon, Central European time, the bids of twenty-seven countries close, hundreds of thousands of orders from sixteen exchanges, and one algorithm has twelve minutes to clear tomorrow’s price for every quarter hour of every bidding zone. Nobody in that room sets a price. The machine looks for the arrangement of buyers and sellers that leaves the continent best off, and once the all-or-nothing orders are settled the price falls out of the answer as a by-product. By a quarter to one it is on every trading screen. That by-product has a history. In 1938 a plywood trust in Leningrad asked a twenty-six-year-old mathematician, not an economist, how to schedule eight lathes across five kinds of veneer. Leonid Kantorovich found that the question had a shape, and that the method for solving it ran on a set of numbers, one per scarce thing, each saying what one more hour of that lathe would be worth to the whole order. He read them and saw what they were. They were prices, though no person had set them. He knew the word was dangerous where he lived, so he called them resolving multipliers. The booklet came out in 1939. The economics he built on it did not: the planning ministry told him in 1942 to do his job and let them do theirs, a colleague warned him that the man who talks about optimum is “the fascist Pareto”, and the book he wrote during the war waited seventeen years in a drawer, then appeared with a preface calling its conclusions inconsistent and incorrect. By then he had renamed the prices “objectively determined valuations”. Three names for the same numbers, each chosen for political safety. Across the war Tjalling Koopmans found the same numbers routing Allied merchant ships. They shared the 1975 Nobel prize. George Dantzig, who built the working algorithm for the US Air Force in the summer of 1947, was not on the citation; Koopmans gave a third of his prize money to the institute where Dantzig had led the work, in Dantzig’s name. What a solver actually does is easier to picture than to define. Set a cut stone on a table and tilt the table: the highest point is always a corner, never the middle of a face, because a straight wish keeps improving until a limit stops it, and where two limits meet is a corner. Dantzig’s method walks the corners uphill and stops when none is higher. And when it stops, it holds a proof as well as a corner. Searching finds a good answer. Optimizing also proves there is no better one. Nothing better exists, inside the set you declared. Whole numbers break the stone. A power station is on or off, so the solver splits the world in two, bounds each half, and throws away any half that cannot beat the best complete plan it already holds, without ever looking inside it. The problem that decides which stations run tomorrow is solved this way every day in the big American markets, and it is not solved to the end. It stops at a tolerance. Even the proof has a limit written into it. Then the other half of the proof: every limit carries a shadow price, what one more unit of the limited thing would be worth. For a grid the limit is a single line, supply equals demand, here, now, and its shadow price is the electricity price. In the American markets that is literal, per node, once the on-or-off decisions are fixed. In Europe it is one price per zone per quarter hour, read off the proof once the block orders are settled. Nobody chose the price on the screen. It fell out of the constraint. Which is the true ending of the Leningrad story. The planners did come round to the method, decades late, and it did not save the plan, because the solver had been handed an objective, tonnes and gross output and a number from above, and it priced that objective faithfully. What you are allowed to forget Richard Bellman’s idea, at RAND around 1950, fits in one sentence: whatever you did first, the rest of an optimal plan must itself be an optimal plan from wherever you now stand. So you can solve a chain of decisions backwards. But to do that you have to say what “wherever you now stand” means. You have to name the state, and a state is a decision about what you are allowed to forget. At my house, at four in the afternoon, the plan needs one thing: how full the battery is. How it got there, the cheap hour at three in the morning, the cloud at eleven, the car that came home early, can all be thrown away. The battery level is the whole of the state. And here is the turn: the state forgets how the battery got to forty percent. The cells do not. Every cycle leaves a mark in the chemistry, and the warranty keeps a count of the cycles the plan has already forgotten. The plan covers two days in quarter hours, 192 steps, with 401 possible power settings at each step, a number with hundreds of digits if you tried every path. Bellman’s curse of dimensionality. The state lifts it: keep every level instead of every path, and merge every path that reaches the same level. What comes out is not a plan but a table, what it is worth to be standing at each battery level at each quarter hour. The table has to start somewhere: at the far end of the two days, somebody has to write down what a kilowatt-hour in the battery is worth. That number is the seed the whole calculation grows from. Hold on to it. I have been inside a hull where the whole world arrived as sound. A submarine does not look. It listens, and nothing goes out. Even its own position is a guess, counted forward from the last fix by heading and speed. I worked in the engine room, not at the hydrophones, but I knew what the boat was: a machine for keeping a guess alive without ever giving itself away. The mathematics has a name for a state that is a belief, and one rule every plan has to obey: you may use only what you knew when you chose. At my house the rule is physical. Two days are planned, and fifteen minutes are real. The right decision that looks wrong The planner is told to weigh the worst tenth of outcomes, not the average. It plans on less sun than the forecast promises and pays a little on most days to be ready for the day that hurts. On most days that day does not come, and by morning you can add up what the caution cost. The feeling says it was wrong. What you see afterwards is what you had, plus information that did not exist yet. Psychology has measured how hard this is to hold on to. In 1975 Baruch Fischhoff showed that people told how a story ended rated that ending as having been likelier all along, from about a third to well over half, and could not undo it when asked. In 1988 Baron and Hershey showed that the same surgical decision, with the same odds, was graded better when the patient lived; asked whether the outcome should count, the same people said no, and used it anyway. Poker players call it resulting. The act’s real question is where a value lives inside the problem. Write “minimize travel time plus a fine for every kilometre over the limit” and the speed limit is for sale: a big enough hurry buys it. Write “minimize travel time, subject to never passing the limit” and no hurry can. Where you put a value is where you decide whether it is for sale. My planner does this in two rounds, and both planners I have run have done it from the first day: first it minimizes how far the house falls short of the comfort it promised, then it freezes that answer as a constraint and only inside it starts minimizing money. The house is served first and is not for sale. The bill is. And one value I have not placed anywhere is the battery itself. The cells are rated for more than eight thousand cycles, two a day for ten years. Should the plan count what it costs them to be used? I do not put a wear cost on the washing machine or the sauna. If I save the battery for later I am betting a cycle in ten years is worth more than one tonight, and maybe it is the other way around; maybe flexibility is worth more now than it will ever be again. Leave wear out, and the plan runs the battery as if it were free, because that is what I will have asked for. This summer I spent a month on which solver should sit in the loop and measured parity: everything argued about came to under one krona across three days of real decisions. Then I gave the planner perfect knowledge of the sun, the house and the prices, and it was worth a fifth of the bill. The money was in the objective and in the information, and no
