Hello and welcome to Geek Orthodox. I’m Father Justin, geek from my youth. AI has taken the world by storm. Whether we like it or not, it’s embedded in pretty much everything. My wife was just on the phone with an AI agent. This technology is being promoted, pushed, and distributed in all areas of our society. It comes from the world of code and is becoming an essential part of it. In fact, it’s part of the reason why I started this “World of Code” arc of my Geek Orthodox podcast. I want to try and explain in the simplest possible layman’s terms what this world of code is that we are immersed in, so that we can understand how it is affecting our lives in a better-informed way. However, undertaking to explain even the Large Language Model (LLM) version of AI is a massive task. Some of the introductory videos I’ve seen are three hours long, and I don’t want to get that deep. I’m not that much of an expert myself, but I do have a long-standing relationship with AI, thanks in part to the oldest AI chat-bot that took the world by storm back in the 1970s: Eliza. The Legacy of Eliza Eliza was a chat-bot written by Joseph Weizenbaum. It was the first chat-bot to “kind of, sort of” pass the Turing test—the informal test of whether a computer can fool a user into thinking it’s human. It shares a number of attributes with modern AI. It was originally written in a relatively obscure programming language called MAD-SLIP, but it was popularized in my favorite language, BASIC. Weizenbaum didn’t release the original MAD-SLIP code of ELIZA, but he did publish a paper documenting what the program did and how it worked, which inspired reconstructions in a number of different computer languages. BASIC was the universal language of the early computer revolution, and a BASIC implentation of Eliza was published in Creative Computing Magazine in 1977. My own first exposure to it was a version customized for my beloved TRS-80 Color Computer by my friend Bruce. I actually have some of the handwritten code he used to customize it, including sheets containing all the variables—something you had to do back when variables were limited to single or double letters. Most of the program consisted of “DATA” statements, which were essentially fragments of conversation. Eliza worked by taking your input, looking for keywords, and then rearranging the relevant portions of your input into a grammatically appropriate response which incorporated one of the conversation fragments which was most relevant to your most important keyword. It functioned best as a “Rogerian” psychologist—the kind of psychologist who asks questions to get the patient to provide more information. For example, if I said, “I have bad dreams,” Eliza might respond, “What does that dream suggest to you?” This illusion tricked many early users into thinking a real person was on the other end of the terminal. However, the illusion fell apart quickly. If you repeated yourself or challenged the bot, it would fall back on canned responses like, “We were discussing you, not me.” From Keywords to Transformers If Eliza was the height of AI back then, current versions are like Eliza on steroids. Well, that’s oversimplifying, of course, but we have to oversimplify something as complex as AI in order to understand. What modern LLM AIs are doing now is not simply looking for single keywords, but using conversation patterns to respond to combinations of keywords. The core of my understanding of AI comes from this wonderful video by Andrej Karpathy, who was one of the founders of OpenAI and also worked with AI for Tesla. He’s more recently famous for inventing the term “vibe coding,” which brings us back to the world of code. And in this video, which is now getting very old—it’s two years since it was posted—he builds a large language model (LLM) AI from scratch using Python. Python is, of course, the modern programming language that's fairly easy to use which we've touched on before in previous episodes of the World of Code. In this case, what he does is he creates what's called a transformer. The “GPT” in ChatGPT stands for Generative Pre-trained Transformer. You train it ahead of time on data with patterns, and the computer analyzes those patterns to generate new text that follows them. In this video, Karpathy builds a transformer that analyzes the works of Shakespeare character by character (i.e., letter-by-letter), which then is able, by statistically analyzing the relation of each letter to the characters that tend to follow it, to predictively generate Shakespearean sounding language, producing sentences like, “Verily, my lord. The sights have left thee again the king.” Like this, modern AI doesn’t just look for single keywords; it looks for relationships between common combinations of words. It “tokenizes” text, giving numerical representations to words, parts of words, and punctuation. To illustrate this, I’ve “vibe coded” with AI a tool that analyzes text patterns. If you feed it some “training text”—preferably something with a limited vocabulary that is somewhat repetitive, like the first chapter of Genesis in Basic English—it analyzes the relationship of each word to the next word and then to the third and fourth following words. It creates a statistical map of relationships between words and word-parts that allows it to function, much like a modern LLM AI, as a next-word predictor. So, if I type “God saw,” into my Genesis 1-trained mini-AI, the model predicts “that” as the next word with high likelihood (“God saw that”), and “everything” with a lower likelihood (“God saw everything”). And this is essentially how large language models work. The example I always use is the classic typewriter test-phrase, “The quick brown fox jumps over a lazy dog”, since as soon as we say “the quick brown,” anybody who knows that phrase will assume that the next word is most likely going to be “fox”—or possibly “dog” because people sometimes switch the test-phrase around. But what large language models are doing is not simply predicting the next word, but generating whole phrases and compositions based not on a single small passage of Scripture, but on patterns observed across a massive corpus—all of Scripture, all of Shakespeare, and much of the internet. LLMs, as they analyze the whole corpus of a language for the patterns inherent in such data structures, are essentially mining language for representations of meaning. Stochastic Parrots and the World of Code Another important concept to understand with modern LLM AIs is temperature. Temperature determines how random the next word selection is, which is important because usually we don’t want every AI generated response to a particular query to be exactly the same. In order to generate new patterns that are meaningful to us, we need to introduce, alongside the pattern-derived response, some element of randomness. High temperature introduces more variety but also leads to “hallucinations” or weird responses. I once ran the “quick brown fox” test-phrase through ChatGPT, and it started talking about a “quick brown table.” It’s a legitimate grammatical construction, of course, but tables—even brown ones—aren’t usually “quick” unless they’re on the back of a truck! AI models are often called stochastic parrots. “Stochastic” basically means semi-random. If you roll a single six-sided die (1d6), you'll get a value between one and six, and, on multiple rolls of that die, those values should be fairly evenly distributed. But if you roll two dice, while the outcome is random it follows a pattern, in that you are much more likely to roll a seven (1+6, 2+5, 3+4) than a two (only 1+1). AI follows patterns derived from language, but, given that those patterns are nothing more than vast arrays of statistical interrelationships between words and parts of words represented as numbers, AI doesn’t actually “understand” anything. It just semi-randomly puts together patterns of text that, statistically speaking, are more frequently associated with one another, which generally results in our perceiving meaning—and often useful meaning—in those generated texts. In the coding world, we’ve moved from machine language to higher-level languages like BASIC to reduce complexity. AI now offers a way to reduce that complexity even further by using our own native languages to generate code. My “vibe coded” tool was created by giving the AI English instructions, which it then translated into functional JavaScript and HTML. AI is actually getting pretty good at this, aided, in part, by the deliberately disambiguated nature of computer languages, which, in some ways, makes them statistically much simpler to master. However, there is a catch. In BASIC, a command is deterministic—it always translates to the same machine code. An LLM is non-deterministic. Because of that semi-random element, we can’t always be sure what the translation will be. So, as we offload these tasks—any tasks—to AI, we must ask: What does this mean for us? What impact does this offloading have on our intellect, our character, and our morality? These are questions we that we—as people who care about things like faith, morality, and meaning—must consider as we think about the impact of AI on the world of code in which we “live and move and breathe and have our being.” References * an excellent site on the history of the ELIZA chat-bot * the BASIC port of ELIZA in Creative Computing’s Big Computer Games * Jeff Shrager’s original BASIC implementation of ELIZA * a playable version of the BASIC version of ELIZA at Archive.org * a great list of visualizations of how GPT AIs work * a relatively simple visualization of how GPT “transformers” work * an in-depth interactive visualization of how LLM AIs work * the
