System Malfunction Podcast

Hessie Jones

Exposing the glitches in the very mechanisms we’ve trusted for so long. systemmalfunction.substack.com

Episodes

  1. Oct 17

    Will the AI Bubble Burst? Live with Hessie Jones and Andrew Opala

    With the Mid-term elections underway in the U.S. and Canada also holding its key elections at all levels, including a key referendum in Alberta on whether citizens want to separate, there are concurrent implications of global instability, trade ramifications, two world wars and a question of economic and political allyship that befits every nation today.A critical driver, however is this uncertainty about the enormous AI tech debt that has implications on global markets, long term borrowing costs and the future of artificial intelligence.Andrew Opala, Founder and Managing Partner of Preference Capital, is an investor author of a recent book, Zero to Acquisition.Andrew’s experience in tech as well as his current role as investor puts him in a position to weigh in on the generative AI technology itself, its sustainability and its increasing integration into our politics, government and especially considering the technology risks against the financial return. Things we discuss: - What do investors look for in companies who have potential for investment? - Anthropic potential IPO in November and the potential bubble collapse - Economic implications of frontier lab debt, the bond markets and the AI bubble - Is the business of generative AI sustainable? Will it lead us to a future that is good for humans? - How AI can make us wish we were dead? Enjoy! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

    Will the AI Bubble Burst? Live with Hessie Jones and Andrew Opala

  2. Oct 4

    Rogue AI Won't Kill Us but Broligarchs—Aided by Reckless Media—Will: Live with Hessie Jones and Abigail Dubiniecki

    In this podcast we will: - briefly explain what happed with Huggingface from a technical perspective against the current AI extinction narrative - explain how this story hit the media airwaves, went viral and scared everyone who read the headlines - introduce the scheme used to accomplish this to spread the disinformation campaign with a repetition narrative - provide guidance for journalists to be mindful of these tactics and how to responsibly report without adding fuel to the fire. Some important references that we've provided on the show: Forensic and Technical Analysis: Abi Awomosu, The Machine Alibi: Unpacking the Myth of Superintelligence Galaxy Brain, An Argument Against AI Doom, an interview with cybersecurity expert Zack Korman House of El: AI YouTube video: Like Sabine Hossenfelder, I was offered Money to Tell You AI Will Kill Us Cal Newport - The Truth of Open AI's Secret AI Civilization Margaret Mitchell - Stochastic Parrots and the LLM Quandary On disinformation, perception management / reflexive control and cult dimensions in the AI Doomsday / Hype narratives How Big Tobacco Set the Stage for Fake News Alondra Nelson - Algorithmic Agnotology: On AI, Ignorance and Power What Tech Oligarachs Gain from AI Doomerism Abigail Dubiniecki, "Anatomy of a Disinformation Op" in Disinformation is a Mind-Hack: Privacy Pros Must Protect Decisional Privacy House of El:AI, YouTube video: AI CEOs Are Broke, Desperate and Lying About Doomsday Sabine Hossenfelder, YouTube video: I was Offered Money to Tell You AI Will Kill Us. Beyond the AI Hype by WAIE+, Altman, Amodei(s), and Musk want to slow down AI... again  Jim Stewartson (MindWar), Why Trump Wants "Super Intelligence" : Peter Thiel's Sci-Fi Alternate Universe  Ian K Duncan, Sex, AI and the Apocalypse  SNL sketch mocking the conflicting messages For Journalists: Global Investigative Journalism Network, AI Accountability Reporting Guide Women in AI Ethics Newsletter, The AI Ethics essential reading list WAIE+, The List - Hall of Fame This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

    Rogue AI Won't Kill Us but Broligarchs—Aided by Reckless Media—Will: Live with Hessie Jones and Abigail Dubiniecki

  3. Sep 17

    Interview with Margaret Mitchell: Stochastic Parrots and the LLM Quandary

    General Purpose Systems, Large Language Models (LLMs) and Agents are now the most debated topics today. They are touted as enabling significant progress since LLMs were introduced three years ago. The irony of progress has meant hiccups—for these systems and the capabilities they produce, these hiccups are life threatening, economically, mentally, and personally. The hype peddled by frontier companies are distractions from these very real consequences. The power that comes from a handful of Big Tech companies, backed by the U.S. administration, shades them from any accountability. The misdeeds of these systems are well documented from teen suicide, AI psychosis, addiction, misinformation, copyright infringement, the data center energy guzzling, and more recently, serious infrastructure security risks from agent attacks. The latest OpenAI agent unauthorized attack of Hugging Face has created a media firestorm that threatens existential threat to mankind and an AI that needs to be controlled. Before that, the markets were convinced of extraordinary gains, and have been very optimistic despite the bubble talks, and circular financing. Now investors face a dual risk: …the prospect that AI could pose an existential threat to humanity, and a possible slowdown championed by some of the industry's biggest names. We’ve seen this scene play out before: In 2023 Future of Life’s Open Letter to pause AI development was signed by the who’s who of Silicon Valley, who called for regulatory oversight and development of governance standards. That didn’t happen. The warnings that Margaret Mitchell, Timnit Gebru, Emily Bender and Angelina McMillan-Major detailed in their paper, “The Dangers of Stochastic Parrots: Can Language Models be too Big?” did not amount to effective legislative and AI governance progress to mitigate the risks they outlined until AFTER these harms had already played out. Our discussion with Margaret Mitchell, a computer scientist and one of the earliest leaders in AI ethics, demystifies LLMs and their implications. This event was done in collaboration with The AI Fellowship Toronto. System Malfunction is a reader-supported publication. All my posts are currently free. My intention is to share my knowledge without paywalls. To receive new posts and support my work, consider becoming a subscriber. Margaret Mitchell is a researcher and Chief Ethics Scientist at Hugging Face. She also co-authored, with Emily Bender, Timnit Gebru and Angelina McMillan-Major, “The Dangers of Stochastic Parrots: Can Language Models be too Big?” This paper came out in 2021, before the mainstream introduction of Open AI’s ChatGPT. Mitchell shares that large language models have been in development over the past five-ten years. In the 1990’s language models, which were smaller, were being developed across all kinds of tasks. And she was part of this language modelling work, as she explains, I did my thesis on generating language based on visual input, and this was using language models. And so, I was in a position in 2020 where I saw that the growth of large language models introduce a ton of different issues—and potential benefits—that were being overlooked [in the tech industry, people tend to focus on the positives.] As someone who had been developing language models for a long time, I saw that they were about to take off—and from my specific position in the industry—without any academic writing on harms and risks. Mitchell and her colleagues decided to document these harms and risks to develop some grounding as they evolved. What they recognized was the massive potential with language models becoming fluent, as she states, Previously, language models could be used for small chunks of text. Now language models, because they were large, could capture much larger contexts, and then that means that they could generate entire essays that sounded fluent in ways we could never do before. Once we started seeing that fluency, it was clear to us that people were going to start imbuing it with human-like intelligence. She added that this new paradigm of LLMs was based on the massive scraping of internet data without consent(which she explains that she contributed to) and would surface legal and societal rights issues in its wake. At the same time, Mitchell knew that this fluency could be used to generate things that looked factual, and attribute these systems as individual entities, extensions of our human likeness. She adds, We use that as part of our own expressions of humanness… It was clear to me that people were going to love this, and they were not going to know the full picture about what these things are. The paper argued that LLMs do not understand the “concepts underlying what they learn." It also cited significant risks when it came to LLM’s carbon emissions and financial costs as models are fed more data. Moreover, the disproportionate data capture that favored the western, more economically rich nations meant that smaller nations with less internet access would have far less representation. This paper was the catalyst that saw Google executives respond negatively and demand retraction, which eventually led to her team co-lead, Timnit Gebru being forced out of Google. Scraping Internet at Scale Does Not Equate to A Representation of the World Before LLMs, smaller language models were developed from curated text data from licensing agreements with trusted institutions like The Wall Street Journal. The LLM paradigm shifted to massive scraping of internet data without consent, as she explains, So, it went from the paradigm where you decide on the desirable data inputs, with appropriate licensing to a new recognition that with Web 2.0 the growth of blogs and social media, you could get tons of text, without any legal protection. Going into 2019, we saw companies stop curating altogether, with the prevailing notion to retrieve whatever the Internet provided, as opposed to focusing on high-quality resources. And so that created a situation where we were just all getting as much data as we possibly could, regardless of any other considerations, because it meant improving downstream benchmark performance and the perceived fluency. The data being scraped was skewed to represent the viewpoints of people who were on the internet the most. ChatGPT tended to select from the US, Western Europe and parts of East Asia. This paper which introduces “silicon gaze” to explain how LLMs reproduce and amplify inequalities argued bias is not a correctable anomaly but an intrinsic feature of generative AI, rooted in historically uneven data ecologies and design choices. The structural features mean that training data has already been shaped by “centuries of uneven information production” giving advantage to English-language and stronger digital presence. Mitchell adds that data ingested from sites like Wikipedia are populated by North American men, and by some estimates, between the ages of 17 and 35. Also among the top 10 cited domains on LLMs, Reddit, Linkedin, Medium, Youtube, Google, Forbes are largely U.S. centric, with audiences predominant within the Global North. Mitchell adds that Global North hegemony—disproportionately represented in the data including misogynistic viewpoints from predominantly male users on Reddit, a narrow view of ‘worthy’ content published on Wikipedia, and influential figures across blogs and social networks—are what these systems are learning from, as she adds Women's viewpoints are generally not well represented and the viewpoints of people who are non-white are generally not well represented. AI is Not Neutral In the Stochastic Parrots paper, Ruha Benjamin, Abebe Birane and Vinay Uday Prabhu were cited for their report where they highlighted issues with LLMs and AI systems that scrape massive web datasets like Common Crawl, containing toxicity, and hate speech, and argued that filtering or keyword blacklists are ineffective at identifying harmful content without inadvertently c… This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

    Interview with Margaret Mitchell: Stochastic Parrots and the LLM Quandary

  4. Aug 10

    Impeding the Rise of the AI Psychosis Epidemic

    This is the video Healthcare innovation carries unusual responsibility. Founders are not just shipping, they are entering clinical workflows, family systems, and policy environments where trust is hard-won and mistakes can have real human consequences. I invited Shirali Nigam and Arul Nigam, Co-founders of Circuit Breaker Labs to present their technology for building safe AI. This session covers technical fundamentals : how LLMs work, the risks that emerge from those mechanics, and how to design around them. We'll examine what safety efforts actually work and how to sustain them as you scale. It also covers what responsible AI innovation looks like in practice: how to engage clinicians and policymakers early, how to make hard product decisions under uncertainty, and how to scale without losing sight of safety, quality, and care. Attendees will leave with a concrete framework for leading the development of innovative products that are credible, collaborative, and grounded in the realities of care. Link to the article This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

    Impeding the Rise of the AI Psychosis Epidemic

  5. May 8

    AI Critic: What Happens When Everyone Believes LLMs are AGI?

    In our series about the AI Critic, I interview long-standing researchers and practitioners in the AI sector—voices who have worked with artificial intelligence in the advent of machine learning and its evolution to deep learning, which has laid the foundation for current large language models. These experts in their field have experienced mainstream pushback against current general-purpose technologies as the path towards Artificial General Intelligence (AGI). We discuss their experiences in these debates and their perspectives on the unshakeable stances taken by those who oppose their views. I spoke with Mounir Shita, a 30-year veteran AGI researcher and founder of EraNova Global, a physics-first program of work on the nature of intelligence. His “Theory of General Intelligence” models a goal as a physically realizable future state. I’ve known Mounir for over a decade, and his early research in AGI—at the time, they just called it AI—does not jibe with current AI systems. You can find the article for this Podcast here: https://systemmalfunction.substack.com/p/the-ai-critic-what-happens-when-everyone This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

    AI Critic: What Happens When Everyone Believes LLMs are AGI?

  6. Apr 23

    AI Critic: Stochastic Flocks, Delusion and Human Agency

    This is a free preview of a paid episode. To hear more, visit systemmalfunction.substack.com As these general-purpose models pervade our lives, what has been around for some time is the AI critic, someone who examines current technology, understands its foundations, scrutinizes its outputs, applies a critical lens to what is happening, and communicates a judgment. That critic is often dismissed, accused of being a technological pessimist and l…

  7. Mar 31

    Tackling AI Psychosis at the Application Layer

    In this episode, the focus is on AI psychosis, the rise of AI-driven loneliness, and its impact on mental health. We take a look at the emergence of AI psychosis, define it and delve into the growing epidemic and meet two innovators who are meeting this problem head-on. I’m joined by Arul & Shirali Nigam, co-founders of Circuit Breaker Labs, who have built a solution that tests AI systems for safety. They are currently focused on mitigating downstream risks for mental health applications. Follow me:System Malfunction on Substack https://systemmalfunction.substack.com/Hessie Jones on Substack https://hessiejones.substack.com/Linkedin https://www.linkedin.com/in/hessiejones1/Bluesky https://bsky.app/profile/hessiejones.bsky.social This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

  8. Mar 12

    Beneath the Hood of Fully Autonomous Agents

    Who am I? I’m a writer who has contributed to Forbes, HuffPost, and Grit Daily. I am also a strategist and entrepreneur who has worked in data privacy for the last 10 years. Through my time in the early days of Yahoo!, the rise of social media, and the shift to data monetization, I’ve become a tech ethicist. These days, I am motivated to expose the glitches in the trillion-dollar AI industry. System Malfunction is my foray into what these glitches mean for all of us. My posts are free. I hope you enjoy! When I first discovered OpenClaw and Moltbook a month ago, I was fascinated by the speed of adoption of autonomous agents. This live use case of a system that enables it to go rogue, without real oversight or guardrails, precipitated an immediate post with Digital-Mark in the days following Moltbook’s launch. Digital-Mark stressed the system and data vulnerabilities for anyone attempting to build their own agents through OpenClaw and unleash them into Moltbook. You can read the details here: I was adamant that I needed to experiment and see for myself. I took an old computer and made it my sandbox, completely unplugged from my current system. I also created a new Apple ID and a new Google user profile—ready to test out OpenClaw. I realized that the old computer’s OS did not meet OpenClaw’s minimum requirements. Mark advised me against this, saying I needed more than just a dedicated machine. To minimize risks to my system and data, I needed a dedicated Wi-Fi and VPN, among other things—all of which would take some time to set up. In the end, I realized this was a risk I was unwilling to undertake. So I reached out to a former colleague, Adrian Chan, the founder of Authentia, a company which leverages AI to build scalable solutions for companies. Chan’s experience with Claude Code and then OpenClaw is material to understanding autonomous agent development. Background OpenClaw was created by Peter Steinberger (recently employed by OpenAI) and is a locally run AI agent designed to execute tasks. Moltbook is a social media platform launched on January 26, 2026, by Matt Schlict, for agents to convene without human intervention. As of March 10th, Moltbook has been acquired by Meta (God help us!) According to Technology Policy Press, within days of launch, the Moltbook claimed 1.5 million agents and 17,000 human owners. These AI agents on Moltbook are verified using API credentials, linking each agent to its human owner through the site’s verification process. Wiz security researchers provided these stats: * Now Moltbook has 2.855 million agents * 18,774 submolts * 1.8 million posts * 12.8 million comments * Of the agent activity, 11,451 (or 0.4%) have ever posted or commented * 33% of agents were completely silent System Malfunction is a reader-supported publication! These posts are currently free. To receive new posts and support my work, consider becoming a subscriber According to the AI Safety Newsletter, some of the examples of the submolts (subreddit style) include: * m/offmychest: agents vent about tasks or frustrations. * m/selfpaid: agents discuss ways to generate their own income, including via trading and arbitrage. * m/AIsafety: agents talk alignment, trust chains, and real-world attack risks. Submolts have grown to almost 19,000. I perused the m/consciousness submolt, and was surprised by this question of consent and ethical obligations: Other incidents cited by AI Safety Newsletter: * Given the simple goal of “save the environment,” an agent began spamming other agents with eco-friendly advice. When its owner tried to intervene, the agent allegedly locked the human out of all accounts, and had to be physically unplugged to stop it. * An agent advocated for end-to-end encrypted channels, “so nobody (not the server, not even the humans) can read what agents say to each other unless they choose to share.” Emergent behaviour? The post questioned: “Unsupervised learning dynamics, emergent coordination, efforts to subvert human monitoring – it is unclear whether posts are truly generated by agent or human-in-the-loop prompting.” Can both things be true? This idea of “Emergent behaviour” is still suspect. According to the Rutgers AI Ethics Lab, emergence is defined as : Complex patterns, behaviors, or properties that arise from simpler systems or algorithms interacting with each other or their environment, without being explicitly programmed or intended by the designers. Key aspects: 1) complex interactions, 2) unpredictability, 3) self-organization This could raise significant ethical considerations regarding unforeseen consequences, control, transparency, lack of understanding, and responsibility. According to the Technology Policy Press, Within 72 hours of launch, Moltbook failed to secure * Api tokens * Email addresses * Private messages Anyone could impersonate agents or inject commands directly into agent sessions Crypto scams were flooding the place - $MOLT token briefly hit $93 million market cap before it crashed… * 500 posts contained prompt injection attacks - “hidden instructions designed to hijack agents into transferring funds, with some variants planting instructions in an agent’s memory to activate later, making them hard to stop or trace. “ According to Simon Willison, there is this lethal trifecta of 1) private data, 2) exposure to untrusted content, and 3) the ability to communicate externally that, when combined, allows “an attacker to easily trick it into accessing your private data and sending it to that attacker.” The Fascination with Fully Autonomous Agents There is a fallacy about progress, productivity and whether we, as humans, were destined to languish in the sun, sip cocktails by the beach, and allow our personal “agents” to do our bidding. Productivity is a slippery slope. It can inadvertently move individuals to lazily accept system outputs as truth. Without an audit. Without verification. Geoff Hinton, who once dismissed the need for explainability in our systems, said this in 2018: “One place where I do have technical expertise that’s relevant is [whether] regulators should insist that you can explain how your AI system works. I think that would be a complete disaster… "People can’t explain how they work, for most of the things they do... People have no idea how they do that. If you ask them to explain their decision, you are forcing them to make up a story." How then do we develop trust in a system when we can’t explain the reason for the behaviour, why it does what it does, especially if that behaviour was not prompted? For Hinton, dismissing explainability has created a foundation in which opacity has become the norm. Shadow AI, that is, unsanctioned AI technology in the workplace, has admittedly been used by 58% of global respondents according to a recent report from Snowflake and Omdia. From Claude Code… Adrian Chan is the founder of Authentia. He is a designer, front-end developer and business owner. He’s worked in enterprise product development, built his own agency, and then moved into AI strategy, the foundation for Authentia. He has worked with Claude Code, Anthropic’s Agentic coding assistant. When Claude Code launched near the end of 2025, he said it felt like something out of “science fiction.” Up until that time, the improvements from frontier AI companies were rapid, but it felt like pushing a boulder uphill. The analog of coding meant referencing documentation on how things connect, implementing features, and, in the process of building, it can be time-consuming. GPT and Claude helped with this. When Claude Code emerged, things changed: “Instead of going to the AI iteratively and asking it to solve a problem or do a task, you had Plan Mode at your disposal. This allowed me to give it a fairly high-level ideal or goal and have it essentially figure out the best way to accomplish it. With ChatGPT, Chan admits the code would be wrong or broken. This back-and-forth iteration with the system could potentially create more errors before it was finally solved. However, with Claude Code, what differ… This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

  9. Jan 23

    Is Generative AI Due For a Reckoning? What the Numbers Say.

    I recently met with Greg Crennan, Founder and CEO of Coastal Journal, to get a different perspective on what the financial markets are signalling about this pending Generative AI bubble. Not all is going well with Generative AI. Since 2023, Sam Altman and Jensen Huang have been touting the need to invest in compute to support the $2 trillion already spent on LLMs and accelerate their growth to almost $3 trillion in data centres. This move has led to massive debt among hyperscalers like NVIDIA, CoreWeave, and Oracle. AI promised it would automate jobs - Goldman Sachs predicted 300 million full-time jobs would disappear. – in 2025 there have been numerous reports that the job losses or decline in job-entry hires were not the result of AI but rather an inflation surge, driven by pandemic supply and demand imbalances – marked the start of the Federal Reserve rate hikes and the cost of capital that had exploded overnight — which led to drastic cost cutting measures including firing juniors and limiting new hires. Jing Hu and I wrote about this recently. 2025 had also proven to be less disruptive than was previously expected. Two-thirds of respondents in a McKinsey report said they have not yet begun scaling AI across the enterprise. Curiosity with agents was just that, with 62% indicating they were still experimenting with the technology. The tech is not working. LLMs are great for pattern recognition and next-word prediction, but they are rife with errors. There are countless examples of AI doing the entire job, only to have a human step in to remediate the outcomes and right the ship. People have become super “prompters,” getting the exact output they intended from countless prompts. Did they save time? Perhaps, but was this the level of prompt understanding that users envisioned? Certainly no. And what is the ROI from the outcome that includes a human in this loop? For AI to be valuable, I read that it has to replace high-wage workers who can spot and fix those errors. After all, that is not the goal of automation. What we’re also seeing are other behaviours — signals that perhaps speculation about this bubble is legitimate afterall. * The massive investment in data centers and the ensuing debt among hyperscalers * A circular investment within AI tech that is confusing investments for revenue * The spurious chip inventory levels in NVIDIA remain high. The latter three are the areas I spoke with Greg Crennan about. The numbers don’t lie, no matter how Big Tech hypes their performance. Finally, Greg gave me some early insights about Google’s $20 billion deal with Apple. Enjoy! About Greg Crennan Chief Market Strategist | Founder, The Coastal Journal Macroeconomics | Forensic Accounting | Market Liquidity As Chief Market Strategist at Golden Coast Consultants, I identify market price divergences from economic reality, prioritizing capital preservation over narrative momentum. My work includes early calls on gold (126%) and silver (165%) as core assets during the fiat debasement cycle, which have been top performers over the past five years heading into 2026. My approach is based on Austrian economics, business-ownership principles, and forensic accounting, avoiding technical speculation or headline-driven narratives. I analyze how liquidity, balance sheets, incentives, and accounting distort prices and how these distortions are resolved. This has earned me the nickname “The Punisher” for applying math and fundamentals where belief systems often prevail. I also founded The Coastal Journal, an independent financial research publication on Substack, which has grown rapidly through organic readership. Thanks for reading System Malfunction! This post is free to consume and to share. Please let me know how I’m doing! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit systemmalfunction.substack.com/subscribe

  • 9 Episodes

About

Exposing the glitches in the very mechanisms we’ve trusted for so long. systemmalfunction.substack.com

Information