Content Operations

Scriptorium - The Content Strategy Experts
  • 4.3 (7)
  • BUSINESS
  • UPDATED BIWEEKLY

The Content Operations podcast from Scriptorium delivers industry-leading insights for scalable, global, AI-optimized content.

  1. 3D AGO

    Taming AI: Using AI for content conversion at scale

    AI promises to transform content conversion, but what does it actually look like when you’re processing thousands of documents a day? In this episode, Sarah O’Keefe (Scriptorium) and Rich Dominelli (DCL) dig into the real-world challenges of using AI for large-scale structured content conversion. Rich Dominelli: If you have millions of articles and you’re asking the AI, ‘What did we do for this project six months ago?” The AI has to find those articles, pull the relevant information out of those articles, summarize it, and hand it back to you. The best way of doing that is to give extra signals to the AI, structured relevant bits of information, front matter, back matter, publication date, keywords, abstract, that allows the AI to query the corpus and get the relevant chunks out of that corpus in a very quick manner. Then, it can summarize what those chunks are. So the AI almost becomes the user interface over that corpus. But to find that data in the first place, structured content is key. Structured content is key when you’re dealing with big indexes and the web, and it’s the same with AI. Related links: Defeating Nondeterminism in LLM Inference (white paper) Data Conversion Laboratory (DCL) Scriptorium, Machine experience (MX): Making content work for humans and machines (podcast) LinkedIn: Host: Sarah O’Keefe Guest: Rich Dominelli Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe and I’m here today with Rich Dominelli who is a Senior Developer and Architect at DCL. Rich, welcome. Rich Domineli: Hi, thank you for having me. SO: Glad to have you. We were talking before we hit the record button, and you described yourself as a perhaps hopeful AI evangelist. RD: Yeah, I am well and thoroughly immersed in the AI game at DCL and using it and plus I play with AI assistants at home. I’m enthusiastic about the future of AI, sometimes disappointed about the present. SO: So DCL, as I think many of our listeners know, is focused on conversion at scale, which to me makes a great use case for AI because ultimately conversion is about edge cases and about inconsistency, right? If everything was 100% consistent, conversion would be pretty easy. RD: Yeah, no, DCL does a lot of structured content generation out of unstructured data, and the creativity, especially in the academic space, of what that unstructured data looks like is sometimes nightmarish. So the AI lets us, does a lot of the heavy lifting for us when it comes to looking for particular items, identifying concrete data points within the documents, pulling things like authors and affiliation, front matter type information, and back matter type information out of the documents and in automated fashion. It can be painful from time to time, but it’s definitely helped. SO: Yeah, so this is, think, you know, the reality of working with AI and working with it in a production environment in order to address all these weird edge cases and what’s going on. So tell us a little bit about how you’re using AI in, you know, these conversion use cases. What does it look like to go in there and start applying some of these tools that we have? RD: So, I mean, typically our flows work in a way where we’re coming in with a PDF or a Word document or some other unstructured format. We take it, we reformat it into a version that’s more AI-friendly, like Markdown, for example. And that’s usually the first step we’re doing when we’re looking for information to pull out of it like front matter. It’s a very common use case. If you look at academic papers, the front matter, the authors and the affiliations that are on that paper can be formatted in more ways than I could list out during the course of this podcast. It’s kind of crazy. So what we’ve started doing, and we’ve been doing this for a couple of years now, is we’re using the AI, we’re handing it the Markdown document, and we’re saying we need to list authors and affiliations, please extract it for us.  Now, naively, when we started that process, we assumed that the AI would give us a consistent list of authors and affiliations. And sometimes it does. But every time you do that call, you’ll get it in a different format. So then you have to start tightening things down. So OK, give me a list of authors and affiliations. I want it to be structured exactly like this. And typically, we have a JSON structure that we’re presenting to the AI, along with our prompt, and saying, give it to us. Well, okay, and that gets you a good chunk of the way there. And that was very exciting when we had that working consistently, we were getting things out of the system on a consistent basis. Awesome. But then you start looking at the results, and every once in a while, you get an author that was missed, or there would be too many authors on that paper.  We had one test paper, which I loved, which had 600 collaborative authors in it. And the AI would just choke after about 280-ish. So then you have to start dealing with things like paging through the data and formatting the data. And then you have to figure out, well, did it miss anything? You have 600 authors. Good luck. So now you have to take what the AI did and compare it against your own representation of it and write a program to do that comparison to say, OK, is it good? Is it good? You have to take a step back and you look at it and you say, okay, we have the information that’s in the non-structured format. We’re handing it to the AI. The AI is gonna give us a structured version of it and we need to validate it. Well, the first validation is very easy. Does that structured version match the schema that we gave it? Yes or no, that’s easy. Well, then you have to say, okay, is everybody there? Well, is there anybody added? Because the nice thing about AI is they occasionally get very creative. Even if you have that temperature dial turned all the way down to zero, it will pull names out of thin air and then come back to you with some random name and stick it in the middle of the data where it’s not obvious, of course, and then hand it back to you. So then you have to start saying, are all the names that appear in this list actually in the document? Are the counts matching? And if it’s not, you go back to the AI and you ask it again, and usually you’ll get a better answer the second or sometimes the third or fourth time.  But you need to be able to catch that, especially if you’re doing this at scale, because if you’re doing a few, it’s easy, you can eyeball it. If you’re doing 1,000 of these a day, you can eyeball all of them. You can say, you can ask the AI, OK, give me a confidence level, but if you can’t trust it in the first place about what it’s returning, yeah, I’m very confident about what I’m giving you right now. It’s really the truth, I promise you this time. I don’t know how trustworthy that would be. So you have to write tools to validate what the AI is producing, or you have to use the AI to validate what it’s producing. So coming in the first time, obviously, we did the count, we did the schema validation. We then said, okay, we’re going to check to make sure all the names appear in the document, we’re going to have landmarks in the document that we can refer back to. So if you start with Microsoft Word and you have track changes on, you can have paragraph IDs that are supplied. So you can make sure that you can find all of the authors in that list and they all have a paragraph ID and you can have your landmarks and that’s great. Or you can even hand the results to a separate AI call and say, proofread this. Is this accurate? Is this the best answer that could be for each of these? I know we’ll come back with an answer. And you can use that as a signal to gauge accuracy and to gauge repeatability and make sure it’s correct. SO: So you’re, let’s see, generating an AI, not a test bed, but an AI environment that’s doing this conversion or that’s processing the files for you for conversion. And then you have to go in and do all this validation to make sure that the output that you’re getting is actually correct. As compared to, I’m gonna say old-fashioned, but you know, as compared to scripting, deterministic, pretty straightforward, if A then B kinds of scripting. What are the differences between that and AI-driven conversion in testing and validation? What are the test plans? How are they different conceptually? RD: So from our perspective, the frustrating thing sometimes is the AI is completely non-deterministic.  SO: Mm-hmm. RD: It can give you a name formatted one way today, and then tomorrow, its formatting might be subtly different, where in the paper it has “Richard Dominelli, Junior.” The AI may decide, well, that comma probably shouldn’t be there, or junior should be followed by a period, and it wasn’t in the paper originally. And you can try prompting around that and tell it to prompt around that and make sure that it’s accurate. But it doesn’t always follow your instructions exactly when that’s the case. SO: And why is that? Why is it non-determin

    25 min

  2. MAY 4

    Machine experience (MX): Making content work for humans and machines

    Your website may look great to humans, but can machines understand it? In this episode, Sarah O’Keefe (Scriptorium) and Tom Cranstoun (Digital Domain Technologies) explore the emerging discipline of machine experience (MX). Sarah and Tom discuss what AI agents actually encounter when they visit your web pages, why microdata and metadata are critical, and what content creators must do to ensure content is consumable for both human and machine audiences. Tom Cranstoun: Humans are looking for pictures, they’re looking for text, and they can infer. You may think, “Well, we’ve already added information on the page,” but by putting it in as microdata, it doesn’t appear on the page for the humans. It appears on the page for the machine. I think that that’s a critical distinction. We are trying to design for both. We don’t want to overload a human with information, but we do want to give the machine as much information as it can take. Related links: The Gathering Digital Domain Technologies MX books The Scriptorium Content Ops manifesto LinkedIn: Host: Sarah O’Keefe Guest: Tom Cranstoun Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe. Today, our guest is Tom Cranstoun, who is founder of a machine experience, or MX community, called The Gathering. He has a couple of books on MX and is currently a consultant operating as Digital Domain Technologies. Tom, after 53 years in the business, some experience with AEM at very, very large companies, including a huge project at Nissan, has turned his attention to the question of how machines, which is to say AI agents, interoperate with the current public-facing web. And so today, Tom, I’m delighted to have you on to talk with you about machine experience, or MX, and what this all means as we move forward in this brave new AI world. So welcome. Tom Cranstoun: Thank you, Sarah. I’m very pleased to be with you today. SO: I am delighted to have you. So I guess we’ll start with the extreme basics here, which is what is machine experience, or MX? TC: Yeah. MX, well, to my definition, machine experience is like user experience, but it’s for machines. Machines cannot ask a friend for help if something goes wrong when they’re browsing a website. They can’t turn to a partner and say, “What do you think this means?” They can’t retry a failing form input because they will just go through the same mechanical patterns to try and carry on throughout the web journeys. Therefore, machine experience is thinking about what elements one must put on a webpage to help a machine understand and action the final goal of the webpage, whether that be a CTA that lets you purchase something, or an information document that lets you know about a government policy, or a charity good, whatever the author of the page is trying to get across to the audience. SO: And so at a high level, what does it look like to build out machine experience? What are some examples of things that you need to put onto a webpage to accommodate the machine that’s reading it? TC: Well, the very first level is the disabilities angle, things like the Americans with Disabilities Act, that kind of WCAG, W-C-A-G, the accessibility work. The more accessibility information is on the page, the more the machine can understand the background of the page. So machine experience and accessibility are pretty much at the top level, the same sort of thing. If you put in JSON-LD, microdata, and you enrich your pages with the things that Americans with Disabilities Act would like, you’re actually helping a machine understand the page. So that is the top-level constraint. When you go below that level, you need to give the machine lots of information about your product, not just the thing that a human wants when it’s glancing at the page now, and as you go through the journeys, things will be added on. Humans can only take in two or three items at a time, so we design pages to reveal what is happening. You go to a catalog, to a product, to a variation, to a purchase, four different steps. Each step introduces different pricing and concepts. It’s best to feed the machine on the page that the machine lands on with all of the information that it needs. This may not necessarily be surfaced to the human reading the page, but it’s there for the machine. This helps the machine when it arrives at your webpage. SO: So I’m really enjoying this concept that a properly organized page with proper accessibility WCAG or ADA compliance and support then results in the machine being better able to parse the page for essentially the same reason, right? It’s properly structured, it’s predictable. The things that are labeled are labeled correctly. I don’t know that we should be driving accessibility in order to enable AI, but on the other hand, if it gets us more accessible pages, then let’s certainly do that. Can you give some examples of what happens when pages are not machine-compatible? What are the kinds of problems that people run… Or not people. What are the kinds of problems that the AIs run into when they try to parse a page that has not been labeled properly or encoded properly? TC: Yeah, I collect these examples from real life. Whenever I use the web as a normal person, I say, “Well, how would a machine interpret this?” Recently, I was looking for a holiday, and I asked an LLM to give me a list of five companies that offer cruises up the Mekong Delta. The machine came back with one offer at $200,000 for a week’s holiday, and the rest of them were $2,000 for a week’s holiday. What had happened there was that the machine had found a European website. Now, the Europeans changed the comma and the dot in monetary labels differently from what the Anglo-Americans do. We use a comma separator between thousands and a full stop between fractions. The Europeans actually put the full stop as the thousand separator and a comma between the fractions. This meant that when the LLM built a table of prices for holidays, it didn’t understand the distinction, and it tripped up. The agent hadn’t been instructed to compare prices and make sure that they were all within the same range and were reasonable. It just produced them as a matter of a fact. “Here’s a holiday for you. One of them is $200,000. The rest of them are 2,000.” There was no knowledge, no information that could tell the agents what was happening. If those pages had been decorated with currency and they had microdata with the… microdata always says that you should use commas as a separator and full stops as the fractional separator. If these things had been in the page, the machine wouldn’t have flipped up. Now, a human could have read a page and seen the locale values shown on the page, and both people would be able to understand what was going on. So that’s a typical trip-up from an undecorated page. SO: And so essentially, the presentational component that says, because I’m serving this page to somebody in, for example, Germany, they are expecting a comma separator between the full Euro amount and the cents, the Euro cents. But that comma is essentially formatting, as opposed to data, and so here we are. TC: Yes, correct. And the microdata has got the thing in a proper machine-readable way. The other things that we always get problems with in the world are English and American date formats. We swap the month and year around when doing short form. The machine-readable version uses ISO dates, and ISO dates put in as a microdata tells the machine categorically. It doesn’t matter what the locale is, this is the date and time. SO: Yeah. And so as the expression of the date, whether April 1st is 1-4 or 4-1 is essentially a formatting problem. TC: Correct. And these are not visibility problems. These are machine experience problems. So it’s layering up. You start with fixing the disability by doing machine experience, and then you fix the locality and the community values, the human factors, display factors. SO: And so I think we’re all familiar with the concept of a customer journey, but you’re now talking about a machine or an MX journey. What does that look like? I mean, how is the machine processing of a website? How do you explore that journey and what it looks like? TC: The machines will not discover your website, come in through your landing page, and then look for offers or products. A machine will have an idea of where it wants to go and will land straight in at a page. It will arrive five pages into your journey, and read the webpage as it is. The owner of the website has lost all of the signals about what the dwell time was on each page, how’s the reader arrived at the end location. Did they go sideways and look at other things? Those things don’t happen with machines. They go straight in, see if they can get what they can. If they can get what they can, they will action it. If they can’t, they will move on, and go to another page or another person’s website and do exactly the same to them. So when a machine arrives at your webpage, it will not be giving you any referral details. It will not tell you what the jou

    20 min

  3. APR 27

    Make the move successful: Replatforming content ops

    Replatforming your content operations isn’t just about swapping systems. In this episode, Alan Pringle and Bill Swallow share what organizations must consider to successfully replatform. From navigating technical debt, system integration, and the people caught in the middle, they discuss change management, technical debt, and why your exit strategy should be part of the plan from day one. Software isn’t forever. Systems come, systems go, they get improved. Your requirements are ever changing with the content that you need to manage. Not thinking about your next jump is really to your detriment. — Bill Swallow Related links: Replatforming structured content Your tech expertise + our CCMS knowledge = replatforming success (case study) Cutting technical debt with replatforming (podcast) Replatforming with localization in mind LinkedIn: Alan Pringle Bill Swallow Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Alan Pringle: Hey everybody, I am Alan Pringle, and today I want to talk with Bill Swallow about content operations and replatforming. Hey Bill, how are you?  Bill Swallow: Good, how are you doing? AP: Good. So I guess we should start this by saying the reason why we want to talk about replatforming is really we have done a few replatforming projects. We’ve had some prospects reach out who are interested in doing it. So I guess we need to explain what it is and some of the things you have to think about when you’re going through the process. So if you would not mind, would you define what we mean by replatforming content operations? BS: Sure. So generally when I talk about replatforming, it’s in the context of a company having one system in place and maybe it’s time has come and they need to move into a new one. So it’s the entire process of determining what type of system you’re going to need, what your requirements are for that and being able to lift everything up from the old system that you want to carry forward and put it in the new system, configuring it and what have you to get it to work going forward. AP: So we’re not talking about using a whole new technology or a whole new platform. It’s shifting to a similar platform for some of the reasons that you just mentioned. And I think that’s another thing. There are several reasons why a company might want to do this. And I know our clients have had various reasons for doing this. Let’s focus on those for a little bit. One of them, I know you kind of already touched on this. Sometimes you just outgrow a system. It just… that’s how it is. So let’s start with that kind of, it’s not sustainable anymore because you’re bigger, too big now for what that system can do. BS: Sure, either you’ve outgrown it or it’s approaching end-of-life or it’s just not meeting the needs that you had five or ten years ago when you bought the system. So there are a lot of different factors there, but basically it comes down to what are your requirements and is it meeting your requirements?  AP: Right. BS: Are you able to get the things done that you need to do given the fact that, you know, the world is quite different now than it was five or 10 years ago. AP: Exactly, and there’s another angle here too that I think we need to briefly mention is that sometimes you’re gonna have two sets of requirements because two companies can merge or there can be an acquisition and then all of a sudden you’ve got two content operations platforms that are pretty much doing the same exact thing and I guarantee you the IT department is not gonna have that.  BS: Absolutely. AP: So there could be a situation where you’ve got two, and one of those is going to go away. And in some cases, and we should talk about this too, it’s not necessarily about picking one. It’s not uncommon to go to a whole other one. So there is quote, “No loser.” That’s also an option. BS: That’s very common because usually in the case of a merger, you have two established groups with two established systems that may be starting to age out on both sides. And it doesn’t make sense to spend the time and the effort to move one group into the other system when that system is probably going to be replaced in a few years anyway. AP: Yep. So basically, the circumstances of the merger have provided a perfect opportunity to do something that’s painful. Replatforming is not magical. There’s still going to be technical hiccups and everything else. But at least it’s not as painful because you’re both moving out of systems that maybe aren’t optimal into something that will basically treat your content creators and all the people managing content much better because it’s going to support their needs better. BS: Or at least everyone goes through the same pain together of learning a new system. It’s team building. AP: And so you bond through shared pain experiences. Exactly. All right. Yeah. huh. We’ll go with that. We will go with that. There’s some other aspects here too that are kind of related to that. And that are the idea that things, because things are getting maybe rickety, that things are getting too expensive to maintain. You keep making these little tweaks and changes in things that become less and less repeatable. And that adds up. That’s time and money that you have invested. BS: It certainly does. Aside from the hard costs of licensing and just the general time to use the system, produce things, you do have, after you’ve used the system for so long, you’ve got your workarounds built in and they may not be a best practice and the workaround may solve the problem on its face, but you’re doing a lot of things that you really shouldn’t be doing with that, you know, with that workaround in place. You really should be doing something that’s a little bit more streamlined. And, you know, as you’re bringing new people in and new groups in, whether it’s a merger or whether it’s just another department that realizes that, Hey, you know, they have this, you know, shiny system over here. Why don’t we start using it too? If you have workarounds in place, it takes a lot longer to get people up and running in a new system because not only do they have to learn the system, but they have to learn how you’ve worked around it. AP: Right, and that’s where you start talking about technical debt because all of those workarounds that you’re describing, that equates to technical debt. And one day, you’re gonna get your backside handed to you because you have all of this technical debt. And replatforming is the perfect time to press that reset button and say, we’re gonna get rid of those things. We’re gonna have a system that addresses those problems in an official, correct way, and none of these weird workarounds.  BS: Mm-hmm. AP: And by the way, those workarounds, what if the person who did them leaves and hasn’t been documented well?  BS: Yeah, that’s a big problem. My guess is that if you’ve been using one particular system for, you know, 10 years or even more, you probably have a lot of content just sitting there that hasn’t been touched in years, hasn’t been needed in years, but it’s still sitting in the system. And it’s still coming up in search results as people looking to, you know, find topics that they need to edit for a new release. And it’s just getting in the way. It’s a good time to, you know, cut clean and, you know, ditch all of that old content that you no longer, that you know, you no longer need and focus on, you know, what you need to produce going forward. AP: Yeah, I think maybe sometime on the show Hoarders, they can do episodes on content people and their technical debt, and basically just hoarding all of this content in various digital forms all over that nobody’s actually looked at. I’m sure we would all laugh and be horrified at the same time by such a show.  BS: It wouldn’t make for exciting TV, though. AP: Yeah. So let’s talk about the overall process for how this kind of works. We’ve talked about what it is, a lot of the reasons for it. So let’s talk about how to do it. And it’s not a one-size-fits-all thing. We can tell you about our experiences that we’ve had. But I know one of the first things you got to do, for example, is choose your new system where you’re going to be moving into. BS: Right. And out of the gate, the knee-jerk reaction is to go with something new and shiny, but you really need to sit back and figure out what it is you need that system to be able to do and how you need that system to be able to do it. you know, we’ve had a lot of clients who’ve come to us after setting up a system, maybe two or three years prior, who just are like, this is just not working for us. And as we talk with them, we realized that, you know, they, essentially, you know, had a square peg and they bought a round hole to put it in. And here they are three years later, still trying to force that peg into the hole. So you need to sit back and really think about your requirements and not the requirements that you have today, but the requirements that you have today and anticipate having at least five years down the r

    23 min

  4. MAR 30

    Who controls your content? AI and content governance

    What does it actually mean to govern your content in the age of AI, and who’s really in control? In this episode, Sarah O’Keefe sits down with Patrick Bosek, CEO of Heretto, to unpack why the quality, accuracy, and structure of your content may be the most critical factors in what your users experience on the other side of an AI model. Patrick Bosek: In today’s world, you don’t have 100% control. There are a couple of different places where this needs to be broken up. One is the end user: what they physically get and what control they have versus what control you have. Then, there’s what control you have of how the AI model is going to behave based on your information and your inputs. Whether or that model is public, like a user accessing your documentation through Claude Desktop, or private, like a user accessing your documentation through your app or website, the governance piece comes down to what control you have immediately before the model. And that breaks down into a couple of things: completeness, accuracy, and structure of the content. Related links: AI and content: Avoiding disaster AI and accountability Structured content: a backbone for AI success Heretto Questions for Sarah and Patrick? Register for the Ask Me Anything session on April 8th at 11 am Eastern. LinkedIn: Sarah O’Keefe Patrick Bosek Transcript: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe. I’m here today with Patrick Bosek, who is the CEO of Heretto. Hey Patrick. Patrick Bosek: Hey, Sarah. Long time no chat. SO: That is, I guess for certain values of long time. We decided today that we wanted to talk about AI and governance, except I promptly tried to come up with a synonym for governance because I’m afraid that when I say that particular word, our audience just walks off. So, okay, Patrick, what is governance? PB: Well, so first of all, thanks for having me on, and second of all, I’m excited about this one because based on our little bit of chat before the show, it sounds like we’re actually gonna have some things to argue about this time around.  SO: I would never. PB: Well, usually we tend to agree right like I think that we’re generally pretty on the same page about stuff. So I’m excited. I’m pumped. Okay, so governance. I mean, obviously it has a ton of different meanings to different people but in the way that I want to talk about it today, because it was my suggestion. It’s related to the governance of content, specifically in the way of the inputs to AI systems. So you can think about the process of controlling for quality, accuracy, the things that matter in the actual content and information before it gets into the AI system. So it’s kind of the upstream quality, totality, structure, all of that checking and assurance ahead of whatever your experience is going to be downstream, of which one is the most contemporary and most interesting is AI. SO: Okay, so this is making sure that it is not garbage in so as to avoid garbage out.  PB: Yeah, I would say that’s a fair statement. SO: Yeah. Okay. And can we use AI to do governance of the content we’re producing? PB: Well, that’s actually a very interesting question. And I think the short answer is somewhat right now. So before I go, okay, before I like fully answer that, I want to put a little disclaimer in here. The stuff with AI is changing so quickly that we should date-stamp this episode. SO: It is March 19th, 2026. And it’s nine-ish Eastern time. PB: Yeah, we are recording this on March 19th, 2026. Now I feel, yeah. Okay, so now that people know when it is that we’re talking about this, I feel a little bit safer in answering. So there are aspects of governance you can do with AI today, for sure. And there’s new capabilities coming online all the time. I actually think, broadly speaking, the thing that’s going to be most challenging about governance is going to be the pieces that can’t be done with AI continuing to not continuing to do them because it becomes like as the human part of the loop becomes smaller and smaller, it becomes so much easier and easier for the human to just click accept because like the AI gets it right, does it, the automation works that kind of thing. And you know, I’ll use like an AI coding analogy because that’s what I spend a lot of time with AI on.  So I use Claude CLI. That’s my primary method of vibe coding or whatever you want to say. And I even find myself like just clicking accept sometimes. But I’m still forcing myself to like, get it, and read the code. And like, I had it write a shell script yesterday. And I was almost about to run it, and I was like, this is a shell script. I should not do that. I should definitely read what’s going on inside of this shell script, but it, gets to a point where like you start to trust it.  SO: Yeah. PB: And as we start to inject AI into the governance layer. So like we build skills that check certain parts of our information architecture or, you know, they kind of act as linters if we’re in docs as code or, you know, whatever it might be. There’s going to be like a form of trust that gets built up. And because we kind of like, tend to think of these agents as like human, they’re not, we tend to prescribe like a human form of trust, you know, like when you have a coworker that does the right thing all the time, you tend to just let them work. And I think that’s kind of the challenge and in the human side of governance. So that’s a really long way of saying. You can build tools and skills and patterns and things like that in AI that will help with governance. But fundamentally, it’s my belief that for the type of documentation or content that you and I work on, and I think most of our audience works on, which is has to be right, has to be accurate, has to conform to standards, et cetera, et cetera, right? It’s product documentation. It’s critical information. I still think that every single word needs to be read and considered by a human being. So really long answer to that question. SO: Right, and then fundamentally, if the AI is right half the time, then I’m going to read everything pretty carefully, knowing that 50% is wrong and I need to fix it. The problem, I think, is when it gets to be 90% correct, you just sort of glaze over because you’re looking for that last 10%, right? So it’s the difference between like doing a developmental edit, where you’re going deep into the words and just rearranging everything and fundamentally changing everything, versus doing a final proofread, where it is far more difficult to read 100 pages and find one typo than it is to read 100 pages that are just trash. And you’re like, start over, rearrange this, reformat everything. We’re not even worried about the typos yet because this is just fundamentally wrong. And so to your point, as it gets closer and closer, you start to believe in the output that it’s generating, which then means almost certainly that one typo, which in your example could be a shell script gone rogue, could be really, really problematic. PB: Yeah. And that’s going to be the challenge of our times in a lot of ways. I think there’s still going to be some aspect of origination that’s going to be necessary for quite some time. even with like automated drafting and pipelines like that, coming online, because in certain places, those work really, really well. but in other places, they, they don’t really work very well yet. It’s going to be the process of like becoming orchestrators in a way where, you know, we’re not rubber stamps, and we’re like really truly adding value and actually defending against the challenges that are going to come up with the automation that we build. SO: Fundamentally, like I saw a reference to this this morning and somebody said, you can write essentially an extractor that’s going to generate your release notes, right? So there’s code updates and you just automate the generation of release notes. Now, I personally am not so sure that you actually need AI for this. Given properly commented code, you could just generate the release notes, right? But setting aside that particular small argument in here. You automate, you can automate the generation of release notes because release notes are essentially, this is the delta between version one and version 1.01 or, know, and here are the changes. It’s a change log. What that means though is that the changes were captured in the code. They’re in the code, like the logic or the information is already there. What we’re doing is extracting it and reformatting it into something that a human can look at on a single page and say, okay, I understand what the changes are and how these apply to me as the user of the software and whether or not I should upgrade. That’s different than we’re going to introduce a new feature into this code and I need to write about why this feature is interesting and relevant to you. The question to me is where is new information being introduced into the system? Where is that information encoded? And then once it’s encoded, we can extract it and process and do thin

    40 min

  5. MAR 23

    Good content = good AI: The fundamentals that never change

    Good content fundamentals have been the foundation of effective product content for decades, and those same principles are exactly what make content AI-ready today. In this episode, Bill Swallow and Alan Pringle explain how attending to your hierarchy of content needs is the key to AI success. Alan Pringle: Right now, AI is not going to fix bad content problems. It is going to regurgitate that bad information, giving your end users information that’s flat out wrong. If your content at the basic source level is wrong, your AI by extension is going to be wrong. And that is the unglossy, unvarnished, hard truth that is still, I don’t think, seeping in like it should across the corporate world. Bill Swallow: It really does come back to the fact that, despite the world changing on a day-to-day basis, the fundamentals have not changed. Related links: A hierarchy of content needs Technical Writing 101, 3rd edition Structured content: a backbone for AI success LinkedIn: Alan Pringle Bill Swallow Transcript: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Bill Swallow: Hi, I’m Bill Swallow. Alan Pringle: And I’m Alan Pringle. BS: And in this episode, surprise surprise, we’re going to talk about content. AP: Really? Who would have thought? BS: But more specifically, what good content means today. Today, everything is all about AI. There is lots of change in progress with regard to AI tooling and content delivery with AI. But have the needs for content really changed? And I would say that off the bat, if you’re doing content right, you really don’t have to reinvent the wheel to make it AI acceptable. AP: No, in this crazy AI-hyped world we’re in, there’s some very basic foundational things that tend to get overlooked because they’re not sexy, and they’re not special and hot and whatever else. All that kind of marketing garbage that just sets me on complete edge and makes me want to say profane things in podcasts.  The bottom line is, there are things that the content world, and especially our little subdomain of it, product content world, has been doing for decades now. And I mean decades.  BS: Or should have been doing. AP: Correct. There are basic tenants that have been in place for decades. That if you’re following them, you are starting down the road of success with AI. I think to kind of prove our point, we’re going to step back and look at some of the things that Scriptorium has talked about and written in the past and see how it stacks up. And Bill, you found one. And let’s talk about that blog post that Sarah O’Keefe wrote. What was the date on that again? BS: It was 2014. And that is when we came up with the hierarchy of content needs. And it really wasn’t so much an invention as it was just a regurgitation of what it means to create good content. So we have a pyramid of content needs. At the bottom, we have available. So is content available? Does it exist? Can someone get to it? I think that we’ve mostly solved that problem given the dearth of information we have out on the internet. But as we know, that information is not always useful. So we go up a rung or a layer on that pyramid and see whether or not the content is accurate. And if it’s accurate, if it provides the correct information, that’s fantastic. Then we go up another level and see whether or not the content is actually appropriate. So it can be correct. It can exist. But is it appropriate? Does it meet a reader’s needs? And is it formatted in a way that works for the reader to ingest? Then we go up a step further and see whether or not the content is connected. And this is where we kind of get to the more modern aspect of content. Does it link out to correct additional resources? Is it available to people in a variety of means? And does it engage with the audience? And then finally, at the top of the pyramid, we have intelligent content. Is the content intelligent? And we’re not talking about AI here at all, but we are really talking about is the content fashioned in a way that it can be used intelligently across different media? AP: That it can be manipulated for different purposes. And that is quoting Sarah directly. And I think that is key here, because that is what AI does. It takes information and basically chops, slices, dices it, and provides it in a new way via a chatbot, for example. So that is that whole manipulation that Sarah is talking about. And we will post a link to the post in the show notes so you can read this at a greater detail to see how well this hierarchy of content needs has stood up. And she even talks about, for example, integrating database content, how you can pull in other information product specifications. If you think about it from an AI lens, I think that parallels pretty closely to the idea of retrieval augmented generation, where you are pulling content from other sources and kind of weaving it in with what an AI engine is providing you. So RAG is, I think, could be kind of interpreted as another way of integrating other information into the way that AI is processing that content. BS: Right, mean, because AI, I mean, it’s not really an audience, but it is a delivery point. There are some structural needs that need to happen there. But ultimately, you’re still writing for people. You might be writing in a way that it allows the AI to repurpose and refactor the information so that the audience gets exactly what they’re looking for. But it still needs to be somewhat tailored to the needs of people because AI in itself, it doesn’t care what the content is, but it’s going to try to produce something for an eventual person to be able to read. AP: I think that then in turn points to something else in our vast compendium of Scriptorium content. And that is a book that Sarah and I wrote, the first edition in 2000, which just kind of makes me shake my head. I know this is not a video podcast yet, but I’m shaking my head in disbelief. The book, Technical Writing 101, has three editions, published between 2000 and 2009. We will put a link in the show notes. You can still download the third edition. And by the way, it’s free. You can get a PDF or EPUB. It’s free. You can get it from our store with some more recent resources from the store.  But to me, I flipped through that book this morning. And I was genuinely surprised at how much of the advice on how to create good product content still is true in this AI era. Everything of talking about modular, writing things in a modular way, being very systematic and structuring things, even if you’re not using a structured authoring tool, use a template, make things very standardized. These are all things that, yes, they make for better, consistent, standard, tech-com, product content for the person reading it. But let’s pretend like AI is the person reading, and I’m doing air quotes here, reading it. It is going to do a better job of understanding, again, I’m sort of personifying here, and I know that’s sort of a no-no. But if you feed AI, a large language model, content that is very structured, that is very templatized, that is standardized, that is in bite-sized chunks, and also, this is very important, the idea of metadata, which we do talk about in that book briefly. We do talk about it. Because you need to be able to label it for different audiences, because I’m thinking about someone sitting, trying to use a product, trying to use a piece of software, talking to a chatbot. And the chatbot is going to ask it, what product are you using? What’s the model number? All of those kinds of things. And now we’re getting to this whole idea of labeling and breaking things apart so that a chatbot, just like a user of a product. Let’s say somebody has a printer that’s on the highest end of the scale. They’re going to have a lot more features that apply to their model than to someone who bought a more basic one. But the thing is, if your product content has not clearly labeled what are features in each of the models, the chatbot is going to spit out the wrong thing. So again, this idea of breaking things up in discrete chunks and labeling them in a way where someone who wants specific information about a specific model, they can get it. And it doesn’t matter if it’s from a web page, it’s from a PDF, a printed book, God forbid in 2026, or from an AI chatbot. Those rules still apply. Those fundamental principles are still there. BS: Mm-hmm. AP: I think one of the biggest problems here is when people do not have those fundamentals already in place, right? BS: If they don’t have those fundamentals in place, they can’t get to the top of that pyramid that Sarah was talking about. And really those fundamentals are those first three layers. Content is available, content is accurate and content is appropriate. If you can actually nail those three layers of the hierarchy of content needs, you are set to then jump to connected and intelligent fairly quickly because your content is already well written, standardized, and appropriate for different audiences. AP: So we’re right back to talking about the way you put content together, your content operations, and how you ha

    15 min

  6. FEB 16

    Check in on AI: The true measure of success for AI initiatives

    In this episode, Sarah O’Keefe and Alan Pringle explore how AI transforms content delivery from static documents into dynamic, consumer-driven experiences. However, the need for human-led governance is critical, and Sarah and Alan explore issues of accuracy, accountability, governance, and more. They challenge organizations to define AI success by its ability to deliver accurate, high-impact outcomes for the end user. Sarah O’Keefe: The metrics that are being used to measure the success of AI are all wrong. We should be measuring the success of various AI efforts based on, “Are people getting what they need? Are they having a successful outcome with whatever it is that they’re trying to do?” The metric we actually seem to be using is, “What percentage of your workflow is using AI? How many people can we get rid of because we’re automating everything with AI?” It’s the wrong metric. The question is, how good are the outcomes? Related links: Sarah O’Keefe: AI and content: Avoiding disaster Sarah O’Keefe: AI and accountability Alan Pringle: Structured content: a backbone for AI success Questions for Sarah and Alan? Register for our upcoming webinar, Ask Me Anything: AI in content ops. LinkedIn: Sarah O’Keefe Alan Pringle Transcript: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Alan Pringle: Hey everybody, I’m Alan Pringle, and today I’m here with Sarah O’Keefe, and we want to do something I’ve kind of dreaded to be honest, to do a check-in on AI in the content space. I’m very ambivalent about this topic. There’s still even two, three years in, there’s still a lot of hype, but there’s also been some good things that have emerged. We need to talk about it fairly realistically. So, Sarah, get ready. Let’s see if I can not curse during this. We’ll try. I’ll try my best not to be like that in this. Legitimately, there are some things that we need to talk about, and also about the challenges because I don’t think the content world is completely ready for a lot of what’s going on right now. Sarah O’Keefe: You know that we have AI that can remove cursing from podcasts, so I feel like we’re good here. AP: Well, also, it’s a challenge to me to behave in a PG-13 more family-friendly kind of way. So I’ll do my best.  SO: I have no idea what you’re talking about. AP: Yeah. So let’s start with the good and where things are right now with the positives. What is AI doing well right now? And let’s kind of get beyond the summarization. I think we can say objectively right now, in general, AI does a very good job of summarizing existing content. But I think it’s doing a lot more beyond that, and we should touch on those things instead. SO: The first thing that I would say is that summarization, but specifically the use case of a chatbot or a large learning model, an LLM, so now we’re talking about Claude, Gemini, ChatGPT and all the rest of them, which has the ability to provide an end user with a way of accessing information, an information access point that is different than what we had previously. In the olden days, you had a book, and you had to sort of flip it open and look at a table of contents or maybe an index and navigate to a page. Fine. Then along comes online content, and you can do full text search, or you can then go into an internet search, right? You type into the search bar, you get a bunch of results, you click, and you sort of, no, that’s not quite it. You modify your search string, you search again, and you sort of navigate your way to where you’re trying to go. With the interactivity of the, you know, ChatGPT class of tools. What happens is that I ask it a question and it gives me an answer. And then I say, that’s not quite what I wanted. And I can sort of zero in on exactly what I’m looking for and tell it, but actually make this easier. Or I don’t understand the words you’re using. Use simpler language. Give me more. Give me less. Give me a summary. Use this as a source. Do not use that as a source. It’s a new way to access information. People love it. There is something psychologically helpful about a conversational search. Now, there’s obviously huge issues with this, particularly around people, you know, using chatbots as their therapists, which introduces all sorts of horrifying, horrifying ethical issues.  AP: Personifying them as a person on the other end. Right. SO: But in the big picture, used well, it allows you to get to the information you’re looking for and get at it in the way that you want.  AP: There’s a control issue here. I don’t think the content consumer has ever had this level of control. SO: Yeah, and as a content consumer, that speaks to me. That is helpful. We’re seeing increasing use of, I would say, guardrails. So, not just slam out the AI with a bunch of stuff, but rather we’ve put some guardrails around it, and there’s various kinds of technologies that you can employ there. And that has been very helpful. And then the third thing I would point to is when we talk about generative AI and generating content, there’s a lot you can do in that sort low fidelity bucket. And what I mean here is I need an image for a presentation, but the background is the wrong color, so I can just swap it out. Now, I can do that with Photoshop. Well, some people can do that with Photoshop. AP: Well, I was about to say, don’t think you or I should be saying we can do the Photoshop because we kind of can’t. SO: Right. Well, and that’s exactly it. So it’s lowered the bar, right? Because I can tell the AI to swap out the background, and it will. And it applies a mid-level Photoshop capability to this image. And now I have the image that I need with a dark background so that the white text shows up in my presentation, that kind of thing.  AP: Right. Yeah. SO: We can do low-stakes synthetic audio if this podcast, which for the record we are recording with actual human beings, but let’s say that Alan curses extensively and we need to swap it out well, we could pretty easily generate some synthetic audio that sounds like him and that PG of eyes the original wording into something that is You know cleaner it would be way funnier to just bleep it. So I don’t know why we would do this but… AP: Correct. Well, and it may come to that. The bottom line is what you’re talking about here is things that have very low risk. This is more fun stuff, the thought of doing some of what we’re talking about and stuff that describes how to use a medical device, for example. Not sure I want to go there with that. But for something low stakes like some one-off presentation that you’re giving, maybe some humor is involved, I totally think that’s an acceptable use because there’s no risk there. SO: That’s really the key point because let’s say you’re writing content for a new medical device. Now you probably have a version one of said medical device, and you’re doing a version two. So, okay, fine. We take the version one content and we sort of, you know, say add color because that’s what we added, you know, in version two, and update all this stuff automatically. But it then becomes very important to actually read that, look at that information, look at all the images, make sure that everything is correct. And by the time you do that super carefully, you may have given back all the time that you saved on the back end when you basically made a copy and said generate the new version. There’s some, you have to be really careful with that, especially depending on what your stakes are in terms of regulating regulatory or compliance stuff.  You can, of course, get away with using AI, as you said, for low-stakes stuff. Now, the big risk you run there, and we’re seeing this in my favorite example of low-stakes content, which is video games, the video game industry has seen huge amounts of pushback against AI-generated game content, because it’s not fun. It’s not creative. It feels flat. It’s not art, and it’s not fun to play. And so it just becomes a slog. Again, same thing. Did you use it for maybe some backgrounds here and there? Okay. Did you use it to drive the story that you’re trying to establish or set up? You know, the enemies that you’re hypothetically fighting, and then they all have a certain sameness, or they all, you know, you’re sort of stealthing your way around the map. And it turns out that the AI-generated things are really dumb in that once they turn their back, you can do literally anything and they won’t notice because it was poorly designed. AP: Right, yeah. And that’s true even in the film entertainment industry. There’s been a tremendous amount of pushback for the very reason I read a review recently talking about a series of clips about history on, I believe it’s on YouTube, by a fairly well-known director I will not name. SO: Mm-hmm. AP: And some of the AI is frankly not done well. And one reviewer basically said that a lot of the people, when you look at the back of these AI-generated, like an AI-generated King George, the back of his head looks like a melted candle. This is not what we want here. If you’re so foc

    32 min

  7. JAN 26

    From black box to business tool: Making AI transparent and accountable

    As AI adoption accelerates, accountability and transparency issues are accumulating quickly. What should organizations be looking for, and what tools keep AI transparent? In this episode, Sarah O’Keefe sits down with Nathan Gilmour, the Chief Technical Officer of Writemore AI, to discuss a new approach to AI and accountability. Sarah O’Keefe: Okay. I’m not going to ask you why this is the only AI tool I’ve heard about that has this type of audit trail, because it seems like a fairly important thing to do. Nathan Gilmour: It is very important because there are information security policies. AI is this brand-new, shiny, incredibly powerful tool. But in the grand scheme of things, these large language models, the OpenAIs, the Claudes, the Geminis, they’re largely black boxes. We want to bring clarity to these black boxes and make them transparent, because organizations do want to implement AI tools to offer efficiencies or optimizations within their organizations. However, information security policies may not allow it. Related links: Sarah O’Keefe: AI and content: Avoiding disaster Sarah O’Keefe: AI and accountability Writemore AI LinkedIn: Nathan Gilmour Sarah O’Keefe Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone. I’m Sarah O’Keefe. Welcome to another episode. I am here today with Nathan Gilmour, who’s the Chief Technical Officer of Writemore AI. Nathan, welcome. Nathan Gilmour: Thanks, Sarah. Happy to be here. SO: Welcome aboard. So tell us a little bit about what you’re doing over there. You’ve got a new company and a new product that’s, what, a year old? NG: Give or take, yep. SO: Yep. So what are you up to over there? Is it AI-related? NG: It is actually AI-related, but not AI-related in the traditional sense. Right now, we’ve built a product or tool that helps technical authoring teams convert from traditional Word or PDF formats, which would make up the bulk of much of the technical documentation ecosystem and help convert it to structured authoring. Meaning that they can get all of the benefits of reuse, easier publishing, high compatibility with various content management systems. And can do it in minutes where traditional conversions could take hours. So it really helps authoring teams get their content out to the world at large in a much more efficient and regulated fashion. SO: So I pick up a corpus of 10 or 20 or 50,000 pages of stuff, and you’re going to take that, and you’re going to shove it into a magic black box, and out comes, you said, structured content, DITA? NG: Correct. SO: Out comes DITA. Okay. What does this actually … Give us the … That’s the 30,000-foot view. So what’s the parachute level view? NG: Perfect. Underneath the hood, it’s actually a very deterministic pipeline. Deterministic pipeline means that there is a lot more code supporting it. It’s not an AI inferring what it should do. There’s actual code that guides a conversion process first. So going from, let’s say, Word to DITA, there are tools within the DITA Open Toolkit that allow and facilitate that much more mechanically, rather than trusting an AI to do it. We know that AI does struggle with structure, especially as context windows expand. It becomes more and more inaccurate. So if we feed these models with far more mechanically created content, they become much more accurate. You’re not trusting them to do much more, more of the nuanced parts of the process. So there’s a big difference between determinism and probabilism. Where determinism is the mechanical conversion of something, probabilism is allowing the AI to infer a process. So that’s where we differ is our process is much more deterministic versus allowing the AI to do everything on its own. SO: So is it fair to say that you combined the … And for deterministic, I’m going to say scripting. But is it fair to say that you combined the DITA OT scripting processing with additional AI around that to improve the results? NG: Correct. It also expedites the results so that instead of having a human do much of the semantic understanding of the document, we allow the AI to do it in a far more focused task. Machines can read faster. SO: Okay. And so for most of us, when we start talking about AI, most people think large language model and specifically ChatGPT, but that’s not what this is. This is not like a front-end go play with it as a consumer. This is a tool for authors. NG: Correct. And even further to that, it’s a partner tool for authors. It allows them to continue authoring in a format that they’re familiar with. Well, let’s take Microsoft Word, for example. Sometimes the shift from Word to structured authoring could be considered an enormous upheaval. Allowing authors to continue authoring in a format that they’re good at and they’re familiar with, and then have a partner tool that allows them to expedite the conversion process to structured authoring so that they can maintain a single source of truth, makes things a little bit better, more manageable, and more reliable in the long run. So instead of having to effectively cause a riot with the authoring teams, we can empower them to continue doing what they’re good at. SO: Okay. So we drop the Word file in and magically DITA comes out. What if it’s not quite right? What if our AI doesn’t get it exactly right? I mean, how do I know that it’s not producing something that looks good, but is actually wrong? NG: Great question. And that’s where, prior to doing anything further, there is a review period for the human authors. So in the event that the AI does make a mistake, it is not only completely transparent, so the output, the payload, as we describe it, comes with a full audit report. So every determination that the AI makes is traced and tracked and explained. And then further to that, the humans are even able to take that payload out anyway, open it up in an XML editor. So at this point in time, the content is converted, it is ready to go into the CCMS. Prior to doing that, it can go into a subject matter expert who is familiar with structured authoring to do a final validation of the content to make sure that it is accurate. The biggest differentiator, though, is the tool never creates content. The humans need to create content because they are the subject matter experts within their field. They create the first draft. The tool takes it, converts it, but doesn’t change anything. It only works with the material as it stands. And then once that is complete, it goes back into another human-centered review so that there are audit trails, it is traceable. And there is a final touchpoint by a human prior to the final migration into their content management system. SO: So you’re saying that basically you can diff this. I mean, you can look at the before and the after and see where all the changes are coming in. NG: Correct. SO: Okay. I’m not going to ask you why this is the only AI tool I’ve heard about that has this type of audit trail, because it seems like a fairly important thing to do. NG: It is very important because there are information security policies. AI is this brand-new, shiny, incredibly powerful tool. But in the grand scheme of things, these large language models, the OpenAIs, the Claudes, the Geminis, they’re largely black boxes. Where we want to come in is to bring clarity to these black boxes. Make them transparent, I guess you can say. Because organizations do want to implement AI tools to offer efficiencies or optimizations within their organizations. However, information security policies may not allow it. One of the added benefits that we have baked into the tool from a backend perspective is its ability to be completely internet-unaware. Meaning if an organization has the capital and the infrastructure to host a model, this can be plugged directly into their existing AI infrastructure and use its brain. Which, realistically, is what the language model is. It’s just a brain. So if companies have invested the time, invested the capital in order to build out this infrastructure, the Writemore tool can plug right into it and follow those preexisting information security policies. Without having to worry about something going out to the worldwide web. SO: So the implication is that I can put this inside my very large organization with very strict information security policies and not be suddenly feeding my entire intellectual property corpus to a public-facing AI. NG: That is entirely correct. SO: We are not doing that. Okay. So I want to step back a tiny bit and think about what it means, because it seems like the thing that we’re circling around is accountability, right? What does it mean to use AI and still have accountability? And so, based on your experience of what you’ve been working on and building, what are some of the things that you’ve uncovered in terms of what we should be looking for generally as we’re building out AI-based things? What should we be looking for in terms of accountability of AI? NG: The major accountability of AI is what could it look like if a business model changes? Let’s kind of focus on the large players in the market right now. There

    21 min

  8. 11/17/2025

    Futureproof your content ops for the coming knowledge collapse

    What happens when AI accelerates faster than your content can keep up? In this podcast, host Sarah O’Keefe and guest Michael Iantosca break down the current state of AI in content operations and what it means for documentation teams and executives. Together, they offer a forward-thinking look at how professionals can respond, adapt, and lead in a rapidly shifting landscape. Sarah O’Keefe: How do you talk to executives about this? How do you find that balance between the promise of what these new tool sets can do for us, what automation looks like, and the risk that is introduced by the limitations of the technology? What’s the roadmap for somebody that’s trying to navigate this with people that are all-in on just getting the AI to do it? Michael Iantosca: We need to remind them that the current state of AI still carries with it a probabilistic nature. And no matter what we do, unless we add more deterministic structural methods to guardrail it, things are going to be wrong even when all the input is right. Related links: Scriptorium: AI and content: Avoiding disaster Scriptorium: The cost of knowledge graphs Michael Iantosca: The coming collapse of corporate knowledge: How AI is eating its own brain Michael Iantosca: The Wild West of AI Content Management and Metadata MIT report: 95% of generative AI pilots at companies are failing LinkedIn: Michael Iantosca Sarah O’Keefe Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe. In this episode, I’m delighted to welcome Michael Iantosca to the show. Michael is the Senior Director of Content Platforms and Content Engineering at Avalara and one of the leading voices both in content ops and understanding the importance of AI and technical content. He’s had a longish career in this space. And so today we wanted to talk about AI and content. The context for this is that a few weeks ago, Michael published an article entitled The coming collapse of corporate knowledge: How AI is eating its own brain. So perhaps that gives us the theme for the show today. Michael, welcome. Michael Iantosca: Thank you. I’m very honored to be here. Thank you for the opportunity. SO: Well, I appreciate you being here. I would not describe you as anti-technology, and you’ve built out a lot of complex systems, and you’re doing a lot of interesting stuff with AI components. But you have this article out here that’s basically kind of apocalyptic. So what are your concerns with AI? What’s keeping you up at night here?  MI: That’s a loaded question, but we’ll do the best we can to address it. I’m a consummate information developer as we used to call ourselves. I just started my 45th year in the profession. I’ve been fortunate that not only have I been mentored by some of the best people in the industry over the decades, but I was very fortunate to begin with AI in the early 90s when it was called expert systems. And then through the evolution of Watson and when generative AI really hit the mainstream, those of us that had been involved for a long time were… there was no surprise, we were already pretty well-versed. What we didn’t expect was the acceleration of it at this speed. So what I’d like to say sometimes is the thing that is changing fastest is the rate at which the rate of change is changing. And that couldn’t be more true than today. But content and knowledge is not a snapshot in time. It is a living, moving organism, ever evolving. And if you think about it, the large language models, they spent a fortune on chips and systems to train the big large language models on everything that they can possibly get their hands and fingers into. And they did that originally several years ago. And the assumption is that, especially for critical knowledge, is that that knowledge is static. Now they do rescan the sources on the web, but that’s no guarantee that those sources have been updated. Or, you know, the new content conflicts or confuses with the old content. How do they tell the difference between a version of IBM database 2 of its 13 different versions, and how you do different tasks across 13 versions? And can you imagine, especially when it comes to software where most of us, a lot of us work, the thousands and thousands of changes that are made to those programs in the user interfaces and the functionality? MI: And unless that content is kept up-to-date and not only the large language models, reconsume it, but the local vector databases on which a lot of chatbots and agenda workflows are being based. You’re basically dealing with out-of-date and incorrect content, especially in many doc shops. The resources are just not there to keep up with that volume and frequency of change. So we have a pending crisis, in my opinion. And the last thing we need to do is reduce the people that are the knowledge workers to update, not only create new content, but deal with the technical debt, so that we don’t collapse on this, I think, is a house of cards. SO: Yeah, it’s interesting. And as you’re saying that, I’m thinking we’ve talked a lot about content debt and issues of automation. But for the first time, it occurs to me to think about this more in terms of pollution. It’s an ongoing battle to scrub the air, to take out all the gunk that is being introduced that has to, on an ongoing basis, be taken out. Plus, you have this issue that information decays, right? In the sense that when, I published it a month ago, it was up to date. And then a year later, it’s wrong. Like it evolved, entropy happened, the product changed. And now there’s this delta or this gap between the way it was documented versus the way it is. And it seems like that’s what you’re talking about is that gap of not keeping up with the rate of change. MI: Mm-hmm. Yeah. I think it’s even more immediate than that. I think you’re right. But now we need to remember that development cycles have greatly accelerated. Now, when you bring AI for product development into the equation, we’re now looking at 30 and 60-day product cycles. When I started, a product cycle was five years. Now it’s a month or two. And if we start using AI to draft new content, for example, just brand new content, forget about the old content or update the old content. And we’re using AI to do that in the prototyping phase. We’re moving that more left upfront. We know that between then and CodeFreeze that there’s going to be a numerous number of changes to the product, to the function, to the code, to the UI. It’s always been difficult to keep up with it in the first place, but now we’re compressed even more. So we now need to start looking at AI to how does it help us even do that piece of it, let alone what might be a corpus that is years and years old, that’s not ever had enough technical writers to keep up with all the changes. So now we have a dual problem, including new content with this compressed development cycle. SO: So the, I mean, the AI hype says we essentially, we don’t need people anymore and the AI will do everything from coding the thing to documenting the thing to, I guess, buying the thing via some sort of an agentic workflow. But what, I mean, you’re deeper into this than nearly anybody else. What is the promise of the AI hype, and what’s the reality of what it can actually do? MI: That’s just the question of the day. Because those of us that are working in shops that have engineering resources, I have direct engineers that work for me and an extended engineering team. So does the likes of Amazon, other serious, not serious, but sizable shops with resources. We have a lot of shops that are smaller. They don’t have access to either their own dedicated content systems engineers or even their IT team to help them. First, I want to recognize that we’ve got a continuum out there, and the commercial providers are not providing anything to help us at this point. So it’s either you build it yourself today, and that’s happening. People are developing individual tools using AI where the more advanced shops are looking at developing entire agentic workflows.  And what we’re doing is looking at ways to accelerate that compressed timeframe for the content creators. And I want to use content creators a little more loosely because as we move the process left, and we involve our engineers, our programmers in the early, earlier in the phase, like they used to be, by the way, they used to write big specifications in my day. Boy, I want to go into a Gregorian chant. “Oh, in my day!” you know, but, but they don’t do that anymore. And basically the, the role of the content professional today is that of an investigative journalist. And you know what we do, right? We, we scrape and we claw. We test, we use, we interview, we use all of the capabilities of learning, of association, assimilation, synthesis, and of course, communication. And turns out that writing’s only 15% roughly of what the typical writer does in an information developer or technical documentation professional role, which is why we have a lot of different roles, by the way, that if we’re gonna replace or accelerate with people with AI, have to handle all those capabilities of a

    33 min
See All (197)

Ratings & Reviews

4.3
out of 5
7 Ratings

About

The Content Operations podcast from Scriptorium delivers industry-leading insights for scalable, global, AI-optimized content.

Information

  • Creator
    Scriptorium - The Content Strategy Experts
  • Years Active
    2017 - 2026
  • Episodes
    197
  • Rating
    Clean
  • Copyright
    © Scriptorium Publishing Services, Inc.
  • Show Website
    Content Operations

You Might Also Like