Psych Tech @ Work

Charles Handler

Science 4-Hire is now Psych Tech @ Work! - a podcast about safe innovation at the intersection of psychological science, technology, and the future of work. Psych Tech @ Work promotes safe technological innovation and human/machine partnerships as an essential force in creating equilibrium and between psychology and commerce.  Maintaining this balance in a time of unprecedented change is essential for ensuring that the future of work is ethical, positive, and prosperous.   Creating such a future requires an unprecedented level of interdisciplinary collaboration.  With the goal of educating, engaging, and inspiring others through thoughtful and practical discussions with guests from a wide variety of backgrounds and specialties, Psych Tech @ Work provides a smorgasbord of food for thought and practical takeaways about the issues that will make or break the future of work! charleshandler.substack.com

  1. 4d ago

    Plausible Isn't Valid: Why AI Still Needs Measurement Science

    “There’s tons of plausible summaries of information out there… but is it valid? Is it structured? Is it giving the same opportunity to each of the individuals that are participating? That’s where that science comes in.” — Tony Boyce Why is this topic important?: AI now automates sensitive portions of high stakes people processes: hiring, employee listening, job analysis, customer feedback. Its output almost always looks credible, whether or not it is accurate, consistent, or fair, and the people relying on it usually can’t tell. What is the one thing I will learn from listening?: Three questions to ask any AI tool that gathers information from people. Does it follow a structured guide? Does everyone get the same chance to be heard? Can every result be traced back to what the person actually said? Episode Overview I’m joined by my old friend Tony Boyce, an IO psychologist who has always worked at the frontier of science and technology: partner at Aon, principal research scientist at Amazon, and now co-founder of SAVO, a startup using AI-moderated conversations to gather information from people at scale. He also co-chaired this year’s SIOP Leading Edge Consortium on AI and assessment. We cover how Tony proved AI could reliably score spoken responses at Amazon (and the six-page pitch that won him a 40-developer team), how he builds rigor into AI-moderated conversations at SAVO, why surveys, focus groups, and multiple choice tests may be on their way out, and why tools this powerful still need scientists. Organizations are always trying to learn from people: what a job requires, why engagement dropped, what customers think, who is ready for a role. AI is moving into all of it. The gap between output that looks credible and output that is defensible is what this conversation is about. Topics Discussed & Key Insights 1. Plausible Is Not Valid AI tools that gather information from people are easy to build, and a vendor with no science team can produce output that looks identical to output built the right way. The difference is underneath: the same opportunity for everyone, a structured conversation, and results that can be traced and defended. Since the good and the bad look alike, buyers have to ask how the output was produced. “You can have a bot and a bot can ask you questions… but in order to actually do it in a way that it stays on the rails, that it follows the right threads, that it structures the conversation, is where the measurement science comes in.” 2. Where the Rigor Lives in an AI-Moderated Conversation SAVO isn’t a pre-hire assessment tool. It uses AI drive interviews to gather credible information from employees, customers, job experts, and candidates. The rigor lives in three places: * The spec. An instruction sheet for each conversation: the information needed, the follow-ups, what to listen for. Writing one by hand took three to four days, so an AI tool now drafts about 80% and a scientist finishes it. * The delivery. One agent builds rapport and runs the conversation. A second “metacognitive” agent tracks what has been learned and steers toward the gaps, within set guardrails. * The analysis. Each response is broken into evidence units and scored by multiple judges, building a report you can hang your hat on. “You’ve got all of this amazing unstructured data… but how do you process it in a reliable, valid, traceable way?” 3. What the Evidence Looks Like At Amazon, Tony’s team trained a model on roughly 2,000 hours of human-labeled recordings to score candidates’ spoken answers to a situational judgment scenario. The evidence had two parts. A number anyone can check: the model agreed with the human raters more closely than they agreed with each other. And known limits: when LLMs arrived in early 2025, the scoring carried over to new content and populations without retraining, but only within similarity limits the team defined. “Holy moly, this stuff actually works… and it’s reliable.” 4. The Old Trade-offs Are Breaking Gathering information from people has always meant trading depth (interviews, focus groups, surveys) against scale. AI is removing that trade-off. Within two years, Tony expects employee listening, customer listening, and user testing to see fewer focus groups, fewer human interviews, and almost no surveys. When an engagement score drops, a survey that held a short conversation, instead of offering an empty text box, will already tell you why. Job analysis interviews can run at scale instead of one at a time. And in hiring, he expects AI conversations and role plays to all but kill the multiple choice test. “For a century, anytime we’ve wanted to grab some information from humans, we’ve had to choose either depth and rigor… or scale and rigor… But you could never get all three together.” 5. Fragile Systems Need Scientists AI is powerful and brittle. Tony asked a decision model how confident it was that it would give the right answer: 90%. The wrong answer: also 90%. I’ve seen the same in AI pipelines for writing assessment items, where fixing one thing quietly breaks another. Trust is unresolved too: executive 360 raters said the experience felt fine but didn’t trust the result. None of this argues against AI. It argues for people whose job is to test it. “A simple turn of phrase and the decision quality or confidence quality of this thing just fell apart.” Final Takeaway AI makes it easy to produce output about people that looks right. Science determines whether it is right. The tools to gather richer information from people at scale exist today. What separates useful from dangerous is whether someone designed for validity, tested for failure, and can show the evidence. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    Plausible Isn't Valid: Why AI Still Needs Measurement Science

  2. Oct 15

    Calling All TA Leaders: Success Requires Embedded Science

    “We need to have rigor adjusted for reality.” — Allie Wehling Episode Overview I am lucky to be joined by my friend Allie Wehling. I consider Allie a superstar. She has achieved what I call the gold standard for internal I/Os. Allie is a rare example of having an I/O helping guide the business of hiring. At Cisco, she owns both the IO content and the end-to-end hiring process. She sees the whole system and how every piece affects the others. Most hiring functions treat science as a bolt-on: a test added to a process nobody designed with measurement in mind. That approach has value, but it leaves results to chance. Consistent, defensible hiring requires a holistic approach, where science, process, and business needs are designed together. Join us as we dig into what an embedded science lens brings to hiring, and why TA can’t afford to go without it. Topics Discussed & Key Insights 1. TA Leaders: Embed Science, Don’t Bolt It On Assessment usually gets added to hiring without anyone thinking about the whole process — piecemeal jamming assessments into legacy tools and processes. Allie’s role is built differently. Her teams cover both what gets measured and how it’s implemented, from requisition through pre-boarding. Having one leader across both is rare, and it gets results. “We have the WHAT, the content, the assessments from our COE, and then the HOW. How are we actually implementing it?” 2. Science Sees the Whole Funnel, Not Just the Test The assessment is one input. Every handoff, from assessment to interview to decisioning, introduces the potential for bias AND the potential for ROI. The trick is pulling both levers in the right direction. And we can’t forget that candidate experience is critical. When top candidates drop out because of slow scheduling or untrained interviewers, the process fails, not the test. “because you can have the most beautifully designed and validated assessment and still walk away with a bad hire because the assessment is just one input to the much longer process.” 3. Science Has to Be in Lockstep With the Business What’s scientifically best and what’s operationally viable are different questions, and both have to be solved. Process and technology have to be designed together. Allie’s team works “two in a box” with recruiting tech, and Cisco has hired an AI orchestration lead so the tools feel like one system. “they’re two different optimization problems. And if we pretend like they’re one, our systems are going to die quiet deaths.” 4. Science Prepares the Org. for Transformation Allie listens for the problems leaders haven’t voiced yet. When executives kept asking how Cisco assesses for AI, her team built a library of work samples to measure it. Bespoke assessments built from validated items give hiring managers a hand in the content. An internal AI tool sets cut scores against predicted pass rates instead of gut thresholds. “I try to listen to what problem are they not even voicing yet.” 5. Science Keeps You Defensible Job analysis documents job relatedness. Vendor claims get tested, not trusted. If a vendor can’t explain what its score predicts, or who’s on its science team, it’s not the right partner. “They’re predicting something, but is it the thing that we want to predict?” Final Takeaway Speed, volume, and AI are making hiring harder to get right. Science is what turns a pile of tools into a system that works effectively, consistently and holds up to scrutiny. It’s not a test you add. It’s a lens you build in. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    Calling All TA Leaders: Success Requires Embedded Science

  3. Sep 24

    You're Not Losing Your Job to AI. You're Handing It Over.

    “If you automate chaos you get chaos at scale.” — Tomas Chamorro-Premuzic Episode Overview In this episode I’m joined for a third time by Tomas Chamorro-Premuzic — organizational psychologist, author, and now the first Chief Science Officer at Russell Reynolds Associates, one of the world’s top executive leadership advisory firms. We set out to role play this conversation as if it were happening in 2027. We didn’t stick to it. We kept sliding between the future, the present, and whatever is right around the corner — which says more about this moment than the role play ever could. We’ll come back in a year to see how we did. Underneath the predictions is one argument: machines can only take what has already been standardized. Topics Discussed & Key Insights 1. The Future Is Already Here. It’s Just Not Evenly Distributed. Tomas recently returned from Shenzhen — stores staffed by humanoid robots, selling humanoid robots, with shoppers carrying them out in boxes. No human employees. His point, borrowing William Gibson: “the future is already here, just not evenly distributed.” What looks weird and niche today becomes mainstream fast. That’s the honest baseline for any one-year prediction: 95% of work will look the same next year. The question is which 5% won’t. 2. You Can’t Automate What Hasn’t Been Standardized The core flip of the episode. Automation doesn’t take jobs — it takes standardized work. As Tomas puts it: “you cannot automate what hasn’t been standardized already.” Which raises his uncomfortable question: are knowledge workers standardizing themselves? Leaning on AI for everything produces what he calls “work slop and cognitive surrender” — the intellectual version of elevator music. Designed to blend in. Designed to be ignored. The more predictable your output, the easier you are to model, and the easier you are to replace. 3. Hiring for Potential, Not Track Record His first concrete prediction: organizations will hire leaders less for what they have done and more for what they could do. AI enriches the signals needed to model curiosity, humility, coachability, and EQ — the things good assessment science has been chasing for decades. Technical skills, including the AI literacy everyone is worried about, “will be far less important than we think.” Human and humane skills will matter more. Track record measures what’s already been commoditized. Potential is what’s left. 4. Autonomous Hiring Splits the Market I ran a live query on my TA Tech Navigator during the episode. TA Tech Navigator is my online market research portal that contains 540 profiles of candidate evaluation and assessment tech. It has several frameworks that classify AI use and Science and can run trends reports using the aggregate data. The trend it flagged: fully autonomous hiring stacks are moving from fringe to viable. There are now vendors where no human touches the process until the final decision. Employers are splitting into two camps — buying efficiency or buying defensibility. Is the tool validated? Is there something here you can trust? That gap will widen, with regulation as the wildcard. The Navigator’s one-year read: faster, cheaper, more automated — but not fairer, more valid, or more defensible unless somebody demands it. 5. The Zombie Problems Tomas closes with a warning. Obsessing about the future is often an excuse for not dealing with present problems. Two have been with us for decades, and AI doesn’t touch either one: measuring leadership performance objectively, so tools predict actual performance instead of a popularity contest — and our stubborn preference for intuition over validated assessment. The same world that worships data, science, and validity is the world in which the MBTI is the number one personality assessment. New technology doesn’t kill these problems. Only deciding to solve them does. Final Takeaway Machines don’t take work. They take standardized work. The defensible position — for a knowledge worker, a leader, or a hiring process — is to stay hard to predict: keep thinking, keep surprising, and measure the things that actually differentiate people. That’s been the job all along. AI just raised the price of skipping it. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    You're Not Losing Your Job to AI. You're Handing It Over.

  4. Aug 11

    AI Hiring Law: What Changed While You Weren't Looking?

    “The way the law is working — the Workday case is exhibit A here — is that vendors are going to be on the hook if their systems are producing biased outcomes. Period.” — Marc Weinstein Episode Overview In this episode I’m joined by Marc Weinstein, an attorney with 27+ years of practice focused on organizations that use tests to make high-stakes decisions about people — licensure, certification, admissions, and employment selection. His work centers on the legal rights of people impacted by AI systems and the organizations that use them. The legal ground under AI hiring is moving fast. In the past few months a state AI law got sued into a rewrite, a court handed down a ruling that shields bias audit data from view, and a new lawsuit opened a line of attack nobody was watching. If you buy, sell, or use AI hiring tools, this episode helps you stay on top of the shifting sands. Topics Discussed & Key Insights 1. First, how to roll: governance before headlines. Marc’s advice starts above the regulations. Your own governance is the key to successful risk mitigation. Get this part right and the rest gets much easier. * Before you track any law, ask why you want to use AI in the first place. If you have good answers, then ask how * Set principles for your organization — not for the whole world, for your organization. Let them drive your use cases, your vendor selection, and your evaluation criteria * Operate by them for real. A policy that lives on paper is worth nothing * Do this and you’re already in compliance with most of the laws that could touch you. Everything below is what’s shifting underneath that foundation 2. Vendor liability is shifting fast — get ready now. For a long time, vendors selling hiring tools got a pass. How the customer used the tool was the customer’s problem. That arrangement is breaking down. * In Mobley v. Workday, the court is letting claims proceed on the theory that the platform acts as an agent of its employer customers — which puts the same federal anti-discrimination laws that apply to employers on the platform * Plaintiffs go where the deep pockets are. Mega vendors like Workday and 8fold are the deep pockets * Employers are still independently on the hook. Vendor liability adds to yours, it doesn’t replace it * Newer legislation points the same direction, and California’s new regulations make bias testing itself evidence in court — more on that below * Since we recorded: the court refused to dismiss most of the remaining claims against Workday. The case keeps moving, and the core theory keeps surviving 3. Bias audit data can be withheld — and that says a lot. A discovery ruling in the Workday case protected the company’s bias testing data from the plaintiffs. Worth sitting with what that means. * The court held the data was privileged because lawyers curated it and the testing was done to provide legal advice — not for business use * Publicly advertising that you conduct bias testing did not waive the privilege * So the people alleging harm from the system may never see the data that would show whether the system harmed them * The bigger question: if the most important evidence about how these tools perform can be shielded, what does accountability actually look like? Proof may have to come from other directions 4. There are new ways to hold these tools accountable. The Eightfold case shows creative lawyering finding routes into AI hiring that didn’t exist a year ago. * The claim is built on the Fair Credit Reporting Act and privacy law — the argument is that AI-generated candidate scores are consumer reports, compiled without the required disclosures and consent * These theories don’t require proving discrimination as a legal element, which makes them easier cases to bring * The target hasn’t changed. It’s still about how these tools treat people. There are just more ways to get there now 5. Colorado is the cautionary tale: get aggressive, get crushed. Colorado passed the most rigorous state AI law in the country — mandatory risk management, impact assessments, a real duty of care. Then it got taken apart. * xAI sued to block the law. The DOJ intervened on xAI’s side — the first time the federal government moved to invalidate a state AI law * The legislature rewrote the law under pressure. The replacement is a notice-and-transparency framework, delayed to January 2027 * The lesson for everyone else: states that regulate AI aggressively are going to get sued by the federal government and the private sector. Plan around that reality 6. Meanwhile, California quietly became the strictest. While everyone watched Colorado, California put real requirements on the books — and nobody has sued to stop them. * Under the FEHA regulations, conducting a documented bias test supports an employer’s defense in discrimination litigation. Not conducting one supports the plaintiff. Records must be kept four years * The privacy agency’s automated decision-making rules take effect January 2027: pre-use notice to applicants and employees, opt-out rights, and the right to demand how a system reached a decision about you * That last one matters. Can anyone actually explain how a frontier AI model reached its decision? For most of these tools, the honest answer is no Final Takeaway The laws will keep shifting. Colorado got rewritten, California’s deadlines are coming, and the courts are redrawing vendor liability case by case. You can’t chase all of it. Build the governance foundation, make vendors show you their evidence — your data, not just their study — and treat every recommendation a machine makes as the decision it really is. Good governance gets you more than 90% of the way there, as long as it’s not just lip service. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    AI Hiring Law: What Changed While You Weren't Looking?

  5. Jun 29

    You Bought the AI. So Why Aren't You Using It?

    “You buy the hiring platform, and then you don’t have these AI workflows enabled... here you are three, four years later. You’re still not able to do that.” — Nicole Mundy Episode Overview In this episode I’m joined by Nicole Mundy, Senior Research Analyst at Talentech Labs — a research and advisory firm that helps its enterprise clients evaluate and procure hiring technology systems. Nicole brings a valuable perspective to the table because she sees the dynamic between vendor and buyer up close and personal. When combined with my experience in this same realm from the science side- our discussion shines light on the reality of what is happening in AI tech adoption for TA. Topics Discussed & Key Insights 1. Companies are buying AI hiring tools at scale — and then leaving them switched off. Among the world’s top enterprise organizations, Nicole estimates a surprisingly small amount are actually using AI to automatically assess active applicants. Why? Approvals never come - Companies buy the platform intending to enable the AI, “once the right approvals are in place- but years often pass without any change. Pilots underperform - Big companies test these tools and often conclude they can’t really use them, or they just didn’t work. Lack of solid ROI evidence - Despite the best intentions- most companies do not do the follow up work needed to demonstrate the impact of these tools on the bottom line. Legal ambiguity freezes decisions - With regulations constantly in flux, risk management often takes priority over business needs. 2. Validation is misunderstood and absent. What vendors without I/O science guidance call validation isn’t what legal compliance actually requires. * Vendors are quick to speak about the validity of their solution and talk endlessly about validating their AI models — running statistical checks that the model predicts consistently and de-biasing its outputs across groups. This is purely empirical work. * But that’s IT-style validation. It confirms the system runs as built; it says nothing about whether the tool is fair or job-related * Validation for legal compliance, and sound science, demands a blend of rational and empirical work to document the job-relatedness of any tool used to make employment decisions 3. “Skills” are everywhere, and nowhere. Skills-based hiring is the headline everyone wants. The problem is what counts as a skill. * Most platforms apply the “skill” label with no objective framework to define it. A skill ends up being little more than a tag like “Excel” or “communication” * The definitions behind these labels are usually poorly organized and loosely constructed. * There is no connection between the skills a platform claims to measure and any outcome on the job. Without that link, there is nothing for a buyer to trust or defend. * The companies doing skills-based hiring well are not buying one vendor and flipping a switch. They run multi-year programs: define the skills objectively, inventory what the organization has against what it needs, curate tools carefully on the front end, and collect assessment data at multiple points. 4. Cheating is a zero-sum game, so let’s change the rules AI-assisted candidate fraud, from AI completing assessments to coaching candidates through interviews, is driving enterprises toward more dynamic evaluation that is harder to game. But chasing detection is largely a losing battle. * Trying to catch and block AI use is whack-a-mole. It’s a zero-sum game, and it’s frustrating, because the tools keep getting better and the detection never really gets ahead * There’s a more useful way to think about it. People are going to use AI on the job, so why not let them use it in the application process in a controlled way? The question stops being “did they use AI” and becomes “how well do they use it” * Most large organizations are still just trying to get visibility on how much cheating is happening and what it looks like, which says how early everyone is on this. * Across the approaches Nicole sees, one thing tends to hold up whether a process is locked down or fully AI-assisted: competency-based follow-up questions that make candidates explain their own reasoning in their own words. Final Takeaway Enterprise isn’t slow on AI in hiring because it doesn’t understand the technology. It’s slow because the tools are bought on vendor claims that were never reviewed against real science, and the danger only becomes clear once the tool is in play. The companies getting it right aren’t chasing tools. They’re building programs, science first. Held to that bar, many of the tools on the market today wouldn’t survive the review, and the ones that would wouldn’t be sitting switched off. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    You Bought the AI. So Why Aren't You Using It?

  6. Jun 4

    “The answer isn’t more AI — it’s better signal.”

    In this episode I’m joined by Robert Newry, Founder & CEO of the assessment company Arctic Shores and long time champion of doing assessment right! Robert and I (and my AI co-host Mayda Tokens!) dig into one of the most urgent problems in hiring right now: the complete breakdown of traditional hiring signals. We ponder the question- “How do we find the truth in an age where AI has flooded the top of the funnel, made credentials and resumes unreliable, and put enormous pressure on organizations to find new ways to identify talent?” And we come up with some pretty good answers! 1. The Top of the Funnel Is in Chaos The numbers are staggering. Accenture’s global resourcing lead told Robert they’re on pace for 12 million applications this year for roughly 100,000 hires — up from 4 million just three years ago. Same size team. Two and a half times the volume. The culprit isn’t a surge in qualified candidates; it’s AI-powered application tools that let candidates apply to jobs while they sleep. The moral contract between candidates and employers has been broken: candidates assume companies are using AI to screen, so they’re using AI to apply. “It’s chaos out there. Candidates are using AI to fight AI — and we’re in a no-win scenario.” 2. Traditional Assessment Is Increasingly Gameable Arctic Shores’ research from 18 months ago showed what most people didn’t want to admit: AI can ace virtually any traditional assessment format — personality tests, cognitive reasoning, multiple choice — with ease. And it’s not just about having a second screen open. Candidates can now point a phone at their screen, have the AI read the item, and get the answer instantly. Proctoring doesn’t solve this. The old protection mechanisms are obsolete. 3. The Answer Is Better Signal, Not More AI The solution isn’t to ban AI from the process — it’s to design assessments that AI can’t easily game because they’re rooted in authentic behavior. Robert’s framework: if AI is being used to evaluate signals, those signals have to be grounded in high-fidelity behavioral data — not scraped from job descriptions, not inferred from keyword matching, not built on garbage in. Job descriptions themselves are often the first failure point, and no amount of downstream AI sophistication fixes a weak foundation. 4. Stop Counting Leaves — Look at the Roots Robert’s tree analogy is one of the sharpest frameworks in this episode. For decades, hiring has been obsessed with leaves — the skills on a resume, the credentials on a LinkedIn profile. But with the average shelf life of a skill now estimated at two and a half years, leaves are increasingly unreliable. What matters is the root system: the durable human capabilities that allow someone to grow new skills, adapt to changing roles, and thrive in uncertainty. 5. Skills-Based Hiring Needs a Clearer Definition of “Skill” Both Robert and I agree: the skills-based hiring movement is directionally right, but conceptually messy. Calling “innovation” or “persistence” a skill conflates what can be learned with what is innate. Durable traits — personality, cognitive style, learning orientation — don’t expire the way technical skills do. Measurement strategy has to account for these differences, or skills-based hiring just becomes the next echo chamber. Final Takeaway The hiring signal crisis is real — and it’s accelerating. AI has made it trivially easy to fake credentials, game traditional assessments, and flood the funnel with noise. The organizations that receive the best signal won’t be the ones that deploy the most AI. They’ll be the ones that invest in the right signal: behavior-based, validated, and rooted in the durable human traits that no machine can fake. *Claude.ai assisted with the creation of these show notes This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    “The answer isn’t more AI — it’s better signal.”

  7. Apr 27

    Going All In: Using AI to Build Better Assessments

    “By the time you dot the final I’s and cross the final T’s, the assessment is already out of date.” — Taylor Sullivan Episode Overview In this episode I’m joined by rising I/O rockstar Taylor Sullivan, IO psychologist and the architect of Workera’s assessment strategy. With Taylor’s guidance Workera, a verified skills intelligence platform, is doing something most of the industry is still afraid to do: going all in on using AI to build, deliver, and validate AI-based assessments. Taylor and I (and my AI co-host Mayda Tokens) dig into how this actually works, why it’s scientifically defensible, and why the industry needs to stop waiting and start moving. Topics Discussed & Key Insights 1. Traditional Assessment Development Is Already Broken By the time a traditional assessment clears all the I-dotting and T-crossing, it’s often already out of date. AI changes that — enabling dynamic content generation, richer construct understanding, and real-time iteration that keeps pace with how work actually evolves. 2. Codifying Measurement Science Into a Multi-Agent System Workera didn’t just bolt AI onto existing processes. They embedded IO psychology’s core principles — evidence-centered design, validity frameworks, job analysis — directly into a multi-agent authoring system. Experts define the standards. Agents execute to those standards. The science drives the machine, not the other way around. Here’s a brief sketch of how it works in practice * Define the purpose — Tell the agent what you’re measuring and why. This grounds everything that follows. * Extract the construct — The agent probes the skill space using critical incident techniques, identifying what great performance actually looks like. * Design the assessment — The agent selects question formats (multiple choice, drag and drop, voice interaction, sequencing) based on what will best elicit evidence of the skill. * Automated quality review — Before anything goes live, the system checks for bias, language issues, and content alignment to the original skill definition. * Monitor and improve — Once deployed, the agent tracks response patterns, flags problems, and learns from score appeals adjudicated by humans. The skill domain is flexible — it works for cheeseburgers or cybersecurity. The methodology behind it is the same either way. 3. The “Harness” — Why This Is Safe The key to responsible agentic AI isn’t less autonomy — it’s a well-designed harness (the constrained ecosystem where the agents do their thing). Human experts define what good looks like, set quality thresholds, and build in escalation points. The agents work within those constraints and loop back when they hit uncertainty. As Taylor puts it: “It’s not running completely autonomously unchecked.” 4. This Is About Development, Not Just Hiring Workera’s primary focus is post-hire — workforce development, upskilling, and learning. Once an assessment identifies verified gaps in a person’s skills, the platform connects those gaps directly to personalized learning plans, curating from an organization’s existing content library. Two people can get the same score on an assessment and walk away with completely different development paths based on their specific pattern of strengths and gaps. 5. Verified Skills Intelligence — What It Actually Means In a world where AI can write a perfect resume and LinkedIn profile for anyone, credentials are noise. Verified skills intelligence cuts through that — using assessment to generate actual evidence of what someone can do, fit for the stakes of the decision being made. Final Takeaway The tools to move beyond multiple choice, beyond static assessments, and beyond slow validation cycles exist today. The bottleneck isn’t technology — it’s the will to trust well-designed systems. When the science is built into the machine from the start, speed and rigor aren’t in conflict. They’re the same thing. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    Going All In: Using AI to Build Better Assessments

  8. Mar 30

    Quality Research Shows the Real Impact of AI @ Work

    Quote: “If you know what you’re doing, AI makes you faster. If you don’t, it just makes you wrong faster.” –Louis Hickman In this episode I’m joined by esteemed Psych Tech @ Work, Alumnus and AI research machine, Louis Hickman. Our incredible conversation taps into Louis’ myriad research studies to unpack AI’s direct impact on work, domain expertise, and talent assessment. And of course, this episode also marks the return of the now new and improved AI podcast co-host Mayda Tokens (2.0). Besides telling dumb jokes- Mayda’s job is to remind us that AI isn’t just a tool — it’s becoming an active participant in how we think, question, and explore ideas. In the course of our conversation Mayda and I coax some PROFOUND take aways from our friend Louis as he shares the practical outcomes of his research: 1. AI is not removing the need for expertise — it’s making it more visible. Scaling intelligence is easy.Scaling judgment is not. The organizations that succeed won’t be the ones that adopt AI the fastest. They’ll be the ones that: * Understand what they’re measuring * Use AI to enhance — not replace — that understanding’ * Maintain control over how decisions are made 2. AI allows us to scale both good science and bad measurement Louis pushes back on the idea that recent advances represent a fundamental shift in how we measure people. Instead, what we’re seeing is: * Better models * Faster processing * More scalable systems But none of that replaces the need for valid, reliable, and job-relevant measurement. 3. AI doesn’t level the playing field — it often rewards those who already understand the game. One of the most interesting ideas in this episode is how AI interacts with individual differences in expertise. At a high level: * For simple tasks, AI helps novices perform closer to experts * For complex tasks, AI actually widens the gap- allowing experts to perform better Why? Because experts know how to ask better questions, recognize when AI is wrong, and refine its outputs—while novices often lack the ability to judge quality, diagnose errors, or course-correct when things go off track. 4. Replicability in LLMs Is Possible — if you know how to set it up right A major “wow” moment in Louis’ research: By running the model locally on the same class of hardware, fixing the model and prompt, and turning off sampling/randomness in the settings, you can make the system produce the same output for the same input every time. 5. AI should be used to scale decisions, but those decisions still need to be grounded in clearly defined constructs At this point, AI adoption isn’t optional—it’s expected. Organizations are being pushed to move faster and scale, while vendors are rapidly building and deploying solutions, often without deep validation. The resolution isn’t to slow down adoption—it’s to ensure we add and maintaining rigor. 6. AI makes it easy to scale assessment, but if the underlying design is weak, we’re just scaling bad measurement faster. The resolution is to ensure what gets scaled is built on clear constructs, strong design, and validated measurement, so speed amplifies quality—not noise. 7. Working with AI is no longer just about what you can do—it’s about how effectively you can partner to make what you do better! The tension is clear: AI can accelerate work, but over-reliance without critical evaluation leads to lower quality, missed errors, and reduced trust. This shows up in real ways—unchecked outputs, declining attention to detail, and growing skepticism in collaborative work. The resolution is that AI doesn’t replace accountability—users still need to apply judgment, review outputs, and take ownership of the final result. Tune in to get the full story on these profound revelations and hear Mayda’s stand up comedy routine. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com

    Quality Research Shows the Real Impact of AI @ Work
See All (114)

Ratings & Reviews

5
out of 5
6 Ratings

About

Science 4-Hire is now Psych Tech @ Work! - a podcast about safe innovation at the intersection of psychological science, technology, and the future of work. Psych Tech @ Work promotes safe technological innovation and human/machine partnerships as an essential force in creating equilibrium and between psychology and commerce.  Maintaining this balance in a time of unprecedented change is essential for ensuring that the future of work is ethical, positive, and prosperous.   Creating such a future requires an unprecedented level of interdisciplinary collaboration.  With the goal of educating, engaging, and inspiring others through thoughtful and practical discussions with guests from a wide variety of backgrounds and specialties, Psych Tech @ Work provides a smorgasbord of food for thought and practical takeaways about the issues that will make or break the future of work! charleshandler.substack.com

Information

  • Creator
    Charles Handler
  • Years Active
    2018 - 2026
  • Episodes
    114
  • Rating
    Explicit
  • Copyright
    © Charles Handler
  • Show Website
    Psych Tech @ Work

You Might Also Like