92 episodes

How do you create innovative machine learning and analytics products? Brian T. O’Neill reveals the strategies and activities that CxOs and innovative leaders in technical product management, data science and analytics are using to deliver indispensable experiences around data. From traditional analytics to machine learning and AI, Brian and his guests explore how extraordinary value can be created when the outputs of data science and analytics are turned into engaging, valuable decision support applications and user experiences centered around the humans in the loop. Experiencing Data also features special guests on design, ethics, explainable AI (XAI), and innovation who relate their expertise to the world of data-driven software. If you're in charge of creating simple, valuable, human-centered data products that produce business value in the last mile, you'll enjoy #ExperiencingData.

Transcripts available at: https://designingforanalytics.com/ed

ABOUT THE HOST
Brian T. O’Neill is a consulting product designer who helps companies create innovative ML and analytics solutions. He is also the founder and principal of Designing for Analytics.…and a professional percussionist/drummer.

Experiencing Data with Brian T. O'Neill Brian T. O'Neill from Designing for Analytics

    • Technology
    • 5.0 • 26 Ratings

How do you create innovative machine learning and analytics products? Brian T. O’Neill reveals the strategies and activities that CxOs and innovative leaders in technical product management, data science and analytics are using to deliver indispensable experiences around data. From traditional analytics to machine learning and AI, Brian and his guests explore how extraordinary value can be created when the outputs of data science and analytics are turned into engaging, valuable decision support applications and user experiences centered around the humans in the loop. Experiencing Data also features special guests on design, ethics, explainable AI (XAI), and innovation who relate their expertise to the world of data-driven software. If you're in charge of creating simple, valuable, human-centered data products that produce business value in the last mile, you'll enjoy #ExperiencingData.

Transcripts available at: https://designingforanalytics.com/ed

ABOUT THE HOST
Brian T. O’Neill is a consulting product designer who helps companies create innovative ML and analytics solutions. He is also the founder and principal of Designing for Analytics.…and a professional percussionist/drummer.

    091 - How Brazil’s Biggest Fiber Company, Oi, Leverages Design To Create Useful Data Products with Sr. Exec. Design Manager, João Critis

    091 - How Brazil’s Biggest Fiber Company, Oi, Leverages Design To Create Useful Data Products with Sr. Exec. Design Manager, João Critis

    Today I talked with João Critis from Oi. Oi is a Brazilian telecommunications company that is a pioneer in convergent broadband services, pay TV, and local and long-distance voice transmission. They operate the largest fiber optics network in Brazil which reaches remote areas to promote digital inclusion of the population. João manages a design team at Oi that is responsible for the front end of data products including dashboards, reports, and all things data visualization. 

    We begin by discussing João’s role leading a team of data designers. João then explains what data products actually are, and who makes up his team’s users and customers. João goes on to discuss user adoption challenges at Oi and the methods they use to uncover what users need in the last mile. He then explains the specific challenges his team has faced, particularly with middle management, and how his team builds credibility with senior leadership. In conclusion, João reflects on the value of empathy in the design process. 

     

    In this episode, João shares:  

    A data product  (4:48)
    The research process used by his data teams to build journey maps for clients (7:31)
    User adoption challenges for Oi (15:27)
    His answer to the question “how do you decide which mouths to feed?” (16:56)
    The unique challenges of middle management in delivering useful data products (20:33)
    The importance of empathy in innovation (25:23)
    What data scientists need to learn about design and vice versa (27:55)

     

    Quotes from Today’s Episode

    “We put the final user in the center of our process. We [conduct] workshops involving co-creation and prototyping, and we test how people work with data.” - João (8:22)


    "My first responsibility here is value generation. So, if you have to take two or three steps back, another brainstorm, rethink, and rebuild something that works…. [well], this is very common for us.” - João (19:28)


    “If you don’t make an impact on the individuals, you’re not going to make an impact on the business. Because as you said, if they don’t use any of the outputs we make, then they really aren’t solutions and no value is created. - Brian (25:07)


    “It’s really important to do what we call primary research where you’re directly interfacing as much as possible with the horse’s mouth, no third parties, no second parties. You’ve really got to develop that empathy.” - Brian (25:23)


    “When we are designing some system or screen or other digital artifact, [we have to understand] this is not only digital, but a product. We have to understand people, how people interact with systems, with computers, and how people interact with visual presentations.” - João (28:16)

    Links
    Oi: https://www.oi.com.br/
    LinkedIn: https://www.linkedin.com/in/critis/
    Instagram: https://www.instagram.com/critis/

    • 31 min
    090 - Michelle Carney’s Mission With MLUX: Bringing UX and Machine Learning Together

    090 - Michelle Carney’s Mission With MLUX: Bringing UX and Machine Learning Together

    Michelle Carney began her career in the worlds of neuroscience and machine learning where she worked on the original Python Notebooks. As she fine-tuned ML models and started to notice discrepancies in the human experience of using these models, her interest turned towards UX. Michelle discusses how her work today as a UX researcher at Google impacts her work with teams leveraging ML in their applications. She explains how her interest in the crossover of ML and UX led her to start MLUX, a collection of meet-up events where professionals from both data science and design can connect and share methods and ideas. MLUX now hosts meet-ups in several locations as well as virtually. 

    Our conversation begins with Michelle’s explanation of how she teaches data scientists to integrate UX into the development of their products. As a teacher, Michelle utilizes the IDEO Design Kit with her students at the Stanford School of Design (d.school). In her teaching she shares some of the unlearning that data scientists need to do when trying to approach their work with a UX perspective in her course, Designing Machine Learning.

    Finally, we also discussed what UX designers need to know about designing for ML/AI. Michelle also talks about how model interpretability is a facet of UX design and why model accuracy isn’t always the most important element of a ML application. Michelle ends the conversation with an emphasis on the need for more interdisciplinary voices in the fields of ML and AI. 

     

    Skip to a topic here:

    Michelle talks about what drove her career shift from machine learning and neuroscience to user experience (1:15)
    Michelle explains what MLUX is (4:40)
    How to get ML teams on board with the importance of user experience (6:54)
    Michelle discusses the “unlearning” data scientists might have to do as they reconsider ML from a UX perspective (9:15)
    Brian and Michelle talk about the importance of considering the UX from the beginning of model development  (10:45)
    Michelle expounds on different ways to measure the effectiveness of user experience (15:10)
    Brian and Michelle talk about what is driving the increase in the need for designers on ML teams (19:59)
    Michelle explains the role of design around model interpretability and explainability (24:44)

     

    Quotes from Today’s Episode
    “The first step to business value is the hurdle of adoption. A user has to be willing to try—and care—before you ever will get to business value.” - Brian O’Neill (13:01)


    “There’s so much talk about business value and there’s very little talk about adoption. I think providing value to the end-user is the gateway to getting any business value. If you’re building anything that has a human in the loop that’s not fully automated, you can’t get to business value if you don’t get through the first gate of adoption.” - Brian O’Neill (13:17)


    “I think that designers who are able to design for ambiguity are going to be the ones that tackle a lot of this AI and ML stuff.” - Michelle Carney (19:43)


    “That’s something that we have to think about with our ML models. We’re coming into this user’s life where there’s a lot of other things going on and our model is not their top priority, so we should design it so that it fits into their ecosystem.” - Michelle Carney (3:27)


    “If we aren’t thinking about privacy and ethics and explainability and usability from the beginning, then it’s not going to be embedded into our products. If we just treat usability of our ML models as a checkbox, then it just plays the role of a compliance function.” - Michelle Carney (11:52)


    “I don’t think you need to know ML or machine learning in order to design for ML and machine learning. You don’t need to understand how to build a model, you need to understand what the model does. You need to understand what the inputs and the outputs are.” - Michelle Carney (18:45)




    Links
    Twitter @mluxmeetup: https://twitter.com/

    • 31 min
    089 - Reader Questions Answered about Dashboard UX Design

    089 - Reader Questions Answered about Dashboard UX Design

    Dashboards are at the forefront of today’s episode, and so I will be responding to some reader questions who wrote in to one of my weekly mailing list missives about this topic. I’ve not talked much about dashboards despite their frequent appearance in data product UIs, and in this episode, I’ll explain why. Here are some of the key points and the original questions asked in this episode:



    My introduction to dashboards (00:00)
    Some overall thoughts on dashboards (02:50)
    What the risk is to the user if the insights are wrong or misinterpreted (4:56)
    Your data outputs create an experience, whether intentional or not (07:13)
    John asks:
    How do we figure out exactly what the jobs are that the dashboard user is trying to do? Are they building next year's budget or looking for broken widgets?  What does this user value today? Is a low resource utilization percentage something to be celebrated or avoided for this dashboard user today?  (13:05)
    Value is not intrinsically in the dashboard (18:47)
    Mareike asks:
    How do we provide Information in a way that people are able to act upon the presented Information?  How do we translate the presented Information into action? What can we learn about user expectation management when designing dashboard/analytics solutions? (22:00)
    The change towards predictive and prescriptive analytics (24:30)
    The upfront work that needs to get done before the technology is in front of the user (30:20)
    James asks:
    How can we get people to focus less on the assumption-laden and often restrictive term "dashboard", and instead worry about designing solutions focused on outcomes for particular personas and workflows that happen to have some or all of the typical ingredients associated with the catch-all term "dashboards?” (33:30)
    Stop measuring the creation of outputs and focus on the user workflows and the jobs to be done (37:00)
    The data product manager shouldn’t just be focused on deliverables (42:28)

     

    Quotes from Today’s Episode
    “The term dashboards is almost meaningless today, it seems to mean almost any home default screen in a data product. It also can just mean a report. For others, it means an entire monitoring tool, for some, it means the summary of a bunch of data that lives in some other reports. The terms are all over the place.”- Brian (@rhythmspice) (01:36)


    “The big idea here that I really want leaders to be thinking about here is you need to get your teams focused on workflows—sometimes called jobs to be done—and the downstream decisions that users want to make with machine-learning or analytical insights. ” - Brian (@rhythmspice) (06:12)


    “This idea of human-centered design and user experience is really about trying to fit the technology into their world, from their perspective as opposed to building something in isolation where we then try to get them to adopt our thing.  This may be out of phase with the way people like to do their work and may lead to a much higher barrier to adoption.” - Brian (@rhythmspice) (14:30)


    “Leaders who want their data science and analytics efforts to show value really need to understand that value is not intrinsically in the dashboard or the model or the engineering or the analysis.” - Brian (@rhythmspice) (18:45)


    “There's a whole bunch of plumbing that needs to be done, and it’s really difficult. The tool that we end up generating in those situations tends to be a tool that’s modeled around the data and not modeled around [the customers] mental model of this space, the customer purchase space, the marketing spend space, the sales conversion, or propensity-to-buy space.” - Brian (@rhythmspice) (27:48)


    “Data product managers should be these problem owners, if there has to be a single entity for this. When we’re talking about different initiatives in the enterprise or for a commercial software company, it’s really sits at this product management function.”  - Brian (@rhythmspice) (34:42)


    “It’s

    • 48 min
    088 - Doing UX Research for Data Products and The Magic of Qualitative User Feedback with Mike Oren, Head of Design Research at Klaviyo

    088 - Doing UX Research for Data Products and The Magic of Qualitative User Feedback with Mike Oren, Head of Design Research at Klaviyo

    Mike Oren, Head of Design Research at Klaviyo, joins today’s episode to discuss how we do UX research for data products—and why qualitative research matters. Mike and I recently met in Lou Rosenfeld’s Quant vs. Qual group, which is for people interested in both qualitative and quantitative methods for conducting user research. Mike goes into the details on how Klaviyo and his teams are identifying what customers need through research, how they use data to get to that point, what data scientists and non-UX professionals need to know about conducting UX research, and some tips for getting started quickly. He also explains how Klaviyo’s data scientists—not just the UX team—are directly involved in talking to users to develop an understanding of their problem space.



    Klaviyo is a communications platform that allows customers to personalize email and text messages powered by data. In this episode, Mike talks about how to ask research questions to get at what customers actually need. Mikes also offers some excellent “getting started” techniques for conducting interviews (qualitative research), the kinds of things to be aware of and avoid when interviewing users, and some examples of the types of findings you might learn. He also gives us some examples of how these research insights become features or solutions in the product, and how they interpret whether their design choices are actually useful and usable once a customer interacts with them. I really enjoyed Mike’s take on designing data-driven solutions, his ideas on data literacy (for both designers, and users), and hearing about the types of dinner conversations he has with his wife who is an economist ;-) . Check out our conversation for Mike’s take on the relevance of research for data products and user experience. 

     

    In this episode, we cover:

    Using “small data” such as qualitative user feedback  to improve UX and data products—and the #1 way qualitative data beats quantitative data  (01:45)
    Mike explains what Klaviyo is, and gives an example of how they use qualitative information to inform the design of this communications product  (03:38)
    Mike discusses Klaviyo data scientists doing research and their methods for conducting research with their customers (09:45)
    Mike’s tips on what to avoid when you’re conducting research so you get objective, useful feedback on your data product  (12:45)
    Why dashboards are Mike’s pet peeve (17:45)
    Mike’s thoughts about data illiteracy, how much design needs to accommodate it, and how design can help with it (22:36)
    How Mike conveys the research to other teams that help mitigate risk  (32:00)
    Life with an economist! (36:00)
    What the UX and design community needs to know about data (38:30)

     

    Quotes from Today’s Episode
    “I actually tell my team never to do any qualitative research around preferences…Preferences are usually something that you’re not going to get a reliable enough sample from if you’re just getting it qualitatively, just because preferences do tend to vary a lot from individual to individual; there’s lots of other factors. ”- Mike (@mikeoren) (03:05)


    “[Discussing a product design choice influenced by research findings]: Three options gave [the customers a] feeling of more control. In terms of what actual options they wanted, two options was really the most practical, but the thing was that we weren’t really answering the main question that they had, which was what was going to happen with their data if they restarted the test with a new algorithm that was being used. That was something that we wouldn’t have been able to identify if we were only looking at the quantitative data if we were only serving them; we had to get them to voice through their concerns about it.” - Mike (@mikeoren) (07:00)


    “When people create dashboards, they stick everything on there. If a stakeholder within the organization asked for a piece of data, that goes on the dashboar

    • 42 min
    087 - How Data Product Management and UX Integrate with Data Scientists at Albertsons Companies to Improve the Grocery Shopping Experience

    087 - How Data Product Management and UX Integrate with Data Scientists at Albertsons Companies to Improve the Grocery Shopping Experience

    For Danielle Crop, the Chief Data Officer of Albertsons, to draw distinctions between “digital” and “data” only limits the ability of an organization to create useful products. One of the reasons I asked Danielle on the show is due to her background as a CDO and former SVP of digital at AMEX, where she also managed  product and design groups. My theory is that data leaders who have been exposed to the worlds of software product and UX design are prone to approach their data product work differently, and so that’s what we dug into this episode.   It didn’t take long for Danielle to share how she pushes her data science team to collaborate with business product managers for a “cross-functional, collaborative” end result. This also means getting the team to understand what their models are personalizing, and how customers experience the data products they use. In short, for her, it is about getting the data team to focus on “outcomes” vs “outputs.”

    Scaling some of the data science and ML modeling work at Albertsons is a big challenge, and we talked about one of the big use cases she is trying to enable for customers, as well as one “real-life” non-digital experience that her team’s data science efforts are behind.

    The big takeaway for me here was hearing how a CDO like Danielle is really putting customer experience and the company’s brand at the center of their data product work, as opposed solely focusing on ML model development, dashboard/BI creation, and seeing data as a raw ingredient that lives in a vacuum isolated from people.  


     


    In this episode, we cover:


    Danielle’s take on the “D” in CDO: is the distinction between “digital” and “data” even relevant, especially for a food and drug retailer? (01:25)

    The role of data product management and design in her org and how UX (i.e. shopper experience) is influenced by and considered in her team’s data science work (06:05)

    How Danielle’s team thinks about “customers” particularly in the context of internal stakeholders vs. grocery shoppers  (10:20)

    Danielle’s current and future plans for bringing her data team into stores to better understand shoppers and customers (11:11)

    How Danielle’s data team works with the digital shopper experience team (12:02) 

    “Outputs” versus “Outcomes”  for product managers, data science teams, and data products (16:30)

    Building customer loyalty, in-store personalization, and long term brand interaction with data science at Albertsons (20:40)

    How Danielle and her team at Albertsons measure the success of their data products (24:04)

    Finding the problems, building the solutions, and connecting the data to the non-technical side of the company (29:11)


     


    Quotes from Today’s Episode
    “Data always comes from somewhere, right? It always has a source. And in our modern world, most of that source is some sort of digital software. So, to distinguish your data from its source is not very smart as a data scientist. You need to understand your data very well, where it came from, how it was developed, and software is a massive source of data. [As a CDO], I think it’s not important to distinguish between [data and digital]. It is important to distinguish between roles and responsibilities, you need different skills for these different areas, but to create an artificial silo between them doesn’t make a whole lot of sense to me.”- Danielle  (03:00)



    “Product managers need to understand what the customer wants, what the business needs, how to pass that along to data scientists and data scientists, and to understand how that’s affecting business outcomes. That’s how I see this all working. And it depends on what type of models they’re customizing and building, right? Are they building personalization models that are going to be a digital asset? Are they building automation models that will go directly to some sort of operational activity in the store? What

    • 37 min
    086 - CED: My UX Framework for Designing Analytics Tools That Drive Decision Making

    086 - CED: My UX Framework for Designing Analytics Tools That Drive Decision Making

    Today, I’m flying solo in order to introduce you to CED: my three-part UX framework for designing your ML / predictive / prescriptive analytics UI around trust, engagement, and indispensability. Why this, why now? I have had several people tell me that this has been incredibly helpful to them in designing useful, usable analytics tools and decision support applications. 


     


    I have written about the CED framework before at the following link:


     


    https://designingforanalytics.com/ced


     


    There you will find an example of the framework put into a real-world context. In this episode, I wanted to add some extra color to what is discussed in the article. If you’re an individual contributor, the best part is that you don’t have to be a professional designer to begin applying this to your own data products. And for leaders of teams, you can use the ideas in CED as a “checklist” when trying to audit your team’s solutions in the design phase—before it’s too late or expensive to make meaningful changes to the solutions. 



    CED is definitely easier to implement if you understand the basics of human-centered design, including research, problem finding and definition, journey mapping, consulting, and facilitation etc. If you need a step-by-step method to develop these foundational skills, my training program, Designing Human-Centered Data Products, might help. It comes in two formats: a Self-Guided Video Course and a bi-annual Instructor-Led Seminar.


    Quotes from Today’s Episode
    “‘How do we visualize the data?’ is the wrong starting question for designing a useful decision support application. That makes all kinds of assumptions that we have the right information, that we know what the users' goals and downstream decisions are, and we know how our solution will make a positive change in the customer or users’ life.”- Brian (@rhythmspice) (02:07)



    “The CED is a UX framework for designing analytics tools that drive decision-making. Three letters, three parts: Conclusions; C, Evidence: E, and Data: D. The tough pill for some technical leaders to swallow is that the application, tool or product they are making may need to present what I call a ‘conclusion’—or if you prefer, an ‘opinion.’ Why? Because many users do not want an ‘exploratory’ tool—even when they say they do. They often need an insight to start with, before exploration time  becomes valuable.” - Brian (@rhythmspice) (04:00)



    “CED requires you to do customer and user research to understand what the meaningful changes, insights, and things that people want or need actually are. Well designed ‘Conclusions’—when experienced in an analytics tool using the CED framework—often manifest themselves as insights such as unexpected changes, confirmation of expected changes, meaningful change versus meaningful benchmarks, scoring how KPIs track to predefined and meaningful ranges, actionable recommendations, and next best actions. Sometimes these Conclusions are best experienced as charts and visualizations, but not always—and this is why visualizing the data rarely is the right place to begin designing the UX.” - Brian (@rhythmspice) (08:54)



    “If I see another analytics tool that promises ‘actionable insights’ but is primarily experienced as a collection of gigantic data tables with 10, 20, or 30+ columns of data to parse, your design is almost certainly going to frustrate, if not alienate, your users. Not because all table UIs are bad, but because you’ve put a gigantic tool-time tax on the user, forcing them to derive what the meaningful conclusions should be.”   - Brian (@rhythmspice) (20:20)

    • 27 min

Customer Reviews

5.0 out of 5
26 Ratings

26 Ratings

Kevin EBG ,

Excellent for data and product professionals

I just discovered Experiencing Data and it is tremendously helpful in helping to understand and shape contemporary data efforts and the emerging field of data product management. Find someone champion both the value of data, ML, design thinking and UX is needed more and more. The podcast is great for product managers as well as data engineers and scientists.

Josh Gratsch ,

Fantastic and thought provoking!

This podcast has helped our team at Ascend Innovations guide conversations around how we continue to strengthen the bond between our data science and UX functions, combining the art of design thinking with the power of data science.

@play ,

How to amplify data value

“How to amplify data value” is alternative title for the show. Brian and his guest continually provide insights on how to make data mean more to users and provide decision support.

Top Podcasts In Technology

Jason Calacanis
Lex Fridman
PJ Vogt
NPR
Jack Rhysider
Gimlet

You Might Also Like

Jon Krohn and Guests on Machine Learning, A.I., and Data-Career Success
Kyle Polich
Changelog Media
DataCamp
Sam Charrington
Melissa Perri