In this episode, Beah (Chief Product Officer) and Zac (SVP, Insights) discuss our approach to user research, how we run robust research while respecting user privacy, and why moving quickly matters. Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy. Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. In today’s episode, we’re going to talk about user research. And so my guest here is Zac. Zac, do you want to introduce yourself? Zac: Yeah, hi, Zac Pappas. I am on the Insights team at DuckDuckGo. I’ve been here for coming up on 14 years and pretend to participate in a lot of market research, user research, and other conversations that we have where we try to infer what users want and whether or not we’re giving it to them. Beah: Sweet, thanks. And I’m Beah. If you haven’t met me, I’m on the product team at DuckDuckGo. All right, let’s launch into it. I have questions for you, Zac. Are you ready? Zac: Ready as you’ll ever be. Beah: All right, so let’s start. Why don’t you give me an example? Like when we’re talking about validating things with user research, just like let’s set up an example of what that means and what that would look like. Zac: I think so. Yeah. So I guess starting with what is user research and why do we do it? I think the obvious answer that people tend to think about are, you know, sitting in an interview or doing some sort of focus group and learning from people. But really, for us, user research is kind of all of that plus, you know, internal dialogue and real testing like A/B testing. So to put it, you know, simply, I guess, user research for us is just trying to be wrong very productively and being efficient at being wrong productively over time. So for us, everything really starts out as a guess or an assumption about users or their behavior about, you know, what is upcoming in the market and research checks that, you know, our assumptions are correct or at least as close to correct as possible. And we try to do that as cheaply as possible, meaning quickly without spending more time on it than what we’re gaining in terms of the product that we’re creating. So an example could be our homepage at DuckDuckGo.com because our homepage has been around for as long as I think DuckDuckGo has been around, but has gone through many, many, many forms. We’ve done tons of A/B testing on it, showing one version of it to some users and a different version to other users. We’ve done a lot of user testing on it, like sitting down with actual consumers of the homepage or users of the homepage. Everything from discussing with parents how they perceive the homepage. So a lot of subtlety that goes into what we think maybe appears as a pretty simple layout or design. Beah: I had a little connection issue there for a second, but I think it’s alright. Let’s forge ahead. Sorry, audience. So, the homepage, I think, as you pointed out, sort of like deceptively, seems deceptively simple, it’s not highly functional software, it’s a page, there’s a few boxes on it, a field, but like what makes it so tricky and worthy of user research. Zac: Yeah, good question. Because it is kind of subtle, or I guess it doesn’t seem very obvious. So if you think about it, the homepage is really where we have a big mix of different user types coming in. Making any assumption about a part of our product usually means that you have to infer who is using it, meaning are they younger or older? Do they have a lot of context about what they’re doing or very little? So our homepage is actually a mix of different user types. There’s a lot of returning users who are very familiar with DuckDuckGo, they’ve been coming to it for a very long time. And so when they come to the homepage, it’s a pretty like rote act for them. The other group that really comes to our homepage a lot are new users or new visitors to the brand and product space in general. So they have maybe very little context about what DuckDuckGo is or what products we have. And so you can imagine, you have these two very diametrically opposite groups that come in, people who have a kind of a strong habit formation and muscle memory on how to use the page and then others who don’t have anything and they kind of, you know, maybe say meander or explore more than they are, you know, directly going to the thing that they know that they want. So what we learned over years of A/B testing, user testing, and kind of regularly reviewing what the experience is like for real people, we found something kind of interesting, if I can just generalize it, which is new users — like kind of people first visiting DuckDuckGo for the first time — really look at the center of the page. And if you see DuckDuckGo.com today, it has these two prominent boxes in the middle. It depends on the browser that you’re in or the form factor that you’re on. But primarily what we know is new users will tend, their eyes kind of follow the middle of the page. And so when they land, we really want to show them an overview or a good sense of what DuckDuckGo can do for them. Whereas returning users, people who’ve been coming back for a long time, tend to want to find the search box. And so we can, or at least we have some confidence that we can move the search box around, you know, the top of the page where it’s at now, to the bottom, to the middle. And if you’re a returning user, because you kind of expect or know that it’s there, you will more easily be able to hunt and find it. So the configuration that you see today, and maybe changes as things develop, really does to us strike a very good balance between new and returning user needs, both something that a brand new user can figure out and use efficiently and kind of get a lay of the land of what DuckDuckGo is and what we do. And then returning users can quickly get to the search box or other more mechanical parts of the product without too much friction. Beah: So how did we actually learn that? Like what testing methodology did we use? Can you walk me through it? Zac: Yeah, a lot of, I’m sure, pretty standard things for people who are used to product development, but A/B testing primarily. A couple years ago we were running, I think, like at least one A/B test of the homepage, like one alternative version of our homepage every week for pretty much the entirety of 2024, 2025. That year we had a ton and tons of homepage tests. And then independent of the actual A/B experiments that we’re doing, we will regularly run diary studies. So recruiting 10 to 20 DuckDuckGo users or other different user types, even non-users, to give them some background on the product. And then really give them a chance to use it for multiples of weeks and have them do journal entries and take notes of the things that they’re running into that they’ve experienced that are emotionally supercharged or maybe had caused some great duress in how they tried to use the product. So anything that’s really notable to that user can come up in a diary study. What we’ve also done in the past is to try to create alternative versions of our homepage from designs or a design system that we can give to users to play around with in real time. So we can actually simulate what an alternative version of the homepage might look like and how it should act and give them a clickable prototype that they can use, get some feedback before we ever invest too heavily in the platform. So it’s been a mix of really just doing a lot of testing pretty much every week since the conception of the company on some run. The homepage was a big focus for us for the last few years to try to nail something that could eloquently show what all of our kind of product offering is between search and now the expansion into Duck AI and of course the apps and email and other things. Beah: Yeah, so like there’s like, sounds like we’ve been marrying, and I think we do this elsewhere in the company, a mix of like quantitative research where we get to hear how people think, you know, people who are opted into some kind of study with us, like we actually get to understand what they’re thinking, what they’re doing in some detail and then just like quantitatively looking at results in an A/B test. Zac: Yeah, absolutely. And probably the maybe thing I didn’t say was we were very active on our social media. We point most of our users, at least if you click around our website, a lot of things point to our subreddit, which is reddit.com slash r slash DuckDuckGo. If you have product feedback, if you run into a product issue. So we do try to, and we have lots of people here that every day are checking comments or posts to that subreddit just to see if we released something and it’s causing some undue user harm. Tons of feedback boxes around our SERP, like the search results page within the app. And so we’re constantly trying to encourage that if a user has a negative experience, they’re able to get it pretty frictionlessly by a message to us. So we’re looking at social media, we’re really running really structured user tests, like intentional user tests, A/B testing and just kind of a what they call dogfooding or like internally using the product a lot and trying to be very critical about our understanding of what users really need. Beah: Yeah, I’ve said it before and I’ll say it again. If you’re writing feedback to us, a human is probably not just reading it, but like taking it to heart and thinking about it. So, um, how about like, tell me what are the priva
