Thomas Dunn, Product Researcher at Buffer, shared in the first part of the interview more about Buffer’s visions, working with data & tools to understand the customers and useful tools for a product manager. Read on for the second part of this interview, made by Monica Obogeanu, product manager MozaicLabs.

MO: Is there an error that you made, based on a wrong assumption? Or a sort of red flag that any product manager should keep in mind when doing research?

TD: Yeah, oh, many many times :). To go back to the theme of data: in the past, I’ve made a product proposal to the team based just on the qualitative research side of things. Due to the fact that I had targeted the wrong group of users, I was hearing the same thing over and over again. This led me to believe that the whole company was facing a big problem.

Joel made a great recommendation that I zoom out and look at the wider scope of things. Ie. how many people use this feature? I realized then that this issue was affecting around 1% of our user base and wasn’t as serious as I had thought. It raised further questions such as “why don’t 99% of our customers use this feature?” and “how does this feature align with Buffer’s vision moving forward?

To me, my initial approach here would be an example of a red flag – raising alarm bells based purely on qualitative research. Whilst qualitative research is invaluable, it’s important to add further validation by analyzing data and measuring the potential impact of an iteration or feature. Since then, I’ve made sure to get a data lens on my research and it has helped a ton for prioritization.

MO: Could you say that you’ve since found a guideline to help you choose customers when doing qualitative research?

TD: It’s something I always try to improve. I don’t know if we have a guideline because targeting changes depending on what we’re working on. I guess we try to avoid bias, like just speaking to our power users. It’s super helpful to learn from successful customers and where relevant I think it’s even more helpful to speak to customers that we let down.

I try to speak to churned or inactivated users more now, even though it’s a bit more difficult to recruit them for calls. On these calls, we learn plenty about what’s not working so well, and it’s a completely different experience from what our happy customers are going through. This helps give me a more rounded picture.

Quick note – we’re very lucky at Buffer because people are very generous to offer their insights and their feedback to us. Due to this, I find that there is often a very positive bias towards our Brand so it’s important to make customers feel comfortable sharing their frustrations and challenges with me. It’s always wonderful hearing praise and success stories but of course, we can improve so so much. Good targeting helps here I find.

MO: You mentioned that right now you are focused on on-boarding. Can you tell me what that means exactly and how it differs from customer development?

TD: In the last few months I’ve transitioned into a bit of a generalist research role. Research is my foundation skill but maybe at the moment 20-30% of what I do, so I’ve done a bit more product, a bit more design, just helping things get shipped, deployed, and moving along with a much smaller team. Right now there are 4 of us focusing on on-boarding as our day to day, but then we have lots of advice from other team members to help guide us along as well.

MO: What do you do in a day’s work?

TD: That’s a good question. It changes so much at the moment… and literally just changed yesterday haha.

So right now we’re trying to make sure that we help customers to experience value. Above everything else we want customers to reach successful moments in the app very early on. In the past, I’ve made the mistake of looking at on-boarding as funneling users through certain actions, because we know that these actions lead to retention. I believe that was the wrong perspective. I wasn’t thinking: these customers have a goal that they want to achieve in their lives and have come to Buffer to hit these goals and improve their lives. One of the biggest reasons small businesses come to Buffer is because they want to win back time in their day to focus on other things: so how can we get help them experience these benefits as quickly as possible so they’re likely to stick around?

One of our biggest responsibilities is just to clear the path to Aha moments. Because we’ve got a lot of noise, sometimes we’ll run customers through flows that don’t provide value to them, so we’re constantly just tweaking things, trying to get customers to those Aha moments. Almost like we’re guiding them through a jungle with a machete. We’re striving to cut the clearest path to success possible.

MO: So to do this and to get an understanding of each customer’s experience, do you look at data, do you also continue to do interviews with people?

TD: It’s a combination of data and research. We want to be very thorough with iterations as these changes affect millions of people now. With our on-boarding flow for our example, we might have 8 steps. We know, at every single step, what’s the drop-off rate, what’s our activation rate, how many customers churned after 3 months, how many of these customers converted to paid plans etc. – so we try to get the whole picture.

Hotjar, a really great usability tool, lets us record user sessions so that we can watch customers signing up for the app for the first time. For example, a couple of days ago I watched 80 customers sign up for the app for the first time. By doing that, I identified some fascinating patterns and some dodgy bugs too.

This way it’s almost as if we were watching the customers signing up over their shoulder like we’re in the same room as them. It’s a cheap and fast way of doing research too.

We might wait until somebody signed up for the app and speak to them the next day. This way we find out what they wanted to achieve when they signed up for Buffer, whether they achieved that, and what was stopping them from getting to those moments in the app. Speaking to customers so quickly is crucial because the experience is still fresh in their minds.

MO: What other tools do you use to get insight into your users? You already mentioned Looker and Hotjar.

TD: On the signals side of things – UserVoice, which is a suggestions forum for customers, which has always been very helpful in casting a wide net and giving our community the chance to have a say in our roadmap.

As a researcher I absolutely love Calendly – it’s made my job so much easier booking calls and making it easier for customers to book a session with me. The other main one would be Qualaroo, which is a really neat in-app survey tool. What else, there’s just so many tools… Typeform as well is a great research tool that we use for surveys, beta recruitment and other things. And to be honest, just Google Docs. The beauty of the work we do is that we don’t need a lot of complicated software, we can work from anywhere as long as there’s a connection, so that tends to be mainly what we use as researchers.

MO: Is there anything that I didn’t ask you about your work that you specifically enjoy?

TD: I just love seeing patterns more than anything, and for me, it’s exciting to go into an exploration not knowing what I’m going to find. It’s so exciting to speak to customers and start seeing things coming up over and over again. If it’s a surprise, that’s even better and it always excites me about what we do.

Eager to find out more about how to build useful products, users love? Pre-register for Product Leaders, 8-9 June, Bucharest.