Recently we held a meetup where Dashly and other IT companies talked internal researches. We invited Alisa Velminskaya, user researcher at Skyeng, who told us about how Skyeng’s research team manages to combine user research and data analysis. She also shared some research organizing experience. Other representatives who joined her to talk about their companies’ research methods were Ekaterina Syuma, product designer at Miro, and Maxim Golovkin, head of web & mobile development at ER-Telecom. The one shooting questions at them was Dmitrii Sergeev, the CEO & founder of Dashly. Why product companies like researches even more than colleges do While conducting qualitative researches on the development stage lets you save time and resources you would otherwise waste implementing useless features, later stage researches may let you see how to increase your key metrics and simplify main processes of your company. "9 out of 10 hypotheses do not work out. They need to be verified right off the bat. And that’s what qualitative researches are good for. Researches are about making your job with highest quality, within smallest amount of time. This makes everyone satisfied," – Dmitrii Sergeev, CEO & founder of Dashly The key of a product research is in it taking as shortest cycle as possible – the sooner your idea meets harsh reality (with your team receiving valuable feedback on the way) the less losses in time, money and other resources your company will eventually suffer. How do you know if you need a research Don’t research general issues. For example, questions like “why aren’t people buying our product?” is not a good research goal. Ask more in-depth questions like “How do users go from reading our blog to registering in our service?”. Questions like that may serve as a good research starter. The goal of your research must be clear. "At some point of analyzing your metrics you inevitably enter the unknown. You’ve used up every metric you have, what comes next in uncertain. Here’s where qualitative researches come in handy," – Maxim Golovkin, Head of web & mobile development at ER-Telecom With that in mind, by discarding useless and unsound researches. it’s important not to go researching every bit you can possibly reach "We decide to start a research only when we lack knowledge on a subject," – Ekaterina Syuma, Product designer at Miro We know a task is set correctly when we have a clear understanding of what we need the information we may uncover for and what we’ll do with it. A research task is faulty when don’t know what we’ll get from doing the research. How do you find out if a task is faulty? Who must do researches It depends on what tasks you set for your researches and how big your company is. In larger companies, researches are usually held by a separate team. Skyeng, for example, has a research team that fully focuses on gathering insights for their product teams, and each product team has researchers assigned to them. Researchers are lead by product managers and are a part of the research team which is run by a horizontal manager. Product managers are responsible for business in this chain of command, and a horizontal manager is responsible for staff development. ER-Telecom, too, has a research team that leads large researches for the product and marketing team. Another way is to assemble a cross-functional team for each research. In Miro, researches are lead by teams that consist of product managers, designers and developers who conduct researches as an additional task. Miro also has a data team that actively takes part in qualitative researches and conducts its own quantitative researches. In Dashly, researches are made by different teams depending on the task of a research. Dashly’s team constantly verifies hypotheses and does its own researches too. growth hacking If a company is data-informed, its research culture is bound to flourish and prosper Ekaterina Syuma Product designer at Miro What skills does a researcher need curiosity; critical thinking; analytical skills; well developed empathy; communication skills; product knowledge; UX analytic skills and basics; counting ability;non-perfectionism; sociological imagination (an ability to interpret reality); an ability to abstract from your product, an ability to treat it critically (seeking out its flaws will help you improve your product); active listening techniques; Don’t panic just looking at this list: nobody expects you to have all of these skills at the very start, but doing product researches will give you a chance to develop them. "Researches help me grow as a designer. They help me criticize my own decisions, see them in abstract, they help me iterate more. Researches develop my skills and empathy," – Ekaterina Syuma, Product designer at Miro Choosing methods and tools Product companies use and combine different methods: Note: we took the classification suggested in Alisa’s presentation as a basis and added a bunch of method examples to it. When it comes to product researches, qualitative methods are used more often as this method is more convenient to use for short iterations. However, Skyeng and other companies have proven that sometimes it can be useful to complement the data you’ve gotten during a qualitative research with the knowledge you can get with a quantitative research. For example, you can take some in-depth interviews and analyze the qualitative data on the segment you included in your interview at the same time. It also works the other way around, quantitative data can help you find your users’ pain point and qualitative data can help you cover it. "Combined methods are an awesome thing, " – Alisa Velmiskina, User researcher at Skyeng Sometimes you can do with using just one method, like in-depth interviews or feedback gathering. But the thing is, you can combine them to reach an even better result. You can complement usability tests with short interviews and feedback gathering – with desk researches. Choosing the right methods depends on the goals of your research and the capabilities of your team. Sometimes your team is in one hemisphere and your users are in the other one. In this case, you have to use remote tools that will serve you the same function. Fundamentally, the tests you’ll be having will be the same as corridor testing, except done using a video chat app, or phone calls, in case with interviews. That’s how these processes are usually done in product companies. How to get ready for a research You’ll need to set the following: problem, goal, tasks, hypotheses. You’ll also need to choose the segment you’ll be testing these hypotheses on and make a guideline (a list of questions) that will encompass all of the research tasks. Tip: don’t ask your respondent about what they would do, ask them about their past experience in a similar situation. This way, they won’t misinform you and you won’t end up developing your product in the wrong direction. How to find respondents You can invite your users to take a part in your interview or a survey by messaging them or sending them an email campaign. Combined methods will come in handy here as well: you can pick your internal respondents at the start of your research based on the quantitative data you have. "Want to hear the good news? Unhappy users are always happy to share their negative feedback. Make use of it! Users who dropped off like sharing with us why they did that. Recently I got a message from one of those guys: “I dropped off, I CAN COME AND TELL YOU WHY," – Dmitrii Sergeev, CEO & founder of Dashly It’s hard to find people who have never used your product before and fit certain criteria at the same time. Still, how do you find them? In most cases, you’ll need to look for respondents of your desired segment individually. You can do it via official communities in Slack, Telegram or Facebook, for example. You can also ask your users for a way to reach out to their co-workers. In any case, just don’t forget to take time looking for respondents into account when planning a research. What questions to ask and how to ask them Write an introduction and think of some introductory questions. Tell people about what you do. Tell them about how important it is for you to make your product the best you can for your users. Make sure being a part of this research is a positive experience for your users. Basic rules: Ask open-ended questions. Avoid flattery. Don’t let your respondents lie to you. Don’t ask for opinions, ask about experiences. Talk less, listen more. Be happy to receive critique. Examples given by Alisa or taken from the book “The Mom Test” by Rob Fitzpatrick. Don’t forget to thank the respondents who took part in your research! After your conversation (a must-do) and in the process, if that fits. We highly recommend reading “The Mom Test” by Rob Fitzpatrick at least once and getting more practice interviewing if you want to learn asking new questions, answers to which will not misdirect you and your product. How to make sure your statistical sampling is enough It depends on the nature of your research. For quantitative researches, it is based on the saturation concept: the sampling must be large enough for you to seek each possible meaning and every experience option important for your research. But remember not to make it too large as that will make your research counterproductive. The principle is simple: stop once the information you’re receiving starts to repeat. "Remember that in qualitative researches, contrary to quantitative researches, you can’t evaluate your sampling based on statistics," – Alisa Velmiskina, User researcher at Skyeng Qualitative researches are aimed at reconstructing the sense of your interviews, not statistical generalization. How to filter insights out of the pile of gathered data The sequence of methods and the framework set for analyzing the data you’ve gathered have to be picked based on your preferences, there is not universal formula to it. "When I was trying out different frameworks like CJM and JTBD, I’d always end up getting some monster framework as a result because mixing them would result in something totally new and custom," – Ekaterina Syuma, Product designer at Miro The earlier you start making your own researches – the sooner you’ll learn picking and combining frameworks that suit your needs. How to measure research effectiveness It’s not easy to measure research effectiveness. Calculating if research results are worth the resources you spend on them is even harder. When do you consider your research successful? You can look at the insights you’ve gathered. "If we held a research that gave us answers to all of the key research questions, confirmed or disproved every hypothesis we had – our research is successful," – Alisa Velmiskina, User researcher at Skyeng If all of your questions had been answered but getting them answered brought you no insights, it means your questions were defined incorrectly. That’s why it’s so important to filter out faulty material out of your researches at the very beginning. "I’m always thinking of new hypotheses, even when I’m testing something in the office with our guys. Then I check these hypotheses and score them," – Ekaterina Syuma, Product designer at Miro In the best case scenario, you’ll get a list of recommendations i.e. certain actions you’ll have to complete in order to improve your product. Your researchers will then pass the list to your product team, product managers will work that list by making additions and prioritizing it, then the list gets passed to the devs. That’s how it works in Skyeng. "It’s important to make sure that these insights are not gathered just for show, but are used to come up with a potential solution," – Ekaterina Syuma, Product designer at Miro The result of this research was Skyeng launching a new feature and getting a boost in metrics. How to implement research results Your researchers’ aim is to give your product team not some purposeless manuscript, but working conclusions and recommendation. Skyeng have developed a report template – a set of questions that their research team has to find answers to by the end of the research. After that, the product team is given a structured list of insights (with quotations and references to source materials – to keep product team safe from the temptation of misinterpreting the data) that confirm or disprove the hypotheses. If you’re doing many researches and learn a lot of stuff from them, think of a way to share your discoveries with your team. Guys at Miro do it by holding regular Product Insights meetups, and Dashly has Fish Thursdays. Inner meetups like that let each team keep other teams informed about what interesting stuff they have discovered. In Skyeng, research results are discussed at stand-up meetings and presented to other teams’ clients. How to score research results It’s important to figure out the most convenient way to score your results so that it’s always easy for you to come back to them and analyze the data from a new perspective, or even correlate these results with your new insights. Researches may be repeated. Your product is evolving, consequences are changing, new problems arise and sometimes you have to repeat your researches on the same aspects of your product, this time – with new goals and new tasks. You may decide to make some of your researches regular. If you research often, you’ll need to make a research catalogue. Our colleagues at Skyeng have one as well. Theirs is a table organized in a way that allows to conveniently look for researches on different topics. They keep notes and conclusions on each research in . Notion Dashly uses to prioritize hypotheses and score data related to them. You may choose any method you like. Favro Dashly’s growth hacking board in Favro What are the risks? Getting low-quality data. You may forget to ask something due to the lack of experience, feeling stressed during the interview if it’s the first one you’re taking, or your respondent being too silent or too talkative. Making a list of questions may prevent it. Finding out what you’ve already known. To avoid this, you have to know your product and inner processes well. Finding something that will bring you no benefit or something you can’t implement in your product. Remember that your resources are limited and that it’s your user you’re making your product for, not your team. Wasting too much time and energy. Before starting a research, compare what would be easier to do: a research or an A/B test. If you don’t have enough data or insights, do a research to add some new hypotheses to your hypothesis jar. Checklist Look at your analytics. Does looking at all the data make you question “why”? Make a qualitative research on it! Or the other way around, say, you held a custdev session and found something new. Test this insight on the quantitative data you have. Find its weak spots and decide on the research methods. Choose a user segment. If you can do a desk research – do it. If you can make do with it – make do with it. Define a research goal. Divide the goal into tasks. If it’s still unclear to you what you want to get from your research – don’t even start it. Make sure that the task of your research is not faulty (i.e. you know for sure how you can use the information you’ll get as a result of this research). Think of some hypotheses for every task. Discuss your hypotheses with your team. Think of questions, answers to which will let you confirm or disprove each hypothesis. Don’t let your research take too long. It must have the shortest cycle possible. Don’t ask users about their theoretical experience, ask them about their real-life experience – not what they would have done, but how they are solving a problem now or how they were solving it before. Talk less, listen more. Don’t listen to flattery, aim to get more critique. Make a list of questions: from general to specific, from easiest to hardest. Don’t be the only source of truth about your customers. Discuss research results with your team. Compose cross-functional teams and synchronize your experiments so they don’t affect one another. Think of a convenient way to score results of your researches. "The core of a company whose corporate culture is built on bringing usefulness to its users must have a spark that ignites a research. Sometimes you feel like you’re not sure about something. At this point, you have to talk to your user to find out exactly why you’re not sure about it. That’s how we’re striving to approach it," – Dmitrii Sergeev, CEO & founder of Dashly And remember what guys at Skyeng told us, and researches are their speciality: combined methods are an awesome thing, operational processes are your growth point, and internal researches are your source of insights. Amen to that. We thank Alisa, Maxim, Ekaterina and Dmitrii for sharing their knowledge and experiences.
