WTF is Mixed Methods Research?

The rules of user research have changed. The distinction between qualitative and quantitative research is becoming blurry as research at scale becomes more common. Fast-growing companies are prioritising researchers fluent in both qualitative and quantitative methods that can adapt to any research challenge. We have entered the era of Mixed Methods Research.

Traditionally these qualitative and quantitative research methods were viewed in opposition. Each side would proudly stand and proclaim that their methods were more important for the company’s success. The researcher conducted the study using either a qual or quant research method and interpreted the findings on their own.

Then it all started to change. The importance of user experience became increasingly important as companies adopted product-led and lean startup growth strategies. Founders initially took on this responsibility for user research. As companies grew, this work was passed down to technically-minded product managers. Product managers quickly learned that big data could tell them what users were doing but it couldn’t tell them why.

Mixed Methods Research was the ideal toolkit. Technical product teams were already using quantitative research to justify decisions and qualitative research became the natural fit to inform their quant studies. Before long the combination of the two made Mixed Methods Research part of the natural vocabulary at scaling companies.

And then we arrive at the present day. Mixed Methods Research has started to find its footing outside of young startups and is quickly emerging on its own as a field of expertise. Let’s dive in and learn more about what Mixed Methods Research is all about and the many advantages it offers.

Quantitative research methods are remarkably useful for gathering hard data to measure, validate and inform crucial decisions. This quant data is the currency of internal decision-making in product-led companies. Stats from this research can answer questions like ‘Which feature do users like the most?’ or ‘Which design do users prefer?’.

However, quantitative data will likely surface new questions that can only be answered with qualitative research. Qual insights can help you to understand why questions such as ‘Why do my users like to use my product?’, ‘Why do they churn?’ and ‘Why do they behave this way?’.

Additionally, qualitative research provides an opportunity for users to share insights you never would have thought to ask about. At OpinionX, we call these insights unknown unknowns and they’re a core output of successful discovery research.

Mixed Methods Research is unique because the researcher can address multiple research objectives in one project — understanding the pains and motivations driving user behaviour as well as the scale of their impact on the total userbase. A Full-Stack Researcher has the breadth of expertise to select the perfect research methods to answer any question, regardless of whether it sits on the qual or quant side of the fence.

It is only in more recent years that user researchers and product teams have joined the mixed methods movement. The view on hiring user researchers has changed since product teams have adopted Mixed Methods Research. A researcher specialised in a single area will require a complimentary researcher, which many of the most ambitious early stage companies just can’t afford.

This is why product leaders love to hire researchers that can turn their hand to any research project at the drop of a hat. These are the people that be called upon for any challenge. They uniquely fit the research needs of fast-moving and scaling companies. The demand for such people has given rise to the Full-Stack Researcher.

In 2016, Blake Bartlett from venture capital firm OpenView coined the idea of “Product-Led Growth”. This is the strategy that fuels rocketship trajectory companies like Slack, Calendly and Dropbox. The defining feature of Product-Led Growth is the “End User Era”. In the 1980s and 1990s we had the CIO Era, where deals were done over dinner and 18 holes of golf. Then came the Exec Era in the 2000s, where outbound sales and marketing targetted the non-technical executive of the company. The End User Era suggests that purchasing has shifted down to the end user. Their primary decision-making criteria is not “how will this product help the business’ bottom line?” but “how will this product help me in my day-to-day?”.

The default strategy for most companies over the past decade has been a sales-driven approach that targets execs with cost saving and efficiency messaging. The salesperson was the driver of this exec-focused strategy. As companies adopt Product-Led Growth, their messaging shifts to focus on unmet needs and pains of the end user. In this Product-Led Growth model, the product and UX functions are the ones that interact with the end user and it is therefore the responsibility of these teams to understand the user’s motivation to purchase.

Understanding unmet needs, pains and undiscovered motivations requires a different mindset than the mindset of a salesperson or traditional market researcher. The oldschool qualitative researcher’s objective is to craft detailed and relatable customer personas. They act reactively to the needs of the product team and work in the world of perspective rather than numbers. In the world of Product-Led Growth, a researcher must do so much more than that. Research must shift to become a continuous function that isn’t confined to pre-determined project timelines. Building and shipping quickly is the lifeblood of the company so researchers must be responsive and adaptive. Identifying the core pains driving user behaviour must be proactive instead of a reactive response to the output of big data.

The need for user researchers to be skilled internal networkers grows as companies continue to become more integrated cross-functionally. They become the go-to resource for everyone, from the marketing and sales teams to the product and design functions. Each of these teams speak a different language and value different types of data for decision-making.

The increased focus that Mixed Methods Research puts on understanding user experience gives researchers the flexibility to address the needs of every internal stakeholder. However, in order to deliver the output of those projects, user researchers must become multilingual influencers capable of wielding any type of data in order to convert internal skeptics to the view of the end user.

The highly sought Full-Stack Researchers we see today have mastered these three Mixed Methods Research designs:

Exploratory Mixed Methods Research research puts the qualitative step first. This typically involves a user interview sprint but can also take the form of an online survey with open-ended questions. The findings of this first step inform the quantitative research that follows. This usually means taking the insights from your qual step and using them to inform a quantitative project, such as a quant survey or unmoderated usability testing. The results of your quant step help you to quantify the significance of your qualitative insights.

This approach is great when you are working on a research problem with a lot of unknowns and you don’t know what you should ask in your quant study (or as we call them at OpinionX, ‘unknown unknowns’). Qualitative insights will help you to form a hypothesis which can then be validated or invalidated during the quantitative step. To explain, let’s introduce you to Alison Berent-Spillson, a UX researcher, designer and Medium content creator.

In 2018, Alison was tasked with conducting research for Slingshot, an application that helps competitive foosball players to improve how they practice. To kick off her research, Alison needed to get a basic understanding of the opinions and perspectives that foosball players held. This led her to conduct exploratory interviews with foosball players. It was through these interviews that Alison was first able to understand the frustrations that players felt.

After discovering some of the common foosball frustrations, Alison felt like she had gained enough insights to develop the product features that would be most useful for players. However, rather than jumping in and building features based on anecdotal evidence from a handful of interviews, she instead brought these insights to a quantitative study to measure their validity and to see how much impact each feature would really have on the player’s practice.

By using mixed methods, Alison was able to distinguish between consensus insights and those that were held only by a minority of users. As a result, Slingshot was able to prioritise features that were meaningful to the most amount of players. Following this exploratory mixed methods process meant the team could avoid technical debt by only building solutions that would add the most value for customers.

🔬 2. Explanatory: Quant → Qual

Explanatory user research starts with collecting and analysing quantitative data. This step often starts by looking at user product behaviour at scale using big data tools like Segment or MixPanel. You can then bring your insights to a qualitative deep dive to learn more about the context behind the figures through focus groups, in-depth interviews or online community discussions. Explanatory research integrates all of your findings for a broader and deeper understanding of your users.

This research design is especially useful when there is a need to explain and interpret the quantitative findings. To explain, let’s introduce you to Clement Kao, a Product Manager at a consumer fintech startup called Blend.

Prior to Blend, Clement was working at a CRM startup that struggled with high user churn. Clement noticed that the most engaged users interacted with the app very differently than the disengaged cohort. He decided to segmented the user base by engagement levels to pinpoint the specific events and actions that were indicators of which users were the most and least engaged.

Clement started with quantitative research. He analysed the user base to determine their top quartile of “highly engaged” users (with a median login rate of 31.7 times per month) and their bottom quartile of “highly disengaged” users (with a median login rate of 4.9 times per month). Comparing these two groups, they found that highly engaged users regularly reassigned leads to teammates, rejected leads and set vacations in the app. Clement was very surprised that highly engaged users would reassign leads to their teammates — this insight directly contradicted their hypothesis of what a highly engaged user was.

Confused by these results, Clement’s team turned to qualitative user research to find the root cause behind the behaviors they saw in their analysis. Through 1:1 interviews with highly engaged and highly disengaged users, they unearthed a range of insights that they used to improve their UX. They increased the visibility of key features, introduced new user views to make it easier for highly engaged users to perform their most important tasks and launched an experiment to algorithmically determine CRM lead value for their users. These changes increased median logins from 10.6 times per month to 18.3 times per month, drove a reduced churn rate and created an increased referral rate — boosting both top-line revenue and bottom-line profits.

Dynamic research methods blend qualitative and quantitative data together at the same time within one research study. Unlike exploratory and explanatory research, an entire Mixed Methods Research project can be carried out in just one step using a dynamic mixed methods research tool.

A number of core principles define dynamic user research methods. They are fundamentally user-driven. By putting the user at the centre, the researcher takes more of an observation role. This means that dynamic research is often unmoderated or reactively controlled so that participants can shape the evolving data themselves. Dynamic research is particularly suited for teams that are using a Product-Led Growth strategy as the user’s ability to share their experience is prioritised over the researcher’s interference.

The most common method of dynamic research is unmoderated usability testing at scale. At an individual level, this is inherently qualitative and observational. However at scale, quantitative data begins to surface trends in user behaviour.

Another method of dyamic Mixed Methods Research is OpinionX (hey, that’s us! 👋). OpinionX takes more of a deliberative approach to dynamic research. On the surface, OpinionX looks like a combination of lots of features that you are already familiar with. Open-ended questions and free-response textboxes enable you to gather rich verbatim opinions from users. Structured voting formats like agree -vs- disagree and pair-choice selection translates the unstructured user opinions into organised quantitative data. The result is a harmonious blend of qualitative user opinions that are easy to prioritise with sorting methods such as consensus, importance and divergence.

Dynamic research methods like OpinionX remove the need for the researcher to manually translate insights from one step to the next, like in exploratory or explanatory Mixed Methods Research. Mixed Methods Research is often seen as a time-consuming process due to the added steps. However, platforms like OpinionX are great at gathering rich and actionable insights even faster than a traditional solo qual or quant study.

When COVID first hit back in March 2020, a group of Irish entrepreneurs launched a nonprofit called Feed the Heroes to provide a way for ordinary people stuck at home to support frontline workers. The project was a rapid success and reached over €1M in donations, mainly from individuals, within a few weeks. The Feed the Heroes team were keen to understand what supporters’ core movitations were for donating to their cause.

The Feed the Heroes team had drafted a traditional multiple choice survey but they felt that a significant amount of people would end up falling into the “Other” box. They also worried that they would end up missing out on some really rich insights from passionate donors that way. Instead, they launched an OpinionX survey. They added over 30 interesting sample opinions that were directly from donors tweets, emails and testimonials to kickstart the survey. The final thing to do was to share their survey link with their 17,000-strong donor base.

👆 This user-generated statement (qual) was the most important opinion to participants (quant). It completely changed how the Feed the Heroes team understood their value proposition and messaging.

Check out the full case study for more info on the surprising insights that helped the Feed the Heroes team get even closer to their donors here.

-

-

If you are a research specialist, a part-time researcher, a wannabe researcher or even a ‘why do I have to do research in my job?’ researcher, Mixed Methods Research can help you get the insights you need.

The goal of becoming a Full-Stack Researcher is not simply to improve your ability to gather meaningful insights. Instead, Mixed Methods Research enables you to improve your ability to collaborate with your internal stakeholders. Rather than being a Swiss Army Knife that your team delegates research work to, a Full-Stack Researcher is a guide that understands which language to use to guide each internal function to the perspective of the user.

Everybody’s path to becoming a Full-Stack Researcher will be different. I've compiled some of my favourite resources below that are perfectly suited if you’re about to undertake a mixed methods project for the first time. If dynamic research sounds like the right fit for your project, get started for free on OpinionX today or check out the OpinionX knowledge base to learn more about how to get the most value out of your OpinionX surveys.

Mixed Methods Research used to be an academic affair that was convoluted and inaccessible. That is no longer the case. As the leaders of the Product-Led Growth movement attest; we are now in the era of the end user — and Mixed Methods Research is your toolkit to understand those users.

The more research resources you can have, the better! For many companies this simply isn’t possible. While some may only be able to afford to bring onboard their first UX research hire today, others may find it difficult to even find a user researcher to hire. User research is a rapidly growing profession (the UX industry reached 1M people in 2017 and will grow to 100M by 2050). As a result, there are more user research roles available than there are user researchers to fill them! With this talent shortage in mind, the broader the expertise of your researcher the better.

Companies only hear from 4% of their dissatisfied customers. Without user research, understanding the user experience problems that caused these users to become dissatisfied and churn is a guessing game. Customer experience is the battleground for competitive advantage. Companies with superior CX bring in 5.7x more revenue than their competitors with comparatively poor CX.

A great place to start is with User Interviews’ annual State of User Research report.

UX collective has created a great resource on all things related to running user interviews. Hotjar has everything you need to know about usability testing.

UX Planet have a great guide to conducting user surveys. Hubspot can help you get started with A/B tests.

Analysing open-ended questions from a traditional survey is a manual process that requires significant time to do. If you’ve already run your open-ended survey, check out Louis Grenier’s guide and template for analysing open-ended questions in 5 steps. If you haven’t run your project yet, OpinionX automates the analysis of open-ended responses by letting participants vote on each other’s opinions to surface the statements that are most important to them.

Academia has had a 15 year headstart on Mixed Methods Research compared to tech companies. Mixed Methods Research started taking off in academia around 2006. Early studies found that ‘Mixed methods give a voice to study participants, ensure that study findings are grounded in participants’ experiences and uncover more information than can be obtained in only quantitative research’ [Wisdom & Creswell, 2013]. In recent years, however, Mixed Methods Research has rapidly expanded beyond academia as the UX profession grows and specialised tools enable easier access to this new field of user research.

If you’ve enjoyed this deep dive into the future of user experience research, consider sharing the Twitter thread summary.

I write about the future of UX research on the OpinionX blog, the Full-Stack Researcher. Subscribe to get an email every two weeks that includes my most recent posts and my favourite links found around the web over the last fortenight.

(Disclaimer: The author is the Co-Founder & CEO at OpinionX)