How To Run The Right Kind Of Research Study With The Double-Diamond Model
How To Run The Right Kind Of Research Study With The Double-Diamond Model
Steve Bromley
Product and design teams make a lot of decisions. Early on in the development of a product, they will be thinking about features — such as what the product should do, and how each feature should work. Later on, those decisions become more nuanced — such as ‘what should this button say? Each decision introduces an element of risk — if a bad decision is made, it will reduce the chance for the product to be successful.
The people making these decisions rely on a variety of information sources to improve the quality of their decision This includes intuition, an understanding of the market, as well as an understanding of user behavior. Of these, the most valuable source of information to put evidence behind decisions is understanding our users.
Being armed with an understanding of the appropriate user research methods can be very valuable when developing new products. This article will cover some appropriate methods and advice on when to deploy them.
A Process For Developing Successful Products
The double diamond is a model created by the UK’s Design Council which describes a process for making successful products. It describes taking time to understand a domain, picking the right problem to solve, and then exploring potential ideas in that space. This should prove that the product is solving real problems for users and that the implementation of the product works for users.
To succeed at each stage of the process requires understanding some information about your users. Some of the information we might want to understand from users when going through the process is as follows:
Each stage has some user research methods that are best suited to uncovering that information. In this article, we’ll refer to the double diamond to highlight the appropriate research method throughout product development.
Diamond 1: Exploring The Problem And Deciding What To Fix
The first diamond describes how to come up with a suitable problem that a new product or feature should fix. It requires understanding what problems users have, and prioritizing them to focus on a high-value area. This avoids the risk of building something that no-one is going to use.
The most effective way of understanding the problem is to get true first-hand experience of users performing real tasks in context. This is best done by applying ethnographic and observational methods to identify the range of problems that exist, then prioritizing them using methods such as surveys.
| Double Diamond Phase | Appropriate Method | Why? |
| Explore the problem | Ethnographic and Observational studies | Gives deep insight into what problems people have that can inspire product decisions |
| Decide what to fix | Surveys | Discovers how representative problems are, and helps prioritise them |
We’ll review each method, in turn, to describe why it’s appropriate.
Explore The Problem With Ethnographic And Observational Methods
The first phase of the double diamond is to ‘explore the problem’. User research can build up an understanding of how people act in the real world and the problems they face. This allows the problem space to be fully explored.
This valuable behavioral information is only uncovered only by watching people do real tasks and asking them questions to uncover their motivations and issues. Doing early qualitative research will help identify the problems that people have. These problems can inspire product ideas, features, and help teams understand how to successfully solve user’s problems. This information will also help disregard poor product ideas by revealing that there is no real need for it. This leads to a more useful product being developed and increasing the chance of success.
The most appropriate methods for doing this are ethnographic. This can include diary studies, where a user’s interaction with the subject matter is captured over a number of weeks. This reveals issues that wouldn’t turn up in a single session or that people wouldn’t remember to talk about in a lab-based interview.
This isn’t the only way of uncovering this kind of in-depth information though. Other suitable observational methods include watching people use existing software or products, both in the lab or in the wild. These are quicker and easier to run than diary studies, but are limited to only capturing a single interaction or what the participant will remember when prompted. For some subject matters, this might be enough (e.g. navigating an online shop can be done and explored in a single session). More complex interactions over time, such as behavior with fitness trackers, would be more sensible to track as a diary study.
Decide What To Fix With Surveys
The second half of the first diamond comes next. Having understood real user’s contexts and what problems they have, these can then be documented and prioritized to ‘decide what to fix’.
This prioritization will be done by product managers who take into account many factors, such as “what do we have the technical ability to do” and “what meets our business goals”. However, user research can also add valuable information by uncovering the size of the issues users have. Surveys are a sensible approach for this, informed by the true understanding of user behavior uncovered in the earlier studies. This allows teams to size the uncovered issues and reveal how representative the behaviors discovered are.
Combining quantitative methods with generative user research studies help inspire early decisions about what a product should do. For example, Spotify’s discovery work on how people consume music analyzed primary research fieldwork to create personas and inform their development work. This allows a team to complete the first diamond with a clear understanding of what problem their product is trying to solve.
Diamond 2: Test And Refine Potential Solutions
The second diamond describes how to end up on a successful implementation of a product to fix the problem. Having decided which problem to fix, research can then explore different ways of fixing that problem, and help refine the best method.
| Double Diamond Phase | Appropriate Method | Why? |
| Test potential solutions | Moderated usability testing | Creates a deep understanding of why the solution works, to inform iteration |
| Refine final solution | Unmoderated usability testing | Can get quick results on small questions, such as with the UI |
Test Potential Solutions With Moderated Usability Testing
The second diamond in the double diamond design process starts with evaluating a variety of solutions in order to decide the best possible implementation of a product. To achieve this with rigor requires usability testing — creating representative prototypes and then observing if users can successfully complete tasks using them.
This kind of study takes time to do properly, and attention on each individual’s user experience to understand what causes the behavior that is observed during usability testing. A moderated session, with the researcher present, can ask probing questions to uncover things that participants won’t articulate unprompted such as “what are you thinking currently” or “ why did you decide to do that?”. These kinds of studies reveal more data when a moderator is able to ask participants these questions, and avoids missing the opportunity to gather more data from each study, which can be used to evaluate and iterate the product. A single moderated research session potentially reveals more useful information than a series of unmoderated tests.
This kind of in-depth exploration of the problem has been a key part of AirBNB’s early success. In 2009 the company was close to bankruptcy and desperate to understand why people were not booking rooms. By spending time with users reviewing the ads on their website, they were able to uncover that the pictures were the problem. This then allowed them to focus their iteration on the process for gathering photos of rooms, which put them on the path for changing hotel booking forever. As the global pandemic changes people’s behavior with holidays in the future, in-depth qualitative research will be essential as they continue to adapt to new challenges.
This doesn’t mean that the moderator has to be in the same room as the participant. Often it can be very valuable to find participants who are geographically remote, and avoid over-sampling people who live in major cities, which is often where research teams are based. Screen sharing software, such as google hangouts or zoom can make remote sessions possible, while still having the session run live with a moderator.
Refine Final Solution With Unmoderated Usability Testing
The final stage of the double diamond describes refining the final solution, which can require a lot of small iterative tests. A shortcut to the deep insight from moderated testing is remote unmoderated research. This includes tools like usertesting.com which allow teams to put their software in front of users with little effort. By sending a website URL to their panel of users, they send back videos of their participants using the website and commenting on their experience.
This method can be popular because it is quick (multiple sessions can run simultaneously without a moderator present) and cheap (participants aren’t paid a huge amount to take part). Because of this, it is often considered an appropriate tool by companies looking to start doing user research, however, care needs to be taken.
This method has constraints which means that it’s only sensible for later on in the design process. Because the participants on these kinds of websites are all people who test multiple websites regularly, they become increasingly different to a normal user. This changes their behavior while using websites and makes it dangerous to draw conclusions from their behavior about what other users would understand or do. This is called a sampling bias — creating a difference between ‘real’ users, and the users being tested.
Because of these risks, these studies may be most appropriate late in development, testing content or UI changes, when the risks of getting decisions wrong are much lower. Iterative studies ensure that users understand what has been created, and are able to use it in the way the designer intended. An example of this is the iterative usability testing the UK’s Government Digital Service ran to ensure citizens could successfully identify themselves and access government services.
After The Double Diamond
As we’ve covered, it is important to select the right method to de-risk product decisions before launch. When a product is launched, it will be immediately obvious whether there is an audience for it, and whether people understand and can use the product — both through how well the product sells, and through reviews and customer feedback.
Nevertheless, launching the right product doesn’t mean that the opportunity for research is over. New opportunities to explore real user behavior will continue to inspire adding or removing features, or changes to how the product works.
| Double Diamond Phase | Appropriate Method | Why? |
| Solution delivered | Analytics + moderated usability testing combined | Inform future updates post-launch with qualitative and quantitative insight. |
Combining some of the methods we’ve described previously with new data from analytics will continue to drive high-quality decision making.
Research After The Solution Is Delivered With Analytics
Post-launch analytics are an important part of building a complete understanding of the behavior of users.
Analytics will reveal what people are doing on a website. However, this information is most valuable when combined with understanding why that behavior is occurring. It is also important to be aware that analytics are only seeing a short section of a user’s experience, the part that happens on your website and their whole end-to-end journey also includes a lot of things that happened off the site, or in the real world. Building a research strategy that combines insight from analytics with an understanding of motivations from qualitative studies is a powerful combination to inform decision making.
This requires close collaboration between the analytics team and the user research team — regular community events, skills sharing and project updates will create awareness of the priorities of each team, the type of research questions they can support one another with and identify opportunities to work together, leading to a stronger combined team.
Optimize Your Research Process
In this article, we’ve covered some appropriate methods to use to inform product development. However, there can still be resistance to running the right kind of study.
New research teams may be asked to cut corners. This can include suggesting participants who are convenient, such as friends, without taking the time to screen them to ensure they represent real users. This can be suggested by colleagues who are unaware of the risks caused by taking decisions based on unrepresentative research.
In addition to running research studies, a researcher has to be an educator and advocate for running the right kind of studies and help their colleagues understand the differences in quality between the type of information gathered from different research methods. Presentations, roadshows, and creating posters are some techniques that can help achieve this.
Incorporating user research into decision making can be quite radical at some organizations, particularly those with a history of deferring to client wishes or listening to the highest-paid person in the room. A lot of hard work and creativity are needed to bring about change in how people work. This requires understanding the decision maker’s current incentives, and describing the benefits of research in a way that shows how it makes their life easier.
If an organization understands and accepts why running studies using appropriate methods it shows a real desire for improving the quality of decision making within the organization. This is an encouraging sign that a new research team has the potential to be successful.
The next step for new researchers will be to establish the logistics of running research, including creating a research process, building out the tools and software needed, and identifying the highest priority research questions for your organization. There is a lot of great guidance from the research community on techniques to do this, for example, the work being done by the research ops community.
(ah, ra, il)
Articles on Smashing Magazine — For Web Designers And Developers
