Cleaning while Cooking: How to Effectively Synthesize Qualitative Data
Deriving meaning from large, complex sets of interview data is the bread and butter of a qualitative researcher’s job. It’s a gratifying task — but a tedious one too.
We spend hours reading and re-reading conversations we’ve had with participants, attempting to pull out patterns and through-lines (“coding”, as we call it in the social sciences) from the things people are saying. We want to dig deeper, to get past the surface-level ripples of their opinions and preferences and find what Indi Young calls the “deeper current” that guides their behavior.
In User Research, we then share our research findings with our company with the aim of informing design.
Having been in the field for two years now, I’ve started fine-tuning my own process for analyzing interview data in an efficient but thorough manner. I call it “cleaning while cooking.”
But first, a proper research argument
Let us start by laying out the skeleton of what a valid research argument looks like, as formulated in Wayne Booth’s “The Craft of Research”:
- The sentence that sums up the results of your research is your claim. It asserts that something must be true or false and so needs support.
TV can have harmful psychological effects on children
2. The sentence supporting your claim is your reason.
because when they are constantly exposed to violent images, they come to think violence is natural.
3. The second kind of support on which you base your reasons is your evidence.
Smith (1997) found that children ages 5–7 who watched more than three hours of violent television a day were 25 percent more likely to say that what they saw on television was “really happening.”
Therefore,
CLAIM [because of] REASON [based on] EVIDENCE
TV can have harmful psychological effects on children (claim) because when they are constantly exposed to violent images, they come to think violence is natural (reason). Smith (1997) found that children ages 5–7 who watched more than three hours of violent television a day were 25 percent more likely to say that what they saw on television was “really happening” (evidence).
In User Research, our research “arguments” take on a variety of frameworks, and can look more like this:
CLAIM 1, because REASON
EVIDENCE (quote)
EVIDENCE (quote)
CLAIM 2, because REASON
EVIDENCE (quote)
EVIDENCE (quote)
and so on.
Now, let’s look at how we assemble our claims, reasons, and evidence in our day-to-day research work.
The User Research process
If we take a look at the typical steps in a research project, it looks something like this (note that they are not always linear):
Cleaning while cooking is a method for the fourth step of the research journey, the synthesis step. It brings clarity and efficiency to the sometimes tedious process of analyzing people’s words, of answering, “what did we find?”
Questions to Answer
Deciding what to find in the interviews is always connected to what we asked in the first place: “what should we learn?” As in, while we want to leave as much space as possible for unexpected patterns, or knowledge that we didn’t see coming, the answers we want to pull out should be connected to the questions we want to answer. In User Research, we call these “Questions to Answer.” They’re not the questions we’re asking the participants, they’re the questions we have as a team and want to answer in that study.
They can be questions such as, how do users interact with our paywall? Or, why do college students use Spotify and not Apple Music? These questions then guide the crafting of your interview script.
So what is the “cooking” and what is the “cleaning”?
What happens when you’re cooking a meal and you save all the dishes for after you’ve finished cooking? You turn to your sink and see a large pile of pots and pans. Even though you want to rest since you’ve spent all this time cooking, you now have to roll up your sleeves and spend 30 minutes washing the dishes. Or worse, you have to live with the mess until you have the energy to tackle it later.
When I say “cooking” in research, I’m referring to conducting an interview, or watching a recorded unmoderated interview.
“Cleaning,” on the other hand, is taking notes, writing quotes, coding, and aligning findings with our Questions to Answer.
Just like washing the dishes, if you wait to “clean” your data until after you’ve finished, say, 10 one-hour interviews, you now have a mountain of unstructured data you have to go through. It’s easy to get lost — to lose track of what Question to Answer you’re focusing on, because oftentimes the participants are addressing these questions out of order. You might also be depending on memory to go back to different participants and grab various quotes, and you can lose a lot of good nuggets of information in the process.
Instead, clean while you’re cooking. In other words, debrief iteratively. Conduct an interview, process your data. Conduct a second interview, process your data. It’s like having mini debriefs in between sessions.
How to cook and clean your research
Step 1: Write out all of your Questions to Answer.
Question to Answer #1 Why did users choose Spotify and not Apple Music?
Question to Answer #2 Do they want privacy for their playlists?
Question to Answer #3 How did they first hear of Spotify?
Step 2: Conduct your interview. If you ran an unmoderated study, watch your recorded interview.
Step 3: Take notes on that one interview. Write down important quotes.
Step 4: Go to each Question to Answer and write down the “insight” for that participant.
I put insight here in quotes because it’s not a validated insight yet — we have to look at all of the data for all participants before we elevate it as a true insight.
Now your list should look like this:
Question to Answer #1
Participant 1: said she liked library of Spotify more than Apple Music. Radio feature showed her more songs she liked than Apple’s did.
Question to Answer #2
Participant 1: wants privacy for playlists — “I don’t want anyone seeing the playlists I make”
Question to Answer #3
Participant 1: introduced to Spotify by friend while studying abroad
Step 4: Move on to your next interview and continue your list, so it starts to look like this:
Question to Answer #1
Participant 1: said she liked library of Spotify more than Apple Music. Radio feature showed her more songs she liked than Apple’s did.
Participant 2: observation observation “quote”
Participant 3: another observation
Question to Answer #2
Participant 1: wants privacy for playlists — “I don’t want anyone seeing the playlists I make”
Participant 2: observation
Participant 3: observation ”quote”
Question to Answer #3
Participant 1: introduced to Spotify by friend while studying abroad
Participant 2: observation
Participant 3: observation
…
Participant 10: observation “quote”
and so on, until you’ve done this cooking and cleaning process for all of the participants.
At the very end of your interviews, you now have a clear list of all of the participants actions, sentiments, and quotes grouped according to each Question to Answer. We’ll call these your mini insights:
Question to Answer #1
Participant 1: mini insight
Participant 2: mini insight “quote”
Participant 3: mini insight “quote”
Participant 4: mini insight
…
Participant 10: mini insight
One thing to note here: this isn’t just some organizational hack. It’s also a way of holding yourself accountable as an objective researcher when you might otherwise rely on what Daniel Kahneman calls the “ease of availability” heuristic in intuition studies; believing something to be true because we can easily recall other points of evidence by memory.
With cleaning while cooking, we avoid falling into this fallacy. Instead of claiming that participants felt a certain way because I remember (but didn’t write down verbatim) many of them echoing a particular sentiment, I’m forcing myself to look at the concrete evidence. Writing them down also helps alleviate any biases that can occur in normal social interactions, like when a person’s tone of voice, charisma, or emphatic response threatens to supersede truth.
Now, you can look at your Questions to Answer and code these mini-insights–pull out that high-level pattern you’re seeing. Put that high-level pattern at the top of all the mini-insights with supporting evidence. This is our Claim 1, or Main Insight.
Question to Answer #1
Most participants stated they preferred Spotify because it introduced them to new songs they liked. (Claim 1 + Reason)
“I tried Apple Music first but I hated the songs it would automatically go to after I played a song.” (Evidence)
“Spotify has a Radio feature where I can choose a song I like and it creates its own playlist from that song, and I’ve learned so much new cool music from that.” (Evidence)
Participant 1: mini insight
Participant 2: mini insight
Participant 3: mini insight
Participant 4: mini insight
…
Participant 10: mini insight
Question to Answer #2
Most participants stated they want the option for their playlists to be private because they don’t want everyone to see their music tastes. (Claim 2 + Reason)
“Music is very personal to me. Sometimes I find music I love that’s kinda weird or strange and I don’t want anyone else to see.” (Evidence)
Participant 1: mini insight
Participant 2: mini insight
Participant 3: mini insight
Participant 4: mini insight
…
Participant 10: mini insight
Finally, you can take all of your Claims with their supporting Evidence and format them into your company’s preferred report method, be it Powerpoint or Google Docs. The list of individual participant mini insights can be saved in your repository of notes for that study .
An added benefit of cleaning while cooking is that it can enable you to share the workload across your research team and even accelerate your ability to report findings. In a strategic design research project conducted at Course Hero in November 2020, we had our director of UX research, George Zhang, act as our main “dishwasher,” if you will. As my colleague and I conducted 8–10 interviews in the span of two days, George listened in and cleaned as we cooked. He took notes and extracted quotes and insights as we ran the interviews. This was, of course, a luxury, but was necessary with a short deadline and a high number of interviews by the researchers. By the time we finished our 10th interview, George had compiled the list of main insights, and we spent only one more hour writing up additional insights.
Overall, this method of cleaning while cooking has saved me about 5–6 hours of synthesis time per study. It’s also brought me much closer to my findings. I walk away from a study now knowing that I looked at it thoroughly, and the results are much more “sticky” to me in the long run.
