User testing is inherently time consuming. It takes time to identify appropriate interviewees, schedule and conduct moderated sessions, pore through feedback, identify patterns, and make decisions based on that analysis.
But it is essential to learning design because it supports a core objective of the process. Learning design seeks to create experiences that draw a learner in and make it easy and enjoyable for them to absorb the material. User testing puts learning design experts in direct contact with real users to understand their likes, dislikes, mindsets, and frictions. It’s this firsthand knowledge of the users they’re designing for that allows experts to build engaging, effective programs.
In the era of AI, much of the anxious dialogue surrounding the technology stems from a fear that robots will replace humans. In the best cases, though, artificial intelligence is not supplanting people, but rather freeing them up to do what they do best — and with greater speed, efficiency, and accuracy.
Lauren Gould, senior learning strategist here at Studion, has uncovered ways to incorporate AI into her user testing workflows to do just that. She leverages specialized AI tools to unlock better insights faster and turns to AI to manage the elements where it excels, so she’s free to bring her human expertise to the parts of the process that demand her knowledge, discernment, and humanity.
“In years past, the user testing process would have taken a team of us a week or more. From running dozens of half-hour or hour-long moderated sessions to combing through meeting recordings by hand and tallying responses, it was laborious,” Gould explains.
“Now, we can do it all in two to three days.”
Finding the right fit for AI in the user testing process
At its core, user testing is a deeply human operation. It’s the act of one person, the learning strategist, coming together with another person, the participant, to understand them more fully.
Through testing and challenging their own assumptions, chasing interesting lines of conversation, and finding the connecting thread across interviews, people like Gould design learning experiences that address genuine user needs and create Signature Moments that ultimately make a lasting impression on learners.
Because of this interpersonal, intuitive work at the heart of what she does, Gould knew AI could never run the process independently and that certain steps were off-limits for automation.
“When you’re doing research with users in the beginning phases, it’s as much about educating you — helping you figure out if you’re even asking the right questions and exploring the best areas — as it is getting answers. So that’s a collaborative process I knew an AI could never do,” she explains.
However, there are plenty of other areas where tedium stood in the way of Gould’s ability to do her highest-order work. “Having to comb through transcripts and physically tally the results of questions where we asked participants to rate things on a scale of one to five? That took up a lot of time,” she shares.
In thinking through her user testing workflow, Gould identified areas where AI could fit and others where a human expert could simply never be replaced.
How AI enhances user testing
So how has AI changed the way Gould approaches user testing? It has reshaped the process in several ways.
AI for qualitative data analysis
Perhaps most notably, it radically streamlined the process of digesting qualitative data. “After we’ve recorded interviews, we feed anonymized versions of the transcripts into a tool called Dovetail, which instantly creates visually editable transcriptions,” Gould says.
On the platform, she can tag parts of the conversation, like user mindsets, frictions, or most-liked features, and easily surface patterns across interviews. Not only that, the tool can then create video clips of similar feedback and assemble them into a highlight reel for the client. Before, time constraints would have made it impossible to edit video compilations. Dovetail does it automatically.
AI for quantitative data analysis
Quantitative data is another area where the right AI tool shines. Gould shares, “We’ve started using Julius, which is built to run Python directly, unlike a standard ChatGPT chat, so it can actually count much more reliably.” Julius not only accurately tallies the quantitative results, it also creates visualizations of patterns in minutes.
“I’ve used Julius to make a really cool heatmap of features — it was a complicated story to tell, and the visual helped the client understand how users were reacting to the concepts we presented,” she says.
AI to conduct unmoderated testing
In later-stage user interviews during the build phase, Gould is integrating AI in two new ways to increase the volume of feedback they’re getting on products. The first is unmoderated testing.
“Listen Labs allows us to run interviews without us in the room,” she says. “Participants speak with the AI, and you can pre-program the tool to dig deeper when it needs more from an answer.” This allows the team to include a wider swath of participants in this phase of research without demanding additional interview hours from the team.
AI to accelerate prototyping
AI is also enabling faster prototyping. “AI can make really realistic-looking interfaces very fast. Instead of testing with wireframes, we’re able to create concepts that look more fully formed — with the caveat to users that it’s not design,” Gould explains.
Recently, she showed 20 concepts to more than 100 people over the course of a couple of weeks. Without AI, it would have taken many more resources to complete the same amount of work in that short amount of time.
By incorporating AI into these later stages, Gould has more flexibility to experiment without the fear of losing precious time on untested concepts.
“We can put out an idea and throw it away if it doesn’t work, because it didn’t take us weeks and weeks to build,” she says. “The speed at which we can learn is so much faster.”
The importance of human expertise
While there are dozens of artificial intelligence tools cropping up to support the user testing process, there is no substitute for human knowledge and expertise.
Gould finds that the tools for assessing research have sped up her processes, but especially for digesting qualitative sentiments, they still require accuracy checks. “It’s right much of the time,” she says. Beyond assessing for correctness, Gould is still bringing human intelligence and an expert eye to qualitative results. While the AI tool can synthesize data and identify patterns, a human finds meaning in those results and makes the ultimate decisions about how to move forward based on what they see.
Humans also play a vital role in maintaining oversight and compliance. Gould and the team work within the compliance parameters of each client, identifying secure AI solutions and finding alternatives when a partner’s AI rules require it. She also anonymizes data before feeding it into third-party platforms, removing all identifying information about clients, participants, and the project at hand.
Delivering better learning experiences with AI
Ultimately, the purpose of AI adoption isn’t to make our internal processes more efficient; it’s to enable our expert team to deliver better results for clients.
As Gould puts it,
“Better data equals better design.”
And she’s seen firsthand how AI has created better client outcomes. In a recent project, Gould faced the challenge of creating a learning experience for an organization that had gone through an acquisitive growth spurt. Along the way, they’d picked up new sets of users and found that their learning solution wasn’t meeting the needs of these diverse groups.
“They had so much opportunity; the challenge was in knowing where to focus across all their end users,” she says. AI allowed them to connect quickly with many more participants and to synthesize the transcripts from those conversations in days, not weeks. Gould also used AI in prototyping, so she could share more creative outputs with users in the testing phase.
Ultimately, AI helped her surface more nuanced insights across a wider range of participants, then made it easier for her to visualize results for the client so they could clearly see the best decisions to take.
As we continue to integrate artificial intelligence into the user testing processes, we remain focused on surfacing more high-quality data, iterating more effectively, and providing clients with a roadmap to design better learning experiences in less time.
