We’ve been in the learning technology space for over 15 years, and we’ve seen our fair share of new tech come onto the scene. Our approach to new technology remains the same even as the technology itself changes radically: successful implementation aligns technology with both learning and business objectives. Which brings us to AI.
Lay the Groundwork for Successful AI Implementation
AI in online learning should amplify learning, not substitute for the critical thinking and engagement that drive real learning. How? We have two main approaches.
- We use our High Engagement at Scale™ framework to evaluate potential AI use cases using a systematic, standardized rubric. Currently, we are seeing the strongest results in Active Learning and Real World Outcomes, as well as in using AI for data-driven content authoring.
- We apply research-based strategies to ensure AI enhances rather than hinders learning.
Example AI Application: AI Roleplay for Hard-to-Practice Skill Building
Our long-tenured dev team is working on a wide range of AI-related projects–from AI search to analytics to digital twins. We’ll focus on one recent project, the lessons we learned, what AI made surprisingly easy, and what was–equally surprisingly–challenging.
Our latest AI project is an AI video roleplaying tool that allows learners to develop hard-to-practice skills. Designed for busy professionals, it allows financial advisors to rehearse high-stakes client conversations. The AI model uses detailed client scenarios to coach learners on how to work with these clients, specifically the content and tenor of the conversation.
We took this AI pilot a step further, and decided to use AI for as many applications as possible: UI/UX, project management, and documentation. Our goal was to find out what works and what doesn’t to both find efficiencies for our own business and best practices for our clients. Here’s what we uncovered.
Lessons learned in building AI applications for digital learning
1. Everyone’s a product manager.
We are constantly surprised both by how much AI can accomplish and how much direction it requires. When using AI to help build software, the humans using it need to deeply understand the product goals and user needs to effectively guide the AI. As Marin Bareta, Software Developer at Studion, observes, “You become the director, the leading role, the product manager.” LLMs might contain a large swath of collective human knowledge, but they don’t understand your product vision by default. They require a sort of constant, clear direction that’s not immediately intuitive.
2. Iterate, test, and iterate again.
An iterative approach to software development is not a new concept. However, it is particularly relevant to AI, where results are immediate. In our workflows, we engineered a prompt, evaluated the results, and immediately refined our prompt or code. It took many, many tries to get the results that we wanted. But AI is so very fast that this iterative approach actually saves us time.
The key here is to thoughtfully design test scenarios. For example, we tested a video with abjectly terrible client service. What would happen if we were arrogant, rude, and advised investing solely in Bitcoin? Would the AI respond appropriately and consistently?
3. Prompt engineering is still human.
Because our goal for this project was to use AI in as many ways as possible, we used ChatGPT for prompt engineering. Very meta, we know. While this approach worked to solve the “blank page problem,” the AI-generated prompts required a lot of reworking. Frankly, we turned up a lot of slop. We found that prompt engineering, the high-level strategy, required an equally high-level of human input. Nevertheless, it’s often easier to correct a draft prompt or workflow rather than creating one from scratch. So again, this approach saved time.
4. AI tools on top of AI tools.
We were surprised by just how many AI tools exist. “Explosion” is the generally accepted term for this phenomenon, but it doesn’t truly do it justice. It was challenging both to find something that hadn’t already been built and to find the exact tools we needed.
When we launched the AI Roleplaying project, we thought that video transcription and emotional analysis were going to be the heaviest lift. We were wrong, because OpenAI and AssemblyAI already had a ready-to-use AI transcription. For behavioral and emotional analysis of video data, HumeAI provided so much analysis that we needed an additional AI tool to simplify its output for end users. The build-versus-buy question becomes increasingly complex as the interaction of different AI agents and tools proliferates. We use N8N to organize and document workflows. Our latest N8N workflow is available here.
5. Content authoring for AI
We found an important distinction between content designed for human consumption and content optimized for AI systems. Most text-based content that exists today–books, research papers, blog posts, white papers–was created for a human reader. While AI tools can certainly process text-based data, they require guidance to reason, analyze, and draw conclusions. So, for an AI agent or application to rely on a specific content library, that content needs to be refactored for AI’s consumption in order to produce accurate results. This involves strategic chunking, clear formatting, and careful instructions on exactly how the AI should interpret the content. These techniques are key components of retrieval-augmented generation (RAG) systems.
6. What actually is RAG anyway?
RAG–Retrieval-Augmented Generation–is quite the buzzword. It’s generally defined as optimizing the output of an LLM so that it references an authoritative knowledge base outside of its training data sources. We explored several RAG techniques, and found each to have its own pros and cons.
At the end of the day, RAG is really about injecting facts into a database that you can retrieve later. Deciding which facts, and how to inject them, is the challenge.
“You cannot feed AI a car service manual and expect it to fix a car. But what you can do is structure the service module in ways that help the AI provide reasonable options for fixing a car,” says Bareta.
7. Wow, AI really cannot add. But MCP can.
While newer AI models have improved significantly, LLMs have well-documented challenges with reliable mathematical reasoning. We found the best way to address this is with MCP (Model Context Protocol), a standard for connecting AI assistants to external systems where data lives. Essentially, you have to give AI a calculator and tell it how to use it.
Beyond basic math, MCP is a powerful enabler in building AI tools. It’s like a connector–a USB-C key–for linking AI to content repositories, business tools, and development environments. Use cases include connecting to workplace apps like Google Suite or retrieving information from a website.
8. AI systems don’t automatically self-improve
One of the common misconceptions is that a custom AI tool will automatically get better from user interactions. Unlike what many people expect, accomplishing this requires thoughtfully designed processes. In our case, this meant carefully orchestrating updates to our AI tool: fetching user data, feeding it into the system, updating the knowledge database, and creating detailed prompts.
Paradoxically, our latest AI project highlighted just how essential human creativity, oversight, judgement, and intuition are when creating effective AI applications for digital learning. AI clearly has the ability to transform learning by making previously impossible pedagogical approaches practical at scale. But the opportunity comes with risk, both to learners and the organizations serving them, which is why a detailed understanding of the nuts-and-bolts of how AI works is critical for creating successful AI-enabled learning experiences.
