At Southpaw Insights, we’re always on the lookout for tools and technologies that help us deliver the best possible outcomes for our clients. AI for qualitative analysis and reporting has become a game-changer in our field, offering a streamlined way to tackle the often-overwhelming task of interpreting mountains of unstructured data. This summer, we’ve been exploring how AI can complement our work, and recently, we took a deep dive into evaluating a few AI tools for qualitative analysis and reporting.

The Study: What We Did

We set out to understand both our clients—insights professionals across sectors like healthcare, retail, and consumer packaged goods—and the AI tools themselves. (Check out a summary of our learning about insights professionals here!) We conducted five in-depth interviews with insights professionals and then used two AI tools to analyze the same content that we had already gone through manually.

AI vs. Humans: What We Found

The AI tools did a pretty good job—they aligned with about 80% of the manually generated content. (Anecdotally, we’ve heard from industry colleagues doing similar experiments that tools they are using have a similar success rate.) However, some of the AI output had to be discarded; it either missed the mark or focused too heavily on insignificant findings. On the other hand, the AI did manage to identify some themes and nuances that our manual analysis had initially overlooked, showing us that AI can offer value beyond just saving time.

The Upsides of Using AI in Qualitative Analysis

Through this study, we found three key benefits of incorporating AI into qualitative analysis:

1. Time is on Your Side: AI can dramatically cut down the time it takes to identify and summarize themes.
2. Getting to the Gold in Data Mining: AI is fantastic at digging through data to find supporting evidence for themes. This makes it easier for researchers to back up their insights with solid data points and streamlines the tedious task of finding the right quote.
3. Our New Collaborator: AI can act as a thought partner, allowing researchers to interact with the data in new ways. This back-and-forth can help uncover different angles and lead to richer insights.
4. Variety is the Spice of AI: There are so many AI tools for qual—and new ones coming on the scene every day! It’s important to test out different tools and see what works for you. For our purposes, a straightforward, user-friendly tool was the best bet; tools with too many bells and whistles end up being over-engineered and actually less helpful for us.

Proceed with Caution: The Limitations of AI

As promising as AI is, it’s not without its pitfalls. Here are a few things to keep in mind:

1. AI is Accurate…Ish: AI might sometimes get things wrong, exaggerating or misinterpreting themes. It’s crucial to verify AI-generated output before taking it at face value.
2. Size Matters: AI can struggle with small datasets, often creating themes based on limited input, which can be misleading.
3. Talking About Our Feelings: While some AI platforms offer sentiment analysis, they often misinterpret tone and emotion—key elements in qualitative research.
4. A Lack of 3D: AI’s results can be flat, lacking the richness and nuance that a human touch provides. That’s where our expertise as researchers comes in.

The Indispensable Human Touch

AI’s limitations remind us of the irreplaceable value of human insight. While AI can help summarize data and suggest themes, it’s up to us to interpret these findings within the broader context of a client’s business challenge—and the world we live in. Only then can we turn data into actionable, meaningful insights.

Top Tips: How to Make AI Work for You

If you’re thinking about incorporating AI into your qualitative analysis and reporting, here are a few tips:

1. Start with AI: Use AI to generate preliminary themes and overcome initial writer’s block.
2. Iterate and Refine: Use AI as a thought partner, iterating on its outputs to enrich your analysis. Play around with different prompts to see how AI responds and to spark new ideas.
3. Human Oversight: Ensure that someone familiar with the research reviews and refines the AI-generated content.
4. Contextual Interpretation: Frame AI’s findings within your research objectives, crafting a narrative that makes sense for your client.
5. Develop Deeper Insights: Use AI as a starting point, but rely on your expertise to build out actionable insights and recommendations.

Conclusion

AI holds incredible potential for transforming how we approach qualitative analysis and reporting. It can save us time and offer new perspectives, but the human element remains essential. At Southpaw, we see AI as a powerful tool in our research toolkit—one that’s most effective when used alongside our own expertise and intuition. TLDR: embrace AI, but never lose sight of the irreplaceable value that human insight brings to the table.