Determined Testers in Non-Deterministic Systems: 4 Invaluable Characteristics for GenAI Testers
By Stacy Kirk, CEO & Founder, QualityWorks Consulting Group
Freshman year at Stanford, I couldn’t wait to leave my small Texas town and step into a world of brilliant computer scientists. My plan was simple: write code, build cool things, and make enough money to buy a Lexus. To make sure we weren’t complete nerds, we were required to take a few humanities courses—and that’s how I found myself in a year-long philosophy class. It was the first time I was challenged to question ideas, beliefs, and the human experience. I remember my professor asking what I thought of Freud’s ideas, and my immediate reaction was, “I’m a 17-year-old girl from Hurst, Texas. Who am I to question Freud?” Over the course of that year, though, I built my confidence in analyzing concepts using reason, logic, and critical thinking. That remains one of the greatest gifts of my education and the foundation of how I think about technology today. Now, as AI reshapes our industry, I realize that learning to question—rather than just confirm—is what makes testing non-deterministic systems both possible and powerful. The best testers I’ve worked with share four defining traits that will make them thrive in the unpredictable world of generative AI.
1. Domain Knowledge: The New Superpower
For those entering tech—or coming from another field—domain knowledge is pure gold. In AI testing, what used to be 200 pages of business rules is now buried in data, model weights, and inference logic. Testers who understand the underlying business, its ethical boundaries, and the regulatory context can assess whether AI outputs fall within an acceptable range of understanding.
Stephanie Jones, a consultant at QualityWorks, captures this challenge perfectly:
“One of the challenges for testing GenAI is testing for edge cases. Because of the non-deterministic nature of the application, traditional boundary value analysis techniques don’t apply, and it can be hard to predict outputs—or even harder to trigger consistent responses with the same input. One way to help alleviate this issue is to sit with developers and really deep dive into the decision-making process of the application. This then helps testers to better curate test data. Domain knowledge is crucial for testing GenAI—the better you understand the system under test, the better you can create robust test cases and predict system behaviors.”
The job now extends far beyond accuracy to specs; it’s about validating compliance, fairness, bias, safety, and ethical use. Those who bring real-world expertise into heuristic testing become indispensable—the critical human in the loop who can judge when an output is not just correct, but responsible.
2. Empathy: The Edge of Human Insight
Empathy has always mattered in QA, but with generative AI, it’s everything. Our work is no longer about preventing bugs—it’s about ensuring that the AI’s “truths” make sense for real people. I often ask my teams to create detailed user personas for every system under test. We explore not just user demographics but their frustrations, motivations, and goals. When testers understand how an AI’s output will be received—whether it enlightens, frustrates, or misleads—they can better define what success looks like. Empathy turns testing from a technical checklist into a human advocacy practice, ensuring AI systems serve, not alienate, their users.
3. Critical Thinking: Becoming the Judge
Critical thinking is more than a buzzword—it’s a survival skill for AI testers. In traditional QA, we could measure quality against expected results. In the world of AI, there often is no single expected result. Testers must evaluate sources, weigh input variance, and make reasoned judgments about what constitutes a “good” output. The role has evolved from enforcing rules to interpreting context. The best testers integrate multiple perspectives—business intent, user needs, ethical implications—and render informed decisions. In many ways, we’ve become both scientists and philosophers: comfortable with ambiguity, disciplined in reasoning, and courageous enough to say, “This is good enough—and this is not.”
4. Adaptability: Comfort with the Chaos
If consistency was our north star in traditional QA, adaptability is our compass in generative AI. Outputs may vary from one test run to another—and that’s not necessarily a defect. Testers must learn to distinguish between meaningful deviations and harmless variations. Structured testing approaches like “golden master” baselines, metamorphic testing, and continuous refinement cycles help us stay grounded in this flux. But beyond tools and techniques, adaptability is a mindset: accepting that uncertainty is not the enemy, but the environment. The testers who thrive are those who can pivot, learn, and keep moving forward—steady in the storm of change.
As AI systems become more powerful and less predictable, testers have the privilege—and responsibility—of shaping how humans will trust and interact with these technologies. Philosophy taught me that questioning the greats wasn’t arrogance; it was growth. The same is true for us today. In a non-deterministic world, our determination to think, question, and adapt is what turns uncertainty into excellence.
Written by: Stacy Kirk, CEO of QualityWorks
Ready to advance your AI Testing Strategy and Tooling?
Let’s have a conversation about your testing goals and how training, infrastructure, and automation can fast track it.
Schedule a free consultation with Stacy Kirk
Request a demo of our Managed AI Test Squad.