5 Questions CTOs ask about QA

By Stacy Kirk

Over the past 10 years, I have consulted with dozens of tech organizations on how to architect high-quality web and mobile applications. I’ve been fortunate to talk with a lot of CTOs, VPs, and Directors of Engineering who have experienced testing challenges and who have asked my insights on their QA function within their organization. These are the top 5 questions asked or sentiments shared:

1. Why does QA give me such a headache?

Only one CTO was blunt enough to say “Dealing with QA gives me a migraine” but from numerous strained voices of frustration and the rubbing of temples that occur when CTOs explain product quality or testing delays, I think it’s fair to say he’s not the only one with this feeling. So why the headache?

My answer is “Baggage”. If you’ve ever dealt with a person with a lot of baggage, you’ve probably had a few massive headaches. QA is usually one of the last stages before delivery to Production, it carries along the baggage of all the earlier stages: dependencies, missing requirements, costly setups to mimic Production environment, schedule delays, unhappy teams feeling overwhelmed and undervalued …bag…bag…bag.

My recommendation: Keep reading.

2. Do I Need QA or Can Development Do the Testing?

There has been a trend over the past ten years to reduce the number of manual testers and put the ownership of testing on the developers that write the code. CTOs have been very passionate for or against the level of QA needed to deliver products. The question is, do I need QA if we are a test-driven development organization?

My answer is “It depends on the readiness and willingness of your development team; most teams are not there yet.” It’s like deciding between a new tech stack for Development. You look at how it will make your team more effective, how it relates to the product being built, and the success and willingness of your current team in being able to ramp up on the technologies. That same level of diligence is necessary before making the decision to cut QA.

We have all seen organizations that have decreed, “Developers will be the testers” and quickly cut their QA departments. Well, I’ve seen it work and I’ve seen it fail. I’ve seen QA teams cut and then ramped back up two years later. Just because it works at Facebook doesn’t mean it will work with your organization. There is a cultural transformation to testing that cannot be overlooked.

My recommendation: Do an internal assessment. In my work with several clients who faced this same dilemma, my first step is always to guide them through a QA/Agile assessment which helps them to find a solution that best suits their needs and the organization’s priorities and goals.

One way to kick this off is to ask your development team these questions: On a scale from 1-5 (1 being “Hell No… Not I” and 5 being “Absolutely, I’ve been game since last year”),

• Do you feel comfortable with fully testing your code?
• What is your interest in learning?
• Do you think testing should be done by Dev?

If your team scores 3 or lower on those questions, you need to either keep your QA team or create a test transition plan at the same level of diligence required to move your organization to a new tech stack. The transition plan should include identifying QA Champions (from Dev and QA), gaining cultural and technical training & coaching, and pair testing from your QA leaders or consultancy that specializes in these types of testing transformations.

3. What is the best way to hire top-notch SDETs/Automation Developers?

Hiring top-notch experienced automation developers is hard. Very hard. There are very few that make it more than a couple of years before they are called into the ocean by those attractive mermaids called “Dev”, inevitably leaving their career of QA to become a developer. I was coaxed many times… “Stacy, you are such a good developer, why waste your time with QA?” As I grew my consultancy firm of test-driven automation developers, I experienced the same retention problem.

So what have I recommended? My answer is Build Top Notch.

My Recommendation: Hire brilliant developers with only a few years of Dev experience and pair them with great test automation developers. This will provide them with a mentor excited by a career path in QA; because they are brilliant, they will ramp up faster than the time it will take to fill that Senior Test Automation.

If you don’t have experienced test automation developers in house, I suggest bringing on 3-6 months of consultancy to help ramp up their skills and to build out a strong QA Test Automation Architecture for your team. Many consultancy firms are open to having a part-time ongoing relationship for continued mentorship.

If you are determined to hire a Senior SDETs, understand that their world is a lonely one. Sweeten your benefits to include time off for conferences and training, to attend relevant meetups, and provide subsidized membership fees for test automation groups.

4. The test reports returned good results, why are there so many bugs in Production?

When I hear this question, there is definitely a headache forming, mostly because someone has just called or knocked on the CTO’s door screaming “What’s going on? Our Customer is pissed! Why was this missed? How could this be missed, it is so obvious?”

My answer: It’s complicated. Okay, before you roll your eyes (yes that happens with this answer), oftentimes it’s not the simple answer “Tester Jo didn’t test the feature properly”. There are many nuances from Production bugs from environmental differences, deployment misconfigurations, to inconsistent reproducibility, or missing requirements covered in heavy assumptions.

My Recommendation: Without exception, for every high severity bug missed, there should be a retrospective to carry out root cause analysis of the issue (not just assumption). The investigation team should consist of Dev, QA, Operations, and Product (not just QA). Once the root cause is identified, the team should build a solution and add the required work to the technical debt backlog. As a CTO, you have to ensure that those improvements get priority. This exercise will be uncomfortable at first (but bugs in Production make us all uncomfortable) so please ensure that the team understands it’s not a “blame game” exercise for you. It’s about building better customer experiences, which is everyone’s responsibility.

5. How do I raise the bar of our QA capability?

Some QA teams were initially hired to only execute manually. According to PractiTest’s State of Testing Report 2019, over 74% of testers are doing some form of test automation or scripting within their organization. The challenge comes in ensuring that manual test teams can transition to an automation-driven testing model. Other teams have automation but leadership isn’t clear if this saves time, identifies bugs, or increases coverage. As technologies change, many CTOs feel that their QA team is not keeping up or has a clear strategy to improve their technical capabilities to keep pace with their development team. How is this solved?

My answer: It starts with investing in an assessment and road mapping activity.

My Recommendation: The fastest and most comprehensive way is to hire an external firm to come in and conduct an assessment team that will offer comparative data and a suggested improvement roadmap. For example, QualityWorks offers an Agile Test and DevOps Maturity Assessment (ATDMA) that provides gap analysis, industry comparison scoring, and a maturity development roadmap inclusive of process, tools, and training. Here’s a sample of our comparison chart that shows how a company ranks against industry standards based on  the results of an assessment:

If there’s no budget for an external assessment, you can work with your current team to answer the following question: Where are the technical, process, visibility, and cultural gaps within the QA organization? Once identified, they should prepare a roadmap to improvement.

I’m about to make a generalization about QA …they like to find bugs. Not just in code, but in life. Therefore, if you task them with figuring out the bugs in the QA organization, architecture, and process, you’ll be amazed at what they can present, especially if you have a more experienced team. When my team and I assess organizations, we are always blown away by the readiness of the QA organization to share their own internal bugs and to recommend areas of improvement. They share because they know their information is anonymous. So as their leader, you will need to provide a platform that allows them to be open to share without fear of retaliation or judgment.

STILL IN NEED OF AN ADVIL?

I hope this will help relieve a few CTO headaches and provide techniques for continuous improvements and visibility within the organization.