On Demand Regression Testing for Security Firewall Product
A security firewall that had a dedicated engineering team with manual and automation QA engineers. The product had the coverage of about 1K automated and 3.5K manual tests. Releases happened every 1.5 months where the 3 weeks before release were reserved for regression testing.
Industry:
Security
Services:
QA as a Service
Location:
Canada
Challenge
The manual team members were not enough to execute the whole regression test suite during that time, so our partner requested additional manual QAs for 3 weeks of each release.
To be able to test, additional manual QA engineers needed access to the testing environment, test management tool and bug-tracking system. To access the bug-tracking system (Jira) and regular testing environment located in internal infrastructure (under VPN access) QA service participants needed to sign the NDA agreement. Since we had so many participants, this requirement could have made the process inefficient and slow.
Solution
The QA Service team dedicated 1 person to all regression phases of each release. This person executed service orchestration and all communication and collaboration between the partner and the Techstack QA service team. This person signed an NDA with the client and got access to the bug-tracking system.
During regression execution, QA service team participants were adding found defect reports into the internal bug-tracker. The orchestrator was checking if these bugs were already added in Jira on the partner side and added new ones. We created one general account for the entire QA service team in the test-management tool (TestRail) and restricted it to a particular regression test set. After each regression phase, we changed the password for this account. We also added a copy of the testing environment for the QA service team in a separate internal network with a route to the Techstack office network.
Technologies Used
Building testing processes from scratch is a tedious and multi-stage process. Our QA engineers assembled a set of important tools required for setting up a testing environment.
The workflow
Investigation
Investigate testing environment, accessibility, and check quality and readiness of regression testing documentation.
Adaptation
Adapt test-cases, define terms and number of QA service participants.
Tools setup
Setup intermediate tools for QA service execution.
Reporting
Post intermediate results during service run.
About the team
We dedicated one QA Lead to all regression phases of each release for service orchestration and the QA service team to run tests.
Impact
The QA team took part in 8 regression cycles, until the project was restructured and sold to another company. The service team executed about 65% of tests during each regression phase and found about 50% of regression unique bugs. Each phase was executed in 2-2.5 weeks vs 3 weeks dedicated to the regression testing phase.