Developer testing – effective QA against the rules

Yes, you heard right! Developer testing. It means testing done by developers! And yes, I’m talking about the confirmation testing, which is known as “The changelog” in our R&D department. The result – improvement from 50 % to 95 % of tickets closed at the end of a sprint and all sprint goals completed on time 4 sprints in a row! [1]

Our development team has recently been trying very hard to shorten the development cycle of features and fixes. One of the biggest delays we have identified was caused by the tickets waiting in the “To Test” state. It means that the implementation part has been completed and is waiting for a QA Engineer to confirm it’s functionality by testing it. As I was the only tester for 7 developers on the team, the tickets with lower severity had to simply wait, often more than a week. Moreover, the testing activities were concentrated at the end of a sprint. A ticket reopened too late can easily be a cause of a failure in reaching team’s sprint goals.

Generally, it is strongly recommended by the literature to not let developers test their own work. The common reasoning is, that everybody is in love with their creations and would not like to damage them. Also, some defects are approach-related and thus cannot be discovered by the same person with the same approach. Moreover, test planning, test case writing and test data definition need special skills, which developers generally do not possess. Our mindset was changed by our CTO, who saw this as an opportunity to improve efficiency of the development.

In our team, we kept all of the aforementioned risks in mind and tailored the process to negate all of them. We have tried several versions of the process. In a short time we found the most important thing – that we need to differentiate tasks (new development) and defects (bug fixes). You’ll later see why.

Generally, it is much easier to write test cases for known functionality. This is the case for the defects which only fix or slightly modify an already tested feature. Experiences with an existing (sub-)system, where a feature is only updated and testing approaches are well known, help the QA engineer to define a set of all necessary test cases, edge-case scenarios and also expected visual aspects. Therefore, based on the well-defined test cases, a developer should be able to develop the fix and test it in a way, which will ensure it meets quality requirements. Later, a QA engineer interviews the developer to find out his confidence level about the testing and also asks several direct questions about the most critical areas. Based on this information and on the experiences with the particular developer[2], the QA engineer then decides which defects need to be retested and which can be closed right away.

On the other hand, tasks usually represent development of new features. Since the people writing the code in Y Soft are not “coders” but developers, they have to propose solutions for many low-level issues throughout the development process. Therefore, it frequently happens, that some aspects and limitations are discovered later in the sprint, making it very challenging for a QA engineer to define a complete set of test cases in advance. Also, without a proper hands-on the final work, it is also very difficult to define requirements for visual aspects and to judge user friendliness. Therefore, tasks have to always be retested by a QA engineer.

Nevertheless, defining at least some of these tests brings certain benefits that are also common for the defects:

• A QA engineer can discover inconsistencies between his/her understanding compared to the developer’s understanding of the work to be done. It is generally better to find them before significant time has been spent on development of something undesired and reworking it later.
• A partial test suite can be very helpful to the developer during the development, as it can be used as a checklist to cover many possible scenarios.
• Some other scenarios can be discovered during the development and can be added to the test suite. These test cases would otherwise probably not exist.
• As the developer performs tests himself, many issues are found and fixed in a much shorter time and with less effort than it would when they are found and reported back by the QA department (several days later). This way we can assure higher quality of the developed product in the development phase.
• Based on the current state of work and human resources, the team can flexibly agree on the extent to which developers will test their work. Either they do extensive tests to help QA engineers, or they perform only a basic set of tests in order to move on to the next development task sooner.

These result in:

• shorter development cycles (Open to Closed status)
• less reopened tickets
• better understanding of the whole solution for all members of the team

The process itself consists of the following steps:

1. When the sprint backlog is defined:
   1. A QA Engineer creates a set of test cases for each of the tickets (pre-)planned for the sprint;
   2. The test cases are defined in a subtask of each ticket. The subtasks are named “Testing of [ticketID]”;
2. At the sprint planning meeting:
   1. The QA engineer consults on the technical details of the solution and proposed tests with other team members and the current product manager;
   2. The effort for each ticket is estimated including the testing part;
3. Developers have to test their work and switch the ticket to status “To Test”:
   1. All defects are tested in a standardized testing environment (ideally prepared by a QA engineer);
   2. Tasks can be tested in a development environment (e.g. running on a developer’s machine built by a development tool);
4. When a ticket has the “To Test” status, the QA engineer:
   1. Evaluates which defects need to be tested again and which do not;
   2. Retests all tasks;

It is important to note that the aforementioned benefits are only subjectively observed by the members of the team, as none of them has been measured in any systematic way. Doing that would require returning back to the old way of work, in order to make the necessary measurements. Since members of the team are satisfied with the new process , there is no need or motivation to revert back to the old method. A change from less than 50 % to 95 % of closed tickets from the end of Sprint 17 to the end of Sprint 18 and the 100 % fulfillment of sprint goals in the last four sprints presents a sufficiently strong argument to try this process in other teams.

[1] The orange line represents the percentage of developers’ time was planned for a sprint. The time is estimated for each ticket. The green line represents the percentage of that planned estimated time, for which the tickets were closed at the end of the sprint. The first attempt to use developer testing was in Sprint 18. In Sprint 22, we resumed the process. The trend of about 70 % of work planned and more than 90 % finished remains to date.

[2] In order to gain experiences with the developers, there has to be a period of several sprints, where every defect is retested. The QA engineer needs to measure the ratio of closed and reopened defects per developer. During this period the QA engineer can also find out, whether he/she is able to define all necessary test cases beforehand.