1. Background
Leaked test bugs refer to product logic defects that are not found during the testing process (especially those that can be reproduced in the test environment), and are found and fed back after the release of the online version or after the user experience. It may cause online failure or asset loss. In the process of product testing, it is inevitable that some bugs will be missed. Therefore, some thoughts on bugs and missed tests will be made and summarized.
2. Reason Analysis
Bugs are actually a problem that any application product will have. Not all bugs can be found, including senior testing, more or less online defects will appear. No one can think about all the functional operations and application scenarios of the software. . Although it is impossible to achieve completely zero defects, every time we release a product, we need to pursue fewer and fewer defects, higher and higher product quality, and reduce feedback on online problems.
There are mainly the following reasons why defect detection occurs:
2.1 In the requirements review stage, the understanding of the details of the business requirements is not clear, the design is unreasonable, and the hidden expansion requirements have not been deeply excavated
In the actual product development process, the product requirements are actually in the process of refinement, optimization, and drill-down. When the interactive document output of the PRD document is required for review, some product details and implicit requirements cannot be exposed, and the test cases The writing is based on PRD, interactive documents, and the test cases involved in their own experience and understanding of the requirements.
- Before the requirements review, we should carefully read the PRD and interactive documents, first form our own thinking about the product, list the doubts about the product design through the brain map, and find out the product design defects from the perspective of users or from the industry.
- In the demand review meeting, communicate your doubts and doubts about the product to the product and development with the listed doubts, and ask more why? How to achieve? Data acquisition source? How to deal with data that exceeds expectations? How is the cache handling mechanism? Where is the data stored? Is the logic handled by the frontend or is it served by the backend? Is the backend service logic associated with a third party?
- After the requirements review is completed, according to certain functions, the requirements are divided into several large modules, and the large modules are divided into small function points, and then the specific implementation process of the function points is considered, and the module functions are subdivided through the mind map, from the page to the interaction , Boundary processing, interface logic, environment configuration and other dimensions to sort out the requirements, dig out the hidden and expandable demand points as much as possible, and then conduct a demand review and technical review in the test group, let the collaborating members supplement the implicit requirements together, and make the product design Defects are exposed as early as possible and maximized.
- In the post-technical review, discuss logical interaction, upstream and downstream data trends, and message sending and flow, and connect doubts on the technical side.
2.2 Incomplete coverage of test cases and missing scenarios
In the process of test case design, it is easy to have limited thinking or blind spots in demand. It is impossible for us to completely cover all the scenarios used by users. It is not realistic to write them all as test cases to simulate and cover.
Before the start of use case design, make a mind map
List business processes, front-end and back-end interface logic through mind maps. Then according to the PRD and interactive documents, according to the UI interface, it is divided into large functional blocks, and then in the large functional blocks, and then in the large functional blocks and then into small functional blocks, and finally to the function point, each function point through the UI, basic functions, Develop a use case design map in dimensions such as boundary, memory, data, interaction, and interface logic, and list doubts that need to be found for product development and confirmation.
Organize use case review after use case design is completed
a. Organize development and product review of test cases, raise questions about use case design, and review and supplement use cases from the perspectives of product implementation, data storage, users, and product experience.
b. It is also very necessary to pre-review the test cases in the test group. Before the formal use case review, the pre-review will be carried out in the group. After the version is over, the full use case collection will be organized and the use cases will be cross-talked, especially for those who are experienced or familiar with the business. Veteran drivers, you can quickly help point out the missing points of the use case in the use case review, which will help testers open their minds and cover as many user scenarios as possible. It is worth noting that if you encounter uncertainties in the use case review, you should record them immediately Take it down as a to-do item, and find relevant personnel to confirm it in time after the end, to avoid guesswork and uncertainty.
Summarize user feedback and improve the test case process – drill down to build test cases to be prepared
a. After the product test is released and launched, for the defects reported by users, if the defects are caused by incomplete scene design, we first analyze whether the scene with the problem is inevitable or occasional. If it is inevitable, we can communicate with technical students , confirm some specific steps to reproduce the scenario, confirm the reason for the introduction, and the solution.
b. If there is a defect in the line, the test case needs to be improved: in addition to supplementing the scenario case, consider some scenarios related to the scenario, improve and review the test cases in a variety of scenarios in a timely manner, and add them to the use case library.
c. Make a review and summary of the specific reasons for online defect analysis, pay attention to the online problem feedback group, find problems in time, locate the problem, analyze the cause, judge whether the problem is caused by the introduction of old logic or new functions, and accurately supplement the corresponding use cases For special scenarios, interface automation, asset loss prevention data dog verification, and full use case collection BVT use cases are supplemented.
2.3 The test phase is not strictly executed according to the test cases
Executing the test according to the test case allows us to avoid missing some test points as much as possible. It is not right to simplify the test cases because of a certain person or familiarity with a certain business, and not strictly follow the test cases to perform the test, so that some missing bugs should not occur.
improvement measures
Test cases may not guarantee that all scenarios and functional points can be covered, but strictly executing tests according to test cases can ensure product quality to the greatest extent and avoid defects as much as possible.
Cultivate the habit of test records: For test blocking use cases and test Fail use cases, you should focus on and record them, and perform precise regression testing in the regression testing phase to ensure that new bugs introduced by related functions can also be found when bugs are fixed.
Although the testing process is very standardized, the software quality is still unsatisfactory.
2.4 The test environment and test resources are limited, resulting in missing defects
The testing environment of Dewu at this stage is extremely complicated. The business system does not exist in isolation, and the related parties are intertwined, and the related systems are often unstable. In addition, it involves scarce resources such as ID cards and bank cards. The use is limited, and a valid data is often discarded after testing a valid data, so we can restore the customer’s actual environmental problems through mocking as much as possible.
After all, the reality is not a real environment. Due to the difference in the environment, many unexpected problems may arise, such as: configuration problems, data source problems, and data synchronization problems. It will be exposed, and it cannot be restored in our test environment. We can only verify the problem based on the pre-release environment or production environment, which may lead to hidden risks in quality.
1) Introduce grayscale release test
The test team conducts regression testing on the pre-release environment, which can basically simulate the real environment to execute use cases that cannot be tested in the test environment without affecting the normal use of online users.
2) Do a good job of case screening in the production verification link
First, sort out the production verification cases. In addition to screening the p0+p1 level cases for regression, the production verification cases should also include test cases where the test environment mock or baffle is blocked, and cases where the back-end interface responds to the front-end. Execute according to the production verification case to cover the real online environment scenario.
3) Strengthen the understanding of back-end and business logic of related parties
The front-end not only needs to understand the interaction business logic of the front-end and back-end interfaces, but also needs to understand the interface interaction logic between the back-end interface and other related parties, verify and judge whether the interface data given by it is correct, and have an overall coverage of test cases in the test environment The degree of control to ensure that the test case coverage of the production environment is comprehensive.
2.5 New bugs introduced by developers
Some developers will only solve the problem described in the bug you submitted, and will not check all the points that may be involved in the problem. It may appear that the problem has been solved, but a new problem has been introduced. A developer who is not familiar with functional modules comes to fix bugs, because he is not familiar with the business, and his inconsideration leads to the unintentional introduction of new bugs.
1) code review
- From the level of code management: develop and fix a bug and submit the code for self-testing. When the development team submits the code for code review, the possibility of introducing new bugs will be small and the risk will be reduced.
2) Accurate regression testing
- From the level of self-cultivation in testing: After development and testing, understand the code change points, accurately analyze the impact of the change points on the associated function points, confirm and verify the bugs fixed by developers, and traverse the associated function points as much as possible regression testing to
3) Talk to the developer about how the developer fixed this function*
- It is easy to talk to the development and implement. From the design of the development, you can grasp the attention points of the test and record them in the use cases. For example, the development of A once made function B in a certain way, and a certain bug appeared. Now that function B is implemented in the same way, it is very likely that the previous bug will appear in function C.
4) Practice and application of coverage
- Increase the code coverage of development and smoke execution. According to the coverage data analysis, which smoke use cases are not covered, whether the method is not covered, or the class is not covered, or the verification of abnormal logic is not returned. Use the development self-test and Coverage reduces the introduction of new bugs.
2.6 Lack of exploratory testing
Many of the bugs we found were not discovered by the execution of test cases, but were found by random testing during the test process, and these steps were not reflected in the test cases, and our test cases could not cover all scenarios.
1) ET test after passing the admission test
- Enter the SIT test stage after the test access test is completed: Generally speaking, ET test is the easiest to find bugs, so after the test access test is completed and enters the SIT test stage, a round of exploratory testing is performed first to eliminate most of the bugs. Expose it in the early stage of testing, let the cumulative number of bugs reach a certain peak, and find bugs as soon as possible, the higher the quality.
2) UAT Intragroup ET testing prior to testing
- The SIT test is coming to an end. Before the UAT test, organize an ET test within the group. Let different tests in the group use different test methods, test thinking, test experience, and test habits to conduct exploratory tests. Tested bugs, weird bugs, or unreasonable usage.
3) Accurate testing
The test case clustering analysis function of the precise test can effectively find “test errors”. For example, if there is an error in the execution steps of a use case, its clustering result will inevitably change. The manager can find and correct this type of error through the results of system analysis. Previously, it may be necessary to return to the site for repeated confirmation.
The core technical point of precise testing is the traceability technology of test cases and codes. Simply put, this technology means that when the function is executed, the corresponding overall code execution will be generated immediately, that is, when a test case is clicked, the corresponding code and module will be tracked immediately.
Accurate testing test vulnerability analysis function, suitable for agile testing. Based on program static data and dynamic operating data, it can automatically analyze the location of the highest risk of software defects, guide the first to complete the coverage of high-risk modules, and complete the coverage test of the most risky modules within a limited time.
3. Thinking from the perspective of development
3.1 Self-test background
It is very necessary and a general trend for developers to do a good job of self-testing. The early stage is the development self-test, and the later stage is the user experience test. In terms of cost and time, the later the bug is discovered, the higher the repair cost; in terms of modification efficiency, the sooner the bug is dealt with, the faster it will be. For an excellent developer, there will be more bugs in the self-test than the bugs found in the test, that is, the number of bugs is quite small when it is the turn of the test.
3.2 Thinking about Difficult Problems
The time and schedule are too tight, and the schedule is tight.
I am too confident in my own code, I think it has strong robustness, and I can’t bear to modify it.
Think it is the responsibility of the test and rely heavily on the test.
I don’t know how to effectively do a good job of self-test and cover it comprehensively.
The development of smoke tests does not have a thorough understanding of the use cases specified by QA creation, and the execution is simple.
3.3 Mindset shift
Code quality and project quality are our responsibility.
Testing is different from how developers think about problems. Development is making software, and testing is breaking software, trying to find out the problem.
Any function has normal scenarios and abnormal scenarios, and most of them use equivalence classes and boundary values to select data, covering comprehensively.
Don’t trust any developed code to be bug-free.
Go out of the development thinking used in the specific implementation, and test whether it passes from the perspective of requirements and users. If you are a user, test your function.
3.4 The pain points and hidden dangers caused by not careful self-test
Omission of requirements: Once users discover this problem, the user impression will be greatly reduced, and they may give up using it directly from the beginning, which will bring about a very large loss of customers.
Functional accident: The main process function is not tested in place, or the abnormal scene is not tested in place, resulting in frequent online error reporting and extremely bad experience, which is directly regarded as an accident.
Delayed launch of requirements: If the self-test is not sufficient, the test will spend a lot of time to communicate low-level bugs, and even the main process cannot go on. This will undoubtedly bring rework, repeated testing, time-consuming, demand delays, project delays, etc. to the development. series of questions.
3.5 Develop self-test report specifications
- Functional module introduction and background introduction
Function, background introduction
Introduction to user groups
version number
Hosts, code release branch
pre-release or official
Functional design documents and UI design drawings, etc.
Database data synchronization, environment configuration, switch setting, etc.
Business points and test points recorded when writing code
Self-Test Points for Requirement Changes
Forward, reverse, abnormal scene test points
compatibility
Will the development of this function affect other functions, and will a line of code cause new problems?
The number of high-level bugs, affecting the core process of smoking
Number of medium-level bugs, serial process links
Number of low-level bugs, page display UI effect
Development Smoke Self-Test Phase Coverage
One round, integration stage coverage
Comply with the standard of test SOP regulations
Smoke self-test and integration stage coverage criteria
Controlling the number of bugs in the testing phase
Control the number of bugs after launch, and the monthly review of quality meets the quantity control standards
Four. Summary
After the missed defect detection occurs, we need to analyze the missed bug in depth, and think about which aspects are not enough. Is it an error in understanding the business logic? Use case evaluation missing? Is the technical solution unreasonable? Is there a deviation in the direction of thinking about design use cases? Ask a few more why, think about the problem from another perspective, and design the evaluation reasonably. Ensure that similar bugs can be prevented from being discovered and exposed in advance, so as to reduce the occurrence of defects as much as possible and improve product quality. To do a good job in the implementation of use case test plans at different stages, and to add refined testing and exploratory testing links, it is necessary to develop new testing thinking and get out of the usual conventional testing thinking. At the same time, it is also necessary to stand on the development side and consider the thinking logic of writing code design to reduce the possibility of bugs and omissions in the testing phase. The development side also needs to strictly implement the self-test and coverage SOP requirements.
*arts /Viki
If you think the article is helpful to you, please comment, forward and like~
#missed #test #bug #Dewu #technologys #personal #space #News Fast Delivery