Test Automation Metrics
If you are tracking the number of the lines of code and the cyclomatic complexity of the test automation code, what type of metric are you gathering?
a. Tool scripting metrics
b. Automation code defect density
c. Trend metrics
d. Number of false fail results
A) Tool scripting metrics
What is the purpose of the test execution report?
a. To provide the details of the test execution at the test case level to project management
b. To provide information on failures so the TAE can address and fix the failures
c. To provide summary and trend information for interested stakeholders
d. To provide feedback to the SUT developers regarding the data used in the tests
C) To provide summary and trend information for interested stakeholders
The test execution report is a summary report showing trends and summarized information for the stakeholders.
A, B and D are examples of information that would be found in the test logs rather than in the execution report.
it is generally agreed that more coverage is better as it reduces overall risk of software deployment.
A) True
B) False
A) True
TRUE or FALSE these are Internal TAS metrics ...
1. Tool scripting metrics
2. Automation code defect density
3. Speed and efficiency of TAS components
TRUE
What would be a good option to provide a good overview of the overall execution results?
a. Test Logs
b. Test automation report
c. Execution report
d.Report generator
B) Test automation report
The logs alone cannot provide a good overview of the overall execution result. For this, it is necessary to have in place reporting functionality
These are measures of benefits of a TAS
a. Savings of time or effort
b. Increase in the amount of testing performed and repeatability
c. Greater use of resources
d. All the above
D) All the above
What is the purpose of the External metrics?
a. External metrics are those used to measure the TAS's impact on other activities, in particular the testing activities.
b. External metrics are those used to measure the effectiveness and efficiency of the TAS in fulfilling its objectives.
c. External metrics are used to monitor the SUT and the functional testing of the SUT
A) External metrics are those used to measure the TAS's impact on other activities, in particular the testing activities.
These are valid reasons for publishing the reports
a. The report should be published for everyone interested in the execution results
b. It can be uploaded on a website
c. It can be sent to a mailing list or uploaded to another tool such as a test management tool
d. All the above
D) All the above
Scripting approach used, familiarity with the test tool, the environment, and the skill level of the test automation engineer .. are factors for the "Effort to build automated tests"
a. TRUE
b. FALSE
A) TRUE
If you are tracking the frequency with which the test automation code reports a defect that is not really a defect, what metric are you gathering?
a. Tools scripting metrics
b. Automation code defect density
c. Trend metrics
d. Number of false-fail results
D) Number of false-fail results
This is an example of a false-fail – when the code indicates that a failure occurred but it wasn’t really a failure.
You have finished the test automation for a financial project. During the automation effort, significant time was spent automating the validation of the report results. This was particularly difficult because the developers used a new tool to create the reports and dashboards and significant coding was required to allow the automation to recognize the fields within the report tables and to verify the appearance of the dashboard.
The developers have now told you they are planning to re-work the reports modules to change the way the reports are rendered and to change the dashboard displays.
Your management wants to know how much time this will require and if it should be done or if that part of the application should not be tested by the automation. You are estimating that the time required will be three days per test script and there are 20 test scripts involved in this testing.
What other information do you need to know if you should make these changes or not?
a. The ratio of failures to defects
b. The number of times the tests will be run over the life of the reporting software
c. The comparison of the 60 days to the EMTE
d. The time required to analyze SUT failures
C) The comparison of the 60 days to the EMTE
Since the options are either to update the automation or resort to manual testing for the reports, you need to know how the 60 days compares to the equivalent manual test effort per execution (and the number of planned executions over the life of the project)
The available logging of the SUT and the TAS play a crucial role in analyzing failures. The logging should provide enough information to perform this analysis efficiently. Important logging features include:
1. SUT logging and TAS logging should be synchronized
2. The TAS should log the expected and actual behavior
3. The TAS should log the actions to be performed
4. The TAS and SUT should log all the actions that are performed
a. 1,2,3
b. 2,1,4
c. 1
d. 1,2,3,4
A) 1, 2, 3
You have been asked to implement test automation for a project that is not meeting its deadlines. After further analysis you discover that the manual testers are not able to keep up with the new feature testing because the regression testing is taking 75% of their time. As a result, the new features are being released with many defects and customers are complaining about the quality.
Given this information, what metric should you be tracking to show the value of the test automation for this project?
a. Percentage of code covered by the test automation
b. Equivalent manual test effort for the automated tests
c. Number of defects found by test automation
d. Percentage of build accepted/rejected by the automated tests
B) Equivalent manual test effort for the automated tests
Because regression testing is taking the most time from the manual testers, automating the regression tests should be the first task for the test automation.
You have been running a test automation suite on a SUT and it is failing to run to completion. You have tried this five nights in a row and it has failed each night. Unfortunately, the logs are deleted when the test fails, so you have no idea what is causing the failure. The one clue you have is that the SUT is logging “out of memory” errors.
You have found and corrected the problem that was causing the TAS logs to be deleted. You have the ability to log the following information.
1. The start and stop time of each test
2. The memory reading from the SUT at the start. and stop of each test
3. The test case that is currently executing
4. The status of each test case that is executed
5. The number of cycles that have been completed for each test run
6. The random data used by each test case
7. The screenshots from every successfully completed test case
What information do you need to be sure is being logged in order to help identify the problem?
a. 1,2,4,5
b. 2,3,4,5
c. 2,3,6,7
d. 1,2,6,7
B) 2,3,4,5
- You need to know the memory reading at the start and stop of each test, you need to know which test is currently executing (to see which one is failing)
- you need to know which tests are passing and failing and you need to know if multiple cycles of the same tests are being executed (since the problem might only occur during the 3rd cycle).
- 1 is not needed because the start and stop time of each test is not helpful for figuring out the memory problem. 6 is not needed because there is no indication that random data is being used.
- 7 is not needed and would take up a lot of space because a screen shot is unlikely to be helpful for every failed case. If you could get a screen shot of just tha last one, that would be more interesting particularly if there is an error message displayed, but once the failling case is defined, more instrumentation for that test could be added.
Differences in the time it takes to perform the same test steps in the same environment can indicate a problem in .. ?
a. SUT
b. TAS
c. SUT and TAS
d. None of the above
A) SUT
If the SUT is not performing the same functionality in the same elapsed time, investigation is needed.