Test Progress Monitoring and Control

hello everyone welcome to session 3 of module 5 test progress monitoring and control in this session we will learn about test progress monitoring and several control activities so what we will learn we will learn about the techniques and metrics commonly used for monitoring test preparation and execution then we will see about test reporting using test matrix and the main topics that we will cover in this session will be monitoring the progress of test activities then reporting the test status and finally how to control the testing activities if something goes wrong in the test process so monitoring the progress of test activities after defining test strategy and plan once you start the test execution it is very necessary that you track what you are doing what execution is going on on day to day basis what you are trying to achieve eventually so tracking is very important once you start your execution so it's not necessary it's it's not only specific to the test execution phase tracking is kind of important in the complete software testing life cycle while you are designing your test case while you are developing the test case while you are doing execution tracking of activities is very important in each and every phase but during test execution this has its own importance because during test execution you are you are collecting all the execution data based on the test passed how many defects raised how many defects got faced so this all this data needs to be um tracked so that you can report to the stakeholders what the exact outcome of the test execution is for that specific phase so uh test monitoring is a test management task which tracks the test progress and keeps an eye on how testing is progressing so this test monitoring activity is a test management task which basically checks how the test is progressing how the test execution is going on how everything in the test life cycle is going on the reports are periodically prepared that compare the actual result with the plan so in the test monitoring there are continuous reports being generated which compare the plan with the actual so that you can find out if there are any deviations from planned versus actual result so test progress monitoring can be done based on different metrics like test executions done how many test cases have been executed how many among those passed and how many among those failed or it can be done based on defects how many defects open closed or in progress so test progress monitoring these are kind of metrics that you can capture while test pro test execution is going on so based on test cases based on defects raised you can collect all these matrix and once you are reporting the test summary report you can com you can combine all these metrics into a report to give a fair bit of idea about the test progress to the stakeholders the test progress monitoring technique varies depending on the preferences of testers and stakeholders depends on needs and goals of the project or the regulatory requirements so test progress monitoring technique will vary depending on what the stakeholders want what they want to see in the test progress report or test test summary report uh if there is any regulatory requirement that needs to be fulfilled then your test progress report or test summary report will be different for that so this this test progress monitoring technique depends on certain factors so what is the purpose of test monitoring um test monitoring provides feedback to test team and test manager how the testing is being progressed how testing is progressing in the software testing life cycle so your test monitoring activities give feedback to the whole team where we stand today as as in software testing life cycle while the test execution or all test activities are being progressed as well as it gives fair bit of idea to test manager and other stakeholders as well it provides visibility to project team about test results so other purpose of test monitoring is that it provides visibility to project team about the testing that is that is done in the project then it measures the status of test execution against the exit criteria defined in test plan the other purpose of test monitoring is that it measures the status of test execution against the exit criteria so you monitor the test progress in stlc and then monitor whether that progress has fulfilled the exit criteria whether the criteria that is mentioned as an exit criteria in your plan has been met with the um test execution with the status that that that you get after doing test monitoring activities then you're gathering data for use in future test estimations so that's another purpose of test monitoring you gather the test monitoring data which is used in further test estimations so suppose you did some estimation in in release a you did some test estimation for for doing a test execution of 100 test cases you did an estimation of 30 days uh and you were able to finish all those 100 test cases in less than 20 days so you the estimation was not proper in this release i mean you you were able to do the execution well in advance as compared to the estimated time so this data that you have got after test monitoring that you were able to finish the testing 10 12 days in advance can be used in future estimation future test estimations of further test cycles so test progress monitoring metrics so common metrics used for test progress monitoring are total test cases prepared that's the very common metric then how many test cases have been run till now how many test cases have been passed and how many have been failed so these are these are some of the matrix which you you capture on day-to-day basis in software testing life cycles once test execution starts or um the testing activities start you capture these metrics on day-to-day basis um then you you capture total number of incidence rays in the test cycle uh how many of them have been fixed what was the planned effort for suppose executing 100 test cases and what is the actual effort that you have put to execute those 100 test cases to capture all these metrics then what is the percentage of test environment completed suppose um there there will be some effort required to uh complete the test environment setup where you will do the test execution so you can cap you can calculate that completion into percentage that's another matrix then extent of the test coverage achieved in terms of requirements or risks or code etc so the test coverage in terms of requirement or in terms of risks or in terms of code coverage that you have achieved with the test cases that you are executing then what is the status of testing um for example test analysis design and implementation so all this status is also monitored by test progress monitoring matrix how much test plan has been created how much analysis has been done how much uh design has been complete and how many test cases have been documented or implemented till now uh and how much more time is required to document all the test cases um then any subjective level of confidence testers have on the project product so that's another um metric that can be used uh to make the release decisions basically uh so for example if tester says that yes he is confident of the quality of the software that's that's his own um level of confidence that he can give to the stakeholders then stakeholders um also can take that matrix as as a decision making point to do a release or not then reporting test status so reporting test status is about effectively communicating test monitoring findings to other project stakeholders so once you find uh the test monitoring matrix then you just do not make a note of those matrix and keep it you have to report those matrix in a proper way to the stakeholders so that they can understand what's going on in the test life cycle so reporting the test status is about communicating those monitoring metrics effectively to the project stakeholders then test status reports help project stat project stakeholders to understand results of test cycle and ensure if exit criteria was met so based on the test status report that is provided to project stakeholders they understand they are able to understand what the status of the test cycle is and whether exit criteria has been met or not then it also helps stakeholders to to decide the project release decisions so based on the test status report stakeholders make release decisions for the project or the software if the test status report or test summary report says that exit criteria was met that that was defined initially and every all of the points in exit criteria have been met and it is good to go then stakeholders can make decision to release the product in the market if the exit criteria has not been met based on the test summary report then they they will delay the release decision uh then ieee 829 standard test summary report template so there is there is a ieee 829 standard test summary report template which includes um test summary report identifier um then what a bit of summary about the test summary a bit of summary about the product project and any variances then um the comprehensive assessment about the project and summary of results what results have been seen while in all the test life cycle or in test execution cycle then evaluation based on the results and then summary of activities and finally the approval of the document so this is this is test basic test summary report ieee 829 test summary report template so based on different needs and requirements of the organization so organizations have modified test summary report according to their needs and requirements so there are even one page summary test summary report because stakeholders do not want to read all 50 pages test summary report document so it depends what kind of software you are testing if it is it has to follow all the standards and it has to follow all the process then you might be required to deliver a 20 30 page test summary report based on the complexity of the project but if it's a simple project then stakeholders might want a single page summary report which just contains um the number of test cases uh total number of test cases how many passed failed the number of defects raised how many were fixed how many are still in progress whether the exit criteria was met or not so based on certain monitoring findings you can have a test summary report which which suits best for the project requirements but the standard test summary report is the one that is from ieee 829 and let's see an example um so in case of risk-based testing major criteria for test reporting is to subject the important product risks with appropriate extent of testing so if you are doing a risk-based testing so what will be the major criteria for test reporting the major criteria for test reporting will be that you subject uh the product risk with appropriate extent of testing so if feature a was the highest risk feature in the product then that feature you you have to show in your coverage report that that feature has been tested thoroughly with number of how many number of test cases and what percentage of defects have been fixed for that feature so in this table if you see so suppose if you're doing a risk-based testing you can have a template and divide the functionality product risk areas based on the uh risk so suppose functionality uh performance security interfaces compatibility so based on the risks you have identified these are the high risk areas and you have planned some test cases you have some planned test cases and then how many actual test cases have been run what is the percentage of test cases still that need to be executed and what are the outstanding defects against these functionalities so this this risk-based testing report a simple report or table will help the stakeholders to decide what is the residual risk in the product what is the residual risk in the different risk areas or different modules of the product so test control is about bringing the project under control in case of any diversions so you have done the monitoring you have done the reporting so after doing a monitoring activity so you just you just do not keep monitoring in in the test life cycle if something is going wrong in the life cycle there has to be some control which can control these diversions in the test life cycle so test control is all about bringing the project testing under control in case of any diversions in the project test control is a test management tasks in which you apply corrective actions to get the test project on track so in case something is going wrong in the project you are not able to achieve what you plan for then you take corrective measures so that you can deliver what you plan for within that specified within the planned date or plan schedule that that you had initially then examples of test control activities so for example um there is a there is a e-commerce website which is uh which when initially launched there was a set of performance tests that was being executed every night because there was no traffic or activity during night for that e-commerce portal but suddenly it became popular and there a lot of end-users users started using that website 24 um on all five days and 24 hours so in in that case you cannot make you cannot have a downtime for the site during the weekdays because it will affect the revenue of the company so in in such scenarios you need to reschedule the testing to some other time slot which is free or when the traffic is minimal so initially performance testing was executed every night um in the weekdays so resha dealing the test to weekends solve this issue so that's one control activity that can sort out these issues because you have to do performance testing you cannot avoid it but you need to reschedule the testing to some other time slot in case uh there is some issue with particular sign time slot then suppose if there are there are defects there are a lot of defect fixes that you are getting from the development team and those defects when they are being retested are found that they are not actually fixed and the problem still exists so in this case rather than developers fixing those defects and directly coming back to testers and again again getting back uh opened the best thing is to taking measures so that developers themselves verify those defects first um thoroughly before releasing it into the test environment so that will save a lot of test effort and time in retesting those defects and again reopening and so it will save a lot of that effort then suppose um there was um there was a module that was expected to be released today but somehow that module has been delayed and module xyz has been successfully developed and is ready to be delivered and can be delivered today so re-prioritizing test execution so suppose test team was supposed to start execution of module a today but module a has been delayed instead of module a there is another module that has been the coding of which has been finished and can be delivered today so you can take that module test team can take that module and re-prioritize the test execution for the module that is ready today and can push the test execution of the delayed module further then in case release date cannot be moved and lots of tests need to be executed then you need to have more resources to get the testing done so in case there is an extreme time pressure and release cannot be moved release date cannot be moved then you need more resources that's another control activity that you can take to release the software within time frame so these are all these are some of the test control activities that are required to bring the project under control in case of any diversions so to conclude in this session we learn about test progress monitoring and control we learn about monitoring the test progress um of testing activities then we learn about reporting test status and the standard template ieee 829 template for reporting test status and then finally we learn about the controlling of test activities in case of any deviations from planned activities thank you

hello everyone welcome to session 3 of module 5 test progress monitoring and control in this session we will learn about test progress monitoring and several control activities so what we will learn we will learn about the techniques and metrics commonly used for monitoring test preparation and execution then we will see about test reporting using test matrix and the main topics that we will cover in this session will be monitoring the progress of test activities then reporting the test status and finally how to control the testing activities if something goes wrong in the test process so monitoring the progress of test activities after defining test strategy and plan once you start the test execution it is very necessary that you track what you are doing what execution is going on on day to day basis what you are trying to achieve eventually so tracking is very important once you start your execution so it&#39;s not necessary it&#39;s it&#39;s not only specific to the test execution phase tracking is kind of important in the complete software testing life cycle while you are designing your test case while you are developing the test case while you are doing execution tracking of activities is very important in each and every phase but during test execution this has its own importance because during test execution you are you are collecting all the execution data based on the test passed how many defects raised how many defects got faced so this all this data needs to be um tracked so that you can report to the stakeholders what the exact outcome of the test execution is for that specific phase so uh test monitoring is a test management task which tracks the test progress and keeps an eye on how testing is progressing so this test monitoring activity is a test management task which basically checks how the test is progressing how the test execution is going on how everything in the test life cycle is going on the reports are periodically prepared that compare the actual result with the plan so in the test monitoring there are continuous reports being generated which compare the plan with the actual so that you can find out if there are any deviations from planned versus actual result so test progress monitoring can be done based on different metrics like test executions done how many test cases have been executed how many among those passed and how many among those failed or it can be done based on defects how many defects open closed or in progress so test progress monitoring these are kind of metrics that you can capture while test pro test execution is going on so based on test cases based on defects raised you can collect all these matrix and once you are reporting the test summary report you can com you can combine all these metrics into a report to give a fair bit of idea about the test progress to the stakeholders the test progress monitoring technique varies depending on the preferences of testers and stakeholders depends on needs and goals of the project or the regulatory requirements so test progress monitoring technique will vary depending on what the stakeholders want what they want to see in the test progress report or test test summary report uh if there is any regulatory requirement that needs to be fulfilled then your test progress report or test summary report will be different for that so this this test progress monitoring technique depends on certain factors so what is the purpose of test monitoring um test monitoring provides feedback to test team and test manager how the testing is being progressed how testing is progressing in the software testing life cycle so your test monitoring activities give feedback to the whole team where we stand today as as in software testing life cycle while the test execution or all test activities are being progressed as well as it gives fair bit of idea to test manager and other stakeholders as well it provides visibility to project team about test results so other purpose of test monitoring is that it provides visibility to project team about the testing that is that is done in the project then it measures the status of test execution against the exit criteria defined in test plan the other purpose of test monitoring is that it measures the status of test execution against the exit criteria so you monitor the test progress in stlc and then monitor whether that progress has fulfilled the exit criteria whether the criteria that is mentioned as an exit criteria in your plan has been met with the um test execution with the status that that that you get after doing test monitoring activities then you&#39;re gathering data for use in future test estimations so that&#39;s another purpose of test monitoring you gather the test monitoring data which is used in further test estimations so suppose you did some estimation in in release a you did some test estimation for for doing a test execution of 100 test cases you did an estimation of 30 days uh and you were able to finish all those 100 test cases in less than 20 days so you the estimation was not proper in this release i mean you you were able to do the execution well in advance as compared to the estimated time so this data that you have got after test monitoring that you were able to finish the testing 10 12 days in advance can be used in future estimation future test estimations of further test cycles so test progress monitoring metrics so common metrics used for test progress monitoring are total test cases prepared that&#39;s the very common metric then how many test cases have been run till now how many test cases have been passed and how many have been failed so these are these are some of the matrix which you you capture on day-to-day basis in software testing life cycles once test execution starts or um the testing activities start you capture these metrics on day-to-day basis um then you you capture total number of incidence rays in the test cycle uh how many of them have been fixed what was the planned effort for suppose executing 100 test cases and what is the actual effort that you have put to execute those 100 test cases to capture all these metrics then what is the percentage of test environment completed suppose um there there will be some effort required to uh complete the test environment setup where you will do the test execution so you can cap you can calculate that completion into percentage that&#39;s another matrix then extent of the test coverage achieved in terms of requirements or risks or code etc so the test coverage in terms of requirement or in terms of risks or in terms of code coverage that you have achieved with the test cases that you are executing then what is the status of testing um for example test analysis design and implementation so all this status is also monitored by test progress monitoring matrix how much test plan has been created how much analysis has been done how much uh design has been complete and how many test cases have been documented or implemented till now uh and how much more time is required to document all the test cases um then any subjective level of confidence testers have on the project product so that&#39;s another um metric that can be used uh to make the release decisions basically uh so for example if tester says that yes he is confident of the quality of the software that&#39;s that&#39;s his own um level of confidence that he can give to the stakeholders then stakeholders um also can take that matrix as as a decision making point to do a release or not then reporting test status so reporting test status is about effectively communicating test monitoring findings to other project stakeholders so once you find uh the test monitoring matrix then you just do not make a note of those matrix and keep it you have to report those matrix in a proper way to the stakeholders so that they can understand what&#39;s going on in the test life cycle so reporting the test status is about communicating those monitoring metrics effectively to the project stakeholders then test status reports help project stat project stakeholders to understand results of test cycle and ensure if exit criteria was met so based on the test status report that is provided to project stakeholders they understand they are able to understand what the status of the test cycle is and whether exit criteria has been met or not then it also helps stakeholders to to decide the project release decisions so based on the test status report stakeholders make release decisions for the project or the software if the test status report or test summary report says that exit criteria was met that that was defined initially and every all of the points in exit criteria have been met and it is good to go then stakeholders can make decision to release the product in the market if the exit criteria has not been met based on the test summary report then they they will delay the release decision uh then ieee 829 standard test summary report template so there is there is a ieee 829 standard test summary report template which includes um test summary report identifier um then what a bit of summary about the test summary a bit of summary about the product project and any variances then um the comprehensive assessment about the project and summary of results what results have been seen while in all the test life cycle or in test execution cycle then evaluation based on the results and then summary of activities and finally the approval of the document so this is this is test basic test summary report ieee 829 test summary report template so based on different needs and requirements of the organization so organizations have modified test summary report according to their needs and requirements so there are even one page summary test summary report because stakeholders do not want to read all 50 pages test summary report document so it depends what kind of software you are testing if it is it has to follow all the standards and it has to follow all the process then you might be required to deliver a 20 30 page test summary report based on the complexity of the project but if it&#39;s a simple project then stakeholders might want a single page summary report which just contains um the number of test cases uh total number of test cases how many passed failed the number of defects raised how many were fixed how many are still in progress whether the exit criteria was met or not so based on certain monitoring findings you can have a test summary report which which suits best for the project requirements but the standard test summary report is the one that is from ieee 829 and let&#39;s see an example um so in case of risk-based testing major criteria for test reporting is to subject the important product risks with appropriate extent of testing so if you are doing a risk-based testing so what will be the major criteria for test reporting the major criteria for test reporting will be that you subject uh the product risk with appropriate extent of testing so if feature a was the highest risk feature in the product then that feature you you have to show in your coverage report that that feature has been tested thoroughly with number of how many number of test cases and what percentage of defects have been fixed for that feature so in this table if you see so suppose if you&#39;re doing a risk-based testing you can have a template and divide the functionality product risk areas based on the uh risk so suppose functionality uh performance security interfaces compatibility so based on the risks you have identified these are the high risk areas and you have planned some test cases you have some planned test cases and then how many actual test cases have been run what is the percentage of test cases still that need to be executed and what are the outstanding defects against these functionalities so this this risk-based testing report a simple report or table will help the stakeholders to decide what is the residual risk in the product what is the residual risk in the different risk areas or different modules of the product so test control is about bringing the project under control in case of any diversions so you have done the monitoring you have done the reporting so after doing a monitoring activity so you just you just do not keep monitoring in in the test life cycle if something is going wrong in the life cycle there has to be some control which can control these diversions in the test life cycle so test control is all about bringing the project testing under control in case of any diversions in the project test control is a test management tasks in which you apply corrective actions to get the test project on track so in case something is going wrong in the project you are not able to achieve what you plan for then you take corrective measures so that you can deliver what you plan for within that specified within the planned date or plan schedule that that you had initially then examples of test control activities so for example um there is a there is a e-commerce website which is uh which when initially launched there was a set of performance tests that was being executed every night because there was no traffic or activity during night for that e-commerce portal but suddenly it became popular and there a lot of end-users users started using that website 24 um on all five days and 24 hours so in in that case you cannot make you cannot have a downtime for the site during the weekdays because it will affect the revenue of the company so in in such scenarios you need to reschedule the testing to some other time slot which is free or when the traffic is minimal so initially performance testing was executed every night um in the weekdays so resha dealing the test to weekends solve this issue so that&#39;s one control activity that can sort out these issues because you have to do performance testing you cannot avoid it but you need to reschedule the testing to some other time slot in case uh there is some issue with particular sign time slot then suppose if there are there are defects there are a lot of defect fixes that you are getting from the development team and those defects when they are being retested are found that they are not actually fixed and the problem still exists so in this case rather than developers fixing those defects and directly coming back to testers and again again getting back uh opened the best thing is to taking measures so that developers themselves verify those defects first um thoroughly before releasing it into the test environment so that will save a lot of test effort and time in retesting those defects and again reopening and so it will save a lot of that effort then suppose um there was um there was a module that was expected to be released today but somehow that module has been delayed and module xyz has been successfully developed and is ready to be delivered and can be delivered today so re-prioritizing test execution so suppose test team was supposed to start execution of module a today but module a has been delayed instead of module a there is another module that has been the coding of which has been finished and can be delivered today so you can take that module test team can take that module and re-prioritize the test execution for the module that is ready today and can push the test execution of the delayed module further then in case release date cannot be moved and lots of tests need to be executed then you need to have more resources to get the testing done so in case there is an extreme time pressure and release cannot be moved release date cannot be moved then you need more resources that&#39;s another control activity that you can take to release the software within time frame so these are all these are some of the test control activities that are required to bring the project under control in case of any diversions so to conclude in this session we learn about test progress monitoring and control we learn about monitoring the test progress um of testing activities then we learn about reporting test status and the standard template ieee 829 template for reporting test status and then finally we learn about the controlling of test activities in case of any deviations from planned activities thank you

Transcript for:Test Progress Monitoring and Control

Transcript for:
Test Progress Monitoring and Control