How is Agile Testing Different
With the Agile approach, testing is not just the end of the waterfall, but the planning and execution of testing takes place continuously. In addition to concepts such as automated testing, User Acceptance testing and Integration testing must happen on an ongoing basis. The responsibility of testing rests will development teams as well as the User Acceptance teams. However, UAT is responsible for providing ongoing feedback about the software and its functionality. therefore E2E UAT testing must built into the development process and must be done at intervals as the software implementation is underway. End to end User Acceptance Testing get complicated because even though the development teams do a great job of unit and functional testing, integration is sometimes not tested as extensively, and at times the real end to end is done for the first time, by the user acceptance testing. Given a compressed timeframe for UAT, and pressure to meet release dates, combined with pressure to do the testing faster and more cost effectively, add to the UAT complexity.
Types of Testing We Do
At Strategic Business Decisions we bring strong capabilities and expertise in testing. Following is an overview of types of testing that we perform:
A System is made of components and modules that may include a software platform, client operating system, and other tools and components. System testing verifies the interaction between the various components that constitute the system with a focus on identifying interface defects. System tests are used to fully integrate apps and ensure effective interaction between components. End to end testing scenarios are used to thoroughly test that inputs in the app are tested for the desired outputs.
End to End Testing
End-to-end testing determines if the flow of information in an application works from beginning to the end. end-to-end testing reveals system dependencies and ensures that information integrity between systems and applications is maintained. The complete application is tested for critical functionalities such as communicating with the other systems, interfaces, database, network, and other applications.
User Acceptance Testing (UAT)
UAT or end-user testing is generally the last phase of the software testing process. It’s intended to ensure that the software/portals can perform the required tasks according to approved functional specifications. UAT sign-off is required prior to software being generally available and rolled out to the market. User Acceptance Testing determines if the software meets the specifications laid out in the requirements and complies with the business needs and can be delivered to the end users free of defects. The acceptance test cases are executed with test data and outcomes are compared against expected outcomes. The acceptance criteria include functional completeness, data integrity, user experience, security, and readiness. User acceptance tests are implemented in phases. The initial stage covers the happy path and if the initial scenarios pass, then the test team will execute more complex test cases. Delays is development of software can delay or compress the UAT period.
Test Project Planning
Good planning is critical to a successful UAT. A good understanding of the UAT scope and the high level scenarios is key to a successful UAT execution. Key topics covered during planning include scope, tiemlines, key scenarios, high level deliverables, and overall staffing and skills needs for the testing.
The test plan can help get the team get a consistent level of understanding. It’s also a means to get an understanding of high level risks and potential issues that can impact UAT such as code completion and development team deliverables and test environment readiness.Test plans include the test objectives, testing scope and test schedule plus resources required for carrying out the testing. Each test effort requires a readiness checklist that will cover all the test team requirements.
Design for Testing
Design for testing involves understanding the testing scenarios, developing test cases, designing test methods, defining testing pre-requisites, defining detailed test steps, linking test cases to functional scenarios and development deliverables, deploying test tracking artifacts (for example Visual Studio) for tracking each test case.
Defining test scenarios is key to testing activity. Scenarios vary in complexity. In general, a scenario covers a business issue that the software is attempting to solve. For example, a scenario can be “customer purchases an online online”. To evaluate if this is a suitable scenario, ask the following question: Does the scenario cover the business issue being addressed? Is it easy and straight forward to evaluate the outcome of the scenario? Is it clear when the objective is achieved? Can we discern the persona(s) getting to the expected outcome? Can we verify that the expected outcome has been achieved? A test scenario generally covers a number of test cases. From a testing perspective, test scenarios are developed based on functional and business requirements provided by the development team and the business stakeholders.
Each test case is made up of one or more testing steps or testing activities along with expected outcomes.
The testing environment consists of core platform software, portal software, database servers, API, networks, systems and applications, client operating system, browsers, and hardware, etc. Testers execute the test cases on this environment. In many large organizations the testing often happens on a production environment with test flights. Test flights manage the flow of information, make the new features visible, and prevent leakage of information into production systems or other business critical systems. Some of the testing may also take place in test or pre-prod environments that must seamlessly integrate with the production environment during the testing phase. At times, unforeseen delays due to test environment readiness, whitelisting, test flights, etc. may impact or impede testing progress.
Test outcomes are always documented. During test execution, the actual outcome is compared with the expected outcome and defects or deviations are filed as a bug. Based on the test outcome the test case can pass or fail. A test case includes test steps and the expected outcome for each test step. Test results are recorded as screenshots, and detailed defect logs, capturing detailed information to reduce debugging time.
Acceptance Test Report
Daily and weekly test reports include summary of test results, defects identified, variations from expected results, recommendations for path forward and approval decision criteria.
During test execution, the test plan and the test cases meet the reality of what’s being developed. Some of the test cases may no longer be applicable because of timing of delivering some functionality. Also, some of the functionality may be delivered in a different manner due to technical debt and therefore require adjustment to the test plans and test scenarios and the test cases.
Bug tracking and triage requires a number of different teams to be involved. Bug triage meetings require attendance of the right resources and executive visibility to drive bug resolution.
Test Progress Tracking and Reporting
The project dashboard must be comprehensive yet easy to understand and include actionable insights for the development teams and for management.
Testing Reporting: The project is kept on track with timely reports to monitor the results of testing through test execution dashboard that deploys key performance indicators and metrics to monitor and report on the progress of operations verification efforts and comparison between project current and historical results
- Pie charts showing active, pass & fail
- Line charts showing progress against plan
Tracking bugs filed, bugs in progress and bugs fixed
Daily and weekly communications and updates to project teams and senior management
Updates and bug triage
- Updates on high level project progress via PowerPoint decks
Some of the ongoing administrative activities involved include:
- Deploy, update and manage SharePoint site Documentation Library
- Maintain test change management logs
- Conduct daily reporting and weekly project audits to ensure test efforts are running efficiently, effectively and on-plan
- Utilize appropriate project execution tools
- Organize, document, and preserve all key test results and operations (Visual Studio, SharePoint, Excel, Power BI and other tools)
Test appraisal is the process of ongoing gating of the testing execution. At each stage of testing there are different criteria that must be considered. The start of testing depends on completion of the development and stability of the code and the test environment. Without a stable environment it’s difficult to find valid test cases and the development team time will be wasted on figuring out which bugs are valid and which are due to the state of the code. Once testing starts, there are some low hanging fruit that can help build momentum. At the same time some of the more complex test cases will require additional set up that is more time consuming and are required to set up other test cases. The number and quality of the bugs being filed is an indicator of the quality of the testing. Without good and relevant bugs that are covering the key areas of development and functionality it’s impossible to know if key issues are being missed and if the test team is going deep enough to find critical issues. As the end of UAT draws closer, reviewing sign off criteria are key to understanding if the end of UAT tmelines can be met and if the business can sign-off on UAT.
Project resources are onboarded and trained to act as SME on specific project topics under their domain. Project resources will need some onboarding to gain mastery of the apps being developed and the features being released. Project resources are responsible to meet key testing milestones, provide weekly updates to Program Management Office on progress being made. Teams provide updates on test scenario execution, update test execution logs, capture and document step-by-step test execution activities and identify and document risks and issue logs. The test resources along with the program management office identify mitigation solutions. Test resources must maintain detailed and high quality documentation at all time. It’s key that test resources build positive trustful relationships with all stakeholders.
Test Program Management
To make testing successful, good program management methodology implementation will support the ability of the team to scale. The importance of program management in UAT is often overlooked and as a result UAT can get affected negatively. Like any other effort, successful UAT requires adherance to good program management practices: good planning, strong execution and project management, resource management and coaching, knowledge sharing and knowledge management. The test team will work with Program Managers to define and document end-to-end testing scenarios. UAT efforts also depend on team work and leadership. Without leadership the team work across UAT teams will break down and teams start to compete instead of collaborating.
A key role of program manager is to provide daily and weekly status communications to various stakeholders including project stakeholders, development teams, business stakeholders and other key stakeholders as needed. The role of the program manager is to respond to ad hoc requests if needed and identify opportunities for improvement.