Environment Suitability
Is my baseline for all my lower lane environments?
Environment Suitability should be completed for your “production minus one”
environment, where you approve changes prior to production deployment.
What is required in a test plan?
All test plans must include:
Test Type (Feature, Integration, Regression, Performance, Acceptance) story based in JIRA
Test Scope (feature type only)
Acceptance Criteria required to be scenario or rules based- Applicable for all test type
Testing Dependencies people, data, and environment
Impacted Applications
Testing Tool, if applicable
What are mandatory fields in Test results?
Test results are the output of the testing done for each feature story (if applicable) in your release.
Work Item ID: JIRA story ID defines traceability between JIRA stories and test evidence
Test Case ID: unique test case identifier
Test Case Name: High level name/description of what the test case is covering
Test Scripts: only required when not using an approved Test Case Management Tool (ALM, qTest)
Execution Date/Time: Test execution timing of final test result, which should occur after your test plan completion
•Result: must be “PASS” or “FAIL”
•Defect ID (Optional) Failed test cases do not require defect linkage but can be populated where needed.
•Test Tool (Optional) what tool was used to document test results
•Limitation ID (Optional) if your test is used for env. suitability, you can add the limitation here for traceability.
Review the work items (stories and bugs) in the release container how is it different from what we review today for stories and bugs?
OTA will now have visibility into the components of the test plan including acceptance criteria, no testing justification, environment suitability limitations, etc. which are required elements for you to review as part of your approval.
Can release key be tagged to multiple NM CRQs
No. NM CRQ should always be 1:1 with release key
How many days do we have to upload environment suitability execution results?
Once you have certified in AppHQ, by selecting the CERTIFY button, you have 21 business days to upload your Environment Suitability Test Suite execution results to the Global Change Policy Artifact Repository (GCP AR)
What is the test plan for project when we are not testing
You will still be required to have each of the 4 testing types feature, performance, user acceptance, and regression (integration is optional). You can identify within the test type that it is out of scope of the release.
How to determine Test Evidence Traceability Dedicated Stories?
Test Planned Field (Yes, No) will determine when test evidence is required for Dedicated Testing Stories.
•If planned, evidence should be captured.
•If not planned, no evidence required.(proper justification will be required)
Will OTA need to review Environment Suitability limitation?
Yes. Today you identify if there are test environment differences, including a risk statement. Going forward, you will review the environment suitability limitations identified for the applications with the release container scope. (Data pulled form AppHQ based on impacted AITs) .
Review with the testing team to determine if any new limitation were identified as part of testing and request application owner to ensure the new limitation is document in AppHQ
$$$$DOUBLE JEOPARDY$$$$
0
(Clue: Think of Recent Incidents)
Points scored by Giants against Dallas Cowboys in opening game of 2023 NFL season :)
What are the 3 categories for environment suitability baseline?
Functional
: This is the app’s main functionality and what you generally test before deploying. For example, login, enter a trade, view statements.
Data
: This is data not naturally present or creatable that is needed for testing. For example, point of service generated credit card transactions.
Infrastructure
: These are differences in the test environment that keep you from testing functionality (like the example below)
For infrastructure, the test environment is always different from production. Do I need to list all infrastructure differences?
You are not required to list all infrastructure differences such as server counts, networking setups, etc. Infrastructure limitations should be listed if they will result in the inability to sufficiently test in your lower lane environment
What are the 2 primary Orientation for Acceptance Criteria
Scenario Oriented Acceptance Criteria
Derived from behavior driven development, this format purposes to describe how a function or general application task operates from a user perspective. Example: Given, When, Then
Rule Oriented Acceptance Criteria
Works as a check list for criteria and leaves no room for ambiguity. The practitioner would provide the user story and then the “checklist” of rules that it needs to pass. Example: Step 1, Step 2, Step 3
Test Evidence Traceability Feature Stories.
What are the applicable test scope values in feature story where evidence will be required?
Dev + Test
•Default option: When development and testing of the story are completed as apart of the work for that story. Evidence will need to be provided at the story level.
Test Only
•When no development work is required in the story.
This should not be used as a dedicated testing story. Example: when an upstream application makes a code change that requires downstream applications to test within sprints
How will OTA know what type of testing is in scope?
The previous view showed testing by project (PPRT ID) and non material work efforts. This view shows testing by each individual dedicated test story. Summary of , all 4 testing types or an explanation for out of scope will be available under Release Level Summary
Can we do OTA once CRQ is scheduled?
Yes As long as it is before CRQ start date
When should these test cases be executed?
and
Do I have to certify before my next release?
Environment Suitability test cases must be run no more than 3 months prior to certification due date
No. Certification is part of the AppHQ process. Since AppHQ requires certification every 180 days, your first certification will occur in the next 6 months depending on your application’s last certification date.
What are the various test scope applicable to a feature story?
Dev + Test
•Default option: When development and testing of the story are completed as part of the work for that story. Evidence will need to be provided at the story level.
Test Only
•When no development work is required in the story. This should not be used as a dedicated testing story. Example: when an upstream application makes a code change that requires downstream applications to test within sprints
Testing Deferred
•When only development is completed and testing of the story will be done later. Testing Deferred Story will need to be linked, via the Related To field, to another story with testing.
No Testing
•When only development is completed and no testing is required. If no testing takes place, justification will be required.
What are the different ways to upload evidence into Landing Zone?
•Manual using upload template
•Manual edit on screen
•Automated test case management tool
How is this NEW MRA process different than what we review \approve today for High Severity Defects?
It isn’t. You will continue to review high severity defects the same way you do today. If any open high severity defects are included in the scope of the release container, you will identify whether the stakeholder(s) is aware and if an available workaround exists.
Can a non Material change be tagged to a PPRT?
No. PPRT is normally tied to DEP event which is applicable only for Material changes
The Environment Suitability (ES) Suite must include test cases covering these?
High Level Application Regression Test
Exercises all major functions, including data requirements associated with each.
Interfaces to other applications or services
If not covered above, exercises interfaces to external applications or services required for proper functioning.
Previously mitigated environment limitations
If not covered already above, known limitations that have been closed, to provide evidence that mitigations are working as expected.
How is environment suitability linked with test plan activities
You will identify applications in scope for testing while completing the test plan, using the impacted application field.
The impacted application will also define the Environment Testing Dependency , which identifies whether integration across LLEs for multiple applications and/or other systems is needed to achieve successful testing.
In the test plan tab for dedicated test stories, you will confirm you have reviewed the Environment Suitability limitations for all applications involved in testing of that story before moving status to Ready for Testing
Can the OTA sign off without all test evidence in Landing Zone for the release?
No all test evidence is now required before OTA can be complete. Users can upload their evidence at any time into the Landing Zone.
What is OTA Agreeing to while providing approval?
The Change Objectives have been realized and demonstrated via testing outcomes (testing acceptance/Success criteria) for all applicable test types. OTAs have evaluated the accuracy and adequacy of testing, including variations between planning and execution, failed test results, and environmental suitability limitations
All testing has been completed in accordance with test plan(s)
Defects have been reviewed with stakeholders and dispositioned
For No testing
Review testing is not applicable for this change
Appropriate Justification is provided for No testing rationale
Can Dedicated Testing Story be tagged to multiple Impacted AITS?
Yes As long as the CRQ to which the release event is tied also has same list of AITs
If CRQ is tied only to one AIT then this will throw error in release manager