Scientia Prudentia et Valor: Planning for a Successful Test Plan
This content discusses the importance and process of test planning in the STAT Center of Excellence, emphasizing efficient techniques for designing experiments and the role of a diverse test planning team. It covers motivation, foundations of test planning, test program organization, and the STAT process for decision quality information.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
STAT COE: Scientia Prudentia et Valor Planning for a Successful Test Plan Aaron Ramert Scientific Test and Analysis Techniques Center of Excellence www.AFIT.edu/STAT 937-255-3636 x 4736 Scientia Prudentia et Valor DISTRIBUTION STATEMENT
Agenda STAT COE: Scientia Prudentia et Valor Motivation The STAT Process Foundations of test planning The 5 W s Planning Session Tasks Resources Conclusions Questions 2
Motivation STAT COE: Scientia Prudentia et Valor A 2004 report from the National Research Council stated, Within the statistical community there has been much more attention paid to the development of efficient techniques for design of experiments than on the planning process that precedes it. In a 2013 article, Dr. Geoff Vining noted, Most modern textbooks focus primarily on the analysis of the experimental results rather than the proper protocol As a result, most modern experimental design textbooks are truly little more than expositions on the analysis of variance (ANOVA). Pop open your laptop and create some DOEs for us. 3
Todays Discussion STAT COE: Scientia Prudentia et Valor Test Planning Session Comprehensive Test Program Organization and planning for the Test Planning Session Goals Objectives Responses Factors Etc.. From requirements to decision quality information 4
Some Progress STAT COE: Scientia Prudentia et Valor Prior work has focused on what the required deliverables are, some discussion on how to produce them, no discussion on who is needed Answer: A diverse test planning team which is prepared to exchange knowledge and produce deliverables which are mutually acceptable to all stakeholders 5
STAT in Test & Evaluation Requirement(s) Plan Program Level ID/Clarify/Quantify STAT Candidates Understand Requirements Basic Test Process STAT COE: Scientia Prudentia et Valor Decompose Mission/System STAT Infusion Verification Optimization Screening Determine Objective(s) Characterization Define Design Space Define Response(s) Design Factors Recorded Conditions Randomization Blocking Disallowed Combinations Select Factor(s) / Level(s) Identify Constraint(s) Noise Design Confidence & Power Signal & Noise Create Test Matrix Orthogonality Model Specification Optimality Criteria Coverage Replication Execute Run Card Generation Experimental Protocol/Training Record Deviations Execute Test Analyze Distribution Fitting Hypothesis Testing Monte Carlo Simulation Optimization Analysis of (Co)Variance Response Surface Methodology Regression Modeling Multivariate Analysis Report Decision Quality Info Program Decision(s) 6
Focus on Planning STAT COE: Scientia Prudentia et Valor Requirement(s) Plan ID / Clarify / Quantify STAT Candidates Understand Requirements Decompose Mission/System Screening Verification Optimization Determine Objective(s) Characterization Define Response(s) Design Factors Recorded Conditions Randomization Blocking Disallowed Combinations Define Design Space Identify Constraint(s) Select Factor(s) / Level(s) Noise Design Execute Analyze Program Decision(s) 7
Test Planning Meeting STAT COE: Scientia Prudentia et Valor Will likely require multiple days Must conclude with deliverables Convert requirements into well defined goals and objectives Define the responses Define the factors and levels Identify the constraints Will only succeed if the right people are brought together and are prepared to think hard and make decisions Defined through a System Decomposition 8
A Plan for Test Planning STAT COE: Scientia Prudentia et Valor An e-mail with Test planning meeting, next Thursday, 9:00 sharp will not yield good results Timely announcement which covers the 5 W s (who, what, when, where, why) A plan to plan for Test Planning Becoming recursive? Start with a small core group and have a brainstorming session to come up with an initial estimate of the 5 W s Be as deliberate and as thorough as possible Clear communication is essential 9
Who STAT COE: Scientia Prudentia et Valor A diverse test planning team will include people familiar with all relevant aspects of the system under test (SUT) and the test effort Will likely include: Program Representative System engineers Operational users Lab/range operators STAT experts Designated Leader May also include: Maintenance personnel Human Systems Integration Industry experts Logisticians (schedulers) Safety personnel More Maximum participation for the entire process desired 10
What STAT COE: Scientia Prudentia et Valor A designated planning session Stakeholders dedicate time to come together, cross brief the entire planning team in their specific area of expertise and cooperatively produce the elements necessary for a test plan It s not enough to merely have some people in a room. They have to be the right people with the right skills and who have done their homework prior to the planning session Post an agenda with assignments and due dates 11
When, Where STAT COE: Scientia Prudentia et Valor Have briefs before the event Allow enough time for a complete evolution Consider the complexity of the SUT Location where maximum number of stakeholders can attend Should it be near a specific test area? Should it be near an example of the SUT? Considerations for a virtual test planning meeting 12
Why STAT COE: Scientia Prudentia et Valor An effective test, which produces defensible decision quality information, begins with a comprehensive and well founded plan A good plan doesn t guarantee a good test, but a poor plan will nearly always lead to a poor test Identify barriers to testing early so there is time to address them Establish rules, roles, and responsibilities 13
Steps to Plan the Planning STAT COE: Scientia Prudentia et Valor Begin with the end in mind. Dr. Stephen Covey s 2nd habit Convey to the to the test team what will be accomplished at the planning session. If you can communicate a clear vision to the test team in a timely manner they can show up prepared to teach, learn, make decisions, and produce deliverables. When the planning session ends the test team should have: 1. A thorough understanding of the requirements 2. Identified all governing documents 3. A thorough understanding of the system under test (SUT) 4. A thorough understanding of SUT and operator interactions 5. A thorough understanding of the testing resources and capabilities 14
Steps to Plan the Planning STAT COE: Scientia Prudentia et Valor When the planning session ends the test team should have: 6. Identified the decision(s) to be made and who will make them 7. Testing deadlines and program timeline 8. Agreed upon testing goals and objectives 9. A list of responses and factors 10. A communications plan who can release information 11. A data collection, data storage, and data security plan 12. A record of all assumptions 13. A list of any missing information and planned corrective actions 14. Any identified testing shortfalls 15. Any action items and next steps (a timeline?) 16. Anything else unique to your test effort 15
Example Agenda STAT COE: Scientia Prudentia et Valor The STAT COE was involved with testing and OBOGS unit 1. Review the task order 2. Review the MIL STD and SFF documents 3. OBOGS decomposition 4. OBOGS maintenance review 5. Human breathing and OBOGS interactions 6. Aircrew flight equipment review 7. Test laboratory 1 overview and history 8. Test laboratory 2 overview and history 9. Problem statement development 10. Test goals and objectives development 11. Responses, factors, and limitations development 12. Review of action items 16
STAT COE STAT COE: Scientia Prudentia et Valor The STAT COE consists of a highly educated and trained interdisciplinary group of DoD T&E professionals who possess expert knowledge and experience of DoD T&E planning, execution and assessment, warfare areas, and statistical expertise. In addition to injecting more STAT into program T&E strategy planning, the STAT COE Experts ensure the organic T&E team gain a better understanding of STAT and how to employ it to optimize test rigor, effectiveness, and efficiencies. More at: www.afit.edu\STAT 17
STAT COE Resources www.afit.edu/STAT STAT COE: Scientia Prudentia et Valor Currently have 75 documents Best Practices T&E Strategy Test Design Data Analysis and Statistics Reliability Software Testing Test Planning Guide Case Studies 18
STAT COE Resources www.afit.edu/STAT STAT COE: Scientia Prudentia et Valor Tools that assist with T&E Strategy Test Design Data Analysis and Statistics Reliability Additional Resources 19
STAT COE Resources www.afit.edu/STAT STAT COE: Scientia Prudentia et Valor 20
Conclusion STAT COE: Scientia Prudentia et Valor Begin with the end in mind. When this planning session is over we will have Communicate your vision Have the right people with the right information in a single space and make sure they are all prepared to brief their subjects and make decisions Document everything 21
STAT COE: Scientia Prudentia et Valor Questions? 22
