April 24, 2017 Agenda Review and Action Items Update

Agenda Agenda 24 April 2017 0900 1000 1000 1015 1015 1100 1100 1130 1130 1145 1145 1200 1200 1245 1245 1345 1345 1445 1445 1500 1500 1530 1530 1600 1600 1615 Around the room Review Jan meeting action items ESH update MRL 1-4 update Website changes DMC 2018 Lunch Cyber security in MRL S&T Workshop MRL assessments in Pre-Award Surveys User s guide update OSD BoK path forward Review action items/Adjourn All Jim/Gary Jack/Tom L. Jim Jim Jim/All All Kaye/Scott Jim/Harts/Scott Karr Ali Steve G. Gary/Jim

January MRL WG Action Item Update 1. Jordan M. will get with Dave Karr and work on an Army/Air Force briefing on AS6500, OPR Jordan M. due 13 April 2018. 2. Mark G. to identify some POCs for MEP, OPR Mark G., Done! 3. Jason will look at six sigma and the MRL criteria and share his findings, OPR Jason S., due 13 April 2018. 4. Mark G. will provide instructions on how to use www.dodmrl.com and send to Jim, OPR Mark G. Done! 5. Jim to send instructions to the team for comment, OPR Jim M., due 19 Jan 2018. Done! 6. Get senior leaders to brief at DMC general session, OPR Scott P., due 19 Oct 2108. 7. Tom L. will collect feedback on his ESH recommendation and report back in April, OPR Tom L., due 13 April 2018. Done! 8. Jim M. will collect feedback on his MRL 1-3 recommendation and report back in April, OPR Jim M., due 13 April 2018. Done! 9. Jim will make sure that the team receives Al Sanders MRL 1-4 matrix and presentation, OPR Jim M., due 26 Jan 2018. Done! 10. Harts and Scott to take the lead set the agenda and get 6.1/6.2 folks to come to the MRL WG meeting to address their needs. Jim to coordinate a meeting in April or July, OPR Harts, Scott P. and Jim M. due 13 April 2018. Pushed to July. 11. Tom F. to provide POCs for the ask an expert from MDA and Space, OPR Tom F., due 26 Jan 2018. Space? 12. Jim to follow up with Industry POCs and finalize industry POC list and add to website, OPR Jim M., due 26 Jan 2018. Done! 13. Jack, Tom L. and Josh to look at the language in the User s Guide and propose changes, OPR, Jack, Tom F. , and Josh R., due 13 April 2018. 14. Scott to get a POC in cyber security from Al Sanders (done) and have them brief in April, OPR Scott, due 6 April 2018. Done! 15. Tom L. to send our MRL 6 artifacts for team to review and discuss in April, OPR Tom L, Done! 16. Steve G. to brief the path forward on the OSD BOK, OPR Steve G., due 24 April 2018. This meeting! 17. Scott to report what version of the MRL criteria the DOCENT tool is built on, OPR Scott P., due 26 due 2018. 2016 DB 18. Tom L. to call David G. and discuss work on sustainment, OPR Tom L. due 13 April 2018. 19. Jack to look at matrix and update language, OPR Jack, due ASAP! Green In progress Red needs completed Black - done

ESH Committee Tom Lastoskie Chair Jack Galuardi Vice Chair

Overview 2Q2018 Presentation/Discussion Items of Discussion 1. 2017 Workshop Results (same as Jan 2018 mtg) 2. Additional Guidance from OSD (same as Jan 2018 mtg) 3. Implementation Plan 4. Additional Potential Actions Comments

1. 2017 Workshop Results Overwhelmingly, participants agreed that Environmental, Safety and Health (ESH) criteria should be enhanced or added 5

2. Additional Guidance from OSD A. OSD wants to keep the current Matrix Thread format as is B. That means that Threads/Sub- Threads to stay the same (no adding of Sub-Threads to the current structure) C. Criteria enhancements or new criteria can be added to the existing structure! 6

3. Implementation Plan A. Took the 2017 Workshop outputs for ESH from the 3 Teams, adjudicated those outputs and provided to the MRL WG ESH Committee B. ESH Committee had deadline of 16 Feb to provide comments C. Majority of the MRL WG ESH Committee voted to use the adjudicated language ( ESH Committee Language in Tab 2 of attached spreadsheet, ESH Committee Criteria Wording for MRL Matrix Apr2018.xlsx ) and incorporate the ESH Committee Language into the MRL Deskbook criteria matrix 7

4. Additional Potential Actions A. Update MRL Users Guide to reflect changes to criteria and incorporate the Industry Comments/Recommendations in Tab 2 of attached spreadsheet, ESH Committee Criteria Wording for MRL Matrix Apr2018.xlsx B. Incorporate new wording in the MRL Deskbook to address situations where Industry may look at some areas of ESH differently than DoD? 1. This is an additional activity which the MRL Working Group would need to decide if action is appropriate C. Define/Explain Special Handling with more definitive wording in the MRL Deskbook and User Guide 1. This is an additional activity which the MRL Working Group would need to decide if action is appropriate 8

Comments

ESH Committee Members Tom Lastoskie Chair Jack Galuardi Vice Chair Michael Ganowsky Member, Boeing Jason Schmidt Member, Boeing Greg Krieger Member, BAE Eric Whitman Member, Draper Harry Diaz-Agosto Member, DCMA Steve Anderson Member, Navy (NAVAIR) David Karr Member, USAF (LCMC) All members verified their membership

MRL 1-4 Update Jim Morgan

MRL 1 MRL 1- -4 Update 4 Update MRL working group has been working this since 2016 Taken to 2016/2017 workshops and input provided APT/Jim created new version as input to 2017 workshop MRL WG S&T practitioners help sought Another version is available Current changes are fairly minimal Al Sanders work? NO request to do this!, are we fixing something that is not broken? Recommendations Accept latest changes, add to criteria Leave well enough alone! Wait until after July S&T meeting

MRL 1 MRL 1- -4 Update 4 Update Summary of Changes Summary of Changes MRL 4 changes to create logical flow from 1-3 to 4 Some criteria added where none existed Some simple word changes to current criteria Words added for clarification MRL 1-3 identified used instead of assessed/surveyed

MRL 2 MRL 4 MRL 3 Potential sources identified to address technology needs. Understand state of the art. A.1 Capability instead of technology needs. Concepts under consideration added Understand the state of the art. Potential sources identified to address capability needs for concepts under consideration. New manufacturing concepts and potential solutions identified. Manufacturing Science (concepts and technologies) requirements identified to support potential solutions. Manufacturing technology concepts identified through experiments/models. Advanced manufacturing technology concepts and capabilities identified. Manufacturing Science & Advanced Manufacturing Technology requirements identified. Advanced manufacturing technology development requirements identified. A.2 Man Sci. Requirements ID d To support potential solutions Advanced ManTech Capabilities added. ID d only Man Sci to ManTech Relevant materials/processes evaluated for manufacturability using experiments/models. B.1 Identify potential elements (materials, processes, capabilities, limitations) that impact producibility. Relevant elements evaluated for producibility using experiments, models or simulations. Elements for producibility Added

MRL 2 MRL 4 MRL 3 Applications defined. Broad performance goals identified that may drive manufacturing options. Top level performance requirements defined. Trade-offs in design options assessed based on experiments. Product lifecycle and technical requirements evaluated. B.2 Manufacturing and technical requirements identified that may drive manufacturing options. Broad performance goals identified. Top level product performance, lifecycle, and technical requirements defined and evaluated. Trade-offs in design options assessed based on experiments or models. Mfg and tech Requirements ID d That drive mfg options Simple rewording Initial cost targets and risks identified. High level process chart model developed. Technology cost models developed for new process steps and materials based on experiments. C.1 Initial cost models developed for process steps and materials (high-level process chart). Manufacturing cost estimates for potential solutions initiated. Cost models instead of targets Mfg cost estimated

MRL 2 MRL 4 MRL 3 Sensitivity analysis conducted to define cost drivers and production development strategy (i.e. lab to pilot to factory). Producibility cost risks assessed. Initial cost models support Analysis of Alternatives (AoA) and Alternative Systems Review (ASR). C.2 Sensitivity analysis conducted to identify and quantify cost drivers, risks, and development strategy (e.g., lab to pilot line to factory). Producibility and lifecycle cost risks assessed. Initial cost analysis supports Analysis of Alternatives (AoA) and Alternative Systems Review (ASR) or trade studies. ID d instead of defined Production eliminated Lifecycle added Analysis instead of models MRL 1 Potential investments identified to Potential manufacturing investments identified. C.3 Material properties identified for research MRL 1 D.1 to Material characteristics and properties surveyed and identified for research. Material availability assessed. Material scale-up issues identified. Projected lead times have been identified for all difficult to obtain, difficult to process, or hazardous materials. Quantities and lead times estimated. D.2 Materials sources and availability identified. Material availability and sourcing assessed. Projected lead times have been identified. All difficult to obtain, difficult to process, or hazardous materials have been identified. Quantity and lead time estimated. Material scale-up risks identified and mitigation plans created. Sources ID d Sources assessed Scale up risk ID d and mitigation plans Sentences combined

MRL 2 MRL 4 MRL 3 Survey completed for potential supply chain sources. D.3 Supply chain capabilities analyzed with gaps and risks identified for preferred concept. Gaps and risks for preferred concepts added

MRL 2 MRL 4 MRL 3 List of hazardous materials identified. Special handling procedures applied in the lab. Special handling concerns assessed. List of hazardous materials updated. Special handling procedures applied in the lab. Special handling requirements identified. D.4 Environmental, safety, and health (ESH) compliance risk identified. Potential special handling concerns defined and regulatory requirements assessed and alternatives evaluated. Special handling procedures assessed. SH concerns defined In the lab eliminated ESH added List of hazardous materials updated and alternatives assessed. Special handling risks and issues identified and disposal procedures evaluated. Special handling procedures are being followed. Lifecycle added SH risks ID d, in the lab eliminated Identification of proposed manufacturing concepts or producibility needs based on high-level process flow chart models. Production modeling/simulation approaches for process or product are identified. E.1 Production modeling/simulation approaches for process or product are identified. Modeling and simulations utilized to model behavior/performance of potential manufacturing solutions and defined manufacturing concepts. Key predictive process variables extracted from manufacturing trials or DOEs. Big change Big change

MRL 2 MRL 4 MRL 3 Identification of material and/or process approaches. Complete a survey to determine the current state of critical processes. E.2 Identification of potential processing approaches. Current state of key/critical processes assessed against program requirements. Potential added Assessed instead of surveyed Quality strategy identified as part of the Acquisition Strategy and included in Systems Engineering Plan (SEP). F.1 Quality management strategy included in technology development and in systems engineering activities. Quality management strategy included Instead if ID d Product inspection and acceptance testing strategy identified as part of the Acquisition Strategy and included in Systems Engineering Plan (SEP). F.2 Initial product quality requirements and risks identified. Inspection technologies identified. Added Product inspection and acceptance testing strategy identified as part of the technology development and systems engineering activities. Tech development instead of acq strategy

MRL 2 MRL 4 MRL 3 Tooling/Special Test Equipment (STE)/Special Inspection Equipment (SIE) requirements are considered as part of AoA. Tooling, Special Test Equipment (STE), Special Inspection Equipment (SIE) requirements, including modifications, funding, and schedule, are considered as part of AoA or trade studies. H.1 Including modifications, funding, and schedule Trade studies added

MRL 2 MRL 4 MRL 3 Availability of manufacturing facilities for prototype development and production evaluated as part of AoA. H.2 Capability and availability of manufacturing facilities for prototype development and production evaluated as part of AoA or trade studies. Human Factors & Ergonomics /Safety requirements identified for manufacturing (personnel, processes & equipment). Capability added Trade studies added I.1 Manufacturing concepts to support trade studies identified. Manufacturing strategy development (notional plans and schedules) initiated. Added Technology development article component list developed with associated lead time estimates. I.2 Initial plans and strategies for potential materials identified. Specific considerations (availability, lead time, handling, storage) for materials and components idenitified. Materials and components list developed with estimates of availability, lead time, handling, storage, etc. Added Added Handling and storage added

Website Changes Website Changes New - How to use this MRL Body of Knowledge Contact List for Ask-An-Expert on MRLs Jordan Masters, Army acquisition Mike Deitchman, Navy S&T Rob Arthur, DAU Space POCs need to be added (MDA provided, will be added) Harry and Tony, DCMA (Will be added) Contact List for Industry MRL Leadership Updated per member recommendations 2018 MRL Working Group Information All MRL WG presentations posted Agenda/Meeting info Minutes

DMC 2018 DMC 2018 Suggested agenda Day 1 MRL Training Tom L. AS6500 training Dave K. or some other relevant training Day 2 OSD Perspective Mr. Gold MRL WG Activities Scott P. Industry Industry industry Other ideas? OSD BoK?

Cyber Security Kaye Ortiz

MRL Working Group S&T Meeting PURPOSE Gather various stakeholders as part of the MRL Working Group continuous process improvement efforts to refine the MRL 1-3 criteria Determine their needs, expectations and what they need from the team Have S&T stakeholders become more active members of the MRL Working Group Explore MRL WG next steps towards refinement of MRL 1-3 BACKGROUND Development of the MRL criteria began in 2000 and has been continually refreshed by the MRL WG The goal of the MRL working group is to continue to refresh the MRL body of knowledge to reflect the best practices To assist in identification and mitigation of manufacturing, production, quality, and industrial risks and issues Annual workshops attended by government, industry, and academia to socialize recommended changes DISCUSSION SECDEF and DoD services and agencies are pursuing Enhanced Readiness, Reducing Acquisition Development Cycle Times, Moving Manufacturing to the Left Developmental Planning and Early Systems Engineering, Tech Transition to Avoid the Valley of Death The MRL working group began to refine the MRL 1-3 criteria and met with government and industry SMEs to refine in 2016/2017 At January MRL Working Group meeting a one day workshop was proposed WHO S&T, R&D, Acquisition, ManTech, Developmental Planning/Early SE, and manufacturing & quality operations SMEs WHAT Participate in a one day discussion to establish S&T community MRL needs/wants/etc. WHEN 25 July, after the 24 July MRL WG meeting, Vencore, 4100 North Fairfax Dr., Arlington, VA 22203, Suite 800

MRL Working Group S&T Meeting MRL Working Group S&T Meeting Key questions What is the best way to get stakeholders to provide needs, expectations? Workshop, survey, etc.? Does S&T community perceive MRL as value added? Do they use MRL? Is existing criteria acceptable? Why or why not? What is the focus? What is output from today? Rough agenda Who should attend?, Who will invite?, How to get them there? Who will coordinate all activities? Who is going to help? Chairperson? List of actions and OPRs Target 20 30 people (Vencore) Services, Industry, Academia, MRL WG

MRLs & Pre-Award Surveys Dave Karr

MRLs & Pre-Award Surveys Situation: Numerous competitive RFPs released with requirements for offerors to report the results of MRL self-assessments in their proposals Challenge: Source Selection evaluators must rely only upon the claims of the offerors regarding their MRL scores Solution: Engage DCMA to verify MRL self-assessments through Pre- Award Survey process Problem: PAS factors are not aligned to MRL threads/criteria 28

MRLs & Pre-Award Surveys Existing Pre-Award Survey checklists: General (SF1403) Technical (SF1404) Production (SF1405) Quality Assurance (SF1406) Financial Capability (SF1407) Accounting System (SF1408) DCMA s process seems very open to tailoring the survey to the needs of the customer How can we best request DCMA to verify MRL assessments? Tailor existing checklist? New checklist? MRL matrix? Would change by phase/target MRL 29

Users Guide Update Ali Al-Hamdani

OSD BoK Path Forward Steve Gray