Technical Service Test Purpose Discussion
Explore the purpose of tests by Technical Service for vehicle type approval in the context of CSMS compliance. Various comments and perspectives highlight the importance of verification and validation in confirming safety measures implementation. The discussion delves into the interaction between testing, security objectives, and regulatory compliance.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Agenda & Discussion Points 12th Workshop on UNR155 Implementation 8th November 2022
Agenda 8th Nov. 2022. 09.30 (CET)/ 17.30 (KST,JST) 09.35 (CET)/ 17.35 (KST,JST) Opening Proposal amendments to the interpretation document regarding "reciprocal recognition of CoC of CSMS" TFCS-23-04 Review on Q&A(C) - Testing - "How to handle non critical elements?" Next steps - Conditions for "reciprocal recognition of CoC" Closing 10.00 (CET)/ 18.00 (KST,JST) 11:00 (CET)/ 19.00 (KST,JST) 11:30 (CET)/ 19.30 (KST,JST)
Q&A(C) Table Testing
Item 6: What is the purpose of test by Technical Service? Background Initial comment from the Word Document created by GRVA secretariat: The purpose of tests for vehicle type approval is not intended to discover another vulnerability. Results of the tests should be reviewed in the perspective of CSMS compliance. It should be confirmed that test is rational referring to the manufacture's process and results of risk assessment.
Item 6: What is the purpose of test by Technical Service? Comments <NL> The overarching expectation is that the CS-requirements will be verified and validated by the tests. So an assessment of how the requirement is covered by the test is one element. There is a hierarchical relation to the TARA, where the security objective is translated to the separate requirements.
Item 6: What is the purpose of test by Technical Service? Comments <FR> Based on TARA evaluation and declared countermeasures, testing aims to confirm safety measures implementation with regards to annex 5 as declared by the OEM.
Item 6: What is the purpose of test by Technical Service? Comments <DE> KBA agrees with the combined feedback of NL and FRA. TS shall confirm by sampling if the OEM meets 7.3 (in particular 7.3.3 and following) homologation (annual reporting). - before and after Are (or shall) audits be included in this question?
Item 6: What is the purpose of test by Technical Service? Answer(Comment) as the group The purpose of tests for vehicle type approval is not intended to discover new vulnerability. Results of the tests should be reviewed in the perspective of CSMS compliance. It should be confirmed that test is rational referring to the manufacture's process and results of risk assessment. The overarching expectation is that the CS-requirements will be verified and validated by the tests. So an assessment of how the requirement is covered by the test is one element. There is a hierarchical relation to the TARA, where the security objective is translated to the separate requirements.(?) (Do we need to mention the relation with Annex 5?)
Item 7: How many tests? Chosen on what basis ? Background
Item 7: How many tests? Chosen on what basis ? Comments <NL> Based on the TARA and other input the TS might select certain tests to witness. The focus has to be on mitigations that the security heavily relies on and an assessment of the performed test that were used to determine that the CS- requirements for the TARA were met. OEM and TS should work together to produce convincing proof that the design will be effective in its security performance.
Item 7: How many tests? Chosen on what basis ? Comments <FR> Our initial prescription is 3 to 5 tests (subject to change) depending on TARA and Annex 5 assessment results. We believe that this achieves a balance between having enough time allocated to each test and keeping the total test time reasonable.
Item 7: How many tests? Chosen on what basis ? Comments <DE> Same as NL with the extention "Based on the TARA and other input the TS might select certain tests to witness or to perform the tests on its own."
Relevant Item 26: Are critical elements same as cyber- relevant elements/ECUs? If yes, does all cyber relevant ECUs should undergo an exhaustive TARA? Comments <DE>The testing itself should of course be concentrated on the main risks. This can include elements which, when taken separately, can be seen as non- critical, but form a critical system. The manufacturer is obliged to perform testing which covers all necessary action to guarantee the accepted by him remaining risk. The approval authority in our opinion can limit its testing (or supervising of tests) to a sampling regarding serious risks (regarding health, environment, safety; perhaps also other like privacy/rather not protection of property), where the accepted remaining risk might by much bigger than this of the manufacturer (but acceptable with regard to the general protection objectives).
Item 7: How many tests? Chosen on what basis ? OEM Answer(Comment) as the group The testing itself should be concentrated on the main risks defined by the relevant TARA. This can include elements which, when taken separately, can be seen as non-critical, but form a critical system. The manufacturer is obliged to perform testing which covers all necessary action to guarantee the accepted by him remaining risk under the CSMS. When testing is outsourced to the suppliers, the OEM should explain the TS how the OEM examined the test results delivered in the light of TARA.
Item 8: How will the sensitive information related vehicle type be treated? Background
Item 8: How will the sensitive information related vehicle type be treated? Comments <NL> It depends on the level of sensitivity and how the OEM defined their processes for information sharing. There must be sufficient information manufacturer to assess CSMS and the cyber security of the vehicle type. If certain information is not available to be shared with the TS/TA, then the audit/assessmnet should include such items at the manufacturer's facility. And confidentiality items should be avoided from the test report. Only names/pseudo names can be used in the reports. from the vehicle
Item 8: How will the sensitive information related vehicle type be treated? Comments <FR> The TS should have access to all information and material needed to perform their audits, assessments and tests. When a document is too confidential to be sent to the TS, it should be shown on the manufacturer's premises. Similarly, if a particular test were to include equipment or a program that cannot leave the manufacturer's facilities, the technical service may choose to either perform witness testing or to perform the test themselves on-premise.
Item 8: How will the sensitive information related vehicle type be treated? Comments <DE> TS and TAA shall have access to all necessary information (if appropriate under the precondition of no specific records).
Item 8: How will the sensitive information related vehicle type be treated? Answer(Comment) as the group The TAA and TS should have access to all information and material needed to perform their audits, assessments and tests. When a document is too confidential to be sent to the TS, it should be shown on the manufacturer's premises. Confidentiality items should be avoided from the test report. Only names/pseudo names can be used in the reports.
Item 17: Destructible test methods allowed? Background
Item 17: Destructible test methods allowed? Comments <NL> See question 7, aim of tests performed or witnessed by TS is not to discover new vulnerabilities. In the proces of witnessing or perfroming tests the TS may encounter some untested cases, vulnerabilities in the implemntation of the solution.
Item 17: Destructible test methods allowed? Comments <DE> Agree to the comment from NL but the question was originally raised by an OEM to avoid that the TS performs destructable test. From our point of view "destructable test" should be possible and required if deemed to be necessary, but not be restricted (for both OEM and TS/TAA).
Item 17: Destructible test methods allowed? Comments <FR> We have no such kind of restriction on tests (legitimate destructive tests could be needed, e.g. erasing data or soldering) but verification tests as we envision them for R155 will seldom be destructive.
Item 17: Destructible test methods allowed? Answer(Comment) as the group Destructible test methods will not be restricted by TAA or TS. However, the purpose of tests for vehicle type approval is not intended to discover another vulnerability and results of the tests should be reviewed in compliance. Verification tests as we envision them for R155 will seldom be destructive. the perspective of CSMS
Item 18: How much effort (time) shall be spent (in particular on pen-testing)? Background
Item 18: How much effort (time) shall be spent (in particular on pen-testing)? Comments <NL> Not sure if this question understood correctly, but if it comes to time spend by the designated TS for the TA. The expected time for the assessment should be determined beforehand based on complexity and others. A reference for thought is about 1 week with 2 specialists, but practice might turn-up with better solutions. The result should give the convincing proof of performance during live. effectiveness of the CS-
Item 18: How much effort (time) shall be spent (in particular on pen-testing)? Comments <DE> Depends on extent and complexity of e/e architecture (or modification in stage 2)
Item 18: How much effort (time) shall be spent (in particular on pen-testing)? Comments <FR> See our answer to question 7 above. Depending of the test matrix defined, a test phase should generally take between 1 and 2 weeks.
Item 18: How much effort (time) shall be spent (in particular on pen-testing)? Answer(Comment) as the group Depending of the test matrix defined by the tester. (Fuzzing?) test phase should generally take between 1 and 2 weeks.
Q&A(C) Table Risk Assessment
Item 26: Are critical elements same as cyber-relevant elements/ECUs? If yes, does all cyber relevant ECUs should undergo an exhaustive TARA? How to handle non critical elements? Answer(Comment) as the group Critical elements in the vehicle type approval considerations should be focused. Critical elements may not In the CSMS, a standardized process of having such standard/basic security requirements, testing, and monitoring activities for non-critical elements can be included by the manufacturer. Such implementation of CSMS processes for non-critical elements can be spot checked at the manufacturer s location by TS/TAA through sampling. This way we are covering both critical and non-critical elements aspect of the regulation.
Next steps Conditions for "reciprocal recognition of CoC"
