Overview of Fuego Test System Projects and Initiatives
Towards enhancing the Fuego Test System, this document explores various projects, industry initiatives, and vision discussed in June 2018 by Tim Bird, the Fuego Maintainer at Sony Electronics. It delves into the outline of projects, features in progress, things missing from a previous release, documentation conversion progress, LTS provisioning support, and pre-built Docker image development.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Fuego Test System Projects, Industry Initiatives, and Vision June 2018 Tim Bird Fuego Maintainer Sony Electronics 1 1 10/23/2014 PA1 Confidential
Outline Projects Industry Initiatives Vision 2 2 10/23/2014 PA1 Confidential
Outline Features in progress 1.3 stuff that got missed underutilized features: dependencies, dynamic variables, charting, criteria sharing dynamic documentation Industry initiatives Board automation standards Definition of Automated Testing stack Automated Testing Summit Kernelci Linux Foundation project Vision easy customization test server 3 10/23/2014 PA1 Confidential
Missing from 1.3... Things that slipped from the 1.3 release: Documentation conversion LTS Provisioning support Pre-built docker 4 10/23/2014 PA1 Confidential
Documentation conversion Conversion of docs to reStructuredText Replace PDF and wiki docs with rst Move all docs under source repository Use sphynx to create multiple formats Publish on readthedocs.io Made some progress Have sphynx templates in place Got stuck on markup conversion Considered automation, but hit some hurdles See http://fuegotest.org/wiki/rst_docs 5 10/23/2014 PA1 Confidential
LTS Provisioning support Provisioning Ability to provision board with new system software (particularly the kernel) Fuego historically has left this as an exercise for the user Did some work on this in my lab usb keyboard automation teensy-usb host-controlled keyboard for target LTS download and build Ubuntu kernel replacement Haven t generalized the feature Some support was put into ttc 6 10/23/2014 PA1 Confidential
Pre-built docker image Ability to use Fuego without building the docker image Create a pre-built Fuego docker image, and host it at docker.io e.g. docker run fuego Requires automatic container customization Network proxy User and group Volume mount customization Includes refactoring the Fuego directory layout Turned out to be too intrusive for 1.3 release 7 10/23/2014 PA1 Confidential
Underutilized features Overlay system Ability to override any ov_ function Dynamic board variables Intended for automatic test customization Dependency information cache Automatic installation Customized (per-board) pass-criteria Specs 8 10/23/2014 PA1 Confidential
Dynamic Documentation Provide documentation on a per-suite, per- testset and per-testcase basis Allow users to share well-structured information about a test test outline, expected results, notes links to resources Integrate dynamic report (generated with ftc gen-report ) into text Have .rst template system Not generating anything from rst yet. 9 10/23/2014 PA1 Confidential
Outline Projects Industry Initiatives Vision 10 10 10/23/2014 PA1 Confidential
Test System problems No lego blocks for test system infrastructure Current systems are monolithic e.g. Hard for Fuego to use LAVA as board control software Have mismatches in models, artifacts Lots of islands of work Nobody handles off-DUT hardware orchestration Maybe LAVA, but it s not generalized (e.g. LAVA multi-node tests) 11 10/23/2014 PA1 Confidential
Automated Test Standards Would be good to define: objects, methods, interfaces, protocols Want to mix and match test stack layers, and allow separate implementations to compete board control test orchestration results parsing results aggregation analysis, etc. Reuse features from other domains e.g. log results visualization e.g. libvirt for hardware board control 12 10/23/2014 PA1 Confidential
Previous discussions Presentation at Linaro Connect See http://fuegotest.org/ffiles/Test-Standards- LC-2017.pdf Lots of meetings at ELCE 2017 on this Pengutronix introduced labgrid Linutronix demonstrated r4d and libvirt BOF resulted in some collaboration: See https://elinux.org/Board_Farm Mailing list for discussion: https://lists.yoctoproject.org/listinfo/automated-testing Please join this discussion 13 10/23/2014 PA1 Confidential
Automated Testing Summit October 25, Edinburgh Scotland See http://elinux.org/Automated_Testing_Summit Sponsored by Linux Foundation Core Embedded Linux Project Attempt to assemble wide variety of Linux test stakeholders and practitioners Likely a by-invitation meeting Add name to wiki page or e-mail me if you are interested in attending 14 10/23/2014 PA1 Confidential
KernelCI Linux Foundation Project Kevin Hillman has proposal for making a Linux Project to support KernelCI Project is being done by individuals in spare time (is underfunded, can t expand) 15 10/23/2014 PA1 Confidential
Outline Projects Industry Initiatives Vision 16 16 10/23/2014 PA1 Confidential
Vision super high level Do for testing what open source has done for coding Significant parts of the test process are unshared, ad hoc, private, etc. For no good reason most QA doesn t need to be proprietary There are OSS frameworks and test programs but parts are missing to create a open testing community. Fuego Goal: Promote the sharing of tests, test methods, and results, the way code is shared now Make it easy to create, share and discover tests Make test results easy to share and evaluate 17 10/23/2014 PA1 Confidential
Core principles Useful Scales Actually find bugs or prevent regressions Allow sharing Usable by wide audience Minimal requirements Customizable Easy to use Modular Applicable to embedded 18 10/23/2014 PA1 Confidential
Sharing, Generalization, and Customization Sharing is key to reduce effort Tests have to be generalized so others can run them in different environments Customization is important to be able to leverage what is shared Fuego has good artifacts that can be shared However, people are only sharing tests so far Fuego has good customization in 3 of 4 areas: test applicability (dependencies) test instantiation (specs) test results analysis (pass-criteria) expected values (???) 19 10/23/2014 PA1 Confidential
Easy test customization Customizable system state check Ability to save a snapshot of system status or behavior Example: I want the system to still have X I want the system to continue to be able to do Y Make it very easy to capture state or behavior as an expected value Reduces maintenance for tests 20 10/23/2014 PA1 Confidential
Work in progress clitest Functional.fuego_board_status Functional.fuego_compare_reports meta-test tests results of other tests save_baseline spec Run to establish baseline report default spec Run to compare current report with baseline seddiff compare with regex masking 21 10/23/2014 PA1 Confidential
Demo of tbwiki regression test Update expected value very easily Just a few mouse clicks 22 10/23/2014 PA1 Confidential
Test Server Fuego centralized test server Test artifact sharing hub tests, specs, criteria, boards, results, run-requests Test store Request dispatcher Use cases: Share ad-hoc test (test package) Request test on someone else s board Allow developer to see results from a wide variety of boards Test on hardware that a developer doesn t have locally Mine data for patterns Use customizations for your testing 23 10/23/2014 PA1 Confidential
Fuego 24 10/23/2014 PA1 Confidential
Fuego Features Pre-Built docker image Eliminate long Fuego install step Test program binary cache Remove need for SDK in order to test Focus on pass-criteria customization and sharing For testplan_lts tests, to remove false positives 25 10/23/2014 PA1 Confidential
Provisioning and scaling the testing effort Automated provisioning Requires hardware control for 100% reliability Less than 1% of users will use hardware to automate their kernel installs Want to support semi-automated provisioning Trying hard in Fuego to avoid requiring hardware board control Semi-automated means: Try software board control, and fall back to user intervention 26 10/23/2014 PA1 Confidential
