Overview of Fuego Test System
The Fuego Test System is a comprehensive platform that combines a Linux distribution, host scripts, pre-packaged tests, and Jenkins within a container. It offers core features such as distribution of Linux for testing, an architecture-neutral build system, test execution scripts, result analysis, visualization, and report generation. The system's architecture includes components like Docker containers, Jenkins, test programs, and target boards. With features covering installation, test execution, analysis, and visualization, the Fuego Test System proves to be a versatile tool for testing and automation tasks.
Uploaded on Mar 08, 2025 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Fuego Test System Status Summary June 2018 Tim Bird Fuego Maintainer Sony Electronics 1 1 10/23/2014 PA1 Confidential
Outline Introduction Feature List Feature Status 2 2 10/23/2014 PA1 Confidential
Micro-Introduction Fuego = (Fuego Linux distribution + host scripts + pre-packaged tests + Jenkins) all inside a container 3 10/23/2014 PA1 Confidential
Architecture Diagram Host machine: Container build system Docker container: Fuego Linux distribution Jenkins Test programs Scripts Target board Volume Mount Web control interface Toolchains Config Builds Logs 4 10/23/2014 PA1 Confidential
Core features Distribution of Linux for testing Build system Architecture-neutral & inherently cross-platform Includes a collection of tests Scripts for test execution Results parsing, analysis, and visualization Report generation Multiple transports Jenkins front end Also has a command line tool 5 10/23/2014 PA1 Confidential
Outline Introduction Feature List Feature Status 6 6 10/23/2014 PA1 Confidential
Slide key Italic header = reference material 7 10/23/2014 PA1 Confidential
Feature Areas Installation Test Execution Results Analysis User Interface Visualization Reports Tests Fuego Distribution Infrastructure Usability Documentation Marketing 8 10/23/2014 PA1 Confidential
Installation Docker build Pre-built Docker image (partially completed) Problems with container customization Adding a board Adding a toolchain Populating Jenkins artifacts Provisioning setup (none) 9 10/23/2014 PA1 Confidential
Pre-built docker image Ability to use Fuego without building the docker image Create a pre-built Fuego docker image, and host it at docker.io e.g. docker run fuego Requires automatic container customization Network proxy User and group Volume mount customization Includes refactoring the Fuego directory layout Turned out to be too intrusive for 1.3 release 10 10/23/2014 PA1 Confidential
Execution Launch jenkins job or command line main.sh, ftc run-test Test customization (plans, specs, criteria) Overlay system (ovgen.py) Test dependency system (need_check.sh) Transports (serial, ssh, local, ttc, lava) Build system Source retrieval Deployment Remote execution Log retrieval Error handling/Abort Timeouts 11 10/23/2014 PA1 Confidential
Results Analysis Log parsing Unified Output Format log_compare Pass-criteria per-board pass-criteria Jenkins history 12 10/23/2014 PA1 Confidential
User Interface Jenkins UI Job launching ftc to manage Jenkins objects ftc add-node, add-job, add-view History, status, result indicators Not very rich (no icon for abort or skip) log links, results description Failure reason (missing) 13 10/23/2014 PA1 Confidential
Visualization Charts Log splitting by testcase Tables Plots (jquery flot module) chart_config A few items missing from JTA (hide/unhide data sets) 14 10/23/2014 PA1 Confidential
Log splitting by testcase Ability to split the test log into pieces, according to testcase boundaries Only works for some logs Requires slight modification to test s parser.py Addition to UI Can click on testcase in Jenkins UI, and see section of log related to that testcase Very handy for examining results details 15 10/23/2014 PA1 Confidential
Reports ftc list-runs, gen-report field control output formats --where (filtering) (time comparisons are a bit confusing) (could use a batch id) Dynamic documents chart insertion (missing) coverage (very little) 16 10/23/2014 PA1 Confidential
Report generation improvements More output formats html, rst, pdf, excel, csv Control of fields displayed header_fields report fields More filtering (--where options) especially tguid:result Try this: ftc gen-report where tguid:result!=PASS 17 10/23/2014 PA1 Confidential
Tests LTP see next slide kselftest Realtime, Filesystem, Network POSIX System benchmarks Security/vulnerability autopkg ptest Fuego self-tests / release test 18 10/23/2014 PA1 Confidential
LTP lots of specs skiplists Autodetect skiplist items supports separate build and manual deploy one_test execute a single test program Problems: realtime parsing is not good should separate POSIX from other tests 19 10/23/2014 PA1 Confidential
New tests Realtime benchmarks: backfire, deadlinetest, migratetest, pmqtest, ptsematest, sigwaittest, svsematest Other tests (updated or new): dbench4, dd, iperf3, vuls, autopkgtest, year2038 LTP_one_test fuego_board_status Fuego self-tests: fuego_lint, fuego_tguid_check, fuego_ftc_check fuego_release_test 20 10/23/2014 PA1 Confidential
Fuego release test Complicated test to do a full release test Builds docker container Runs docker container for release under test , alongside test-runner container Checks Jenkins web interface Using HTML DOM element checks Using comparisons of web page rendered images Adds capabilities to Fuego distribution for testing of other DUT web or image features 21 10/23/2014 PA1 Confidential
Fuego Distribution Additions clitest Selenium Web page HTML testing Chromium, imagemagick, python-pillow Web page image testing pexpect, flake8 (python lint) Fuego self-test python-reportlab Report formats iperf3 Network testing Tests expected output of commands 22 10/23/2014 PA1 Confidential
Web page and image comparison tools Added Selenium to Fuego distribution For web page testing Added Chromium to Fuego distribution For web page rendering automation Added tools for: Comparison of returned HTML Web page image capture Image comparison With support for masked regions Note: This is not generalized yet Need to read Functional.fuego_release_test scripts and use as example 23 10/23/2014 PA1 Confidential
Infrastructure bitbucket git repositories Hardware board control Individual test phases ftc outside the docker container 24 10/23/2014 PA1 Confidential
Hardware board control General feature is ability to control board under automation Added in 1.3: Hook for hardware board reboot Shows method for adding board control hooks Goal is to support provisioning and other hardware functions, as well ex. off-DUT test hardware control and multiplexing Would rather re-use some other board control layer e.g. LAVA, labgrid, Dryad (from SLAV), ttc, etc. 25 10/23/2014 PA1 Confidential
Individual test phases Ability to run test phases individually Main purpose is to allow separation of: Test program build (on host) Test execution on target Can use ftc option: ftc run-test p pcb Executes pre_test, pre_check and build phases, then stops Can use environment variable FUEGO_TEST_PHASES= pre_test pre_check build 26 10/23/2014 PA1 Confidential
ftc outside the docker container docker adds unneeded overhead to some commands Some commands can now be run outside the docker container New fuego.conf file to specify directory locations list-runs, gen-report can be done directly on host Partial step towards use of low-level Fuego functionality with alternate UIs and frameworks 27 10/23/2014 PA1 Confidential
Usability End user License Web site Ease of installation/use Consistency Obviousness Robustness Error Handling Developer support Coding style Languages (bash, python, json) Debugging Documentation 28 10/23/2014 PA1 Confidential
Contributor guidelines Recently added to wiki Coding style guide Mostly indentation (4 spaces, no tabs) See http://fuegotest.org/wiki/Coding_style License guide Patch submission tips See http://fuegotest.org/wiki/License_And_Contribution_Policy 29 10/23/2014 PA1 Confidential
Documentation Installation (quickstart guide) End-user docs (wiki, pdf) User s guide (adding...) Walkthroughs (none) Jenkins help (none) Setting up triggers (needed) Setting up notifications (needed) Developer docs (wiki) Architecture API reference Tutorials ?? (Some presentations exist, but they re old) 30 10/23/2014 PA1 Confidential
Documentation conversion Conversion of docs to reStructuredText Replace PDF and wiki docs with rst Move all docs under source repository Use sphynx to create multiple formats Publish on readthedocs.io Made some progress Have sphynx templates in place Got stuck on markup conversion Considered automation, but hit some hurdles See http://fuegotest.org/wiki/rst_docs 31 10/23/2014 PA1 Confidential
Marketing Outreach Visibility User growth (slower than expected) 32 10/23/2014 PA1 Confidential
Test server (prototype) Test store (put-test, get-test) Distributed test requests (put-request) Aggregation Results sharing (put-run) Data mining (nothing) 33 10/23/2014 PA1 Confidential
Outline Introduction Feature List Feature Status 34 34 10/23/2014 PA1 Confidential
Feature Status Feature Area Installation Test Execution Results Analysis User Interface Visualization Reports Tests Fuego Distribution Infrastructure Usability Documentation Marketing Test server Status Pretty Good Good Very Good Good Good OK OK Good OK OK OK Good Not Good Potential Improvements Pre-built image, provisioning help Handle timeouts better Report of failure reason ?? Add more chart_config options Integration into dynamic docs More tests Ability to load packages as needed Move to gitlab ?? Need tutorials and walkthroughs ?? Finish implementation, provide server code 35 10/23/2014 PA1 Confidential
Final Thoughts Don t want to focus on negatives Fuego is a very rich, capable, flexible system An amazing amount of features It s often easy to implement a new feature, because of rich existing infrastructure Can usually support what you want to do Very friendly community 36 10/23/2014 PA1 Confidential
Fuego 37 10/23/2014 PA1 Confidential
