Visual Analytic Application Toolset for Large-Scale Heterogeneous Data
This project focuses on developing a Python Dash application for visual analytics, aiming to provide better insights into large-scale heterogeneous data collected by the Army Research Laboratory. The toolset involves methods to acquire, query, and visualize data from a Postgres database, as well as data exploration through Jupyter Notebooks. The project also involves containerizing the Dash app and utilizing R-scripts for data analysis. The goal is to create interactive and user-friendly visualizations to aid in automated vehicle control research within the Army.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
APPROVED FOR PUBLIC RELEASE U.S. ARMY COMBAT CAPABILITIES DEVELOPMENT COMMAND ARMY RESEARCH LABORATORY Visual Analytic Application Toolset for Large-Scale Heterogeneous Data: Python Dash Application for Visual Analytics Mariya Occorso Intern Distribution A Army Research Laboratory 13 Aug 2020 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE About Me Mariya Occorso Arcadia University Undergraduate (Rising Junior) Computer Science Major Member of Women s Lacrosse team Mentor Vincent Perry Intern Collaborators Aiden Sage James Selena 2 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE PROJECT OVERVIEW ExLF- Expedient Leader Follower Army is working towards automated vehicle control A lot of data that needs to be turned into something useful Data Stored on Postgres Not easily accessible A lot of information for singular day Goal: Create a visualization that will offer better insights into the data Dash app Visualize what Aiden s r-scripts/analysis produce Containerize the dash app 3 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE VISUAL ANALYTICS PLATFORM OVERVIEW 4 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE POSTGRES DATABASE Methods: Acquired data through querying postgres database Manually selected tables/schemas Tables/Schemas: Data dictionary Information/explanation on database Based exploration on data dictionary Not a complete picture Comms_analysis 5 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE DATA EXPLORATION Jupyter Notebooks Cleaned and investigated specific columns Converted time stamps Checked for unique values Merged data frames TIRs Comms_analysis Summary Statistics Used in my own plots Explained some patterns 6 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE DATA VISUALIZATION R-Scripts: Aiden s scripts provide analysis Faster than python Python: Dash app coded in pyhton Creates visualizations based on data output Dash App: Interactive Visualization User friendly Adaptable for other data tables Displays everything in one place 7 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE DASH INSPIRATION 8 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE DASH APP OVERVIEW Two main tabs: Data Overview and Data Analysis Various graphs/charts Each build based on functions- flexible Generic framework code Comments document set up and functions Callbacks connect user interface to visualizations 9 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE DASH APP: TAB 1 Data Overview: Overview Graph: Checklist, Radio Items and Dropdown Filter data Multiple columns plotted Parameters supplied by column names Marks failure points Call backs allow for filtration of the options Bar charts: Breakdowns data User insight to run_ids 10 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE DASH APP: TAB 2 Data Analysis: Data provided by Aiden Analysis Graph: User picks parameters for one column Plots data and summary statistics Marks failure points TIRs Graph: Allows user to plot TIR specific values for each parameter 11 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE FUTURE WORK Containerize App: Accessible on Centennial Uses postgres directly Packages already installed Large Scale Data: Multiple data tables within single day Multiple days available for user to choose Optimize App: R-Scripts to do majority of calculations Use Dash Core Component Store to store data locally Improves Rendering times Big Picture: Visualize data and Aiden s analysis in a user friendly format that can be used to determine factors that lead up to a vehicles failure Integrate work of Sage, Selena, and James into dashboard 12 APPROVED FOR PUBLIC RELEASE
APPROVED FOR PUBLIC RELEASE ACKNOWLEDGMENTS I would like to thank the High Performance Computing Modernization Program (HPCMP) and the HPC Internship Program (HIP) for their sponsorship I would like to thank my mentor Vince for this opportunity and guidance throughout the internship I would also like to thank Aiden for all of his help during our internship. Sources: https://dash.plotly.com/https://dash.plotly.com/https://dash.plotly.com/https://dash.plotly.com/ https://github.com/plotly/dash-sample-apps/tree/master/apps/dash-manufacture-spc- dashboardhttps://github.com/plotly/dash-sample-apps/tree/master/apps/dash-manufacture-spc- dashboard https://github.com/plotly/dash-sample-apps/tree/master/apps/dash-oil-and-gas 13 APPROVED FOR PUBLIC RELEASE
