16/05/2018 ODI Status Report and Project Overview
"Learn about the Open Data Interface (ODI) database system for space environment data retrieval, processing, and storage. Explore the ODI server software, project maintenance details, and server functionalities for Earth-orbiting spacecraft coordinates and data processing triggers."
Uploaded on | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
16/05/2018 ODI: Open Data Interface (Status report) ESA FP Days, ESTEC D. Heynderickx, DH Consultancy, Belgium P. Wintoft, Solar Analytics, Sweden 1
ODI overview 16/05/2018 Open Data Interface (ODI) is a database system for retrieving, processing and storing space environment (and other) data and metadata in a MySQL (MariaDB) database. Development started in 2008 (Swedish Institute of Space Physics, DH Consultancy, Solar Analytics) Compliant with CDF and SPASE data models Server and client software for data downloading and pre/post- processing, and data retrieval. Currently support for ~150 datasets. Extensible with user added functionality. A licence can be requested at http://space- env.esa.int/index.php/tec-ees-odi-repository-registration.html ESA FP Days, ESTEC 2
Project overview 16/05/2018 Activity title: ODI database maintenance Funding programme: INFRA Contract value: 25 k (4000117649/16/NL/LF/as CCN1) Start and end dates: Nov 2017 Oct 2018 Contractor team: DH Consultancy, Solar Analytics Name of the TO in TEC-EES: Hugh Evans Current developments are marked green in slides ESA FP Days, ESTEC 3
ODI server software (I) 16/05/2018 PHP engine for database communication and process flow Requires read/write access to MySQL database Download and parser scripts NASA/GSFC CDFexport tool to ingest CDF data Python tool to ingest NetCDF data Ingestion of FITS and PDS datasets cron/quartz setup for automatic download and ingestion User may add functionality triggered by hooks on download and ingestion ESA FP Days, ESTEC 4
ODI server software (II) 16/05/2018 Coordinates for Earth orbiting spacecraft Download of TLEs from https://space-track.org Generation of spacecraft coordinates (GEI) using NASA/JPL SPICE library (http://naif.jpl.nasa.gov/) Magnetic coordinates for spacecraft in Earth s magnetosphere UNILIB library for calculation of L, L*, MLT, For fixed or record varying pitch angle(s) IGRF + OPQ field models A single command triggers the whole processing suite (manually or in cron/quartz job), e.g.: php get_ingest.php goes_gp_mag_1m_rt ESA FP Days, ESTEC 5
ODI server software: hooks 16/05/2018 raw_data_dir=GOES/RT/geomag file_name_pattern=*Gp_mag_1m.txt platform=GOES_RTP platform_type=satellite instrument=SEM skeleton_file=Gp_mag_1m.skt parser_file=../../parser/GOES_SEM_RT_TLE.php availability=public download_script=wget_GOES_RT Gp_mag_1m geomag SPACETRACK_satnum=29155 UNILIB_PREFIX= cron_schedule=8 * * * User defined process hooks Scripts (batch, shell or php) started by triggers Pre/post download, ingestion post_ingest.bat example: copy GOES GEI and magnetic coordinates from one dataset to others ESA FP Days, ESTEC 6 @php %ODI_DATASETS%\dataset\goes_rt_coordinates.php goes_gp_mag_1m_rt
ODI client software (I) 16/05/2018 Direct local connection (php/MySQL) requires read access to database Remote connection over HTTP/REST (server/client), JSON output HAPI (NASA Heliospheric API: https://github.com/hapi- server/data-specification) server/client Java SE and MySQL Connector/J JDBC driver APIs for php, Java, IDL, Matlab, Jython, Python Standardised procedure syntax Outputs in language specific objects Excel interface ESA FP Days, ESTEC 7
16/05/2018 Excel interface ESA FP Days, ESTEC 8
Database structure 16/05/2018 ESA FP Days, ESTEC 9
Datasets supported in ODI distribution (~150) 16/05/2018 ACE: archive and real time EPAM, SWEPAM, MAG, SIS DSCOVR: real time IMF and plasma data (JSON streams) GOES: archive and real time SEM data SREM: PROBA1, Integral, GioveB, Rosetta, Herschel, Planck Magnetic and solar indices (Kp, Dst, F10.7, ISN, OMNI, ) Interplanetary particle datasets: HELIOS, IMP8, Voyager, Pioneer, Wind Radiation belt missions: AZUR/EI-88, CRRES/MEA/HEEF/PROTEL, UARS/PEM, SAMPEX/PET, NOAA/POES/SEM2, XMM/ERMD, PROBA-V/EPT, RBSP/HOPE/MAGEIS/REPT/RPS, HIMAWARI/SEDA Proprietary datasets: MIR/REM, STRV1B/REM, AMPTE/UKS, EQUATOR-S, ISEE1/WIM/KED, Meteosat/SEM ESA FP Days, ESTEC 10
ODI applications 16/05/2018 ESA projects SEPEM, SEDAT, SPENVIS, SAAPS, JHelioViewer, HERMES, GALEM SRREM, RENELLA, VALIRENE SSA P2-SWE-II, VSWMC2 (P2-SWE-XIV), P2-SWE- XIII EC FP7 projects SEPServer SPACECAST, SPACESTORM ESA FP Days, ESTEC 11
Current activity: summary 16/05/2018 Updates of the client interfaces Enhancements of the REST interface, new HAPI interface Functionality for ingesting FITS and PDS data files Parsing of SPASE metadata (http://spase-group.org) Support for new datasets (e.g. RBSP/RPS) Software maintenance Dataset maintenance ESA FP Days, ESTEC 12
