Innovative Observation Filters in Data Assimilation Integration

The Joint Effort for Data assimilation Integration (JEDI) Observation Filters - Quality Control Joint Center for Satellite Data Assimilation (JCSDA) JEDI Academy - 10-13 June 2019

UFO Observation filters JEDI/UFO calls abstract observation filters before and after the actual operator Filters are written once and used for many observation types Several generic filters already exist - Entirely controlled from yaml configuration file(s) More generic filters will be developed to cover most needs

Observations Processing Flow CostFunction::evaluate CostJo::initialize ObsFilters::ObsFilters Observer::Observer ObsOperator::variables ObsFilters::requiredGeoVaLs CostFunction::runNL Model::forecast Observer::initialize GeoVaLs::GeoVaLs loop over time steps Observer::process State::getValues end loop over time steps Observer::finalize ObsFilters::priorFilter ObsOperator::simulateObs ObsFilters::postFilter CostJo::finalize ObsErrors::ObsErrors ydep=ysimul-yobs Create and configure filters Metadata Gather list of required GeoVaLs Create GeoVaLs Fill GeoVaLs ObsFilters base class manages individual filters (Models and ObsOperators don t need to know). Call prior filters GeoVaLs Simulate observations Call post filters H(x) Setup R, compute Jo

Observation Processing The observation operator is called from the Observer Observation filters are called immediately before/after the operator template <typename MODEL> void Observer<MODEL>::doFinalize() { Log::trace() << "Observer::doFinalize start" << std::endl; filters_->priorFilter(*gvals_); hop_.simulateObs(*gvals_, yobs_, ybias_); filters_->postFilter(yobs_); Log::trace() << "Observer::doFinalize done" << std::endl; } filters_ contains the list of filters and calls them

UFO Observation filters Observation filters are generic and have access to - Observation values and metadata - Model values at observations locations (GeoVaLs) - Simulated observation value (for post-filter) - Their own private data Filters modify - QC flags - Observation error - Their own data (anything passed to the constructor)

UFO Generic Observation Filters PreQC - Handles flags set by pre-processing in IODA BackgroundCheck - Currently based on oonly ( bwill be added) ObsBoundsCheck - Rejects observations values outside some bounds ObsDomainCheck - Rejects observations with metadata values outside some bounds (metadata can be location or any other attribute) BlackList - Unconditionally rejects data for some variables, channels

UFO Generic Observation Filters All filters except the BlackList take an optional where clause to restrict their scope of application - The where clause is very generic and can be expressed in terms of observation metadata, value or GeoVaLs value Filters are controlled by the yaml configuration file - Filters are applied in the order in which they appear in the yaml - Most scientific work happens in the yaml file Another filter (QCmanager) is used internally for book-keeping GeoVaLsWriter (in OOPS) saves GeoVaLs to file is a filter

yaml examples ObsTypes: - ObsSpace: name: Radiosonde ObsDataIn: obsfile: Data/obs/sondes_obs_2018041500_m.nc4 ObsDataOut: obsfile: Data/hofx/sondes_hyb-4dvar-gfs_2018041500_m.nc4 simulate: variables: [eastward_wind, northward_wind, air_temperature] ObsOperator: name: Radiosonde ObsFilters: - Filter: PreQC threshold: 3 apply_at_iterations: 0 - Filter: Background Check variables: [eastward_wind,northward_wind,air_temperature] threshold: 6.0

yaml examples ObsTypes: - ObsOperator: name: CRTM n_Absorbers: 3 n_Clouds: 2 n_Aerosols: 0 Sensor_ID: amsua_n19 EndianType: little_endian CoefficientPath: Data/ ObsSpace: name: amsua_n19 ObsDataIn: obsfile: Data/amsua_n19_obs_2018041500_m.nc4 simulate: variables: [brightness_temperature] channels: 1-15 ObsFilters: - Filter: Background Check variables: [brightness_temperature] channels: 1-5 threshold: 2.0

yaml examples - ObsOperator: name: GnssroRef ObsOptions: use_compress: 1 ObsFilters: - Filter: Domain Check where: - variable: altitude minvalue: 0 maxvalue: 30000 - variable: earth_radius_of_curvature minvalue: 6250000 maxvalue: 6450000 - variable: geoid_height_above_reference_ellipsoid minvalue: -200 maxvalue: 200 - Filter: Background Check variables: - refractivity threshold: 3.0

yaml examples - ObsSpace: name: SeaSurfaceSalinity ObsDataOut: {obsfile: ./Data/sss.out.nc} ObsDataIn: {obsfile: ./Data/sss.nc} simulate: variables: [sea_surface_salinity] ObsOperator: name: SeaSurfaceSalinity ObsFilters: - Filter: Domain Check where: - variable: sea_area_fraction@GeoVaLs minvalue: 0.5 - Filter: Domain Check where: - variable: sea_surface_temperature@GeoVaLs minvalue: 15

Filter Example The Domain Check filter is not very complicated: void ObsDomainCheck::priorFilter(const GeoVaLs & gv) const { const oops::Variables vars(config_.getStringVector("observed")); std::vector<bool> inside = processWhere(obsdb_, gv, config_); for (size_t jv = 0; jv < vars.size(); ++jv) { for (size_t jobs = 0; jobs < obsdb_.nlocs(); ++jobs) { if (!inside[jobs] && flags_[jv][jobs] == 0) { flags_[jv][jobs] = QCflags::domain; } } } }

Observation Filters: Next Developments Thinning (code exists) Buddy check Duplicate check Profile check (inversions ) VarQC Maximum speed of ships, buoys, sondes, aircrafts Force use of @MetaData Save more information in diagnostics file One default yaml file per observation type

Final Comments Think generic!