Developing Repository Assessment Framework: Collaboration and Progress Report

Repository Audit and Certification DSA WDS Partnership WG RDA Working Groups Meeting at NIST November 13-14, 2014

Working Group Members Lesley Rickards (UK, PSMSL, WDS-SC) [Co-chair] Mary Vardigan (USA, ICPSR, DSA Board) [Co-chair] Kevin Ashley (UK, Digital Curation Centre) Michael Diepenbroek (Germany, Pangaea, WDS-SC) Ingrid Dillo (The Netherlands, DANS, DSA Board) Fran oise Genova (France, CDS, WDS-SC) Herv L Hours (UK, UK Data Archive, DSA Board) Guoqing Li (China, CEODE, WDS-SC) Jean-Bernard Minster (USA, UCSD, Chair of WDS Scientific Committee) Paul Trilsbeek (The Netherlands, MPI for Psycholinguistics, DSA Board) Eleni Panagou, Ph.D. Candidate in Web Engineering, Democritus University of Thrace, Greece [RDA Early Career Researcher]

Context and Background Data Seal of Approval and World Data System both lightweight mechanisms for repository assessment DSA began in social science and humanities, WDS in natural and physical sciences but both expanding in scope Over past two years, both groups began to see commonalities and synergies When RDA Audit and Certification Interest Group established, exploring a partnership seemed natural

Working Group Goals Develop common catalog of criteria for basic repository assessment and certification Develop common procedures for assessment Implement a shared testbed for assessment Ultimately, create a shared framework for certification that includes other standards as well

Our Work So Far Began virtual meetings early in 2014 to map DSA and WDS criteria to each other Officially recognized as an RDA working group in May 2014 Considered an example for a non-technical group -- RDA is about building bridges In August created a summary mapping with draft common requirements

Procedures for Mapping Created comprehensive Google spreadsheet to have all information in one place Mapped the DSA criteria to the WDS criteria, and the WDS to the DSA Held lengthy discussions on each guideline Group members noted areas of agreement and gaps and documented them

General Findings The two catalogs have similarities and differences DSA guidelines more concise; WDS has multi-part criteria DSA focus on data management, not organizational stability WDS certification includes membership in the WDS and certification of services, not in scope for the DSA Overall, working together has been great

Mapping Summary Shows mappings along with notes on level of the match (good match, partial, gap, etc.) Reconciles the two standards with suggested common language for requirements Assigns a concept to each common requirement, e.g., Discovery, Appraisal, Continuity of Access Assigns ISO/TRAC label(s): Organizational Infrastructure, Digital Object Management, Technology

Mapping Summary Walk-through Summary Document

Context Please provide context for your repository. (1) Repository type (select from a typology -- e.g., domain repository). (2) Brief description of the repository s Designated Community, an identified group of potential Consumers who should be able to understand a particular set of information (from OAIS). (3) Level of curation performed (select from a list). Common Requirement DSA Guideline WDS Criterion Please provide context for your repository. (1) Repository type (select from a typology -- e.g., domain repository). (2) Brief description of the repository's Designated Community, "an identified group of potential Consumers who should be able to understand a particular set of information" (from OAIS). (3) Level of curation performed (select from a list). 3. Which roles(s) do you apply for within WDS? Role(s) and scope within WDS; 10. What is the scientific background of your facility? Please name your specific field(s). 0. Repository context and outsourcing Partial Match

Appraisal The repository accepts data based on defined criteria to ensure relevance and understandability for data users. Common Requirement DSA Guideline WDS Criterion The repository accepts data based on defined criteria to ensure relevance and understandability for data users. [The criteria should explain clearly the philosophy and the technical approach to implement it.] 2. The data producer provides the data in formats recommended by the data repository. 3. The data producer provides the data together with the metadata requested by the data repository. 22. The facility accepts data sets from its producers based on defined criteria for collection, selection, and evaluation. V. Management of data, products, and services Partial Match

Mission/Scope The repository has an explicit mission to provide access to and preserve data in its domain. Common Requirement DSA Guideline WDS Criterion 16.1. The facility has defined: the scope of the data and/or product (services) it offers. IV. Organisational framework. 14. Promote active communication with research community and other users III. General requirements The repository has an explicit mission to provide access to and preserve data in its domain. 4. The data repository has an explicit mission in the area of digital archiving and promulgates it. Partial Match

Documented storage procedures The repository applies documented processes and procedures in managing archival storage of the data. Common Requirement DSA Guideline WDS Criterion The repository applies documented processes and procedures in managing archival storage of the data. 6. The data repository applies documented processes and procedures for managing data storage. 23. Archival storage of the data sets is undertaken to defined specifications. V. Management of data, products, and services Good Match

Preservation plan The repository assumes responsibility for long- term preservation and manages this function in a planned and documented way. Common Requirement DSA Guideline WDS Criterion The repository assumes responsibility for longterm preservation and manages this function in a planned and documented way. 16.2. The facility has defined: its responsibility for the longterm preservation of its data, products and services. IV. Organisational framework 7. The data repository has a plan for long-term preservation of its digital assets. Partial Match

Workflows Archiving takes place according to defined workflows from ingest to dissemination. Common Requirement DSA Guideline WDS Criterion 23. Archival storage of the data sets is undertaken to defined specifications. V. Management of data, products, and services Archiving takes place according to defined workflows from ingest to dissemination. 8. Archiving takes place according to explicit work flows across the data life cycle. Partial Match

Data discovery and identification The repository enables users to discover the data and to refer to them in a persistent way through proper citation. Common Requirement DSA Guideline WDS Criterion 24. The facility permits efficient usage of archived data sets, products and services based on defined criteria and preferably open standards (searchable, accessible, and usable objects and services). V. Management of data, products, and services The repository enables users to discover the data and to refer to them in a persistent way through proper citation. 10. The data repository enables the users to discover and use the data and refer to them in a persistent way. Partial Match

Data integrity and authenticity The repository guarantees the integrity and authenticity of the data. Common Requirement DSA Guideline WDS Criterion 21. The facility ensures integrity and authenticity of data sets during ingest, archival storage, data quality assessment and analysis, product generation, access, and delivery. V. Management of data, products, and services 11. The data repository ensures the integrity of the digital objects and the metadata. 12. The data repository ensures the authenticity of the digital objects and the metadata. The repository guarantees the integrity and authenticity of the data. Good Match

Technical infrastructure The technical infrastructure of the repository supports the tasks and functions necessaryto effectively perform the mission. Common Requirement DSA Guideline WDS Criterion 25. Facility functions on well supported operating systems and other core infrastructural software. VI. Technical infrastructure. 26. Facility is using hardware and software technologies appropriate to the services it provides to its designated community(ies) VI. Technical infrastructure. 27. Security: Technical infrastructure for protection of the facility and its data, products, services, and users. VI. Technical infrastructure 13. The technical infrastructure explicitly supports the tasks and functions described in internationally accepted archival standards like OAIS. The technical infrastructure of the repository supports the tasks and functions necessary to effectively perform the mission. Partial Match

Security The repository maintains a careful plan to protect the safety of its holdings, the security of its facility, and the privacy of its users. OR The repository addresses security needs across its data, systems, personnel, and physical plant. Common Requirement DSA Guideline WDS Criterion The repository maintains a careful plan to protect the safety of its holdings, the security of its facility, and the privacy of its users. OR The repository addresses security needs across its data, systems, personnel, and physical plant. See line above See line above

Licenses The repository maintains all applicable licenses covering data access and use and monitors compliance. Common Requirement DSA Guideline WDS Criterion 14. The data consumer complies with access regulations set by the data repository. 16. The data consumer respects the applicable licences of the data repository regarding the use of the data. The repository maintains all applicable licenses covering data access and use and monitors compliance. [16.4] The facility has defined: the rights of its users to access and use data. IV. Organisational framework Partial Match

Continuity of access The repository has a continuity plan to ensure ongoing access to and preservation of its holdings. Common Requirement DSA Guideline WDS Criterion 9. The data repository assumes responsibility from the data producers for access and availability of the digital objects. 19. Maintenance of a continuity plan in the event of a host institution shift of interests or reaction to substantial changes. IV. Organisational framework The repository has a continuity plan to ensure ongoing access to and preservation of its holdings. Poor Match

Data quality Please provide a description of the mechanism used to ensure (to the largest extent possible) data quality, recognizing that there is a difference between scientific and technical quality. Alternative wordy version: The repository has appropriate internal expertise to address data and metadata quality through assessment of acquisitions, setting quality-related deposit criteria, and enriching data and metadata quality when appropriate to the mission and ensures sufficient information is available for end users to make quality-related evaluations. Common Requirement DSA Guideline WDS Criterion Please provide a description of the mechanism used to ensure (to the largest extent possible) data quality, recognizing that there is a difference between scientific and technical quality. Alternative "wordy" version: The repository has appropriate internal expertise to address data and metadata quality through assessment of acquisitions, setting quality-related deposit criteria, and enriching data and metadata quality when appropriate to the mission and ensures sufficient information is available for end users to make quality-related evaluations. 1. The data producer deposits the data in a data repository with sufficient information for others to assess the quality of the data, and compliance with disciplinary and ethical norms. 12. Have relevant external experts to provide advice and guidance to WDS node.III. General requirements Poor Match

Confidentiality/Ethics When appropriate, the repository protects the subjects of research to the extent possible, taking into account disciplinary norms. Common Requirement DSA Guideline WDS Criterion 1. The data producer deposits the data in a data repository with sufficient information for others to assess the quality of the data, and compliance with disciplinary and ethical norms. 15. The data consumer conforms to and agrees with any codes of conduct that are generally accepted in the relevant sector for the exchange and proper use of knowledge and information. When appropriate, the repository protects the subjects of research to the extent possible, taking into account disciplinary norms. Gap

Open access See statements approved by 2014 ICSU General Assembly. Common Requirement DSA Guideline WDS Criterion 15. Provide full, open, timely, non- discriminatory and unrestricted access to metadata, data, products, and services, no cost or at the Cost of Fulfilling User Request (COFUR). III. General requirements See statements approved by 2014 ICSU General Assrmbly Gap

Organizational infrastructure The organization has adequate funding and sufficient numbers of qualified staff to effectively carry out the mission. Common Requirement DSA Guideline WDS Criterion 17.1 through 17.4. The organizational form is adequate for the facility in terms of: funding; sufficient numbers of qualified staff; organizational structure; long-term planning. The organization has adequate funding and sufficient numbers of qualified staff to effectively carry out the mission. Gap

Scientific guidance The repository adopts mechanism(s) to secure ongoing scientific guidance and feedback from recognized experts, and maintains publicly accessible documentation of such guidance. Common Requirement DSA Guideline WDS Criterion The repository adopts mechanism(s) to secure ongoing scientific guidance and feedback from recognized experts, and maintains publicly accessible documentation of such guidance. 12. Have relevant external experts to provide advice and guidance to WDS node. III. General requirements. Gap

Next Steps Map to Nestor and ISO Finalize the harmonized requirements and put them out to the community as Version 1 Begin to work on aligning procedures Determine relationship of DSA and WDS to each other Create testbed for certification Investigate shared pool of reviewers

Links to Other RDA Groups Practical Policy There may be a way to share policies across repositories and to integrate them into the assessment process (e.g., checks for integrity). Domain Repositories IG This is a natural fit. We can work with the IG to get basic certification on the agenda of repositories and to test our new criteria.

Questions? Mary Vardigan vardigan@umich.edu Lesley Rickards -- ljr@bodc.ac.uk