Breaking Shackles: GPhL Leading MX Metadata Revolution

ISPyB Status Report for Global Phasing Ltd. Gerard Bricogne & Collaborators Global Phasing Ltd, Cambridge, UK MXCuBE-ISPyB meeting, 20-22 November 2024, ELETTRA

Our position on ISPyB as of ALBA 2023 We are not a synchrotron, we are application developers, the relevant ones in the ISPyB context being the data collection Workflow and the processing programs autoPROC and STARANISO. We do have users, though, but very many of them only see autoPROC results as produced by the synchrotron auto-processing pipelines, archived in the ISPyB backend, and presented by its frontend(s). ISPyB therefore plays a central role in our ability to deliver the best value to our users, both academic and industrial, in a way that is inextricably linked to synchrotron auto-processing facilities. Through the development and deployment of our Workflow, we are also generators of data collection protocols whose execution creates several datasets that are inter-related through high-level metadata. These inter-relations cannot be represented within ISPyB, and the processing by autoPROC of such composite datasets produces a body of results for which no adequate presentation exists so far in the ISPyB frontends this has been a long-standing issue. The outdated one sample -> one dataset -> one processing result paradigm still prevailed

A wind of change at the ALBA meeting The outcome of ESRF s Pilot Project of rebasing ISPyB on ICAT, as announced at the previous (SOLEIL) meeting, had been keenly awaited; but no support for multi-sweep datasets was forthcoming yet. As our work with the group of Ashwin Chari (MPG, Goettingen) at the EMBL-HH P14 beamline (Gleb Bourenkov) using the GPhL Workflow had kept producing an abundance of multi-sweep datasets (today 4,000 and counting) and associated autoPROC results, for the book-keeping of which ISPyB was unusable, we (Rasmus, with Gleb & Ashwin) had produced a hand-crafted facility to do this (Rasmus s talk at ALBA). The idea of an Abstract LIMS , proposed by the MXCuBE Developers, seemed to offer the possibility of preserving the natural structure of data and metadata produced by an experiment and of its processing results. This would also help in presenting as uniform an external appearance as possible to users whose projects involve using multiple synchrotrons. We (GPhL) thought we should seize this opportunity to propose a new LIMS, starting from the definition of a Data Model, that would at last break the shackles of ISPyB.

A nod from the Steering Committee: go for it! (Excerpt of the Committee s Minutes written up by Andy Goetz, ESRF) There was consensus that the comprehensive definition of MX metadata and of how they should be displayed is a topic that interests all partners and that will be at the heart of the continuation of the iCAT-ISPyB Pilot Project presented by ESRF at this meeting. It was agreed that GPhL would lead this activity in the field of rotation MX, on account of (1) GPhL s long-standing but still unfulfilled requests for extensions of ISPyB that could accommodate multi-orientation datasets and their full processing results, (2) the ongoing (or predictable) accumulation of such datasets as a result of the deployment of the GPhL workflow at several MXCuBE-driven beamlines, and (3) GPhL s implementation, with the EMBL-HH P14 and G ttingen MPI teams, of a capability to organise all objects produced by the collection and processing of multi- orientation datasets that could provide a useful reference. Crucially, this activity would closely interact with the Abstract LIMS project within MXCuBE. ACTION:GPhL to organise and lead the definition of metadata and its embodiment in a Data Model. Results of this activity to be presented at the next ISPyB meeting. OUTCOME: The MXLIMS initiative, led by Rasmus.

Details of our reliance on ISPyB et al. (1) Regarding autoPROC+STARANISO (aP+SA for short): optional user input information is not reliably made available to the program in the various auto-processing facilities hence our proposal, at the previous ELETTRA meeting in 2018, that auto-processing at synchrotrons be incorporated into the scope of an extended MXCuBE-ISPyB collaboration aimed at standardising how processing programs are invoked and their results are made accessible to users an added difficulty is that aP+SA produce unique output (anisotropic statistics) whose display is not satisfactorily supported by front-ends user access to full aP+SA results can be whimsical, with decisions made at various synchrotrons being out of our sphere of influence

Details of our reliance on ISPyB et al. (2) Regarding the Global Phasing Workflow (WF for short): this is intended for use in collecting special-quality datasets, not every dataset: it is important that this should be user-selectable, e.g. via the ISPyB Diffraction Plan or equivalent sample-specific information The execution of an on-the-fly designed WF strategy breaks the one sample -> one dataset -> one processing result book-keeping paradigm both for dataset and processing results storage and for the display of these results, beyond the previously mentioned problems for displaying full aP+SA results The implementation of our proposed ISPyB extensions and the standardisation of auto-processing have been blocked by the rigidity of the ISPyB back-end and the lack of resources

Hopes invested in MXLIMS Extensible book-keeping of samples, datasets, processing results, and all associated metadata suitable for archiving and user presentation Seamless connection between experiments and processing, fulfilling our wish for a standardisation of auto-processing. Uniformity in user-level interaction across synchrotrons, from sample shipping (with associated sample-specific info and instructions) to access to datasets, results and graphical summaries. Viable basis for a collaborative activity that would enable an improvement of the scientific basis for the software functionalities. Rasmus has produced a logical data model, created a GitHub repository for it convened a small working group that has held regular VCs. However the ESRF ICAT initiative has led to a wait and see situation until the full extent of the fragmentation of ISPyB-related efforts is clear.

A final remark (from a non-synchrotron!) Does the dichotomy between ISPyB and MXCuBE still make sense? It seems a bureaucratic and managerial artifact (call it legacy) to have separate collaborations, with one of them (MXCuBE) being richly resourced and the other (ISPyB) chronically under-resourced. The meetings are only partially overlapping in terms of timings and attendance. From this perspective, the MXLIMS work would provide an opportunity to highlight how closely the two sets of capabilities in these packages are closely intertwined, and progress towards further abstraction (i.e. to non-MXCuBE BCSs) and rationalization.

Acknowledgements Rasmus Fogh, Peter Keller, Clemens Vonrhein (GPhL) Kate Smith, Ezequiel Panepucci (formerly SLS, soon ANSTO and MAX IV) Karl Levik, James Hall, Eliot Hall, Neeli Katti, Irakli Sikharulidze (DLS) Jacob Oldfield (ANSTO) Alberto Nardella, Carla Takahashi (MAX IV) Edward Daniel (Icebear, Oulu) Max Burian, Pascal Storzbach, Daphne van Dijken (Dectris) Alex de Maria, Marcus Oskarsson, Olof Svensson (ESRF)