Common Provenance Framework for Multi-organizational Environments (preprint after revision)

CPF Technical Supplementary 1 Rudolf Wittner1,2,3, *, Matej Gallo2, Francesca Frexia4, Simone Leo4, Luca Pireddu4, Cecilia 2 Mascia4, Markus Plass5, **, Niina Eklund3, Stian Soiland-Reyes6,7, Heimo Müller3,5, Jörg 3 Geiger8, and Petr Holub1,2,3 4 1Institute of Computer Science, Masaryk University, Brno, CZ 5 2Faculty of Informatics, Masaryk University, Brno, CZ 6 3BBMRI-ERIC, Graz, AT 7 4CRS4 – Center for Advanced Studies, Research and Development in Sardinia, Pula, IT 8 5Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research 9 Institute of Pathology, Medical University of Graz, Graz, Austria 10 6Department of Computer Science, The University of Manchester, UK 11 7Informatics Institute, University of Amsterdam, Amsterdam, NL 12 8Interdisciplinary Bank of Biomaterials and Data Würzburg (ibdw), University and University 13 Hospital of Würzburg, Würzburg, DE 14 *Correspondence: Rudolf Wittner ([email protected])) 15 *Correspondence: Markus Plass (markus[email protected] 16 Overview 17 This document serves as a supplementary document for the "Traceability of samples and 18 data in multi-organizational environments with the Common Provenance Framework" paper, 19 and provides detailed technical overview of the Common Provenance Framework (CPF). 20 The document is structured as follows: 21 1. Section 1 defines the Common Provenance Model (CPM), an underlying PROV-DM 22 based data model of the CPF. 23 2. Section 2 describes general provenance provenance chain traversal algorithm. 24 3. Section 3 conceptualizes approaches for exchanging described objects, their prove25 nance, and additional provenance metadata (e.g., versioning information or hash of 26 provenance). 27 4. Section 4 describes the implementation of a Provenance Management System, a sys28 tem for storing CPM-compliant provenance. 29 5. Section 5 shows an application of the CPM on a real world use case from digital pathol30 ogy. 31 6. Section 6 shows how a provenance chain can be incrementally built by adding new 32 components into a distributed provenance chain. 33 1 The Common Provenance Model 34 The CPM1,2 is underlying data model of the CPF – a data model for representing prove35 nance information that enables the integration of provenance from various sources. The 36 main goal of the model is to enable the creation of distributed provenance chains across 37 heterogeneous multi-organizational environments, with support for unified Service Oriented 38 1 Architecture (SOA)-based traversal and querying mechanisms, independently from the par39 ticular processes or research objects documented by the provenance. In the context of the 40 CPF, the CPM provides harmonized solution to address the requirements, and offers gen41 eral guidance on how specific requirements can be addressed (e.g., how to link provenance 42 components) in cases when a different data model would be considered to implement the 43 framework. As a result, the CPM directly addresses the traceability of described objects and 44 their provenance in both backward and forward directions by advanced interoperability of 45 distributed provenance pieces, with built-in support for confidentiality and non-repudiation. 46 The CPM is an extension of PROV-DM3, which represents provenance information as a 47 directed graph with semantically annotated nodes and edges. The nodes represent entities, 48 activities, or agents, and the edges represent their mutual relations. The CPM specializes 49 the PROV-DM types (i.e., PROV entities, PROV activities, and PROV agents) and defines 50 constraints on how these should be used in conformance to the context defined by the CPF. 51 The CPM types and attributes are listed in Table 1 and Table 2 respectively. 52 2 Table 1: The table lists core elements of the CPM and their mandatory attributes in a finalized provenance component. The cpm prefix stands for the CPM namespace (https://www. commonprovenancemodel.org/, where the model will be sustained.) CPM Type Type description Attributes Backward connector A connector*that represents a received described object -prov:type: (a mandatory attribute; cpm:backwardConnector is a mandatory value for this attribute) -cpm:referencedBundleId: a mandatory attribute -cpm:referencedMetaBundleId: a mandatory attribute -cpm:referencedBundleSpecV: a mandatory attribute -cpm:referencedmetaBundleSpecV: a mandatory attribute -cpm:referencedBundleHashValue: a mandatory attribute -cpm:hashAlg: a mandatory attribute -cpm:provenanceServiceUri: an optional attribute Forward connector A connector*that represents a described object that can be or has been sent from a sender to a receiver -prov:type: a mandatory attribute; (cpm:forwardConnector is a mandatory value for this attribute) Forward connector specialized A connector*that represents a described object that has been sent from a sender to a receiver -prov:type: a mandatory attribute; (cpm:forwardConnectorSpec is a mandatory value for this attribute) -cpm:referencedBundleId: a mandatory attribute -cpm:referencedMetaBundleId: a mandatory attribute -cpm:referencedBundleSpecV: a mandatory attribute -cpm:referencedmetaBundleSpecV: a mandatory attribute -cpm:provenanceServiceUri: a mandatory attribute -cpm:hashAlg: a mandatory attribute -cpm:referencedBundleHashValue: a mandatory attribute Identifier entity A provenance structure that represents an external identifier -prov:type: a mandatory attribute; cpm:id is a mandatory value for this attribute -cpm:externalId: a mandatory attribute -cpm:externalIdType: a mandatory attribute Main activity A provenance structure that represents the described activity documented in a particular provenance component -prov:type: a mandatory attribute; cpm:mainActivity is a mandatory value for this attribute -cpm:referencedMetaBundleId: mandatory attribute -cpm:referencedmetaBundleSpecV: a mandatory attribute -dct:hasPart: an optional attribute Sender agent A provenance structure that represents a sender in the context of passing a described object between two provenance controllers -prov:type: a mandatory attribute; cpm:senderAgent is a mandatory value for this attribute Receiver agent A provenance structure that represents a receiver in the context of passing a described object between two provenance controllers -prov:type: a mandatory attribute; cpm:receiverAgent is a mandatory value for this attribute Current agent A provenance structure that represents a provenance controller -prov:type: a mandatory attribute; cpm:currentAgent is a mandatory value for this attribute *Aconnector serves as an interconnection between provenance components and represents a described object which can be or has been exchanged between a sender and a receiver. 3 Table 2: The table lists attributes used in the CPM. CPM Attribute Attribute description cpm:externalId External identifier of a described object or a described activity, which is represented by a PROV entity, PROV activity or PROV agent. cpm:externalIdType Type of the external identifier. cpm:hasId An indication whether the PROV entity having this attribute has an external identifier expressed as a standalone entity. cpm:describedObjectType Type of the object represented by the PROV entity. cpm:referencedBundleId Identifier of a PROV bundle referenced by the connector that contains this attribute. cpm:referencedMetaBundleId Identifier of a PROV bundle that contains respective provenance component’s metadata in the standardized representation. cpm:referencedBundleSpecV Identification of the CPM version to which the referenced bundle conforms. cpm: referencedMetaBundleSpecV Identification of the CPM version to which the referenced bundle conforms. cpm:provenanceServiceUri URI of a service where a provenance component can be requested using its identifier. cpm:hashValue A hash value of a described object, which is represented by the PROV entity, presuming that the described object is a digital object and not a physical object. cpm: referencedBundleHashValue A hash value of a PROV bundle referenced by the connector in which the attribute is present. cpm:hashAlg Algorithm used to create a hashValue. cpm:contactIdPid A resolvable persistent identifier dereferencing to actual contact information (e.g. email address) that can be used to contact a person or organization represented by a particular agent. The value of this attribute is a persistent identifier, thus it has the form of a qualified name. 1.1 Provenance Components 53 The CPF distinguishes a provenance component as a manageable unit of provenance infor54 mation, which can be also considered as an intermediate step transformation of provenance 55 information into a finalized provenance component. Each provenance component contain 56 a provenance graph designed according the PROV-DM3. This graph is encapsulated in a 57 PROV bundle4that is assigned a unique identifier, and is valid according to the validation 58 procedure defined in the PROV-CONSTRAINTS5document. 59 As the primary purpose for collecting finalized provenance components is to enable trace60 ability of described objects and related information, a special attention should be put on 61 mechanisms to identify described objects in provenance. In fact, PROV does not stipulate 62 any specific mechanism for identifying documented objects, so that adopters of the standard 63 may determine their own way potentially hindering interoperability of provenance coming 64 from different sources. According to the CPM, a described object is expressed in a prove65 nance component as a PROV entity and its identifier shall be associated with that entity 66 4 using one of the following two mechanisms. 67 1. Using the cpm:externalId attribute. The value of this attribute is the identifier of the 68 described object, and no additional information is associated with this identifier. 69 2. Using the cpm:hasId attribute for PROV entity representing a described object and 70 an alternate PROV entity of type cpm:id. The pattern is shown in Figure 1. This 71 mechanism allows for attachment of additional information related to the identifier, such 72 as identifier type (as an analogy to Content-Type in email headers). Using the type, 73 an application processing such provenance component may later determine means to 74 access the referenced object using the identifier or additional attributes. The purpose 75 of this is to enable harmonised means to refer from a provenance component to an 76 external is defined. 77 described object identifierprov:alternateOf cpm:hasId 'identifier' prov:type 'cpm:id ' cpm:externalId ID_VALUE cpm:externalIdType ID_TYPE Figure 1: A pattern for expression of identifiers with additional information. Each described object can have multiple identifiers, but the mechanisms used to ex78 press the identifier should not be combined with a single described object. The identifier 79 property (entity(id)) of a PROV entity should not be used to express the identifier of of 80 the described object. The only exception is entity representing a provenance component in 81 meta-component. 82 1.2 Finalized Provenance Components 83 Each finalized component consists of two types of information: 1) traversal information that 84 is used for navigation and traversal of a provenance chain; and 2) domainor scenario85 specific information documenting described activities. A simplified schema of a provenance 86 chain with the traversal and domain-specific information is depicted in Figure 2. The domain87 specific information may correspond to a provenance component to which the traversal in88 formation is properly attached during the finalization event. 89 5 Provenance component 2 Traversal information Domain-specific provenance Provenance component 1 Traversal information Domain-specific provenance Provenance component 3 Traversal information Domain-specific provenance Hospital IS Laboratory IS (Electronic) Lab Notebook Biobank IS Software logs Computational workflows Device logs ...Repositories Reference Reference Reference Provenance information in any format Figure 2: Simplified schema of a provenance chain with references to provenance information. 1.2.1 Traversal Information – Basic Structure 90 The general structure of the traversal information is depicted in Figure 3, and forms the 91 core of the presented research. In fact, the main innovative idea was to separate the low92 granular traversal information, and to prescribe its mandatory structure. In other words, 93 the idea requires resulting provenance graph to contain a sub-graph with strictly prescribed 94 structure of derivation paths between graph nodes (PROV entities) that represent traceable 95 inputs (backward connectors) and traceable outputs (forward connectors) of a documented 96 process (main activity). 97 Applying the structure of the traversal information on the examination step (Step 2) in the 98 motivating example, the backward connector can represent a sample, the main activity the 99 examination process, and the forward connector the digital slide produced by the scanning.1100 Each finalized provenance component contains exactly one main activity that should gen101 erate at least one forward connector. The forward connectors represents outputs of the 102 described activity that can be subsequently used by a receiver, and serve as a means to 103 connect subsequent provenance components to the chain. The connection is realized by in104 cluding the PROV entity with the same identifier in the receivers provenance component and 105 providing values for all mandatory attributes of the backward connector, providing the link to 106 sender’s finalized provenance component and respective meta-component. The mechanism 107 is depicted in Figure 4. 108 1Detailed examples of how the CPM is applied to the motivating example are included in the Supplementary document. 6 Figure 3: General structure of the traversal information in a provenance component. connector_id prov:wasDerivedFrom sender:bundleId connector_id prov:type cpm:backwardConnector cpm:referencedBundleId sender:bundleId cpm:referencedMetaBundleId #URI# cpm:referencedBundleHashValue #HASH# cpm:rederencedBundleSpecV #SPEC ID# cpm:rederencedMetaBundleSpecV #SPEC ID# cpm:hashAlg #HASH# receiver:bundleId prov:wasDerivedFrom ReceiverSender Described object Described object provided by a sender to a receiver Represented in the Sender's finalized provenance component as a forward connector. Represented in the Receiver's finalized provenance component as a backward connector. prov:type cpm:forwardConnector Figure 4: A schema of the interconnection between two provenance components with the support for backward traceability. This way, the backward traceability of a described object without the necessity of updating 109 the sender’s finalized provenance component is enabled. A finalized provenance component 110 without a backward connector represents the beginning of a provenance chain (e.g., sample 111 acquisition), as there is no traceable input to represent. 112 To enable the forward traceability of described objects, the forward link is included in 113 the sender’s component. This is technically realized by adding a the specialized forward 114 connector entity. The attributes associated with the specializing entity provide the link to re115 7 ceiver’s finalized provenance component and respective meta-component. The mechanism 116 is depicted in Figure 5. 117 Using this mechanism is not mandatory, as the attribute values of the specialized forward 118 connector may not be known to the sender at the time of the sender’s finalization event, and 119 neither later. An example of this situation could be when a sender makes an open dataset 120 available together with the respective finalized provenance component and does not know 121 who will use or used the dataset (e.g., as the Camelyon dataset6used in the evaluation of 122 the framework). In such scenario, receivers can use the dataset, generate corresponding 123 finalized provenance component, and connect it to the existing forward connector to enable 124 backward traceability. 125 connector_id prov:wasDerivedFrom sender:bundleId ReceiverSender Described object Described object provided from a sender to a receiver Represented in the Sender's finalized provenance component as a forward connector. Represented in the Receiver's finalized provenance component as a backward connector. prov:type cpm:forwardConnector prov:specializationOf connector_id2 prov:type cpm:specForwardConnector cpm:referencedBundleId receiver:bundleId cpm:referencedMetaBundleId #URI# cpm:referencedBundleHashValue #HASH# cpm:rederencedBundleSpecV #SPEC ID# cpm:rederencedMetaBundleSpecV #SPEC ID# cpm:hashAlg #HASH# connector_id prov:type cpm:backwardConnector cpm:referencedBundleId sender:bundleId cpm:referencedMetaBundleId #URI# cpm:referencedBundleHashValue #HASH# cpm:rederencedBundleSpecV #SPEC ID# cpm:rederencedMetaBundleSpecV #SPEC ID# cpm:hashAlg #HASH# receiver:bundleId prov:wasDerivedFrom Figure 5: A schema of the interconnection between two provenance components with the support for both backward and forward traceability. The CPM presumes that all the information necessary to create the links between final126 ized provenance components (component’s id, meta-components id, connector’s id, etc.) 127 are exchanged between organizations as part of their communication protocol (required by 128 the Requirements). No other provenance structures (types of nodes), except the types pre129 sented in this section are allowed in the traversal information. 130 1.2.2 Traversal Information – Extended Structure 131 It may happen that a part of the process involving the described object does not have a 132 corresponding provenance component, because, for example, the organization carrying out 133 8 particular activity does not adhere to the CPF, or it ceased to exist and the respective prove134 nance component is no longer available. For these reasons, the CPM establishes a mech135 anism for connecting components that describe non-adjacent activities, to support tracking 136 precursors and successors of a given described object even in these cases. The mecha137 nism is based on connector redundancy, which consists of adding one (or more) additional 138 backward or specialized forward connectors in a finalized provenance component directly 139 connected to each other by derivation relations, as depicted in Figure 6. Each of these con140 nectors shall be present in an existing finalized provenance component, where it will be used 141 or generated by the corresponding main activity. Forward traceability can be added to the 142 chain by adding respective specialized forward connectors. 143 wasDerivedFrom x:conn_1 y:conn_1 wasGeneratedBy used y:main wasDerivedFrom prov:type cpm:backwardConnector prov:type cpm:forwardConnector prov:type cpm:mainActivity v:conn_1 prov:type cpm:backwardConnector wasDerivedFrom w:conn_1 prov:type cpm:backwardConnector wasDerivedFrom z:conn_1 prov:type cpm:forwardConnector wasDerivedFrom a:conn_1 prov:type cpm:forwardConnector Figure 6: A schema of connectors redundancy to enhance the integrity of a provenance chain with backward traceability, when a provenance controller cease to exist or does not generate finalized provenance information at all. 1.3 Domain-specific Information 144 Domain-specific information can be represented as a PROV graph and attached to the 145 traversal information using the Compound Activities Pattern7, i.e., using entities that are con146 nected to backward and forward connectors using the PROV specialization relation. These 147 entities may then be part of a subgraph connected only in the domain-specific part, as de148 picted in Figure 7. The domain-specific information shall conform to the requirements on 149 provenance components and shall not contain any backward or forward connector, main 150 activity, or sender, current, or receiver agent. Domain-specific provenance information may 151 add details about the main activity (e.g., sub-activities, intermediate or side products, or links 152 to external systems) taking into account the context semantics, existing ontologies, and rel153 evant standards in the specific area. In addition, the level of detail of the domain-specific 154 information may differ from case to case, since it depends on the reasons for which the 155 finalized provenance is being collected. 156 Confidentiality of information in the the traversal information and the domain-specific part 157 may differ. While domain-specific information may contain much more detailed information 158 about a described activity, the traversal information may contain potentially sensitive links to 159 other finalized provenance components. Division of the provenance graph this way enables 160 enforcing the least privilege principle (partially addressing R6). For example, a general 161 request to get information about the source of all the precursors of a given described object 162 does not necessarily need access to highly-granular domain-specific description. Such a 163 query can be resolved only using the traversal information. 164 This approach is aligned with common mechanisms for provenance access control – 165 transformation of a provenance graph into a valid and consistent view appropriate for an 166 9 3.1.1 Attached Scheme 315 In the attached scheme, provenance and its metadata are part of the communication be316 tween organizations. In particular, when the object is exchanged between organizations, the 317 copy of the corresponding provenance and its metadata are also exchanged. This may be 318 done either by providing it as a standalone piece of information outside the object (Figure 10 319 (a)), or by embedding the information directly in the described object (Figure 10 (b)). As a re320 sult, when the process of the object exchange is finished, the receiver has an actual copy of 321 the provenance and its metadata. For instance, an RO-Crate carrying data with provenance, 322 or data formats that include provenance in their header fall within this category. 323 3.1.2 Semi-attached Schemes 324 In the semi-attached scheme, either provenance or its metadata is not part of the communi325 cation between organizations (only one is present). As a consequence, the receiver of the 326 object has to make an additional request to get a copy of the missing part after the object 327 exchange is finished. For instance, an RO-Crate that encapsulate data with a reference to 328 its provenance stored externally fall within the semi-attached scheme. 329 1. Semi-attached scheme with provenance attached: provenance is part of the com330 munication between a sender and a receiver, but the corresponding metadata is not 331 part of the communication (Figure 10 (c)). 332 2. Semi-attached scheme with metadata attached: metadata of provenance is part of 333 the communication between a sender and a receiver, but provenance is not part of the 334 communication (Figure 10 (d)). 335 3.1.3 Detached Scheme 336 In the detached scheme, neither provenance nor metadata of provenance are part of the 337 communication between organizations (Figure 10 (e)). As a consequence, the receiver of 338 the object has to make additional requests to obtain copies of both the provenance and the 339 metadata after the object exchange is finished. For instance, an RO-Crate that encapsulate 340 data with a reference to its provenance and the metadata stored externally to the object 341 fall within the detached scheme (see11 for further details on possible configurations of byte 342 sequences and metadata references in FAIR Digital Objects). 343 3.2 Properties of the Schemes 344 The available literature presents various properties of provenance coupling schemes12,13 345 which is a categorization of how provenance is coupled with described objects while they 346 are stored. In contrast, provenance exchange schemes presented in this manuscript focus 347 on how provenance (and its metadata) is coupled with described objects when they are 348 exchanged. Various properties of provenance coupling schemes are adopted, amended 349 and described in the context of the provenance exchange schemes in Table 3. 350 16 Table 3: The table compares properties of provenance and its metadata in the context of the attached, semi-attached, and the detached scheme. The text in the Table should be interpreted following way: is the property on the given row affected by the selection of an exchange scheme between a sender and a receiver? The properties are described solely in the context of the exchange of the information between a sender and a receiver, not in the context of their subsequent storage by the receiver or potential subsequent receivers. The referenced properties (e.g., “Access control”) were presented in an existing work. The unreferenced properties (e.g. “Accessibility”) are introduced by me. Property of provenance and provenance metadata in a scheme* Attached scheme Semi-attached and detached schemes Accessibility As the actual copy of provenance and its metadata are provided together with the exchanged object, the scheme is less prone to accessibility errors in comparison with the semi-attached and detached schemes. The reason is that no additional request is needed to get the actual copies. As the actual copy of provenance or its metadata** is not immediately provided to the receiver after the exchange of the object is finished, the scheme is more prone to accessibility errors than the attached scheme. The reason is that there is a need to make an additional request to get the actual copies. Confidentiality (by access control16 and encryption***) The attached scheme requires that authorization decision related to provenance and its metadata is made when accessing the exchanged object, as they are provided to a receiver together. Encryption can be used to protect confidential information in the attached scheme, but this would introduce additional complexity related to encryption/decryption keys management and pose additional risks related to keys leakage or keys/scheme deprecation. The semi-attached and the detached scheme separate the moment of making the authorization decision (especially in the case of the separation of access control strategies for exchanged objects and provenance), as the object, provenance, and its metadata are exchanged between a sender and a receiver at different time instances. Interoperability Interoperability should be not affected by the determination of the exchange scheme, as the provenance and its metadata must be interoperable in every scheme so that the receiver can use it. However, the schemes differ in how the interoperability is achieved. If provenance and its metadata is expressed in a standardized way, the receiver can access it directly. Otherwise, the receiver must use an appropriate tool that transform the received information into an intelligible representation. Interoperability should be not affected by the determination of the exchange scheme, as the provenance and its metadata must be interoperable in every scheme so that the receiver can use it. The transformation into the standardized representation may be realized by an API, through which the receiver requests provenance or its metadata, or must translate the result by himself using an appropriate tool. Size & Ease of Distribution Since provenance and its metadata may be relatively big, their inclusion inside the communication between a sender and a receiver may negatively affect the ease of their distribution. Providing provenance or its metadata in a response to a separate request may ease up their distribution. Performance and scalability properties16 of queries over provenance Performance of queries over attached provenance and its metadata may be better, as the information is always already directly accessible to the user. Detaching the information may cause degradation of queries performance in case that the queries run over “remote” information. 17 Physical objects Provenance and its metadata can not be part of physical objects, such as biological samples. For the description of physical objects, an attached scheme with the provenance and its metadata outside the object, a semi-attached scheme, or the detached scheme can be applied. The detached scheme can be used for the description of physical objects. *Particular scheme has no effect on the its feasibility in distributed & heterogeneous environments13,14, as the “distributed” property is fulfilled per se by the fact that the object, its provenance and provenance metadata are exchanged between a sender and a receiver. Availability is not affected by the selection of an exchange scheme, as it is related to the storage and preserving of information, not to the exchange. Each of the Consistency15,16, Integrity, and Non-repudiation (Trustworthiness16) requires verification of some information (e.g., hash, digital signature, trusted timestamp, or other evidence) with respect to received provenance and its metadata. Presuming that all the information is interoperable, available and accessible for the receiver, achieving these properties is not affected by the selection of an exchange scheme. ** Depending on which one is not included in the communication between a sender and a receiver. This comment applies to each “provenance or its metadata” phrase occurrence in this table. *** Access control and encryption17 are common ways of achieving confidentiality of information. 3.3 Implications 351 Once the exchanged object, its provenance, and metadata of provenance or references to 352 them are received and processed by a receiver, the receiver may decide where the respec353 tive provenance record (i.e., finalized provenance component in the CPF) documenting its 354 described activity will be stored, and which appending strategy should be used. 355 In the attached scheme, the two general appending strategies, i.e., versioning or adding 356 provenance pieces to the provenance chain, do not differ significantly. If an object and re357 lated metadata are meant to “travel together” (e.g., as part of the data file header), then a 358 receiver of the object can append it directly, with no regard to which of the two appending 359 strategies is used. On the other hand, if any of the information is referenced in a semi360 attached or detached scheme, the properties of the two appending strategies vary. If a new 361 version is created in the chain, the receiver must be able to append the versioning informa362 tion. On the other hand, if the receiver adds a new provenance piece to the provenance 363 chain, he is not bound to use any specific metadata of the provenance piece. 364 One of the most important questions is whether a receiver of an object should even be 365 able to create a new version of an existing provenance record that was created by another 366 organization. Since a provenance piece document part of a research object’s life cycle when 367 a particular organization handles it, I suggest that this documentation should not be updated 368 by another organization, and suggest enabling the creation of new versions by different or369 ganizations (different from the organization that originally created the finalized provenance 370 component) only in justified cases, e.g., when an organization ceases to exist and an error in 371 provenance is detected later. In this situation, it would be beneficial if another organization, 372 e.g., an authority, could create a corrected version. As a result, the versioning mechanism 373 should not be used to append new information to a chain without additional integrity assur374 ances when an object crosses organizational boundaries. The assurance must guarantee 375 that the new version only appended new content, and that the original provenance content 376 was not modified. 377 Another aspect that affects the determination of the appending strategy is the intended 378 usage of the formats of the exchanged objects. For instance, genomic data and their meta379 data are part of the MPEG-G standard18, which currently covers documentation of steps 380 starting from raw sequence reads up to their alignment to a reference sequence. Each time 381 a new dataset is derived from an MPEG-G file (e.g., a raw genomic dataset is stored as an 382 18 MPEG-G file, and another dataset with aligned reads will be derived from that raw genomic 383 dataset), it is expected that the derived dataset is represented as a new MPEG-G file. In this 384 scenario, the new file is a new object that can intuitively be documented in a new piece of 385 provenance chain. 386 The determination of provenance appending strategy might also be affected by the as387 signment of identifiers for exchanged described objects. For example, Zenodo19, a popular 388 open repository for storing digital research objects, distinguishes identifiers for the objects 389 themselves and for their specific versions. Using such identifiers for the objects might in390 dicate situations when appending a new provenance piece is more appropriate over the 391 provenance versioning mechanism (or vice versa). In particular, when a derived object is 392 assigned the new object identifier, creating a new provenance piece in a chain intuitively 393 seems to be the preferred version. On the contrary, if the described object is assigned an 394 identifier of a new version of another existing object, creating a new version of a provenance 395 piece may be the preferred option. 396 3.4 Further notes 397 An important aspect that must be taken into consideration when adopting a provenance 398 solution is the determination of an appropriate provenance exchange method. This work 399 describes the general properties of the provenance exchange schemes. However, the prop400 erties of the resulting provenance chain are determined by the combination of all prove401 nance exchange schemes between different organizations in a chain and the application 402 of provenance coupling schemes for provenance storage within each organization. For ex403 ample, if the entire workflow adopts the attached provenance exchange scheme and tight 404 coupling12,13 of the object with provenance – e.g., all the provenance and described object 405 is present in an exchanged RO-Crate, which is iteratively appended – each consecutive 406 organization will have access to it. On the other hand, if a detached scheme is used for 407 a single segment of the workflow, an authorized receiver can access related provenance 408 through a reference and can redistribute this provenance on its own, providing access to 409 other organizations in the chain similarly to the attached scheme. This bottleneck cannot 410 be simply prevented by architectural decisions but must be addressed, e.g., by contractual 411 agreements between organizations involved in distributed provenance handling. 412 In the case of the detached scheme, neither the standardized provenance nor its meta413 data is part of the communication between a sender and a receiver. In this scheme, how 414 the references to respective provenance component and meta-component are designed and 415 represented is within the constituency of the exchanged information format or communica416 tion protocol between a sender and a receiver. The method for referencing provenance com417 ponents using the connectors and the main activity’s attribute may serve as a starting point to 418 design representations of references in the communication. They can be potentially reused 419 when designing interlinking provenance and its metadata outside the standardized prove420 nance. For instance, if a given organization provides a described object via /objects/<ID>in 421 a REST API, it could provide (meta)provenance via /(meta)provenance/<ID>. Another option 422 could be to point to the object, provenance component link, and meta-component link from 423 appropriate fields in the JSON object returned by the API endpoint, which would serve as a 424 higher-level wrapper around both the object and the CPM artifacts. 425 19 4 Implementation 426 The CPF Provenance Management System4is a prototype implementation to support stor427 age, management, and access to finalized provenance components as defined in the CPF. 428 The implementation also supports non-repudiation of origin of stored finalized provenance 429 components and their traversal. The main features of the CPF Provenance Management 430 System are: 431 1. Storing and versioning of finalized provenance components; 432 2. Retrieving stored finalized provenance components; 433 3. Maintaining meta-components; 434 4. Support for non-repudiation of origin of stored finalized provenance components; 435 5. Multi-instance deployment of the system; 436 6. Registering and authentication of organizations and their certificates management. 437 The system is implemented as two web services: a Provenance Management Service 438 itself and a Trusted Party Service. Both services are designed as REST API web applications 439 which communicate through https protocol. The services are designed in a way that can be 440 deployed in multiple instances operated by different organizations. 441 4.1 Provenance Management Service 442 In this section, the main functionalities of the Provenance Management Service are de443 scribed. 444 445 Provenance Component Storage. Once a provenance component is finalized during 446 a finalization event, it is ready to be archived and considered immutable. The Provenance 447 Management Service accepts provenance components as an input, which can be serialized 448 into any of the supported formats (PROV-JSON, PROV-XML, rdf). The input must also be 449 digitally signed, and the signature and a timestamp of the document creation must be pro450 vided together with the stored provenance component. The Management Service performs 451 series of checks on the provenance component (e.g., validity of the signature using the 452 users’ certificate or the uniqueness of identifiers), and if these are successful, the compo453 nent is stored and corresponding provenance component metadata (including the extension 454 to support non-repudiation of origin) is generated in the corresponding meta-component. 455 Versioning of a stored provenance component is performed in the same way with the differ456 ence of generated provenance component metadata. 457 458 Provenance Component Retrieval. Once the finalized provenance component is stored, 459 users can retrieve it through the Management Service with various granularities: complete 460 component, just the traversal information, or just the domain-specific part. The component 461 is provided together with its non-repudiation token. If only a subset of the component is pro462 vided (traversal or domain-specific part), a new non-repudiation token is generated for that 463 purpose. 464 465 4The system was developed as part of two master’s theses that were supervised by Rudolf Wittner, the main author of the presented paper. Parts of the text are taken from the master thesis. 20 Modes of Operation. The Provenance Management Service supports three modes of 466 operation for different types of usage in the context of the organizational roles defined in the 467 cpf: 468 1. Provenance controller. Organization hosting the Service is a provenance controller. 469 This provenance controller is allowed to store and retrieve components from the stor470 age, whereas all other organizations are allowed only to retrieve the documents. 471 2. Provenance processor for finalized provenance components’ storage and provision: 472 A provenance processor hosts the service. The provenance processor does not store 473 any provenance components, and other organizations are allowed to store and retrieve 474 provenance components. 475 3. Combined: The organization hosting the service is a provenance controller, and is 476 also providing the service for other organizations. As a result, the controller and other 477 organizations are allowed to store and retrieve provenance components. 478 The storage of the documents is the subject of authorization, which is based on the 479 client certificates stored in the system. It is up to the decision of organizations adopting the 480 framework which mode of operation is the most suitable for particular usage or scenario. 481 4.2 Trusted Party Service 482 The purpose of the Trusted Party service is to support trustworthiness of stored provenance 483 components by providing non-repudiation of origin. The trusted party is supposed to commu484 nicate with the Provenance Storage Service. The Trusted Party service is always assumed 485 to be run by another organization, e.g., an authority. The trusted party service provides the 486 following features. 487 488 Management of client certificates and digital signatures’ verification. The service 489 stores all the certificates used to verify digital signatures of the documents stored in the 490 Storage Service, and checks the signatures. 491 492 Issuing non-repudiation tokens. The token is issued when a signature is verified by 493 the service to provide a proof that the signature was valid. This token is then stored by the 494 Provenance Storage Service in provenance metadata (See the Supplementary document 495 for further details about the token structure), so that users accessing the stored provenance 496 components have confirmation from an authority that the digital signature associated with 497 the component was valid. The token can also be issued for a subset of a provenance com498 ponent, which is aligned with the granularity of access control of the Provenance Storage 499 Service. 500 501 Non-repudiation tokens storage. The Trusted Party Service stores all evidence in case 502 of a potential dispute about the origin of the provenance component. This includes copies of 503 client certificates (and their intermediates), provenance components, and all tokens issued 504 for the components. 505 21 5 Provenance Chain Example 506 507 This section an example of a provenance chain documenting a distributed multi-organizational508 research pipeline from digital pathology. The purpose of this example is to show how the 509 logic defined by the CPM can be applied and its structures used to build up a distributed 510 provenance chain. Finalized provenance components presented in this Appendix support 511 backward traceability and do not support forward traceability. For that reason, no specializa512 tions of forward connectors in the traversal information are shown. The example presents 513 the final state of a provenance chain and does not show how the resulting chain incremen514 tally constructed. Incremental build of a generic provenance chain including the support for 515 forward traceability is shown in Section 6. 516 The traversal information – composed of standardized expression of the interconnections 517 and derivations paths – enables traversal and navigation in a distributed provenance chain 518 using a common algorithm regardless of the kind of described activity documented in a par519 ticular provenance component. The purpose of the traversal information is not to capture 520 any domain-specific semantic documenting particular described activity, the exclusive pur521 pose is to capture traversal/navigation-related information. The domain-specific semantic is 522 attached to the backbone using a standardized method, which provides further details about 523 the component’s main activity. 524 The example provided in this Section focuses primarily on finalized provenance compo525 nents coming from different sources, i.e. the different provenance controllers participating in 526 the considered research process, to build the provenance chain. The example does not pre527 scribe how domain-specific part of finalized provenance components should be expressed 528 and therefore the given representations are used only for illustrative purposes. Domain529 specific modeling, in fact, can be highly variated regarding the different purposes for prove530 nance collection that could lead to the recording of different information. The level of detail 531 of captured provenance information depends on the qualified decision of a user of the CPM. 532 Figures provided in the example use a simplified expression of provenance structures 533 identifiers, meaning that the identifiers are rather descriptive. In reality, fully qualified names 534 composed of a prefix and a local part would be used. The figures also depict the usage of 535 the simplified method for described objects identification. Each of the described activities 536 also contain startTime and endTime property. Proper representation all the information can 537 be found in the technical evaluation implementation of the framework. 538 The presented example considers the motivating example presented in the main manuscript539 – a research pipeline for digital pathology specialized in the detection of cancer. The pipeline 540 consists of multiple steps, each of which can be carried out by different provenance con541 trollers. 542 The general schema of the resulting provenance chain5is depicted in Figure 11. The 543 chain contains five types of finalized provenance components, each documenting a different 544 part of the use case. There may be more “Sample acquisition” and “Sample processing 545 and WSI Generation” bundles in the pipeline. The two “WSI pre-processing” components 546 integrate all the images from the previous steps. The training and testing datasets are then 547 used in the subsequent steps. 548 5The same provenance chain structure was used for the evaluation of the CPF. 22 Figure 11: Overall schema of a provenance chain documenting the running example. In the following sections, each step of the research pipeline is described, and an appli549 cation of the CPM (especially the traversal information part) is explained. 550 5.1 Biological sample acquisition 551 As the first step, a tissue specimen is collected from a patient during a clinical intervention 552 – breast biopsy – and sent to pathology for examination. Finalized provenance component 553 describing the acquisition of a sample is depicted in Figure 12. 554 Figure 12: Provenance component documenting sample acquisition. 23 This provenance component represents the beginning of the chain, so the traversal in555 formation does not contain a backward connector nor a sender agent. The start of the chain 556 is expressed as a main activity, which generates a PROV entity – forward connector – rep557 resenting the acquired sample that will be sent to the next organization for the following 558 step. 559 If there would be a previous step documenting the sample source according to the CPM 560 specifications – e.g., a finalized provenance component documenting patient related clinical 561 data using HL7 Fast Healthcare Interoperability Resources standard (FHIR) – this would be 562 linked using the backward connector and the sender agent provenance structures. 563 5.2 Sample processing and WSI generation 564 In this part of the research pipeline, the Pathology Laboratory receives the sample, prepares 565 it for a consecutive analysis, perform the diagnostics, and generate image data out of the 566 samples. After having passed an initial quality check and being registered in the in the Lab567 oratory Information System, a sequence of laboratory procedures is put in place to produce 568 a collection of glass slides for the case (i.e., a patient). First, the tissue sample is reduced 569 to smaller fragments and placed into labeled bio-cassettes ready for washing cycles and 570 inclusion in paraffin block (Formalin-Fixed Paraffin-Embedded, FFPE). Once cooled, the re571 sulting paraffin block is cut into thin slices which are bathed, placed on a labeled glass slide 572 and finally dried for some time depending on the specific glass preparation protocol. Tissue 573 slices are then stained to enhance the different cellular structures and covered with a cov574 erslip. After a few hours of drying the glass slides are ready to be analyzed, scanned, and 575 annotated for the purpose of training and validation of machine learning models. 576 Finalized provenance component describing the preparation of a glass slide is depicted 577 in Figure 13. 578 24 Figure 13: Provenance component documenting glass slides preparation. The figure is simplified in a way that each sample can be split into multiple blocks, and each block can be split into multiple slices that are put on a glass. All the consecutive activities are then carried out for each glass individually. As a result, the main activity generates multiple WSI connectors, each for a different image generated from a different glass slide. 25 Figure 21: Initial state of a provenance chain. The presented existing component does not include any backward connectors. In general, the component can contain backward connectors, but these would be not used to add a preceding component to the chain, as the addition of a new backward connector is the goal of this example. Figure 22: Creation of a new provenance component to be linked. Respective meta-component content is depicted in Figure 23. 678 Figure 23: State of the corresponding meta-component. 32 At this point, the two independent provenance components can be linked. This is only 679 possible when someone (not necessarily a provenance controller/processor) initiates the 680 linkage. The structure of the traversal information included in the resulting linked provenance 681 components is then dependent on the order of operations, which is affected by which link, 682 either forward or backward link, is created first. Indeed, different options may better fit dif683 ferent scenarios. For instance, consider an open repository with a publicly available dataset 684 which has already been provided and used by users. At this point, the users have already 685 created their provenance component, and the provenance component of the repository is 686 going to be built retrospectively (so we are adding the preceding provenance component to 687 the chain). As the repository is open, presume that the repository provider does not neces688 sarily know who the receivers are. In such a scenario, it may be more feasible that once the 689 receivers learn about the existence of the preceding provenance component (including all 690 the necessary information to create the link), they create the backward links first and they 691 notify the the sender to create the forward links. On the other hand, when a biobank provides 692 samples to organizations, the biobank typically tracks the receivers of the samples. In such 693 a scenario, the biobank may create a provenance component with the forward link first and 694 then notify the receivers to create the backward links. 695 Presume that the backward link is created first (Figure 24), so that the receiver creates a 696 new version of the provenance component with a link to the sender’s component. Respective 697 meta-component content is depicted in Figure 25. 698 Figure 24: Creating a subsequent provenance component. Figure 25: State of the corresponding meta-component. Optionally, a forward link from the Bundle_C_v1 to the Bundle_B_v1 can be created if 699 necessary, information is provided to the receiver (Figure 26). Respective meta-component 700 content is depicted in Figure 27. 701 33 Figure 26: Creating a forward link. Figure 27: State of the corresponding meta-component. 34 Alternatively, the forward link can be also created immediately before creating the back702 ward link in cases, when all the information necessary is already known to the receiver (so 703 that there would be just a single version of the Bundle_C). The other option is that the sender 704 is notified about the existence of the component before the receiver, so that the forward link 705 is created before the backward link (the biobank’s example). Similarly, depending on what 706 information is available to the sender at the moment of the component creation, there could 707 be two versions of the provenance component (without the forward link and another one with 708 the forward link added) or a single version (with the forward link already included). 709 6.3 Adding a subsequent provenance component with redundant con710 nectors without forward traceability 711 This section describes a procedure which adds subsequent provenance components to an 712 existing provenance chain with the support of redundant connectors for backward traceabil713 ity. Meta-component is not provided here, as it just simply follows the examples of meta714 component as already shown in the previous examples. 715 Assume a provenance chain that consists of two provenance components (Figure 28). 716 Assume that each forward connector present in the example is derived from all backward 717 connectors in the provenance components (the derivations are omitted in figures for better 718 readability). 719 Figure 28: A provenance chain consisting of two provenance components. To add a subsequent provenance component to the end of the chain (after z:bundle1), 720 the following procedure is applied: 721 1. Add a subsequent provenance component to the chain by applying the procedure for 722 “Adding a subsequent provenance component”. 723 2. Duplicate all the backward connectors (with respective derivations) from the compo724 nent in the “original” end of the chain (z:bundle1) that are used as a source of deriva725 tion between the forward connector used to connect the new component into the newly 726 added component (the derivation is there by presumption stated in the beginning of 727 this subsection). 728 3. Do not duplicate any other backward connector (this is not shown in this example). 729 By applying the procedure on the assumed provenance chain, the following chain is 730 obtained (Figure 29). 731 35 Figure 29: Adding a provenance component with redundant backward connectors. The green box shows how the backward connectors from the original bundle are duplicated in the newly added component to show the redundant link to the component y:bundle1. To add a subsequent provenance component to the end of the chain (after a:bundle1)732 with redundant connectors, the described procedure is generalized: 733 1. Add a subsequent provenance component to the chain by applying the procedure for 734 “Adding a subsequent provenance component”. 735 2. Duplicate the whole derivation chain of all the backward connectors from the the com736 ponent in the chain (z:bundle1 or a:bundle1 depending on where the subsequent 737 component will be added) that are used as a source of derivation between the forward 738 connector used to connect the new component in the newly added component (it is 739 there by presumption stated in the beginning of this subsection). 740 3. Do not duplicate any other connectors. 741 Application of the generalized procedure (either for adding a new component after the 742 component a:bundle1 and after the component z:bundle1) is presented in Figure 30. The 743 green box shows how the backward connectors from the original bundle are duplicated in 744 the newly added component. 745 Figure 30: Adding provenance components with redundant backward connectors. The blue box shows how the derivation chain of the respective backward connectors from the original bundle are duplicated in the newly added component. Similarly to the previous step, the green box shows how the backward connectors from the original bundle are duplicated in the newly added component. Now, if the component z:bundle1 disappears for some reason, the respective links to the 746 bundle z:bundle1 becomes invalid, however, the redundant links can be used to access the 747 component y:bundle1 from respective provenance components (Figure 31). 748 36 Figure 31: Provenance component z:bundle1 disappears. The red color indicates the invalidity of the connectors that linked to the disappeared component. 6.4 Adding a subsequent provenance with redundant connectors with 749 forward traceability 750 This section describes a procedure which adds subsequent/preceding provenance compo751 nents to an existing provenance chain with the support of redundant connectors for backward 752 and forward traceability. Assume that each forward connector present in the example is de753 rived from all backward connectors in provenance component (the derivations are omitted in 754 figures for better readability). Assume a provenance chain that consists of two provenance 755 components linked in the both directions (Figure 32). 756 Figure 32: Initial state of a provenance chain. Respective meta-component content is Figure 33. 757 37 Figure 33: State of the corresponding meta-component. Another component can be added to the chain using the mechanism for “Adding a subse758 quent provenance component with redundant connectors without forward traceability” (Fig759 ure 34). 760 Figure 34: Adding a provenance component with redundant backward connectors. The green box shows how the backward connectors from the original bundle are duplicated in the newly added component to show the redundant link. 38 Respective meta-component content is depicted in Figure 35. 761 Figure 35: State of the corresponding meta-component. The “middle” component of the chain is now updated to create the forward link to the 762 newly added component. This is realised by applying the mechanism for “Adding a subse763 quent provenance component” to the connector used by the main activity in the newly added 764 component (Figure 36). 765 Figure 36: Creating a forward link, the red box shows the “local change” to create the forward link. Respective meta-component content is depicted in Figure 37. 766 Figure 37: State of the corresponding meta-component. As the respective provenance processor in the middle of the chain now has the informa767 tion about the new component (a:bundle1), the information can be propagated by to other 768 provenance components linked to the “middle” provenance component (y:bundle1v2 in this 769 step) (Figure 38). 770 39 Figure 38: Propagating the information about the forward link to create the redundant forward links. The blue box highlights how the existing forward link in component z:bundle1v2 is propagated to create the redundant link in component y:bundle1v3. Respective meta-component content is depicted in Figure 39. 771 Figure 39: State of the corresponding meta-component. The whole procedure can be repeated to add another subsequent component can be 772 added to the chain (Figure 40). The links between components are omitted for readability. 773 Meta-component would be updated analogously to the presented meta-components ex774 amples – the “revision chain” between respective provenance components would be pro775 longed where necessary. 776 40 Figure 40: Adding another subsequent provenance component to the provenance chain. The blue boxes show the propagation of respective forward link. The green boxes show the propagation of the backward link. 6.5 Forking a provenance component by adding a subsequent or pre777 ceding provenance component with redundant connectors with 778 forward traceability 779 The whole procedure can be also repeated to add another subsequent component to the 780 chain in “non-linear” way. This may happen, for instance, in a situation when the component 781 z:bundle1v3 from the previous example (Figure 40) would represent a dataset deposited in 782 a public repository that has already been used by other organization with generated prove783 nance component. The “non-linear” component then represent a new organization using the 784 dataset with generated provenance component (Figure 41). 785 Figure 41: Adding a “non-linear” provenance component to a provenance chain. The red box depicts the local change in component z:bundle1v3 to create a forward link to the new component. The blue box shows how the forward link is propagated to another provenance component to create a redundant forward link. The blue and red arrows show the new links. The other links between provenance components are omitted for readability. Similarly, the mechanism for “Adding a preceding provenance component” can be used 786 to add a “non-linear” provenance component. Figure 42 depicts how a new preceding com787 ponent d:bundle1 is added before the component a:bundle1v2. The redundant forward 788 connectors are created analogously to the previous steps. 789 41