Data Description for Integration#
This version of the CDIF recommendations does not contain a full profile for the description of data integration, but this topic has received a lot of attention from the working group. An exploration into different data integration scenarios has been conducted, notably an effort to integrate data from ILO, the World Health Organization (WHO), and the SDG Indicators, with a goal of publishing this data into Google Data Commons and the knowledge graph model used there. This work is on-going, but it has demonstrated the set of metadata needed to fully describe a data integration.
This set of metadata requirements can be summarised as follows, based on the assumption that there is access to the data being integrated:
Detailed data description (variable-level)
Structural metadata of the data sets
Enumerated values (codelists and classifications)
Mappings between sets of enumerated values
Processing description, indicating how mappings were implemented in transformations, and what other operations were performed for data integration
While this seems like a daunting set of information, the exploratory work has shown that if the metadata are available in a sufficiently detailed form, then the actual integration itself is straightforward.
A data provider is expected to describe the data as they manage and present it, along with information about its logical contents. The user can then re-structure the data as needed for their own use, and do so programmatically. Sufficient metadata must be available to support this programmatic restructuring, without losing any of the information about the data - especially its links to semantic definitions.
The concept definitions that specify semantics must be separated from the structural description of data for a useful cross-domain data description scheme, along with an indication of where the semantics for both the field and the values come from.
CDIF recommends a subset of the classes in the DDI-CDI specification for data description. For a static set of data there are four steps. For a service, where the structure and physical format of the data will depend on the service, the last two steps are not required.
The process for providing such detailed descriptions of data can be broken down into a series of steps:
Describe the Data Set or Service: Identify the logical variables in the data, where each ‘variable’ measures a single characteristic of a single unit type, using a consistent set of values. The possible values must be enumerated or otherwise described in a detailed fashion. Representations must be able to identify domain-agnostic semantic descriptions for each possible value, and the variable definitions themselves must similarly be independent of any domain specificity.
Describe the Variables: Indicate how the logical variables fit into the structure of the file, by specifying also any ‘presentational’ variables used for structuring the data serialization in a file. Relationship between presentation and logical variables must be specified. The complete set of variables can then be described as a ‘logical record’.
Describe the Data Structure: Including the fields used to identify a record (the ‘primary key’).
Describe the Physical Format of the Data: Describe the encoding of all variables physically present in the file, and how they are sequenced and stored for programmatic retrieval.