sandro.labruzzo
/
openaire-graph-docs
forked from D-Net/openaire-graph-docs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

WIP: updated text in the datacite section

This commit is contained in:
Claudio Atzori 2022-10-21 13:44:45 +02:00
parent ae41daf81d
commit 93bad11a04
1 changed files with 36 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

75
docs/data-provision/aggregation/datacite.md
View File

@ -1,69 +1,66 @@
# Datacite # Datacite
This section describes the aggregation workflow of Datacite and the mapping implemented for it. This section describes the aggregation workflow used to gather the bibliographic material from Datacite and the relative mapping.
## Datacite datasource ## Datacite datasource
[Datacite](https://datacite.org/index.html) is a leading global non-profit organisation that provides persistent identifiers (DOIs) for research data and other research outputs. [Datacite](https://datacite.org/index.html) is a leading global non-profit organisation that provides persistent identifiers (DOIs) for research data and other research outputs.
## Datacite API ## Datacite API
The [DataCite REST API](https://support.datacite.org/docs/api) allows users to retrieve, query, and browse DataCite DOI metadata records. In particular, it exposes a method for incremental harvesting new datacite records. The [DataCite REST API](https://support.datacite.org/docs/api) allows users to retrieve, query, and browse Datacite metadata records. In particular, it exposes a method for harvesting new records incrementally.
``` ```
https://api.datacite.org/dois?page[cursor]=$CURSOR&page[size]=$NUMBER_OF_ITEM_PER_PAGE&query=updated:[$FROM_DATE_TIMESAMP TO $TO_DATE_TIMESAMP] https://api.datacite.org/dois?page[cursor]=$CURSOR&page[size]=$NUMBER_OF_ITEM_PER_PAGE&query=updated:[$FROM_DATE_TIMESAMP TO $TO_DATE_TIMESAMP]
``` ```
On this API Request, we introduce some variables: On this API Request, we introduce some variables:
- **CURSOR**: The value of the cursor to iterate the pages - **CURSOR**: The value of the cursor to iterate the pages; the cursor is extracted from each API response and used in the next request.
- **NUMBER_OF_ITEM_PER_PAGE**: (max 1000) defines how many records we can download for each page. - **NUMBER_OF_ITEM_PER_PAGE**: (max 1000) defines how many records must be returned within each API response.
- **FROM_DATE_TIMESAMP, TO_DATE_TIMESAMP** interval timestamp of the updated record - **FROM_DATE_TIMESAMP, TO_DATE_TIMESAMP** interval timestamp of the updated record.
Each record contains two pieces of information needed for incremental harvesting: Each record contains two pieces of information needed for incremental harvesting:
- **isActive**: tells if the record is deleted (`isActive:false`) - **isActive**: tells if the record is deleted (`isActive:false`)
- **updated**: timestamp of last update - **updated**: timestamp of last update
## Collection Workflow ## Collection Workflow
The collection workflow is responsible for aggregating new datacite records. Each record is stored on a table called Native Datacite Store with the following schema: The collection workflow is responsible for aggregating new records. Each record is stored locally on a table with the following schema:
- **DOI**: The DOI PID of the datacite record (It is a primary key) - **DOI**: The DOI of the Datacite record (it is the primary key)
- **update_timestamp**: the last update date timestamp - **update_timestamp**: the last update date timestamp
- **json**: the native record JSON - **json**: the native record JSON
During the collection workflow, we identify the most updated record date, and the collection phase downloads all new datacite records and update the existing one through the API using this date as **FROM_DATE_TIMESAMP** variable. The metadata collection process identifies the most recent record date available locally and uses such date to requests the records to the Datacite API, populating the **FROM_DATE_TIMESAMP** variable. The records in the API response are included in the local storage in upsert mode.
## Datacite Mapping ## Datacite Mapping
The table below describes the mapping from the XML baseline records to the OpenAIRE Graph dump format. The table below describes the mapping from the XML baseline records to the OpenAIRE Graph dump format.
| OpenAIRE Result field path | Datacite record JSON path | # Notes | | OpenAIRE Result field path | Datacite record JSON path | # Notes |
|------------------------------------|-------------------------------|-------------------| |--------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `id` | `\attributes\doi`|the identifier will be created by folloing the openaire PID generation policy | | `id` | `\attributes\doi` | the identifier will be created by folloing the openaire PID generation policy |
| <ul><li>`instance`</li> <li>`instance.type`</li></ul> | <ul><li>`\attributes\types\resourceType`</li> <li> `\attributes\types\resourceTypeGeneral` </li> <li>`attributes\types\schemaOrg`</li></ul> | Use the vocabulary **_dnet:publication_resource_** to find a synonym to one of these terms and get the `instance.type`. Using the **_dnet:result_typologies_** vocabulary, we look up the `instance.type` synonym to generate one of the following main entities: <ul><li>`publication`</li> <li>`dataset`</li> <li> `software`</li> <li>`otherresearchproduct`</li></ul> | | <ul><li>`instance`</li> <li>`instance.type`</li></ul> | <ul><li>`\attributes\types\resourceType`</li> <li> `\attributes\types\resourceTypeGeneral` </li> <li>`attributes\types\schemaOrg`</li></ul> | Use the vocabulary **_dnet:publication_resource_** to find a synonym to one of these terms and get the `instance.type`. Using the **_dnet:result_typologies_** vocabulary, we look up the `instance.type` synonym to generate one of the following main entities: <ul><li>`publication`</li> <li>`dataset`</li> <li> `software`</li> <li>`otherresearchproduct`</li></ul> |
| `pid` | `\attributes\doi` | `scheme = doi` | | `pid` | `\attributes\doi` | `scheme = doi` |
| `originalid` | `\attributes\doi` | | | `originalid` | `\attributes\doi` | |
| `dateofcollection` | `attributes\updated` | the timestamp is defined in milliseconds we convert to "yyyy-MM-dd'T'HH:mm:ssZ" format | | `dateofcollection` | `attributes\updated` | the timestamp is defined in milliseconds we convert to "yyyy-MM-dd'T'HH:mm:ssZ" format |
| `author` | `\attributes\creators` | Each creator field will be mapped in the author entity below the subfield. **If the record has no Creator it will be skipped**| | `author` | `\attributes\creators` | Each creator field will be mapped in the author entity below the subfield. **If the record has no Creator it will be skipped** |
| `author.fullname` | `\attributes\creators\name` | if name is not defined, we construct from given and family name | | `author.fullname` | `\attributes\creators\name` | if name is not defined, we construct from given and family name |
| `author.rank` | | Incremental index starting from 1 | | `author.rank` | | Incremental index starting from 1 |
| `author.name` | `\attributes\creators\givenName` | | | `author.name` | `\attributes\creators\givenName` | |
| `author.surname` | `\attributes\creators\familyName` | | | `author.surname` | `\attributes\creators\familyName` | |
| `author.pid` | `\attributes\creators\nameIdentifiers` | this is a list of pids associated to the creator | | `author.pid` | `\attributes\creators\nameIdentifiers` | this is a list of pids associated to the creator |
| `author.pid.scheme` | `\attributes\creators\nameIdentifiers` | mapping with vocabulary **dnet:pid_types** | | `author.pid.scheme` | `\attributes\creators\nameIdentifiers` | mapping with vocabulary **dnet:pid_types** |
| `author.pid.value` | `\attributes\creators\nameIdentifiers/nameIdentifier` | the pid value | | `author.pid.value` | `\attributes\creators\nameIdentifiers/nameIdentifier` | the pid value |
| `maintitle` | `\attributes\titles` | Titles whose title type is null or title type is Main | | `maintitle` | `\attributes\titles` | Titles whose title type is null or title type is Main |
| `subtitle` | `\attributes\titles` | Titles whose title type is Subtitle since the title type vocabulary in OpenAIRE use the datacite title type vocabulary | | `subtitle` | `\attributes\titles` | Titles whose title type is Subtitle since the title type vocabulary in OpenAIRE use the datacite title type vocabulary |
| **date section** | | for each date in particular for DOI starting with _10.14457_ we Apply a fix thai date convert a date to ThaiBuddhistDate and reformat to local one see ticket [#6791](https://support.openaire.eu/issues/6791) | | **date section** | | for each date in particular for DOI starting with _10.14457_ we Apply a fix thai date convert a date to ThaiBuddhistDate and reformat to local one see ticket [#6791](https://support.openaire.eu/issues/6791) |
|`publicationdate` | `\attributes\dates` | where `dateType` is **issued** | | `publicationdate` | `\attributes\dates` | where `dateType` is **issued** |
|`publicationdate` | `\attributes\publicationYear` | we create this date format `01-01-publicationYear` | | `publicationdate` | `\attributes\publicationYear` | we create this date format `01-01-publicationYear` |
|`embargoenddate` | `\attributes\dates` | where `dateType` is **available** | | `embargoenddate` | `\attributes\dates` | where `dateType` is **available** |
| `subjects` | `\attributes\subject` | `scheme=keywords` | | `subjects` | `\attributes\subject` | `scheme=keywords` |
| `description` | `\attributes\descriptions` | | | `description` | `\attributes\descriptions` | |
| `publisher` | `\attributes\publisher` | | | `publisher` | `\attributes\publisher` | |
| `language` | `\attributes\language` | cleaned by using vocabulary `dnet:languages` | | `language` | `\attributes\language` | cleaned by using vocabulary `dnet:languages` |
| `publisher` | `\attributes\publisher` | | | `publisher` | `\attributes\publisher` | |
| `instance.license` | `\attributes\rightsList` | if right value starts with http and matches a particular regex | | `instance.license` | `\attributes\rightsList` | if right value starts with http and matches a particular regex |
| `instance.accessright` | `\attributes\rightsList` | <ul> <li>if not present :`unknown`</li><li>if datasource is _figshare_:`open`</li><li>If `embargo_date < today()`: _OPEN_ </li> </ul> | | `instance.accessright` | `\attributes\rightsList` | <ul> <li>if not present :`unknown`</li><li>if datasource is _figshare_:`open`</li><li>If `embargo_date < today()`: _OPEN_ </li> </ul> |
### Mapping Relation ### Mapping Relation