forked from D-Net/openaire-graph-docs
WIP: authoritative data sourcesag section
This commit is contained in:
parent
797f7c21f7
commit
ec3d23ce88
|
|
@ -8,7 +8,7 @@ OpenAIRE materializes an open, participatory research graph (the OpenAIRE Resear
|
|||
|
||||
## What does OpenAIRE collect?
|
||||
|
||||
OpenAIRE aggregates metadata records describing objects of the research life-cycle from content providers compliant to the [OpenAIRE guidelines](https://guidelines.openaire.eu/) and from entity registries (i.e. data sources offering authoritative lists of entities, like OpenDOAR, re3data, DOAJ, and funder databases). After collection, metadata are transformed according to the OpenAIRE internal metadata model, which is used to generate the final OpenAIRE Research Graph that you can access from the OpenAIRE portal and the APIs.
|
||||
OpenAIRE aggregates metadata records describing objects of the research life-cycle from content providers compliant to the [OpenAIRE guidelines](https://guidelines.openaire.eu/) and from entity registries (i.e. data sources offering authoritative lists of entities, like [OpenDOAR](https://v2.sherpa.ac.uk/opendoar/), [re3data](https://www.re3data.org/), [DOAJ](https://doaj.org/), and various funder databases). After collection, metadata are transformed according to the OpenAIRE internal metadata model, which is used to generate the final OpenAIRE Research Graph, accessible from the [OpenAIRE EXPLORE portal](https://explore.openaire.eu) and the [APIs](https://graph.openaire.eu/develop/).
|
||||
|
||||
The transformation process includes the application of cleaning functions whose goal is to ensure that values are harmonised according to a common format (e.g. dates as YYYY-MM-dd) and, whenever applicable, to a common controlled vocabulary. The controlled vocabularies used for cleansing are accessible at http://api.openaire.eu/vocabularies. Each vocabulary features a set of controlled terms, each with one code, one label, and a set of synonyms. If a synonym is found as field value, the value is updated with the corresponding term.
|
||||
Also, the OpenAIRE Research Graph is extended with other relevant scholarly communication sources that do not follow the OpenAIRE Guidelines and/or are too large to be integrated via the “normal” aggregation mechanism: DOIBoost (which merges Crossref, ORCID, Microsoft Academic Graph, and Unpaywall).
|
||||
|
|
@ -45,9 +45,7 @@ Objects and relationships in the OpenAIRE Research Graph are extracted from info
|
|||
|
||||
OpenAIRE collects metadata records describing objects of the research life-cycle from content providers compliant to the OpenAIRE guidelines and from entity registries (i.e. data sources offering authoritative lists of entities, like OpenDOAR, re3data, DOAJ, and funder databases).
|
||||
|
||||
The OpenAIRE aggregator collects metadata records in the majority of cases via OAI-PMH, but also supports other standard exchange protocols like FTP(S), SFTP, and RESTful API.
|
||||
|
||||
After collection, metadata are transformed according to the OpenAIRE internal metadata model, which is used to generate the final OpenAIRE Research Graph that you can access from the OpenAIRE portal and the APIs.
|
||||
The OpenAIRE aggregator collects metadata records in the majority of cases via [OAI-PMH](https://www.openarchives.org/pmh/), but also supports other standard exchange protocols like FTP(S), SFTP, and some RESTful API.
|
||||
|
||||
For additional details about the aggregation workflows, please refer to [2].
|
||||
|
||||
|
|
|
|||
|
|
@ -3,3 +3,278 @@ sidebar_position: 1
|
|||
---
|
||||
|
||||
# Authoritative data sources
|
||||
|
||||
One of the challenges towards the stability of the contents in the OpenAIRE Research Graph consists of making its identifiers and records stable over time. The barriers to this scenario are many, as the Graph keeps a map of data sources that is subject to constant variations: records in repositories vary in content, original IDs, and PIDs, may disappear or reappear, and the same holds for the repository or the metadata collection it exposes. Not only, but the mappings applied to the original contents may also change and improve over time to catch up with the changes in the input records.
|
||||
|
||||
One of the fronts regards the attribution of the identity to the objects populating the graph. The basic idea is to build the identifiers of the objects in the graph from the PIDs available in some authoritative sources while considering all the other sources as by definition “unstable”. Examples of authoritative sources are Crossref and DataCite. Examples of non-authoritative ones are institutional repositories, aggregators, etc. PIDs from the authoritative sources would form the stable OpenAIRE ID skeleton of the Graph, precisely because they are immutable by construction.
|
||||
|
||||
Such a policy defines a list of data sources that are considered authoritative for a specific type of PID they provide, whose effect is twofold:
|
||||
* OpenAIRE IDs depend on persistent IDs when they are provided by the authority responsible to create them;
|
||||
* PIDs are included in the graph according to a tight criterion: the PID Types declared in the table below are considered to be mapped as PIDs only when they are collected from the relative PID authority data source.
|
||||
|
||||
| *PID Type* | *Authority* |
|
||||
|---|------------------------------------------------------------------------------------|
|
||||
| doi | [Crossref](https://www.crossref.org), [Datacite](https://datacite.org) |
|
||||
| pmc, pmid | [Europe PubMed Central](https://europepmc.org/), [PubMed Central](https://www.ncbi.nlm.nih.gov/pmc) |
|
||||
| arXiv | [arXiv.org e-Print Archive](https://arxiv.org/) |
|
||||
|
||||
There is an exception though: Handle(s) are minted by several repositories; as listing them all would not be a viable option, to avoid losing them as PIDs, Handles bypass the PID authority filtering rule.
|
||||
In all other cases, PIDs are be included in the graph as alternate Identifiers.
|
||||
|
||||
When a record is aggregated from multiple sources considered authoritative for minting specific PIDs, different mappings could be applied to them and, depending on the case,
|
||||
this could result in inconsistencies in the attribution of the field values.
|
||||
To overcome the issue, the intuition is to include such records only once in the graph. To do so, the concept of "delegated authorities" defines a list of datasources that
|
||||
assigns PIDs to their scientific products from a given PID minter.
|
||||
|
||||
This "selection" can be performed when the entities in the graph sharing the same identifier are grouped together. The list of the delegated authorities currently includes
|
||||
|
||||
| *Datasource delegated* | *Datasource delegating* | *Pid Type* |
|
||||
|------------------------------|---------------------------|-----|
|
||||
| [Zenodo](https://zenodo.org) | [Datacite](https://datacite.org) | doi |
|
||||
| [RoHub](https://reliance.rohub.org/) | [W3ID](https://w3id.org/) | w3id |
|
||||
|
||||
## DOIBoost: Crossref, Unpaywall, Microsoft Academic Graph, ORCID
|
||||
|
||||
DOIBoost is a dataset that combines research outputs and links among them from a selection of data sources. It enriches the records available on Crossref with what's available on Unpaywall, Microsoft Academic Graph, ORCID intersecting all those datasets by DOI. As consequence, DOIBoost does not contain any record from MAG, Unpaywall, or ORCID that doesn't provide a DOI available in Crossref.
|
||||
|
||||
The idea behind DOIBoost and its origin can be found in the paper (and related resources) at:
|
||||
|
||||
* La Bruzzo S., Manghi P., Mannocci A. (2019) OpenAIRE's DOIBoost - Boosting CrossRef for Research. In: Manghi P., Candela L., Silvello G. (eds) Digital Libraries: Supporting Open Science. IRCDL 2019. Communications in Computer and Information Science, vol 988. Springer, doi:10.1007/978-3-030-11226-4_11 . Open Access version available at: [10.5281/zenodo.1441071](https://doi.org/10.5281/zenodo.1441071)
|
||||
|
||||
Each Crossref record is enriched with:
|
||||
* ORCID identifiers of authors from ORCID
|
||||
* Open Access instance (with OA color/route and license) from Unpaywall
|
||||
* the following information from MAG:
|
||||
* abstracts
|
||||
* MAG identifiers of authors
|
||||
* affiliation (result - organization) relationships
|
||||
* subjects (MAG FieldsOfStudy)
|
||||
* conference or journal information
|
||||
|
||||
The Open Access status is also set by intersecting the journal information of a record with the journal lists available from DOAJ and the Gold ISSN list.
|
||||
|
||||
### Inputs
|
||||
|
||||
* *Crossref*: dump available to Crossref subscribers via MetadataPlus service, updated once a month.
|
||||
* *Microsoft Academic Graph*: downloaded version on 2021-02-15. We plan to take the latest version in Dec 2021 before MAG will be retired.
|
||||
* *ORCID*: baseline dump obtained in 2020-10-13, regularly updated every week from the [ORCID public API](https://info.orcid.org/documentation/features/public-api).
|
||||
* *Unpaywall*: public database snapshot downloaded in March 2021. Unpaywall updates it twice a year (https://unpaywall.org/products/snapshot)
|
||||
|
||||
The construction of the DOIBoost dataset consists of the following phases:
|
||||
|
||||
### 1 Filtering
|
||||
|
||||
Records in Crossref are ruled out according to the following criteria
|
||||
|
||||
* have blank title, examples:
|
||||
* `10.1093/rheumatology/41.7.837`
|
||||
* `10.1093/qjmed/95.7.430`
|
||||
* `10.1371/journal.pone.0171434.g005`
|
||||
* have one of the following publishers: `"Test accounts"`, `"CrossRef Test Account"`
|
||||
* Examples from https://api.crossref.org/works?query.publisher-name=%22Test%20accounts%22
|
||||
* `10.1007/bf00344543`
|
||||
* `10.1007/bf00186154`
|
||||
* `10.1306/64ed947a-1724-11d7-8645000102c1865d`
|
||||
* have no authors with valid names, where valid means: not blank and different from all strings in this list: `List(",", "none none", "none, none", "none &na;", "(:null)", "test test test", "test test", "test", "&na; &na;")`
|
||||
* Examples for blank authors:
|
||||
* `10.1108/00070709810247807`
|
||||
* `10.1016/s1074-9098(02)00346-5`
|
||||
* `10.1136/heart.88.1.6`
|
||||
* Examples for `"none"` author from https://api.crossref.org/works?query.author=%22none%22
|
||||
* `10.4007/annals.2016.184.3.11`
|
||||
* `10.4007/annals.2012.176.1.6`
|
||||
* `10.2172/6393585`
|
||||
* Examples for `"test"` author from https://api.crossref.org/works?query.author=%22test%22
|
||||
* `10.5116/ijme.54ca.a5ae`
|
||||
* `10.5755/j01.ss.71.2.544`
|
||||
* `10.5755/j01.ee.22.2.319`
|
||||
* have `"Addie Jackson"` as author and `"Elsevier BV"` as publisher (empirically we say they are test records)
|
||||
* Examples from https://api.crossref.org/works?query.author=Addie+Jackson&query.publisher-name=%22Elsevier%20BV%22
|
||||
* `10.2139/ssrn.2082156`
|
||||
* `10.2139/ssrn.2202300`
|
||||
* `10.2139/ssrn.2255657`
|
||||
* have not one of the following values in the field `type` : `"book-section"`, `"book"`, `"book-chapter"`, `"book-part"`, `"book-series"`, `"book-set"`, `"book-track"`, `"edited-book"`, `"reference-book"`, `"monograph"`, `"journal-article"`, `"dissertation"`, `"other"`, `"peer-review"`, `"proceedings"`, `"proceedings-article"`, `"reference-entry"`, `"report"`, `"report-series"`, `"standard"`, `"standard-series"`, `"posted-content"`, `"dataset"`,
|
||||
* Example:
|
||||
* `10.1371/journal.pone.0171434.g005`
|
||||
* `10.7554/elife.21052.049`
|
||||
* `10.1371/journal.pcbi.1005379.s006`
|
||||
|
||||
Records with `type=dataset` are mapped into OpenAIRE results of type dataset. All others are mapped as OpenAIRE results of type publication.
|
||||
|
||||
### 2 Mapping Crossref properties into the OpenAIRE Research Graph
|
||||
|
||||
Properties in OpenAIRE results are set based on the logic described in the following table:
|
||||
|
||||
| OpenAIRE Result field path | Crossref path(s) | Notes |
|
||||
|----------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `id` | `doi` | id in the form `doi_________::md5(doi)` |
|
||||
| `dateofcollection` | `indexed.datetime` | |
|
||||
| `lastupdatetimestamp` | `indexed.timestamp` | |
|
||||
| `type` | `type` | `dataset` if the Crossref type is dataset, `publication` otherwise (based on the filtering logics described above) |
|
||||
| `originalId` | `doi, clinical-trial-number, alternative-id` | |
|
||||
| `pid` | | The scheme tells the type of PID, the value contains the actual value |
|
||||
| `pid.scheme` | | Default value: doi |
|
||||
| `pid.value` | `doi` | The doi is normalised and lower-cased |
|
||||
| `maintitle` | `title` | |
|
||||
| `subtitle` | `subtitle` | |
|
||||
| `author` | `author` | if available the sequence is mapped to rank and the ORCID is also mapped |
|
||||
| `author.name` | `author.given` | |
|
||||
| `author.surname` | `author.family` | |
|
||||
| `author.fullname` | `author.given author.family` | |
|
||||
| `author.rank` | | based on the order, starts from 1 |
|
||||
| `author.pid` | | only if the ORCID is available |
|
||||
| `author.pid.id.scheme` | | Default `'pending_orcid'` (meaning that it is not an id confirmed by ORCID) |
|
||||
| `author.pid.id.value` | `author.ORCID` | |
|
||||
| `author.pid.provenance.provenance` | | Default 'Harvested' |
|
||||
| `author.pid.provenance.trust` | | Default '0.9' |
|
||||
| `description` | `abstract` | |
|
||||
| `subject` | `subject` | with `classid='keywords'`, i.e. no controlled vocabularies for Crossref subjects |
|
||||
| `publicationdate` | `issued.datetime` or, if not available, `created.datetime` | |
|
||||
| `publisher` | `publisher` | |
|
||||
| `source` | `source` | only if the record is not of type `book` |
|
||||
| `source` | concatenation of `container-title.head` + `"ISBN: "` + `ISBN.head` | only if the record is of type @book@ |
|
||||
| `container` | | It is set only for publications with information about the journal it was published in. |
|
||||
| `container.name` | `container-title.head` | |
|
||||
| `container.issnOnline` | `issn-type.value` | if `issn-type.type='electronic'` |
|
||||
| `container.issnPrinted` | `issn-type.value` | if `issn-type.type='print'` |
|
||||
| `container.vol` | `volume` | |
|
||||
| `container.sp` | `page` | before `'-'` |
|
||||
| `container.ep` | `page` | after `'-'` |
|
||||
| `instance` | | One instance is created with the DOI URL |
|
||||
| `instance.accessright` | | Values in `instance.accessright.code` and `instance.accessright.label` are set based on license and dateofacceptance:<br/>- `UNKNOWN`: if the license is blank<br/>- `OPEN ACCESS`: if the license is a CC license or an ACS license or an APA license (considered OPEN also by Unpaywall, see [Unpaywall FAQ](https://support.unpaywall.org/support/solutions/articles/44002063718-what-is-an-oa-license-) for details) or if OUP license, but only after 12 months from the publication date<br/>- `EMBARGO`: OUP license, before 12 months from the publication date<br/>- `CLOSED`: if there is a license not covered by the previous cases |
|
||||
| `instance.accessright.code` | | Code from the [COAR vocabulary for access right](http://vocabularies.coar-repositories.org/documentation/access_rights/) |
|
||||
| `instance.accessright.label` | | One of: `OPEN`, `RESTRICTED`, `CLOSED`, `EMBARGO` |
|
||||
| `instance.accessright.scheme` | | Scheme that defines the code and label, i.e. the URL to the [COAR vocabulary for access right](http://vocabularies.coar-repositories.org/documentation/access_rights/) |
|
||||
| `instance.accessright.openAccessRoute` | | only if `instance.accessright.value = 'OPEN ACCESS'`. Default is `hybrid`. The route is fixed in subsequent phases of DOIBoost, namely when intersecting with Unpaywall and patching the hostedby via DOAJ and the Gold-ISSN list. |
|
||||
| `instance.license` | `license.URL ` | If there is a `license.content-version='vor'`, then this is used. Otherwise the first license entry is used. |
|
||||
| `instance.pid` | | The scheme tells the type of PID, the value contains the actual value |
|
||||
| `instance.pid.scheme` | | Default value: `doi` |
|
||||
| `instance.pid.value` | `doi` | The doi is normalised and lower-cased |
|
||||
| `instance.publicationdate` | `issued.datetime` or, if not available, `created.datetime` | |
|
||||
| `instance.refereed` | | set to `peerReviewed` only if `relation.has-review.id` is not empty, `UNKNOWN` otherwise. |
|
||||
| `instance.type` | `subtype` | mapped using the [OpenAIRE vocabulary for result typologies](https://api.openaire.eu/vocabularies/dnet:result_typologies) |
|
||||
| `instance.url` | `doi` | Full URL of the DOI |
|
||||
|
||||
All other fields of the Json schema not mentioned in the table contain empty values.
|
||||
|
||||
All the records from Crossref are related to the datasource with `name=Crossref` and `id=openaire____::081b82f96300b6a6e3d282bad31cb6e2`
|
||||
|
||||
Possible improvements:
|
||||
* map `clinical-trial-number` and `alternative-id` in `alternateIdentifiers`?
|
||||
* Verify if Crossref has a property for `language`, `country`, `container.issnLinking`, `container.iss`, `container.edition`, `container.conferenceplace` and `container.conferencedate`
|
||||
* Different approach to set the `refereed` field and improve its coverage?
|
||||
|
||||
h3. 2 Map Crossref links to projects/funders
|
||||
|
||||
Links to funding available in Crossref are mapped as funding relationships (`result -- isProducedBy --> project`) applying the following mapping:
|
||||
|
||||
| *funder* | *grant code* | *Link to* |
|
||||
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---|---|
|
||||
| DOI: `{10.13039/100010663, 10.13039/100010661, 10.13039/501100007601, 10.13039/501100000780, 10.13039/100010665}` or name: `'European Union’s Horizon 2020 research and innovation program'` | series of `4-9` digits in `award` | Link to H2020 project |
|
||||
| DOI: `{10.13039/100011199, 10.13039/100004431, 10.13039/501100004963, 10.13039/501100000780}` | series of `4-9` digits in `award` | Link to FP7 project |
|
||||
| DOI: `10.13039/501100000781` OR name: `'European Union's'` | series of `4-9` digits in `award` | Link to FP7 or H2020 project |
|
||||
| DOI: `10.13039/100000001` | `award` | Link to NSF project |
|
||||
| DOI: `10.13039/501100001665` OR name: `{'The French National Research Agency (ANR)', 'The French National Research Agency'}` | `award` | Link to ANR project |
|
||||
| DOI: `10.13039/501100002341` | `award` | Link to Academy of Finland project |
|
||||
| DOI: `10.13039/501100001602` | `award`, removing the initial 'SFI' if present | Link to SFI project |
|
||||
| DOI: `10.13039/501100000923` | `award` | Link to ARC project |
|
||||
| DOI: `10.13039/501100000038` | `award` ignore: we cannot map the project codes in Crossref to project codes in OpenAIRE | Link to NSERC (@unidentified@ project) |
|
||||
| DOI: `10.13039/501100000155` | `award` ignore: we cannot map the project codes in Crossref to project codes in OpenAIRE | Link to SSHRC (@unidentified@ project) |
|
||||
| DOI: `10.13039/501100000024` | `award` ignore: we cannot map the project codes in Crossref to project codes in OpenAIRE | Link to CIHR (@unidentified@ project) |
|
||||
| DOI: `10.13039/501100002848` OR name :`'CONICYT, Programa de Formación de Capital Humano Avanzado'` | `award` | Link to CONICYT project |
|
||||
| DOI: `10.13039/501100003448` | series of `4-9` digits in award | Link to GSRT project |
|
||||
| DOI: `10.13039/501100010198` | `award` | Link to SGOV project |
|
||||
| DOI: `10.13039/501100004564` | series of `4-9` digits in award | Link to MESTD project |
|
||||
| DOI: `10.13039/501100003407` | `award` | Link to MIUR project. Since OpenAIRE has a small subset of MIUR projects, a link to the MIUR funder (@unidentified@ project) is also generated |
|
||||
| DOI: `{10.13039/501100006588, 10.13039/501100004488}` | `award`, removing `'Project No'` and `'HRZZ'` prefix, if present | Link to HRZZ or MZOS project |
|
||||
| DOI: `10.13039/501100006769` | `award` | Link to Russian Science Foundation project |
|
||||
| DOI: `10.13039/501100001711` | `award` after `'_'` and before `'/'` | Link to SNSF project |
|
||||
| DOI: `10.13039/501100004410` | `award` | Link to TUBITAK project |
|
||||
| DOI: `10.10.13039/100004440` or name: `Wellcome Trust Masters Fellowship` | `award` | Link to Wellcome Trust specific project and to the `unidentified` project.|
|
||||
|
||||
### 3 Intersect Crossref with UnpayWall by DOI (DOIBoost1)
|
||||
|
||||
The fields we consider from UnpayWall are:
|
||||
* `is_oa`
|
||||
* `best_oa_location`
|
||||
* `oa_status`
|
||||
|
||||
The results of Crossref that intersect by DOI with UnpayWall records are enriched with one additional `instance` with the following properties:
|
||||
|
||||
| *OpenAIRE Result field path* | *Unpaywall field path* | *Notes* |
|
||||
|---|---|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `instance` | | created only if @is_oa@ and a `best_oa_location` is available |
|
||||
| `instance.accessright` | | default value `Open Access`: we do not add instances if UnpayWall says there is no open version |
|
||||
| `instance.accessright.code` | | Open Access code from the [COAR vocabulary for access right](http://vocabularies.coar-repositories.org/documentation/access_rights/) |
|
||||
| `instance.accessright.label` | | Always `OPEN` |
|
||||
| `instance.accessright.scheme` | | Scheme that defines the code and label, i.e. the URL to the [COAR vocabulary for access right](http://vocabularies.coar-repositories.org/documentation/access_rights/) |
|
||||
| `instance.accessright.openAccessRoute` | `oa_status` | |
|
||||
| `instance.url` | `best_oa_location` | |
|
||||
| `instance.license` | `best_oa_location.license` | |
|
||||
| `instance.pid` | | The scheme tells the type of PID, the value contains the actual value |
|
||||
| `instance.pid.scheme` | | Default value: `doi` |
|
||||
| `instance.pid.value` | `doi` | The doi is normalised and lower-cased |
|
||||
|
||||
For the definition of UnpayWall's @oa_status@ refer to the [Unpaywall FAQ](https://support.unpaywall.org/support/solutions/articles/44001777288-what-do-the-types-of-oa-status-green-gold-hybrid-and-bronze-mean-)
|
||||
|
||||
The record will also feature a relation to the UnpayWall data source: `name="UnpayWall"`, `id=openaire____::8ac8380272269217cb09a928c8caa993`.
|
||||
|
||||
### 4 Intersect DOIBoost1 with ORCID (DOIBoost2)
|
||||
|
||||
The fields we consider from ORCID are:
|
||||
* `doi`
|
||||
* `authors`, a list of authors, each with optional `name`, `surname`, `creditName`, `oid`
|
||||
|
||||
| *OpenAIRE field path* | *ORCID path* | *Notes* |
|
||||
|-------------------------------------|---|-------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `pid` | `doi` | |
|
||||
| `author.name` | `capitalize(name)` | only mapped if not blank |
|
||||
| `author.surname` | `capitalize(surname)` | only mapped if not blank |
|
||||
| `author.fullname` | | if name and surname are not blank, they are concatenated (`capitalize(name) capitalize(surname)`), otherwise we use the `creditName` |
|
||||
| `author.pid` | | only if the `ORCID` is available |
|
||||
| `author.pid.id.scheme` | | Default `orcid` (meaning that it is confirmed by ORCID, (in contrast to the `orcid_pending` set from Crossref and Unpaywall) |
|
||||
| `author.pid.id.value` | `oid` | |
|
||||
| `author.pid.provenance.provenance` | | Default `Harvested` |
|
||||
| `author.pid.provenance.trust` | | Default `0.9` |
|
||||
|
||||
The records are enriched with the ORCID identifiers of their authors.
|
||||
|
||||
[//]: # (TODO: Update with the new approach implemented by Miriam.)
|
||||
|
||||
The current approach is:
|
||||
* if the number of authors from Crossref equals the size of authors from ORCID, then we pick the list of authors with more PIDs and try to enrich it with the PIDs from the other list, based on JaroWrinkler distance on authors' names, surnames, or fullnames, depending on which properties are available;
|
||||
* if the number of authors are different, then we take the longest and try to enrich it with the PIDs from the other author list, based on JaroWrinkler distance on authors' names, surnames, or fullnames, depending on which properties are available
|
||||
|
||||
Miriam will modify the process to ensure that:
|
||||
* the list of authors from Crossred always "win"
|
||||
* the identifiers from ORCID "win"
|
||||
|
||||
### 5 Intersect DOIBoost2 with Microsoft Academic Graph (DOIBoost3)
|
||||
|
||||
*Important Notes*
|
||||
* Only papers with DOI are considered
|
||||
* Since for the same DOI we have multiple version of item with different MAG PaperId, we only take one per DOI (the last one we process). We call this dataset @Papers_distinct@
|
||||
|
||||
When mapping MAG records to the OpenAIRE Research Graph, we consider the following MAG tables:
|
||||
* `PaperAbstractsInvertedIndex`: for the paper abstracts
|
||||
* `Authors`: for the authors. The MAG data is pre-processed by grouping authors by PaperId
|
||||
* `Affiliations` and `PaperAuthorAffiliations`: to generate links between publications and organisations
|
||||
* `Journals` and `ConferenceInstances`: joined with @Papers_distinct@ to have the information about the venues where the paper was published
|
||||
* TO BE REMOVED `PaperUrls`: to create one instance for the OpenAIRE publication
|
||||
* `FieldsOfStudy`: to add subjects
|
||||
|
||||
The records are enriched with:
|
||||
* abstracts
|
||||
* MAG identifiers of authors
|
||||
* affiliation relationships
|
||||
* subjects (MAG FieldsOfStudy)
|
||||
* conference or journal information (in the @journal@ field) TODO: or @container@, in case of the dump?
|
||||
* [TO BE REMOVED] instances with URL from MAG
|
||||
|
||||
### 6 Enrich DOIBoost3 with hosting data sources (`hostedby`) and access right information
|
||||
|
||||
In this phase, we intersect DOIBoost3 with a dataset composed of journals from OpenAIRE, Crossref, and the ISSN gold list. Each journal comes with its International Standard Serial Numbers (`issn`, `eissn`, `lissn`) and, when available, a flag that tells if the journal is open access. The intersection is done on the basis of the International Standard Serial Numbers. The records with a `journal.[l|e]issn` that match are enriched as follows:
|
||||
* Each instance gain the `hostedby` information corresponding to the journal
|
||||
* If the journal is open access, the access rights of the instances are also set to `Open Access` with `gold` route (because by construction, the journals we know are open are from DOAJ or Gold ISSN list)
|
||||
|
||||
The hostedby of records that do not match are set to the `Unknown Repository`.
|
||||
|
|
|
|||
Loading…
Reference in New Issue