add content

.env

@@ -1,2 +0,0 @@
-URL="http://snf-23385.ok-kno.grnetcloud.net"
-BASE_URL="/"

README.md

@@ -9,11 +9,6 @@ $ git clone https://code-repo.d4science.org/D-Net/openaire-graph-docs.git
 
 ## Local installation and deployment
 
-From https://docusaurus.io/docs/installation#requirements 
-> Node.js version 16.14 or above (which can be checked by running node -v) 
-
-
-
 To install the required packages use:
 ```
 $ npm install

docs/data-model/data-model.md

@@ -1,11 +1,11 @@
 # Data model
 
-The OpenAIRE Research Graph comprises several types of [entities](../category/entities) and [relationships](./relationships) among them.
+The OpenAIRE Research Graph comprises several types of entities and [relationships](./relationships) among them.
 
-The latest version of the JSON schema can be found on the [Downloads](../downloads/full-graph) section.
+The latest version of the JSON schema can be found on [Bulk downloads](../download).
 
 <p align="center">
-    <img loading="lazy" alt="Data model" src={require('../assets/img/data-model.png').default} width="80%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
+    <img loading="lazy" alt="Data model" src="/img/docs/data-model.png" width="80%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
 </p>
 
 The figure above, presents the graph's data model. 

docs/data-model/entities/community.md

@@ -37,7 +37,7 @@ _Type: String • Cardinality: ONE_
 Description of the research community/research infrastructure
 
 ```json
-"description": "This portal provides access to publications, research data, projects and software that may be relevant to the Corona Virus Disease (COVID-19). The OpenAIRE COVID-19 Gateway aggregates COVID-19 related records, links them and provides a single access point for discovery and navigation. We tag content from the OpenAIRE Graph (10,000+ data sources) and additional sources. All COVID-19 related research results are linked to people, organizations and projects, providing a contextualized navigation."
+"description": "This portal provides access to publications, research data, projects and software that may be relevant to the Corona Virus Disease (COVID-19). The OpenAIRE COVID-19 Gateway aggregates COVID-19 related records, links them and provides a single access point for discovery and navigation. We tag content from the OpenAIRE Research Graph (10,000+ data sources) and additional sources. All COVID-19 related research results are linked to people, organizations and projects, providing a contextualized navigation."
 ```
 
 ### name

docs/data-model/entities/other.md

@@ -646,12 +646,7 @@ A measure computed for this instance (e.g. those provided by [BIP! Finder](https
 ### key
 _Type: String • Cardinality: ONE_
 
-The specified measure. Currently supported one of: 
-* `influence` (see [PageRank](/data-provision/indicators-ingestion/impact-scores#pagerank-pr))
-* `influence_alt` (see [Citation Count](/data-provision/indicators-ingestion/impact-scores#citation-count-cc))
-* `popularity` (see [AttRank](/data-provision/indicators-ingestion/impact-scores#attrank))
-* `popularity_alt` (see [RAM](/data-provision/indicators-ingestion/impact-scores#ram))
-* `impulse` (see ["Incubation" Citation Count](/data-provision/indicators-ingestion/impact-scores#incubation-citation-count-icc))
+The specified measure. Currently supported one of: `{ influence, influence_alt, popularity, popularity_alt, impulse, cc }` (see [the dedicated page](../../data-provision/enrichment/impact-scores) for more details).
 
 ```json
 "key": "influence"

docs/data-model/entities/result.md

@@ -311,7 +311,7 @@ _Type: [Subject](other#subject) • Cardinality: MANY_
 Subject, keyword, classification code, or key phrase describing the resource.
 
 ```json
-"subjects": [
+"subjecsts": [
     {
         "provenance": {
             "provenance": "Harvested",

docs/data-model/pids-and-identifiers.md

@@ -70,5 +70,5 @@ Currently, the following data sources are used as "PID authorities":
 | arXiv     | `arXiv_______`      	  | arXiv.org e-Print Archive             	 |
 | handle    | `handle______`      	  | any repository                        	 |
 
-OpenAIRE also perform duplicate identification (see the [dedicated section for details](/data-provision/deduplication)).
+OpenAIRE also perform duplicate identification (see the [dedicated section for details](../../data-provision/deduplication/)).
 All duplicates are **merged** together in a **representative record** which must be assigned a dedicated OpenAIRE identifier (i.e. it cannot have the identifier of one of the aggregated record).

docs/data-provision/aggregation/aggregation.md

@@ -4,17 +4,34 @@ sidebar_position: 1
 
 # Aggregation
 
-OpenAIRE materializes an open, participatory research graph (the OpenAIRE Research Graph) where products of the research life-cycle (e.g. scientific literature, research data, project, software) are semantically linked to each other and carry information about their access rights (i.e. if they are Open Access, Restricted, Embargoed, or Closed) and the sources from which they have been collected and where they are hosted. The OpenAIRE Research Graph is materialised via a set of autonomic, orchestrated workflows operating in a regimen of continuous data aggregation and integration. [1]
+OpenAIRE materializes an open, participatory research graph (the OpenAIRE Research graph) where products of the research life-cycle (e.g. scientific literature, research data, project, software) are semantically linked to each other and carry information about their access rights (i.e. if they are Open Access, Restricted, Embargoed, or Closed) and the sources from which they have been collected and where they are hosted. The OpenAIRE research graph is materialised via a set of autonomic, orchestrated workflows operating in a regimen of continuous data aggregation and integration. [1]
 
 ## What does OpenAIRE collect?
 
-OpenAIRE aggregates metadata records describing objects of the research life-cycle from content providers compliant to the [OpenAIRE guidelines](https://guidelines.openaire.eu/) and from entity registries (i.e. data sources offering authoritative lists of entities, like [OpenDOAR](https://v2.sherpa.ac.uk/opendoar/), [re3data](https://www.re3data.org/), [DOAJ](https://doaj.org/), and various funder databases). After collection, metadata are transformed according to the OpenAIRE internal metadata model, which is used to generate the final OpenAIRE Research Graph, accessible from the [OpenAIRE EXPLORE portal](https://explore.openaire.eu) and the [APIs](https://graph.openaire.eu/develop/).
+OpenAIRE aggregates metadata records describing objects of the research life-cycle from content providers
+compliant to the [OpenAIRE guidelines](https://guidelines.openaire.eu/) base on the
+[OpenAIRE Content Acquisition Policies](https://doi.org/10.5281/zenodo.1446408)
+from 2018. And from entity registries (i.e. data sources offering authoritative lists of entities,
+like [OpenDOAR](https://v2.sherpa.ac.uk/opendoar/), [re3data](https://www.re3data.org/),
+[DOAJ](https://doaj.org/), [DRIS](https://dspacecris.eurocris.org/cris/explore/dris) from
+[euroCRIS](https://www.openaire.eu/openaire-and-eurocris-sign-a-memorandum-of-understanding), and
+various funder databases).
 
-The transformation process includes the application of cleaning functions whose goal is to ensure that values are harmonised according to a common format (e.g. dates as YYYY-MM-dd) and, whenever applicable, to a common controlled vocabulary. The controlled vocabularies used for cleansing are accessible at [api.openaire.eu/vocabularies](https://api.openaire.eu/vocabularies/). Each vocabulary features a set of controlled terms, each with one code, one label, and a set of synonyms. If a synonym is found as field value, the value is updated with the corresponding term. 
-In addition, the OpenAIRE Research Graph is extended with other relevant scholarly communication sources that need special handling, either because they do not strictly follow the OpenAIRE Guidelines or due to the vast amount of data of data they offer (e.g. DOIBoost, that merges Crossref, ORCID, Microsoft Academic Graph, and Unpaywall).
+After collection, metadata are transformed according to the OpenAIRE internal metadata model, which is
+used to generate the final OpenAIRE Research Graph, accessible from the
+[OpenAIRE EXPLORE portal](https://explore.openaire.eu) and the [APIs](https://graph.openaire.eu/develop/).
+
+The transformation process includes the application of cleaning functions whose goal is to ensure 
+that values are harmonised according to a common format (e.g. dates as YYYY-MM-dd) and, whenever applicable, 
+to a common controlled vocabulary. The controlled vocabularies used for cleansing are accessible at 
+[api.openaire.eu/vocabularies](https://api.openaire.eu/vocabularies/). 
+Each vocabulary features a set of controlled terms, each with one code, one label, and a set of synonyms. 
+If a synonym is found as field value, the value is updated with the corresponding term. 
+Also, the OpenAIRE Research Graph is extended with other relevant scholarly communication sources that do 
+not follow the OpenAIRE Guidelines and/or are too large to be integrated via the “normal” aggregation mechanism: DOIBoost (which merges Crossref, ORCID, Microsoft Academic Graph, and Unpaywall).
 
 <p align="center">
-    <img loading="lazy" alt="Aggregation" src={require('../../assets/img/aggregation.png').default} width="65%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
+    <img loading="lazy" alt="Aggregation" src="/img/docs/aggregation.png" width="65%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
 </p>
 
 The OpenAIRE aggregation system collects information about objects of the research life-cycle compliant to the [OpenAIRE acquisition policy](https://www.openaire.eu/content-acquisition-policy) from [different types of data sources](https://explore.openaire.eu/search/find/dataproviders):
@@ -32,7 +49,7 @@ Relationships between objects are collected from the data sources, but also auto
 
 Objects and relationships in the OpenAIRE Research Graph are extracted from information packages, i.e. metadata records, collected from data sources of the following kinds:
 
-- *Literature, Institutional and thematic repositories*: Information systems where scientists upload the bibliographic metadata and full-texts of their articles, due to obligations from their organization or due to community practices (e.g. ArXiv, Europe PMC);
+- *Institutional or thematic repositories*: Information systems where scientists upload the bibliographic metadata and full-texts of their articles, due to obligations from their organization or due to community practices (e.g. ArXiv, Europe PMC);
 - *Open Access Publishers and journals*: Information system of open access publishers or relative journals, which offer bibliographic metadata and PDFs of their published articles;
 - *Data archives*: Information systems where scientists deposit descriptive metadata and files about their research data (also known as scientific data, datasets, etc.).;
 - *Hybrid repositories/archives*: information systems where scientists deposit metadata and file of any kind of scientific products, incuding scientific literature, research data and research software (e.g. Zenodo)
@@ -46,13 +63,13 @@ Objects and relationships in the OpenAIRE Research Graph are extracted from info
 OpenAIRE collects metadata records describing objects of the research life-cycle from content providers compliant to the OpenAIRE guidelines and from entity registries (i.e. data sources offering authoritative lists of entities, like OpenDOAR, re3data, DOAJ, and funder databases).
 
 The OpenAIRE aggregator collects metadata records in the majority of cases via [OAI-PMH](https://www.openarchives.org/pmh/), but also supports other standard exchange protocols like FTP(S), SFTP, and some RESTful API.
-The whole list of available and used collectors could be found in the [RedMine Wiki - API Protocols](https://support.openaire.eu/projects/openaire/wiki/API_protocols)
 
 For additional details about the aggregation workflows, please refer to [2].
 
 
+The whole list of available and used collectors could be found in the [RedMine Wiki - API Protocols](https://support.openaire.eu/projects/openaire/wiki/API_protocols).
 ## References
 
-[1] Manghi, P., Artini, M., Atzori, C., Bardi, A., Mannocci, A., La Bruzzo, S., Candela, L., Castelli, D. and Pagano, P. (2014), “The D-NET software toolkit: A framework for the realization, maintenance, and operation of aggregative infrastructures”, Program: electronic library and information systems, Vol. 48 No. 4, pp. 322-354. [doi:10.1108/prog-08-2013-0045](http://doi.org/10.1108/prog-08-2013-0045)
+[1] Manghi P. et al. (2014) "The D-NET software toolkit: A framework for the realization, maintenance, and operation of aggregative infrastructures", Program, Vol. 48 Issue: 4, pp.322-354, [10.1108/PROG-08-2013-0045](https://doi.org/10.1108/PROG-08-2013-0045)
 
-[2] Atzori, C., Bardi, A., Manghi, P., & Mannocci, A. (2017, January). "The OpenAIRE workflows for data management". In Italian Research Conference on Digital Libraries (pp. 95-107). Springer, Cham. [doi:10.1007/978-3-319-68130-6_8](https://doi.org/10.1007/978-3-319-68130-6_8)
+[2] Atzori, Claudio, Bardi, Alessia, Manghi, Paolo, & Mannocci, Andrea. (2017). The OpenAIRE workflows for data management. Zenodo. [10.5281/zenodo.996006](http://doi.org/10.5281/zenodo.996006)

docs/data-provision/aggregation/collector.md

@@ -0,0 +1,548 @@
+# Collectors - API protocols
+
+The section is a collection from the [Redmine Wiki](https://support.openaire.eu/projects/openaire/wiki/API_protocols) page and describes the API protocols that are supported by D-NET for the collection of metadata and full-texts.
+We also give a description of the plugins associated to the API protocols for metadata collection. Plug-ins for the full-text download are described in the wiki section [[Fulltext_Download]].
+
+The living document is at https://support.openaire.eu/projects/openaire/wiki/API_protocols
+
+## Protocols for metadata collection
+
+### OAI
+
+* Protocol name: oai
+* Environment:
+  * non-Hadoop: PROD+BETA+DEV
+  * Hadoop: PROD+BETA+DEV
+* Plugin class: eu.dnetlib.data.collector.plugins.oai.OaiCollectorPlugin
+* Plugin description: Harvests records from an OAI-PMH endpoint
+* Support incremental collection: Yes
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** format: OAI metadataPrefix (mandatory)
+** set: list of sets to be harvested (optional). When not provided all records are harvested.
+
+
+### OAI sets
+
+* Protocol name: oai_sets
+* Plugin class: eu.dnetlib.data.collector.plugins.oaisets.OaiSetsCollectorPlugin
+* Plugin description: Collects metadata about OAI sets available from an OAI-PMH endpoint
+* Support incremental collection: No
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+### HTTP
+
+* Protocol name: http
+* Plugin class: eu.dnetlib.data.collector.plugins.HttpCollectorPlugin
+* Plugin description: Collects metadata records from one XML file available at a remote location. The plugin will split the input file into several XML records based on the splitOnElement parameter. For file available from the local file system see protocol `file`. For file located in the web application classpath see protocol `classpath`. For the collection of distinct files from a remote folder see `httpList` and `httpWithFileName`.
+* Support incremental collection: No
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** splitOnElement: name of the XML field that identifies the root of each records in the input file (mandatory). Example: for an input file like @<container><record><a>aaa</a></record><record><a>2222</a></record></container>@ the splitOnElement parameter must be set to `record`. Please note that the value must not be an xpath, but the name of the element. This plugin will not work properly if the element can be repeated inside the same XML record, as in :  @<container><record><a>aaa</a><record>I am inside another record field</record></record></container>@
+
+### HTTP LIST
+
+* Protocol name: httpList
+* Available since: Feb 2015
+* Plugin class: eu.dnetlib.data.collector.plugins.httplist.HttpListCollectorPlugin
+* Plugin description: Collects metadata records from a remote endpoint when the path of each record is provided via a text file (one by line). In other words, the data source has an "index file" where the paths to the metadata records are listed one per line. The plugin reads each line of the index file and downloads from the URL provided in each line. The index line can also contain a partial URL as the baseURL is used as prefix to each line in the index file.
+* Support incremental collection: No
+* Supported file formats: XML for the metadata records, textual file for the list of locations of the records.
+* Plugin parameters:
+** baseUrl: base URL to construct the final URLs to the metadata records to download (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** listUrl: URL to the the index file listing the locations (mandatory)
+
+### HTTP WITH FILENAME
+
+* Protocol name: HTTPWithFileName
+* Available since: May 2018
+* Plugin class: eu.dnetlib.data.collector.plugins.httpfilename.HTTPWithFileNameCollectorPlugin
+* Plugin description: Collects metadata records from a remote HTTP endpoint where the available XML files are listed in a standard index file. 
+* Support incremental collection: No
+* Supported file formats: XML, json
+* Plugin parameters:
+** baseUrl: URL to the list of files  (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory). In case of json format, refer to the xpath that is valid after the conversion from json to xml via the org.json library.
+** filter : MIRIAM???
+
+### FILE
+
+* Protocol name: file
+* Plugin class: eu.dnetlib.data.collector.plugins.FileCollectorPlugin
+* Plugin description: Collects metadata records from one XML file available at a local location on the file system. The plugin will split the input file into several XML records based on the splitOnElement parameter. For file available from a remote location see protocol `http`. For file located in the web application classpath see protocol `classpath`. For collecting all files in a folder see protocol `filesystem`.
+* Support incremental collection: No
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** splitOnElement: name of the XML field that identifies the root of each records in the input file (mandatory). Example: for an input file like @<container><record><a>aaa</a></record><record><a>2222</a></record></container>@ the splitOnElement parameter must be set to `record`. Please note that the value must not be an xpath, but the name of the element. This plugin will not work properly if the element can be repeated inside the same XML record, as in :  @<container><record><a>aaa</a><record>I am inside another record field</record></record></container>@
+
+### FILESYSTEM
+
+* Protocol name: filesystem
+* Plugin class: eu.dnetlib.data.collector.plugins.filesystem.FilesystemCollectorPlugin
+* Plugin description: Collects metadata records from one folder the file system. Each file must be a single metadata record. In case of json files, they are collected and transformed in XML. For the collection of one XML file containing several metadata records see protocol `file`. For the collection of files from archives see protocols `zip`, `tarGz`.
+* Support incremental collection: No
+* Supported file formats: XML, json
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory). In case of json format, refer to the xpath that is valid after the conversion from json to xml via the org.json library.
+** extensions: comma separated list of extensions. (optional) 
+** fileFormat: xml or json. Default to xml. (optional)
+** setObjIdentifierFromFileName: true if you want the dnet identifier to be forged based on the file name instead of the ID XPath.
+
+### CLASSPATH
+
+* Protocol name: classpath
+* Plugin class: eu.dnetlib.data.collector.plugins.ClasspathCollectorPlugin
+* Plugin description: Collects metadata records from one XML file available in the classpath of the web application. The plugin will split the input file into several XML records based on the splitOnElement parameter. For file available from a remote location see protocol `http`. For file available from the local file system see protocol `file`.
+* Support incremental collection: No
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** splitOnElement: name of the XML field that identifies the root of each records in the input file (mandatory). Example: for an input file like @<container><record><a>aaa</a></record><record><a>2222</a></record></container>@ the splitOnElement parameter must be set to `record`. Please note that the value must not be an xpath, but the name of the element. This plugin will not work properly if the element can be repeated inside the same XML record, as in :  @<container><record><a>aaa</a><record>I am inside another record field</record></record></container>@
+
+### File CSV (deprecated, use HTTP CSV instead)
+
+* Protocol name: fileCSV
+* Plugin class: eu.dnetlib.data.collector.plugins.FileCSVCollectorPlugin
+
+### HTTP CSV
+
+* Protocol name: httpCSV
+* Plugin class: eu.dnetlib.data.collector.plugins.HttpCSVCollectorPlugin
+* Plugin description: Reads one CSV file with header and generate one XML metadata record per line. Lines with invalid quotes are skipped.
+* Support incremental collection: No
+* Supported file formats: csv, tsv
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** separator: string used to separate columns (optional). When empty `\t` is assumed.
+** identifier: name of the column where the value to be used to forge dnet identifier for each record can be found (mandatory)
+** quote: char used in the CSV for quoting (optional). When empty, no quotation char is considered.
+
+### EXCEL
+
+* Protocol name: excelFile
+* Available since: June 2017
+* Plugin class: eu.dnetlib.data.collector.plugins.excel.ReadExcelPlugin
+* Plugin description: Collects metadata records from one sheet of an excel file with headers. The data is converted from excel to csv and the same classes used by `httpCSV` are used.
+* Support incremental collection: No
+* Supported file formats: Excel with header
+* Plugin parameters: *@MIRIAM can you please add param descriptions below?*
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** argument: 
+** header_row:
+** file_to_save
+** remove_empty_lines
+** remove_lines_with_id
+** col_id
+** tmp_file
+** remove_tmp_file
+** sheet_number
+** separator
+** quote
+
+### FTP (S)
+
+* Protocol name: ftp
+* Plugin class: eu.dnetlib.data.collector.plugins.ftp.FtpCollectorPlugin
+* Plugin description: Collects XML metadata records with a given extension from an FTP site with login authentication. The plugin can be configured to: (i) go recursively into subfolders; (ii) collect files with multiple extensions.
+* Support incremental collection: Yes
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint, must include the path on the SFTP site where to start the collection (mandatory) 
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** username and password: for SFTP authentication (mandatory)
+** recursive: set to true to enable recursion. False to only visit the folder identified by baseUrl. (mandatory)
+** extensions: comma separated list of extensions. (mandatory)
+
+### SFTP
+
+* Protocol name: sftp
+* Available since: March 2016
+** Authentication via private key supported since June 2022
+* Plugin class: eu.dnetlib.data.collector.plugins.sftp.SftpCollectorPlugin
+* Plugin description: Collects XML metadata records with a given extension from an SFTP site with login or public/private key authentication. The plugin can be configured to: (i) go recursively into subfolders; (ii) collect files with multiple extensions.
+* Support incremental collection: Yes
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint, must include the path on the FTP site where to start the collection (mandatory) 
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** username : for authentication (mandatory) 
+** authMethod: method for the authentication (optional, default value "login". For private key authentication enter the value "key")
+** password: mandatory when using the default authentication method (authMethod is blank or "login")
+** privateKeyPath: mandatory when using the private key authentication method (authMethod is "key") - it is the path to the file system where to find the private key (ppk file)
+** recursive: set to true to enable recursion. False to only visit the folder identified by baseUrl. (mandatory)
+** extensions: comma separated list of extensions. (mandatory)
+
+### REST
+
+* Protocol name: rest_json2xml
+* Available since: Dec 2017
+* Plugin class: eu.dnetlib.data.collector.plugins.rest.RestCollectorPlugin
+* Plugin description: Collects metadata records from a REST endpoint. It features a number of parameters to support its adoption for the collection for almost any REST API, regardless their peculiarities for handling pagination and returning json or XML.
+* Support incremental collection: No
+* Supported file formats: Json, XML
+* Plugin parameters: 
+    e.g. https://v2.sherpa.ac.uk/cgi/retrieve?api-key=12345-67890-ABCD&format=Json&item-type=repository&offset=0&order=id&limit=10
+** baseUrl: base URL of the endpoint (mandatory), e.g. _https://v2.sherpa.ac.uk/cgi/retrieve_
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory), e.g. _//*[local-name()='system_metadata']/*[local-name()='id']_
+** resumptionType: {scan, count, discover, page, deep-cursor} (mandatory), e.g. _discover_
+*** scan: evaluated the resumptionXPath
+*** count: evaluate the resultTotalXPath
+*** discover: unknown of total results
+*** page: for paging items
+*** deep-cursor (>= v1.4.0/2020): handling of Solr cursor (like on CrossRef), _queryParams_ must have '&cursor=*'
+** resumptionParam: argument key (mandatory), e.g. _offset_
+** resumptionXpath: used by scan, e.g. ___
+** resultTotalXpath: used by count, e.g. _//*[local-name()='system_metadata']/*[local-name()='id'][1]_
+** resultFormatParam: argument key, e.g. _format_
+** resultFormatValue:  (mandatory), value of the parameter _resultOutputParam_ of the REST request e.g. _Json_
+** resultOutputFormat: {json,xml} (optional) when not specified it defaults to the value specified in _resultFormatValue_. It is expected, in that case, that _resultFormatValue_ is set to _json_ or _xml_. (param added in February 2021 to support use cases like the one of the EU Open Data Portal: it returns json but the request should be made with the param _output_format=standard_) 
+** resultSizeParam: argument key (mandatory), e.g. _limit_
+** resultSizeValue: argument value >=2 , empty input field -> default is _100_
+** queryParams: more specific argument like _api-key_, _item-type_, _order_, ...
+** entityXpath: xslt entity path (mandatory), e.g. _//*[local-name()='items']_
+** authMethod: {bearer} (optional), together with *authToken*
+** authToken: [string] (optional), actually only in combination with _authMethod: bearer_ because on use of the HTTP-Header authorization method.
+
+> The *Param(s) arguments are depended on the API endpoint.
+
+### Mongo Dump
+
+* Protocol name: mongoDump
+* Available since: Feb 2015
+* Plugin class: eu.dnetlib.data.collector.plugins.mongo.MongoDumpPlugin
+* Plugin description: Collects metadata records from a mongo dump stored in the local file system. The dump must be generated from a D-Net mdstore. This plugin is useful when we have collected a very big collection of records in one infrastructure (e.g. BETA) and we want to add it to another infrastructure (e.g. PRODUCTION) without having to rely on the external data source again.
+* Support incremental collection: No
+* Supported file formats: mongo dump in json format generated from a D-Net mdstore. To be more precise, each json object must contain an element called `body` containing the XML metadata record in a format compatible with D-NET (i.e. an OAI record with `dri:*` elements in the header).
+* Plugin parameters:
+** baseUrl: path to the dump on the local file system (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory but ignored, as the dnet identifier is assumed to be already in the XML records.)
+
+Note that this protocol is not yet been registered in the OpenAIRE infrastructures.
+
+### TARGZ
+
+* Protocol name: targz
+* Available since: Jan 2015
+* Plugin class: eu.dnetlib.data.collector.plugins.archive.targz.TarGzCollectorPlugin
+* Plugin description: Collects metadata records from a tar gz archive located in the local file system. Each file in the archive is assumed to be one XML metadata record
+* Support incremental collection: No
+* Supported file formats: targz archive containing XML files
+* Plugin parameters:
+** baseUrl: path to the archive in the local file system (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+### ZIP
+
+* Protocol name: zip
+* Available since: Jan 2015
+* Plugin class: eu.dnetlib.data.collector.plugins.archive.zip.ZipCollectorPlugin
+* Plugin description: Collects metadata records from a zip archive located in the local file system. Each file in the archive is assumed to be one XML metadata record
+* Support incremental collection: No
+* Supported file formats: zip archive containing XML files
+* Plugin parameters:
+** baseUrl: path to the archive in the local file system (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+
+### GZIP
+
+* Protocol name: fileGzip
+* Plugin class: eu.dnetlib.data.collector.plugins.FileGZipCollectorPlugin
+* Plugin description: Collects metadata records from a gzipped file located in the local file system. The plugin will split the input file into several XML records based on the splitOnElement parameter.
+* Support incremental collection: No
+* Supported file formats: XML file in gz format
+* Plugin parameters:
+** baseUrl: path to the file in the local file system (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+** ** splitOnElement: name of the XML field that identifies the root of each records in the input file (mandatory). Example: for an input file like @<container><record><a>aaa</a></record><record><a>2222</a></record></container>@ the splitOnElement parameter must be set to `record`. Please note that the value must not be an xpath, but the name of the element. This plugin will not work properly if the element can be repeated inside the same XML record, as in :  @<container><record><a>aaa</a><record>I am inside another record field</record></record></container>@
+
+<pre><code class="html">
+Note: HADOOP fileGzip plugin: eu.dnetlib.dhp.collection.plugin.file.FileGZipCollectorPlugin
+
+availability: DEV and BETA
+
+The GZIP plugin for the HADOOP environment has a slightly different naming in labels but the functionality should be the same.
+
+</code></pre>
+
+### schema.org/Dataset
+
+* Protocol name: Schema.org
+* Available since: November 2018
+* Plugin class: eu.dnetlib.data.collector.plugins.schemaorg.SchemaOrgPlugin
+* Plugin description: Collects schema.org/Dataset records published in accessible html pages. Supported microformats are JSON-LD. In a high level description, the plugin is comprised of the following modules:
+** Repository - A Java Iterable that iterates over an external repository, collecting available html web page endpoints that must be parsed to extract schema.org/Dataset records
+** Endpoint Access - A Java Iterator that polls the repository queue for enspoints to process. It identifies the JSON-LD records containing schema.org/Dataset description and extracts them
+** Dataset Mapping - A Java iterator that retrieves the extracted schema.org/Dataset entries and maps them to OpenAIRE compatible dataset xml descriptions
+* Supported Repository protocols:
+** sitemapindex - Given a sitemapindex.xml file, the plugin will access sequentially all sitemap files and extract the endpoints to later be processed by the transformation pipeline
+** HTTP API listing - Given a GET URL, the plugin will sequentially retrieve page by page all listed entries, extract endpoints from the result and provide them for processing to the transformation pipeline
+*** The HTTP API listing process is sensitive to the exposed repository API. Currently one implementation targeting the Keggle (https://www.kaggle.com/) API is provided
+* Support incremental collection: No
+* Supported file formats:
+** Repository Access - sitemaps & http API for listing records
+** Repository Document endpoints - html web pages
+** Dataset records - schema.org/Dataset microformat
+* Plugin parameters:
+** Repository Agnostic
+*** consumerBlockPolling: (Boolean) Whether the consumer threads should block waiting for next endpoint to be retrieved from the repository or not. If false, the final consumer should handle waiting exterenally. If false, the final consumer should handle NoSuchElementException exception if no endpoint was retrieved within the configured timeout. The value defaults to true if left empty
+*** consumerBlockPollingTimeout: (Long) The amount of units (see consumerBlockPollingTimeoutUnit) to block waiting for an endpoint to be retrieved from the repository. The value defaults to 2 if left empty
+*** consumerBlockPollingTimeoutUnit: (java.util.concurrent.TimeUnit) The unit to measure the timeout in (consumerBlockPollingTimeout). The value defaults to TimeUnit.MINUTES if left empty
+*** endpointCharset: (java.nio.charset.Charset) The Charset to use reading the html web page. The value defaults to UTF-8 if left empty
+*** updatedDateFormat: (String) the format to use when parsing the "dateModified" property. The value defaults to "YYYY-MM-DD" if left empty
+*** createdDateFormat: (String) the format to use when parsing the "dateCreated" property. The value defaults to "YYYY-MM-DD" if left empty
+*** publicationDateFormat: (String) the format to use when parsing the "datePublished" property. The value defaults to "YYYY-MM-DD" if left empty
+*** contributorFallbackType: (DatasetDocument.Contributor.ContributorType) the contributor type to fall back to for identified contributors for which no direct inference can be made on their contribution type. The value defaults to "Other" if left empty
+*** identifierMappingARK: (CSV String) a coma separated list of strings that if found in an Identifier @PropertyValue data type should indicate that the identifier value should be classified as an ARK identifier.  The value defaults to "null" if left empty
+*** identifierMappingDOI: (CSV String) a coma separated list of strings that if found in an Identifier @PropertyValue data type should indicate that the identifier value should be classified as an DOI identifier.  The value defaults to "null" if left empty
+*** identifierMappingHandle: (CSV String) a coma separated list of strings that if found in an Identifier @PropertyValue data type should indicate that the identifier value should be classified as an Handle identifier.  The value defaults to "null" if left empty
+*** identifierMappingPURL: (CSV String) a coma separated list of strings that if found in an Identifier @PropertyValue data type should indicate that the identifier value should be classified as an PURL identifier.  The value defaults to "null" if left empty
+*** identifierMappingURN: (CSV String) a coma separated list of strings that if found in an Identifier @PropertyValue data type should indicate that the identifier value should be classified as an URN identifier.  The value defaults to "null" if left empty
+*** identifierMappingURL: (CSV String) a coma separated list of strings that if found in an Identifier @PropertyValue data type should indicate that the identifier value should be classified as an URL identifier.  The value defaults to "null" if left empty
+*** identifierFallbackType: (DatasetDocument.Identifier.IdentifierType) the fallback type to use as the type of identifier in case no other inference can be made. The value defaults to "null" if left empty
+*** identifierFallbackURL: (Boolean) a value indicating that if no identifier could be retrieved, the URL property (if available) should be used as the identifier. The value default to "true" if left empty
+** Sitemap Repository
+*** repositoryAccessType: (String) the directive to use to drive the repository access plugin to use the sitemap index methodology. The value must be set to "sitemapindex"
+*** baseUrl: (String) the url where the sitemapindex.xml file is available
+*** sitemap_queueSize: (Integer) the size of the blocking queue that will hold the extracted endpoints from the repository and will then be used to feed the transformation pipeline
+*** sitemap_IndexCharset: (java.nio.charset.Charset) The Charset to use reading the sitemap index file. The value defaults to UTF-8 if left empty (this is also the sitemap specification value)
+*** sitemap_FileCharset: (java.nio.charset.Charset) The Charset to use reading the individual sitemap files. The value defaults to UTF-8 if left empty (this is also the sitemap specification value)
+*** sitemap_FileSchema: (SitemapFileIterator.Options.SitemapSchemaType) The schema of the individual sitemap files. Can be one of Text or Xml as per the sitemap specification. The value default to "Xml" if left empty
+*** sitemap_FileType: (SitemapFileIterator.Options.SitemapFileType) The schema of the individual sitemap files. Can be one of Text or GZ as per the sitemap specification. The value default to "Text" if left empty
+*** sitemap_producerBlockPollingTimeout: (Long) The amount of units (see sitemap_producerBlockPollingTimeoutUnit) to block waiting for a slot in the queue to put a retrieved endpoint. The value defaults to 20 if left empty
+*** sitemap_producerBlockPollingTimeoutUnit: (java.util.concurrent.TimeUnit) The unit to measure the timeout in (sitemap_producerBlockPollingTimeout). The value defaults to TimeUnit.MINUTES if left empty
+** HTTP API (Kaggle) Repository
+*** repositoryAccessType: (String) the directive to use to drive the repository access plugin to use the sitemap index methodology. The value must be set to "httpapi-kaggle"
+*** baseUrl: (String) the base url that should be used to construct the individual dataset endpoint url given the API listing response for each page
+*** httpapi-kaggle_queueSize: (Integer) the size of the blocking queue that will hold the extracted endpoints from the repository and will then be used to feed the transformation pipeline
+*** httpapi-kaggle_APICharset: (java.nio.charset.Charset) The Charset to use reading the API response. The value defaults to UTF-8 if left empty
+*** httpapi-kaggle_APICharset: (java.nio.charset.Charset) The Charset to use reading the API response. The value defaults to UTF-8 if left empty
+*** httpapi-kaggle_queryUrl: (String) The query GET url to call to access pages of results. The query string should contain a page placeholder token to facilitate paged retreival of results
+*** httpapi-kaggle_queryPagePlaceholder: (String) The query GET url page placeholder token. The value defaults to "{PAGE}" if left empty
+*** httpapi-kaggle_responsePropertyTotalDataset: (String) The API response json property that holds the number of total records available. The value defaults to "totalDatasetListItems" if left empty
+*** httpapi-kaggle_responsePropertyDatasetList: (String) The API response json property that holds the list of page results. The value defaults to "datasetListItems" if left empty
+*** httpapi-kaggle_responsePropertyDatasetUrl: (String) The API response json property that holds the url fragments for each dataset in the response. The value defaults to "datasetUrl" if left empty
+*** httpapi-kaggle_responseBaseDatasetUrl: (String) The base url that should be used to construct the individual dataset endpoint url given the API listing response for each page. The value defaults to the value of the "baseUrl" configuration property if left empty
+*** httpapi-kaggle_producerBlockPollingTimeout: (Long) The amount of units (see httpapi-kaggle_producerBlockPollingTimeoutUnit) to block waiting for a slot in the queue to put a retrieved endpoint. The value defaults to 20 if left empty
+*** httpapi-kaggle_producerBlockPollingTimeoutUnit: (java.util.concurrent.TimeUnit) The unit to measure the timeout in (httpapi-kaggle_producerBlockPollingTimeout). The value defaults to TimeUnit.MINUTES if left empty
+
+Usage example
+<pre><code class="java">
+public static void main(String[] args) throws Exception {
+	ConsoleAppender console = new ConsoleAppender();
+	console.setLayout(new PatternLayout("%d [%p|%c|%C{1}] %m%n"));
+	console.setThreshold(Level.DEBUG);
+	console.activateOptions();
+	Logger.getLogger("eu.dnetlib.data.collector.plugins").addAppender(console);
+
+	HashMap<String,String> params = new HashMap<>();
+	params.put("consumerBlockPolling", Boolean.toString(true));
+	params.put("consumerBlockPollingTimeout", "2");
+	params.put("consumerBlockPollingTimeoutUnit", TimeUnit.MINUTES.toString());
+	params.put("endpointCharset", StandardCharsets.UTF_8.name());
+	params.put("updatedDateFormat", "YYYY-MM-DD");
+	params.put("createdDateFormat", "YYYY-MM-DD");
+	params.put("publicationDateFormat", "YYYY-MM-DD");
+	params.put("contributorFallbackType", DatasetDocument.Contributor.ContributorType.Other.toString());
+	params.put("identifierFallbackType", null);
+	params.put("identifierFallbackURL", Boolean.toString(true));
+	params.put("identifierMappingARK", "ark, ARK");
+	params.put("identifierMappingDOI", "doi, DOI");
+	params.put("identifierMappingHandle", "Handle, HANDLE");
+	params.put("identifierMappingPURL", "purl, PURL");
+	params.put("identifierMappingURN", "urn, URN");
+	params.put("identifierMappingURL", "url, URL");
+
+	params.put("repositoryAccessType", "sitemapindex");
+	params.put("sitemap_queueSize", "100");
+	params.put("sitemap_IndexCharset", StandardCharsets.UTF_8.name());
+	params.put("sitemap_FileCharset", StandardCharsets.UTF_8.name());
+	params.put("sitemap_FileSchema", SitemapFileIterator.Options.SitemapSchemaType.Text.toString());
+	params.put("sitemap_FileType", SitemapFileIterator.Options.SitemapFileType.GZ.toString());
+	params.put("sitemap_producerBlockPollingTimeout", "20");
+	params.put("sitemap_producerBlockPollingTimeoutUnit", TimeUnit.MINUTES.toString());
+	//params.put("repositoryAccessType", "httpapi-kaggle");
+	//params.put("httpapi-kaggle_queueSize", "100");
+	//params.put("httpapi-kaggle_APICharset", StandardCharsets.UTF_8.name());
+	//params.put("httpapi-kaggle_queryUrl", "https://www.kaggle.com/datasets_v2.json?sortBy=updated&group=public&page={PAGE}&pageSize=20&size=sizeAll&filetype=fileTypeAll&license=licenseAll");
+	//params.put("httpapi-kaggle_queryPagePlaceholder", "{PAGE}");
+	//params.put("httpapi-kaggle_responsePropertyTotalDataset", "totalDatasetListItems");
+	//params.put("httpapi-kaggle_responsePropertyDatasetList", "datasetListItems");
+	//params.put("httpapi-kaggle_responsePropertyDatasetUrl", "datasetUrl");
+	//params.put("httpapi-kaggle_responseBaseDatasetUrl", "https://www.kaggle.com");
+	//params.put("httpapi-kaggle_producerBlockPollingTimeout", "20");
+	//params.put("httpapi-kaggle_producerBlockPollingTimeoutUnit", TimeUnit.MINUTES.toString());
+
+	InterfaceDescriptor descriptor = new InterfaceDescriptor();
+	descriptor.setId("schema.org - reactome");
+	descriptor.setBaseUrl("https://reactome.org/sitemapindex.xml");
+	//descriptor.setId("schema.org - kaggle");
+	//descriptor.setBaseUrl("https://www.kaggle.com");
+
+	descriptor.setParams(params);
+
+	SchemaOrgPlugin schemaOrgPlugin = new SchemaOrgPlugin();
+
+	Iterable<String> iterable = schemaOrgPlugin.collect(descriptor, null, null);
+
+	for(String item : iterable) {
+		if(item == null) continue;
+		System.out.println(item);
+	}
+}
+</code></pre>
+
+## Collector plugins for specific data sources
+
+For the collection of metadata from some data sources, specific plugins were required and have been implemented. Unlikely they can be re-used for other data sources.
+
+### PANGAEA: Datasets by Project
+
+* Protocol name: datasetsbyproject
+* Plugin class: eu.dnetlib.data.collector.plugins.datasets.DatasetsByProjectPlugin
+* Plugin description: Given a list of projects provided in a csv file, the plugin search the target endpoint for metadata records related to those projects. 
+* Support incremental collection: No
+* Supported file formats: Response from the endpoint must be in XML. The input csv file has the following format: @project id on the endpoint;project acronym;project name;OpenAIRE info:eu project string@, example: @4106;EPOCA;European Project on Ocean Acidification;info:eu-repo/grantAgreement/EC/FP7/211384@.
+* Plugin parameters:
+** baseUrl: URL to the CSV. Can be local or remote (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+### PANGAEA: Datasets by Journal
+
+* Protocol name: datasetsbyjournal
+This protocol has no plugin associated but it is rather used as marker for the assignment of the correct workflows that calls `eu.dnetlib.data.collector.plugins.datasets.DatasetsByJournalIterator` for the collection from PANGAEA of publications published in Open Access journals (based on an intersection with DOAJ) and the relative datasets.
+
+
+### Re3Data
+
+* Protocol name: re3data
+* Available since: Sept 2015
+* Plugin class: eu.dnetlib.data.collector.plugins.datasources.Re3DataCollectorPlugin
+* Plugin description: Collects metadata about data repositories from re3data API.
+* Support incremental collection: No
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+### GRIST (europePMC API)
+
+* Protocol name: gristProjects
+* Available since: Apr 2016
+* Plugin class: eu.dnetlib.data.collector.plugins.projects.grist.GristCollectorPlugin
+* Plugin description: Plugin to collect metadata record about projects and fundings via the europePMC GRIST API. Currently used in OpenAIRE only for WellcomeTrust projects.
+* Support incremental collection: No
+* Supported file formats: XML in grist core format
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+### GTR-2 for RCUK projects
+
+* Protocol name: gtr2Projects
+* Available since: Dec 2016
+* Plugin class: eu.dnetlib.data.collector.plugins.projects.gtr2.Gtr2CollectorPlugin
+* Plugin description: Collects metadata about projects, organizations, principal investigators of RCUK projects via the Gtr-2 HTTP API
+* Support incremental collection: Yes, but not supported by the endpoint
+* Supported file formats: XML
+* Plugin parameters:
+** baseUrl: base URL of the endpoint (mandatory)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory)
+
+### OpenTrials API 
+
+The API protocol has been deleted as the OpenTrials API were not maintained. The plugin code is still available just in case.
+Currently OpenTrials content is collected from a csv generated by processing a database dump.
+
+
+### remoteMdStore
+
+* Protocol name: remoteMdStore
+* Environment:
+  * non-Hadoop: PROD+BETA+DEV
+* Available since: April 2020
+* Plugin class: eu.dnetlib.data.collector.plugins.remotemdstore.RemoteMdstorePlugin
+* Plugin description: Collects metadata records from a given mdStore at a different environment like PROD, BETA, DEV.
+* Support incremental collection: No
+* Supported file formats: XML
+* Plugin parameters: 
+** baseUrl: base URL of the endpoint (mandatory), e.g. _mongodb://services.openaire.eu:27017/_
+** remote_database: _mdstore_
+** remote_mdstore_id: _id of remote mdstore native or transformed_
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory), e.g. _//*[local-name()='system_metadata']/*[local-name()='id']_
+
+
+### BioSchemas
+
+* Protocol name: Bioschemas
+* Environment:
+  * non-Hadoop: PROD+BETA+DEV
+* Available since: 2021
+* TODO
+
+
+### SparQL
+
+* Protocol name: sparql
+* Environment:
+  * non-Hadoop: PROD+BETA+DEV
+* Available since:
+* TODO
+
+
+### openDAP
+
+* Protocol name: sparql
+* Environment:
+  * non-Hadoop: PROD+BETA+DEV
+* Available since:
+* TODO
+
+
+### sword
+
+* Protocol name: sword/SWORD
+* Environment:
+  * PROD: non-Hadoop
+* Available since:
+* TODO
+
+
+## Protocols for full-text collection
+
+### Files from metadata
+
+* Protocol name: files_from_metadata
+* Plugin class: eu.dnetlib.data.collector.plugins.filesfrommetadata.FilesFromMetadataCollectorPlugin
+* Plugin description: This protocol supports the collection of full-texts based on information available in already collected metadata records. Typical example is the case when the metadata records contain links to the PDF of the relative full-texts.
+* Plugin parameters:
+** baseUrl: (mandatory but ignored)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory but ignored)
+** basePath: path on the local file system where the full-text files must be stored. (mandatory) 
+
+When you associate a download workflow to API with this protocol, you'll have to select the proper download plugins. See  [[Fulltext_Download]] for details on the available plugins.
+
+### Files from mdstore
+
+* Protocol name: files_from_mdstore
+* Plugin class: eu.dnetlib.data.collector.plugins.filesfrommetadata.FilesFromMetadataCollectorPlugin
+* Plugin description: This protocol supports the collection of full-texts in cases when they have already been collected as metadata records. Typical example is the case of JATS XML records: they can be seen as metadata records, but they also contains the full-texts of publications. Instead of collecting twice from a remote location, this plugins enable the copy of records from the metadata stores of a data source to an object store.
+* Plugin parameters:
+** baseUrl: (mandatory but ignored)
+** ID XPath: xpath where the value to be used to forge the dnet identifier for each record can be found (mandatory but ignored)
+** basePath: path on the local file system where the full-text files must be stored. (mandatory)
+

docs/data-provision/aggregation/compatible-sources.md

@@ -1,5 +0,0 @@
----
-sidebar_position: 1
----
-
-# OpenAIRE compatible sources

docs/data-provision/aggregation/datacite.md

@@ -33,7 +33,7 @@ The metadata collection process identifies the most recent record date available
 
 ### Entity Mapping
 
-The table below describes the mapping from the XML baseline records to the OpenAIRE Research Graph dump format.
+The table below describes the mapping from the XML baseline records to the OpenAIRE Graph dump format.
 
 | OpenAIRE Result field path                             | Datacite record JSON path                                                                                                                       | # Notes                                                                                                                                                                                                                                              |
 |--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -56,14 +56,13 @@ The table below describes the mapping from the XML baseline records to the OpenA
 | **date section**                                       |                                                                                                                                                 | for each date in particular for DOI starting with _10.14457_ we Apply a fix thai date convert a date to ThaiBuddhistDate and reformat to local one see ticket [#6791](https://support.openaire.eu/issues/6791)                                       |
 | `publicationdate`                                      | `\attributes\dates`                                                                                                                             | where `dateType` is **issued**                                                                                                                                                                                                                       |
 | `publicationdate`                                      | `\attributes\publicationYear`                                                                                                                   | we create this date format `01-01-publicationYear`                                                                                                                                                                                                   |  
-| `embargoenddate`                                       | `\attributes\dates`                                                                                                                             | where `dateType` is **available**                                                                                                                                                                                                                    |
 | `subjects`                                             | `\attributes\subject`                                                                                                                           | `scheme=keywords`                                                                                                                                                                                                                                    |
 | `description`                                          | `\attributes\descriptions`                                                                                                                      |                                                                                                                                                                                                                                                      |
 | `publisher`                                            | `\attributes\publisher`                                                                                                                         |                                                                                                                                                                                                                                                      |
 | `language`                                             | `\attributes\language`                                                                                                                          | cleaned by using vocabulary `dnet:languages`                                                                                                                                                                                                         |
 | `publisher`                                            | `\attributes\publisher`                                                                                                                         |                                                                                                                                                                                                                                                      |
-| `instance.license`                                     | `\attributes\rightsList`                                                                                                                        | if the rights value starts with http and matches a particular regex                                                                                                                                                                                  |
-| `instance.accessright`                                 | `\attributes\rightsList`                                                                                                                        | <ul><li>if not present :`unknown`</li><li>if datasource is Figshare:`open`</li><li>If `embargo_date < today()`: OPEN</li></ul>                                                                                                                       |
+| `instance.license`                                     | `\attributes\rightsList`          | We filter the datacite accessRights matching this regex: `.*(/licenses|/publicdomain|unlicense\.org/|/legal-and-data-protection-notices|/download/license|/open-government-licence).*` |
+| `instance.accessright`                                 | `\attributes\rightsList`                                                                                                                        | <ul><li>if we found a match in a vocabulary `dnet:access_modes` we put the synonyms</li><li>if datasource is Figshare:`open`</li><li>otherwise UNKNOWN</li></ul>                                                                                                                       |
 
 ### Relation Mapping
 

docs/data-provision/aggregation/doiboost.md

@@ -131,7 +131,7 @@ Possible improvements:
 * Verify if Crossref has a property for `language`, `country`, `container.issnLinking`, `container.iss`, `container.edition`, `container.conferenceplace` and `container.conferencedate`
 * Different approach to set the `refereed` field and improve its coverage?
 
-### Map Crossref links to projects/funders
+h3. 2 Map Crossref links to projects/funders
 
 Links to funding available in Crossref are mapped as funding relationships (`result -- isProducedBy -- project`) applying the following mapping:
 

docs/data-provision/aggregation/ebi.md

@@ -69,7 +69,7 @@ curl -s "https://www.ebi.ac.uk/europepmc/webservices/rest/MED/33024307/datalinks
 ```
 
 ## Mapping
-The table below describes the mapping from the EBI links records to the OpenAIRE Research Graph dump format.
+The table below describes the mapping from the EBI links records to the OpenAIRE Graph dump format.
 We filter all the target links with pid type **ena**, **pdb** or **uniprot**
 For each target we construct a Bioentity with the following mapping
 

docs/data-provision/aggregation/pubmed.md

@@ -12,7 +12,7 @@ Pubmed exposes an entry point FTP with all the updates for each one. [ftp baseli
 
 ## Entity Mapping
 
-The table below describes the mapping from the XML baseline records to the OpenAIRE Research Graph dump format.
+The table below describes the mapping from the XML baseline records to the OpenAIRE Graph dump format.
 
 | OpenAIRE Result field path     | PubMed record field xpath                                                                                                                     | Notes                                                                                                                                                                                                                                                 |
 |--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

docs/data-provision/cleaning.md

@@ -1,37 +0,0 @@
-# Cleaning
-
-
-<!-- ## Vocabulary based cleaning  -->
-
-The aggregation processes run independently one from another and continuously. Each aggregation process, depending on the characteristics of the records exposed by the data source, makes use of one or more vocabularies to harmonise the values available in a given field.
-In this page, we describe the *vocabulary-based cleaning* operation performed to harmonise the data of the different data sources.
-A vocabulary is a data structure that defines a list of terms, and for each term defines a list of synonyms:
-
-```xml
-<TERMS>
-    <TERM native_name="Annotation" code="0018" english_name="Annotation" encoding="OPENAIRE">
-        <SYNONYMS>
-            <SYNONYM term="Comentario" encoding="CSIC"/>
-            <SYNONYM term="Comment/debate" encoding="Aaltodoc Publication Archive"/>
-            <SYNONYM term="annotation" encoding="OPENAIRE-PR202112"/>
-            [...]
-        </SYNONYMS>
-        <RELATIONS/>
-    </TERM>
-    <TERM native_name="Article" code="0001" english_name="Article" encoding="OPENAIRE">
-        <SYNONYMS>
-            <SYNONYM term="A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä" encoding="Aaltodoc Publication Archive"/>
-            <SYNONYM term="A4 Artikkeli konferenssijulkaisussa" encoding="Aaltodoc Publication Archive"/>
-            <SYNONYM term="Article" encoding="OTHER"/>
-            <SYNONYM term="Article (author)" encoding="OTHER"/>
-            [...]
-```
-
-Each vocabulary is typically used to control and harmonise the values available in a specific field characterising the bibliographic records. The example above provides a preview of the vocabulary used to clean the [result's instance typology](/data-model/entities/result#instance).
-
-The content of the vocabularies can be accessed on [api.openaire.eu/vocabularies](https://api.openaire.eu/vocabularies/).
-
-Given a value provided in the original records, the cleaning process looks for a synonym and, when found, resolves the corresponding term which is used in turn to build the cleaned record.
-Each aggregation process applies vocabularies according to their definitions in a given moment of time, however, it could be the case that a vocabulary changes after the aggregation of one data source has finished, thus the aggregated content does not reflect the current status of the controlled vocabularies.
-
-In addition, the integration of ScholeXplorer and DOIBoost and some enrichment processes applied on the raw and on the de-duplicated graph may introduce values that do not comply with the current status of the OpenAIRE controlled vocabularies. For these reasons, we included a final step of cleansing at the end of the workflow materialisation. 

docs/data-provision/data-provision.md

@@ -1,8 +1,7 @@
-# Graph production workflow
+# Data provision 
 
 OpenAIRE collects metadata records from more than 70K scholarly communication sources from all over the world, including Open Access institutional repositories, data archives, journals. All the metadata records (i.e. descriptions of research products) are put together in a data lake, together with records from Crossref, Unpaywall, ORCID, Grid.ac, and information about projects provided by national and international funders. Dedicated inference algorithms applied to metadata and to the full-texts of Open Access publications enrich the content of the data lake with links between research results and projects, author affiliations, subject classification, links to entries from domain-specific databases. Duplicated organisations and results are identified and merged together to obtain an open, trusted, public resource enabling explorations of the scholarly communication landscape like never before.
 
 <p align="center">
-    <img loading="lazy" alt="Data provision" src={require('../assets/img/architecture.png').default} width="100%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
+    <img loading="lazy" alt="Data provision" src="/img/docs/architecture.png" width="80%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
 </p>
-

docs/data-provision/deduction-and-propagation/bulk-tagging.md

@@ -1,37 +0,0 @@
-# Deduction
-
-The Deduction process (also known as “bulk tagging”) enriches each record with new information that can be derived from the existing property values.
-
-This process is used to associate results to community/research initiatives that are part of OpenAIRE. 
-As of November 2022, three procedures are in place to relate a research product to a research initiative, infrastructure (RI) or community (RC) based on:
-
-* subjects: it is possible to specify a list of subjects that are relevant for the RC/RI. Every time one of the subjects is found among the subjects of a result, the result is linked to the RC/RI.
-
-<p align="center">
-    <img loading="lazy" alt="Bulktagging Subject" src={require('../../assets/img/enrichment/bulktagging_subject.png').default} width="50%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-
-* data sources: it is possible to list a set of data sources relevant for the RC/RI. All the results collected from these data sources will be linked to the RC/RI
-<p align="center">
-    <img loading="lazy" alt="Bulktagging Data source" src={require('../../assets/img/enrichment/bulktagging_datasource.png').default} width="50%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
- When only some results collected from a datasource are relevant for the RC/RI, it is possible to specify a set of selection constraints (SC) that have to be verified before linking the result to the 
-community. The selection constraint has the form <strong>SC = S1 or S2 or ... or Sn</strong>. The generic Si has the form <strong>Si = s<sub>i1</sub> and s<sub>i2</sub> and ...and s<sub>in</sub></strong> and each s<sub>ij</sub> is a condition on a specific field of the result. The set of fields that can be specified is <strong>F={title, author, contributor, description, orcid}</strong>, 
-while the set of condition can be among <strong>V={contains, equals, not_contains, not_equals, contains_ignorecase, equals_ignorecase, not_contains_ignorecase, not_equal_ignorecase}</strong>, and the value is free text.
-A possible selection criteria can be: “All the products whose contributor contains DARIAH “
-
-<p align="center">
-    <img loading="lazy" alt="Bulktagging Data source" src={require('../../assets/img/enrichment/bulktagging_selconstraints.png').default} width="70%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-* Zenodo community: it is possible to list a set of Zenodo communities relevant for the RC/RI. All the products collected from the listed Zenodo communities are linked to the RC/RI
-
-
-<p align="center">
-    <img loading="lazy" alt="Bulktagging Zenodo Community" src={require('../../assets/img/enrichment/bulktagging_zenodo.png').default} width="50%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-
-The list of subjects, Zenodo communities and data sources used to enrich the products are defined by the managers of the community gateway or infrastructure monitoring dashboard associated with the RC/RI.

docs/data-provision/deduction-and-propagation/propagation.md

@@ -1,55 +0,0 @@
-# Propagation
-
-This process enriches the graph by adding new links and/or new properties. The new information is added by exploiting existing semantic 
-relationships and values between the involved entities 
-
-As of November 2022, the following procedures are in place:
-
-* Country propagation: updates the property “country” of a results. This happens when the result is collected from an institutional datasource or when the datasource hosting the result is inserted in a whitelist. For all the results whose hosting datasource verifies one of the conditions above, the country of the organization providing the datasource is added to the country of the result: e.g. publication collected from an institutional repository maintained by an italian university will be enriched with the property “country = IT”.
-<p align="center">
-    <img loading="lazy" alt="Country Propagation" src={require('../../assets/img/enrichment/propagation_country.png').default} width="50%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-* Project propagation: adds a "isProducedBy" relationship (and its inverse) between a Project P and Result R1, if R1 has a strong semantic relationship with another Result R2 and P produces R2: e.g. publication linked to project P “is supplemented by” a dataset D. Dataset D will get the link to project P. The relationships considered for this procedure are “isSupplementedBy” and “isSupplementTo”.
-<p align="center">
-    <img loading="lazy" alt="Project Propagation" src={require('../../assets/img/enrichment/propagation_resulttoproject.png').default} width="40%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-* Result to RC/RI through organization propagation. The manager of the RC/RI can specify a set of organizations whose product are relevant for the 
-community.  
-Each result having such a relation of affiliation with at least one organization relevant for the RC/RI will be linked to it.
-<p align="center">
-    <img loading="lazy" alt="Result to community through organization propagation" src={require('../../assets/img/enrichment/propagation_resulttocommunitythroughorganization.png').default} 
-    width="50%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-* Result to RC/RI through semantic relation: extends the set of products linked to a RC/RI by exploiting strong semantic relationships between the results;
-e.g. if a result R1 is associated to the community C and is supplemented by a result R2 then the result R2 will be linked to the community. The relationships considered for this procedure are “isSupplementedBy” and “supplements”.
-<p align="center">
-    <img loading="lazy" alt="Result to community through semantic relation propagation" src={require('../../assets/img/enrichment/propagation_resulttocommunitythroughsemrel.png').default} width="40%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-* ORCID identifiers to result through semantic relation. This propagation enriches the results by adding ORCID identifiers to authors. The added ORCID will be marked as "potential" since they have been inserted through propagation. 
-The process considers the set of overlapping authors between results (R1 and R2) linked with a strong semantic relationship (IsSupplementedBy, IsSupplementTo). 
-For each author A in the overlapping set, if R1 provides the ORCID value for A and R2 does not, then the author A in R2 will be enriched with the information of the ORCID found in R1.
-
-<p align="center">
-    <img loading="lazy" alt="Orcid propation through semantic relation" src={require('../../assets/img/enrichment/propagation_orcid.png').default} width="40%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-* affiliation to organization through institutional repository. This propagation adds one "hasAuthorInstitution" relationship (and its inverse) 
-between a Result R and Organization O, 
-if R was collected from a datasource D with type institutional repository, and D was provided by O. 
-<p align="center">
-    <img loading="lazy" alt="Affiliation propagation through institutional repository" src={require('../../assets/img/enrichment/propagation_affiliationistrepo.png').default} width="40%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-* affiliation to organization through semantic relation. This propagation adds one "hasAuthorInstitution" relationship (and its inverse) between a 
-Result R and an Organization O, 
-if R has an affiliation relation with an organization O1 that is in relation "isChildOf" with O.
-
-<p align="center">
-    <img loading="lazy" alt="Affiliation propagation through semantic relation" src={require('../../assets/img/enrichment/propagation_organizationsemrel.png').default} width="40%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
- The algorithm exploits only the organization leaves that are in a "IsChildOf" relation with another organization. So far one single step is done
-<p align="center">
-    <img loading="lazy" alt="propagation strategy" src={require('../../assets/img/enrichment/organization_tree.png').default} width="40%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>

docs/data-provision/deduplication/clustering-functions.md

@@ -3,91 +3,18 @@ sidebar_position: 3
 ---
 # Clustering functions 
 
-## Ngrams
-
-It creates ngrams from the input field. <br />
-```
-Example:
-Input string: “Search for the Standard Model Higgs Boson”
-Parameters: ngram length = 3, maximum number = 4
-List of ngrams: “sea”, “sta”, “mod”, “hig”
-```
-
 ## NgramPairs
-
 It produces a list of concatenations of a pair of ngrams generated from different words.<br />
-```
-Example:
-Input string: “Search for the Standard Model Higgs Boson”
-Parameters: ngram length = 3
-Ngram pairs: “seasta”, “stamod”, “modhig”
-```
+*Example:*<br />
+Input string: `“Search for the Standard Model Higgs Boson”`<br />
+Parameters: ngram length = 3<br />
+List of ngrams: `“sea”`, `“sta”`, `“mod”`, `“hig”`<br />
+Ngram pairs: `“seasta”`, `“stamod”`, `“modhig”`
 
 ## SuffixPrefix
 
-It produces ngrams pairs in a particular way: it concatenates the suffix of a string with the prefix of the next in the input string. A specialization of this function is available as SortedSuffixPrefix. It returns a sorted list. <br />
-```
-Example:
-Input string: “Search for the Standard Model Higgs Boson”
-Parameters: suffix and prefix length = 3, maximum number = 2
-Output list: “ardmod”` (suffix of the word “Standard” + prefix of the word “Model”), “rchsta” (suffix of the word “Search” + prefix of the word “Standard”)
-```
-
-## Acronyms
-
-It creates a number of acronyms out of the words in the input field. <br />
-```
-Example:
-Input string: “Search for the Standard Model Higgs Boson”
-Output: "ssmhb"
-```
-
-## KeywordsClustering
-
-It creates keys by extracting keywords, out of a customizable list, from the input field. <br />
-```
-Example:
-Input string: “University of Pisa”
-Output: "key::001" (code that identifies the keyword "University" in the customizable list)
-```
-
-## LowercaseClustering
-
-It creates keys by lowercasing the input field. <br />
-```
-Example:
-Input string: “10.001/ABCD”
-Output: "10.001/abcd"
-```
-
-## RandomClusteringFunction
-
-It creates random keys from the input field. <br />
-
-## SpaceTrimmingFieldValue
-
-It creates keys by trimming spaces in the input field. <br />
-```
-Example:
-Input string: “Search for the Standard Model Higgs Boson”
-Output: "searchstandardmodelhiggsboson"
-```
-
-## UrlClustering
-
-It creates keys for an URL field by extracting the domain. <br />
-```
-Example:
-Input string: “http://www.google.it/page”
-Output: "www.google.it"
-```
-
-## WordsStatsSuffixPrefixChain
-
-It creates keys containing concatenated statistics of the field, i.e. number of words, number of letters and a chain of suffixes and prefixes of the words. <br />
-```
-Example:
-Input string: “Search for the Standard Model Higgs Boson”
-Parameters: mod = 10
-Output list: "5-3-seaardmod" (number of words + number of letters % 10 + prefix of the word "Search" + suffix of the word "Standard" + prefix of the word "Model"), "5-3-rchstadel" (number of words + number of letters % 10 + suffix of the word "Search" + prefix of the word "Standard" + suffix of the word "Model")
-```
+It produces ngrams pairs in a particular way: it concatenates the suffix of a string with the prefix of the next in the input string.<br />
+*Example:*<br />
+Input string: `“Search for the Standard Model Higgs Boson”`<br />
+Parameters: suffix and prefix length = 3<br />
+Output list: `“ardmod”` (suffix of the word `“Standard”` + prefix of the word `“Model”`), `“rchsta”` (suffix of the word `“Search”` + prefix of the word `“Standard”`)

docs/data-provision/deduplication/deduplication.md

@@ -1,28 +1,29 @@
 # Deduplication
 
-Metadata records about the same scholarly work can be collected from different providers. Each metadata record can possibly carry different information because, for example, some providers are not aware of links to projects, keywords or other details. Another common case is when OpenAIRE collects one metadata record from a repository about a pre-print and another record from a journal about the published article. For the provision of statistics, OpenAIRE must identify those cases and “merge” the two metadata records, so that the scholarly work is counted only once in the statistics OpenAIRE produces.
+## Clustering 
 
-## Methodology overview
+Clustering is a common heuristics used to overcome the N x N complexity required to match all pairs of objects to identify the equivalent ones. The challenge is to identify a clustering function that maximizes the chance of comparing only records that may lead to a match, while minimizing the number of records that will not be matched while being equivalent. Since the equivalence function is to some level tolerant to minimal errors (e.g. switching of characters in the title, or minimal difference in letters), we need this function to be not too precise (e.g. a hash of the title), but also not too flexible (e.g. random ngrams of the title). On the other hand, reality tells us that in some cases equality of two records can only be determined by their PIDs (e.g. DOI) as the metadata properties are very different across different versions and no clustering function will ever bring them into the same cluster. To match these requirements OpenAIRE clustering for products works with two functions:
+* DOI: the function generates the DOI when this is provided as part of the record properties;
+* Title-based function: the function generates a key that depends on (i) number of significant words in the title (normalized, stemming, etc.), (ii) module 10 of the number of characters of such words, and (iii) a string obtained as an alternation of the function prefix(3) and suffix(3) (and vice versa) o the first 3 words (2 words if the title only has 2). For example, the title “Entity deduplication in big data graphs for scholarly communication” becomes “entity deduplication big data graphs scholarly communication” with two keys key “7.1entionbig” and “7.1itydedbig” (where 1 is module 10 of 54 characters of the normalized title.
+To give an idea, this configuration generates around 77Mi blocks, which we limited to 200 records each (only 15K blocks are affected by the cut), and entails 260Bi matches. Matches in a block are performed using a “sliding window” set to 80 records. The records are sorted lexicographically on a normalized version of their titles. The 1st record is matched against all the 80 following ones, then the second, etc. for an NlogN complexity.
 
-The deduplication process can be divided into three different phases:
-* Candidate identification (clustering)
-* Duplicates identification (pair-wise comparisons)
-* Duplicates grouping (transitive closure)
+## Matching and election
 
-<p align="center">
-    <img loading="lazy" alt="Deduplication Workflow" src={require('../../assets/img/deduplication-workflow.png').default} width="85%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
+Once the clusters have been built, the algorithm proceeds with the comparisons. Comparisons are driven by a decisional tree that:
+1. Tries to capture equivalence via PIDs: if records share a PID then they are equivalent
 
-### Candidate identification (clustering) 
+2. Tries to capture difference:
 
-Clustering is a common heuristics used to overcome the N x N complexity required to match all pairs of objects to identify the equivalent ones. The challenge is to identify a [clustering function](./clustering-functions) that maximizes the chance of comparing only records that may lead to a match, while minimizing the number of records that will not be matched while being equivalent. Since the equivalence function is to some level tolerant to minimal errors (e.g. switching of characters in the title, or minimal difference in letters), we need this function to be not too precise (e.g. a hash of the title), but also not too flexible (e.g. random ngrams of the title). On the other hand, reality tells us that in some cases equality of two records can only be determined by their PIDs (e.g. DOI) as the metadata properties are very different across different versions and no [clustering function](./clustering-functions) will ever bring them into the same cluster.
+  a. If record titles contain different “numbers” then they are different (this rule is subject to different feelings, and should be fine-tuned);
 
-### Duplicates identification (pair-wise comparisons)
+  b. If record contain different number of authors then they are different;
 
-Pair-wise comparisons are conducted over records in the same cluster following the strategy defined in the decision tree. A different decision tree is adopted depending on the type of the entity being processed.
+  c. Note that different PIDs do not imply different records, as different versions may have different PIDs.
 
-To further limit the number of comparisons, a sliding window mechanism is used: (i) records in the same cluster are lexicographically sorted by their title, (ii) a window of K records slides over the cluster, and (iii) records ending up in the same window are pair-wise compared. The result of each comparison produces a similarity relation when the pair of record matches. Such relations will be consequently used as input for the duplicates grouping stage.
+3. Measures equivalence:
 
-### Duplicates grouping (transitive closure)
+  a. The titles of the two records are normalised and compared for similarity by applying the Levenstein distance algorithm. The algorithm returns a number in the range [0,1], where 0 means “very different” and 1 means “equal”. If the distance is greater than or equal 0,99 the two records are identified as duplicates.
 
-Once the similarity relations between pairs of records are drawn, the groups of equivalent records are obtained (transitive closure, i.e. “mesh”). From such sets a new representative object is obtained, which inherits all properties from the merged records and keeps track of their provenance. 
+  b. Dates are not regarded for equivalence matching because different versions of the same records should be merged and may be published on different dates, e.g. pre-print and published version of an article.
+  
+Once the equivalence relationships between pairs of records are set, the groups of equivalent records are obtained (transitive closure, i.e. “mesh”). From such sets a new representative object is obtained, which inherits all properties from the merged records and keeps track of their provenance. The ID of the record is obtained by appending the prefix “dedup_” to the MD5 of the first ID (given their lexicographical ordering). A new, more stable function to generate the ID is under development, which exploits the DOI when one of the records to be merged includes a Crossref or a DataCite record.

docs/data-provision/deduplication/organizations.md

@@ -3,68 +3,4 @@ sidebar_position: 2
 ---
 
 # Organizations
-
-The organizations in OpenAIRE are aggregated from different registries (e.g. CORDA, OpenDOAR, Re3data, ROR). In some cases, a registry provides organizations as entities with their own persistent identifier. In other cases, those organizations are extracted from other main entities provided by the registry (e.g. datasources, projects, etc.).
-
-The deduplication of organizations is enhanced by the [OpenOrgs](https://orgs.openaire.eu), a tool that combines an automated approach for identifying duplicated instances 
-of the same organization record with a "humans in the loop" approach, in which the equivalences produced by a duplicate identification algorithm are suggested to data curators, in charge for validating them.
-The data curation activity is twofold, on one end pivots around the disambiguation task, on the other hand assumes to improve the metadata describing the organization records 
-(e.g. including the translated name, or a different PID) as well as defining the hierarchical structure of existing large organizations (i.e. Universities comprising its departments or large research centers with all its sub-units or sub-institutes).
-
-Duplicates among organizations are therefore managed through three different stages:
-  * *Creation of Suggestions*: executes an automatic workflow that performs the deduplication and prepare new suggestions for the curators to be processed;
-  * *Curation*: manual editing of the organization records performed by the data curators;
-  * *Creation of Representative Organizations*: executes an automatic workflow that creates curated organizations and exposes them on the OpenAIRE Research Graph by using the curators' feedback from the OpenOrgs underlying database.
-
-The next sections describe the above mentioned stages.
-
-### Creation of Suggestions
-
-This stage executes an automatic workflow that faces the *candidate identification* and the *duplicates identification* stages of the deduplication to provide suggestions for the curators in the OpenOrgs.
-
-#### Candidate identification (clustering)
-
-To match the requirements of limiting the number of comparisons, OpenAIRE clustering for organizations aims at grouping records that would more likely be comparable. 
-It works with four functions:
-* *URL-based function*: the function generates the URL domain when this is provided as part of the record properties from the organization's `websiteurl` field;
-* *Title-based functions*: 
-	* generate strings dependent to the keywords in the `legalname` field;
-	* generate strings obtained as an alternation of the function prefix(3) and suffix(3) (and vice versa) on the first 3 words of the `legalname` field;
-	* generate strings obtained as a concatenation of ngrams of the `legalname` field; 
-
-#### Duplicates identification (pair-wise comparisons)
-
-For each pair of organization in a cluster the following strategy (depicted in the figure below) is applied.
-The comparison goes through the following decision tree:
-1. *grid id check*: comparison of the grid ids. If the grid id is equivalent, then the similarity relation is drawn. If the grid id is not available, the comparison proceeds to the next stage;
-2. *early exits*: comparison of the numbers extracted from the `legalname`, the `country` and the `website` url. No similarity relation is drawn in this stage, the comparison proceeds only if the compared fields verified the conditions of equivalence;
-3. *city check*: comparison of the city names in the `legalname`. The comparison proceeds only if the legalnames shares at least 10% of cities;
-4. *keyword check*: comparison of the keywords in the `legalname`. The comparison proceeds only if the legalnames shares at least 70% of keywords;
-5. *legalname check*: comparison of the normalized `legalnames` with the `Jaro-Winkler` distance to determine if it is higher than `0.9`. If so, a similarity relation is drawn. Otherwise, no similarity relation is drawn.
-
-<p align="center">
-    <img loading="lazy" alt="Organization Decision Tree" src={require('../../assets/img/decisiontree-organization.png').default} width="100%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-[//]: # (Link to the image: https://docs.google.com/drawings/d/1YKInGGtHu09QG4pT2gRLEum4LxU82d4nKkvGNvRQmrg/edit?usp=sharing)
-
-### Data Curation
-
-All the similarity relations drawn by the algorithm involving the decision tree are exposed in OpenOrgs, where are made available to the data curators to give feedbacks and to improve the organizations metadata.
-A data curator can:
- * *edit organization metadata*: legalname, pid, country, url, parent relations, etc.;   
- * *approve suggested duplicates*: establish if an equivalence relation is valid;
- * *discard suggested duplicates*: establish if an equivalence relation is wrong;
- * *create similarity relations*: add a new equivalence relation not drawn by the algorithm.
-
-Note that if a curator does not provide a feedback on a similarity relation suggested by the algorithm, then such relation is considered as valid.
-
-### Creation of Representative Organizations
-
-This stage executes an automatic workflow that faces the *duplicates grouping* stage to create representative organizations and to update them on the OpenAIRE Research Graph. Such organizations are obtained via transitive closure and the relations used comes from the curators' feedback gathered on the OpenOrgs underlying Database.
-
-#### Duplicates grouping (transitive closure)
-
-Once the similarity relations between pairs of organizations have been gathered, the groups of equivalent organizations are obtained (transitive closure, i.e. “mesh”). From such sets a new representative organization is obtained, which inherits all properties from the merged records and keeps track of their provenance.
-
-The IDs of the representative organizations are obtained by the OpenOrgs Database that creates a unique ``openorgs`` ID for each approved organization. In case an organization is not approved by the curators, the ID is obtained by appending the prefix ``pending_org`` to the MD5 of the first ID (given their lexicographical ordering).
+<span className="todo">TODO</span>

docs/data-provision/deduplication/research-products.md

@@ -4,66 +4,48 @@ sidebar_position: 1
 
 # Research results
 
+Metadata records about the same scholarly work can be collected from different providers. Each metadata record can possibly carry different information because, for example, some providers are not aware of links to projects, keywords or other details. Another common case is when OpenAIRE collects one metadata record from a repository about a pre-print and another record from a journal about the published article. For the provision of statistics, OpenAIRE must identify those cases and “merge” the two metadata records, so that the scholarly work is counted only once in the statistics OpenAIRE produces.
+
 Duplicates among research results are identified among results of the same type (publications, datasets, software, other research products). If two duplicate results are aggregated one as a dataset and one as a software, for example, they will never be compared and they will never be identified as duplicates.
 OpenAIRE supports different deduplication strategies based on the type of results.
 
-The next sections describe how each stage of the deduplication workflow is faced for research results.
+## Methodology overview
 
-### Candidate identification (clustering)
+The deduplication process can be divided into two different phases:
+* Candidate identification (clustering)
+* Decision tree
+* Creation of representative record
 
-To match the requirements of limiting the number of comparisons, OpenAIRE clustering for research products works with two functions:
-* *DOI-based function*: the function generates the DOI when this is provided as part of the record properties;
-* *Title-based function*: the function generates a key that depends on (i) number of significant words in the title (normalized, stemming, etc.), (ii) module 10 of the number of characters of such words, and (iii) a string obtained as an alternation of the function prefix(3) and suffix(3) (and vice versa) on the first 3 words (2 words if the title only has 2). For example, the title ``Search for the Standard Model Higgs Boson`` becomes ``search standard model higgs boson`` with two keys key ``5-3-seaardmod`` and ``5-3-rchstadel``.
+The implementation of each phase is different based on the type of results that are being processed.
 
-To give an idea, this configuration generates around 77Mi blocks, which we limited to 200 records each (only 15K blocks are affected by the cut), and entails 260Bi matches.
+### Publications 
 
-### Duplicates identification (pair-wise comparisons)
+#### Candidate identification (clustering)
 
-Comparisons in a block are performed using a *sliding window* set to 50 records. The records are sorted lexicographically on a normalized version of their titles. The 1st record is compared against all the 50 following ones using the decision tree, then the second, etc. for an NlogN complexity.
-A different decision tree is adopted depending on the type of the entity being processed. 
-Similarity relations drawn in this stage will be consequently used to perform the duplicates grouping.
+Clustering is a common heuristics used to overcome the N x N complexity required to match all pairs of objects to identify the equivalent ones. The challenge is to identify a [clustering function](./clustering-functions) that maximizes the chance of comparing only records that may lead to a match, while minimizing the number of records that will not be matched while being equivalent. Since the equivalence function is to some level tolerant to minimal errors (e.g. switching of characters in the title, or minimal difference in letters), we need this function to be not too precise (e.g. a hash of the title), but also not too flexible (e.g. random ngrams of the title). On the other hand, reality tells us that in some cases equality of two records can only be determined by their PIDs (e.g. DOI) as the metadata properties are very different across different versions and no [clustering function](./clustering-functions) will ever bring them into the same cluster. To match these requirements OpenAIRE clustering for products works with two functions:
+DOI: the function generates the DOI when this is provided as part of the record properties;
+Title-based function: the function generates a key that depends on (i) number of significant words in the title (normalized, stemming, etc.), (ii) module 10 of the number of characters of such words, and (iii) a string obtained as an alternation of the function prefix(3) and suffix(3) (and vice versa) o the first 3 words (2 words if the title only has 2). For example, the title “Entity deduplication in big data graphs for scholarly communication” becomes “entity deduplication big data graphs scholarly communication” with two keys key “7.1entionbig” and “7.1itydedbig” (where 1 is module 10 of 54 characters of the normalized title.
 
-#### Publications
+#### Decision tree
 
 For each pair of publications in a cluster the following strategy (depicted in the figure below) is applied.
-The comparison goes through different stages:
-1. *trusted pids check*: comparison of the trusted pid lists (in the `pid` field of the record). If at least 1 pid is equivalent, records match and the similarity relation is drawn.
-2. *instance type check*: comparison of the instance types (indicating the subtype of the record, i.e. presentation, conference object, etc.). If the instance types are not compatible then the records does not match. Otherwise, the comparison proceeds to the next stage
-3. *untrusted pids check*: comparison of all the available pids (in the `pid` and the `alternateid` fields of the record). In every case, no similarity relation is drawn in this stage. If at least one pid is equivalent, the next stage will be a  *soft check*, otherwise the next stage is a *strong check*.
-4. *soft check*: comparison of the record titles with the Levenshtein distance. If the distance measure is above 0.9 then the similarity relation is drawn.
-5. *strong check*: comparison composed by three substages involving the (i) comparison of the author list sizes and the version of the record to determine if they are coherent, (ii) comparison of the record titles with the Levenshtein distance to determine if it is higher than 0.99, (iii) "smart" comparison of the author lists to check if common authors are more than 60%.
+Cross comparison of the pid lists (in the `pid` and `alternateid` elements). If 50% common pids, levenshtein distance on titles with low threshold (0.9).
+Otherwise, check if the number of authors and the title version is equal. If so, levenshtein distance on titles with higher threshold (0.99).
+The publications are matched as duplicate if the distance is higher than the threshold, in every other case they are considered as distinct publications.
 
 <p align="center">
-    <img loading="lazy" alt="Publications Decision Tree" src={require('../../assets/img/decisiontree-publication.png').default} width="100%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
+    <img loading="lazy" alt="Deduplication workflow" src="/img/docs/dedup-results.png" width="80%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
 </p>
 
-[//]: # (Link to the image: https://docs.google.com/drawings/d/19SIilTp1vukw6STMZuPMdc0pv0ODYCiOxP7OU3iPWK8/edit?usp=sharing)
+#### Creation of representative record
+<span className="todo">TODO</span>
 
-#### Software
-For each pair of software in a cluster the following strategy (depicted in the figure below) is applied.
-The comparison goes through different stages:
-1. *pids check*: comparison of the pids in the records. No similarity relation is drawn in this stage, it is only used to establish the final threshold to be used to compare record titles. If there is at least one common pid, then the next stage is a *soft check*. Otherwise, the next stage is a *strong check*
-2. *soft check*: comparison of the record titles with Levenshtein distance. If the measure is above 0.9, then the similarity relation is drawn
-3. *strong check*: comparison of the record titles with Levenshtein distance. If the measure is above 0.99, then the similarity relation is drawn
+### Datasets
+<span className="todo">TODO</span>
 
-<p align="center">
-    <img loading="lazy" alt="Software Decision Tree" src={require('../../assets/img/decisiontree-software.png').default} width="85%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
+### Software
+<span className="todo">TODO</span>
 
-[//]: # (Link to the image: https://docs.google.com/drawings/d/19gd1-GTOEEo6awMObGRkYFhpAlO_38mfbDFFX0HAkuo/edit?usp=sharing)
+### Other types of research products
+<span className="todo">TODO</span>
 
-#### Datasets and Other types of research products
-For each pair of datasets or other types of research products in a cluster the strategy depicted in the figure below is applied.
-The decision tree is almost identical to the publication decision tree, with the only exception of the *instance type check* stage. Since such type of record does not have a relatable instance type, the check is not performed and the decision tree node is skipped.
-
-<p align="center">
-    <img loading="lazy" alt="Dataset and Other types of research products Decision Tree" src={require('../../assets/img/decisiontree-dataset-orp.png').default} width="90%" className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module"/>
-</p>
-
-[//]: # (Link to the image: https://docs.google.com/drawings/d/1uBa7Bw2KwBRDUYIfyRr_Keol7UOeyvMNN7MPXYLg4qw/edit?usp=sharing)
-
-### Duplicates grouping (transitive closure)
-
-The general concept is that the field coming from the record with higher "trust" value is used as reference for the field of the representative record.
-
-The IDs of the representative records are obtained by appending the prefix ``dedup_`` to the MD5 of the first ID (given their lexicographical ordering). If the group of merged records contains a trusted ID (i.e. the DOI), also the ``doi`` keyword is added to the prefix.

docs/data-provision/enrichment-by-mining/acks.md

@@ -1,30 +0,0 @@
----
-sidebar_position: 3
----
-
-# Extraction of acknowledged concepts
-
-***Short description:*** Scans the plaintexts of publications for acknowledged concepts, including grant identifiers (projects) of funders, accession numbers of bioetities, EPO patent mentions, as well as custom concepts that can link research objects to specific research communities and initiatives in OpenAIRE.
-
-***Algorithmic details:***
-The algorithm processes the publication's fulltext and extracts references to acknowledged concepts. It applies pattern matching and string join between the fulltext and a target database which contains the title, the acronym and the identifier of the searched concept.
-
-***Parameters:***
-Concept titles, acronyms, and identifiers, publication's identifiers and fulltexts
-
-***Limitations:*** -
-
-***Environment:***
-Python, [madIS](https://github.com/madgik/madis), [APSW](https://github.com/rogerbinns/apsw)
-
-***References:***
-* Foufoulas, Y., Zacharia, E., Dimitropoulos, H., Manola, N., Ioannidis, Y. (2022). DETEXA: Declarative Extensible Text Exploration and Analysis. In: , et al. Linking Theory and Practice of Digital Libraries. TPDL 2022. Lecture Notes in Computer Science, vol 13541. Springer, Cham. [doi:10.1007/978-3-031-16802-4_9](https://doi.org/10.1007/978-3-031-16802-4_9)
-
-***Authority:*** ATHENA RC • ***License:*** CC-BY/CC-0 • ***Code:*** [iis/referenceextraction](https://github.com/openaire/iis/tree/master/iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction)
-
-
-
-
-
-
-

docs/data-provision/enrichment-by-mining/affiliation_matching.md

@@ -1,57 +0,0 @@
----
-sidebar_position: 1
----
-
-# Affiliation matching
-
-***Short description:*** The goal of the affiliation matching module is to match affiliations extracted from the pdf and xml documents with organizations from the OpenAIRE organization database.
-
-***Algorithmic details:***
-
-*The buckets concept*
-
-In order to get the best possible results, the algorithm should compare every affiliation with every organization. However, this approach would be very inefficient and slow, because it would involve the processing of the cartesian product (all possible pairs) of millions of affiliations and thousands of organizations. To avoid this, IIS has introduced the concept of buckets. A bucket is a smaller group of affiliations and organizations that have been selected to be matched with one another. The matching algorithm compares only these affiliations and organizations that belong to the same bucket.
-
-*Affiliation matching process*
-
-Every affiliation in a given *bucket* is compared with every organization in the same bucket multiple times, each time by using a different algorithm (*voter*). Each *voter* is assigned a number (match strength) that describes the estimated correctness of the result of its comparison. All the affiliation-organization pairs that have been matched by at least one *voter*, will be assigned the match strength > 0 (the actual number depends on the voters, its calculation method will be shown later).
-
-It is very important for the algorithm to group the affiliations and organizations properly i.e. the ones that have a chance to match should be in the same *bucket*. To guarantee this, the affiliation matching module allows to create different methods of dividing the affiliations and organizations into *buckets*, and to use all of these methods in a single matching process. The specific method of grouping the affiliations and organizations into *bucket* and then joining them into pairs is carried out by the service called *Joiner*.
-
-Every *joiner* can be linked with many different *voters* that will tell if the affiliation-organization pairs joined match or not. By providing new *joiners* and *voters* one can extend the matching algorithm with countless new methods for matching affiliations with organizations, thus adjusting the algorithm to his or her needs.
-
-All the affiliations and organizations are sequentially computed by all the *matchers*. In every *matcher* they are grouped by some *joiner* in pairs, and then these pairs are processed by all the *voters* in the *matcher*. Every affiliation-organization pair that has been matched at least once is assigned the match strength that depends on the match strengths of the *voters* that pointed the given pair is a match. 
-
-**NOTE:** There can be many organizations matched with a given affiliation, each of them matched with a different match strength. The user of the module can set a match strength threshold which will limit the results to only those matches that have the match strength greater than the specified threshold.
-
-*Calculation of the match strength of the affiliation-organization pair matched by multiple matchers*
-
-It often happens that the given affiliation-organization pair is returned as a match by more than one matcher, each time with a different match strength. In such a case **the match with the highest match strength will be selected**.
-
-*Calculation of the match strength of the affiliation-organization pair within a single matcher*
-
-Every voter has a match strength that is in the range (0, 1]. **The voter match strength says what the quotient of correct matches to all matches guessed by this voter is, and is based on real data and hundreds of matches prepared by hand.**
-
-The match strength of the given affiliation-organization pair is based on the match strengths of all the voters in the matcher that have pointed that the pair is a match. It will always be less than or equal to 1 and greater than the match strength of each single voter that matched the given pair.
-
-The total match strength is calculated in such a way that each consecutive voter reduces (by its match strength) the gap of uncertainty about the correctness of the given match.
-
-***Parameters:***
-
-* input
-    * input_document_metadata: [ExtractedDocumentMetadata](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/metadataextraction/ExtractedDocumentMetadata.avdl) avro datastore location. Document metadata is the source of affiliations.
-    * input_organizations: [Organization](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/importer/Organization.avdl) avro datastore location. 
-    * input_document_to_project: [DocumentToProject](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/importer/DocumentToProject.avdl) avro datastore location with **imported** document-to-project relations. These relations (alongside with inferred document-project and project-organization relations) are used to generate document-organization pairs which are used as a hint for matching affiliations.
-    * input_inferred_document_to_project: [DocumentToProject](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/referenceextraction/project/DocumentToProject.avdl) avro datastore location with **inferred** document-to-project relations. 
-    * input_project_to_organization: [ProjectToOrganization](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/importer/ProjectToOrganization.avdl) avro datastore location. These relations (alongside with infered document-project and document-project relations) are used to generate document-organization pairs which are used as a hint for matching affiliations
-* output
-    * [MatchedOrganization](https://github.com/openaire/iis/blob/master/iis-wf/iis-wf-affmatching/src/main/resources/eu/dnetlib/iis/wf/affmatching/model/MatchedOrganization.avdl) avro datastore location with matched publications with organizations.
-
-***Limitations:*** -
-
-***Environment:*** 
-Java, Spark
-
-***References:*** -
-
-***Authority:*** ICM • ***License:*** AGPL-3.0 • ***Code:*** [CoAnSys/affiliation-organization-matching](https://github.com/CeON/CoAnSys/tree/master/affiliation-organization-matching)

docs/data-provision/enrichment-by-mining/citation_matching.md

@@ -1,41 +0,0 @@
-# Citation matching
-
-***Short description:*** During a citation matching task, bibliographic entries are linked to the documents that they reference. The citation matching module - one of the modules of the Information Inference Service (IIS) - receives as an input a list of documents accompanied by their metadata and bibliography. Among them, it discovers links described above and returns them as a list. In this document we shall evaluate if the module has been properly integrated with the whole
-system and assess the accuracy of the algorithm used. It is worth mentioning that the implemented algorithm has been described in detail in arXiv:1303.6906 [cs.IR]1. However, in the referenced paper the algorithm was tested on small datasets, but here we will focus on larger datasets, which are expected to be analysed by the system in the production environment.
-
-***Algorithmic details:***
-
-*General description*
-
-The algorithm used in citation matching task consists of two phases. In the first one, for each citation string a set of potentially matching documents is retrieved using a heuristic. In the second one, the metadata of these documents is analysed in order to assess which of them is the most similar to given citation. We assume that citations are parsed, i.e. fragments containing meaningful pieces of metadata information are marked in a special way. Note that in the IIS system, the citation parsing step is executed by another module. The following metadata fields are used by the described solution:
-
-* an author,
-* a title,
-* a journal name,
-* pages,
-* a year of publication.
-
-*Heuristic matching*
-
-The heuristic is based on indexing of document metadata by their author names. For each citation we extract author names and try to find documents in the index which have the same author entries. As spelling errors and inaccuracies commonly occur in citations, we have implemented approximate index which enables retrieval of entities with edit distance less than or equal 1.
-
-*Strict matching*
-
-In this step, all the potentially matching pairs obtained in the heuristic step are evaluated and only the most probable ones are returned as the final result. As citations tend to contain spelling errors and differ in style, there is a need to introduce fuzzy similarity measures fitted to the specifics of various metadata fields. Most of them compute a fraction of tokens or trigrams that occur in both fields being compared. When comparing journal
-names, we have taken longest common subsequence (LCS) of two strings into consideration. This can be seen as an instance of the assignment problem with some refinements added. The overall similarity of two citation strings is obtained by applying a linear Support Vector Machine (SVM) using field similarities as features.
-
-***Parameters:***
-
-* input: 
-    * input_metadata: [ExtractedDocumentMetadataMergedWithOriginal](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/transformers/metadatamerger/ExtractedDocumentMetadataMergedWithOriginal.avdl) avro datastore location with the metadata of both publications and bibliorgaphic references to be matched
-    * input_matched_citations: [Citation](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/common/citations/Citation.avdl) avro datastore location with citations which were already matched and should be excluded from fuzzy matching
-* output: [Citation](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/common/citations/Citation.avdl) avro datastore location with matched publications
-
-***Limitations:*** -
-
-***Environment:***
-Java, Spark
-
-***References:*** -
-
-***Authority:*** ICM • ***License:*** AGPL-3.0 • ***Code:*** [CoAnSys/citation-matching](https://github.com/CeON/CoAnSys/tree/master/citation-matching)

docs/data-provision/enrichment-by-mining/cites.md

@@ -1,23 +0,0 @@
----
-sidebar_position: 4
----
-
-# Extraction of cited concepts
-
-***Short description:*** Scans the plaintexts of publications for cited concepts, currently for references to datasets and software URIs.
-
-***Algorithmic details:***
-The algorithm extracts citations to specific datasets and software. It extracts the citation section of a publication's fulltext and applies string matching against a target database which includes an inverted index with dataset/software titles, urls and other metadata.
-
-***Parameters:***
-Title, URL, creator names, publisher names and publication year for each concept to create the target database. Identifier and publication's fulltext to extract the cited concepts
-
-***Limitations:*** -
-
-***Environment:***
-Python, [madIS](https://github.com/madgik/madis), [APSW](https://github.com/rogerbinns/apsw)
-
-***References:***
-* Foufoulas Y., Stamatogiannakis L., Dimitropoulos H., Ioannidis Y. (2017) “High-Pass Text Filtering for Citation Matching”. In: Kamps J., Tsakonas G., Manolopoulos Y., Iliadis L., Karydis I. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2017. Lecture Notes in Computer Science, vol 10450. Springer, Cham. [doi:10.1007/978-3-319-67008-9_28](https://doi.org/10.1007/978-3-319-67008-9_28)
-
-***Authority:*** ATHENA RC • ***License:*** CC-BY/CC-0 • ***Code:*** [iis/referenceextraction](https://github.com/openaire/iis/tree/master/iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction)

docs/data-provision/enrichment-by-mining/classifies.md

@@ -1,22 +0,0 @@
----
-sidebar_position: 5
----
-
-# Classifiers
-
-***Short description:*** A document classification algorithm that employs analysis of free text stemming from the abstracts of the publications. The purpose of applying a document classification module is to assign a scientific text to one or more predefined content classes.
-
-***Algorithmic details:***
-The algorithm classifies publication's fulltexts using a Bayesian classifier and weighted terms according to an offline training phase. The training has been done using the following taxonomies: arXiv, MeSH (Medical Subject Headings), ACM, and DDC (Dewey Decimal Classification, or Dewey Decimal System).
-
-***Parameters:*** Publication's identifier and fulltext
-
-***Limitations:*** -
-
-***Environment:***
-Python, [madIS](https://github.com/madgik/madis), [APSW](https://github.com/rogerbinns/apsw)
-
-***References:***
-* Giannakopoulos, T., Stamatogiannakis, E., Foufoulas, I., Dimitropoulos, H., Manola, N., Ioannidis, Y. (2014). Content Visualization of Scientific Corpora Using an Extensible Relational Database Implementation. In: Bolikowski, Ł., Casarosa, V., Goodale, P., Houssos, N., Manghi, P., Schirrwagen, J. (eds) Theory and Practice of Digital Libraries -- TPDL 2013 Selected Workshops. TPDL 2013. Communications in Computer and Information Science, vol 416. Springer, Cham. [doi:10.1007/978-3-319-08425-1_10](https://doi.org/10.1007/978-3-319-08425-1_10)
-
-***Authority:*** ATHENA RC • ***License:*** CC-BY/CC-0 • ***Code:*** [iis/referenceextraction](https://github.com/openaire/iis/tree/master/iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction)

docs/data-provision/enrichment-by-mining/documents_similarity.md

@@ -1,48 +0,0 @@
-# Documents similarity
-
-***Short description:*** Document similarity module is responsible for finding similar documents among the ones available in the OpenAIRE Information Space. It produces "similarity" links between the documents stored in the OpenAIRE Information Space. Each link has a similarity score from [0,1] range assigned; it is expected that the higher the score, the more similar are the documents with respect to their content.
-
-***Algorithmic details:***
-The similarity between two documents is expressed as the similarity between weights of their common terms (i.e., words being reduced to their root form) within a context of all terms from the first and the second document. In this approach, the computation can be divided into three consecutive steps:
-
-1. selection of proper terms,
-2. calculation of weights of terms for each document,
-3. calculation of a given similarity function on weights of terms corresponding to each pair of documents.
-  
-The document similarity module uses the term frequency inverse-document frequency (TFIDF) measure and the cosine similarity to produce weights for terms and calculate their similarity respectively.
-
-*Steps of execution*
-
-Computation of similarity between documents is executed in the following steps.
-
-1. First, we create a text representation of each document. The text is a concatenation of 3 attributes of document object coming from Information Space: title, abstract, and keywords.
-2. Text representation of each document is split into words. Next, stop words or words which occur in more than the N percent of documents (say 99%) or these occurring in less than M documents (say 5) are discarded as we assume that they carry no important information. 	
-3. Next, the words are stemmed (reduced to their root form) and thus converted to terms. The importance of each term in each document is calculated using TFIDF measure (resulting in a vector of weights of terms for each document). Only the top P (say 20) important terms per documents remain for the further computations.	
-4. In order to calculate the cosine similarity value for the documents, we execute the following steps.	
-    a. Triples [document id, term, term weight] are grouped by a common term and for each pair of triples from the group, term importance is recalculated as the multiplication of terms weights, producing quads [document id 1, document id 2, term, multiplied term weight].	
-    b. Quads are grouped by [document id 1, document id 2] and the values of the multiplied term weight are summed up, resulting in the creation of triples [document id 1, document id 2, total common weight]. 	
-    c. Finally, triples are normalized using product of the norm of the term weights' vectors. The normalized value is the final similarity measure with value between 0 and 1.
-5. For a given document, only the top R (say 20) links to similar documents are returned. The links that are thrown away are assumed to be uninteresting for the end-user and thus storing them would only needlessly take disk space.
-
-***Parameters:***
-* input: 
-    * input_document: [DocumentMetadata](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/documentssimilarity/DocumentMetadata.avdl) avro datastore location
-    * parallel: sets parameter parallel for Pig actions (default=80)
-    * mapredChildJavaOpts: mapreduce's map and reduce child java opts set to all PIG actions (default=Xmx12g)
-    * tfidfTopnTermPerDocument: number of the most important terms taken into account (default=20)
-    * similarityTopnDocumentPerDocument: maximum number of similar documents for each publication (default=20)
-    * removal_rate: removal rate (default=0.99)
-    * removal_least_used: removal of the least used terms (default=20)
-    * threshold_num_of_vector_elems_length: vector elements length threshold, when set to less than 2 all documents will be included in similarity matching (default=2)
-* output: [DocumentSimilarity](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/documentssimilarity/DocumentSimilarity.avdl) avro datastore location
-
-***Limitations:*** -
-
-***Environment:*** 
-Pig, Java
-
-***References:***
-
-* P. J. Dendek, A. Czeczko, M. Fedoryszak, A. Kawa, and L. Bolikowski, "Content Analysis of Scientific Articles in Apache Hadoop Ecosystem", Stud. Comp.Intelligence, vol. 541, 2014.
-
-***Authority:*** ICM • ***License:*** AGPL-3.0 • ***Code:*** [CoAnSys/document-similarity](https://github.com/CeON/CoAnSys/tree/master/document-similarity)

docs/data-provision/enrichment-by-mining/metadata_extraction.md

@@ -1,36 +0,0 @@
-# Metadata extraction
-
-***Short description:*** Metadata Extraction algorithm is responsible for plaintext and metadata extraction out of the PDF documents. It based on [CERMINE](http://cermine.ceon.pl/about.html) project.
-
-CERMINE is a comprehensive open source system for extracting metadata and content from scientific articles in born-digital form. The system is able to process documents in PDF format and extracts:
-
-* document's metadata, including title, authors, affiliations, abstract, keywords, journal name, volume and issue,
-* parsed bibliographic references
-* the structure of document's sections, section titles and paragraphs
-
-CERMINE is based on a modular workflow, whose architecture ensures that individual workflow steps can be maintained separately. As a result it is easy to perform evaluation, training, improve or replace one step implementation without changing other parts of the workflow. Most steps implementations utilize supervised and unsupervised machine-leaning techniques, which increases the maintainability of the system, as well as its ability to adapt to new document layouts.
-
-***Algorithmic details:***
-CERMINE workflow is composed of four main parts:
-
-* Basic structure extraction takes a PDF file on the input and produces a geometric hierarchical structure representing the document. The structure is composed of pages, zones, lines, words and characters. The reading order of all elements is determined. Every zone is labelled with one of four general categories: METADATA, REFERENCES, BODY and OTHER.
-* Metadata extraction part analyses parts of the geometric hierarchical structure labelled as METADATA and extracts a rich set of document's metadata from it.
-* References extraction part analyses parts of the geometric hierarchical structure labelled as REFERENCES and the result is a list of document's parsed bibliographic references.
-* Text extraction part analyses parts of the geometric hierarchical structure labelled as BODY and extracts document's body structure composed of sections, subsections and paragraphs.
-
-CERMINE uses supervised and unsupervised machine-leaning techniques, such as Support Vector Machines, K-means clustering and Conditional Random Fields. Content classifiers are trained on [GROTOAP2 dataset](http://cermine.ceon.pl/grotoap2/). More information about CERMINE can be found in the [presentation](http://cermine.ceon.pl/static/docs/slides.pdf).
-
-***Parameters:***
-* input: [DocumentText](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/metadataextraction/DocumentText.avdl) avro datastore location
-* output: [ExtractedDocumentMetadata](https://github.com/openaire/iis/blob/master/iis-schemas/src/main/avro/eu/dnetlib/iis/metadataextraction/ExtractedDocumentMetadata.avdl) avro datastore location
-
-***Limitations:***
-Born-digital form of PDF documents is supported only. Large PDF documents may require more than 4g of assgined memory (set by default).
-
-***Environment:*** 
-Java, Hadoop
-
-***References:***
-* Dominika Tkaczyk, Pawel Szostek, Mateusz Fedoryszak, Piotr Jan Dendek and Lukasz Bolikowski. CERMINE: automatic extraction of structured metadata from scientific literature. In International Journal on Document Analysis and Recognition, 2015, vol. 18, no. 4, pp. 317-335, doi: 10.1007/s10032-015-0249-8.
-
-***Authority:*** ICM • ***License:*** AGPL-3.0 • ***Code:*** [CERMINE](https://github.com/CeON/CERMINE)

docs/data-provision/enrichment/enrichment.md

@@ -0,0 +1,44 @@
+# Enrichment
+
+## Mining
+
+The OpenAIRE Research Graph is enriched by links mined by OpenAIRE’s full-text mining algorithms that scan the plaintexts of publications for funding information, references to datasets, software URIs, accession numbers of bioetities, and EPO patent mentions. Custom mining modules also link research objects to specific research communities, initiatives and infrastructures. In addition, other inference modules provide content-based document classification, document similarity, citation matching, and author affiliation matching.
+
+**Project mining** in OpenAIRE text mines the full-texts of publications in order to extract matches to funding project codes/IDs. The mining algorithm works by utilising (i) the grant identifier, and (ii) the project acronym (if available) of each project. The mining algorithm: (1) Preprocesses/normalizes the full-texts using several functions, which depend on the characteristics of each funder (i.e., the format of the grant identifiers), such as stopword and/or punctuation removal, tokenization, stemming, converting to lowercase; then (2) String matching of grant identifiers against the normalized text is done using database techniques; and (3) The results are validated and cleaned using the context near the match by looking at the context around the matched ID for relevant metadata and positive or negative words/phrases, in order to calculate a confidence value for each publication-->project link. A confidence threshold is set to optimise high accuracy while minimising false positives, such as matches with page or report numbers, post/zip codes, parts of telephone numbers, DOIs or URLs, accession numbers. The algorithm also applies rules for disambiguating results, as different funders can share identical project IDs; for example, grant number 633172 could refer to H2020 project EuroMix but also to Australian-funded NHMRC project “Brain activity (EEG) analysis and brain imaging techniques to measure the neurobiological effects of sleep apnea”. Project mining works very well and was the first Text & Data Mining (TDM) service of OpenAIRE. Performance results vary from funder to funder but precision is higher than 98% for all funders and 99.5% for EC projects. Recall is higher than 95% (99% for EC projects), when projects are properly acknowledged using project/grant IDs.
+
+**Dataset extraction** runs on publications full-texts as described in “High pass text-filtering for Citation matching”, TPDL 2017[1]. In particular, we search for citations to datasets using their DOIs, titles and other metadata (i.e., dates, creator names, publishers, etc.). We extract parts of the text which look like citations and search for datasets using database join and pattern matching techniques. Based on the experiments described in the paper, precision of the dataset extraction module is 98.5% and recall is 97.4% but it is also probably overestimated since it does not take into account corruptions that may take place during pdf to text extraction. It is calculated on the extracted full-texts of small samples from PubMed and arXiv.
+
+**Software extraction** runs also on parts of the text which look like citations. We search the citations for links to software in open software repositories, specifically github, sourceforge, bitbucket and the google code archive. After that, we search for links that are included in Software Heritage (SH, https://www.softwareheritage.org) and return the permanent URL that SH provides for each software project. We also enrich this content with user names, titles and descriptions of the software projects using web mining techniques. Since software mining is based on URL matching, our precision is 100% (we return a software link only if we find it in the text and there is no need to disambiguate). As for recall rate, this is not calculable for this mining task. Although we apply all the necessary normalizations to the URLs in order to overcome usual issues (e.g., http or https, existence of www or not, lower/upper case), we do not calculate cases where a software is mentioned using its name and not by a link from the supported software repositories.
+
+**For the extraction of bio-entities**, we focus on Protein Data Bank (PDB) entries. We have downloaded the database with PDB codes and we update it regularly. We search through the whole publication’s full-text for references to PDB codes. We apply disambiguation rules (e.g., there are PDB codes that are the same as antibody codes or other issues) so that we return valid results. Current precision is 98%. Although it's risky to mention recall rates since these are usually overestimated, we have calculated a recall rate of 98% using small samples from pubmed publications. Moreover, our technique is able to identify about 30% more links to proteins than the ones that are tagged in Pubmed xmls.
+
+**Other text-mining modules** include mining for links to EPO patents, or custom mining modules for linking research objects to specific research communities, initiatives and infrastructures, e.g. COVID-19 mining module. Apart from text-mining modules, OpenAIRE also provides a document classification service that employs analysis of free text stemming from the abstracts of the publications. The purpose of applying a document classification module is to assign a scientific text one or more predefined content classes. In OpenAIRE, the currently used taxonomies are arXiv, MeSH (Medical Subject Headings), ACM and DDC (Dewey Decimal Classification, or Dewey Decimal System).
+
+## Bulk Tagging/Deduction
+
+The Deduction process (also known as “bulk tagging”) enriches each record with new information that can be derived from the existing property values.
+
+As of September 2020, three procedures are in place to relate a research product to a research initiative, infrastructure (RI) or community (RC) based on:
+
+* subjects (2.7M results tagged)
+
+* Zenodo community (16K results tagged)
+
+* the data source it comes from (250K results tagged)
+
+The list of subjects, Zenodo communities and data sources used to enrich the products are defined by the managers of the community gateway or infrastructure monitoring dashboard associated with the RC/RI.
+
+## Propagation
+
+This process “propagates” properties and links from one product to another if between the two there is a “strong” semantic relationship.
+
+As of September 2020, the following procedures are in place:
+Propagation of the property “country” to results from institutional repositories: e.g. publication collected from an institutional repository maintained by an italian university will be enriched with the property “country = IT”.
+
+* Propagation of links to projects: e.g. publication linked to project P “is supplemented by” a dataset D. Dataset D will get the link to project P. The relationships considered for this procedure are “isSupplementedBy” and “supplements”.
+
+* Propagation of related community/infrastructure/initiative from organizations to products via affiliation relationships: e.g. a publication with an author affiliated with organization O. The manager of the community gateway C declared that the outputs of O are all relevant for his/her community C. The publication is tagged as relevant for C.
+
+* Propagation of related community/infrastructure/initiative to related products: e.g. publication associated to community C is supplemented by a dataset D. Dataset D will get the association to C. The relationships considered for this procedure are “isSupplementedBy” and “supplements”.
+
+* Propagation of ORCID identifiers to related products, if the products have the same authors: e.g. publication has ORCID for its authors and is supplemented by a dataset D. Dataset D has the same authors as the publication. Authors of D are enriched with the ORCIDs available in the publication. The relationships considered for this procedure are “isSupplementedBy” and “supplements”.

docs/data-provision/enrichment/impact-scores.md

@@ -0,0 +1,73 @@
+---
+sidebar_position: 2
+---
+
+# Impact scores
+<span className="todo">TODO - add intro</span>
+
+## Citation Count (CC)
+
+This is the most widely used scientific impact indicator, which sums all citations received by each article. The citation count of a 
+publication $i$ corresponds to the in-degree of the corresponding node in the underlying citation network: $s_i = \sum_{j} A_{i,j}$, 
+where $A$ is the adjacency matrix of the network (i.e., $A_{i,j}=1$ when paper $j$ cites paper $i$, while $A_{i,j}=0$ otherwise). 
+Citation count can be viewed as a measure of a publication's overall impact, since it conveys the number of other works that directly 
+drew on it.
+
+## "Incubation" Citation Count (iCC)
+
+This measure is essentially a time-restricted version of the citation count, where the time window is distinct for each paper, i.e., 
+only citations $y$ years after its publication are counted (usually, $y=3$). The "incubation" citation count of a paper $i$ is 
+calculated as: $s_i = \sum_{j,t_j \leq t_i+3} A_{i,j}$, where $A$ is the adjacency matrix and $t_j, t_i$ are the citing and cited paper's 
+publication years, respectively. $t_i$ is cited paper $i$'s publication year. iCC can be seen as an indicator of a paper's initial momentum 
+(impulse) directly after its publication.
+
+## PageRank (PR)
+
+Originally developed to rank Web pages, PageRank has been also widely used to rank publications in citation
+networks. In this latter context, a publication's PageRank 
+score also serves as a measure of its influence. In particular, the PageRank score of a publication is calculated 
+as its probability of being read by a researcher that either randomly selects publications to read or selects 
+publications based on the references of her latest read. Formally, the score of a publication $i$ is given by: 
+
+$$
+s_i = \alpha \cdot \sum_{j} P_{i,j} \cdot s_j + (1-\alpha) \cdot \frac{1}{N}
+$$
+
+where $P$ is the stochastic transition matrix, which corresponds to the column normalised version of adjacency 
+matrix $A$, $\alpha \in [0,1]$, and $N$ is the number of publications in the citation network. The first addend 
+of the equation corresponds to the selection (with probability $\alpha$) of following a reference, while the 
+second one to the selection of randomly choosing any publication in the network. It should be noted that the 
+score of each publication relies of the score of publications citing it (the algorithm is executed iteratively 
+until all scores converge). As a result, PageRank differentiates citations based on the importance of citing 
+articles, thus alleviating the corresponding issue of the Citation Count.
+
+## RAM
+
+RAM is essentially a modified Citation Count, where recent citations are considered of higher importance compared 
+to older ones. Hence, it better captures the popularity of publications. This "time-awareness" of citations 
+alleviates the bias of methods like Citation Count and PageRank against recently published articles, which have 
+not had "enough" time to gather as many citations. The RAM score of each paper $i$ is calculated as follows:
+
+$$
+s_i = \sum_j{R_{i,j}}
+$$
+
+where $R$ is the so-called Retained Adjacency Matrix (RAM) and $R_{i,j}=\gamma^{t_c-t_j}$ when publication $j$ cites publication 
+$i$, and $R_{i,j}=0$ otherwise. Parameter $\gamma \in (0,1)$, $t_c$ corresponds to the current year and $t_j$ corresponds to the 
+publication year of citing article $j$.
+
+## AttRank
+
+AttRank is a PageRank variant that alleviates its bias against recent publications (i.e., it is tailored to capture popularity). 
+AttRank achieves this by modifying PageRank's probability of randomly selecting a publication. Instead of using a uniform probability,
+AttRank defines it based on a combination of the publication's age and the citations it received in recent years. The AttRank score 
+of each publication $i$ is calculated based on:
+
+$$
+s_i = \alpha \cdot \sum_{j} P_{i,j} \cdot s_j
+    + \beta \cdot Att(i)+ \gamma \cdot c \cdot e^{-\rho \cdot (t_c-t_i)}
+$$
+
+where $\alpha + \beta + \gamma =1$ and $\alpha,\beta,\gamma \in [0,1]$. $Att(i)$ denotes a recent attention-based score for publication $i$, 
+which reflects its share of citations in the $y$ most recent years, $t_i$ is the publication year of article $i$, $t_c$ denotes the current 
+year, and $c$ is a normalisation constant. Finally, $P$ is the stochastic transition matrix.

docs/data-provision/enrichment/mining.md

@@ -0,0 +1,6 @@
+---
+sidebar_position: 1
+---
+
+# Mining algorithms
+<span className="todo">TODO</span>

docs/data-provision/finalisation.md

@@ -1,18 +0,0 @@
-# Finalisation
-
-At the very end of the graph production workflow, a step is dedicated to perform certain finalisation operations, that we describe in this page, 
-aiming to improve the overall quality of the data. 
-The output of this final step is the final version of the OpenAIRE Research Graph.
-
-## Filtering
-
-Bibliographic records that do not meet minimal requirements for being part of the OpenAIRE Research Graph are eliminated during this phase. 
-Currently, the only criteria applied horizontally to the entire graph aims at excluding scientific results whose title is not meaningful for citation purposes.
-Then, different criteria are applied in the pre-processing of specific sub-collections:
-
-* [Crossref filtering](/data-provision/aggregation/non-compatible-sources/doiboost#crossref-filtering)
-
-## Country cleaning
-
-This phase is responsible for removing the country information from result records that match specific criteria. The need for this phase is driven by the fact that some datasources, although referred of national pertinence, they contain material that is not always related to the given country. 
-

docs/data-provision/indexing.md

@@ -1,17 +1,13 @@
+---
+sidebar_position: 5
+---
+
 # Indexing
 
 The final version of the OpenAIRE Research Graph is indexed on a Solr server that is used by the OpenAIRE portals (EXPLORE, CONNECT, PROVIDE) and APIs, the latter adopted by several third-party applications and organizations, such as:
 
-* The OpenAIRE Research Graph APIs and Portals will offer to the EOSC (European Open Science Cloud) an Open Science Resource Catalogue, keeping an up to date map of all research results (publications, datasets, software), services, organizations, projects, funders in Europe and beyond.
+* EOSC --The OpenAIRE Research Graph APIs and Portals will offer to the EOSC an Open Science Resource Catalogue, keeping an up to date map of all research results (publications, datasets, software), services, organizations, projects, funders in Europe and beyond.
 
-* DSpace & EPrints repositories can install the OpenAIRE plugin to expose OpenAIRE compliant metadata records via their OAI-PMH endpoint and offer to researchers the possibility to link their depositions to the funding project, by selecting it from the list of project provided by OpenAIRE.
+* DSpace & EPrints repositories can install the OpenAIRE plugin to expose OpenAIRE compliant metadata records via their OAI-PMH endpoint and offer to researchers the possibility to link their depositions to the funding project, by selecting it from the list of project provided by OpenAIRE
 
-* EC participant portal (Sygma - System for Grant Management) uses the OpenAIRE API in the “Continuous Reporting” section. Sygma automatically fetches from the OpenAIRE Search API the list of publications and datasets in the OpenAIRE Research Graph that are linked to the project. The user can select the research products from the list and easily compile the continuous reporting data of the project.
-
-* ScholExplorer is used by different players of the scholarly communication ecosystem. For example, [Elsevier](https://www.elsevier.com/authors/tools-and-resources/research-data/data-base-linking) uses its API to make the links between 
-publications and datasets automatically appear on ScienceDirect.
-ScholExplorer indexes the links among the four major types of research products (API v3) available in the OpenAIRE Research Graph and makes them available through an HTTP API that allows 
-to search them by the following criteria:
-  * Links whose source object has a given PID or PID type;
-  * Links whose source object has been published by a given data source ("data source as publisher");
-  * Links that were collected from a given data source ("data source as provider").
+* EC participant portal (Sygma - System for Grant Management) uses the OpenAIRE API in the “Continuous Reporting” section. Sygma automatically fetches from the OpenAIRE Search API the list of publications and datasets in the OpenAIRE Research Graph that are linked to the project. The user can select the research products from the list and easily compile the continuous reporting data of the project.

docs/data-provision/indicators-ingestion/impact-scores.md

@@ -1,169 +0,0 @@
-# Impact indicators
-
-This page summarises all calculated impact indicators, which are included into the [measure](/data-model/entities/other#measure) property.
-It should be noted that the impact indicators are being calculated both on the level of the research output as well on the level of distinct DOIs.
-Below we explain their main intuition, the way they are calculated, and their most important limitations, in an attempt help avoiding common pitfalls and misuses.
-
-
-## Citation Count (CC)
-
-***Short description:***
-This is the most widely used scientific impact indicator, which sums all citations received by each article.
-Citation count can be viewed as a measure of a publication's overall impact, since it conveys the number of other works that directly 
-drew on it.
-
-***Algorithmic details:***
-The citation count of a 
-publication $i$ corresponds to the in-degree of the corresponding node in the underlying citation network: $s_i = \sum_{j} A_{i,j}$, 
-where $A$ is the adjacency matrix of the network (i.e., $A_{i,j}=1$ when paper $j$ cites paper $i$, while $A_{i,j}=0$ otherwise). 
-
-***Parameters:*** -
-
-***Limitations:***
-OpenAIRE collects data from specific data sources which means that part of the existing literature may not be considered when computing this indicator.
-Also, since some indicators require the publication year for their calculation, we consider only research products for which we can gather this information from at least one data source.
-
-***Environment:*** PySpark
-
-***References:*** -
-
-***Authority:*** ATHENA RC • ***License:*** GPL-2.0 • ***Code:*** [BIP! Ranker](https://github.com/athenarc/Bip-Ranker)
-
-
-## "Incubation" Citation Count (iCC)
-
-***Short description:***
-This measure is essentially a time-restricted version of the citation count, where the time window is distinct for each paper, i.e., 
-only citations $y$ years after its publication are counted.
-
-***Algorithmic details:***
-The "incubation" citation count of a paper $i$ is 
-calculated as: $s_i = \sum_{j,t_j \leq t_i+y} A_{i,j}$, where $A$ is the adjacency matrix and $t_j, t_i$ are the citing and cited paper's 
-publication years, respectively. $t_i$ is cited paper $i$'s publication year. iCC can be seen as an indicator of a paper's initial momentum 
-(impulse) directly after its publication.
-
-***Parameters:*** 
-$y=3$
-
-***Limitations:***
-OpenAIRE collects data from specific data sources which means that part of the existing literature may not be considered when computing this indicator.
-Also, since some indicators require the publication year for their calculation, we consider only research products for which we can gather this information from at least one data source.
-
-***Environment:*** PySpark
-
-***References:*** 
-* Vergoulis, T., Kanellos, I., Atzori, C., Mannocci, A., Chatzopoulos, S., Bruzzo, S. L., Manola, N., & Manghi, P. (2021, April). Bip! db: A dataset of impact measures for scientific publications. In Companion Proceedings of the Web Conference 2021 (pp. 456-460).
-
-***Authority:*** ATHENA RC • ***License:*** GPL-2.0 • ***Code:*** [BIP! Ranker](https://github.com/athenarc/Bip-Ranker)
-
-
- ## PageRank (PR)
-
-***Short description:***
-Originally developed to rank Web pages, PageRank has been also widely used to rank publications in citation
-networks. In this latter context, a publication's PageRank 
-score also serves as a measure of its influence.
-
-***Algorithmic details:***
-The PageRank score of a publication is calculated 
-as its probability of being read by a researcher that either randomly selects publications to read or selects 
-publications based on the references of her latest read. Formally, the score of a publication $i$ is given by: 
-
-$$
-s_i = \alpha \cdot \sum_{j} P_{i,j} \cdot s_j + (1-\alpha) \cdot \frac{1}{N}
-$$
-
-where $P$ is the stochastic transition matrix, which corresponds to the column normalised version of adjacency 
-matrix $A$, $\alpha \in [0,1]$, and $N$ is the number of publications in the citation network. The first addend 
-of the equation corresponds to the selection (with probability $\alpha$) of following a reference, while the 
-second one to the selection of randomly choosing any publication in the network. It should be noted that the 
-score of each publication relies of the score of publications citing it (the algorithm is executed iteratively 
-until all scores converge). As a result, PageRank differentiates citations based on the importance of citing 
-articles, thus alleviating the corresponding issue of the Citation Count.
-
-***Parameters:*** 
-$\alpha = 0.5, convergence\_error = 10^{-12}$
-
-***Limitations:***
-OpenAIRE collects data from specific data sources which means that part of the existing literature may not be considered when computing this indicator.
-Also, since some indicators require the publication year for their calculation, we consider only research products for which we can gather this information from at least one data source.
-
-***Environment:*** PySpark
-
-***References:*** 
-* Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: Bringing order to the web. Stanford InfoLab.
-
-***Authority:*** ATHENA RC • ***License:*** GPL-2.0 • ***Code:*** [BIP! Ranker](https://github.com/athenarc/Bip-Ranker)
- 
-
-## RAM
-
-***Short description:***
-RAM is essentially a modified Citation Count, where recent citations are considered of higher importance compared to older ones.
-Hence, it better captures the popularity of publications. This "time-awareness" of citations 
-alleviates the bias of methods like Citation Count and PageRank against recently published articles, which have 
-not had "enough" time to gather as many citations.
-
-***Algorithmic details:***
-The RAM score of each paper $i$ is calculated as follows:
-
-$$
-s_i = \sum_j{R_{i,j}}
-$$
-
-where $R$ is the so-called Retained Adjacency Matrix (RAM) and $R_{i,j}=\gamma^{t_c-t_j}$ when publication $j$ cites publication 
-$i$, and $R_{i,j}=0$ otherwise. Parameter $\gamma \in (0,1)$, $t_c$ corresponds to the current year and $t_j$ corresponds to the 
-publication year of citing article $j$.
-
-***Parameters:*** 
-$\gamma = 0.6$
-
-***Limitations:***
-OpenAIRE collects data from specific data sources which means that part of the existing literature may not be considered when computing this indicator.
-Also, since some indicators require the publication year for their calculation, we consider only research products for which we can gather this information from at least one data source.
-
-***Environment:*** PySpark
-
-***References:*** 
-* Ghosh, R., Kuo, T. T., Hsu, C. N., Lin, S. D., & Lerman, K. (2011, December). Time-aware ranking in dynamic citation networks. In 2011 ieee 11^{th} international conference on data mining workshops (pp. 373-380). IEEE.
-
-***Authority:*** ATHENA RC • ***License:*** GPL-2.0 • ***Code:*** [BIP! Ranker](https://github.com/athenarc/Bip-Ranker)
-
-
-## AttRank
-
-***Short description:***
-AttRank is a PageRank variant that alleviates its bias against recent publications (i.e., it is tailored to capture popularity). 
-AttRank achieves this by modifying PageRank's probability of randomly selecting a publication. Instead of using a uniform probability,
-AttRank defines it based on a combination of the publication's age and the citations it received in recent years.
-
-***Algorithmic details:***
-The AttRank score 
-of each publication $i$ is calculated based on:
-
-$$
-s_i = \alpha \cdot \sum_{j} P_{i,j} \cdot s_j
-    + \beta \cdot Att(i)+ \gamma \cdot c \cdot e^{-\rho \cdot (t_c-t_i)}
-$$
-
-where $\alpha + \beta + \gamma =1$ and $\alpha,\beta,\gamma \in [0,1]$. $Att(i)$ denotes a recent attention-based score for publication $i$, 
-which reflects its share of citations in the $y$ most recent years, $t_i$ is the publication year of article $i$, $t_c$ denotes the current 
-year, and $c$ is a normalisation constant. Finally, $P$ is the stochastic transition matrix.
-
-***Parameters:*** 
-$\alpha = 0.2, \beta = 0.5, \gamma = 0.3, \rho = -0.16, convergence\_error = 10^-{12}$
-
-Note that recent attention is based on the 3 most recent years (including current one).
-
-***Limitations:***
-OpenAIRE collects data from specific data sources which means that part of the existing literature may not be considered when computing this indicator.
-Also, since some indicators require the publication year for their calculation, we consider only research products for which we can gather this information from at least one data source.
-
-***Environment:*** PySpark
-
-***References:*** 
-* Kanellos, I., Vergoulis, T., Sacharidis, D., Dalamagas, T., & Vassiliou, Y. (2021, April). Ranking papers by their short-term scientific impact. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) (pp. 1997-2002). IEEE.
-
-***Authority:*** ATHENA RC • ***License:*** GPL-2.0 • ***Code:*** [BIP! Ranker](https://github.com/athenarc/Bip-Ranker)
-
- 

docs/data-provision/indicators-ingestion/usage-counts.md

@@ -1,7 +0,0 @@
-# Usage Statistics Indicators
-
-Usage Statistics indicators for research products, like publications, datasets,etc., are an important complement to other (traditional and alternative) bibliometric indicators to provide a comprehensive and recent view of the impact of such resources but also about their authors, institutions and the platforms themselves. They are taking into account different levels of information: the usage of data sources, the usage of individual items in the context of their resource type and the usage of individual web resources or files.
-
-Usage Statistics Indicators are built by the OpenAIRE's UsageCounts Service. The service collects usage data and consolidated usage statistics reports respectively, from its distributed network of data providers (repositories, e-journals, CRIS) by utilizing open standards and protocols and delivers reliable, consolidated and comparable usage metrics like counts of item downloads and metadata views conformant to COUNTER Code of Practice.
-
-You can find more information about the UsageCounts service [here](https://usagecounts.openaire.eu/).

docs/data-provision/merge-by-id.md

@@ -1,3 +0,0 @@
-# Merge by id
-
-<span className="todo">TODO</span>

docs/data-provision/post-cleaning.md

@@ -0,0 +1,9 @@
+---
+sidebar_position: 4
+---
+
+# Post-cleaning
+
+The aggregation processes are continuously running and apply vocabularies as they are in a given moment of time. It could be the case that a vocabulary changes after the aggregation of one data source has finished, thus the aggregated content does not reflect the current status of the controlled vocabularies.
+
+In addition, the integration of ScholeXplorer and DOIBoost and some enrichment processes applied on the raw and on the de-duplicated graph may introduce values that do not comply with the current status of the OpenAIRE controlled vocabularies. For these reasons, we included a final step of cleansing at the end of the workflow materialisation. The output of the final cleansing step is the final version of the OpenAIRE Research Graph.

docs/data-provision/stats.md

@@ -0,0 +1,7 @@
+---
+sidebar_position: 6
+---
+
+# Stats analysis
+
+The OpenAIRE Research Graph is also processed by a pipeline for extracting the statistics and producing the charts for funders, research initiative, infrastructures, and policy makers that you can see on MONITOR. Based on the information available on the graph, OpenAIRE provides a set of indicators for monitoring the funding and research impact and the uptake of Open Science publishing practices, such as Open Access publishing of publications and datasets, availability of interlinks between research products, availability of post-print versions in institutional or thematic Open Access repositories, etc.

docs/download.md

@@ -0,0 +1,17 @@
+---
+sidebar_position: 4
+---
+
+# Bulk downloads
+
+In order to facilitate users, different dumps are available. All are available under the Zenodo community called [OpenAIRE Research Graph](https://zenodo.org/communities/openaire-research-graph).
+Here we provide detailed documentation about the full dump:
+
+* JSON dump: https://doi.org/10.5281/zenodo.3516917
+* JSON schema: https://doi.org/10.5281/zenodo.4238938
+
+:::note Tip!
+
+For a visual and interactive overview of the JSON schema, we suggest to use a JSON schema viewer like [jsonschemaviewer](https://navneethg.github.io/jsonschemaviewer/) (you just need to copy the schema and then you can easily navigate through the nodes).
+
+:::

docs/downloads/alternative-model/cfhb.md

@@ -1,30 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-# CfHbKeyValue
-
-Information about the sources from which the record has been collected.
-
-
-	@JsonSchema(description = "the OpenAIRE identifier of the data source")
-### key
-_Type: String • Cardinality: ONE_
-
-the OpenAIRE identifier of the data source
-	
-```json
-"key":"10|openaire____::081b82f96300b6a6e3d282bad31cb6e2"
-```
-
-### value
-_Type: String • Cardinality: ONE_
-
-The name of the data source.
-
-```json 
-"value":"Crossref"
-```
-

docs/downloads/alternative-model/communityInstance.md

@@ -1,37 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-# CommunityInstance
-
-It is a subclass of [Instance](../../data-model/entities/result#instance) extended with information regarding the collection and hosting source for this materialization of the result.
-
-### hostedby
-_Type: [CfHbKeyValue](./cfhb) • Cardinality: ONE_
-
-Information about the source from which the instance can be viewed or downloaded.
-
-```json
-
-"hostedby": {
-                "key": "10|issn___print::35ee75a5ad42581d604be113a8f56427",
-                "value": "New Phytologist"
-            },
-            
-```
-
-### collectedfrom
-_Type: [CfHbKeyValue](./cfhb) • Cardinality: ONE_
-
-Information about the source from which the record has been collected
-
-
-```json
-
-"collectedfrom": {
-                "key": "10|openaire____::081b82f96300b6a6e3d282bad31cb6e2",
-                "value": "Crossref"
-            }
-```

docs/downloads/alternative-model/context.md

@@ -1,46 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-# Context
-
-Information related to research initiative/community (RI/RC) related to the result.
-
-### code 
-_Type: String • Cardinality: ONE_
-
-Code identifying the RI/RC.
-
-```json
-"code":"sdsn-gr"
-
-```
-
-
-### label
-_Type: String • Cardinality: ONE_
-
-Label of the RI/RC.
-
-```json
-"label":"SDSN - Greece"
-```
-	
-### provenance 
-_Type: [Provenance](/data-model/entities/other#provenance-2)  • Cardinality: MANY_
-
-Why this result is associated to the RI/RC.
-
-```json
-
-"provenance":[{
-                "provenance":"Inferred by OpenAIRE",
-                "trust":"0.9"
-              },
-              ...
-   ] 
-
-```
-

docs/downloads/alternative-model/extendedresult.md

@@ -1,141 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-
-# Extended Result
-
-
-It is a subclass of [Result](/data-model/entities/result) extended with information regarding projects (and funders), research communities/infrastructure and related data sources.
-
-
-
-### projects
-
-_Type: [Project](project.md)  • Cardinality: MANY_
-
-
-List of projects (i.e. grants) that (co-)funded the production of the research results.
-
-
-```json
-
-
-"projects": [
-        {
-            "id": "40|corda__h2020::94c4a066401e22002c4811a301bb4655",
-            "code": "727929",
-            "acronym": "TomRes",
-            "title": "A NOVEL AND INTEGRATED APPROACH TO INCREASE MULTIPLE AND COMBINED STRESS TOLERANCE IN PLANTS USING TOMATO AS A MODEL",
-            "funder": {
-                "shortName": "EC",
-                "name": "European Commission",
-                "jurisdiction": "EU",
-                "fundingStream": "H2020"
-            },
-            "provenance": {
-                "provenance": "Harvested",
-                "trust": "0.900000000000000022"
-            },
-            "validated": {
-                "validationDate": "2021-0101",
-                "validatedByFunder": true
-            }
-        },
-        ...
-    ]
-
-```
-
-### context
-
-_Type: [Context](./context) • Cardinality: MANY_
-
-
-Reference to relevant research infrastructure, initiative or communities (RI/RC) among those collaborating with OpenAIRE. Please see https://connect.openaire.eu that are publicly visible.
-
-
-```json
-
-
-"context":[
-        {
-            "code":"sdsn-gr",
-            "label":"SDSN - Greece",
-            "provenance":[
-                {
-                    "provenance":"Inferred by OpenAIRE",
-                    "trust":"0.9"
-                }
-            ]
-        },
-        ...
-    ] 
-
-```
-
-
-
-### collectedfrom
-
-_Type: [CfHbKeyValue](./cfhb) • Cardinality: MANY_
-
-
-Information about the sources from which the record has been collected.
-
-
-```json
-
-"collectedfrom":[
-        {
-            "key":"10|openaire____::081b82f96300b6a6e3d282bad31cb6e2",
-            "value":"Crossref"
-        },
-        ...    
-   ]
-
-```
-
-
-### instance
-
-_Type: [CommunityInstance](./communityInstance) • Cardinality: MANY_
-
-Information about the source from which the instance can be viewed or downloaded.
-
-```json
-
-
-"instance": [
-        {
-            "license": "http://doi.wiley.com/10.1002/tdm_license_1.1",
-            "accessright": {
-                "code": "c_16ec",
-                "label": "RESTRICTED",
-                "scheme": "http://vocabularies.coar-repositories.org/documentation/access_rights/",
-                "openAccessRoute": null
-            },
-            "type": "Article",
-            "url": [
-                "https://api.wiley.com/onlinelibrary/tdm/v1/articles/10.1111%2Fnph.15014",
-                "http://onlinelibrary.wiley.com/wol1/doi/10.1111/nph.15014/fullpdf",
-                "http://dx.doi.org/10.1111/nph.15014"
-            ],
-            "publicationdate": "2018-02-09",
-            "refereed": "UNKNOWN",
-            "hostedby": {
-                "key": "10|issn___print::35ee75a5ad42581d604be113a8f56427",
-                "value": "New Phytologist"
-            },
-            "collectedfrom": {
-                "key": "10|openaire____::081b82f96300b6a6e3d282bad31cb6e2",
-                "value": "Crossref"
-            }
-        },
-         ...
-    ]
-
-
-```

docs/downloads/alternative-model/funder.md

@@ -1,72 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-# Funder
-
-
-Information about the funder funding the project.
-
-
-### fundingStream
-
-_Type: String • Cardinality: ONE_
-
-
-Funding information for the project.
-
-
-```json
-
-"funding_stream": "H2020"
-
-
-```
-
-### jurisdiction
-
-_Type: String • Cardinality: ONE_
-
-
-Geographical jurisdiction (e.g. for European Commission is EU, for Croatian Science Foundation is HR).
-
-
-```json
-
-"jurisdiction": "EU"
-
-```
-
-
-### name
-
-_Type: String • Cardinality: ONE_
-
-
-The name of the funder.
-
-
-```json
-
-"name": "European Commission"
-
-```
-
-
-### shortName
-
-_Type: String • Cardinality: ONE_
-
-
-The short name of the funder.
-
-
-```json
-
-"shortName": "EC"
-
-```
-
-

docs/downloads/alternative-model/project.md

@@ -1,134 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-
-
-# Project
-
-
-The information about the projects related to the result.
-
-
-### id
-
-_Type: String • Cardinality: ONE_
-
-
-Main entity identifier, created according to the [OpenAIRE entity identifier and PID mapping policy](../../data-model/pids-and-identifiers).
-
-
-```json
-
-"id": "40|corda__h2020::70ea22400fd890c5033cb31642c4ae68"
-
-```
-
-
-### code
-
-_Type: String • Cardinality: ONE_
-
-
-Τhe grant agreement code of the project.
-
-
-```json
-
-"code": "777541"
-
-```
-
-
-### acronym
-
-_Type: String • Cardinality: ONE_
-
-
-Project's acronym.
-
-
-```json
-
-"acronym": "OpenAIRE-Advance"
-
-```
-
-
-### title
-
-_Type: String • Cardinality: ONE_
-
-
-Project's title.
-
-
-```json
-
-"title": "OpenAIRE Advancing Open Scholarship"
-
-```
-
-
-### funder
-
-_Type [Funder](funder.md)  • Cardinality: ONE_
-
-
-Information about the funder funding the project.
-
-
-```json
-
-
-"funder": {
-                "shortName": "EC",
-                "name": "European Commission",
-                "jurisdiction": "EU",
-                "fundingStream": "H2020"
-            }
-
-
-```
-
-### provenace
-
-
-_Type [Provenance](../../data-model/entities/other#provenance-2)  • Cardinality: ONE_
-
-
-The reason why the project is associated to the result.
-
-
-```json
-
-
-"provenance": {
-                  "provenance": "Harvested",
-                  "trust": "0.900000000000000022"
-              }
-
-```
-
-
-### validated
-
-
-_Type [Validated](validated.md)  • Cardinality: ONE_
-
-
-Specifies it the association between the project and the result was validated.
-
-
-```json
-
-
-"validated": {
-                "validationDate": "2021-0101",
-                "validatedByFunder": true
-            }
-
-```
-

docs/downloads/alternative-model/validated.md

@@ -1,41 +0,0 @@
----
-
-sidebar_position: 1
-
----
-
-# Validated
-
-
-Information about the validtion of the association between the result and the funding information.
-
-
-### validationDate
-
-_Type: String • Cardinality: ONE_
-
-
-When OpenAIRE collected the association between the funding and the result from an authoritative source (i.e. Sygma).
-
-
-```json
-
-"validationDate": "2021-0101"
-
-```
-
-
-### validatedByFunder
-
-_Type: Boolean • Cardinality: ONE_
-
-
-Specifies if the validation comes from the funder.
-
-
-```json
-
-
-"validatedByFunder": true
-
-```

docs/downloads/beginners-kit.md

@@ -1,6 +0,0 @@
----
-sidebar_position: 2
----
-
-# Beginners kit
-

docs/downloads/full-graph.md

@@ -1,48 +0,0 @@
----
-sidebar_position: 1
----
-
-# Full graph dump
-
-You can download the full OpenAIRE Research Graph Dump as well as its schema from the following links: 
-
- Dataset: https://doi.org/10.5281/zenodo.3516917
-    
- Schema: https://doi.org/10.5281/zenodo.4238938
-
-The schema used to dump this dataset mirrors  the one described in the [Data Model](../data-model).
-This dataset is licensed under a Creative Commons Attribution 4.0 International License.
-It is composed of several files so that you can download the parts you are interested into. The files are named after the entity they store (i.e. publication, dataset). Each file is at most 10GB and it is 
-a tar archive containing gz files, each with one json per line.
-
-## How to acknowledge this work
-
-Open Science services are open and transparent and survive thanks to your active support and to the visibility and reward they gather. If you use one of the [OpenAIRE Research Graph dumps](https://doi.org/10.5281/zenodo.3516917) for your research, please provide a proper citation following the recommendation that you find on the dump's Zenodo page or as provided below. 
-
-:::note How to cite
-
-Manghi P., Atzori C., Bardi A., Baglioni M., Schirrwagen J., Dimitropoulos H., La Bruzzo S., Foufoulas I., Mannocci A., Horst M., Czerniak A., Kiatropoulou K., Kokogiannaki A., De Bonis M., Artini M., Ottonello E., Lempesis A., Ioannidis A., Manola N., Principe P. (2022). "OpenAIRE Research Graph Dump", *Dataset*, Zenodo. [doi:10.5281/zenodo.3516917](https://doi.org/10.5281/zenodo.3516917) ([BibTex](/bibtex/OpenAIRE_Research_Graph_dump.bib))
-:::
-
-Please also consider citing [other relevant research products](/publications#relevant-research-products) that can be of interest.
-
-Also consider adding one of the following badges to your service with the appropriate link to [our website](https://graph.openaire.eu); click on the badges below to download the respective badge image files.
-
-
-<div className="row">
-    <div className="col col--4 left-badge">
-        <a target="_blank" href={require('../assets/badges/openaire-badge-1.zip').default} download>
-            <img loading="lazy" alt="Openaire badge" src={require('../assets/badges/openaire-badge-1.png').default} className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module pagination-nav__link" style={{ paddingTop: '1.2em', paddingBottom: '1.2em'}} title="Click to download"/>
-        </a>  
-    </div>
-    <div className="col col--4 mid-badge">
-        <a target="_blank" href={require('../assets/badges/openaire-badge-2.zip').default} download>
-            <img loading="lazy" alt="Openaire badge" src={require('../assets/badges/openaire-badge-2.png').default} className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module pagination-nav__link  dark-badge" style={{ paddingTop: '1.2em', paddingBottom: '1.2em'}} title="Click to download"/>
-        </a>  
-    </div>
-    <div className="col col--4 right-badge">
-        <a target="_blank" href={require('../assets/badges/openaire-badge-3.zip').default} download>
-            <img loading="lazy" alt="Openaire badge" src={require('../assets/badges/openaire-badge-3.png').default} className="img_node_modules-@docusaurus-theme-classic-lib-theme-MDXComponents-Img-styles-module pagination-nav__link" style={{ paddingTop: '1.2em', paddingBottom: '1.2em'}} title="Click to download"/>
-        </a>
-    </div>
-</div>

docs/downloads/related-datasets.md

@@ -1,30 +0,0 @@
----
-sidebar_position: 4
----
-
-# Other related datasets
-
-In this page, we list other related datasets; please refer to their respective schema definitions for the data model they follow.
-
-## The dump of ScholeXplorer
-
- Dataset: https://doi.org/10.5281/zenodo.6338616
- 
- Schema (Scholix version 3): https://doi.org/10.5281/zenodo.1120275
- 
- Schema (Scholix version 4): https://doi.org/10.5281/zenodo.6351557
-
-This dataset is licensed under a CC0 1.0 Universal (CC0 1.0) Public Domain Dedication.
-The dataset contains the GZ-compressed dump of the Scholix links exposed by the OpenAIRE ScholeXplorer service.
-
-## The OpenAIRE LOD dump
-
-Dataset (RDF dump): https://doi.org/10.5281/zenodo.609943
-
-LOD Ontology: http://lod.openaire.eu/vocab
-
-SPARQL Endpoint: http://lod.openaire.eu/sparql
-
-
-The OpenAIRE Linked Open Data (LOD) Services and their integration with the OpenAIRE information space have been released as a beta version. The LOD exporting process started with a specification of the OpenAIRE data model as an RDF vocabulary, and then mapping of the OpenAIRE data to the graph-based RDF data model. To interlink the OpenAIRE data with related data on the Web, we have identified a list of potential datasets to interlinked with, including the DBpedia dataset extracted from Wikipedia and the publication databases DBLP and CiteSeer.
-Please refer [here](http://lod.openaire.eu/documentation) for more details on the LOD documentation.

docs/downloads/subgraphs.md

@@ -1,68 +0,0 @@
----
-sidebar_position: 3
----
-
-# Sub-graph dumps
-
-In order to facilitate users, different dumps are available under the Zenodo community called [OpenAIRE Research Graph](https://zenodo.org/communities/openaire-research-graph).
-This page lists all alternative dumps currently available.
-
-
-## The OpenAIRE COVID-19 dump
-
- Dataset: https://doi.org/10.5281/zenodo.6638745
-
- Schema: https://doi.org/10.5281/zenodo.6372977
-    
- This dataset is licensed under a Creative Commons Attribution 4.0 International License.
-    It contains metadata records of publications, research data, software and projects on the topic of Corona Virus and COVID-19. 
-This dump is part of the activities of OpenAIRE to support the fight against COVID-19 together with the OpenAIRE COVID-19 Gateway. 
-The dump consists of a tar archive containing gzip files with one json per line. Please refer [here](#alternative-sub-graph-data-model) for details on the data model of this dump.
-
-## The dump of funded products
-
- Dataset: https://doi.org/10.5281/zenodo.6634431
-
- Schema: https://doi.org/10.5281/zenodo.6372977
-
- This dataset is licensed under a Creative Commons Attribution 4.0 International License.
-It contains metadata records of research products (research literature, data, software, other types of research products) with funding 
-information available in the OpenAIRE Research Graph. Records are grouped by funder in a dedicated archive file. Each tar archive contains 
-gzip files, each with one json record per line. The model of this dump differs from the one of the whole graph.
-Please refer [here](#alternative-sub-graph-data-model) for details on the data model of this dump.
-
-## The dump of delta projects
-
- Dataset: https://doi.org/10.5281/zenodo.7119633
-
- Schema: https://doi.org/10.5281/zenodo.4238938
-  
- This dataset is licensed under a Creative Commons Attribution 4.0 International License.
-  It contains the metadata records of projects collected by OpenAIRE in a given time frame. Usually one deposition of collected projects is done for each release of the OpenAIRE Research Graph
- The deposition is one tar archive containing gzip files, each with one json record per line.
-
-## The dumps about research communities, initiatives and infrastructures
-
- Dataset: https://doi.org/10.5281/zenodo.6638478
-
- Schema: https://doi.org/10.5281/zenodo.6372977
-
- This dataset is licensed under a Creative Commons Attribution 4.0 International License.
-The dataset contains one file per community/initiative/infrastructure collaborating with OpenAIRE. Check out also their community gateways on 
- CONNECT. Each file is a tar archive containing gzip files with one json per line. The only communities/research initiative/infrastructure we dump are those visible to everyone.
- The model of this dump differs from the one of the whole graph.
-Please refer [here](#alternative-sub-graph-data-model) for details on the data model of this dump.
-
- --- 
-
- ## Alternative sub-graph data model
-
- It should be noted that the dumps for research communities, infrastructures, and products related to projects do not strictly follow the main data model of the OpenAIRE Research Graph. In particular, they differ in the following:
-
- * only research products are dumped (no relations, and entities different from results)
- * the dumped results are extended with information that can be inferred in the whole dump namely:
-   * funding information if present
-   * associated research community/infrastructure 
-   * associated data sources 
-
-So they have just one entity type, that is the [Extended Result](alternative-model/extendedresult.md).

docs/learning-center/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Learning center",
+  "position": 9,
+  "link": {
+    "type": "generated-index",
+    "description": "5 minutes to learn the most important Docusaurus concepts."
+  }
+}

docs/learning-center/open-plato.md

@@ -0,0 +1,7 @@
+---
+sidebar_position: 1
+---
+
+# OpenPlato Webinars
+<span className="todo">TODO</span>
+

docs/learning-center/tutorials.md

@@ -0,0 +1,7 @@
+---
+sidebar_position: 2
+---
+
+# Tutorials
+<span className="todo">TODO</span>
+

docs/license.md

@@ -3,6 +3,4 @@ sidebar_position: 11
 ---
 
 # License
-
-OpenAIRE Research Graph is available for download and re-use as CC-BY (due to some input sources whose license is CC-BY). Parts of the graphs can be re-used as CC-0.
-
+<span className="todo">TODO</span>

docs/publications.md

@@ -2,79 +2,56 @@
 sidebar_position: 7
 ---
 
-# Relevant publications
+# How to cite
 
-Open Science services are open and transparent and survive thanks to your active support and to the visibility and reward they gather. If you use one of the [OpenAIRE Research Graph dumps](https://doi.org/10.5281/zenodo.3516917) for your research, please provide a proper citation following the recommendation that you find on the dump's Zenodo page or as provided below. 
+If you use one of the [OpenAIRE Research Graph dumps](https://zenodo.org/record/6616871), please cite it following the recommendation that you find on the Zenodo page.
 
-:::note How to cite
-
-Manghi P., Atzori C., Bardi A., Baglioni M., Schirrwagen J., Dimitropoulos H., La Bruzzo S., Foufoulas I., Mannocci A., Horst M., Czerniak A., Kiatropoulou K., Kokogiannaki A., De Bonis M., Artini M., Ottonello E., Lempesis A., Ioannidis A., Manola N., Principe P. (2022). "OpenAIRE Research Graph Dump", *Dataset*, Zenodo. [doi:10.5281/zenodo.3516917](https://doi.org/10.5281/zenodo.3516917) ([BibTex](/bibtex/OpenAIRE_Research_Graph_dump.bib))
-:::
-
-## Other relevant research products
-
-Please also consider citing the related research products listed below.
+## Other relevant publications
 
 ### Aggregation system
+Manghi, P., Artini, M., Atzori, C., Bardi, A., Mannocci, A., La Bruzzo, S., Candela, L., Castelli, D. and Pagano, P. (2014), “The D-NET software toolkit: A framework for the realization, maintenance, and operation of aggregative infrastructures”, Program: electronic library and information systems, Vol. 48 No. 4, pp. 322-354. 
+Michele Artini, Claudio Atzori, Alessia Bardi, Sandro La Bruzzo, Paolo Manghi, & Andrea Mannocci. (2016, November 24). The D-NET software toolkit: dnet-basic-aggregator (Version 1.3.0). Zenodo. <i className="fa-solid fa-arrow-up-right-from-square"></i>
 
-Manghi P., Artini M., Atzori C., Bardi A., Mannocci A., La Bruzzo S., Candela L., Castelli D., Pagano P. (2014). "The D-NET software toolkit: A framework for the realization, maintenance, and operation of aggregative infrastructures", Program: electronic library and information systems, Vol. 48 No. 4, pp. 322-354. [doi:10.1108/prog-08-2013-0045](http://doi.org/10.1108/prog-08-2013-0045)
+Atzori, C., Bardi, A., Manghi, P., & Mannocci, A. (2017, January). The OpenAIRE workflows for data management. In Italian Research Conference on Digital Libraries (pp. 95-107). Springer, Cham.
 
-Atzori C., Bardi A., Manghi P., Mannocci A. (2017). "The OpenAIRE workflows for data management", In Italian Research Conference on Digital Libraries (IRCDL), pp. 95-107, Springer, Cham. [doi:10.1007/978-3-319-68130-6_8](https://doi.org/10.1007/978-3-319-68130-6_8)
-
-Artini M., Atzori C., Bardi A., La Bruzzo S., Manghi P., Mannocci A. (2016). "The D-NET software toolkit: dnet-basic-aggregator (Version 1.3.0)". *Software*, Zenodo. [doi:10.5281/zenodo.168356](https://doi.org/10.5281/zenodo.168356) <i className="fa-solid fa-arrow-up-right-from-square"></i>
-
-Mannocci A., Manghi P. (2016). "DataQ: a data flow quality monitoring system for aggregative data infrastructures", International Conference on Theory and Practice of Digital Libraries (TPDL), pp. 357-369, Springer, Cham. [doi:10.1007/978-3-319-43997-6_28](https://doi.org/10.1007/978-3-319-43997-6_28)
+Mannocci, A., & Manghi, P. (2016, September). DataQ: a data flow quality monitoring system for aggregative data infrastructures. In International Conference on Theory and Practice of Digital Libraries (pp. 357-369). Springer, Cham.
 
 ### Deduplication
+Claudio Atzori, & Paolo Manghi. (2017, February 17). gdup: a big graph entity deduplication system (Version 4.0.5). Zenodo.  https://code-repo.d4science.org/D-Net/dnet-dedup/releases 
 
-Vichos K., De Bonis M., Kanellos I., Chatzopoulos S., Atzori C., Manola N., Manghi P., Vergoulis T. (2022). "A preliminary assessment of the article deduplication algorithm used for the OpenAIRE Research Graph", In Italian Research Conference on Digital Libraries (IRCDL), Padua, Italy, CEUR-WS Proceedings. [http://ceur-ws.org/Vol-3160](http://ceur-ws.org/Vol-3160/) 
+Manghi, Paolo, Marko Mikulicic, and Claudio Atzori. "De-duplication of aggregation authority files." International Journal of Metadata, Semantics and Ontologies 7.2 (2012): 114-130. 
 
-De Bonis M., Manghi P., Atzori C. (2022). "FDup: a framework for general-purpose and efficient entity deduplication of record collections", PeerJ Computer Science, 8, e1058. [https://peerj.com/articles/cs-1058](https://peerj.com/articles/cs-1058)
+Manghi, P., Atzori, C., De Bonis, M., & Bardi, A. (2020). Entity deduplication in big data graphs for scholarly communication. Data Technologies and Applications. 
+Manghi, P., & Mikulicic, M. (2011, October). PACE: A general-purpose tool for authority control. In Research Conference on Metadata and Semantic Research (pp. 80-92). Springer, Berlin, Heidelberg. 
 
-Manghi P., Atzori C., De Bonis M., Bardi, A. (2020). "Entity deduplication in big data graphs for scholarly communication", Data Technologies and Applications. [doi:10.1108/dta-09-2019-0163](https://doi.org/10.1108/dta-09-2019-0163)
-
-
-Atzori C., Manghi P., Bardi, A. (2018). "GDup: de-duplication of scholarly communication big graphs", In 2018 IEEE/ACM 5th International Conference on Big Data Computing Applications and Technologies (BDCAT) (pp. 142-151). IEEE. [doi:10.1109/bdcat.2018.00025](https://doi.org/10.1109/bdcat.2018.00025)
-
-Atzori C., & Paolo Manghi. (2017). "GDup: a big graph entity deduplication system" (Version 4.0.5), *Software*, Zenodo. [doi:/10.5281/zenodo.292980](https://doi.org/10.5281/zenodo.292980)
-
-Atzori C. (2016). "GDup: an Integrated, Scalable Big Graph Deduplication System.". [doi:10.5281/zenodo.1454879](https://doi.org/10.5281/zenodo.1454879)
-
-Manghi P., Mikulicic M., Atzori C. (2012). "De-duplication of aggregation authority files." International Journal of Metadata, Semantics and Ontologies 7.2: 114-130. [doi:10.1504/ijmso.2012.050014](https://doi.org/10.1504/ijmso.2012.050014)
-
-Manghi P., Mikulicic M. (2011). "PACE: A general-purpose tool for authority control", In Research Conference on Metadata and Semantic Research, pp. 80-92, Springer, Berlin, Heidelberg. [doi:10.1007/978-3-642-24731-6_8](https://doi.org/10.1007/978-3-642-24731-6_8)
+Atzori, C., Manghi, P., & Bardi, A. (2018, December). GDup: de-duplication of scholarly communication big graphs. In 2018 IEEE/ACM 5th International Conference on Big Data Computing Applications and Technologies (BDCAT) (pp. 142-151). IEEE. 
+Atzori, Claudio. "GDup: an Integrated, Scalable Big Graph Deduplication System." (2016). 
 
 ### Mining
 
-Giannakopoulos T., Foufoulas Y., Dimitropoulos H., Manola N. (2019). "Interactive Text Analysis and Information Extraction", In Italian Research Conference on Digital Libraries (IRCDL), vol 988. Springer, Cham. [doi:10.1007/978-3-030-11226-4_27](https://doi.org/10.1007/978-3-030-11226-4_27)
+M. Kobos, Ł. Bolikowski, M. Horst, P. Manghi, N. Manola, J. Schirrwagen, “Information inference in scholarly communication infrastructures: the OpenAIREplus project experience”, Procedia Computer Science 38, 92-99. 
 
-Foufoulas Y., Stamatogiannakis L., Dimitropoulos H., Ioannidis Y. (2017). "High-Pass Text Filtering for Citation Matching", In International Conference on Theory and Practice of Digital Libraries (TPDL). Springer, Cham. [doi:10.1007/978-3-319-67008-9_28](https://doi.org/10.1007/978-3-319-67008-9_28)
+Tkaczyk, D., Szostek, P., Fedoryszak, M. et al. CERMINE: automatic extraction of structured metadata from scientific literature. IJDAR 18, 317–335 (2015). 
+Giannakopoulos T., Foufoulas Y., Dimitropoulos H., Manola N. (2019) “Interactive Text Analysis and Information Extraction”. In: Manghi P., Candela L., Silvello G. (eds) Digital Libraries: Supporting Open Science. IRCDL 2019. Communications in Computer and Information Science, vol 988. Springer, Cham. 
 
-Chronis Y., Foufoulas Y., Nikolopoulos V., Papadopoulos A., Stamatogiannakis L., Svingos C., Ioannidis Y. E. (2016). "A Relational Approach to Complex Dataflows", In Workshop Proceedings of the EDBT/ICDT 2016 (MEDAL 2016) Joint Conference on CEUR-WS.org (ISSN 1613-0073) [http://ceur-ws.org/Vol-1558/paper45.pdf](http://ceur-ws.org/Vol-1558/paper45.pdf)
+Foufoulas Y., Stamatogiannakis L., Dimitropoulos H., Ioannidis Y. (2017) “High-Pass Text Filtering for Citation Matching”. In: Kamps J., Tsakonas G., Manolopoulos Y., Iliadis L., Karydis I. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2017. Lecture Notes in Computer Science, vol 10450. Springer, Cham. 
 
-Giannakopoulos T., Foufoulas I., Stamatogiannakis E., Dimitropoulos H., Manola N., Ioannidis Y. (2015). "Visual-Based Classification of Figures from Scientific Literature", In Proceedings of the 24th International Conference on World Wide Web (WWW), Association for Computing Machinery, New York, NY, USA, 1059–1060. [doi:10.1145/2740908.2742024](https://doi.org/10.1145/2740908.2742024)
+T. Giannakopoulos, I. Foufoulas, E. Stamatogiannakis, H. Dimitropoulos, N. Manola, and Y. Ioannidis. 2015. “Visual-Based Classification of Figures from Scientific Literature”. In Proceedings of the 24th International Conference on World Wide Web (WWW '15 Companion). Association for Computing Machinery, New York, NY, USA, 1059–1060. 
 
-Giannakopoulos T., Foufoulas I., Stamatogiannakis E., Dimitropoulos H., Manola N., Ioannidis Y. (2014). "Discovering and Visualizing Interdisciplinary Content Classes in Scientific Publications". D-Lib Mag., Volume 20, Number 11/12. [doi:10.1045/november14-giannakopoulos](https://doi.org/10.1045/november14-giannakopoulos)
+Giannakopoulos, T., Foufoulas, I., Stamatogiannakis, E., Dimitropoulos, H., Manola, N., & Ioannidis, Y. (2014). “Discovering and Visualizing Interdisciplinary Content Classes in Scientific Publications”. D-Lib Mag., Volume 20, Number 11/12. 
 
-Giannakopoulos T., Stamatogiannakis E., Foufoulas I., Dimitropoulos H., Manola N., Ioannidis Y. (2014). "Content Visualization of Scientific Corpora Using an Extensible Relational Database Implementation", International Conference on Theory and Practice of Digital Libraries (TPDL), Springer, Cham. [doi:10.1007/978-3-319-08425-1_10](https://doi.org/10.1007/978-3-319-08425-1_10) 
+Giannakopoulos T., Stamatogiannakis E., Foufoulas I., Dimitropoulos H., Manola N., Ioannidis Y. (2014) “Content Visualization of Scientific Corpora Using an Extensible Relational Database Implementation”. In: Bolikowski Ł., Casarosa V., Goodale P., Houssos N., Manghi P., Schirrwagen J. (eds) Theory and Practice of Digital Libraries -- TPDL 2013 Selected Workshops. TPDL 2013. Communications in Computer and Information Science, vol 416. Springer, Cham.  Also in: Google Books 
 
-Giannakopoulos T., Dimitropoulos H., Metaxas O., Manola N., Ioannidis Y. (2013). "Supervised Content Visualization of Scientific Publications: A Case Study on the ArXiv Dataset", Intelligent Information Systems Symposium (IIS) vol 7912, Springer, Berlin, Heidelberg. [doi:10.1007/978-3-642-38634-3_23](https://doi.org/10.1007/978-3-642-38634-3_23)
+Giannakopoulos T., Dimitropoulos H., Metaxas O., Manola N., Ioannidis Y. (2013) “Supervised Content Visualization of Scientific Publications: A Case Study on the ArXiv Dataset”. In: Kłopotek M.A., Koronacki J., Marciniak M., Mykowiecka A., Wierzchoń S.T. (eds) Language Processing and Intelligent Information Systems. IIS 2013. Lecture Notes in Computer Science, vol 7912. Springer, Berlin, Heidelberg.
  
-Tkaczyk, D., Szostek, P., Fedoryszak, M., Jan Dendek P., Bolikowski Ł. (2015). "CERMINE: automatic extraction of structured metadata from scientific literature", International Journal on Document Analysis and Recognition (IJDAR), 317–335. [doi:10.1007/s10032-015-0249-8](https://doi.org/10.1007/s10032-015-0249-8)
-
-Kobos M., Bolikowski Ł., Horst M., Manghi P., Μanola N., Schirrwagen J. (2014). "Information inference in scholarly communication infrastructures: the OpenAIREplus project experience", Procedia Computer Science 38, 92-99. [doi:10.1016/j.procs.2014.10.016](https://doi.org/10.1016/j.procs.2014.10.016)
+Y. Chronis, Y. Foufoulas, V. Nikolopoulos, A. Papadopoulos, L. Stamatogiannakis, C. Svingos, Y. E. Ioannidis, "A Relational Approach to Complex Dataflows", in Workshop Proceedings of the EDBT/ICDT 2016 (MEDAL 2016) Joint Conference (March 15, 2016, Bordeaux, France) on CEUR-WS.org (ISSN 1613-0073) 
 
 ### Portals
-
-Baglioni Μ., Bardi Α., Kokogiannaki Α., Manghi P., Iatropoulou K., Principe P., Vieira A., Nielsen L. H., Dimitropoulos H., Foufoulas I., Manola N., Atzori C., La Bruzzo S., Lazzeri E., Artini M., De Bonis M., Dell’Amico A. (2019). "The OpenAIRE Research Community Dashboard: On Blending Scientific Workflows and Scientific Publishing",
-International Conference on Theory and Practice of Digital Libraries (TPDL). Lecture Notes in Computer Science, vol 11799. Springer, Cham. [doi:10.1007/978-3-030-30760-8_5](https://doi.org/10.1007/978-3-030-30760-8_5)
+Baglioni M. et al. (2019) The OpenAIRE Research Community Dashboard: On Blending Scientific Workflows and Scientific Publishing. In: Doucet A., Isaac A., Golub K., Aalberg T., Jatowt A. (eds) Digital Libraries for Open Knowledge. TPDL 2019. Lecture Notes in Computer Science, vol 11799. Springer, Cham. 
 
 ### Broker Service
+Artini, M., Atzori, C., Bardi, A., La Bruzzo, S., Manghi, P., & Mannocci, A. (2015). The OpenAIRE literature broker service for institutional repositories. D-Lib Magazine, 21(11/12), 1.
 
-Manghi P., Atzori C., Bardi A., La Bruzzo S., Artini M. (2016). "Realizing a Scalable and History-Aware Literature Broker Service for OpenAIRE", Italian Research Conference on Digital Libraries (IRCDL), pp. 92-103, Springer, Cham. [doi:10.1007/978-3-319-56300-8_9](https://doi.org/10.1007/978-3-319-56300-8_9)
-
-Artini M., Atzori C., Bardi A., La Bruzzo S., Manghi P., Mannocci A. (2015). "The OpenAIRE literature broker service for institutional repositories", D-Lib Magazine, 21(11/12), 1. [doi:10.1045/november2015-artini](https://doi.org/10.1045/november2015-artini)
-
-
+Manghi, P., Atzori, C., Bardi, A., La Bruzzo, S., & Artini, M. (2016, February). Realizing a Scalable and History-Aware Literature Broker Service for OpenAIRE. In Italian Research Conference on Digital Libraries (pp. 92-103). Springer, Cham.
 
 

docs/services.md

@@ -0,0 +1,20 @@
+---
+sidebar_position: 8
+---
+
+# Graph-based services
+
+## Explore
+<span className="todo">TODO</span>
+
+## Provide
+<span className="todo">TODO</span>
+
+## Connect
+<span className="todo">TODO</span>
+
+## Monitor
+<span className="todo">TODO</span>
+
+## Develop
+<span className="todo">TODO</span>

docusaurus.config.js

@@ -5,23 +5,14 @@ const lightCodeTheme = require('prism-react-renderer/themes/github');
 const darkCodeTheme = require('prism-react-renderer/themes/dracula');
 const math = require('remark-math');
 const katex = require('rehype-katex');
-const dotenv = require('dotenv');
-
-// load env variables (see .env file)
-const env = dotenv.config();
-if (env.error) {
-  throw env.error;
-}
-
-console.info("ENV VARIABLES:");
-console.info(env.parsed);
+const { filterItems } = require('./sidebar-utils');
 
 /** @type {import('@docusaurus/types').Config} */
 const config = {
-  title: 'OpenAIRE Research Graph Documentation',
+  title: 'OpenAIRE Documentation',
   tagline: 'Open Access Infrastructure for Research in Europe',
-  url: process.env.URL,
-  baseUrl: process.env.BASE_URL, // serve the website at route
+  url: 'http://snf-23385.ok-kno.grnetcloud.net',
+  baseUrl: '/', // serve the website at route
   onBrokenLinks: 'throw',
   onBrokenMarkdownLinks: 'warn',
   favicon: 'img/favicon.ico',
@@ -38,19 +29,7 @@ const config = {
     defaultLocale: 'en',
     locales: ['en'],
   },
-  themes: [
-    [
-      require.resolve("@easyops-cn/docusaurus-search-local"),
-      /** @type {import("@easyops-cn/docusaurus-search-local").PluginOptions} */
-      ({
-        language: ["en"],
-        indexBlog: false,
-        highlightSearchTermsOnTargetPage: true,
-        searchBarShortcutHint: false,
-        docsRouteBasePath: "/",
-      }),
-    ],
-  ],
+
   presets: [
     [
       'classic',
@@ -58,7 +37,18 @@ const config = {
       ({
         docs: {
           routeBasePath: '/', // serve the docs at the site's route
+
           sidebarPath: require.resolve('./sidebars.js'),
+          async sidebarItemsGenerator({ defaultSidebarItemsGenerator, ...args }) {
+            const sidebarItems = await defaultSidebarItemsGenerator(args); 
+
+            const itemsToFilterOut = [ 
+              'data-model/entities/entity-identifiers', 
+              'data-model/entities/other'
+            ];
+
+            return filterItems(sidebarItems, itemsToFilterOut);
+          },
           // Please change this to your repo.
           // Remove this to remove the "edit this page" links.
           // editUrl:
@@ -73,12 +63,6 @@ const config = {
         // },
         theme: {
           customCss: require.resolve('./src/css/custom.css'),
-        },
-	      sitemap: {
-          changefreq: 'monthly',
-          priority: 0.5,
-          ignorePatterns: ['/tags/**'],
-          filename: 'sitemap.xml',
         },
       }),
     ],
@@ -97,45 +81,98 @@ const config = {
     /** @type {import('@docusaurus/preset-classic').ThemeConfig} */
     ({
       navbar: {
-        title: 'documentation',
+        // title: 'OpenAIRE Documentation',
         logo: {
           alt: 'OpenAIRE',
           src: 'img/logo.png',
         },
         items: [
-          // {
-          //   type: 'doc',
-          //   docId: 'intro',
-          //   position: 'left',
-          //   label: 'Research graph v5.0',
-          // },
+          {
+            type: 'doc',
+            docId: 'intro',
+            position: 'left',
+            label: 'Research graph v5.0',
+          },
           // 
           // documentation version in the navbar
-          {
-            type: 'docsVersionDropdown', 
-            position: 'right'
-          },
-          
+          // {
+          //   type: 'docsVersionDropdown', 
+          //   position: 'right'
+          // },
+          // 
           // link to blog, the blog must be enabled first
           // {to: '/blog', label: 'Blog', position: 'left'},
-          
+          // 
           // link to github repo
           // {
           //   href: 'https://github.com/facebook/docusaurus',
-          //   label: 'Issues',
+          //   label: 'GitHub',
           //   position: 'right',
           // },
         ],
       },
       footer: {
-        style: 'light',
+        style: 'dark',
+        links: [
+          {
+            title: 'Docs',
+            items: [
+              {
+                label: 'Research Graph',
+                to: '/',
+              },
+            ],
+          },
+          {
+            title: 'Dashboards',
+            items: [
+              {
+                label: 'Explore',
+                href: 'https://explore.openaire.eu/',
+              },
+              {
+                label: 'Provide',
+                href: 'https://provide.openaire.eu/',
+              },
+              {
+                label: 'Connect',
+                href: 'https://connect.openaire.eu/',
+              },
+              {
+                label: 'Monitor',
+                href: 'https://monitor.openaire.eu/',
+              },
+              {
+                label: 'Develop',
+                href: 'https://graph.openaire.eu/',
+              },
+            ],
+          },
+          {
+            title: 'Community',
+            items: [
+              { 
+                label: 'Facebook',
+                href: 'http://www.facebook.com/groups/openaire/'
+              },
+              {
+                label: 'Linkedin',
+                href: 'https://www.linkedin.com/company/openaire-eu/',
+              },
+              {
+                label: 'Twitter',
+                href: 'https://twitter.com/OpenAIRE_eu',
+              },
+              {
+                label: 'Youtube',
+                href: 'https://www.youtube.com/channel/UChFYqizc-S6asNjQSoWuwjw',
+              },
+            ],
+          },
+          
+        ],
         copyright: `Copyright © ${new Date().getFullYear()} OpenAIRE`,
       },
-      colorMode: {
-        defaultMode: 'light',
-        disableSwitch: true,
-        respectPrefersColorScheme: false,
-      },
       prism: {
         theme: lightCodeTheme,
         darkTheme: darkCodeTheme,

package.json

@@ -4,22 +4,20 @@
   "private": true,
   "scripts": {
     "docusaurus": "docusaurus",
-    "start": "docusaurus start --host 0.0.0.0",
+    "start": "docusaurus start",
     "build": "docusaurus build",
     "swizzle": "docusaurus swizzle",
     "deploy": "docusaurus deploy",
     "clear": "docusaurus clear",
-    "serve": "docusaurus serve --host 0.0.0.0",
+    "serve": "docusaurus serve",
     "write-translations": "docusaurus write-translations",
     "write-heading-ids": "docusaurus write-heading-ids"
   },
   "dependencies": {
-    "@docusaurus/core": "^2.2.0",
-    "@docusaurus/preset-classic": "^2.2.0",
-    "@easyops-cn/docusaurus-search-local": "^0.33.6",
+    "@docusaurus/core": "2.0.1",
+    "@docusaurus/preset-classic": "2.0.1",
     "@mdx-js/react": "^1.6.22",
     "clsx": "^1.2.1",
-    "dotenv": "^16.0.3",
     "hast-util-is-element": "^1.1.0",
     "prism-react-renderer": "^1.3.5",
     "react": "^17.0.2",
@@ -28,7 +26,7 @@
     "remark-math": "^3.0.1"
   },
   "devDependencies": {
-    "@docusaurus/module-type-aliases": "^2.1.0"
+    "@docusaurus/module-type-aliases": "2.0.1"
   },
   "browserslist": {
     "production": [

release.properties

@@ -1,8 +0,0 @@
-#The name of the tag
-tag_name=1.1 
-# A description of the tag
-tag_description=1.1 is our 1st tag
-#The release name
-release_name=release-1.1
-#The release description
-release_description=this is the release 1.1

sidebar-utils.js

@@ -0,0 +1,18 @@
+// filter out specific items from the sidebar
+function filterItems(items, itemsToFilter) {
+
+    // filter out items of categories
+    let result = items.map((item) => {
+      if (item.type === 'category') {
+        return {...item, items: filterItems(item.items, itemsToFilter)};
+      }
+      return item;
+    });
+  
+    // filter out items in current level
+    return result.filter( item => !itemsToFilter.includes(item.id) );
+}
+
+module.exports = {
+    filterItems
+};

sidebars.js

@@ -51,22 +51,12 @@ const sidebars = {
       href: "https://graph.openaire.eu/develop/overview.html"
     },
     {
-      type: 'category', 
-      label: "Downloads",
-      link: {
-        type: 'generated-index',
-        description: 'All resources, available for download, are listed below.'
-      },
-      items: [
-        { type: 'doc', id: 'downloads/full-graph'},
-        { type: 'doc', id: 'downloads/beginners-kit' },
-        { type: 'doc', id: 'downloads/subgraphs' },
-        { type: 'doc', id: 'downloads/related-datasets' },
-      ]
-    }, 
+      type: 'doc', 
+      id: 'download'
+    },
     {
       type: 'category', 
-      label: "Graph production workflow",
+      label: "Data provision",
       link: {type: 'doc', id: 'data-provision/data-provision'},
       items: [
         {
@@ -74,46 +64,12 @@ const sidebars = {
           label: "Aggregation",
           link: {type: 'doc', id: 'data-provision/aggregation/aggregation'},
           items: [
-            {
-              type: 'doc',
-              label: "OpenAIRE compatible sources",
-              id: 'data-provision/aggregation/compatible-sources',
-            },
-            {
-            type: 'category',
-              label: "Non-compatible sources",
-              link: { type: 'generated-index' },
-              items: [
-                { type: 'doc', id: 'data-provision/aggregation/non-compatible-sources/doiboost', label: 'DOIBoost' },
-                { type: 'doc', id: 'data-provision/aggregation/non-compatible-sources/pubmed' },
-                { type: 'doc', id: 'data-provision/aggregation/non-compatible-sources/datacite' },
-                { type: 'doc', id: 'data-provision/aggregation/non-compatible-sources/ebi', label: 'EMBL-EBI' },
-              ]
-            }
+            { type: 'doc', id: 'data-provision/aggregation/doiboost', label: 'DOIBoost' },
+            { type: 'doc', id: 'data-provision/aggregation/pubmed' },
+            { type: 'doc', id: 'data-provision/aggregation/datacite' },
+            { type: 'doc', id: 'data-provision/aggregation/ebi', label: 'EMBL-EBI' },
           ]
         },
-        {
-          type: 'doc', 
-          id: 'data-provision/merge-by-id'
-        },
-        {
-          type: 'category', 
-          label: "Enrichment by mining",
-          link: {
-            type: 'generated-index',
-            description: 'The OpenAIRE Research Graph is enriched using the different Text and Data Mining (TDM) algorithms that are grouped in the following categories.'
-          },
-          items: [
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/affiliation_matching' },
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/citation_matching' },
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/classifies' },
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/documents_similarity' },
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/acks' },
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/cites' },
-              { type: 'doc', id: 'data-provision/enrichment-by-mining/metadata_extraction' },
-            ]
-        },
-        { type: 'doc', id: 'data-provision/cleaning' },
         {
           type: 'category', 
           label: "Deduplication",
@@ -124,49 +80,41 @@ const sidebars = {
           ]
         }, 
         {
-          type: 'category',
-          label: "Deduction & propagation",
-          link: { 
-            type: 'generated-index' ,
-            description: 'The OpenAIRE Research Graph is further enriched by the deduction and propagation processes descibed in this section.'
-
-          },
+          type: 'category', 
+          label: "Enrichment",
+          link: {type: 'doc', id: 'data-provision/enrichment/enrichment'},
           items: [
-            { type: 'doc', id: 'data-provision/deduction-and-propagation/bulk-tagging' },
-            { type: 'doc', id: 'data-provision/deduction-and-propagation/propagation' },
+            { type: 'doc', id: 'data-provision/enrichment/mining' },
+            { type: 'doc', id: 'data-provision/enrichment/impact-scores' },
           ]
         },
-        {
-          type: 'category',
-          label: "Indicators ingestion",
-          link: { 
-            type: 'generated-index' ,
-            description: 'In this step, the following types of indicators are ingested in the OpenAIRE Research Graph.'
-
-          },
-          items: [
-            { type: 'doc', id: 'data-provision/indicators-ingestion/impact-scores' },
-            { type: 'doc', id: 'data-provision/indicators-ingestion/usage-counts' },
-          ]
-        },
-        { type: 'doc', id: 'data-provision/finalisation' },
+        { type: 'doc', id: 'data-provision/post-cleaning' },
         { type: 'doc', id: 'data-provision/indexing' },
+        { type: 'doc', id: 'data-provision/stats' },
       ]
     },
     {
-      type: "link",
+      type: 'doc', 
+      id: 'services'
+    },
+    {
+      type: 'category', 
       label: "Learning center",
-      href: "https://openplato.eu/"
+      link: { type: 'generated-index' },
+      items: [
+        { type: 'doc', id: 'learning-center/open-plato' },
+        { type: 'doc', id: 'learning-center/tutorials' },
+      ]
     },
     {
       type: 'doc', 
       id: 'publications',
       label: "Relevant publications"
     },
-    // {
-    //   type: 'doc', 
-    //   id: 'faq'
-    // },
+    {
+      type: 'doc', 
+      id: 'faq'
+    },
     {
       type: 'doc', 
       id: 'license'

src/css/custom.css

@@ -5,66 +5,58 @@
  */
 
 /* You can override the default Infima variables here. */
+/*
+:root {
+  --ifm-color-primary: #2e8555;
+  --ifm-color-primary-dark: #29784c;
+  --ifm-color-primary-darker: #277148;
+  --ifm-color-primary-darkest: #205d3b;
+  --ifm-color-primary-light: #33925d;
+  --ifm-color-primary-lighter: #359962;
+  --ifm-color-primary-lightest: #3cad6e;
+  --ifm-code-font-size: 95%;
+  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
+}
+*/
+
+/* For readability concerns, you should choose a lighter palette in dark mode. */
+/*
+[data-theme='dark'] {
+  --ifm-color-primary: #25c2a0;
+  --ifm-color-primary-dark: #21af90;
+  --ifm-color-primary-darker: #1fa588;
+  --ifm-color-primary-darkest: #1a8870;
+  --ifm-color-primary-light: #29d5b0;
+  --ifm-color-primary-lighter: #32d8b4;
+  --ifm-color-primary-lightest: #4fddbf;
+  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
+}
+*/
 
 :root {
-  --ifm-color-primary: #e6122e;
-  --ifm-color-primary-dark: #cf1029;
-  --ifm-color-primary-darker: #c30f27;
-  --ifm-color-primary-darkest: #a10d20;
-  --ifm-color-primary-light: #ee233e;
-  --ifm-color-primary-lighter: #ef2f48;
-  --ifm-color-primary-lightest: #f15166;
-  --ifm-background-color: #F5F5F5;
-  --ifm-navbar-background-color: #fff;
+  --ifm-color-primary: #4666ca;
+  --ifm-color-primary-dark: #3757be;
+  --ifm-color-primary-darker: #3353b4;
+  --ifm-color-primary-darkest: #2a4494;
+  --ifm-color-primary-light: #5b77d0;
+  --ifm-color-primary-lighter: #6680d3;
+  --ifm-color-primary-lightest: #859adc;
   --ifm-code-font-size: 95%;
   --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
 }
 
 [data-theme='dark'] {
-  --ifm-color-primary: #f15166;
-  --ifm-color-primary-dark: #ef334c;
-  --ifm-color-primary-darker: #ed243f;
-  --ifm-color-primary-darkest: #d1112a;
-  --ifm-color-primary-light: #f36f80;
-  --ifm-color-primary-lighter: #f57e8d;
-  --ifm-color-primary-lightest: #f8aab5;
-  --ifm-background-color: #2c2e3a;
-  --ifm-navbar-background-color: #2c2e3a;
+  --ifm-color-primary: #5dade2;
+  --ifm-color-primary-dark: #429fdd;
+  --ifm-color-primary-darker: #3498db;
+  --ifm-color-primary-darkest: #227fbd;
+  --ifm-color-primary-light: #78bbe7;
+  --ifm-color-primary-lighter: #86c2e9;
+  --ifm-color-primary-lightest: #aed6f1;
   --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
 }
 
-.navbar__logo {
-    height: 2.5rem;
-}
 
 .todo {
   background-color: yellow;
-}
-
-@media (min-width: 996px) {
-
-  .left-badge {
-    padding-right: 5px;
-  }
-
-  .mid-badge {
-    padding-left: 0;
-    padding-right: 5px;
-  }
-
-  .right-badge {
-    padding-left: 0;
-  }
-}
-
-.dark-badge {
-  background-color: #c6c6c6;
-}
-
-.footer {
-  background-color: var(--ifm-navbar-background-color);
-  padding-bottom: 2em;
-  padding-top: 1em;
-  height: var(--ifm-navbar-height);
-}
-
+}

static/bibtex/OpenAIRE_Research_Graph_dump.bib

@@ -1,33 +0,0 @@
-@dataset{manghi_paolo_2022_6616871,
-  author       = {Manghi, Paolo and
-                  Atzori, Claudio and
-                  Bardi, Alessia and
-                  Baglioni, Miriam and
-                  Schirrwagen, Jochen and
-                  Dimitropoulos, Harry and
-                  La Bruzzo, Sandro and
-                  Foufoulas, Ioannis and
-                  Mannocci, Andrea and
-                  Horst, Marek and
-                  Czerniak, Andreas and
-                  Kiatropoulou, Katerina and
-                  Kokogiannaki, Argiro and
-                  De Bonis, Michele and
-                  Artini, Michele and
-                  Ottonello, Enrico and
-                  Lempesis, Antonis and
-                  Ioannidis, Alexandros and
-                  Manola, Natalia and
-                  Principe, Pedro},
-  title        = {OpenAIRE Research Graph Dump},
-  month        = Jun,
-  year         = 2022,
-  note         = {{A new version of this dataset is published every 6 
-                   months. The content available on the OpenAIRE
-                   EXPLORE and CONNECT portals might be more up-to-
-                   date with respect to the data you find here.}},
-  publisher    = {Zenodo},
-  version      = {4.1},
-  doi          = {10.5281/zenodo.6616871},
-  url          = {https://doi.org/10.5281/zenodo.6616871}
-}