promoting beta branch to master #134

claudio.atzori · 2021-08-09T10:08:42+02:00

claudio.atzori commented

2021-08-09 10:08:42 +02:00

Graph model version [dhp-schemas:2.7.15]

introduces updated labels to qualify the relationships. In earlier provision rounts though, the actionsets, the IIS outputs including them won't be updated for backward compatibility.
added support for delegated authorities (Zenodo).

Raw graph creation

added optional phase for patching the relation identifiers based on an association map on hdfs.

DoiBoost

improved AccessRight according to https://support.openaire.eu/issues/4362

Scholexplorer

data mapped using the same OAF based mapping layers, paving the road to a tighter integration in the OpenAIRE Graph

Cleaning workflow

improved record filtering, based on the library me.xuender:unidecode

Stats workflow

introduced indicators from sprint1 and sprint2

Graph model version [dhp-schemas:2.7.15] - introduces updated labels to qualify the relationships. In earlier provision rounts though, the actionsets, the IIS outputs including them won't be updated for backward compatibility. - added support for delegated authorities (Zenodo). Raw graph creation - added optional phase for patching the relation identifiers based on an association map on hdfs. DoiBoost - improved AccessRight according to https://support.openaire.eu/issues/4362 Scholexplorer - data mapped using the same OAF based mapping layers, paving the road to a tighter integration in the OpenAIRE Graph Cleaning workflow - improved record filtering, based on the library `me.xuender:unidecode` Stats workflow - introduced indicators from sprint1 and sprint2

claudio.atzori added the

enhancement

label 2021-08-09 10:08:42 +02:00

claudio.atzori added 83 commits 2021-08-09 10:08:43 +02:00

bc4b86c27c updated URL in the issueManagement tag

c35c117601 fixed process doiboost workflow:

- splitted OrcidToOAF into two phase preprocess and process
- updated workflow used in production

441701c85c DoiBoost AccessRigh #4362 - If multiple licenses are available, take the one applied to 'vor'

3d8e2aa146 Code refactor:

- removed old workflows in doiboost
 - splitted workflow of doiboost in preprocess and process

981b1018f6 DoiBoost AccessRigh #4362 - decide access right according to licence. Default access right is Unknown

6222adf176 DoiBoost AccessRigh #4362 - added resources and test for crossref mapping (licence part included)

f4f7c6f9d3 DoiBoost AccessRigh #4362 - Unpaywall mapped to OAF with OPEN instance (non oa are filtered out) (unknown hostedby) + map the color as it is

09ad7b2a9e DoiBoost AccessRigh #4362 - Unpaywall mapped to OAF with OPEN instance (non oa are filtered out) (unknown hostedby) + map the color as it is

4da46bb62f mergin with branch beta

7e2caafe84 Scholexplorer: fixed mapping typologies

bf9e0d2d4f Merge pull request 'orcid-no-doi' (#123 ) from enrico.ottonello/dnet-hadoop:orcid-no-doi into beta

Reviewed-on: #123

34506df1b6 DoiBoost AccessRigh #4362 - if the journal is open, the OPEN access right is set to all instances and color is GOLD (overwrite if the color was already set in one of the previous steps)

3bc9a05bc9 mergin with branch beta

199123b74b DoiBoost AccessRigh #4362 - Fixed issue on date formatting. Added test method and associated resource

59530a14fb DoiBoost AccessRigh #4362 - set BestAccessRight with the ususal comparator

662c396354 duplicate the number of partitions in ConvertCrossrefToOaf

0977baf41d contents mapped from the stores with 'claim' interpretation will not change their identifier along their way towards the graph

65934888a1 adding record identifier among the originalIds regardless of what IdentifierFactory produces

b420b11ed3 duplicate the number of partitions in ProcessMag

54acc5373b changed the name of the workflows

dd81c36b60 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

83fe31c92e changed the name of the workflows

10d7b4f0b4 filtering 'old' OpenAIRE ids from the entity.originalId[] array in the OAF -> XML searialization procedure

b226ba4439 mergin with branch beta

31d2d6d41e Scholexplorer: introduction of dedup openaire

d94565862a fixed NPE

1a5b114906 DoiBoost AccessRigh #4362 - refactoring

62ae36a3d2 fixed NPE

058b636d4d added control to check if the entity exists

43e9380cd3 update resolve relation to use the same format of openaire graph

ca74e8dd02 create a separate wf for resolving relation

bc835d2024 [cleaning] fixed filtering function for missing titles

4a439c3863 NPE fixed

cfde63a7c3 fixed resolve relation join

d9e3b89937 implemented last part of workflows to generate scholixGraph

a0393607a7 mapping funding relations from Datacite should be done according to the actual result identifier

f9fbb0f261 added indicators second sprint

3920c69bc8 change implementation of resolve Relation to generate jsonRdd in output

8fac10c91e fixed defintion wf of creation final infospace of scholexplorer

848aabbb6c minor fix

f3b9570354 properly invalidating metadata

ed185fd7ed added missing colons

1a28a69cac changed the citeee in *_citations to cites

5aa7d16d1b updated assertions in eu.dnetlib.dhp.oa.graph.raw.MappersTest

825d9f0289 fixed datacite workflow starting from Importing delta

6c936943aa mergin with branch beta

16c91203bd implemented workflow of creation action set for scholexplorer

43e62fcae9 DoiBoost AccessRigh #4362 - related to https://code-repo.d4science.org/D-Net/dnet-hadoop/pulls/126/files#issuecomment-4193

73ed7374a9 mergin with branch beta

5fe016dcbc DoiBoost AccessRigh #4362 - related to https://code-repo.d4science.org/D-Net/dnet-hadoop/pulls/126/files#issuecomment-4194

80d5b3b4de DoiBoost AccessRigh #4362 - removing commented code

4a9741825d added result_orcid, result_project provenance, issn in datasources

b346feed36 Merge pull request 'Change the access right in DoiBoost' (#126 ) from doiboosi_accessright into beta

Reviewed-on: #126

2fff24df55 code formatting

c806387d4b tests for enermaps

df8715a1ec format code after mvn compile

9b181ffa73 added the h2020 classification scheme for projects

3d8f0f629b implemented workflow of creation action set for scholexplorer

6dddad86ee [cleaning] title cleaning based on the me.xuender:unidecode library

e1797c0a42 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

4c5a71ba2f [broker] updated relation descriptors, making use of constant values

3d1580fa9b fixed a typo

4afa5215a9 fixed a NPE?

6aef3e8f46 Merge pull request '[broker] updated relation descriptors' (#127 ) from broker_relations_upgrade into beta

Reviewed-on: #127

3721df7aa6 refactoring create actionset of scholexplorer, moved on package dhp-aggregation

908f57a475 code formatting

dc55ed4acd Merge pull request '[beta] stats update workflow' (#128 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #128

e87e1805c4 [raw_all] added extra workflow step for patching the identifiers in the relations, given an id mapping dataset

5d08ad86ae [raw_all] patching relation identifier phase to be run at the end, i.e. includes also claimed relations

b1b0cc3f15 fixed wrong package name

c53d106e80 [provision] lowercase relation filter

7b172e7cd9 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

26af0320d0 added the sprint 2 indicators in monitor db

6358f92c3a added sleep to solve problem of lost request of creating index

55e6470f44 Merge pull request 'added the sprint 2 indicators in monitor db' (#129 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #129

19620eed46 applying PR#131, Patch the identifiers (source/target) in the relations, refinements

e244f73165 Update 'README.md'

117c3d5c67 fixed a typo

fd55c77d97 updated dependency dhp-schemas:2.7.15

e826aae848 using constants from ModelConstants

3fc820203b fixed wrong test file

83c04e5d28 mapping test for dataset records adapted to reflect the delegated pid authority (zenodo)

66f398fe6f Merge pull request '[stats] fixed a typo' (#133 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #133

claudio.atzori manually merged commit a45b95ccc1 into master

2021-08-09 10:51:03 +02:00

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#134