Dump Extention #151

miriam.baglioni · 2021-10-19T10:22:42+02:00

miriam.baglioni commented

2021-10-19 10:22:42 +02:00

This PR extends the dump to consider the new model. It provides also a unique workflow that can be parametrized to handle each dump type

miriam.baglioni added the

enhancement

label 2021-10-19 10:22:42 +02:00

claudio.atzori was assigned by miriam.baglioni

2021-10-19 10:22:42 +02:00

miriam.baglioni added 101 commits 2021-10-19 10:22:42 +02:00

e13926cdd0 merged with master

2e70aa43f0 Merge pull request 'H2020Classification fix and possibility to add datasources in blacklist for propagation of result to organization' (#108 ) from miriam.baglioni/dnet-hadoop:master into master

Reviewed-on: #108

The changes look ok, but please drop a comment to describe how the parameters should be changed from the workflow caller for both workflows
* H2020Classification
* propagation of result to organization

ea9b00ce56 adjusted test

4d6c473bf1 removed redundant classes contained now in dhp-schema

ae7bd24d79 removed old workflows

0821d8e97d Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi

1265dadc90 workflow aligned with stable_ids

d0945c3c78 added temporary output folder, because of folder access rights are different on beta and prod

abdd0ade1f added temporary output folder as workflow parameter

783988af06 depending on dhp-schemas:2.6.14 (release)

28a66af425 updated URL in the issueManagement tag

e0061232e9 [aggregation] datacite wf: conditional creation of links, optional resume from intermediate phases

f5486ffb14 Fixed issues to tests

2f66fedfec changed the logic to verify if a community is contained in the list of context of a result

da88c850c6 changed the logic to verify if a community is contained in the list of context of a result

86e50f7311 modified code to split the Croatian funder

69fd40fd30 modified code to split the Croatian funder

87a6e2b967 extended test class

4b432fbee8 extended test class

65a242646d added resource for APC dump

6410ab71d8 added APC in the dump and test method

13c66e16be changed logic to split for communities

9a58f1b93d added logic to select only the valid relations: those not deletedbyinference and having both part of the relation as entities in the graph

39b1a6edf6 added test class for the selection of valid relations and description

8429aed6c6 Added resource for testing selection of valid relations

e9a17ec899 added check to verify not to add void APC

f13e11e3f7 [aggregation] datacite wf: defined parameter declaring the path used to store the OAF objects produced by the transformation phase

9629569e22 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

5295d10691 added check not to dump deletedByInference entities

72397ea1ba Added fix for community of arbitrary name length

8f322a73cb change because of the renaming of originalId in acronym

fa720c1da4 [doiboost] added workflow for the ActionSet update dedicated to production

084b4ef999 added the creation of the openaireId from funder and grant number if the element is not present in the context profile

59615da65e Add test to verify the creation of relation between context and projects

618d2de2da minor changes and refactoring

d418c309f5 removed the part after part-x- in the file name generated by spark. It was too long and created problems while creating the tar entries

734de62474 [doiboost] added workflow for the ActionSet update dedicated to production

5e38c7f42d dumping only communities with status all

d70f8c96fd funding contains and not starts with h2020

c028feef4f workflow for the dump as sub workflows

eae10c5894 modification to allow the dump for a single community

970b387b8d modification to allow dump of a single community

52ce35d57b -

320cf02d96 Changed the way to find results linked to projects. We verify to actually have the project on the graph before selecting the result

886617afd0 One result linked to more than on project is saved just once

774cdb190e changes to mirror the last dump of the graph with the ols data model.

4cb65bc64a fixed process doiboost workflow:

- splitted OrcidToOAF into two phase preprocess and process
- updated workflow used in production

1cdd09cd8e Tentative fix for testing of Jenkins

10068c00ea Code refactor:

- removed old workflows in doiboost
 - splitted workflow of doiboost in preprocess and process

a65667d217 added publication to dataset even if no contributors

7840cc6526 merged with master

66604bb2b4 added absolute path to process folder

2dc50c0999 added default value to process path

9913b6073c Merge pull request 'orcid-no-doi' (#123 ) from enrico.ottonello/dnet-hadoop:orcid-no-doi into master

Reviewed-on: #123

5e5f65a3c3 contents mapped from the stores with 'claim' interpretation will not change their identifier along their way towards the graph

13cf444f85 Merge pull request 'force orginalId for claimed records' (#124 ) from forceOrginalId_claims into master

Reviewed-on: #124

5947cddafc adding record identifier among the originalIds regardless of what IdentifierFactory produces

77e8c6c7f7 filtering 'old' OpenAIRE ids from the entity.originalId[] array in the OAF -> XML searialization procedure

9069958479 tests for enermaps

ffdb2a3ea3 [cleaning] fixed filtering function for missing titles

5b6844b969 mapping funding relations from Datacite should be done according to the actual result identifier

35e395eae8 merge with master

998b66855a updated assertions in eu.dnetlib.dhp.oa.graph.raw.MappersTest

d267dce520 [raw_all] added extra workflow step for patching the identifiers in the relations, given an id mapping dataset

9594343725 code formatting after mvn compile

a9961a1835 [cleaning] title cleaning based on the me.xuender:unidecode library

1923c1ce21 replaced full join + filtering with a left join

5f7330d407 Merge branch 'master' into fct_project_id_replacement

f83dd70e1c Merge pull request 'Patch the identifiers (source/target) in the relations' (#125 ) from fct_project_id_replacement into master

Reviewed-on: #125

e725c88ebb [raw_all] patching relation identifier phase to be run at the end, i.e. includes also claimed relations

6e3554a45e [provision] lowercase relation filter

576693d782 added unit test for PatchRelationsApplication

081fe92a21 Merge branch 'fct_project_id_replacement' of https://code-repo.d4science.org/D-Net/dnet-hadoop into fct_project_id_replacement

2fc89fc9b5 Merge branch 'fct_project_id_replacement' of https://code-repo.d4science.org/D-Net/dnet-hadoop into fct_project_id_replacement

9bc4fd3b69 Patch FCT relations - fixed issue with join

a6a38cca9e fixed implementation of PatchRelationsApplication, refined the relative unit test

4f78565c04 fixed implementation of PatchRelationsApplication, refined the relative unit test

5219d56be5 Merge pull request 'Patch the identifiers (source/target) in the relations, refinements' (#131 ) from fct_project_id_replacement into master

Reviewed-on: #131

11e26c020a Update 'README.md'

74afe43c3a fixed wrong test file

ee13da9258 merge branch with master

73dc082927 added new dumped field (openaccessroute, pid and alternate identifier at the level of the instance) and the bipFinder measure at the level of the result

6bd1eca7e0 merge branch with beta

c3931557e3 extended the logic of the dump to consider the validation date in the relation (also in the dumped result for communities and funders at the level of the project), the extention on the instance for the APC, the pid, the alternate identifiers, and the extention of the AccessRight to store the OpenAccessRoute. Added new resourec for testing and extended the old class to verify the new dump. Fixed also issue on relation dump: only relation whose source and target are entities in the graph are dumped. The same hold for references to projects

5d70f842eb mergin with branch beta

eff499af9f added new tests and changed the test example

bc9e3a06ba Graph Dump - extended the test class

8daaa32e90 Graph Dump - added resources for testing

bd0d7bfba7 Graph Dump - added resources for testing addition of validation date in project for communityresult

a90bac3bc9 Graph Dump - added method to test class to verify addition of validation date in projects for community result

c8321ad31a merge with branch beta

c4ccd7b32c -

9814c3e700 mergin with branch beta

e653756e3d applied some suggestiond from Sonar Lint

83f51f1812 refactoring

fec40bdd95 merging with branch beta - resolved conflicts

63933808d4 added fix for mixing result types, added configuration default to funder subworkflow

46f82c7c8f removed not needed folder deletion

76d41602be Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

1315952702 merge with branch beta

c7f6cd2591 added again the setting for saXReader

claudio.atzori added 1 commit 2021-10-19 16:15:05 +02:00

e9157c67aa Merge branch 'beta' into dump

claudio.atzori manually merged commit d517c71458 into beta

2021-10-19 16:20:17 +02:00

claudio.atzori referenced this pull request

2021-11-26 15:55:18 +01:00

promoting beta branch to master - November 2021 #165

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#151