WIP: dump of the OpenAIRE graph - Changes #103

miriam.baglioni · 2021-04-09T17:12:44+02:00

miriam.baglioni commented

2021-04-09 17:12:44 +02:00

This PR adds:

the information about APC in the instance
relations extracted from context versus projects and vice-versa

fixes:

dump for relations which start and target relations may not exist in the graph
dump for organizations with deletedbyinference = true

This PR adds: * the information about APC in the instance * relations extracted from context versus projects and vice-versa fixes: * dump for relations which start and target relations may not exist in the graph * dump for organizations with deletedbyinference = true

miriam.baglioni added the

bug

enhancement

labels 2021-04-09 17:12:44 +02:00

claudio.atzori was assigned by miriam.baglioni

2021-04-09 17:12:44 +02:00

miriam.baglioni added 59 commits 2021-04-09 17:12:45 +02:00

815c7c11aa merge branch with master

bc09d37e8c used constants in ModelConstants class

7c86e66697 merge branch with master

0d76e039cf changed to logic to verify in community is contained in the list of context of a result

fe36895c53 added datasource blacklist for the organization to result propagation through institutional repositories

9bdadd4ddb merge branch with master

6190465851 merge upstream

2f5e6647c6 merge upstream

f7c35e6311 merge branch with master

896919e735 merge upstream

a684e1065e merge branch with master

d4ad740c98 merge branch with master

9841086ef3 modified code to split the Croazian funders

5d92df0627 tried again to fix issue for croatian funder

871e5bea29 should have fixed for real now

cc11ee1cb9 changed the param value to directly upload on Zenodo

17049f8bde extended the test class - should have done by the start :)

defbb71561 extended resource info to match the new test

330343937c -

4b6e514f02 merge upstream

931b2a2e15 merge branch with master

200e7e9c46 modified description

5dea729de3 added article processing charges and modified description

a896febc02 added APC in the dumped information

d0c94462e4 refactoring

16c1a27852 added test for APC dump

e5463fea01 added resource for apc dump

08f8dd9454 refactoring

870ee28dd6 refactoring

4632795f25 merge branch with master

f7714645d2 merge with dump

f93356f690 refactoring

daabc370c5 changed the workflow to add the step for selecting the valid relations

c0c225f3b2 added logic to select only the valid relations: those not deletedbyinference and having both part of the relation as entities in the graph

152ba8e2ef added description

8d28ca9815 added test class for the selection of valid relations

b3f02083e7 refactoring

2c209e1140 added resources for testing selection of valid relations

0421f5e1d8 added check to verify not to add void APC

5022f1b50d removing organization deletedbyinference from the dump

26cf32c066 changed the test to mirror the change in the logic of the code

ad6d0ca9eb added to all the entities the check that deletedbyinference = false

70e391d427 merge upstream

66d64947af merge branch with master

7f9b7cfcf6 removing from the dump organization that have been deleted by inference

22f4930479 refactoring

e1af14833d refactoring

c52355b516 refactoring

f95ec49a59 changed the substring to be pk for communities of arbitrary name length

46a322b770 changed the name of originalId in acronym

3e3a45d930 refactoring

c58206c3ba added test for the creation of relations with funders

eaf86828e6 refactoring

95c5f97259 added the part for the extraction of relations versus projects

3d94c12d6e refactoring

26b34201ec refactoring

bd4b6b053d changed classid with classname in the construction of provenance for the dump

6b51b69cf7 added the creation of the openaireId from funder and grant number if the element is not present in the context profile

04a0d1ba6e added test method to check the creation of relations between context and projects

miriam.baglioni added 17 commits 2021-04-23 12:24:03 +02:00

6179deb836 removed the part after part-x- in the file name generated by spark. It was too long and created problems while creating the tar entries

8c4c74a640 changed logic to be able to create a dump for a single community at a time

50d13a1d74 changed the workflow for the dump of a single community

08e731916b removed parameter communityMap when sending data to Zenodo

80a7170794 -

bc501f41f6 added test class for community removal from the set to be dumped

7465fa3f20 dumping only the communities with status "all". We decided those with status manager wil be available on demand

7cf1f49d5e if the funding does not start with H2020 but contains it the nsp should be corda__h2020

251178aca8 the new json schema for the result

a469d79b84 test for the creation of relationships between context and projects when the funding contains h2020

eb0762622c added decision node to upload on zenodo or not

8981a82011 -

7d1b8b7f64 merge upstream

4ae6fba01d refactoring

72e5aa3b42 refactoring

1416a49b63 merge branch with master

3c3e3537e0 Merge branch 'singleCommunityDump' into dump

miriam.baglioni commented

2021-04-23 16:06:06 +02:00

The PR also extends the dump with the possibility of dumping a single community content and upload it on Zenodo by its own.

It also adds in the dump only the communities with status "all". Those ith status "manager" are not dumped anymore.

The PR also extends the dump with the possibility of dumping a single community content and upload it on Zenodo by its own. It also adds in the dump only the communities with status "all". Those ith status "manager" are not dumped anymore.

miriam.baglioni added 9 commits 2021-05-20 10:44:20 +02:00

952d6dc2fb modified the code to allow the dump for a single community (indip from its status)

bcbadeb107 added a decision node to choose if to upload results in zenodo

e902699afa refactoring

25c5dd744f new definition of the workflow for the dump as sub workflows

c6cadacd24 Changed the way to find results linked to projects. We verify to actually have the project on the graph before selecting the result

b8f25fbf39 changed the way to find the funders: we select the projects distinct nsp

85bd6ab5d0 added verification tha one result linked to more than on project is saved just onece

f5ecd0d3ab added new parameters

e0ca4cead0 -

miriam.baglioni added 2 commits 2021-05-20 11:28:12 +02:00

02b80cf24f resolved conflicts

197471972c resolved conflicts

miriam.baglioni commented

2021-05-20 16:21:26 +02:00

Added also specification of dump WFs as subworkflows and the possibility to dump community results indipendently from the status

claudio.atzori commented

2021-08-12 17:07:13 +02:00

We shall re-issue the PR for updating the dump procedures as soon as the dhp-schemas:2.7.17 will be released and made available on the beta branch.

We shall re-issue the PR for updating the dump procedures as soon as the `dhp-schemas:2.7.17` will be released and made available on the `beta` branch.

claudio.atzori closed this pull request

2021-08-12 17:07:14 +02:00