Miriam Baglioni
e902699afa
refactoring
2021-04-23 13:48:27 +02:00
Miriam Baglioni
bcbadeb107
added a decision node to choose if to upload results in zenodo
2021-04-23 12:42:06 +02:00
Miriam Baglioni
952d6dc2fb
modified the code to allow the dump for a single community (indip from its status)
2021-04-23 12:41:32 +02:00
Miriam Baglioni
3c3e3537e0
Merge branch 'singleCommunityDump' into dump
2021-04-23 12:12:43 +02:00
Miriam Baglioni
1416a49b63
merge branch with master
2021-04-23 12:11:15 +02:00
Miriam Baglioni
72e5aa3b42
refactoring
2021-04-23 12:10:30 +02:00
Miriam Baglioni
4ae6fba01d
refactoring
2021-04-23 12:09:19 +02:00
Miriam Baglioni
7d1b8b7f64
merge upstream
2021-04-23 11:55:49 +02:00
Miriam Baglioni
8981a82011
-
2021-04-23 11:55:20 +02:00
Miriam Baglioni
eb0762622c
added decision node to upload on zenodo or not
2021-04-23 11:54:54 +02:00
Miriam Baglioni
a469d79b84
test for the creation of relationships between context and projects when the funding contains h2020
2021-04-23 11:52:27 +02:00
Miriam Baglioni
251178aca8
the new json schema for the result
2021-04-23 11:51:27 +02:00
Miriam Baglioni
7cf1f49d5e
if the funding does not start with H2020 but contains it the nsp should be corda__h2020
2021-04-23 11:50:26 +02:00
Miriam Baglioni
7465fa3f20
dumping only the communities with status "all". We decided those with status manager wil be available on demand
2021-04-23 11:49:45 +02:00
Claudio Atzori
906d50563c
Merge pull request 'properly invalidating impala metadata' ( #105 ) from antonis.lempesis/dnet-hadoop:master into master
...
Reviewed-on: #105
2021-04-15 15:06:22 +02:00
Antonis Lempesis
03d36fadea
properly invalidating impala metadata
2021-04-15 13:34:22 +03:00
Miriam Baglioni
bc501f41f6
added test class for community removal from the set to be dumped
2021-04-13 16:40:24 +02:00
Miriam Baglioni
80a7170794
-
2021-04-13 16:39:55 +02:00
Miriam Baglioni
08e731916b
removed parameter communityMap when sending data to Zenodo
2021-04-13 16:38:59 +02:00
Miriam Baglioni
50d13a1d74
changed the workflow for the dump of a single community
2021-04-13 16:33:00 +02:00
Miriam Baglioni
8c4c74a640
changed logic to be able to create a dump for a single community at a time
2021-04-13 16:32:19 +02:00
Miriam Baglioni
6179deb836
removed the part after part-x- in the file name generated by spark. It was too long and created problems while creating the tar entries
2021-04-13 16:30:59 +02:00
dcff9cecdf
bug fix: ids in self mergerels are not marked deletedbyinference=true
2021-04-12 15:55:27 +02:00
Miriam Baglioni
04a0d1ba6e
added test method to check the creation of relations between context and projects
2021-04-09 12:49:51 +02:00
Miriam Baglioni
6b51b69cf7
added the creation of the openaireId from funder and grant number if the element is not present in the context profile
2021-04-09 12:49:07 +02:00
Miriam Baglioni
bd4b6b053d
changed classid with classname in the construction of provenance for the dump
2021-04-09 12:48:09 +02:00
Miriam Baglioni
26b34201ec
refactoring
2021-04-09 12:47:03 +02:00
Miriam Baglioni
3d94c12d6e
refactoring
2021-04-09 12:45:45 +02:00
Miriam Baglioni
95c5f97259
added the part for the extraction of relations versus projects
2021-04-09 11:31:37 +02:00
Miriam Baglioni
eaf86828e6
refactoring
2021-04-09 11:30:30 +02:00
Miriam Baglioni
c58206c3ba
added test for the creation of relations with funders
2021-04-09 11:30:07 +02:00
Miriam Baglioni
3e3a45d930
refactoring
2021-04-08 10:44:37 +02:00
Miriam Baglioni
46a322b770
changed the name of originalId in acronym
2021-04-08 10:40:06 +02:00
Miriam Baglioni
f95ec49a59
changed the substring to be pk for communities of arbitrary name length
2021-04-07 13:22:54 +02:00
Miriam Baglioni
c52355b516
refactoring
2021-04-07 12:13:45 +02:00
Miriam Baglioni
e1af14833d
refactoring
2021-04-07 12:13:00 +02:00
Miriam Baglioni
22f4930479
refactoring
2021-04-07 12:12:04 +02:00
Miriam Baglioni
7f9b7cfcf6
removing from the dump organization that have been deleted by inference
2021-04-07 12:11:36 +02:00
Miriam Baglioni
66d64947af
merge branch with master
2021-04-07 10:38:18 +02:00
Miriam Baglioni
70e391d427
merge upstream
2021-04-07 10:38:08 +02:00
Miriam Baglioni
ad6d0ca9eb
added to all the entities the check that deletedbyinference = false
2021-04-07 10:37:49 +02:00
Claudio Atzori
37b65cc3ad
Merge pull request 'updates on stats-update workflow' ( #100 ) from antonis.lempesis/dnet-hadoop:master into master
...
The workflow integrated in the _stable_ids_ branch has been run correctly on the BETA content, thus IMO this PR can be integrated in the master branch.
Reviewed-on: #100
2021-04-02 16:13:35 +02:00
Miriam Baglioni
26cf32c066
changed the test to mirror the change in the logic of the code
2021-04-01 18:22:57 +02:00
Miriam Baglioni
5022f1b50d
removing organization deletedbyinference from the dump
2021-04-01 18:16:40 +02:00
Miriam Baglioni
0421f5e1d8
added check to verify not to add void APC
2021-04-01 17:38:30 +02:00
Miriam Baglioni
2c209e1140
added resources for testing selection of valid relations
2021-04-01 16:57:34 +02:00
Miriam Baglioni
b3f02083e7
refactoring
2021-04-01 16:56:58 +02:00
Miriam Baglioni
8d28ca9815
added test class for the selection of valid relations
2021-04-01 16:56:32 +02:00
Miriam Baglioni
152ba8e2ef
added description
2021-04-01 16:55:57 +02:00
Miriam Baglioni
c0c225f3b2
added logic to select only the valid relations: those not deletedbyinference and having both part of the relation as entities in the graph
2021-04-01 16:53:33 +02:00
Miriam Baglioni
Claudio Atzori
Antonis Lempesis
miconis