miriam.baglioni
/
dnet-hadoop
forked from D-Net/dnet-hadoop
2
0
Fork 0
Code Issues 2 Pull Requests Releases Wiki Activity
2,320 Commits 24 Branches 0 Tags 607 MiB
Commit Graph

2320 Commits

Author SHA1 Message Date
Miriam Baglioni 3c3e3537e0 Merge branch 'singleCommunityDump' into dump 2021-04-23 12:12:43 +02:00
Miriam Baglioni 1416a49b63 merge branch with master 2021-04-23 12:11:15 +02:00
Miriam Baglioni 72e5aa3b42 refactoring 2021-04-23 12:10:30 +02:00
Miriam Baglioni 4ae6fba01d refactoring 2021-04-23 12:09:19 +02:00
Miriam Baglioni 7d1b8b7f64 merge upstream 2021-04-23 11:55:49 +02:00
Miriam Baglioni 8981a82011 - 2021-04-23 11:55:20 +02:00
Miriam Baglioni eb0762622c added decision node to upload on zenodo or not 2021-04-23 11:54:54 +02:00
Miriam Baglioni a469d79b84 test for the creation of relationships between context and projects when the funding contains h2020 2021-04-23 11:52:27 +02:00
Miriam Baglioni 251178aca8 the new json schema for the result 2021-04-23 11:51:27 +02:00
Miriam Baglioni 7cf1f49d5e if the funding does not start with H2020 but contains it the nsp should be corda__h2020 2021-04-23 11:50:26 +02:00
Miriam Baglioni 7465fa3f20 dumping only the communities with status "all". We decided those with status manager wil be available on demand 2021-04-23 11:49:45 +02:00
Claudio Atzori 906d50563c Merge pull request 'properly invalidating impala metadata' (#105) from antonis.lempesis/dnet-hadoop:master into master 
Reviewed-on: D-Net/dnet-hadoop#105
 2021-04-15 15:06:22 +02:00
Antonis Lempesis 03d36fadea properly invalidating impala metadata 2021-04-15 13:34:22 +03:00
Miriam Baglioni bc501f41f6 added test class for community removal from the set to be dumped 2021-04-13 16:40:24 +02:00
Miriam Baglioni 80a7170794 - 2021-04-13 16:39:55 +02:00
Miriam Baglioni 08e731916b removed parameter communityMap when sending data to Zenodo 2021-04-13 16:38:59 +02:00
Miriam Baglioni 50d13a1d74 changed the workflow for the dump of a single community 2021-04-13 16:33:00 +02:00
Miriam Baglioni 8c4c74a640 changed logic to be able to create a dump for a single community at a time 2021-04-13 16:32:19 +02:00
Miriam Baglioni 6179deb836 removed the part after part-x- in the file name generated by spark. It was too long and created problems while creating the tar entries 2021-04-13 16:30:59 +02:00
miconis dcff9cecdf bug fix: ids in self mergerels are not marked deletedbyinference=true 2021-04-12 15:55:27 +02:00
Miriam Baglioni 04a0d1ba6e added test method to check the creation of relations between context and projects 2021-04-09 12:49:51 +02:00
Miriam Baglioni 6b51b69cf7 added the creation of the openaireId from funder and grant number if the element is not present in the context profile 2021-04-09 12:49:07 +02:00
Miriam Baglioni bd4b6b053d changed classid with classname in the construction of provenance for the dump 2021-04-09 12:48:09 +02:00
Miriam Baglioni 26b34201ec refactoring 2021-04-09 12:47:03 +02:00
Miriam Baglioni 3d94c12d6e refactoring 2021-04-09 12:45:45 +02:00
Miriam Baglioni 95c5f97259 added the part for the extraction of relations versus projects 2021-04-09 11:31:37 +02:00
Miriam Baglioni eaf86828e6 refactoring 2021-04-09 11:30:30 +02:00
Miriam Baglioni c58206c3ba added test for the creation of relations with funders 2021-04-09 11:30:07 +02:00
Miriam Baglioni 3e3a45d930 refactoring 2021-04-08 10:44:37 +02:00
Miriam Baglioni 46a322b770 changed the name of originalId in acronym 2021-04-08 10:40:06 +02:00
Miriam Baglioni f95ec49a59 changed the substring to be pk for communities of arbitrary name length 2021-04-07 13:22:54 +02:00
Miriam Baglioni c52355b516 refactoring 2021-04-07 12:13:45 +02:00
Miriam Baglioni e1af14833d refactoring 2021-04-07 12:13:00 +02:00
Miriam Baglioni 22f4930479 refactoring 2021-04-07 12:12:04 +02:00
Miriam Baglioni 7f9b7cfcf6 removing from the dump organization that have been deleted by inference 2021-04-07 12:11:36 +02:00
Miriam Baglioni 66d64947af merge branch with master 2021-04-07 10:38:18 +02:00
Miriam Baglioni 70e391d427 merge upstream 2021-04-07 10:38:08 +02:00
Miriam Baglioni ad6d0ca9eb added to all the entities the check that deletedbyinference = false 2021-04-07 10:37:49 +02:00
Claudio Atzori 37b65cc3ad Merge pull request 'updates on stats-update workflow' (#100) from antonis.lempesis/dnet-hadoop:master into master 
The workflow integrated in the _stable_ids_ branch has been run correctly on the BETA content, thus IMO this PR can be integrated in the master branch.

Reviewed-on: D-Net/dnet-hadoop#100
 2021-04-02 16:13:35 +02:00
Miriam Baglioni 26cf32c066 changed the test to mirror the change in the logic of the code 2021-04-01 18:22:57 +02:00
Miriam Baglioni 5022f1b50d removing organization deletedbyinference from the dump 2021-04-01 18:16:40 +02:00
Miriam Baglioni 0421f5e1d8 added check to verify not to add void APC 2021-04-01 17:38:30 +02:00
Miriam Baglioni 2c209e1140 added resources for testing selection of valid relations 2021-04-01 16:57:34 +02:00
Miriam Baglioni b3f02083e7 refactoring 2021-04-01 16:56:58 +02:00
Miriam Baglioni 8d28ca9815 added test class for the selection of valid relations 2021-04-01 16:56:32 +02:00
Miriam Baglioni 152ba8e2ef added description 2021-04-01 16:55:57 +02:00
Miriam Baglioni c0c225f3b2 added logic to select only the valid relations: those not deletedbyinference and having both part of the relation as entities in the graph 2021-04-01 16:53:33 +02:00
Miriam Baglioni daabc370c5 changed the workflow to add the step for selecting the valid relations 2021-04-01 16:52:39 +02:00
Miriam Baglioni f93356f690 refactoring 2021-04-01 16:24:08 +02:00
Miriam Baglioni f7714645d2 merge with dump 2021-03-30 16:27:38 +02:00