lsmyrnaios
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Pull Requests Activity
446 Commits 68 Branches 18 Tags 877 MiB
Commit Graph

446 Commits

Author SHA1 Message Date
miconis 4da13e4570 Revert "Merge branch 'master' into deduptesting" 
This reverts commit 772f75d167, reversing
changes made to 5f45f2c77f.
 2020-04-20 16:04:49 +02:00
miconis 772f75d167 Merge branch 'master' into deduptesting 2020-04-20 14:50:12 +02:00
Claudio Atzori d714bfb4d4 collectedfrom field moved in common parent class Oaf.java 2020-04-20 12:25:19 +02:00
Michele Artini 8ff7facfa3 fixed collectedFrom ID 2020-04-20 11:09:27 +02:00
Michele Artini 25307965d2 add a default datainfo if missing 2020-04-20 09:43:27 +02:00
Michele Artini d2058fdc47 tests 2020-04-20 09:31:14 +02:00
Michele Artini 478a958f09 tests 2020-04-20 09:15:27 +02:00
Claudio Atzori 5f45f2c77f Merge branch 'master' into deduptesting 2020-04-18 12:46:40 +02:00
Claudio Atzori ad7a131b18 introduced common project code formatting plugin, works on the commit hook, based on https://github.com/Cosium/git-code-format-maven-plugin, applied to each java class in the project 2020-04-18 12:42:58 +02:00
Claudio Atzori a2938dd059 cleanup 2020-04-18 12:24:22 +02:00
Claudio Atzori 9374ff03ea Merge branch 'master' into deduptesting 2020-04-18 12:06:58 +02:00
Claudio Atzori 71813795f6 various refactorings on the dnet-dedup-openaire workflow 2020-04-18 12:06:23 +02:00
miconis 6450bb0daa test for softwares dedup added. definition of orp, dataset and sw dedup configurations 2020-04-17 17:31:59 +02:00
Claudio Atzori 038ac7afd7 relation consistency workflow separated from dedup scan and creation of CCs 2020-04-17 13:12:44 +02:00
Claudio Atzori c92bfeeaee Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-04-17 13:07:52 +02:00
Sandro La Bruzzo 01ea7721f3 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-04-17 12:12:25 +02:00
Sandro La Bruzzo 5e2fa996aa fixed problem with conversion of long into string 2020-04-17 12:11:51 +02:00
miconis 418cf94642 implementation of the deletedbyinference test in propagating relations 2020-04-17 10:40:21 +02:00
Claudio Atzori cb0952428e Merge branch 'master' into deduptesting 2020-04-16 14:42:25 +02:00
Claudio Atzori cc21bbfb1a Merge branch 'deduptesting' of https://code-repo.d4science.org/D-Net/dnet-hadoop into deduptesting 2020-04-16 14:41:37 +02:00
Claudio Atzori ec5dfc068d added spark.sql.shuffle.partitions=3840 to dedup scan wf 2020-04-16 14:41:28 +02:00
Claudio Atzori 09f356b047 Merge pull request 'Closes #7: subdirs inside graph table dirs' (#8) from przemyslaw.jacewicz/dnet-hadoop:przemyslawjacewicz_7_distcp_configuration_fix into master 
Run the code from this PR in isolation and it worked fine. Thanks!
 2020-04-16 14:38:46 +02:00
Claudio Atzori 3437383112 Merge branch 'master' into deduptesting 2020-04-16 12:46:14 +02:00
miconis 0eccbc318b Deduper class (utilities for dedup) cleaned. Useless methods removed 2020-04-16 12:36:37 +02:00
Claudio Atzori 76d23895e6 Merge branch 'deduptesting' of https://code-repo.d4science.org/D-Net/dnet-hadoop into deduptesting 2020-04-16 12:18:32 +02:00
miconis 6a089ec287 minor changes 2020-04-16 12:15:38 +02:00
Claudio Atzori 376efd67de removed prepare statement in spark action 2020-04-16 12:14:16 +02:00
miconis 9b36458b6a Merge branch 'deduptesting' of code-repo.d4science.org:D-Net/dnet-hadoop into deduptesting 2020-04-16 12:13:58 +02:00
miconis cd4d9a148f creating temporary directories in dedup test 2020-04-16 12:13:26 +02:00
Claudio Atzori b39ff36c16 improving the wf definitions 2020-04-16 12:11:37 +02:00
Claudio Atzori 011b342bc9 trying to avoid OOM in SparkPropagateRelation 2020-04-16 11:13:51 +02:00
Claudio Atzori 069ef5eaed trying to avoid OOM in SparkPropagateRelation 2020-04-15 21:23:21 +02:00
Claudio Atzori 8eedfefc98 try to introduce intermediate serialization on hdfs to avoid OOM 2020-04-15 18:35:35 +02:00
Przemysław Jacewicz da019495d7 [dhp-actionmanager] target dir removal added for distcp actions 2020-04-15 17:56:57 +02:00
miconis 5689d49689 minor changes 2020-04-15 16:34:06 +02:00
Claudio Atzori c439d0c6bb PromoteActionPayloadForGraphTableJob reads directly the content pointed by the input path, adjusted promote action tests (ISLookup mock) 2020-04-15 16:18:33 +02:00
Claudio Atzori ff30f99c65 using newline delimited json files for the raw graph materialization. Introduced contentPath parameter 2020-04-15 16:16:20 +02:00
Sandro La Bruzzo 3d3ac76dda Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-04-15 15:24:01 +02:00
Sandro La Bruzzo 74a7fac774 fixed problem with timestamp 2020-04-15 15:23:54 +02:00
Alessia Bardi 550a9f82ed Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-04-14 17:53:01 +02:00
Alessia Bardi a68fae9bcb now supporting openaire 4.0 compliance 2020-04-14 17:52:48 +02:00
Sandro La Bruzzo c36239e693 fixed incremental indexing 2020-04-14 17:47:36 +02:00
Claudio Atzori 82e8341f50 reorganizing parameter names in the provision workflow 2020-04-14 15:54:41 +02:00
Claudio Atzori 6b5f9ca9cb raw graph creation workflow moved under dhp-graph-mapper, claims integration is included 2020-04-10 17:53:07 +02:00
miconis 0be2e72be5 further implementation of tests for the deduplication of each entity. publication dump added, empty entity files created 2020-04-08 18:02:30 +02:00
Claudio Atzori 47f3d9b757 unit test for GraphHiveImporterJob 2020-04-08 13:24:43 +02:00
Sandro La Bruzzo ba9f07a6fe fixed wrong test 2020-04-08 13:18:20 +02:00
Claudio Atzori d74e128aa6 Utility classes moved in dhp-common and dhp-schemas 2020-04-07 11:56:22 +02:00
Claudio Atzori c57cf679ca Merge branch 'provision_dataset' 2020-04-07 08:56:58 +02:00
Claudio Atzori 1a1a026a18 we do expect to find field bestaccessright already defined. No need to add it again 2020-04-07 08:55:33 +02:00