spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
364 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

364 Commits

Author SHA1 Message Date
Claudio Atzori 56f5a4eeab dhp-build tests upgraded to use junit5 Assertions 2020-03-25 17:38:31 +01:00
Claudio Atzori c02718c10b dhp-build tests upgraded to junit5 2020-03-25 17:23:22 +01:00
Claudio Atzori 23668d4a6a WIP adopting junit5 2020-03-25 16:49:45 +01:00
Claudio Atzori a226198a13 WIP adopting junit5 2020-03-25 16:47:39 +01:00
Michele Artini ebe45003d9 fixed some junit packages 2020-03-25 16:45:03 +01:00
Michele Artini d9bfdcd607 updated poms 2020-03-25 16:31:12 +01:00
Michele Artini 120e823cd1 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-25 16:00:10 +01:00
Claudio Atzori 71ae7dd272 renamed module dnet-dedup to dnet-dedup-openaire 2020-03-25 15:57:09 +01:00
Michele Artini fd57722c69 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-25 15:56:49 +01:00
Claudio Atzori f441f823dd fixed path referencing a test resource file 2020-03-25 15:21:46 +01:00
Claudio Atzori 51d0c9bdd7 integrated changes from branch dedupTest 2020-03-25 15:15:41 +01:00
Claudio Atzori 36f8f2ea66 master set to 'yarn' in spark actions, removed path to rawSet from the dedup scan workflow 2020-03-25 14:16:06 +01:00
Michele Artini 2559299da4 tests 2020-03-25 12:25:00 +01:00
Claudio Atzori 2180cc4fe7 more fields included in result view definition 2020-03-25 11:21:46 +01:00
Claudio Atzori efb0b7d660 master set to 'yarn' in spark actions 2020-03-25 11:15:35 +01:00
Michele Artini 0fda2c3a30 some tests on db records 2020-03-25 09:43:58 +01:00
miconis 02320de371 minor changes 2020-03-24 17:43:51 +01:00
miconis 8e8b5e8f30 roots wf merged in scan wf 2020-03-24 17:40:58 +01:00
Claudio Atzori 51ff68db66 Merge branch 'dedupTest' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dedupTest 2020-03-24 11:18:19 +01:00
Claudio Atzori 1e869e7bed using method available from currently used library 2020-03-24 11:17:44 +01:00
miconis f0d72b76a8 package structure fixed 2020-03-24 10:51:40 +01:00
Claudio Atzori aaedbb1b8b WIP: dedup workflow, stage 2 2020-03-24 09:59:28 +01:00
Michele Artini e3760c7f39 fix a bug with organization countries 2020-03-24 08:43:56 +01:00
Claudio Atzori 8b0ba3d76a posprocessing script correctly run as hive2 action 2020-03-23 17:40:39 +01:00
miconis 93e2291291 minor changes 2020-03-23 17:17:56 +01:00
miconis f7890a90df implementation of the mechanism that checks the existance of a mergerel file 2020-03-23 17:13:30 +01:00
miconis c20e179f5a structure of the workflows updated 2020-03-23 11:43:49 +01:00
Claudio Atzori 658d40ccbe WIP trying to use hive2 actions 2020-03-23 11:14:54 +01:00
Claudio Atzori ecb64e4998 Merge branch 'migration_wfs_regular_all_steps' 2020-03-23 08:57:01 +01:00
Michele Artini 15160032bd fixed a bug setting some organization fields 2020-03-23 08:39:14 +01:00
Claudio Atzori a4c52661a0 WIP: fixing dedup workflows 2020-03-20 19:17:24 +01:00
Claudio Atzori 6cb0a9bff0 dedup wf directory structure aligned with project commons 2020-03-20 16:48:14 +01:00
miconis e16e644faf implementation of the workflow for entity update and for relations update 2020-03-20 13:01:56 +01:00
miconis 6d879e2ee1 integration of the new AtomicAction class 2020-03-19 15:10:42 +01:00
miconis 6e0fb8efa0 minor changes 2020-03-19 15:08:03 +01:00
miconis 4e82a24af2 minor changes and implementation of the create connected components action 2020-03-19 15:01:07 +01:00
Claudio Atzori 36236dd1c1 action migration workflow produces eu.dnetlib.dhp.schema.action.AtomicAction(s) 2020-03-19 14:00:38 +01:00
Claudio Atzori a0ab15a64c need to stick on using guava:11.0.2 as it is the version used by the hadoop components (oozie client for sure). The last version (28.2-jre) breaks the oozie workflow submission 2020-03-19 13:58:58 +01:00
Sandro La Bruzzo 0594b92a6d implemented relation with dataset 2020-03-19 11:11:07 +01:00
Claudio Atzori 1850a02ae4 added simpler, AtomicAction replacement, based on the dhp.Oaf model 2020-03-19 10:44:16 +01:00
miconis 679b5869e5 implementation of the lookup procedure to take dedup conf from the resource profiles 2020-03-18 17:41:56 +01:00
Claudio Atzori abe8fb69a2 added global properties, moved postprocessing script inside the oozie_app directory 2020-03-18 15:43:54 +01:00
miconis f32eae5ce9 implementation of the spark action for the simrel creation 2020-03-18 14:27:49 +01:00
Claudio Atzori c7e0730720 compress the output produced by migration steps 1 and 2 2020-03-18 09:34:57 +01:00
Claudio Atzori 2f11e37602 fixed expansion of path variables 2020-03-17 19:41:07 +01:00
Claudio Atzori 2795b0b096 no need to mkdir a the all_entities file 2020-03-17 17:22:14 +01:00
Claudio Atzori 19746ad308 when reuseContent, reset ${workingPath}/all_entities 2020-03-17 17:17:06 +01:00
Claudio Atzori 2f0c85eeb3 updated parameters for regular_all_steps worfklow, introduced flag 'reuseContent' 2020-03-17 17:04:58 +01:00
Claudio Atzori b8290b5851 updated parameters for regular_all_steps worfklow 2020-03-17 15:45:30 +01:00
Claudio Atzori 4706f24ec5 updated parameters for regular_all_steps worfklow 2020-03-17 15:23:54 +01:00