antonis.lempesis
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 1
Code Issues Pull Requests Projects Releases Wiki Activity
542 Commits 18 Branches 0 Tags 128 MiB
Commit Graph

542 Commits

Author SHA1 Message Date
Claudio Atzori daa26acc9d dataset based provision WIP, fixed spark2EventLogDir 2020-04-02 16:15:50 +02:00
Przemysław Jacewicz 7b2a7e2417 [dhp-actionmanager] missing descriptions added and minor naming and formatting fixes 2020-04-02 11:48:40 +02:00
Claudio Atzori 9c7092416a dataset based provision WIP 2020-04-01 19:07:30 +02:00
Przemysław Jacewicz 80cf43b9c8 [dhp-actionmanager] promoting workflow added 2020-04-01 18:51:25 +02:00
Przemysław Jacewicz 5b459bcc47 [dhp-actionmanager] promoting spark job added 2020-04-01 18:49:08 +02:00
Przemysław Jacewicz e21bb89dbd [dhp-actionmanager] partitioning spark job added 2020-04-01 18:41:29 +02:00
Przemysław Jacewicz f9f7350bb9 [dhp-actionmanager] common package added with utility classes supporting hadoop and spark envs 2020-04-01 18:39:26 +02:00
Przemysław Jacewicz ad70c23b2e [dhp-actionmanager] pom updated 2020-04-01 18:36:00 +02:00
Przemysław Jacewicz 4e910a78d4 [dhp-workflows] spark 2 connection properties added 2020-04-01 18:29:26 +02:00
Przemysław Jacewicz 51ff3b4e81 [dhp-schemas] added safeguard against casting exception in mergeFrom methods and null-safe handling of collectedfrom collection for relation 2020-04-01 18:28:23 +02:00
Claudio Atzori 1402eb1fe7 cleanup 2020-04-01 15:38:50 +02:00
Claudio Atzori 7061d07727 ActionSets migration serialize the output as plain text files instead of SequenceFiles 2020-04-01 14:58:22 +02:00
Claudio Atzori adcdd2d05e WIP: reimplementing the adjacency list construction process using spark Datasets 2020-04-01 14:56:57 +02:00
Sandro La Bruzzo 201d79021e Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-31 14:41:41 +02:00
Sandro La Bruzzo cd7416ae4c first implementation of incremental update of scholix index 2020-03-31 14:41:35 +02:00
przemek 9d1d18d4b9 Merge branch 'master' into przemyslawjacewicz_actionmanager_impl_prototype 2020-03-31 12:04:58 +02:00
Claudio Atzori 377e1ba840 [maven-release-plugin] prepare for next development iteration 2020-03-30 20:06:00 +02:00
Claudio Atzori 76d9315129 [maven-release-plugin] prepare release dhp-1.1.6 2020-03-30 20:05:56 +02:00
Claudio Atzori ef429010ee removed log file and job-override.properties 2020-03-30 20:00:58 +02:00
Claudio Atzori 0fbec69b82 use oozie prepare statement to cleanup working directories 2020-03-30 19:48:41 +02:00
Claudio Atzori 3af2b8d700 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-03-30 13:12:21 +02:00
Claudio Atzori f3f9affd49 allow dynamic executors to build XML records 2020-03-30 13:12:11 +02:00
Claudio Atzori 2e2d4c4c68 adjusted path to template resource 2020-03-30 13:11:49 +02:00
Miriam Baglioni dd011f4a95 to make them visible to Claudio 2020-03-30 10:55:47 +02:00
Miriam Baglioni b1af90a45f to make it visible to Claudio 2020-03-30 10:50:03 +02:00
Sandro La Bruzzo 62cc257e5c fixed step1 workflow 2020-03-27 17:07:34 +01:00
Sandro La Bruzzo 1a7a866861 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-27 15:11:48 +01:00
Sandro La Bruzzo 7cef698f36 reformat code 2020-03-27 15:11:34 +01:00
Claudio Atzori 1767dfaa3f method can be protected, it is meant to be used only in tests 2020-03-27 14:31:26 +01:00
Sandro La Bruzzo a4b6a51168 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-27 13:48:56 +01:00
Sandro La Bruzzo 15d9106b3f FIxed merge of dhp dedup 2020-03-27 13:48:44 +01:00
Claudio Atzori 589d1c1443 ignore *.log files 2020-03-27 13:45:25 +01:00
Claudio Atzori e196fff212 adjusted path for source resource in unit test 2020-03-27 13:45:10 +01:00
Sandro La Bruzzo 8c9a56a0c8 refactored package name 2020-03-27 13:19:33 +01:00
Sandro La Bruzzo 2bd2d6f202 Merge branch 'master' of code-repo.d3science.org:D-Net/dnet-hadoop 2020-03-27 13:16:36 +01:00
Sandro La Bruzzo a9935f80d4 refactor class name and workflow name for graph mapper, added javadoc 2020-03-27 13:16:24 +01:00
Michele Artini ae03948eed Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-27 11:47:07 +01:00
Michele Artini f6e86b44a6 tests 2020-03-27 11:46:37 +01:00
Michele Artini 408be3c632 test and fixed a problem with datacite namespaces 2020-03-27 11:44:50 +01:00
Claudio Atzori 673e744649 moved openaire specific implementations under dedicated package eu.dnetlib.dhp.oa 2020-03-27 10:42:17 +01:00
Claudio Atzori 098fabab3f reorganizing content under dhp-workflows/dhp-graph-mapper 2020-03-26 19:44:19 +01:00
Claudio Atzori 77c4294924 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-03-26 18:26:52 +01:00
Claudio Atzori 43cbcda7ef unit test for SparkGraphImporterJob 2020-03-26 18:26:40 +01:00
Sandro La Bruzzo e04da6d66a merged all oozie wf in one 2020-03-26 14:17:07 +01:00
Sandro La Bruzzo e71e001b58 commented test that doesn't work 2020-03-26 14:15:21 +01:00
Sandro La Bruzzo 0cd022ad6a merge with master 2020-03-26 14:08:29 +01:00
Claudio Atzori abcd3f5bf5 added sample data for unit tests 2020-03-26 11:12:52 +01:00
Sandro La Bruzzo d5f11e27be renamed wf 2020-03-26 09:49:23 +01:00
Sandro La Bruzzo 9a37ad0127 renamed modules 2020-03-26 09:46:46 +01:00
Sandro La Bruzzo a768226e52 updated generate scholix to generate json 2020-03-26 09:40:50 +01:00