lsmyrnaios
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Pull Requests Activity
461 Commits 68 Branches 18 Tags 643 MiB
Commit Graph

461 Commits

Author SHA1 Message Date
Claudio Atzori c439d0c6bb PromoteActionPayloadForGraphTableJob reads directly the content pointed by the input path, adjusted promote action tests (ISLookup mock) 2020-04-15 16:18:33 +02:00
Claudio Atzori ff30f99c65 using newline delimited json files for the raw graph materialization. Introduced contentPath parameter 2020-04-15 16:16:20 +02:00
Sandro La Bruzzo 3d3ac76dda Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-04-15 15:24:01 +02:00
Sandro La Bruzzo 74a7fac774 fixed problem with timestamp 2020-04-15 15:23:54 +02:00
Alessia Bardi 550a9f82ed Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-04-14 17:53:01 +02:00
Alessia Bardi a68fae9bcb now supporting openaire 4.0 compliance 2020-04-14 17:52:48 +02:00
Sandro La Bruzzo c36239e693 fixed incremental indexing 2020-04-14 17:47:36 +02:00
Claudio Atzori 82e8341f50 reorganizing parameter names in the provision workflow 2020-04-14 15:54:41 +02:00
Claudio Atzori 6b5f9ca9cb raw graph creation workflow moved under dhp-graph-mapper, claims integration is included 2020-04-10 17:53:07 +02:00
miconis 0be2e72be5 further implementation of tests for the deduplication of each entity. publication dump added, empty entity files created 2020-04-08 18:02:30 +02:00
Claudio Atzori 47f3d9b757 unit test for GraphHiveImporterJob 2020-04-08 13:24:43 +02:00
Sandro La Bruzzo ba9f07a6fe fixed wrong test 2020-04-08 13:18:20 +02:00
Claudio Atzori d74e128aa6 Utility classes moved in dhp-common and dhp-schemas 2020-04-07 11:56:22 +02:00
Claudio Atzori c57cf679ca Merge branch 'provision_dataset' 2020-04-07 08:56:58 +02:00
Claudio Atzori 1a1a026a18 we do expect to find field bestaccessright already defined. No need to add it again 2020-04-07 08:55:33 +02:00
Claudio Atzori fbdd18a96b using dataset based relation preparation procedure 2020-04-07 08:54:39 +02:00
Claudio Atzori 77f59b1b10 dataset based provision WIP 2020-04-06 19:37:27 +02:00
Claudio Atzori 6177cf36fb Merge pull request 'Closes #4: New action manager implementation' (#5) from przemyslaw.jacewicz/dnet-hadoop:przemyslawjacewicz_actionmanager_impl_prototype into master 
Nothing more to add here. Thanks for your contribution!
 2020-04-06 17:35:07 +02:00
Claudio Atzori e355961997 dataset based provision WIP 2020-04-06 17:34:25 +02:00
miconis 56fbe689f0 implementation of the tests for each spark action 2020-04-06 16:30:31 +02:00
Claudio Atzori ca345aaad3 dataset based provision WIP 2020-04-06 15:33:31 +02:00
Claudio Atzori c8f4b95464 dataset based provision WIP 2020-04-06 08:59:58 +02:00
Claudio Atzori eb2f5f3198 dataset based provision WIP 2020-04-04 17:41:31 +02:00
Claudio Atzori 3d1b637cab dataset based provision WIP 2020-04-04 14:03:43 +02:00
miconis 53fd624c34 implemented test for sparkcreatesimrels 2020-04-03 18:32:25 +02:00
Claudio Atzori 24b2c9012e dataset based provision WIP 2020-04-02 18:44:09 +02:00
miconis a61763d149 structure for sparksimrel changed to be compliant with mockito testing 2020-04-02 18:37:53 +02:00
Claudio Atzori daa26acc9d dataset based provision WIP, fixed spark2EventLogDir 2020-04-02 16:15:50 +02:00
Przemysław Jacewicz 7b2a7e2417 [dhp-actionmanager] missing descriptions added and minor naming and formatting fixes 2020-04-02 11:48:40 +02:00
Claudio Atzori 9c7092416a dataset based provision WIP 2020-04-01 19:07:30 +02:00
miconis bfa5bc74df minor changes 2020-04-01 19:05:48 +02:00
Przemysław Jacewicz 80cf43b9c8 [dhp-actionmanager] promoting workflow added 2020-04-01 18:51:25 +02:00
Przemysław Jacewicz 5b459bcc47 [dhp-actionmanager] promoting spark job added 2020-04-01 18:49:08 +02:00
miconis 9802bcb9fe dedup testing 2020-04-01 18:48:31 +02:00
Przemysław Jacewicz e21bb89dbd [dhp-actionmanager] partitioning spark job added 2020-04-01 18:41:29 +02:00
Przemysław Jacewicz f9f7350bb9 [dhp-actionmanager] common package added with utility classes supporting hadoop and spark envs 2020-04-01 18:39:26 +02:00
Przemysław Jacewicz ad70c23b2e [dhp-actionmanager] pom updated 2020-04-01 18:36:00 +02:00
Przemysław Jacewicz 4e910a78d4 [dhp-workflows] spark 2 connection properties added 2020-04-01 18:29:26 +02:00
Przemysław Jacewicz 51ff3b4e81 [dhp-schemas] added safeguard against casting exception in mergeFrom methods and null-safe handling of collectedfrom collection for relation 2020-04-01 18:28:23 +02:00
Claudio Atzori 1402eb1fe7 cleanup 2020-04-01 15:38:50 +02:00
Claudio Atzori 7061d07727 ActionSets migration serialize the output as plain text files instead of SequenceFiles 2020-04-01 14:58:22 +02:00
Claudio Atzori adcdd2d05e WIP: reimplementing the adjacency list construction process using spark Datasets 2020-04-01 14:56:57 +02:00
Sandro La Bruzzo 201d79021e Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-03-31 14:41:41 +02:00
Sandro La Bruzzo cd7416ae4c first implementation of incremental update of scholix index 2020-03-31 14:41:35 +02:00
przemek 9d1d18d4b9 Merge branch 'master' into przemyslawjacewicz_actionmanager_impl_prototype 2020-03-31 12:04:58 +02:00
Claudio Atzori 377e1ba840 [maven-release-plugin] prepare for next development iteration 2020-03-30 20:06:00 +02:00
Claudio Atzori 76d9315129 [maven-release-plugin] prepare release dhp-1.1.6 2020-03-30 20:05:56 +02:00
Claudio Atzori ef429010ee removed log file and job-override.properties 2020-03-30 20:00:58 +02:00
Claudio Atzori 0fbec69b82 use oozie prepare statement to cleanup working directories 2020-03-30 19:48:41 +02:00
Claudio Atzori 3af2b8d700 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-03-30 13:12:21 +02:00