andreas.czerniak
/
BrBETA_dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
480 Commits 23 Branches 0 Tags 90 MiB
Commit Graph

381 Commits

Author SHA1 Message Date
Miriam Baglioni 8f12292daa changed the way to save the results on filesystem 2020-04-11 16:47:34 +02:00
Miriam Baglioni 87f802821e new workflow for country propagation: it is composed of the preparation step and in the propagation. The propagation part runs in parallel on the result types 2020-04-11 16:40:22 +02:00
Miriam Baglioni a562080b0b parameters to be used in the prepared Job and in the actual country propagation job 2020-04-11 16:39:17 +02:00
Miriam Baglioni 1251ad4455 removed unuseful class 2020-04-11 16:38:13 +02:00
Miriam Baglioni aef9b3aa90 new parametric implementation of country propagation. Exploits information compute before and broadcasts it to each executor 2020-04-11 16:36:59 +02:00
Miriam Baglioni a2d833d5dd step of data preparation before actual country propagation will take palce 2020-04-11 16:36:03 +02:00
Miriam Baglioni 6897c920a2 classes in support of new implementation of country propagation 2020-04-11 16:35:26 +02:00
Miriam Baglioni 85766a02d8 added dependency to use hive on local machine 2020-04-11 16:34:22 +02:00
Miriam Baglioni 79b8ea4fed prepared information to be used in actual country propagation. Subset of info 2020-04-11 16:29:41 +02:00
Miriam Baglioni 1822476613 Test for country propagation 2020-04-11 16:28:09 +02:00
Miriam Baglioni 7783b09c5b new implementation for result to project propagation. Prepare some info to be used in propagation 2020-04-11 16:26:23 +02:00
Miriam Baglioni 90469789b9 two new classes fro new implementation of project to result propagation 2020-04-09 13:29:01 +02:00
Miriam Baglioni 627ad58a8b new wf definition 2020-04-09 11:33:19 +02:00
Miriam Baglioni 9c63c4840d new workflow and parameters for country propagation 2020-04-08 19:13:42 +02:00
Miriam Baglioni a2d309545b new parametrized implementation for country propagation 2020-04-08 19:12:59 +02:00
Miriam Baglioni 6dfdba9ef7 new parametrized implementation for country propagation 2020-04-08 18:14:37 +02:00
Miriam Baglioni 03f7cb6402 new parametrized implementation for country propagation 2020-04-08 18:08:41 +02:00
Miriam Baglioni df2fc4a6d7 Merge remote-tracking branch 'upstream/master' 2020-04-08 18:07:26 +02:00
Miriam Baglioni fcfef4632f input parameters for country propagation preparation job 2020-04-08 18:07:18 +02:00
Miriam Baglioni 61045e84d9 merged conflict in pom 2020-04-08 14:23:30 +02:00
Claudio Atzori 47f3d9b757 unit test for GraphHiveImporterJob 2020-04-08 13:24:43 +02:00
Sandro La Bruzzo ba9f07a6fe fixed wrong test 2020-04-08 13:18:20 +02:00
Miriam Baglioni 540da4ab61 new busuness logic with prepared info before actual job run 2020-04-08 13:04:04 +02:00
Miriam Baglioni 8438702b3d addition in propagation constants 2020-04-08 10:54:01 +02:00
Miriam Baglioni 2afe971816 new implementation for country propagatio 2020-04-08 10:49:09 +02:00
Miriam Baglioni beebbcf66b new config for countrypropagation 2020-04-08 10:31:29 +02:00
Claudio Atzori d74e128aa6 Utility classes moved in dhp-common and dhp-schemas 2020-04-07 11:56:22 +02:00
Claudio Atzori c57cf679ca Merge branch 'provision_dataset' 2020-04-07 08:56:58 +02:00
Claudio Atzori 1a1a026a18 we do expect to find field bestaccessright already defined. No need to add it again 2020-04-07 08:55:33 +02:00
Claudio Atzori fbdd18a96b using dataset based relation preparation procedure 2020-04-07 08:54:39 +02:00
Claudio Atzori 77f59b1b10 dataset based provision WIP 2020-04-06 19:37:27 +02:00
Claudio Atzori 6177cf36fb Merge pull request 'Closes #4: New action manager implementation' (#5) from przemyslaw.jacewicz/dnet-hadoop:przemyslawjacewicz_actionmanager_impl_prototype into master 
Nothing more to add here. Thanks for your contribution!
 2020-04-06 17:35:07 +02:00
Claudio Atzori e355961997 dataset based provision WIP 2020-04-06 17:34:25 +02:00
Claudio Atzori ca345aaad3 dataset based provision WIP 2020-04-06 15:33:31 +02:00
Claudio Atzori c8f4b95464 dataset based provision WIP 2020-04-06 08:59:58 +02:00
Claudio Atzori eb2f5f3198 dataset based provision WIP 2020-04-04 17:41:31 +02:00
Claudio Atzori 3d1b637cab dataset based provision WIP 2020-04-04 14:03:43 +02:00
Claudio Atzori 24b2c9012e dataset based provision WIP 2020-04-02 18:44:09 +02:00
Claudio Atzori daa26acc9d dataset based provision WIP, fixed spark2EventLogDir 2020-04-02 16:15:50 +02:00
Przemysław Jacewicz 7b2a7e2417 [dhp-actionmanager] missing descriptions added and minor naming and formatting fixes 2020-04-02 11:48:40 +02:00
Claudio Atzori 9c7092416a dataset based provision WIP 2020-04-01 19:07:30 +02:00
Przemysław Jacewicz 80cf43b9c8 [dhp-actionmanager] promoting workflow added 2020-04-01 18:51:25 +02:00
Przemysław Jacewicz 5b459bcc47 [dhp-actionmanager] promoting spark job added 2020-04-01 18:49:08 +02:00
Przemysław Jacewicz e21bb89dbd [dhp-actionmanager] partitioning spark job added 2020-04-01 18:41:29 +02:00
Przemysław Jacewicz f9f7350bb9 [dhp-actionmanager] common package added with utility classes supporting hadoop and spark envs 2020-04-01 18:39:26 +02:00
Przemysław Jacewicz ad70c23b2e [dhp-actionmanager] pom updated 2020-04-01 18:36:00 +02:00
Przemysław Jacewicz 4e910a78d4 [dhp-workflows] spark 2 connection properties added 2020-04-01 18:29:26 +02:00
Claudio Atzori 1402eb1fe7 cleanup 2020-04-01 15:38:50 +02:00
Claudio Atzori 7061d07727 ActionSets migration serialize the output as plain text files instead of SequenceFiles 2020-04-01 14:58:22 +02:00
Claudio Atzori adcdd2d05e WIP: reimplementing the adjacency list construction process using spark Datasets 2020-04-01 14:56:57 +02:00