andreas.czerniak
/
BrBETA_dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
490 Commits 23 Branches 0 Tags 90 MiB
Commit Graph

85 Commits

Author SHA1 Message Date
Miriam Baglioni ca2b40952e minor changes 2020-04-14 16:48:02 +02:00
Miriam Baglioni 61d39e659e parameters for the project2result propagation phase 2020-04-14 16:47:39 +02:00
Miriam Baglioni 92f19fa0a0 parameters for the project2result preparation phase 2020-04-14 16:46:57 +02:00
Miriam Baglioni cadab9b81d new implementation for result to project propagation. Use the prepared info in propagation 2020-04-14 16:46:07 +02:00
Miriam Baglioni ceb1f299bf minor changes 2020-04-14 16:45:12 +02:00
Miriam Baglioni e0038bde5b Support class to serialize/deserialize the association project, set of linked results 2020-04-14 15:32:12 +02:00
Miriam Baglioni c0bebb7c35 code to compute the prepared information used in the actual propagation step. This step will produce who files: one with potential updates (association between projects and a list of results), the other already linked entities (association between projects and the list of results already linked to them) 2020-04-14 15:31:26 +02:00
Miriam Baglioni f47ee5b78e directory where to store the prepared infor before actual propagation will take place 2020-04-14 15:29:21 +02:00
Miriam Baglioni 36cc9516d8 the starting relation set for testing 2020-04-14 15:28:34 +02:00
Miriam Baglioni 4b01dc60e6 test unit for result to project propagation 2020-04-14 15:28:00 +02:00
Miriam Baglioni 8f12292daa changed the way to save the results on filesystem 2020-04-11 16:47:34 +02:00
Miriam Baglioni 87f802821e new workflow for country propagation: it is composed of the preparation step and in the propagation. The propagation part runs in parallel on the result types 2020-04-11 16:40:22 +02:00
Miriam Baglioni a562080b0b parameters to be used in the prepared Job and in the actual country propagation job 2020-04-11 16:39:17 +02:00
Miriam Baglioni 1251ad4455 removed unuseful class 2020-04-11 16:38:13 +02:00
Miriam Baglioni aef9b3aa90 new parametric implementation of country propagation. Exploits information compute before and broadcasts it to each executor 2020-04-11 16:36:59 +02:00
Miriam Baglioni a2d833d5dd step of data preparation before actual country propagation will take palce 2020-04-11 16:36:03 +02:00
Miriam Baglioni 6897c920a2 classes in support of new implementation of country propagation 2020-04-11 16:35:26 +02:00
Miriam Baglioni 85766a02d8 added dependency to use hive on local machine 2020-04-11 16:34:22 +02:00
Miriam Baglioni 79b8ea4fed prepared information to be used in actual country propagation. Subset of info 2020-04-11 16:29:41 +02:00
Miriam Baglioni 1822476613 Test for country propagation 2020-04-11 16:28:09 +02:00
Miriam Baglioni 7783b09c5b new implementation for result to project propagation. Prepare some info to be used in propagation 2020-04-11 16:26:23 +02:00
Miriam Baglioni 90469789b9 two new classes fro new implementation of project to result propagation 2020-04-09 13:29:01 +02:00
Miriam Baglioni 627ad58a8b new wf definition 2020-04-09 11:33:19 +02:00
Miriam Baglioni 9c63c4840d new workflow and parameters for country propagation 2020-04-08 19:13:42 +02:00
Miriam Baglioni a2d309545b new parametrized implementation for country propagation 2020-04-08 19:12:59 +02:00
Miriam Baglioni 6dfdba9ef7 new parametrized implementation for country propagation 2020-04-08 18:14:37 +02:00
Miriam Baglioni 03f7cb6402 new parametrized implementation for country propagation 2020-04-08 18:08:41 +02:00
Miriam Baglioni fcfef4632f input parameters for country propagation preparation job 2020-04-08 18:07:18 +02:00
Miriam Baglioni 540da4ab61 new busuness logic with prepared info before actual job run 2020-04-08 13:04:04 +02:00
Miriam Baglioni 8438702b3d addition in propagation constants 2020-04-08 10:54:01 +02:00
Miriam Baglioni 2afe971816 new implementation for country propagatio 2020-04-08 10:49:09 +02:00
Miriam Baglioni beebbcf66b new config for countrypropagation 2020-04-08 10:31:29 +02:00
Miriam Baglioni dd011f4a95 to make them visible to Claudio 2020-03-30 10:55:47 +02:00
Miriam Baglioni b1af90a45f to make it visible to Claudio 2020-03-30 10:50:03 +02:00
Miriam Baglioni 19d7f8b51d decommented execution for some of the result type for testing purposes 2020-03-24 16:49:46 +01:00
Miriam Baglioni ad24c8478f added missing parameter 2020-03-24 16:19:59 +01:00
Miriam Baglioni 46094a3eec bug fixing for implementation with dataset 2020-03-24 16:19:36 +01:00
Miriam Baglioni ad712f2d79 added the needed variables in the config and read the variables in the workflow 2020-03-23 17:11:36 +01:00
Miriam Baglioni f1e9fe9752 changed implementation using dataset and query on hive 2020-03-23 17:11:00 +01:00
Miriam Baglioni f09cd1e911 removed unuseful variable in the configuration 2020-03-23 17:10:14 +01:00
Miriam Baglioni 9418e3d4fa read dataset from files instead of using hive tables 2020-03-23 17:09:27 +01:00
Miriam Baglioni a7bf037306 remove unused class 2020-03-23 14:36:43 +01:00
Miriam Baglioni 8ab8b6b0bf minor 2020-03-23 14:35:23 +01:00
Miriam Baglioni 30d58fd98c change the configuration of the workflow 2020-03-23 14:32:49 +01:00
Miriam Baglioni a440152b46 refactoring 2020-03-23 14:30:56 +01:00
Miriam Baglioni 47561f3597 changed the implementation from rdd to dataset got from sql queries (on hive) 2020-03-23 11:58:32 +01:00
Miriam Baglioni 67ea3cf3ed changed the way to read the file with info on resource or relation. From sequenceFile to textFile 2020-03-17 16:32:05 +01:00
Miriam Baglioni b4652d018c moved the creation of new dir to common class. 2020-03-17 16:31:24 +01:00
Miriam Baglioni e80f80ca93 properties and workflow for new propagation 2020-03-02 17:03:31 +01:00
Miriam Baglioni 50080c1b3c changed the implementation of addAll method. Before adding all the items in a collection, we check if the accumulator set is not empty 2020-03-02 16:41:37 +01:00