antonis.lempesis
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 1
Code Issues Pull Requests Projects Releases Wiki Activity
1,759 Commits 18 Branches 0 Tags 128 MiB
Commit Graph

1759 Commits

Author SHA1 Message Date
Miriam Baglioni d0ff2a56fb - 2020-07-29 17:06:53 +02:00
Miriam Baglioni b96dedb56b changed due to changes in the model 2020-07-29 17:05:31 +02:00
Miriam Baglioni 6d0f08277b classes to implement the dump of the whole graph. 2020-07-29 17:03:19 +02:00
Miriam Baglioni 8d4327b292 input parameters and workflow definition for the dump of the whole graph 2020-07-29 17:00:34 +02:00
Miriam Baglioni b5f995ab12 refactoring 2020-07-29 16:59:48 +02:00
Miriam Baglioni f7a87cc447 added new constants value 2020-07-29 16:58:40 +02:00
Miriam Baglioni 6b63668d3f - 2020-07-29 16:57:45 +02:00
Miriam Baglioni dac9f7cc10 refactorin 2020-07-29 16:56:46 +02:00
Miriam Baglioni 770b71b14a Added new model class to dumop Datasources 2020-07-29 16:55:39 +02:00
Miriam Baglioni c760d16936 removed the collected from element from the entities 2020-07-29 16:55:08 +02:00
Miriam Baglioni 6a499c6b7a added hashCode to relation to avoid to produce multiple identical relations 2020-07-29 16:54:16 +02:00
Miriam Baglioni b71d12cf26 refactoring 2020-07-29 16:52:44 +02:00
Miriam Baglioni a8d65b68cb changed to delete the part to check if it was a test or a real execution 2020-07-29 16:47:57 +02:00
Miriam Baglioni 3ec2392904 Added new class to move the place the split is effectively run 2020-07-29 16:46:50 +02:00
Michele Artini 8ba94833bd added an es prop 2020-07-29 14:16:08 +02:00
Miriam Baglioni 178c2729a7 changed the path to reach the java class to be executed 2020-07-29 12:29:51 +02:00
Miriam Baglioni 437ac12139 removed unused parameter 2020-07-29 12:28:16 +02:00
Claudio Atzori 6f11c0496e fixed typo in module name dhp-worfklow-profiles -> dhp-workflow-profiles 2020-07-28 15:01:58 +02:00
Claudio Atzori f680eb3e12 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-07-28 14:10:56 +02:00
Claudio Atzori 985b360c31 fixed typo in module name dhp-worfklow-profiles -> dhp-workflow-profiles 2020-07-28 14:10:52 +02:00
Claudio Atzori 7fc27bfdd1 Merge pull request 'islookup_timeout' (#30) from islookup_timeout into master 
Thanks, Michele!
 2020-07-28 13:53:12 +02:00
Michele Artini 3acd632123 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-07-28 12:02:30 +02:00
Michele Artini 35e6e9c064 tests 2020-07-28 12:02:15 +02:00
Claudio Atzori 2c4196ab22 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into islookup_timeout 2020-07-27 17:40:58 +02:00
Claudio Atzori ee832f358e Merge pull request 'stats_wf_extensions_and_corrections' (#28) from spyros/dnet-hadoop:stats_wf_extensions_and_corrections into master 
Thank you Guys! The update workflow will be made available to the beta & production orchestration systems under the HDFS path

```/lib/dnet/oa/graph/stats/oozie_app```
 2020-07-27 16:02:03 +02:00
Antonis Lempesis 4ac8ebe427 correctly calculating the project duration 2020-07-24 19:50:40 +03:00
Antonis Lempesis 18d9464b52 creating shadow db only if it not exists... 2020-07-24 19:50:40 +03:00
Antonis Lempesis e217d496ab added the dest db... 2020-07-24 19:50:40 +03:00
Antonis Lempesis b16bb68b9f added the target db name... 2020-07-24 19:50:40 +03:00
Antonis Lempesis 1ee7eeedf3 added the source db name... 2020-07-24 19:50:40 +03:00
Antonis Lempesis cecbbfa0fc added missing tables and views: contexts, creation_date, funder 2020-07-24 19:50:40 +03:00
Antonis Lempesis 25b7a615f5 moved datasource_sources table creating in the datasource section 2020-07-24 19:50:40 +03:00
Antonis Lempesis a8da4ab9c0 years in projects are now integers 2020-07-24 19:50:40 +03:00
Antonis Lempesis c9cfc165d9 not using impala since the resulting tables are not visible 2020-07-24 19:50:40 +03:00
Antonis Lempesis dd3d6a6e15 compute stats for the used and new impala tables 2020-07-24 19:50:40 +03:00
Antonis Lempesis e6f50de6ef Separated impala from hive steps 2020-07-24 19:50:40 +03:00
Antonis Lempesis de49173420 fixed a typo in queries 2020-07-24 19:50:40 +03:00
antleb 391cf80fb8 Added peer-reviewed, green, gold tables and fields in result. Added shortcuts from result-country 2020-07-24 19:50:40 +03:00
antleb 68389d0125 Corrected the script used by the last step of the wf 2020-07-24 19:50:40 +03:00
antleb ec52141f1a changed refereed type from value to clssname 2020-07-24 19:50:40 +03:00
Spyros Zoupanos 63cd797aba Comment out step 15 to make it work with the new schema of Claudio 2020-07-24 19:50:40 +03:00
Spyros Zoupanos 138c6ddffa Insert statement to datasource table that takes into account the piwik_id of the openAIRE graph 2020-07-24 19:50:40 +03:00
Spyros Zoupanos 3630794cef Fix to consider the relationships that have been 'virtually deleted' for project_results - defect #5607 2020-07-24 19:50:40 +03:00
Spyros Zoupanos 5546f29e63 Corrections on the shadow schema and the impala table stats calculation 2020-07-24 19:50:40 +03:00
Spyros Zoupanos adf8a025d2 Adding more relations (Sources, Licences, Additional) and shadow schema as provided and discussed with Antonis Lempesis 2020-07-24 19:50:40 +03:00
Spyros Zoupanos 657a40536b Corrections by Spyros: Scipt cleanup, corrections and re-arrangement 2020-07-24 19:50:40 +03:00
Giorgos Alexiou 477fa6234d Script re-organisation and adding table invalidations needed for impala 2020-07-24 19:50:40 +03:00
Miriam Baglioni 6c2223d1fc added code to get the openaire id for contexts 2020-07-24 17:30:15 +02:00
Miriam Baglioni afd54c1684 removed not needed upload and refactoring 2020-07-24 17:28:56 +02:00
Miriam Baglioni 7b0569d989 changed to map also the result associated to the whole graph 2020-07-24 17:28:11 +02:00