spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
589 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

15 Commits

Author SHA1 Message Date
Spyros Zoupanos 9a1512004f Corrections for LaReferencia ending period 2020-10-05 19:09:31 +03:00
Spyros Zoupanos 48d6bf28eb More control parameters and limits on the lareferncia download 2020-10-04 17:03:01 +03:00
Spyros Zoupanos 2b330dd84c Corrections for correct download of OpenAIRE logs - Date limits 2020-10-04 10:19:44 +03:00
Spyros Zoupanos 7b7075cfdd Changes for proper log downloading (limits on starting and ending period) + loggers to STDOUT 2020-10-04 00:24:55 +03:00
Spyros Zoupanos 07e750939f Adding correct logger everywhere, cleaning code, removing sysouts 2020-10-02 16:25:21 +03:00
Spyros Zoupanos dc6114a24e Introducing impala connections and using the correct connection string 2020-09-27 13:19:45 +03:00
Spyros Zoupanos 5af2abbea5 Moving variable declarations to a more appropriate place, adding drop table code 2020-09-04 19:49:07 +03:00
Spyros Zoupanos cf7b9c6db3 More progress on adding queries to the code. Initial database and table creation seems OK. Downloading logs from available piwik_ids 2020-09-02 21:02:56 +03:00
Spyros Zoupanos 637e61bb0f Getting the right piwik_ids from (graph) stats db 2020-09-01 22:06:16 +03:00
Spyros Zoupanos b213da51c4 Modifying JSON saving procedure to make the files usable by HIVE JsonSerDe 2020-05-21 21:49:33 +03:00
Spyros Zoupanos bf820a98b4 Removing the not needed download code that ignores SSL certificates and uses username/password for authentication. Repository ids are provided manually for the moment until the Hive stats DB provides the correct piwik_id 2020-05-19 18:45:28 +03:00
Spyros Zoupanos 9cdea87c7a More progress on download jsons. All certificates are ignored & authentication is done with username & pass 2020-05-16 13:16:16 +03:00
Spyros Zoupanos 66c7ddfc5e More progress on SQL statements and parameters 2020-05-14 22:27:18 +03:00
Spyros Zoupanos 98ba2d0282 The workflow starts 2020-05-12 20:38:31 +03:00
Spyros Zoupanos c0b509abfb Simple java action added. 
Simple java connection to hive db + basic statements added
 2020-05-09 15:51:22 +03:00