spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
580 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

34 Commits

Author SHA1 Message Date
Spyros Zoupanos c6bbe215e1 Code cleaning and re-organisation 2020-10-02 14:45:19 +03:00
Spyros Zoupanos c1257ac7c5 Correct parameter parsing 2020-10-01 23:24:40 +03:00
Spyros Zoupanos dc6114a24e Introducing impala connections and using the correct connection string 2020-09-27 13:19:45 +03:00
Spyros Zoupanos bc5cf28375 Changing portalStats queries to faster ones 2020-09-23 22:22:43 +03:00
Spyros Zoupanos a2d64b4644 Added lareferencialogtmp_json table creation 2020-09-20 14:03:16 +03:00
Spyros Zoupanos 373f4fdbd8 Better organisation of downloaded logs 2020-09-20 12:56:04 +03:00
Spyros Zoupanos 8a39ec44e0 More logging messages - Code needs cleaning. Duplicate code for LaReferencia table creation. Should it at one place. The same for various methods that are used for the JSON downloading in various classes like getJson 2020-09-20 11:27:27 +03:00
Spyros Zoupanos 2b2bac9b28 More progress on LaReFerenciaLogs 2020-09-20 00:59:33 +03:00
Spyros Zoupanos 053588c365 Adding lareferencia initial files 2020-09-20 00:00:59 +03:00
Spyros Zoupanos 2e701c547d Finished Sarc Stats 2020-09-19 23:43:26 +03:00
Spyros Zoupanos 03fb2b9e01 Sarc stats almost finished. Have to look at sarcStats() method - INSERT INTO downloads_stats 2020-09-17 22:19:10 +03:00
Spyros Zoupanos 958fb1a343 Creation of Sarc JSON tables 2020-09-16 21:46:32 +03:00
Spyros Zoupanos 1dcb197f02 Renamings on Sarcs and deletion of empty files (to be double checked) 2020-09-16 21:28:05 +03:00
Spyros Zoupanos 17acbb7fc6 Schema separation on sarc stats that are downloaded 2020-09-16 20:30:36 +03:00
Spyros Zoupanos ba33df29b4 Workign on Sarc stats 2020-09-14 20:10:53 +03:00
Spyros Zoupanos 08a102a76c processIrusStats done 2020-09-13 16:00:40 +03:00
Spyros Zoupanos 3d5904fb41 More proper naming to methods 2020-09-13 15:01:29 +03:00
Spyros Zoupanos 95fee808fd Downloading Irus Reports works correctly. Need to add limit on downloaded files for testing reasons. Now we have breakpoints 2020-09-13 14:51:45 +03:00
Spyros Zoupanos 196946cd6b Moving JSON Serde jar addition to a better place 2020-09-13 13:01:39 +03:00
Spyros Zoupanos f8e91cdc5c processLogs.updateProdTables. I need feedback for processLogs.portalStats to see wy they never end 2020-09-13 12:23:03 +03:00
Spyros Zoupanos 8ddf1dcc15 processPortalLog finished 2020-09-12 20:13:33 +03:00
Spyros Zoupanos 81102dd791 Removing not needed jar by reflection 2020-09-07 20:54:47 +03:00
Spyros Zoupanos 719f9e3cd9 Adding systout messages (should be transformed to log messages) 2020-09-07 20:44:01 +03:00
Spyros Zoupanos e2c70f64ed More progress on loading JSON Serde jar 2020-09-07 00:01:05 +03:00
Spyros Zoupanos 5af2abbea5 Moving variable declarations to a more appropriate place, adding drop table code 2020-09-04 19:49:07 +03:00
Spyros Zoupanos cf7b9c6db3 More progress on adding queries to the code. Initial database and table creation seems OK. Downloading logs from available piwik_ids 2020-09-02 21:02:56 +03:00
Spyros Zoupanos f3dda9858c More progress - Adding queries to code 2020-08-31 23:19:15 +03:00
Spyros Zoupanos 8db9a7ccdc Changes to download Sarc stats 2020-07-25 13:17:47 +03:00
Spyros Zoupanos c035fa7648 Changes to download Irus Stats 2020-07-22 19:22:04 +03:00
Spyros Zoupanos b213da51c4 Modifying JSON saving procedure to make the files usable by HIVE JsonSerDe 2020-05-21 21:49:33 +03:00
Spyros Zoupanos bf820a98b4 Removing the not needed download code that ignores SSL certificates and uses username/password for authentication. Repository ids are provided manually for the moment until the Hive stats DB provides the correct piwik_id 2020-05-19 18:45:28 +03:00
Spyros Zoupanos 66c7ddfc5e More progress on SQL statements and parameters 2020-05-14 22:27:18 +03:00
Spyros Zoupanos 98ba2d0282 The workflow starts 2020-05-12 20:38:31 +03:00
Spyros Zoupanos c0b509abfb Simple java action added. 
Simple java connection to hive db + basic statements added
 2020-05-09 15:51:22 +03:00