spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
569 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

459 Commits

Author SHA1 Message Date
Spyros Zoupanos a2d64b4644 Added lareferencialogtmp_json table creation 2020-09-20 14:03:16 +03:00
Spyros Zoupanos 373f4fdbd8 Better organisation of downloaded logs 2020-09-20 12:56:04 +03:00
Spyros Zoupanos 8a39ec44e0 More logging messages - Code needs cleaning. Duplicate code for LaReferencia table creation. Should it at one place. The same for various methods that are used for the JSON downloading in various classes like getJson 2020-09-20 11:27:27 +03:00
Spyros Zoupanos 2b2bac9b28 More progress on LaReFerenciaLogs 2020-09-20 00:59:33 +03:00
Spyros Zoupanos 053588c365 Adding lareferencia initial files 2020-09-20 00:00:59 +03:00
Spyros Zoupanos 2e701c547d Finished Sarc Stats 2020-09-19 23:43:26 +03:00
Spyros Zoupanos b3d51a954a Deleting the not needed commented out code 2020-09-19 22:11:40 +03:00
Spyros Zoupanos ed4e9f46d9 Changes to create properly directories for downloaded files 2020-09-19 22:04:42 +03:00
Spyros Zoupanos 03fb2b9e01 Sarc stats almost finished. Have to look at sarcStats() method - INSERT INTO downloads_stats 2020-09-17 22:19:10 +03:00
Spyros Zoupanos 2ae67cfdba Creation of Sarc JSON tables 2020-09-16 21:51:50 +03:00
Spyros Zoupanos 958fb1a343 Creation of Sarc JSON tables 2020-09-16 21:46:32 +03:00
Spyros Zoupanos 1dcb197f02 Renamings on Sarcs and deletion of empty files (to be double checked) 2020-09-16 21:28:05 +03:00
Spyros Zoupanos 17f2748eb4 Merge branch 'usage-stats-export-wf' of code-repo.d4science.org:spyros/dnet-hadoop into usage-stats-export-wf 2020-09-16 20:34:14 +03:00
Spyros Zoupanos 17acbb7fc6 Schema separation on sarc stats that are downloaded 2020-09-16 20:30:36 +03:00
Spyros Zoupanos 49de94c4b1 Removing prefix c: from json 2020-09-15 22:32:14 +03:00
Spyros Zoupanos 015f6e88df Removing prefix c: from json 2020-09-15 21:41:28 +03:00
Spyros Zoupanos 8bb00add0d Minor changes to print the Sarc keys 2020-09-15 18:08:42 +03:00
Spyros Zoupanos ba33df29b4 Workign on Sarc stats 2020-09-14 20:10:53 +03:00
Spyros Zoupanos 55222a2516 processIrusStats done 2020-09-13 16:01:08 +03:00
Spyros Zoupanos 08a102a76c processIrusStats done 2020-09-13 16:00:40 +03:00
Spyros Zoupanos 3d5904fb41 More proper naming to methods 2020-09-13 15:01:29 +03:00
Spyros Zoupanos 95fee808fd Downloading Irus Reports works correctly. Need to add limit on downloaded files for testing reasons. Now we have breakpoints 2020-09-13 14:51:45 +03:00
Spyros Zoupanos 196946cd6b Moving JSON Serde jar addition to a better place 2020-09-13 13:01:39 +03:00
Spyros Zoupanos f8e91cdc5c processLogs.updateProdTables. I need feedback for processLogs.portalStats to see wy they never end 2020-09-13 12:23:03 +03:00
Spyros Zoupanos 9caac3e3e3 portalStats finished - Needs testing. Working on updateProdTables 2020-09-12 21:24:31 +03:00
Spyros Zoupanos 8ddf1dcc15 processPortalLog finished 2020-09-12 20:13:33 +03:00
Spyros Zoupanos 968d53f119 Finished downloadsStats 2020-09-11 20:10:37 +03:00
Spyros Zoupanos f78b5d3f86 More progress on viewsStats 2020-09-10 22:37:48 +03:00
Spyros Zoupanos 2d2d1b9694 More progress on viewsStats 2020-09-10 22:27:19 +03:00
Spyros Zoupanos 1d9f8f79a8 Finished cleanOAI 2020-09-09 21:59:04 +03:00
Spyros Zoupanos 398f1f6f15 More progress. Cleaning view double clicks 2020-09-07 21:57:45 +03:00
Spyros Zoupanos 81102dd791 Removing not needed jar by reflection 2020-09-07 20:54:47 +03:00
Spyros Zoupanos 719f9e3cd9 Adding systout messages (should be transformed to log messages) 2020-09-07 20:44:01 +03:00
Spyros Zoupanos e2c70f64ed More progress on loading JSON Serde jar 2020-09-07 00:01:05 +03:00
Spyros Zoupanos 5af2abbea5 Moving variable declarations to a more appropriate place, adding drop table code 2020-09-04 19:49:07 +03:00
Spyros Zoupanos cf7b9c6db3 More progress on adding queries to the code. Initial database and table creation seems OK. Downloading logs from available piwik_ids 2020-09-02 21:02:56 +03:00
Spyros Zoupanos 637e61bb0f Getting the right piwik_ids from (graph) stats db 2020-09-01 22:06:16 +03:00
Spyros Zoupanos 293d6accd4 More progress on adding piwiklogtmp to the code 2020-09-01 19:05:38 +03:00
Spyros Zoupanos f3dda9858c More progress - Adding queries to code 2020-08-31 23:19:15 +03:00
Spyros Zoupanos 8db9a7ccdc Changes to download Sarc stats 2020-07-25 13:17:47 +03:00
Spyros Zoupanos c035fa7648 Changes to download Irus Stats 2020-07-22 19:22:04 +03:00
Spyros Zoupanos 4c00343bbd More progress 2020-06-05 20:39:51 +03:00
Spyros Zoupanos b213da51c4 Modifying JSON saving procedure to make the files usable by HIVE JsonSerDe 2020-05-21 21:49:33 +03:00
Spyros Zoupanos bf820a98b4 Removing the not needed download code that ignores SSL certificates and uses username/password for authentication. Repository ids are provided manually for the moment until the Hive stats DB provides the correct piwik_id 2020-05-19 18:45:28 +03:00
Spyros Zoupanos 9cdea87c7a More progress on download jsons. All certificates are ignored & authentication is done with username & pass 2020-05-16 13:16:16 +03:00
Spyros Zoupanos 66c7ddfc5e More progress on SQL statements and parameters 2020-05-14 22:27:18 +03:00
Spyros Zoupanos 98ba2d0282 The workflow starts 2020-05-12 20:38:31 +03:00
Spyros Zoupanos 0b6f302652 Adding also an update example with the appropriate table definition 2020-05-11 19:53:41 +03:00
Spyros Zoupanos c0b509abfb Simple java action added. 
Simple java connection to hive db + basic statements added
 2020-05-09 15:51:22 +03:00
Spyros Zoupanos cabe92d155 Changes to make it compile successfully 2020-05-07 21:46:14 +03:00