spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
583 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

583 Commits

Author SHA1 Message Date
Spyros Zoupanos e2748fea95 Download directory automatic deletion & creation 2020-10-03 19:47:57 +03:00
Spyros Zoupanos 2e2e2b8b29 Passing via properties the DB parameters 2020-10-02 19:20:31 +03:00
Spyros Zoupanos 07e750939f Adding correct logger everywhere, cleaning code, removing sysouts 2020-10-02 16:25:21 +03:00
Spyros Zoupanos c6bbe215e1 Code cleaning and re-organisation 2020-10-02 14:45:19 +03:00
Spyros Zoupanos c1257ac7c5 Correct parameter parsing 2020-10-01 23:24:40 +03:00
Spyros Zoupanos dc6114a24e Introducing impala connections and using the correct connection string 2020-09-27 13:19:45 +03:00
Spyros Zoupanos 73656f7f31 More corrections on the portalStats queries 2020-09-26 11:18:03 +03:00
Spyros Zoupanos a497b19b21 More corrections on the portalStats queries 2020-09-26 10:47:08 +03:00
Spyros Zoupanos bc5cf28375 Changing portalStats queries to faster ones 2020-09-23 22:22:43 +03:00
Spyros Zoupanos 69640f5fc4 Finished LaReferencia updateProdTables. WF rewriting finished. Needs optimization 2020-09-21 23:26:55 +03:00
Spyros Zoupanos 1ceb363cb2 lareferencia viewsStats & downloadsStats finished 2020-09-21 23:16:10 +03:00
Spyros Zoupanos 65acece7c4 lareferencia removeDoubleClicks done 2020-09-21 22:27:15 +03:00
Spyros Zoupanos 0369f36776 processlaReferenciaLog finished 2020-09-20 21:16:01 +03:00
Spyros Zoupanos 3c11acde0c More progress on processlaReferenciaLog 2020-09-20 15:07:09 +03:00
Spyros Zoupanos a2d64b4644 Added lareferencialogtmp_json table creation 2020-09-20 14:03:16 +03:00
Spyros Zoupanos 373f4fdbd8 Better organisation of downloaded logs 2020-09-20 12:56:04 +03:00
Spyros Zoupanos 8a39ec44e0 More logging messages - Code needs cleaning. Duplicate code for LaReferencia table creation. Should it at one place. The same for various methods that are used for the JSON downloading in various classes like getJson 2020-09-20 11:27:27 +03:00
Spyros Zoupanos 2b2bac9b28 More progress on LaReFerenciaLogs 2020-09-20 00:59:33 +03:00
Spyros Zoupanos 053588c365 Adding lareferencia initial files 2020-09-20 00:00:59 +03:00
Spyros Zoupanos 2e701c547d Finished Sarc Stats 2020-09-19 23:43:26 +03:00
Spyros Zoupanos b3d51a954a Deleting the not needed commented out code 2020-09-19 22:11:40 +03:00
Spyros Zoupanos ed4e9f46d9 Changes to create properly directories for downloaded files 2020-09-19 22:04:42 +03:00
Spyros Zoupanos 03fb2b9e01 Sarc stats almost finished. Have to look at sarcStats() method - INSERT INTO downloads_stats 2020-09-17 22:19:10 +03:00
Spyros Zoupanos 2ae67cfdba Creation of Sarc JSON tables 2020-09-16 21:51:50 +03:00
Spyros Zoupanos 958fb1a343 Creation of Sarc JSON tables 2020-09-16 21:46:32 +03:00
Spyros Zoupanos 1dcb197f02 Renamings on Sarcs and deletion of empty files (to be double checked) 2020-09-16 21:28:05 +03:00
Spyros Zoupanos 17f2748eb4 Merge branch 'usage-stats-export-wf' of code-repo.d4science.org:spyros/dnet-hadoop into usage-stats-export-wf 2020-09-16 20:34:14 +03:00
Spyros Zoupanos 17acbb7fc6 Schema separation on sarc stats that are downloaded 2020-09-16 20:30:36 +03:00
Spyros Zoupanos 49de94c4b1 Removing prefix c: from json 2020-09-15 22:32:14 +03:00
Spyros Zoupanos 015f6e88df Removing prefix c: from json 2020-09-15 21:41:28 +03:00
Spyros Zoupanos 8bb00add0d Minor changes to print the Sarc keys 2020-09-15 18:08:42 +03:00
Spyros Zoupanos ba33df29b4 Workign on Sarc stats 2020-09-14 20:10:53 +03:00
Spyros Zoupanos 55222a2516 processIrusStats done 2020-09-13 16:01:08 +03:00
Spyros Zoupanos 08a102a76c processIrusStats done 2020-09-13 16:00:40 +03:00
Spyros Zoupanos 3d5904fb41 More proper naming to methods 2020-09-13 15:01:29 +03:00
Spyros Zoupanos 95fee808fd Downloading Irus Reports works correctly. Need to add limit on downloaded files for testing reasons. Now we have breakpoints 2020-09-13 14:51:45 +03:00
Spyros Zoupanos 196946cd6b Moving JSON Serde jar addition to a better place 2020-09-13 13:01:39 +03:00
Spyros Zoupanos f8e91cdc5c processLogs.updateProdTables. I need feedback for processLogs.portalStats to see wy they never end 2020-09-13 12:23:03 +03:00
Spyros Zoupanos 9caac3e3e3 portalStats finished - Needs testing. Working on updateProdTables 2020-09-12 21:24:31 +03:00
Spyros Zoupanos 8ddf1dcc15 processPortalLog finished 2020-09-12 20:13:33 +03:00
Spyros Zoupanos 968d53f119 Finished downloadsStats 2020-09-11 20:10:37 +03:00
Spyros Zoupanos f78b5d3f86 More progress on viewsStats 2020-09-10 22:37:48 +03:00
Spyros Zoupanos 2d2d1b9694 More progress on viewsStats 2020-09-10 22:27:19 +03:00
Spyros Zoupanos 1d9f8f79a8 Finished cleanOAI 2020-09-09 21:59:04 +03:00
Spyros Zoupanos 398f1f6f15 More progress. Cleaning view double clicks 2020-09-07 21:57:45 +03:00
Spyros Zoupanos 81102dd791 Removing not needed jar by reflection 2020-09-07 20:54:47 +03:00
Spyros Zoupanos 719f9e3cd9 Adding systout messages (should be transformed to log messages) 2020-09-07 20:44:01 +03:00
Spyros Zoupanos e2c70f64ed More progress on loading JSON Serde jar 2020-09-07 00:01:05 +03:00
Spyros Zoupanos 5af2abbea5 Moving variable declarations to a more appropriate place, adding drop table code 2020-09-04 19:49:07 +03:00
Spyros Zoupanos cf7b9c6db3 More progress on adding queries to the code. Initial database and table creation seems OK. Downloading logs from available piwik_ids 2020-09-02 21:02:56 +03:00