spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
595 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

485 Commits

Author SHA1 Message Date
Spyros Zoupanos a852dd3a0d Making tables visible to Impala 2020-10-09 22:28:45 +03:00
Spyros Zoupanos dd9df4ae58 Modifications to SARC stats 2020-10-09 21:48:52 +03:00
Spyros Zoupanos 8b08f35dfe Corrections for Irus stats after discussing with Dimitris 2020-10-08 19:15:01 +03:00
Spyros Zoupanos 715bbd487d Flags & limiters for Sarc stats 2020-10-08 00:24:58 +03:00
Spyros Zoupanos 2de17e7f32 Corrections for irus stats 2020-10-07 23:28:09 +03:00
Spyros Zoupanos 8da64d8f54 Adding flags and time limits to Irus stats 2020-10-06 23:48:31 +03:00
Spyros Zoupanos 9a1512004f Corrections for LaReferencia ending period 2020-10-05 19:09:31 +03:00
Spyros Zoupanos 915b3287a8 Minor corrections to LaReferencia stats 2020-10-04 20:01:40 +03:00
Spyros Zoupanos 806a450465 Flags for La Referencia logs 2020-10-04 19:16:42 +03:00
Spyros Zoupanos 48d6bf28eb More control parameters and limits on the lareferncia download 2020-10-04 17:03:01 +03:00
Spyros Zoupanos 2b330dd84c Corrections for correct download of OpenAIRE logs - Date limits 2020-10-04 10:19:44 +03:00
Spyros Zoupanos 7b7075cfdd Changes for proper log downloading (limits on starting and ending period) + loggers to STDOUT 2020-10-04 00:24:55 +03:00
Spyros Zoupanos e2748fea95 Download directory automatic deletion & creation 2020-10-03 19:47:57 +03:00
Spyros Zoupanos 2e2e2b8b29 Passing via properties the DB parameters 2020-10-02 19:20:31 +03:00
Spyros Zoupanos 07e750939f Adding correct logger everywhere, cleaning code, removing sysouts 2020-10-02 16:25:21 +03:00
Spyros Zoupanos c6bbe215e1 Code cleaning and re-organisation 2020-10-02 14:45:19 +03:00
Spyros Zoupanos c1257ac7c5 Correct parameter parsing 2020-10-01 23:24:40 +03:00
Spyros Zoupanos dc6114a24e Introducing impala connections and using the correct connection string 2020-09-27 13:19:45 +03:00
Spyros Zoupanos 73656f7f31 More corrections on the portalStats queries 2020-09-26 11:18:03 +03:00
Spyros Zoupanos a497b19b21 More corrections on the portalStats queries 2020-09-26 10:47:08 +03:00
Spyros Zoupanos bc5cf28375 Changing portalStats queries to faster ones 2020-09-23 22:22:43 +03:00
Spyros Zoupanos 69640f5fc4 Finished LaReferencia updateProdTables. WF rewriting finished. Needs optimization 2020-09-21 23:26:55 +03:00
Spyros Zoupanos 1ceb363cb2 lareferencia viewsStats & downloadsStats finished 2020-09-21 23:16:10 +03:00
Spyros Zoupanos 65acece7c4 lareferencia removeDoubleClicks done 2020-09-21 22:27:15 +03:00
Spyros Zoupanos 0369f36776 processlaReferenciaLog finished 2020-09-20 21:16:01 +03:00
Spyros Zoupanos 3c11acde0c More progress on processlaReferenciaLog 2020-09-20 15:07:09 +03:00
Spyros Zoupanos a2d64b4644 Added lareferencialogtmp_json table creation 2020-09-20 14:03:16 +03:00
Spyros Zoupanos 373f4fdbd8 Better organisation of downloaded logs 2020-09-20 12:56:04 +03:00
Spyros Zoupanos 8a39ec44e0 More logging messages - Code needs cleaning. Duplicate code for LaReferencia table creation. Should it at one place. The same for various methods that are used for the JSON downloading in various classes like getJson 2020-09-20 11:27:27 +03:00
Spyros Zoupanos 2b2bac9b28 More progress on LaReFerenciaLogs 2020-09-20 00:59:33 +03:00
Spyros Zoupanos 053588c365 Adding lareferencia initial files 2020-09-20 00:00:59 +03:00
Spyros Zoupanos 2e701c547d Finished Sarc Stats 2020-09-19 23:43:26 +03:00
Spyros Zoupanos b3d51a954a Deleting the not needed commented out code 2020-09-19 22:11:40 +03:00
Spyros Zoupanos ed4e9f46d9 Changes to create properly directories for downloaded files 2020-09-19 22:04:42 +03:00
Spyros Zoupanos 03fb2b9e01 Sarc stats almost finished. Have to look at sarcStats() method - INSERT INTO downloads_stats 2020-09-17 22:19:10 +03:00
Spyros Zoupanos 2ae67cfdba Creation of Sarc JSON tables 2020-09-16 21:51:50 +03:00
Spyros Zoupanos 958fb1a343 Creation of Sarc JSON tables 2020-09-16 21:46:32 +03:00
Spyros Zoupanos 1dcb197f02 Renamings on Sarcs and deletion of empty files (to be double checked) 2020-09-16 21:28:05 +03:00
Spyros Zoupanos 17f2748eb4 Merge branch 'usage-stats-export-wf' of code-repo.d4science.org:spyros/dnet-hadoop into usage-stats-export-wf 2020-09-16 20:34:14 +03:00
Spyros Zoupanos 17acbb7fc6 Schema separation on sarc stats that are downloaded 2020-09-16 20:30:36 +03:00
Spyros Zoupanos 49de94c4b1 Removing prefix c: from json 2020-09-15 22:32:14 +03:00
Spyros Zoupanos 015f6e88df Removing prefix c: from json 2020-09-15 21:41:28 +03:00
Spyros Zoupanos 8bb00add0d Minor changes to print the Sarc keys 2020-09-15 18:08:42 +03:00
Spyros Zoupanos ba33df29b4 Workign on Sarc stats 2020-09-14 20:10:53 +03:00
Spyros Zoupanos 55222a2516 processIrusStats done 2020-09-13 16:01:08 +03:00
Spyros Zoupanos 08a102a76c processIrusStats done 2020-09-13 16:00:40 +03:00
Spyros Zoupanos 3d5904fb41 More proper naming to methods 2020-09-13 15:01:29 +03:00
Spyros Zoupanos 95fee808fd Downloading Irus Reports works correctly. Need to add limit on downloaded files for testing reasons. Now we have breakpoints 2020-09-13 14:51:45 +03:00
Spyros Zoupanos 196946cd6b Moving JSON Serde jar addition to a better place 2020-09-13 13:01:39 +03:00
Spyros Zoupanos f8e91cdc5c processLogs.updateProdTables. I need feedback for processLogs.portalStats to see wy they never end 2020-09-13 12:23:03 +03:00