spyros
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
606 Commits 13 Branches 0 Tags 63 MiB
Commit Graph

606 Commits

Author SHA1 Message Date
Spyros Zoupanos cfea668999 Duplicate elimination from usage_stats, views_stats and downloads_stats tables 2020-10-25 13:07:05 +02:00
Spyros Zoupanos 6ff5436991 Changes for better connection management. Now we can keep the connection alive all the time or to create a new connection every time that we request the connection object 2020-10-25 11:27:37 +02:00
Spyros Zoupanos 8b591937e1 Small correction for views stats generation 2020-10-23 15:24:33 +03:00
Spyros Zoupanos 6b5b6796b7 Small corrections for problems that Dimitris found and flag for the number of threads 2020-10-20 20:30:26 +03:00
Spyros Zoupanos 1ca74ce830 Better file re-organisation for piwiklogs 2020-10-20 18:05:03 +03:00
Spyros Zoupanos 7fdf994eb6 Multithreaded download for piwiklogs 2020-10-19 22:40:02 +03:00
Spyros Zoupanos 6cc58e2720 Minor corrections to avoid empty replies exceptions 2020-10-15 22:08:36 +03:00
Spyros Zoupanos a1dc930486 More control on directory cleanup 2020-10-15 17:41:36 +03:00
Spyros Zoupanos 8f24a6388e Corrections on final steps 2020-10-10 14:25:20 +03:00
Spyros Zoupanos 8826684130 Corrections for Sarc stats. Heavy queries run on Impala. 2020-10-10 12:06:31 +03:00
Spyros Zoupanos d9cc70d334 Minor corrections to Impala queries 2020-10-10 10:59:05 +03:00
Spyros Zoupanos a852dd3a0d Making tables visible to Impala 2020-10-09 22:28:45 +03:00
Spyros Zoupanos dd9df4ae58 Modifications to SARC stats 2020-10-09 21:48:52 +03:00
Spyros Zoupanos 8b08f35dfe Corrections for Irus stats after discussing with Dimitris 2020-10-08 19:15:01 +03:00
Spyros Zoupanos 715bbd487d Flags & limiters for Sarc stats 2020-10-08 00:24:58 +03:00
Spyros Zoupanos 2de17e7f32 Corrections for irus stats 2020-10-07 23:28:09 +03:00
Spyros Zoupanos 8da64d8f54 Adding flags and time limits to Irus stats 2020-10-06 23:48:31 +03:00
Spyros Zoupanos 9a1512004f Corrections for LaReferencia ending period 2020-10-05 19:09:31 +03:00
Spyros Zoupanos 915b3287a8 Minor corrections to LaReferencia stats 2020-10-04 20:01:40 +03:00
Spyros Zoupanos 806a450465 Flags for La Referencia logs 2020-10-04 19:16:42 +03:00
Spyros Zoupanos 48d6bf28eb More control parameters and limits on the lareferncia download 2020-10-04 17:03:01 +03:00
Spyros Zoupanos 2b330dd84c Corrections for correct download of OpenAIRE logs - Date limits 2020-10-04 10:19:44 +03:00
Spyros Zoupanos 7b7075cfdd Changes for proper log downloading (limits on starting and ending period) + loggers to STDOUT 2020-10-04 00:24:55 +03:00
Spyros Zoupanos e2748fea95 Download directory automatic deletion & creation 2020-10-03 19:47:57 +03:00
Spyros Zoupanos 2e2e2b8b29 Passing via properties the DB parameters 2020-10-02 19:20:31 +03:00
Spyros Zoupanos 07e750939f Adding correct logger everywhere, cleaning code, removing sysouts 2020-10-02 16:25:21 +03:00
Spyros Zoupanos c6bbe215e1 Code cleaning and re-organisation 2020-10-02 14:45:19 +03:00
Spyros Zoupanos c1257ac7c5 Correct parameter parsing 2020-10-01 23:24:40 +03:00
Spyros Zoupanos dc6114a24e Introducing impala connections and using the correct connection string 2020-09-27 13:19:45 +03:00
Spyros Zoupanos 73656f7f31 More corrections on the portalStats queries 2020-09-26 11:18:03 +03:00
Spyros Zoupanos a497b19b21 More corrections on the portalStats queries 2020-09-26 10:47:08 +03:00
Spyros Zoupanos bc5cf28375 Changing portalStats queries to faster ones 2020-09-23 22:22:43 +03:00
Spyros Zoupanos 69640f5fc4 Finished LaReferencia updateProdTables. WF rewriting finished. Needs optimization 2020-09-21 23:26:55 +03:00
Spyros Zoupanos 1ceb363cb2 lareferencia viewsStats & downloadsStats finished 2020-09-21 23:16:10 +03:00
Spyros Zoupanos 65acece7c4 lareferencia removeDoubleClicks done 2020-09-21 22:27:15 +03:00
Spyros Zoupanos 0369f36776 processlaReferenciaLog finished 2020-09-20 21:16:01 +03:00
Spyros Zoupanos 3c11acde0c More progress on processlaReferenciaLog 2020-09-20 15:07:09 +03:00
Spyros Zoupanos a2d64b4644 Added lareferencialogtmp_json table creation 2020-09-20 14:03:16 +03:00
Spyros Zoupanos 373f4fdbd8 Better organisation of downloaded logs 2020-09-20 12:56:04 +03:00
Spyros Zoupanos 8a39ec44e0 More logging messages - Code needs cleaning. Duplicate code for LaReferencia table creation. Should it at one place. The same for various methods that are used for the JSON downloading in various classes like getJson 2020-09-20 11:27:27 +03:00
Spyros Zoupanos 2b2bac9b28 More progress on LaReFerenciaLogs 2020-09-20 00:59:33 +03:00
Spyros Zoupanos 053588c365 Adding lareferencia initial files 2020-09-20 00:00:59 +03:00
Spyros Zoupanos 2e701c547d Finished Sarc Stats 2020-09-19 23:43:26 +03:00
Spyros Zoupanos b3d51a954a Deleting the not needed commented out code 2020-09-19 22:11:40 +03:00
Spyros Zoupanos ed4e9f46d9 Changes to create properly directories for downloaded files 2020-09-19 22:04:42 +03:00
Spyros Zoupanos 03fb2b9e01 Sarc stats almost finished. Have to look at sarcStats() method - INSERT INTO downloads_stats 2020-09-17 22:19:10 +03:00
Spyros Zoupanos 2ae67cfdba Creation of Sarc JSON tables 2020-09-16 21:51:50 +03:00
Spyros Zoupanos 958fb1a343 Creation of Sarc JSON tables 2020-09-16 21:46:32 +03:00
Spyros Zoupanos 1dcb197f02 Renamings on Sarcs and deletion of empty files (to be double checked) 2020-09-16 21:28:05 +03:00
Spyros Zoupanos 17f2748eb4 Merge branch 'usage-stats-export-wf' of code-repo.d4science.org:spyros/dnet-hadoop into usage-stats-export-wf 2020-09-16 20:34:14 +03:00