andreas.czerniak
/
BrBETA_dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
4,640 Commits 23 Branches 0 Tags 90 MiB
Commit Graph

127 Commits

Author SHA1 Message Date
Giambattista Bloisi e64c2854a3 Refactor Dedup process to use Spark Dataframe API and intermediate representation with Row interface 
JsonPath cache contention fixed by using a ConcurrentHashMap
Blacklist filtering performance improvement
Minor performance improvements when evaluating similarity
Sorting in clustered elements is deterministic (by ordering and identity field, instead of ordering field only)
 2023-07-24 15:36:24 +02:00
Michele Artini 554df257ab null values in date range conditions 2023-02-13 16:15:32 +01:00
Claudio Atzori 0a58bc7ba7 [broker] prevent NPEs 2023-01-11 14:44:14 +01:00
Claudio Atzori 04cb96001c [broker] d40e20f437 adapted to the beta graph model 2023-01-11 10:10:12 +01:00
Michele Artini 91b845f611 Considering instance pids and alteternative identifiers 2023-01-11 09:58:54 +01:00
Claudio Atzori 27a91841e7 WIP: cleaning of subjects 2022-08-04 11:39:39 +02:00
Claudio Atzori 1de881b796 resolved conflicts for #165 2021-11-26 16:15:11 +01:00
Sandro La Bruzzo 0c0d561bc4 added public class into tests to create correct javadoc 2021-11-19 09:54:22 +01:00
Claudio Atzori bd9a43cefd Revert to 4094f2bb9a 2021-11-19 09:20:43 +01:00
Sandro La Bruzzo 9c82d670b8 make class public in order to create javadoc 2021-11-17 12:31:02 +01:00
Michele Artini c4fce785ab fixed a compilation problem of a unit test 2021-10-19 16:18:26 +02:00
Claudio Atzori 172363e7f1 [broker] integrating PR#147, notification record creation phase separated from indexing on ES 2021-10-19 15:56:27 +02:00
Claudio Atzori bdffa86c2f undo last commit 2021-10-19 15:39:38 +02:00
Michele Artini d6e1f22408 max numbers of workers for indexing 2021-10-05 15:09:18 +02:00
Michele Artini 210d6c0e6d generateNotificationsJob and indexNotificationsJob 2021-10-05 13:57:46 +02:00
Michele Artini 69008e20c2 log and tests 2021-10-05 11:58:20 +02:00
Michele Artini 8bbaa17335 reimplemented of conditions cache as a non static variable 2021-10-05 09:20:37 +02:00
Michele Artini 0a9ef34b56 test 2021-10-04 15:46:12 +02:00
Michele Artini 31a6ad1d79 optimization of verifySubsriptions() 2021-10-04 12:01:56 +02:00
Claudio Atzori b01cd521b0 removed configuration specifying the limit to 8 for spark.dynamicAllocation.maxExecutors 2021-10-01 11:26:33 +02:00
Claudio Atzori ec94cc9b93 IndexNotificationsJob test: persist contents on HDFS instead of passing them to ES 2021-10-01 09:41:27 +02:00
Claudio Atzori 2ee21da43b suggestions from SonarLint 2021-08-11 12:13:22 +02:00
Claudio Atzori 908f57a475 code formatting 2021-07-29 10:49:39 +02:00
Claudio Atzori 4c5a71ba2f [broker] updated relation descriptors, making use of constant values 2021-07-28 17:11:18 +02:00
Claudio Atzori ae2b47b29d [broker] added coalesce(1) on the stats dataset before storing it on postgres 2021-07-09 15:47:51 +02:00
Claudio Atzori 5e4b91d9ef more pervasive use of constants from ModelConstants, especially for ORCID 2021-05-26 18:20:23 +02:00
Claudio Atzori 23b8883ab1 applied intellij code cleanup 2021-05-14 10:58:12 +02:00
Claudio Atzori 7941d7be29 WIP: using common definitions from ModelConstants 2021-03-31 18:33:57 +02:00
Michele Artini 83d815d0bc only stats 2021-02-11 10:57:23 +01:00
Michele Artini 8c1600398a added resumeFrom parameter 2021-02-11 10:54:16 +01:00
Michele Artini 7249cceb53 switch of 2 nodes 2021-02-11 09:27:08 +01:00
Michele Artini cfbcdc95bc fixed a wf param 2021-01-14 14:45:23 +01:00
Michele Artini 69ba3203c0 fixed a conflict 2021-01-14 14:43:25 +01:00
Michele Artini b230d44411 fixed conflict 2021-01-14 14:32:31 +01:00
Michele Artini b9d90e95b8 Added eventId to ShortEventMessage 2021-01-14 14:32:31 +01:00
Michele Artini 64b0b0bfb3 fixed a bug with invalid subject topic 2021-01-14 14:32:31 +01:00
Michele Artini e3e0ab1de1 fixed a problem with join 2021-01-14 14:32:31 +01:00
Michele Artini 26a941315a openaireId 2021-01-14 14:32:31 +01:00
Michele Artini 6f4d1a37f0 ES wf properties 2021-01-14 14:32:31 +01:00
Michele Artini 1391341d06 mkdir of output dir 2021-01-14 14:32:31 +01:00
Michele Artini 3c9cbd19f3 whitelist of topics 2021-01-14 14:32:31 +01:00
Michele Artini 467aa77279 workingDir and outputDir 2021-01-14 14:32:31 +01:00
Michele Artini 10f3f7eca7 workingDir and outputDir 2021-01-14 14:32:31 +01:00
Michele Artini ff41a7b3a4 gzipped output 2021-01-14 14:32:31 +01:00
Claudio Atzori 80cf55ef2e [Broker] fixed partitionEventsByOpendoarIds workflow parameter names 2021-01-13 16:24:30 +01:00
Michele Artini f9dc1e45fd fixed a bug with invalid subject topic 2020-12-15 14:54:11 +01:00
Michele Artini 12fa5d122a fixed a problem with join 2020-12-15 08:30:26 +01:00
Michele Artini 3e19cf7b4a openaireId 2020-12-14 15:24:33 +01:00
Michele Artini a203aee32a ES wf properties 2020-12-14 12:02:33 +01:00
Michele Artini d03756c962 mkdir of output dir 2020-12-14 11:11:41 +01:00