Claudio Atzori
7f244d9a7a
code formatting
2023-10-02 11:04:36 +02:00
Giambattista Bloisi
e239b81740
Fix defect #8997 : GenerateEventsJob is generating huge amounts of logs because broker entity similarity calculation consistently failed
2023-10-02 11:04:18 +02:00
Giambattista Bloisi
e64c2854a3
Refactor Dedup process to use Spark Dataframe API and intermediate representation with Row interface
...
JsonPath cache contention fixed by using a ConcurrentHashMap
Blacklist filtering performance improvement
Minor performance improvements when evaluating similarity
Sorting in clustered elements is deterministic (by ordering and identity field, instead of ordering field only)
2023-07-24 15:36:24 +02:00
Michele Artini
554df257ab
null values in date range conditions
2023-02-13 16:15:32 +01:00
Claudio Atzori
0a58bc7ba7
[broker] prevent NPEs
2023-01-11 14:44:14 +01:00
Claudio Atzori
04cb96001c
[broker] d40e20f437
adapted to the beta graph model
2023-01-11 10:10:12 +01:00
Michele Artini
91b845f611
Considering instance pids and alteternative identifiers
2023-01-11 09:58:54 +01:00
Claudio Atzori
27a91841e7
WIP: cleaning of subjects
2022-08-04 11:39:39 +02:00
Michele Artini
c4fce785ab
fixed a compilation problem of a unit test
2021-10-19 16:18:26 +02:00
Claudio Atzori
172363e7f1
[broker] integrating PR#147, notification record creation phase separated from indexing on ES
2021-10-19 15:56:27 +02:00
Claudio Atzori
bdffa86c2f
undo last commit
2021-10-19 15:39:38 +02:00
Michele Artini
210d6c0e6d
generateNotificationsJob and indexNotificationsJob
2021-10-05 13:57:46 +02:00
Michele Artini
69008e20c2
log and tests
2021-10-05 11:58:20 +02:00
Michele Artini
8bbaa17335
reimplemented of conditions cache as a non static variable
2021-10-05 09:20:37 +02:00
Michele Artini
31a6ad1d79
optimization of verifySubsriptions()
2021-10-04 12:01:56 +02:00
Claudio Atzori
ec94cc9b93
IndexNotificationsJob test: persist contents on HDFS instead of passing them to ES
2021-10-01 09:41:27 +02:00
Claudio Atzori
2ee21da43b
suggestions from SonarLint
2021-08-11 12:13:22 +02:00
Claudio Atzori
908f57a475
code formatting
2021-07-29 10:49:39 +02:00
Claudio Atzori
4c5a71ba2f
[broker] updated relation descriptors, making use of constant values
2021-07-28 17:11:18 +02:00
Claudio Atzori
ae2b47b29d
[broker] added coalesce(1) on the stats dataset before storing it on postgres
2021-07-09 15:47:51 +02:00
Claudio Atzori
5e4b91d9ef
more pervasive use of constants from ModelConstants, especially for ORCID
2021-05-26 18:20:23 +02:00
Claudio Atzori
23b8883ab1
applied intellij code cleanup
2021-05-14 10:58:12 +02:00
Claudio Atzori
7941d7be29
WIP: using common definitions from ModelConstants
2021-03-31 18:33:57 +02:00
Michele Artini
b9d90e95b8
Added eventId to ShortEventMessage
2021-01-14 14:32:31 +01:00
Michele Artini
64b0b0bfb3
fixed a bug with invalid subject topic
2021-01-14 14:32:31 +01:00
Michele Artini
e3e0ab1de1
fixed a problem with join
2021-01-14 14:32:31 +01:00
Michele Artini
26a941315a
openaireId
2021-01-14 14:32:31 +01:00
Michele Artini
6f4d1a37f0
ES wf properties
2021-01-14 14:32:31 +01:00
Michele Artini
3c9cbd19f3
whitelist of topics
2021-01-14 14:32:31 +01:00
Michele Artini
467aa77279
workingDir and outputDir
2021-01-14 14:32:31 +01:00
Michele Artini
10f3f7eca7
workingDir and outputDir
2021-01-14 14:32:31 +01:00
Michele Artini
ff41a7b3a4
gzipped output
2021-01-14 14:32:31 +01:00
Claudio Atzori
fba11eef2a
cleanup
2020-12-07 21:53:13 +01:00
Michele Artini
5de8a7276f
wf to partition opendoar events
2020-12-07 14:56:06 +01:00
Michele Artini
ab08d12c46
considering abstract > MIN_LENGTH in ENRICH_MISSING_ABSTRACT
2020-11-19 10:42:10 +01:00
Michele Artini
c96598aaa4
opendoar partition
2020-09-25 09:02:58 +02:00
Michele Artini
9e681609fd
stats to sql file
2020-09-17 15:51:22 +02:00
Michele Artini
51321c2701
partition of events by opedoarId
2020-09-17 11:38:07 +02:00
Michele Artini
9b0c12f5d3
send notifications
2020-09-11 12:06:16 +02:00
Michele Artini
028613b751
remove old notifications
2020-09-09 15:32:06 +02:00
Michele Artini
a597a218ab
* forall topics
2020-09-08 16:39:40 +02:00
Michele Artini
bb459caf69
support for all topic subscriptions
2020-08-27 11:01:21 +02:00
Michele Artini
82ed8edafd
notification indexing
2020-08-26 15:10:48 +02:00
Michele Artini
da470422d3
deleting events
2020-08-21 14:52:48 +02:00
Michele Artini
6e60bf026a
indexing only a subset of eventsa
2020-08-19 12:39:22 +02:00
Michele Artini
8ba94833bd
added an es prop
2020-07-29 14:16:08 +02:00
Michele Artini
346a1d2b5a
update eventId generator
2020-07-18 09:40:36 +02:00
Michele Artini
262c29463e
relations with multiple datasources
2020-07-15 09:18:40 +02:00
Michele Artini
e1ae964bc4
stats
2020-07-10 16:12:08 +02:00
Michele Artini
2d742a84ae
DedupConfig as json file
2020-07-09 12:53:46 +02:00