Commit Graph

37 Commits

Author SHA1 Message Date
Michele Artini d6e1f22408 max numbers of workers for indexing 2021-10-05 15:09:18 +02:00
Michele Artini 210d6c0e6d generateNotificationsJob and indexNotificationsJob 2021-10-05 13:57:46 +02:00
Claudio Atzori b01cd521b0 removed configuration specifying the limit to 8 for spark.dynamicAllocation.maxExecutors 2021-10-01 11:26:33 +02:00
Michele Artini 83d815d0bc only stats 2021-02-11 10:57:23 +01:00
Michele Artini 8c1600398a added resumeFrom parameter 2021-02-11 10:54:16 +01:00
Michele Artini 7249cceb53 switch of 2 nodes 2021-02-11 09:27:08 +01:00
Claudio Atzori 80cf55ef2e [Broker] fixed partitionEventsByOpendoarIds workflow parameter names 2021-01-13 16:24:30 +01:00
Michele Artini a203aee32a ES wf properties 2020-12-14 12:02:33 +01:00
Michele Artini d03756c962 mkdir of output dir 2020-12-14 11:11:41 +01:00
Michele Artini 399548f221 whitelist of topics 2020-12-14 11:03:55 +01:00
Michele Artini 2e7df07328 workingDir and outputDir 2020-12-10 14:47:22 +01:00
Michele Artini 5de8a7276f wf to partition opendoar events 2020-12-07 14:56:06 +01:00
Claudio Atzori e503271abe fixed notification workflow name 2020-11-19 10:41:38 +01:00
Michele Artini c96598aaa4 opendoar partition 2020-09-25 09:02:58 +02:00
Michele Artini 9e681609fd stats to sql file 2020-09-17 15:51:22 +02:00
Michele Artini 51321c2701 partition of events by opedoarId 2020-09-17 11:38:07 +02:00
Michele Artini 82ed8edafd notification indexing 2020-08-26 15:10:48 +02:00
Michele Artini da470422d3 deleting events 2020-08-21 14:52:48 +02:00
Michele Artini 6e60bf026a indexing only a subset of eventsa 2020-08-19 12:39:22 +02:00
Michele Artini 35e6e9c064 tests 2020-07-28 12:02:15 +02:00
Michele Artini 346a1d2b5a update eventId generator 2020-07-18 09:40:36 +02:00
Michele Artini 262c29463e relations with multiple datasources 2020-07-15 09:18:40 +02:00
Michele Artini e1ae964bc4 stats 2020-07-10 16:12:08 +02:00
Michele Artini 2d742a84ae DedupConfig as json file 2020-07-09 12:53:46 +02:00
Michele Artini efadbdb2bc fixed a bug with duplicated events 2020-07-07 15:37:13 +02:00
Michele Artini b413db0bff white/blacklists 2020-07-02 12:43:03 +02:00
Michele Artini 59a5421c24 indexing, accumulators, limited lists 2020-06-30 16:17:09 +02:00
Michele Artini 6f13673464 accumulators 2020-06-29 16:33:32 +02:00
Michele Artini 77d2a1b1c4 params to choose sql queries for beta or production 2020-06-25 09:28:13 +02:00
Michele Artini 202f6e62ff Splitted join wf 2020-06-24 15:47:06 +02:00
Michele Artini d13e3d3f68 fixed paths 2020-06-23 11:01:42 +02:00
Michele Artini af2f7705fc partial refactoring of some joins 2020-06-23 08:37:35 +02:00
Michele Artini 3ce20c198e reformatting 2020-06-22 12:14:25 +02:00
Michele Artini ed787398b3 refactoring wf 2020-06-22 11:45:14 +02:00
Michele Artini 16c7a18435 refactoring 2020-06-22 08:51:31 +02:00
Michele Artini 9a847b4557 some wf fixing 2020-06-18 13:14:10 +02:00
Michele Artini 30ea1bda88 oozie workflow 2020-06-12 10:42:35 +02:00