broker #78

This PR corrects the following problems: #67 #68 #70 #74 #75

The oozie wf needs to be updated including only the following parameters:

graphInputPath  = /user/michele.artini/broker_input_graph
workingDir      = /user/michele.artini/broker_events_temp
outputDir       = /user/michele.artini/broker_events

esIndexHost              = 10.19.65.51,10.19.65.52,10.19.65.53,10.19.65.54
esEventIndexName         = oa_events_beta
esNotificationsIndexName = oa_notifications_beta
esBatchWriteRetryCount   = 8
esBatchWriteRetryWait    = 60s
esBatchSizeEntries       = 200
esNodesWanOnly           = true


maxIndexedEventsForDsAndTopic = 100
brokerApiBaseUrl = http://10.19.65.36:9380

brokerDbUrl      = jdbc:postgresql://10.19.65.40:5432/oa_broker 
brokerDbUser     = ...
brokerDbPassword = ...

sparkDriverMemory        = 7G
sparkExecutorMemory      = 7G
sparkExecutorCores       = 6

datasourceIdWhitelist    = 10|openaire____::9ecafa3655143cbc4bc75853035cd432,10|opendoar____::dc6e224a8d74ce03bf301152d6e33e97,10|openaire____::09da65eaaa6deac2f785df1e0ae95a06,10|openaire____::3db634fc5446f389d0b826ea400a5da6,10|openaire____::5a38cb462ac487bf26bdb86009fe3e74,10|openaire____::3c29379cc184f66861e858bc7aa9615b,10|openaire____::4657147e48a1f32637bfe3743bce76c6,10|openaire____::c3267ea1c3f378c456209b6df241624e,10|opendoar____::358aee4cc897452c00244351e4d91f69,10|re3data_____::7b0ad08687b2c960d5aeef06f811d5e6,10|opendoar____::798ed7d4ee7138d49b8828958048130a,10|opendoar____::6f4922f45568161a8cdf4ad2299f6d23,10|opendoar____::4aa0e93b918848be0b7728b4b1568d8a,10|openaire____::02b55e4f52388520bfe11f959f836e68 
datasourceTypeWhitelist  = pubsrepository::unknown,pubsrepository::institutional,pubsrepository::thematic,datarepository::unknown,orprepository,softwarerepository 
datasourceIdBlacklist    = -
topicWhitelist           = *

Extra parameter related only to the workflow that dispatch the events on HDFS by opendoar ID

opendoarIds = 3665,1119,455,614,1227,1329,1341,1375,1436,1479,1529,1550,1553,1557,1584,1610,1737,1769,1786,1815,1845,1868,1871,1896,1932,2042,2108,2116,2120,2123,2124,2135,2145,2154,2180,2187,2220,2221,2268,2361,2363,2373,2374,2404,2441,2487,2510,2518,2546,2572,2585,2681,2718,2729,2758,2774,2844,2867,2903,2946,3056,3277,3344,3465,3498,3636,3915,3930,3944,4108,4184,4185,4186,4189,4256,4322,4369,4419,4445,4465,4509,4510,4545,4678,4680,4683,4771,4871,6315,6318,7108,9402,9419,9421,9441,9517,9535

This PR corrects the following problems: #67 #68 #70 #74 #75 The oozie wf needs to be updated including only the following parameters: ``` graphInputPath = /user/michele.artini/broker_input_graph workingDir = /user/michele.artini/broker_events_temp outputDir = /user/michele.artini/broker_events esIndexHost = 10.19.65.51,10.19.65.52,10.19.65.53,10.19.65.54 esEventIndexName = oa_events_beta esNotificationsIndexName = oa_notifications_beta esBatchWriteRetryCount = 8 esBatchWriteRetryWait = 60s esBatchSizeEntries = 200 esNodesWanOnly = true maxIndexedEventsForDsAndTopic = 100 brokerApiBaseUrl = http://10.19.65.36:9380 brokerDbUrl = jdbc:postgresql://10.19.65.40:5432/oa_broker brokerDbUser = ... brokerDbPassword = ... sparkDriverMemory = 7G sparkExecutorMemory = 7G sparkExecutorCores = 6 datasourceIdWhitelist = 10|openaire____::9ecafa3655143cbc4bc75853035cd432,10|opendoar____::dc6e224a8d74ce03bf301152d6e33e97,10|openaire____::09da65eaaa6deac2f785df1e0ae95a06,10|openaire____::3db634fc5446f389d0b826ea400a5da6,10|openaire____::5a38cb462ac487bf26bdb86009fe3e74,10|openaire____::3c29379cc184f66861e858bc7aa9615b,10|openaire____::4657147e48a1f32637bfe3743bce76c6,10|openaire____::c3267ea1c3f378c456209b6df241624e,10|opendoar____::358aee4cc897452c00244351e4d91f69,10|re3data_____::7b0ad08687b2c960d5aeef06f811d5e6,10|opendoar____::798ed7d4ee7138d49b8828958048130a,10|opendoar____::6f4922f45568161a8cdf4ad2299f6d23,10|opendoar____::4aa0e93b918848be0b7728b4b1568d8a,10|openaire____::02b55e4f52388520bfe11f959f836e68 datasourceTypeWhitelist = pubsrepository::unknown,pubsrepository::institutional,pubsrepository::thematic,datarepository::unknown,orprepository,softwarerepository datasourceIdBlacklist = - topicWhitelist = * ``` Extra parameter related only to the workflow that dispatch the events on HDFS by opendoar ID ``` opendoarIds = 3665,1119,455,614,1227,1329,1341,1375,1436,1479,1529,1550,1553,1557,1584,1610,1737,1769,1786,1815,1845,1868,1871,1896,1932,2042,2108,2116,2120,2123,2124,2135,2145,2154,2180,2187,2220,2221,2268,2361,2363,2373,2374,2404,2441,2487,2510,2518,2546,2572,2585,2681,2718,2729,2758,2774,2844,2867,2903,2946,3056,3277,3344,3465,3498,3636,3915,3930,3944,4108,4184,4185,4186,4189,4256,4322,4369,4419,4445,4465,4509,4510,4545,4678,4680,4683,4771,4871,6315,6318,7108,9402,9419,9421,9441,9517,9535 ```

Labels Milestones

broker #78

Step 1:

Step 2: