D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 12 Projects 2 Releases 2 Wiki Activity

broker #78

Merged
claudio.atzori merged 10 commits from broker into master 2020-12-15 10:03:46 +01:00
Conversation 0 Commits 10 Files Changed 36 +512 -129
michele.artini commented 2020-12-15 09:02:23 +01:00
Member
Copy Link

This PR corrects the following problems: #67 #68 #70 #74 #75

The oozie wf needs to be updated including only the following parameters:

graphInputPath  = /user/michele.artini/broker_input_graph
workingDir      = /user/michele.artini/broker_events_temp
outputDir       = /user/michele.artini/broker_events

esIndexHost              = 10.19.65.51,10.19.65.52,10.19.65.53,10.19.65.54
esEventIndexName         = oa_events_beta
esNotificationsIndexName = oa_notifications_beta
esBatchWriteRetryCount   = 8
esBatchWriteRetryWait    = 60s
esBatchSizeEntries       = 200
esNodesWanOnly           = true


maxIndexedEventsForDsAndTopic = 100
brokerApiBaseUrl = http://10.19.65.36:9380

brokerDbUrl      = jdbc:postgresql://10.19.65.40:5432/oa_broker 
brokerDbUser     = ...
brokerDbPassword = ...

sparkDriverMemory        = 7G
sparkExecutorMemory      = 7G
sparkExecutorCores       = 6

datasourceIdWhitelist    = 10|openaire____::9ecafa3655143cbc4bc75853035cd432,10|opendoar____::dc6e224a8d74ce03bf301152d6e33e97,10|openaire____::09da65eaaa6deac2f785df1e0ae95a06,10|openaire____::3db634fc5446f389d0b826ea400a5da6,10|openaire____::5a38cb462ac487bf26bdb86009fe3e74,10|openaire____::3c29379cc184f66861e858bc7aa9615b,10|openaire____::4657147e48a1f32637bfe3743bce76c6,10|openaire____::c3267ea1c3f378c456209b6df241624e,10|opendoar____::358aee4cc897452c00244351e4d91f69,10|re3data_____::7b0ad08687b2c960d5aeef06f811d5e6,10|opendoar____::798ed7d4ee7138d49b8828958048130a,10|opendoar____::6f4922f45568161a8cdf4ad2299f6d23,10|opendoar____::4aa0e93b918848be0b7728b4b1568d8a,10|openaire____::02b55e4f52388520bfe11f959f836e68 
datasourceTypeWhitelist  = pubsrepository::unknown,pubsrepository::institutional,pubsrepository::thematic,datarepository::unknown,orprepository,softwarerepository 
datasourceIdBlacklist    = -
topicWhitelist           = *

Extra parameter related only to the workflow that dispatch the events on HDFS by opendoar ID

opendoarIds = 3665,1119,455,614,1227,1329,1341,1375,1436,1479,1529,1550,1553,1557,1584,1610,1737,1769,1786,1815,1845,1868,1871,1896,1932,2042,2108,2116,2120,2123,2124,2135,2145,2154,2180,2187,2220,2221,2268,2361,2363,2373,2374,2404,2441,2487,2510,2518,2546,2572,2585,2681,2718,2729,2758,2774,2844,2867,2903,2946,3056,3277,3344,3465,3498,3636,3915,3930,3944,4108,4184,4185,4186,4189,4256,4322,4369,4419,4445,4465,4509,4510,4545,4678,4680,4683,4771,4871,6315,6318,7108,9402,9419,9421,9441,9517,9535
This PR corrects the following problems: #67 #68 #70 #74 #75 The oozie wf needs to be updated including only the following parameters: ``` graphInputPath = /user/michele.artini/broker_input_graph workingDir = /user/michele.artini/broker_events_temp outputDir = /user/michele.artini/broker_events esIndexHost = 10.19.65.51,10.19.65.52,10.19.65.53,10.19.65.54 esEventIndexName = oa_events_beta esNotificationsIndexName = oa_notifications_beta esBatchWriteRetryCount = 8 esBatchWriteRetryWait = 60s esBatchSizeEntries = 200 esNodesWanOnly = true maxIndexedEventsForDsAndTopic = 100 brokerApiBaseUrl = http://10.19.65.36:9380 brokerDbUrl = jdbc:postgresql://10.19.65.40:5432/oa_broker brokerDbUser = ... brokerDbPassword = ... sparkDriverMemory = 7G sparkExecutorMemory = 7G sparkExecutorCores = 6 datasourceIdWhitelist = 10|openaire____::9ecafa3655143cbc4bc75853035cd432,10|opendoar____::dc6e224a8d74ce03bf301152d6e33e97,10|openaire____::09da65eaaa6deac2f785df1e0ae95a06,10|openaire____::3db634fc5446f389d0b826ea400a5da6,10|openaire____::5a38cb462ac487bf26bdb86009fe3e74,10|openaire____::3c29379cc184f66861e858bc7aa9615b,10|openaire____::4657147e48a1f32637bfe3743bce76c6,10|openaire____::c3267ea1c3f378c456209b6df241624e,10|opendoar____::358aee4cc897452c00244351e4d91f69,10|re3data_____::7b0ad08687b2c960d5aeef06f811d5e6,10|opendoar____::798ed7d4ee7138d49b8828958048130a,10|opendoar____::6f4922f45568161a8cdf4ad2299f6d23,10|opendoar____::4aa0e93b918848be0b7728b4b1568d8a,10|openaire____::02b55e4f52388520bfe11f959f836e68 datasourceTypeWhitelist = pubsrepository::unknown,pubsrepository::institutional,pubsrepository::thematic,datarepository::unknown,orprepository,softwarerepository datasourceIdBlacklist = - topicWhitelist = * ``` Extra parameter related only to the workflow that dispatch the events on HDFS by opendoar ID ``` opendoarIds = 3665,1119,455,614,1227,1329,1341,1375,1436,1479,1529,1550,1553,1557,1584,1610,1737,1769,1786,1815,1845,1868,1871,1896,1932,2042,2108,2116,2120,2123,2124,2135,2145,2154,2180,2187,2220,2221,2268,2361,2363,2373,2374,2404,2441,2487,2510,2518,2546,2572,2585,2681,2718,2729,2758,2774,2844,2867,2903,2946,3056,3277,3344,3465,3498,3636,3915,3930,3944,4108,4184,4185,4186,4189,4256,4322,4369,4419,4445,4465,4509,4510,4545,4678,4680,4683,4771,4871,6315,6318,7108,9402,9419,9421,9441,9517,9535 ```
claudio.atzori closed this pull request 2020-12-15 10:03:46 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#78
No description provided.
Delete Branch "broker"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?