Update Solr [PROD]
Data Provision
30
Set the path containing the GRAPH to index
inputGraphRootPath
/tmp/prod_provision/graph/14_graph_blacklisted
Set the target path to store the RAW graph
format
DMF
Set the lookup address
isLookupUrl
http://services.openaire.eu:8280/is/services/isLookUp?wsdl
wait configurations
create the AGGREGATOR graph
executeOozieJob
IIS
{
'inputGraphRootPath' : 'inputGraphRootPath',
'isLookupUrl' : 'isLookupUrl',
'format' : 'format'
}
{
'oozie.wf.application.path' : '/lib/dnet/PROD/oa/provision/oozie_app',
'sourceMaxRelations' : '1000',
'targetMaxRelations' : '10000000',
'relPartitions' : '3000',
'batchSize' : '2000',
'relationFilter' : 'isAuthorInstitutionOf,produces,hasAmongTopNSimilarDocuments,cites,isCitedBy',
'otherDsTypeId' : 'scholarcomminfra,infospace,pubsrepository::mock,entityregistry,entityregistry::projects,entityregistry::repositories,websource',
'resumeFrom' : 'prepare_relations',
'shouldIndex' : 'true',
'outputFormat' : 'SOLR',
'sparkDriverMemoryForJoining' : '3G',
'sparkExecutorMemoryForJoining' : '7G',
'sparkExecutorCoresForJoining' : '4',
'sparkDriverMemoryForIndexing' : '2G',
'sparkExecutorMemoryForIndexing' : '2G',
'sparkExecutorCoresForIndexing' : '64',
'sparkNetworkTimeout' : '600',
'workingDir' : '/tmp/prod_provision/working_dir/update_solr'
}
build-report
wf_20210724_062705_620
2021-07-25T13:25:37+00:00
SUCCESS