andreas.czerniak
/
BrBETA_dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
BrBETA_dnet-hadoop/dhp-workflows/dhp-graph-provision/job-override.properties

12 lines
544 B
Properties
Raw Normal View History

indexing workflow WIP: fixed projects fundingtree xml conversion, prioritized links between results and projects when limiting them to 100 in the join procedure
2020-03-02 17:03:07 +01:00
sparkDriverMemory=10G
sparkExecutorMemory=15G
implemented XML records construction and serialization, indexing WIP
2020-02-13 16:53:27 +01:00
#isLookupUrl=http://services.openaire.eu:8280/is/services/isLookUp
isLookupUrl=http://beta.services.openaire.eu:8280/is/services/isLookUp?wsdl
sourcePath=/tmp/db_openaireplus_services.export_dhp.2020.02.03
outputPath=/tmp/openaire_provision
format=TMF
working procedure for records indexing using Spark, via lib com.lucidworks.spark:spark-solr
2020-02-14 12:28:52 +01:00
batchSize=2000
indexing workflow WIP: fixed projects fundingtree xml conversion, prioritized links between results and projects when limiting them to 100 in the join procedure
2020-03-02 17:03:07 +01:00
sparkExecutorCoresForJoining=128
working procedure for records indexing using Spark, via lib com.lucidworks.spark:spark-solr
2020-02-14 12:28:52 +01:00
sparkExecutorCoresForIndexing=64
indexing workflow WIP: fixed projects fundingtree xml conversion, prioritized links between results and projects when limiting them to 100 in the join procedure
2020-03-02 17:03:07 +01:00
reuseRecords=false
otherDsTypeId=scholarcomminfra, infospace, pubsrepository::mock, entityregistry, entityregistry::projects, entityregistry::repositories, websource