Enrico Ottonello
|
465ce39f75
|
job execution now based on file last_update.txt on hdfs
|
2021-02-04 10:44:04 +01:00 |
Enrico Ottonello
|
b2de598c1a
|
all actions from download lambda file to merge updated data into one wf
|
2020-12-15 10:42:55 +01:00 |
Enrico Ottonello
|
99a086f0c6
|
max concurrent executors set to 10, according to ORCID Director of Technology mail request
|
2020-11-24 17:49:32 +01:00 |
Enrico Ottonello
|
5c17e768b2
|
set wf configuration with spark.dynamicAllocation.maxExecutors 20 over 20 input partitions
|
2020-11-23 16:01:23 +01:00 |
Enrico Ottonello
|
97c8111847
|
action to convert lambda file in seq file; spark action to download updated authors
|
2020-11-23 09:49:22 +01:00 |