hardcoded parameters in broker workflow #70

New Issue

claudio.atzori · 2020-12-10T09:10:42+01:00

claudio.atzori commented

2020-12-10 09:10:42 +01:00

I see in https://code-repo.d4science.org/D-Net/dnet-hadoop/src/branch/master/dhp-workflows/dhp-broker-events/src/main/java/eu/dnetlib/dhp/broker/oa/IndexEventSubsetJob.java#L89

There are some hardcoded Elasticsearch connection properties:

esCfg.put("es.batch.write.retry.count", "8");
esCfg.put("es.batch.write.retry.wait", "60s");
esCfg.put("es.batch.size.entries", "200");

it would be good to have them optionally configurable.

I see in https://code-repo.d4science.org/D-Net/dnet-hadoop/src/branch/master/dhp-workflows/dhp-broker-events/src/main/java/eu/dnetlib/dhp/broker/oa/IndexEventSubsetJob.java#L89 There are some hardcoded Elasticsearch connection properties: ``` esCfg.put("es.batch.write.retry.count", "8"); esCfg.put("es.batch.write.retry.wait", "60s"); esCfg.put("es.batch.size.entries", "200"); ``` it would be good to have them optionally configurable.

claudio.atzori added the

enhancement

label 2020-12-10 09:10:42 +01:00

michele.artini was assigned by claudio.atzori

2020-12-10 09:10:42 +01:00

claudio.atzori commented

2020-12-10 09:29:20 +01:00

The same parameters are hardcoded also in

https://code-repo.d4science.org/D-Net/dnet-hadoop/src/branch/master/dhp-workflows/dhp-broker-events/src/main/java/eu/dnetlib/dhp/broker/oa/IndexNotificationsJob.java#L95

As the action implemented by those two classes is quite similar (index Events and Notifications), perhaps you could consider to extract the common part in an abstract class?

The same parameters are hardcoded also in https://code-repo.d4science.org/D-Net/dnet-hadoop/src/branch/master/dhp-workflows/dhp-broker-events/src/main/java/eu/dnetlib/dhp/broker/oa/IndexNotificationsJob.java#L95 As the action implemented by those two classes is quite similar (index Events and Notifications), perhaps you could consider to extract the common part in an abstract class?

michele.artini commented

2020-12-14 12:05:46 +01:00

The fix has been commited in the broker branch

michele.artini commented

2020-12-14 12:07:42 +01:00

The wf parameters with suggested values are:

esBatchWriteRetryCount   = 8
esBatchWriteRetryWait    = 60s
esBatchSizeEntries       = 200
esNodesWanOnly           = true

The wf parameters with suggested values are: ``` esBatchWriteRetryCount = 8 esBatchWriteRetryWait = 60s esBatchSizeEntries = 200 esNodesWanOnly = true ```

michele.artini referenced this issue

2020-12-15 09:02:55 +01:00

broker #78

claudio.atzori commented

2020-12-18 09:27:03 +01:00

Integrated in PR !78

claudio.atzori closed this issue

2020-12-18 09:27:03 +01:00

Sign in to join this conversation.