atumatic solr collection drop & commit #35

New Issue

claudio.atzori · 2020-08-03T14:27:47+02:00

claudio.atzori commented

2020-08-03 14:27:47 +02:00

Currently the oozie workflow for the XML records indexing requires to manually drop the shadow solr collection before the indexing procedure starts as well as to perform an hard commit after the indexing process is completed to make the changes visible.

This task aims at extending the oozie workflow behaviour by wrapping the to_solr_index action node with

a preceeding node aimed to drop the target collection: drop_solr_collection;
a following node aimed to issue a commit command on the target collection: commit_solr_collection.

The parameters available in the workflow definition should be sufficient to perform the new operations.

Currently the oozie workflow for the XML records indexing requires to manually drop the shadow solr collection before the indexing procedure starts as well as to perform an hard commit after the indexing process is completed to make the changes visible. This task aims at extending the oozie workflow behaviour by wrapping the `to_solr_index` action node with * a preceeding node aimed to drop the target collection: `drop_solr_collection`; * a following node aimed to issue a commit command on the target collection: `commit_solr_collection`. The parameters available in the workflow definition should be sufficient to perform the new operations.

claudio.atzori added the

enhancement

label 2020-08-03 14:27:47 +02:00

claudio.atzori self-assigned this 2020-08-03 14:27:47 +02:00

claudio.atzori commented

2020-08-03 14:29:46 +02:00

A first version of the enhancement is implemented in 3a11a387a9, sending it out for code review.

A first version of the enhancement is implemented in https://code-repo.d4science.org/D-Net/dnet-hadoop/commit/3a11a387a94927c108ff29a36fb686b6d6864c43, sending it out for code review.

claudio.atzori referenced this issue

2020-08-03 14:32:47 +02:00

data provision workflow: add nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed #36

claudio.atzori commented

2020-08-12 14:40:04 +02:00

Workflow deployed in /lib/dnet/{PROD|BETA}/oa/provision. We should close this task as soon as the procedure will be tested in the execution orchestrated via D-NET.

Workflow deployed in ```/lib/dnet/{PROD|BETA}/oa/provision```. We should close this task as soon as the procedure will be tested in the execution orchestrated via D-NET.

claudio.atzori commented

2020-09-29 16:26:01 +02:00

Tested and working. Closing

claudio.atzori closed this issue

2020-09-29 16:26:01 +02:00

Sign in to join this conversation.