data provision workflow: add nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed #36

claudio.atzori · 2020-08-03T14:32:47+02:00

claudio.atzori commented

2020-08-03 14:32:47 +02:00

WIP, currently the oozie workflow for the XML records indexing requires to manually drop the shadow solr collection before the indexing procedure starts as well as to perform an hard commit after the indexing process is completed to make the changes visible.

This task aims at extending the oozie workflow behaviour by wrapping the to_solr_index action node with

a preceeding node aimed to drop the target collection: drop_solr_collection;
a following node aimed to issue a commit command on the target collection: commit_solr_collection.
The parameters available in the workflow definition should be sufficient to perform the new operations.

(description from #35)

WIP, currently the oozie workflow for the XML records indexing requires to manually drop the shadow solr collection before the indexing procedure starts as well as to perform an hard commit after the indexing process is completed to make the changes visible. This task aims at extending the oozie workflow behaviour by wrapping the to_solr_index action node with a preceeding node aimed to drop the target collection: drop_solr_collection; a following node aimed to issue a commit command on the target collection: commit_solr_collection. The parameters available in the workflow definition should be sufficient to perform the new operations. (description from [#35](https://code-repo.d4science.org/D-Net/dnet-hadoop/issues/35))

claudio.atzori added the

enhancement

label 2020-08-03 14:32:47 +02:00

miriam.baglioni was assigned by claudio.atzori

2020-08-03 14:32:47 +02:00

claudio.atzori changed title from ~~data provision workflow enhancement: added nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed~~ to data provision workflow: added nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed

2020-08-03 15:41:47 +02:00

claudio.atzori changed title from ~~data provision workflow: added nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed~~ to data provision workflow: add nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed

2020-08-03 15:42:13 +02:00

miriam.baglioni commented

2020-08-03 15:58:08 +02:00

I have looked at the code and workflow and everything seems fine to me. The PR can be accepted

claudio.atzori referenced this issue from a commit

2020-08-10 11:16:30 +02:00

Merge pull request 'data provision workflow: add nodes to perform DELETE BY QUERY before the indexing begins and COMMIT after the indexing is completed' (#36) from provision_indexing into master

claudio.atzori closed this pull request

2020-08-10 11:16:30 +02:00

claudio.atzori commented

2020-08-10 11:28:25 +02:00

Thanks Miriam! PR integrated and deployed in /lib/dnet/oa/provision