Add support to create/update solr collection aliases #449

schatz · 2024-06-20T15:10:34+02:00

schatz commented

2024-06-20 15:10:34 +02:00

The 'update_solr_aliases' action was added in the 'index_infospace_graph' workflow.
It creates/updates two aliases, with the names public and shadow.

It can be explicitly executed when setting resumeFrom=update_solr_aliases.
Also, publicFormat and publicFormat parameters need to be provided.

Example:
resumeFrom=update_solr_aliases
publicFormat=DMF
shadowFormat=TMF

Tested in GARR: test run here

The 'update_solr_aliases' action was added in the 'index_infospace_graph' workflow. It creates/updates two aliases, with the names `public` and `shadow`. It can be explicitly executed when setting **`resumeFrom=update_solr_aliases`**. Also, **`publicFormat`** and **`publicFormat`** parameters need to be provided. Example: `resumeFrom=update_solr_aliases` `publicFormat=DMF` `shadowFormat=TMF` Tested in GARR: test run [here](https://hadoop-edge2.garr-pa1.d4science.org:8889/hue/jobbrowser#!id=0000003-240527171830460-oozie-oozi-W)

claudio.atzori was assigned by schatz

2024-06-20 15:10:34 +02:00

schatz added 1 commit 2024-06-20 15:10:34 +02:00

9f6e16a03c Add support to cretate/update solr collection aliases

claudio.atzori commented

2024-06-25 08:54:06 +02:00

Thanks @schatz for this contribution. I think it gives the proper building blocks to implement the collection/alias switch functionality, but in the current proposed state I cannot accept it. In fact, the switch operation cannot be part of the index feeding workflow. In between there are the content check operations that might prevent to operate the switch and sometimes imply to fix something in the data and feed the shadow collection again.

Hence I think it would be better to define a separated oozie workflow, a simple one, responsible only to perform the switch. In this way the index feeding and the alias switching would be completely independent operations.

As I said above however, the building blocks seems to be in place, so I can reorganise the changes and define the oozie workflow by myself. Thanks!

Thanks @schatz for this contribution. I think it gives the proper building blocks to implement the collection/alias switch functionality, but in the current proposed state I cannot accept it. In fact, the switch operation cannot be part of the index feeding workflow. In between there are the content check operations that might prevent to operate the switch and sometimes imply to fix something in the data and feed the shadow collection again. Hence I think it would be better to define a separated oozie workflow, a simple one, responsible only to perform the switch. In this way the index feeding and the alias switching would be completely independent operations. As I said above however, the building blocks seems to be in place, so I can reorganise the changes and define the oozie workflow by myself. Thanks!

claudio.atzori commented

2024-06-25 09:43:51 +02:00

I must amend my comment above, I just noticed now your comment

Action that updates the solr core aliases - out of order execution, only using the 'resume_from' param

So you thought about it :)

In this case, I will only have to define the D-Net level collection/alias switching workflow to invoke the provision oozie workflow explicitly setting the proper resume_from parameter value.

I must amend my comment above, I just noticed now your comment ``` Action that updates the solr core aliases - out of order execution, only using the 'resume_from' param ``` So you thought about it :) In this case, I will only have to define the D-Net level collection/alias switching workflow to invoke the provision oozie workflow explicitly setting the proper resume_from parameter value.

👍 1

claudio.atzori added 1 commit 2024-06-25 09:58:03 +02:00

0031cf849e Merge branch 'beta' into 9872-create-solr-collection-aliases

claudio.atzori added 2 commits 2024-06-25 15:27:44 +02:00

6055212f77 merged from the json_payload branch

1c30eacac2 updated index feeding procedure to exploit the collection aliases

claudio.atzori commented

2024-06-26 10:07:17 +02:00

I took the opportunity of this PR to introduce some further changes needed to handle the feeding of the index collection targeting the shadow alias.

claudio.atzori merged commit 1182bca9eb into beta

2024-06-26 10:09:52 +02:00

claudio.atzori referenced this issue from a commit

2024-06-26 10:09:53 +02:00

Merge pull request 'Add support to cretate/update solr collection aliases' (#449) from 9872-create-solr-collection-aliases into beta

claudio.atzori changed title from ~~Add support to cretate/update solr collection aliases~~ to Add support to create/update solr collection aliases

2024-07-03 10:32:27 +02:00

Sign in to join this conversation.

No reviewers