import_new_mdstores #112

michele.artini · 2021-06-01T12:28:09+02:00

michele.artini commented

2021-06-01 12:28:09 +02:00

Import of the new mdstores (hdfs) in the graph

The code has been tested running a simple wf (raw_hdfs_stores), a new unit test (working on a dataset record) is available.

The main workflow (raw_all) has been updated but not tested.

New parameters for raw_all workflow:

reuseODF_hdfs
reuseOAF_hdfs
mdstoreManagerUrl (for BETA: https://beta.services.openaire.eu/mdstoremanager)

Import of the new mdstores (hdfs) in the graph The code has been tested running a simple wf (raw_hdfs_stores), a new unit test (working on a dataset record) is available. The main workflow (raw_all) has been updated but not tested. New parameters for raw_all workflow: * reuseODF_hdfs * reuseOAF_hdfs * mdstoreManagerUrl (for BETA: https://beta.services.openaire.eu/mdstoremanager)

michele.artini added 6 commits 2021-06-01 12:28:09 +02:00

4fa5671d16 first implementation of Hdfs Mdstores Importer

ad56a44fda save as gzipped sequence file

e9f2b6037c patch of mdstore records

03a510859a removed coalesce(1)

e950750262 add nodes to import hdfs mdstores

f0fbfdcfae Merge branch 'stable_ids' into import_new_mdstores

michele.artini added 1 commit 2021-06-01 12:42:50 +02:00

ede2749822 orcid pid type

claudio.atzori merged commit dd19c4ac5a into stable_ids

2021-06-14 09:23:56 +02:00

claudio.atzori referenced this issue from a commit