CODE_OF_CONDUCT.md

@@ -1,43 +0,0 @@
-# Contributor Code of Conduct
-
-Openness, transparency and our community-driven participatory approach guide us in our day-to-day interactions and decision-making. Our open source projects are no exception. Trust, respect, collaboration and transparency are core values we believe should live and breathe within our projects. Our community welcomes participants from around the world with different experiences, unique perspectives, and great ideas to share.
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment include:
-
-- Using welcoming and inclusive language
-- Being respectful of differing viewpoints and experiences
-- Gracefully accepting constructive criticism
-- Attempting collaboration before conflict
-- Focusing on what is best for the community
-- Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-- Violence, threats of violence, or inciting others to commit self-harm
-- The use of sexualized language or imagery and unwelcome sexual attention or advances
-- Trolling, intentionally spreading misinformation, insulting/derogatory comments, and personal or political attacks
-- Public or private harassment
-- Publishing others' private information, such as a physical or electronic address, without explicit permission
-- Abuse of the reporting process to intentionally harass or exclude others
-- Advocating for, or encouraging, any of the above behavior
-- Other conduct which could reasonably be considered inappropriate in a professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), [version 1.4](https://www.contributor-covenant.org/version/1/4/code-of-conduct.html).

CONTRIBUTING.md

@@ -1,10 +0,0 @@
-# Contributing to D-Net Hadoop
-
-:+1::tada: First off, thanks for taking the time to contribute! :tada::+1:
-
-This project and everyone participating in it is governed by our [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to [dnet-team@isti.cnr.it](mailto:dnet-team@isti.cnr.it).
-
-The following is a set of guidelines for contributing to this project and its packages. These are mostly guidelines, not rules, which applies to this project as a while, including all its sub-modules. 
-Use your best judgment, and feel free to propose changes to this document in a pull request.
-
-All contributions are welcome, all contributions will be considered to be contributed under the [project license](LICENSE.md).

README.md

@@ -2,11 +2,6 @@
 
 Dnet-hadoop is the project that defined all the [OOZIE workflows](https://oozie.apache.org/) for the OpenAIRE Graph construction, processing, provisioning.
 
-This project adheres to the Contributor Covenant [code of conduct](CODE_OF_CONDUCT.md).
-By participating, you are expected to uphold this code. Please report unacceptable behavior to [dnet-team@isti.cnr.it](mailto:dnet-team@isti.cnr.it).
-
-This project is licensed under the [AGPL v3 or later version](#LICENSE.md).
-
 How to build, package and run oozie workflows
 ====================
 

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/InputStreamRequestBody.java

@@ -0,0 +1,53 @@
+
+package eu.dnetlib.dhp.common.api;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import okhttp3.MediaType;
+import okhttp3.RequestBody;
+import okhttp3.internal.Util;
+import okio.BufferedSink;
+import okio.Okio;
+import okio.Source;
+
+public class InputStreamRequestBody extends RequestBody {
+
+	private final InputStream inputStream;
+	private final MediaType mediaType;
+	private final long lenght;
+
+	public static RequestBody create(final MediaType mediaType, final InputStream inputStream, final long len) {
+
+		return new InputStreamRequestBody(inputStream, mediaType, len);
+	}
+
+	private InputStreamRequestBody(InputStream inputStream, MediaType mediaType, long len) {
+		this.inputStream = inputStream;
+		this.mediaType = mediaType;
+		this.lenght = len;
+	}
+
+	@Override
+	public MediaType contentType() {
+		return mediaType;
+	}
+
+	@Override
+	public long contentLength() {
+
+		return lenght;
+
+	}
+
+	@Override
+	public void writeTo(BufferedSink sink) throws IOException {
+		Source source = null;
+		try {
+			source = Okio.source(inputStream);
+			sink.writeAll(source);
+		} finally {
+			Util.closeQuietly(source);
+		}
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/MissingConceptDoiException.java

@@ -0,0 +1,8 @@
+
+package eu.dnetlib.dhp.common.api;
+
+public class MissingConceptDoiException extends Throwable {
+	public MissingConceptDoiException(String message) {
+		super(message);
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/ZenodoAPIClient.java

@@ -0,0 +1,365 @@
+
+package eu.dnetlib.dhp.common.api;
+
+import java.io.*;
+import java.io.IOException;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.http.HttpHeaders;
+import org.apache.http.entity.ContentType;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.gson.Gson;
+
+import eu.dnetlib.dhp.common.api.zenodo.ZenodoModel;
+import eu.dnetlib.dhp.common.api.zenodo.ZenodoModelList;
+import okhttp3.*;
+
+public class ZenodoAPIClient implements Serializable {
+
+	String urlString;
+	String bucket;
+
+	String deposition_id;
+	String access_token;
+
+	public static final MediaType MEDIA_TYPE_JSON = MediaType.parse("application/json; charset=utf-8");
+
+	private static final MediaType MEDIA_TYPE_ZIP = MediaType.parse("application/zip");
+
+	public String getUrlString() {
+		return urlString;
+	}
+
+	public void setUrlString(String urlString) {
+		this.urlString = urlString;
+	}
+
+	public String getBucket() {
+		return bucket;
+	}
+
+	public void setBucket(String bucket) {
+		this.bucket = bucket;
+	}
+
+	public void setDeposition_id(String deposition_id) {
+		this.deposition_id = deposition_id;
+	}
+
+	public ZenodoAPIClient(String urlString, String access_token) {
+
+		this.urlString = urlString;
+		this.access_token = access_token;
+	}
+
+	/**
+	 * Brand new deposition in Zenodo. It sets the deposition_id and the bucket where to store the files to upload
+	 *
+	 * @return response code
+	 * @throws IOException
+	 */
+	public int newDeposition() throws IOException {
+		String json = "{}";
+
+		URL url = new URL(urlString);
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+		conn.setRequestProperty(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString());
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setRequestMethod("POST");
+		conn.setDoOutput(true);
+		try (OutputStream os = conn.getOutputStream()) {
+			byte[] input = json.getBytes("utf-8");
+			os.write(input, 0, input.length);
+		}
+
+		String body = getBody(conn);
+
+		int responseCode = conn.getResponseCode();
+		conn.disconnect();
+
+		if (!checkOKStatus(responseCode))
+			throw new IOException("Unexpected code " + responseCode + body);
+
+		ZenodoModel newSubmission = new Gson().fromJson(body, ZenodoModel.class);
+		this.bucket = newSubmission.getLinks().getBucket();
+		this.deposition_id = newSubmission.getId();
+
+		return responseCode;
+	}
+
+	/**
+	 * Upload files in Zenodo.
+	 *
+	 * @param is the inputStream for the file to upload
+	 * @param file_name the name of the file as it will appear on Zenodo
+	 * @return the response code
+	 */
+	public int uploadIS(InputStream is, String file_name) throws IOException {
+
+		URL url = new URL(bucket + "/" + file_name);
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+		conn.setRequestProperty(HttpHeaders.CONTENT_TYPE, "application/zip");
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setDoOutput(true);
+		conn.setRequestMethod("PUT");
+
+		byte[] buf = new byte[8192];
+		int length;
+		try (OutputStream os = conn.getOutputStream()) {
+			while ((length = is.read(buf)) != -1) {
+				os.write(buf, 0, length);
+			}
+
+		}
+		int responseCode = conn.getResponseCode();
+		if (!checkOKStatus(responseCode)) {
+			throw new IOException("Unexpected code " + responseCode + getBody(conn));
+		}
+
+		return responseCode;
+	}
+
+	@NotNull
+	private String getBody(HttpURLConnection conn) throws IOException {
+		String body = "{}";
+		try (BufferedReader br = new BufferedReader(
+			new InputStreamReader(conn.getInputStream(), "utf-8"))) {
+			StringBuilder response = new StringBuilder();
+			String responseLine = null;
+			while ((responseLine = br.readLine()) != null) {
+				response.append(responseLine.trim());
+			}
+
+			body = response.toString();
+
+		}
+		return body;
+	}
+
+	/**
+	 * Associates metadata information to the current deposition
+	 *
+	 * @param metadata the metadata
+	 * @return response code
+	 * @throws IOException
+	 */
+	public int sendMretadata(String metadata) throws IOException {
+
+		URL url = new URL(urlString + "/" + deposition_id);
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+		conn.setRequestProperty(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString());
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setDoOutput(true);
+		conn.setRequestMethod("PUT");
+
+		try (OutputStream os = conn.getOutputStream()) {
+			byte[] input = metadata.getBytes("utf-8");
+			os.write(input, 0, input.length);
+
+		}
+
+		final int responseCode = conn.getResponseCode();
+		conn.disconnect();
+		if (!checkOKStatus(responseCode))
+			throw new IOException("Unexpected code " + responseCode + getBody(conn));
+
+		return responseCode;
+
+	}
+
+	private boolean checkOKStatus(int responseCode) {
+
+		if (HttpURLConnection.HTTP_OK != responseCode ||
+			HttpURLConnection.HTTP_CREATED != responseCode)
+			return true;
+		return false;
+	}
+
+	/**
+	 * To publish the current deposition. It works for both new deposition or new version of an old deposition
+	 *
+	 * @return response code
+	 * @throws IOException
+	 */
+	@Deprecated
+	public int publish() throws IOException {
+
+		String json = "{}";
+
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		RequestBody body = RequestBody.create(json, MEDIA_TYPE_JSON);
+
+		Request request = new Request.Builder()
+			.url(urlString + "/" + deposition_id + "/actions/publish")
+			.addHeader("Authorization", "Bearer " + access_token)
+			.post(body)
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			return response.code();
+
+		}
+	}
+
+	/**
+	 * To create a new version of an already published deposition. It sets the deposition_id and the bucket to be used
+	 * for the new version.
+	 *
+	 * @param concept_rec_id the concept record id of the deposition for which to create a new version. It is the last
+	 *            part of the url for the DOI Zenodo suggests to use to cite all versions: DOI: 10.xxx/zenodo.656930
+	 *            concept_rec_id = 656930
+	 * @return response code
+	 * @throws IOException
+	 * @throws MissingConceptDoiException
+	 */
+	public int newVersion(String concept_rec_id) throws IOException, MissingConceptDoiException {
+		setDepositionId(concept_rec_id, 1);
+		String json = "{}";
+
+		URL url = new URL(urlString + "/" + deposition_id + "/actions/newversion");
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setDoOutput(true);
+		conn.setRequestMethod("POST");
+
+		try (OutputStream os = conn.getOutputStream()) {
+			byte[] input = json.getBytes("utf-8");
+			os.write(input, 0, input.length);
+
+		}
+
+		String body = getBody(conn);
+
+		int responseCode = conn.getResponseCode();
+
+		conn.disconnect();
+		if (!checkOKStatus(responseCode))
+			throw new IOException("Unexpected code " + responseCode + body);
+
+		ZenodoModel zenodoModel = new Gson().fromJson(body, ZenodoModel.class);
+		String latest_draft = zenodoModel.getLinks().getLatest_draft();
+		deposition_id = latest_draft.substring(latest_draft.lastIndexOf("/") + 1);
+		bucket = getBucket(latest_draft);
+
+		return responseCode;
+
+	}
+
+	/**
+	 * To finish uploading a version or new deposition not published
+	 * It sets the deposition_id and the bucket to be used
+	 *
+	 *
+	 * @param deposition_id the deposition id of the not yet published upload
+	 *            concept_rec_id = 656930
+	 * @return response code
+	 * @throws IOException
+	 * @throws MissingConceptDoiException
+	 */
+	public int uploadOpenDeposition(String deposition_id) throws IOException, MissingConceptDoiException {
+
+		this.deposition_id = deposition_id;
+
+		String json = "{}";
+
+		URL url = new URL(urlString + "/" + deposition_id);
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setRequestMethod("POST");
+		conn.setDoOutput(true);
+		try (OutputStream os = conn.getOutputStream()) {
+			byte[] input = json.getBytes("utf-8");
+			os.write(input, 0, input.length);
+		}
+
+		String body = getBody(conn);
+
+		int responseCode = conn.getResponseCode();
+		conn.disconnect();
+
+		if (!checkOKStatus(responseCode))
+			throw new IOException("Unexpected code " + responseCode + body);
+
+		ZenodoModel zenodoModel = new Gson().fromJson(body, ZenodoModel.class);
+		bucket = zenodoModel.getLinks().getBucket();
+
+		return responseCode;
+
+	}
+
+	private void setDepositionId(String concept_rec_id, Integer page) throws IOException, MissingConceptDoiException {
+
+		ZenodoModelList zenodoModelList = new Gson()
+			.fromJson(getPrevDepositions(String.valueOf(page)), ZenodoModelList.class);
+
+		for (ZenodoModel zm : zenodoModelList) {
+			if (zm.getConceptrecid().equals(concept_rec_id)) {
+				deposition_id = zm.getId();
+				return;
+			}
+		}
+		if (zenodoModelList.size() == 0)
+			throw new MissingConceptDoiException(
+				"The concept record id specified was missing in the list of depositions");
+		setDepositionId(concept_rec_id, page + 1);
+
+	}
+
+	private String getPrevDepositions(String page) throws IOException {
+
+		HttpUrl.Builder urlBuilder = HttpUrl.parse(urlString).newBuilder();
+		urlBuilder.addQueryParameter("page", page);
+
+		URL url = new URL(urlBuilder.build().toString());
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+		conn.setRequestProperty(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString());
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setDoOutput(true);
+		conn.setRequestMethod("GET");
+
+		String body = getBody(conn);
+
+		int responseCode = conn.getResponseCode();
+
+		conn.disconnect();
+		if (!checkOKStatus(responseCode))
+			throw new IOException("Unexpected code " + responseCode + body);
+
+		return body;
+
+	}
+
+	private String getBucket(String inputUurl) throws IOException {
+
+		URL url = new URL(inputUurl);
+		HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+		conn.setRequestProperty(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString());
+		conn.setRequestProperty(HttpHeaders.AUTHORIZATION, "Bearer " + access_token);
+		conn.setDoOutput(true);
+		conn.setRequestMethod("GET");
+
+		String body = getBody(conn);
+
+		int responseCode = conn.getResponseCode();
+
+		conn.disconnect();
+		if (!checkOKStatus(responseCode))
+			throw new IOException("Unexpected code " + responseCode + body);
+
+		ZenodoModel zenodoModel = new Gson().fromJson(body, ZenodoModel.class);
+
+		return zenodoModel.getLinks().getBucket();
+
+	}
+
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/CategorySummary.java

@@ -1,39 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-public class CategorySummary {
-
-	private String id;
-
-	private String label;
-
-	private boolean hasConcept;
-
-	public String getId() {
-		return id;
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public boolean isHasConcept() {
-		return hasConcept;
-	}
-
-	public CategorySummary setId(final String id) {
-		this.id = id;
-		return this;
-	}
-
-	public CategorySummary setLabel(final String label) {
-		this.label = label;
-		return this;
-	}
-
-	public CategorySummary setHasConcept(final boolean hasConcept) {
-		this.hasConcept = hasConcept;
-		return this;
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/CategorySummaryList.java

@@ -1,7 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.ArrayList;
-
-public class CategorySummaryList extends ArrayList<CategorySummary> {
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ConceptSummary.java

@@ -1,52 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.List;
-
-public class ConceptSummary {
-
-	private String id;
-
-	private String label;
-
-	public boolean hasSubConcept;
-
-	private List<ConceptSummary> concepts;
-
-	public String getId() {
-		return id;
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public List<ConceptSummary> getConcepts() {
-		return concepts;
-	}
-
-	public ConceptSummary setId(final String id) {
-		this.id = id;
-		return this;
-	}
-
-	public ConceptSummary setLabel(final String label) {
-		this.label = label;
-		return this;
-	}
-
-	public boolean isHasSubConcept() {
-		return hasSubConcept;
-	}
-
-	public ConceptSummary setHasSubConcept(final boolean hasSubConcept) {
-		this.hasSubConcept = hasSubConcept;
-		return this;
-	}
-
-	public ConceptSummary setConcept(final List<ConceptSummary> concepts) {
-		this.concepts = concepts;
-		return this;
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ConceptSummaryList.java

@@ -1,7 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.ArrayList;
-
-public class ConceptSummaryList extends ArrayList<ConceptSummary> {
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ContextSummary.java

@@ -1,50 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-public class ContextSummary {
-
-	private String id;
-
-	private String label;
-
-	private String type;
-
-	private String status;
-
-	public String getId() {
-		return id;
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public String getType() {
-		return type;
-	}
-
-	public String getStatus() {
-		return status;
-	}
-
-	public ContextSummary setId(final String id) {
-		this.id = id;
-		return this;
-	}
-
-	public ContextSummary setLabel(final String label) {
-		this.label = label;
-		return this;
-	}
-
-	public ContextSummary setType(final String type) {
-		this.type = type;
-		return this;
-	}
-
-	public ContextSummary setStatus(final String status) {
-		this.status = status;
-		return this;
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ContextSummaryList.java

@@ -1,7 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.ArrayList;
-
-public class ContextSummaryList extends ArrayList<ContextSummary> {
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Community.java

@@ -0,0 +1,14 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+public class Community {
+	private String identifier;
+
+	public String getIdentifier() {
+		return identifier;
+	}
+
+	public void setIdentifier(String identifier) {
+		this.identifier = identifier;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Creator.java

@@ -0,0 +1,47 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+public class Creator {
+	private String affiliation;
+	private String name;
+	private String orcid;
+
+	public String getAffiliation() {
+		return affiliation;
+	}
+
+	public void setAffiliation(String affiliation) {
+		this.affiliation = affiliation;
+	}
+
+	public String getName() {
+		return name;
+	}
+
+	public void setName(String name) {
+		this.name = name;
+	}
+
+	public String getOrcid() {
+		return orcid;
+	}
+
+	public void setOrcid(String orcid) {
+		this.orcid = orcid;
+	}
+
+	public static Creator newInstance(String name, String affiliation, String orcid) {
+		Creator c = new Creator();
+		if (name != null) {
+			c.name = name;
+		}
+		if (affiliation != null) {
+			c.affiliation = affiliation;
+		}
+		if (orcid != null) {
+			c.orcid = orcid;
+		}
+
+		return c;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/File.java

@@ -0,0 +1,44 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class File implements Serializable {
+	private String checksum;
+	private String filename;
+	private long filesize;
+	private String id;
+
+	public String getChecksum() {
+		return checksum;
+	}
+
+	public void setChecksum(String checksum) {
+		this.checksum = checksum;
+	}
+
+	public String getFilename() {
+		return filename;
+	}
+
+	public void setFilename(String filename) {
+		this.filename = filename;
+	}
+
+	public long getFilesize() {
+		return filesize;
+	}
+
+	public void setFilesize(long filesize) {
+		this.filesize = filesize;
+	}
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(String id) {
+		this.id = id;
+	}
+
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Grant.java

@@ -0,0 +1,23 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class Grant implements Serializable {
+	private String id;
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(String id) {
+		this.id = id;
+	}
+
+	public static Grant newInstance(String id) {
+		Grant g = new Grant();
+		g.id = id;
+
+		return g;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Links.java

@@ -0,0 +1,92 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class Links implements Serializable {
+
+	private String bucket;
+
+	private String discard;
+
+	private String edit;
+	private String files;
+	private String html;
+	private String latest_draft;
+	private String latest_draft_html;
+	private String publish;
+
+	private String self;
+
+	public String getBucket() {
+		return bucket;
+	}
+
+	public void setBucket(String bucket) {
+		this.bucket = bucket;
+	}
+
+	public String getDiscard() {
+		return discard;
+	}
+
+	public void setDiscard(String discard) {
+		this.discard = discard;
+	}
+
+	public String getEdit() {
+		return edit;
+	}
+
+	public void setEdit(String edit) {
+		this.edit = edit;
+	}
+
+	public String getFiles() {
+		return files;
+	}
+
+	public void setFiles(String files) {
+		this.files = files;
+	}
+
+	public String getHtml() {
+		return html;
+	}
+
+	public void setHtml(String html) {
+		this.html = html;
+	}
+
+	public String getLatest_draft() {
+		return latest_draft;
+	}
+
+	public void setLatest_draft(String latest_draft) {
+		this.latest_draft = latest_draft;
+	}
+
+	public String getLatest_draft_html() {
+		return latest_draft_html;
+	}
+
+	public void setLatest_draft_html(String latest_draft_html) {
+		this.latest_draft_html = latest_draft_html;
+	}
+
+	public String getPublish() {
+		return publish;
+	}
+
+	public void setPublish(String publish) {
+		this.publish = publish;
+	}
+
+	public String getSelf() {
+		return self;
+	}
+
+	public void setSelf(String self) {
+		this.self = self;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Metadata.java

@@ -0,0 +1,153 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+import java.util.List;
+
+public class Metadata implements Serializable {
+
+	private String access_right;
+	private List<Community> communities;
+	private List<Creator> creators;
+	private String description;
+	private String doi;
+	private List<Grant> grants;
+	private List<String> keywords;
+	private String language;
+	private String license;
+	private PrereserveDoi prereserve_doi;
+	private String publication_date;
+	private List<String> references;
+	private List<RelatedIdentifier> related_identifiers;
+	private String title;
+	private String upload_type;
+	private String version;
+
+	public String getUpload_type() {
+		return upload_type;
+	}
+
+	public void setUpload_type(String upload_type) {
+		this.upload_type = upload_type;
+	}
+
+	public String getVersion() {
+		return version;
+	}
+
+	public void setVersion(String version) {
+		this.version = version;
+	}
+
+	public String getAccess_right() {
+		return access_right;
+	}
+
+	public void setAccess_right(String access_right) {
+		this.access_right = access_right;
+	}
+
+	public List<Community> getCommunities() {
+		return communities;
+	}
+
+	public void setCommunities(List<Community> communities) {
+		this.communities = communities;
+	}
+
+	public List<Creator> getCreators() {
+		return creators;
+	}
+
+	public void setCreators(List<Creator> creators) {
+		this.creators = creators;
+	}
+
+	public String getDescription() {
+		return description;
+	}
+
+	public void setDescription(String description) {
+		this.description = description;
+	}
+
+	public String getDoi() {
+		return doi;
+	}
+
+	public void setDoi(String doi) {
+		this.doi = doi;
+	}
+
+	public List<Grant> getGrants() {
+		return grants;
+	}
+
+	public void setGrants(List<Grant> grants) {
+		this.grants = grants;
+	}
+
+	public List<String> getKeywords() {
+		return keywords;
+	}
+
+	public void setKeywords(List<String> keywords) {
+		this.keywords = keywords;
+	}
+
+	public String getLanguage() {
+		return language;
+	}
+
+	public void setLanguage(String language) {
+		this.language = language;
+	}
+
+	public String getLicense() {
+		return license;
+	}
+
+	public void setLicense(String license) {
+		this.license = license;
+	}
+
+	public PrereserveDoi getPrereserve_doi() {
+		return prereserve_doi;
+	}
+
+	public void setPrereserve_doi(PrereserveDoi prereserve_doi) {
+		this.prereserve_doi = prereserve_doi;
+	}
+
+	public String getPublication_date() {
+		return publication_date;
+	}
+
+	public void setPublication_date(String publication_date) {
+		this.publication_date = publication_date;
+	}
+
+	public List<String> getReferences() {
+		return references;
+	}
+
+	public void setReferences(List<String> references) {
+		this.references = references;
+	}
+
+	public List<RelatedIdentifier> getRelated_identifiers() {
+		return related_identifiers;
+	}
+
+	public void setRelated_identifiers(List<RelatedIdentifier> related_identifiers) {
+		this.related_identifiers = related_identifiers;
+	}
+
+	public String getTitle() {
+		return title;
+	}
+
+	public void setTitle(String title) {
+		this.title = title;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/PrereserveDoi.java

@@ -0,0 +1,25 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class PrereserveDoi implements Serializable {
+	private String doi;
+	private String recid;
+
+	public String getDoi() {
+		return doi;
+	}
+
+	public void setDoi(String doi) {
+		this.doi = doi;
+	}
+
+	public String getRecid() {
+		return recid;
+	}
+
+	public void setRecid(String recid) {
+		this.recid = recid;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/RelatedIdentifier.java

@@ -0,0 +1,43 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class RelatedIdentifier implements Serializable {
+	private String identifier;
+	private String relation;
+	private String resource_type;
+	private String scheme;
+
+	public String getIdentifier() {
+		return identifier;
+	}
+
+	public void setIdentifier(String identifier) {
+		this.identifier = identifier;
+	}
+
+	public String getRelation() {
+		return relation;
+	}
+
+	public void setRelation(String relation) {
+		this.relation = relation;
+	}
+
+	public String getResource_type() {
+		return resource_type;
+	}
+
+	public void setResource_type(String resource_type) {
+		this.resource_type = resource_type;
+	}
+
+	public String getScheme() {
+		return scheme;
+	}
+
+	public void setScheme(String scheme) {
+		this.scheme = scheme;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/ZenodoModel.java

@@ -0,0 +1,118 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+import java.util.List;
+
+public class ZenodoModel implements Serializable {
+
+	private String conceptrecid;
+	private String created;
+
+	private List<File> files;
+	private String id;
+	private Links links;
+	private Metadata metadata;
+	private String modified;
+	private String owner;
+	private String record_id;
+	private String state;
+	private boolean submitted;
+	private String title;
+
+	public String getConceptrecid() {
+		return conceptrecid;
+	}
+
+	public void setConceptrecid(String conceptrecid) {
+		this.conceptrecid = conceptrecid;
+	}
+
+	public String getCreated() {
+		return created;
+	}
+
+	public void setCreated(String created) {
+		this.created = created;
+	}
+
+	public List<File> getFiles() {
+		return files;
+	}
+
+	public void setFiles(List<File> files) {
+		this.files = files;
+	}
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(String id) {
+		this.id = id;
+	}
+
+	public Links getLinks() {
+		return links;
+	}
+
+	public void setLinks(Links links) {
+		this.links = links;
+	}
+
+	public Metadata getMetadata() {
+		return metadata;
+	}
+
+	public void setMetadata(Metadata metadata) {
+		this.metadata = metadata;
+	}
+
+	public String getModified() {
+		return modified;
+	}
+
+	public void setModified(String modified) {
+		this.modified = modified;
+	}
+
+	public String getOwner() {
+		return owner;
+	}
+
+	public void setOwner(String owner) {
+		this.owner = owner;
+	}
+
+	public String getRecord_id() {
+		return record_id;
+	}
+
+	public void setRecord_id(String record_id) {
+		this.record_id = record_id;
+	}
+
+	public String getState() {
+		return state;
+	}
+
+	public void setState(String state) {
+		this.state = state;
+	}
+
+	public boolean isSubmitted() {
+		return submitted;
+	}
+
+	public void setSubmitted(boolean submitted) {
+		this.submitted = submitted;
+	}
+
+	public String getTitle() {
+		return title;
+	}
+
+	public void setTitle(String title) {
+		this.title = title;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/ZenodoModelList.java

@@ -0,0 +1,7 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.util.ArrayList;
+
+public class ZenodoModelList extends ArrayList<ZenodoModel> {
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/collection/HttpConnector2.java

@@ -8,13 +8,10 @@ import java.io.InputStream;
 import java.net.*;
 import java.util.List;
 import java.util.Map;
-import java.util.concurrent.TimeUnit;
 
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang3.math.NumberUtils;
-import org.apache.commons.lang3.time.DateUtils;
 import org.apache.http.HttpHeaders;
-import org.joda.time.Instant;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -97,16 +94,14 @@ public class HttpConnector2 {
 			throw new CollectorException(msg);
 		}
 
+		log.info("Request attempt {} [{}]", retryNumber, requestUrl);
+
 		InputStream input = null;
 
-		long start = System.currentTimeMillis();
 		try {
 			if (getClientParams().getRequestDelay() > 0) {
 				backoffAndSleep(getClientParams().getRequestDelay());
 			}
-
-			log.info("Request attempt {} [{}]", retryNumber, requestUrl);
-
 			final HttpURLConnection urlConn = (HttpURLConnection) new URL(requestUrl).openConnection();
 			urlConn.setInstanceFollowRedirects(false);
 			urlConn.setReadTimeout(getClientParams().getReadTimeOut() * 1000);
@@ -120,8 +115,9 @@ public class HttpConnector2 {
 					urlConn.addRequestProperty(headerEntry.getKey(), headerEntry.getValue());
 				}
 			}
-
-			logHeaderFields(urlConn);
+			if (log.isDebugEnabled()) {
+				logHeaderFields(urlConn);
+			}
 
 			int retryAfter = obtainRetryAfter(urlConn.getHeaderFields());
 			String rateLimit = urlConn.getHeaderField(Constants.HTTPHEADER_IETF_DRAFT_RATELIMIT_LIMIT);
@@ -136,7 +132,9 @@ public class HttpConnector2 {
 			}
 
 			if (is2xx(urlConn.getResponseCode())) {
-				return getInputStream(urlConn, start);
+				input = urlConn.getInputStream();
+				responseType = urlConn.getContentType();
+				return input;
 			}
 			if (is3xx(urlConn.getResponseCode())) {
 				// REDIRECTS
@@ -146,7 +144,6 @@ public class HttpConnector2 {
 					.put(
 						REPORT_PREFIX + urlConn.getResponseCode(),
 						String.format("Moved to: %s", newUrl));
-				logRequestTime(start);
 				urlConn.disconnect();
 				if (retryAfter > 0) {
 					backoffAndSleep(retryAfter);
@@ -162,50 +159,26 @@ public class HttpConnector2 {
 						if (retryAfter > 0) {
 							log
 								.warn(
-									"waiting and repeating request after suggested retry-after {} sec for URL {}",
-									retryAfter, requestUrl);
+									"{} - waiting and repeating request after suggested retry-after {} sec.",
+									requestUrl, retryAfter);
 							backoffAndSleep(retryAfter * 1000);
 						} else {
 							log
 								.warn(
-									"waiting and repeating request after default delay of {} sec for URL {}",
-									getClientParams().getRetryDelay(), requestUrl);
-							backoffAndSleep(retryNumber * getClientParams().getRetryDelay());
+									"{} - waiting and repeating request after default delay of {} sec.",
+									requestUrl, getClientParams().getRetryDelay());
+							backoffAndSleep(retryNumber * getClientParams().getRetryDelay() * 1000);
 						}
 						report.put(REPORT_PREFIX + urlConn.getResponseCode(), requestUrl);
-
-						logRequestTime(start);
-
 						urlConn.disconnect();
-
 						return attemptDownload(requestUrl, retryNumber + 1, report);
-					case 422: // UNPROCESSABLE ENTITY
-						report.put(REPORT_PREFIX + urlConn.getResponseCode(), requestUrl);
-						log.warn("waiting and repeating request after 10 sec for URL {}", requestUrl);
-						backoffAndSleep(10000);
-						urlConn.disconnect();
-						logRequestTime(start);
-						try {
-							return getInputStream(urlConn, start);
-						} catch (IOException e) {
-							log
-								.error(
-									"server returned 422 and got IOException accessing the response body from URL {}",
-									requestUrl);
-							log.error("IOException:", e);
-							return attemptDownload(requestUrl, retryNumber + 1, report);
-						}
 					default:
-						log.error("gor error {} from URL: {}", urlConn.getResponseCode(), urlConn.getURL());
-						log.error("response message: {}", urlConn.getResponseMessage());
 						report
 							.put(
 								REPORT_PREFIX + urlConn.getResponseCode(),
 								String
 									.format(
 										"%s Error: %s", requestUrl, urlConn.getResponseMessage()));
-						logRequestTime(start);
-						urlConn.disconnect();
 						throw new CollectorException(urlConn.getResponseCode() + " error " + report);
 				}
 			}
@@ -226,27 +199,13 @@ public class HttpConnector2 {
 		}
 	}
 
-	private InputStream getInputStream(HttpURLConnection urlConn, long start) throws IOException {
-		InputStream input = urlConn.getInputStream();
-		responseType = urlConn.getContentType();
-		logRequestTime(start);
-		return input;
-	}
-
-	private static void logRequestTime(long start) {
-		log
-			.info(
-				"request time elapsed: {}sec",
-				TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - start));
-	}
-
 	private void logHeaderFields(final HttpURLConnection urlConn) throws IOException {
-		log.info("Response: {} - {}", urlConn.getResponseCode(), urlConn.getResponseMessage());
+		log.debug("StatusCode: {}", urlConn.getResponseMessage());
 
 		for (Map.Entry<String, List<String>> e : urlConn.getHeaderFields().entrySet()) {
 			if (e.getKey() != null) {
 				for (String v : e.getValue()) {
-					log.info("  key: {} - value: {}", e.getKey(), v);
+					log.debug("  key: {} - value: {}", e.getKey(), v);
 				}
 			}
 		}
@@ -266,7 +225,7 @@ public class HttpConnector2 {
 		for (String key : headerMap.keySet()) {
 			if ((key != null) && key.equalsIgnoreCase(HttpHeaders.RETRY_AFTER) && (!headerMap.get(key).isEmpty())
 				&& NumberUtils.isCreatable(headerMap.get(key).get(0))) {
-				return Integer.parseInt(headerMap.get(key).get(0));
+				return Integer.parseInt(headerMap.get(key).get(0)) + 10;
 			}
 		}
 		return -1;

dhp-common/src/main/java/eu/dnetlib/dhp/oozie/RunSQLSparkJob.java

@@ -1,77 +0,0 @@
-
-package eu.dnetlib.dhp.oozie;
-
-import static eu.dnetlib.dhp.common.SparkSessionSupport.runWithSparkHiveSession;
-
-import java.net.URL;
-import java.nio.charset.StandardCharsets;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Optional;
-
-import org.apache.commons.lang3.time.DurationFormatUtils;
-import org.apache.commons.text.StringSubstitutor;
-import org.apache.spark.SparkConf;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.io.Resources;
-
-import eu.dnetlib.dhp.application.ArgumentApplicationParser;
-
-public class RunSQLSparkJob {
-	private static final Logger log = LoggerFactory.getLogger(RunSQLSparkJob.class);
-
-	private final ArgumentApplicationParser parser;
-
-	public RunSQLSparkJob(ArgumentApplicationParser parser) {
-		this.parser = parser;
-	}
-
-	public static void main(String[] args) throws Exception {
-
-		Map<String, String> params = new HashMap<>();
-		for (int i = 0; i < args.length - 1; i++) {
-			if (args[i].startsWith("--")) {
-				params.put(args[i].substring(2), args[++i]);
-			}
-		}
-
-		/*
-		 * String jsonConfiguration = IOUtils .toString( Objects .requireNonNull( RunSQLSparkJob.class
-		 * .getResourceAsStream( "/eu/dnetlib/dhp/oozie/run_sql_parameters.json"))); final ArgumentApplicationParser
-		 * parser = new ArgumentApplicationParser(jsonConfiguration); parser.parseArgument(args);
-		 */
-
-		Boolean isSparkSessionManaged = Optional
-			.ofNullable(params.get("isSparkSessionManaged"))
-			.map(Boolean::valueOf)
-			.orElse(Boolean.TRUE);
-		log.info("isSparkSessionManaged: {}", isSparkSessionManaged);
-
-		URL url = com.google.common.io.Resources.getResource(params.get("sql"));
-		String raw_sql = Resources.toString(url, StandardCharsets.UTF_8);
-
-		String sql = StringSubstitutor.replace(raw_sql, params);
-		log.info("sql: {}", sql);
-
-		SparkConf conf = new SparkConf();
-		conf.set("hive.metastore.uris", params.get("hiveMetastoreUris"));
-
-		runWithSparkHiveSession(
-			conf,
-			isSparkSessionManaged,
-			spark -> {
-				for (String statement : sql.split(";\\s*/\\*\\s*EOS\\s*\\*/\\s*")) {
-					log.info("executing: {}", statement);
-					long startTime = System.currentTimeMillis();
-					spark.sql(statement).show();
-					log
-						.info(
-							"executed in {}",
-							DurationFormatUtils.formatDuration(System.currentTimeMillis() - startTime, "HH:mm:ss.S"));
-				}
-			});
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java

@@ -312,8 +312,7 @@ public class GraphCleaningFunctions extends CleaningFunctions {
 		}
 
 		if (value instanceof Datasource) {
-			final Datasource d = (Datasource) value;
-			return Objects.nonNull(d.getOfficialname()) && StringUtils.isNotBlank(d.getOfficialname().getValue());
+			// nothing to evaluate here
 		} else if (value instanceof Project) {
 			final Project p = (Project) value;
 			return Objects.nonNull(p.getCode()) && StringUtils.isNotBlank(p.getCode().getValue());

dhp-common/src/main/resources/eu/dnetlib/dhp/oozie/run_sql_parameters.json

@@ -1,20 +0,0 @@
-[
-  {
-    "paramName": "issm",
-    "paramLongName": "isSparkSessionManaged",
-    "paramDescription": "when true will stop SparkSession after job execution",
-    "paramRequired": false
-  },
-  {
-    "paramName": "hmu",
-    "paramLongName": "hiveMetastoreUris",
-    "paramDescription": "the hive metastore uris",
-    "paramRequired": true
-  },
-  {
-    "paramName": "sql",
-    "paramLongName": "sql",
-    "paramDescription": "sql script to execute",
-    "paramRequired": true
-  }
-]

dhp-common/src/test/java/eu/dnetlib/dhp/common/api/ZenodoAPIClientTest.java

@@ -0,0 +1,109 @@
+
+package eu.dnetlib.dhp.common.api;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+@Disabled
+class ZenodoAPIClientTest {
+
+	private final String URL_STRING = "https://sandbox.zenodo.org/api/deposit/depositions";
+	private final String ACCESS_TOKEN = "";
+
+	private final String CONCEPT_REC_ID = "657113";
+
+	private final String depositionId = "674915";
+
+	@Test
+	void testUploadOldDeposition() throws IOException, MissingConceptDoiException {
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+		Assertions.assertEquals(200, client.uploadOpenDeposition(depositionId));
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/COVID-19.json.gz")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "COVID-19.json.gz"));
+
+		String metadata = IOUtils.toString(getClass().getResourceAsStream("/eu/dnetlib/dhp/common/api/metadata.json"));
+
+		Assertions.assertEquals(200, client.sendMretadata(metadata));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+	@Test
+	void testNewDeposition() throws IOException {
+
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+		Assertions.assertEquals(201, client.newDeposition());
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/COVID-19.json.gz")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "COVID-19.json.gz"));
+
+		String metadata = IOUtils.toString(getClass().getResourceAsStream("/eu/dnetlib/dhp/common/api/metadata.json"));
+
+		Assertions.assertEquals(200, client.sendMretadata(metadata));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+	@Test
+	void testNewVersionNewName() throws IOException, MissingConceptDoiException {
+
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+
+		Assertions.assertEquals(201, client.newVersion(CONCEPT_REC_ID));
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/newVersion")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "newVersion_deposition"));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+	@Test
+	void testNewVersionOldName() throws IOException, MissingConceptDoiException {
+
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+
+		Assertions.assertEquals(201, client.newVersion(CONCEPT_REC_ID));
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/newVersion2")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "newVersion_deposition"));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+}

dhp-workflows/dhp-actionmanager/src/main/java/eu/dnetlib/dhp/actionmanager/promote/PromoteAction.java

@@ -1,39 +0,0 @@
-/*
- * Copyright (c) 2024.
- * SPDX-FileCopyrightText: © 2023 Consiglio Nazionale delle Ricerche
- * SPDX-License-Identifier: AGPL-3.0-or-later
- */
-
-package eu.dnetlib.dhp.actionmanager.promote;
-
-/** Encodes the Actionset promotion strategies */
-public class PromoteAction {
-
-	/** The supported actionset promotion strategies
-	 *
-	 * ENRICH: promotes only records in the actionset matching another record in the
-	 *  graph and enriches them applying the given MergeAndGet strategy
-	 * UPSERT: promotes all the records in an actionset, matching records are updated
-	 *  using the given MergeAndGet strategy, the non-matching record as inserted as they are.
-	 */
-	public enum Strategy {
-		ENRICH, UPSERT
-	}
-
-	/**
-	 * Returns the string representation of the join type implementing the given PromoteAction.
-	 *
-	 * @param strategy the strategy to be used to promote the Actionset contents
-	 * @return the join type used to implement the promotion strategy
-	 */
-	public static String joinTypeForStrategy(PromoteAction.Strategy strategy) {
-		switch (strategy) {
-			case ENRICH:
-				return "left_outer";
-			case UPSERT:
-				return "full_outer";
-			default:
-				throw new IllegalStateException("unsupported PromoteAction: " + strategy.toString());
-		}
-	}
-}

dhp-workflows/dhp-actionmanager/src/main/java/eu/dnetlib/dhp/actionmanager/promote/PromoteActionPayloadForGraphTableJob.java

@@ -67,9 +67,8 @@ public class PromoteActionPayloadForGraphTableJob {
 		String outputGraphTablePath = parser.get("outputGraphTablePath");
 		logger.info("outputGraphTablePath: {}", outputGraphTablePath);
 
-		MergeAndGet.Strategy mergeAndGetStrategy = MergeAndGet.Strategy
-			.valueOf(parser.get("mergeAndGetStrategy").toUpperCase());
-		logger.info("mergeAndGetStrategy: {}", mergeAndGetStrategy);
+		MergeAndGet.Strategy strategy = MergeAndGet.Strategy.valueOf(parser.get("mergeAndGetStrategy").toUpperCase());
+		logger.info("strategy: {}", strategy);
 
 		Boolean shouldGroupById = Optional
 			.ofNullable(parser.get("shouldGroupById"))
@@ -77,12 +76,6 @@ public class PromoteActionPayloadForGraphTableJob {
 			.orElse(true);
 		logger.info("shouldGroupById: {}", shouldGroupById);
 
-		PromoteAction.Strategy promoteActionStrategy = Optional
-			.ofNullable(parser.get("promoteActionStrategy"))
-			.map(PromoteAction.Strategy::valueOf)
-			.orElse(PromoteAction.Strategy.UPSERT);
-		logger.info("promoteActionStrategy: {}", promoteActionStrategy);
-
 		@SuppressWarnings("unchecked")
 		Class<? extends Oaf> rowClazz = (Class<? extends Oaf>) Class.forName(graphTableClassName);
 		@SuppressWarnings("unchecked")
@@ -104,8 +97,7 @@ public class PromoteActionPayloadForGraphTableJob {
 					inputGraphTablePath,
 					inputActionPayloadPath,
 					outputGraphTablePath,
-					mergeAndGetStrategy,
-					promoteActionStrategy,
+					strategy,
 					rowClazz,
 					actionPayloadClazz,
 					shouldGroupById);
@@ -132,16 +124,14 @@ public class PromoteActionPayloadForGraphTableJob {
 		String inputGraphTablePath,
 		String inputActionPayloadPath,
 		String outputGraphTablePath,
-		MergeAndGet.Strategy mergeAndGetStrategy,
-		PromoteAction.Strategy promoteActionStrategy,
+		MergeAndGet.Strategy strategy,
 		Class<G> rowClazz,
 		Class<A> actionPayloadClazz, Boolean shouldGroupById) {
 		Dataset<G> rowDS = readGraphTable(spark, inputGraphTablePath, rowClazz);
 		Dataset<A> actionPayloadDS = readActionPayload(spark, inputActionPayloadPath, actionPayloadClazz);
 
 		Dataset<G> result = promoteActionPayloadForGraphTable(
-			rowDS, actionPayloadDS, mergeAndGetStrategy, promoteActionStrategy, rowClazz, actionPayloadClazz,
-			shouldGroupById)
+			rowDS, actionPayloadDS, strategy, rowClazz, actionPayloadClazz, shouldGroupById)
 				.map((MapFunction<G, G>) value -> value, Encoders.bean(rowClazz));
 
 		saveGraphTable(result, outputGraphTablePath);
@@ -193,8 +183,7 @@ public class PromoteActionPayloadForGraphTableJob {
 	private static <G extends Oaf, A extends Oaf> Dataset<G> promoteActionPayloadForGraphTable(
 		Dataset<G> rowDS,
 		Dataset<A> actionPayloadDS,
-		MergeAndGet.Strategy mergeAndGetStrategy,
-		PromoteAction.Strategy promoteActionStrategy,
+		MergeAndGet.Strategy strategy,
 		Class<G> rowClazz,
 		Class<A> actionPayloadClazz,
 		Boolean shouldGroupById) {
@@ -206,9 +195,8 @@ public class PromoteActionPayloadForGraphTableJob {
 
 		SerializableSupplier<Function<G, String>> rowIdFn = ModelSupport::idFn;
 		SerializableSupplier<Function<A, String>> actionPayloadIdFn = ModelSupport::idFn;
-		SerializableSupplier<BiFunction<G, A, G>> mergeRowWithActionPayloadAndGetFn = MergeAndGet
-			.functionFor(mergeAndGetStrategy);
-		SerializableSupplier<BiFunction<G, G, G>> mergeRowsAndGetFn = MergeAndGet.functionFor(mergeAndGetStrategy);
+		SerializableSupplier<BiFunction<G, A, G>> mergeRowWithActionPayloadAndGetFn = MergeAndGet.functionFor(strategy);
+		SerializableSupplier<BiFunction<G, G, G>> mergeRowsAndGetFn = MergeAndGet.functionFor(strategy);
 		SerializableSupplier<G> zeroFn = zeroFn(rowClazz);
 		SerializableSupplier<Function<G, Boolean>> isNotZeroFn = PromoteActionPayloadForGraphTableJob::isNotZeroFnUsingIdOrSourceAndTarget;
 
@@ -219,7 +207,6 @@ public class PromoteActionPayloadForGraphTableJob {
 				rowIdFn,
 				actionPayloadIdFn,
 				mergeRowWithActionPayloadAndGetFn,
-				promoteActionStrategy,
 				rowClazz,
 				actionPayloadClazz);
 

dhp-workflows/dhp-actionmanager/src/main/java/eu/dnetlib/dhp/actionmanager/promote/PromoteActionPayloadFunctions.java

@@ -34,7 +34,6 @@ public class PromoteActionPayloadFunctions {
 	 * @param rowIdFn Function used to get the id of graph table row
 	 * @param actionPayloadIdFn Function used to get id of action payload instance
 	 * @param mergeAndGetFn Function used to merge graph table row and action payload instance
-	 * @param promoteActionStrategy the Actionset promotion strategy
 	 * @param rowClazz Class of graph table
 	 * @param actionPayloadClazz Class of action payload
 	 * @param <G> Type of graph table row
@@ -47,7 +46,6 @@ public class PromoteActionPayloadFunctions {
 		SerializableSupplier<Function<G, String>> rowIdFn,
 		SerializableSupplier<Function<A, String>> actionPayloadIdFn,
 		SerializableSupplier<BiFunction<G, A, G>> mergeAndGetFn,
-		PromoteAction.Strategy promoteActionStrategy,
 		Class<G> rowClazz,
 		Class<A> actionPayloadClazz) {
 		if (!isSubClass(rowClazz, actionPayloadClazz)) {
@@ -63,7 +61,7 @@ public class PromoteActionPayloadFunctions {
 			.joinWith(
 				actionPayloadWithIdDS,
 				rowWithIdDS.col("_1").equalTo(actionPayloadWithIdDS.col("_1")),
-				PromoteAction.joinTypeForStrategy(promoteActionStrategy))
+				"full_outer")
 			.map(
 				(MapFunction<Tuple2<Tuple2<String, G>, Tuple2<String, A>>, G>) value -> {
 					Optional<G> rowOpt = Optional.ofNullable(value._1()).map(Tuple2::_2);

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/promote/promote_action_payload_for_graph_table_input_parameters.json

@@ -41,12 +41,6 @@
     "paramDescription": "strategy for merging graph table objects with action payload instances, MERGE_FROM_AND_GET or SELECT_NEWER_AND_GET",
     "paramRequired": true
   },
-  {
-    "paramName": "pas",
-    "paramLongName": "promoteActionStrategy",
-    "paramDescription": "strategy for promoting the actionset contents into the graph tables, ENRICH or UPSERT (default)",
-    "paramRequired": false
-  },
   {
     "paramName": "sgid",
     "paramLongName": "shouldGroupById",

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/dataset/oozie_app/workflow.xml

@@ -115,7 +115,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Dataset</arg>
             <arg>--outputGraphTablePath</arg><arg>${workingDir}/dataset</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="DecisionPromoteResultActionPayloadForDatasetTable"/>
@@ -168,7 +167,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Result</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/dataset</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="End"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/datasource/oozie_app/workflow.xml

@@ -106,7 +106,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Datasource</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/datasource</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
         </spark>
         <ok to="End"/>
         <error to="Kill"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/organization/oozie_app/workflow.xml

@@ -106,7 +106,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Organization</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/organization</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
         </spark>
         <ok to="End"/>
         <error to="Kill"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/otherresearchproduct/oozie_app/workflow.xml

@@ -114,7 +114,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.OtherResearchProduct</arg>
             <arg>--outputGraphTablePath</arg><arg>${workingDir}/otherresearchproduct</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="DecisionPromoteResultActionPayloadForOtherResearchProductTable"/>
@@ -167,7 +166,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Result</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/otherresearchproduct</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="End"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/project/oozie_app/workflow.xml

@@ -106,7 +106,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Project</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/project</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
         </spark>
         <ok to="End"/>
         <error to="Kill"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/publication/oozie_app/workflow.xml

@@ -115,7 +115,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Publication</arg>
             <arg>--outputGraphTablePath</arg><arg>${workingDir}/publication</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="DecisionPromoteResultActionPayloadForPublicationTable"/>
@@ -168,7 +167,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Result</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/publication</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="End"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/relation/oozie_app/workflow.xml

@@ -107,7 +107,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Relation</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/relation</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
         </spark>
         <ok to="End"/>
         <error to="Kill"/>

dhp-workflows/dhp-actionmanager/src/main/resources/eu/dnetlib/dhp/actionmanager/wf/software/oozie_app/workflow.xml

@@ -114,7 +114,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Software</arg>
             <arg>--outputGraphTablePath</arg><arg>${workingDir}/software</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="DecisionPromoteResultActionPayloadForSoftwareTable"/>
@@ -167,7 +166,6 @@
             <arg>--actionPayloadClassName</arg><arg>eu.dnetlib.dhp.schema.oaf.Result</arg>
             <arg>--outputGraphTablePath</arg><arg>${outputGraphRootPath}/software</arg>
             <arg>--mergeAndGetStrategy</arg><arg>${mergeAndGetStrategy}</arg>
-            <arg>--promoteActionStrategy</arg><arg>${promoteActionStrategy}</arg>
             <arg>--shouldGroupById</arg><arg>${shouldGroupById}</arg>
         </spark>
         <ok to="End"/>

dhp-workflows/dhp-actionmanager/src/test/java/eu/dnetlib/dhp/actionmanager/promote/PromoteActionPayloadFunctionsTest.java

@@ -54,7 +54,7 @@ public class PromoteActionPayloadFunctionsTest {
 				RuntimeException.class,
 				() -> PromoteActionPayloadFunctions
 					.joinGraphTableWithActionPayloadAndMerge(
-						null, null, null, null, null, null, OafImplSubSub.class, OafImpl.class));
+						null, null, null, null, null, OafImplSubSub.class, OafImpl.class));
 		}
 
 		@Test
@@ -104,7 +104,6 @@ public class PromoteActionPayloadFunctionsTest {
 					rowIdFn,
 					actionPayloadIdFn,
 					mergeAndGetFn,
-					PromoteAction.Strategy.UPSERT,
 					OafImplSubSub.class,
 					OafImplSubSub.class)
 				.collectAsList();
@@ -184,7 +183,6 @@ public class PromoteActionPayloadFunctionsTest {
 					rowIdFn,
 					actionPayloadIdFn,
 					mergeAndGetFn,
-					PromoteAction.Strategy.UPSERT,
 					OafImplSubSub.class,
 					OafImplSub.class)
 				.collectAsList();

dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/collection/CollectorWorker.java

@@ -19,7 +19,6 @@ import org.slf4j.LoggerFactory;
 import eu.dnetlib.dhp.aggregation.common.ReporterCallback;
 import eu.dnetlib.dhp.aggregation.common.ReportingJob;
 import eu.dnetlib.dhp.collection.plugin.CollectorPlugin;
-import eu.dnetlib.dhp.collection.plugin.base.BaseCollectorPlugin;
 import eu.dnetlib.dhp.collection.plugin.file.FileCollectorPlugin;
 import eu.dnetlib.dhp.collection.plugin.file.FileGZipCollectorPlugin;
 import eu.dnetlib.dhp.collection.plugin.mongodb.MDStoreCollectorPlugin;
@@ -121,8 +120,6 @@ public class CollectorWorker extends ReportingJob {
 				return new FileCollectorPlugin(fileSystem);
 			case fileGzip:
 				return new FileGZipCollectorPlugin(fileSystem);
-			case baseDump:
-				return new BaseCollectorPlugin(this.fileSystem);
 			case other:
 				final CollectorPlugin.NAME.OTHER_NAME plugin = Optional
 					.ofNullable(api.getParams().get("other_plugin_type"))

dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/collection/plugin/CollectorPlugin.java

@@ -10,8 +10,7 @@ import eu.dnetlib.dhp.common.collection.CollectorException;
 public interface CollectorPlugin {
 
 	enum NAME {
-
-		oai, other, rest_json2xml, file, fileGzip, baseDump;
+		oai, other, rest_json2xml, file, fileGzip;
 
 		public enum OTHER_NAME {
 			mdstore_mongodb_dump, mdstore_mongodb

dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/collection/plugin/base/BaseCollectorIterator.java

@@ -1,171 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import java.io.BufferedInputStream;
-import java.io.ByteArrayInputStream;
-import java.io.InputStream;
-import java.io.StringWriter;
-import java.util.Iterator;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.LinkedBlockingQueue;
-
-import javax.xml.stream.XMLEventReader;
-import javax.xml.stream.XMLEventWriter;
-import javax.xml.stream.XMLInputFactory;
-import javax.xml.stream.XMLOutputFactory;
-import javax.xml.stream.events.EndElement;
-import javax.xml.stream.events.StartElement;
-import javax.xml.stream.events.XMLEvent;
-
-import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
-import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
-import org.apache.commons.compress.compressors.CompressorInputStream;
-import org.apache.commons.compress.compressors.CompressorStreamFactory;
-import org.apache.commons.io.IOUtils;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import eu.dnetlib.dhp.common.aggregation.AggregatorReport;
-
-public class BaseCollectorIterator implements Iterator<String> {
-
-	private String nextElement;
-
-	private final BlockingQueue<String> queue = new LinkedBlockingQueue<>(100);
-
-	private static final Logger log = LoggerFactory.getLogger(BaseCollectorIterator.class);
-
-	private static final String END_ELEM = "__END__";
-
-	public BaseCollectorIterator(final FileSystem fs, final Path filePath, final AggregatorReport report) {
-		new Thread(() -> importHadoopFile(fs, filePath, report)).start();
-		try {
-			this.nextElement = this.queue.take();
-		} catch (final InterruptedException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	protected BaseCollectorIterator(final String resourcePath, final AggregatorReport report) {
-		new Thread(() -> importTestFile(resourcePath, report)).start();
-		try {
-			this.nextElement = this.queue.take();
-		} catch (final InterruptedException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	@Override
-	public synchronized boolean hasNext() {
-		return (this.nextElement != null) & !END_ELEM.equals(this.nextElement);
-	}
-
-	@Override
-	public synchronized String next() {
-		try {
-			return END_ELEM.equals(this.nextElement) ? null : this.nextElement;
-		} finally {
-			try {
-				this.nextElement = this.queue.take();
-			} catch (final InterruptedException e) {
-				throw new RuntimeException(e);
-			}
-		}
-
-	}
-
-	private void importHadoopFile(final FileSystem fs, final Path filePath, final AggregatorReport report) {
-		log.info("I start to read the TAR stream");
-
-		try (InputStream origInputStream = fs.open(filePath);
-			final TarArchiveInputStream tarInputStream = new TarArchiveInputStream(origInputStream)) {
-			importTarStream(tarInputStream, report);
-		} catch (final Throwable e) {
-			throw new RuntimeException("Error processing BASE records", e);
-		}
-	}
-
-	private void importTestFile(final String resourcePath, final AggregatorReport report) {
-		try (final InputStream origInputStream = BaseCollectorIterator.class.getResourceAsStream(resourcePath);
-			final TarArchiveInputStream tarInputStream = new TarArchiveInputStream(origInputStream)) {
-			importTarStream(tarInputStream, report);
-		} catch (final Throwable e) {
-			throw new RuntimeException("Error processing BASE records", e);
-		}
-	}
-
-	private void importTarStream(final TarArchiveInputStream tarInputStream, final AggregatorReport report) {
-		long count = 0;
-
-		final XMLInputFactory xmlInputFactory = XMLInputFactory.newInstance();
-		final XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance();
-
-		try {
-			TarArchiveEntry entry;
-			while ((entry = (TarArchiveEntry) tarInputStream.getNextEntry()) != null) {
-				final String name = entry.getName();
-
-				if (!entry.isDirectory() && name.contains("ListRecords") && name.endsWith(".bz2")) {
-
-					log.info("Processing file (BZIP): " + name);
-
-					final byte[] bzipData = new byte[(int) entry.getSize()];
-					IOUtils.readFully(tarInputStream, bzipData);
-
-					try (InputStream bzipIs = new ByteArrayInputStream(bzipData);
-						final BufferedInputStream bzipBis = new BufferedInputStream(bzipIs);
-						final CompressorInputStream bzipInput = new CompressorStreamFactory()
-							.createCompressorInputStream(bzipBis)) {
-
-						final XMLEventReader reader = xmlInputFactory.createXMLEventReader(bzipInput);
-
-						XMLEventWriter eventWriter = null;
-						StringWriter xmlWriter = null;
-
-						while (reader.hasNext()) {
-							final XMLEvent nextEvent = reader.nextEvent();
-
-							if (nextEvent.isStartElement()) {
-								final StartElement startElement = nextEvent.asStartElement();
-								if ("record".equals(startElement.getName().getLocalPart())) {
-									xmlWriter = new StringWriter();
-									eventWriter = xmlOutputFactory.createXMLEventWriter(xmlWriter);
-								}
-							}
-
-							if (eventWriter != null) {
-								eventWriter.add(nextEvent);
-							}
-
-							if (nextEvent.isEndElement()) {
-								final EndElement endElement = nextEvent.asEndElement();
-								if ("record".equals(endElement.getName().getLocalPart())) {
-									eventWriter.flush();
-									eventWriter.close();
-
-									this.queue.put(xmlWriter.toString());
-
-									eventWriter = null;
-									xmlWriter = null;
-									count++;
-								}
-							}
-
-						}
-					}
-				}
-			}
-
-			this.queue.put(END_ELEM); // TO INDICATE THE END OF THE QUEUE
-		} catch (final Throwable e) {
-			log.error("Error processing BASE records", e);
-			report.put(e.getClass().getName(), e.getMessage());
-			throw new RuntimeException("Error processing BASE records", e);
-		} finally {
-			log.info("Total records (written in queue): " + count);
-		}
-	}
-
-}

dhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/collection/plugin/base/BaseCollectorPlugin.java

@@ -1,159 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import java.io.IOException;
-import java.sql.SQLException;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.Optional;
-import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
-import java.util.stream.Stream;
-import java.util.stream.StreamSupport;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.dom4j.Document;
-import org.dom4j.DocumentException;
-import org.dom4j.DocumentHelper;
-import org.dom4j.Node;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import eu.dnetlib.dhp.collection.ApiDescriptor;
-import eu.dnetlib.dhp.collection.plugin.CollectorPlugin;
-import eu.dnetlib.dhp.collection.plugin.file.AbstractSplittedRecordPlugin;
-import eu.dnetlib.dhp.common.DbClient;
-import eu.dnetlib.dhp.common.aggregation.AggregatorReport;
-import eu.dnetlib.dhp.common.collection.CollectorException;
-
-public class BaseCollectorPlugin implements CollectorPlugin {
-
-	private final FileSystem fs;
-
-	private static final Logger log = LoggerFactory.getLogger(AbstractSplittedRecordPlugin.class);
-
-	// MAPPING AND FILTERING ARE DEFINED HERE:
-	// https://docs.google.com/document/d/1Aj-ZAV11b44MCrAAUCPiS2TUlXb6PnJEu1utCMAcCOU/edit
-
-	public BaseCollectorPlugin(final FileSystem fs) {
-		this.fs = fs;
-	}
-
-	@Override
-	public Stream<String> collect(final ApiDescriptor api, final AggregatorReport report) throws CollectorException {
-		// the path of the dump file on HDFS
-		// http://oai.base-search.net/initial_load/base_oaipmh_dump-current.tar
-		// it could be downloaded from iis-cdh5-test-gw.ocean.icm.edu.pl and then copied on HDFS
-		final Path filePath = Optional
-			.ofNullable(api.getBaseUrl())
-			.map(Path::new)
-			.orElseThrow(() -> new CollectorException("missing baseUrl"));
-
-		// get the parameters for the connection to the OpenAIRE database.
-		// the database is used to obtain the list of the datasources that the plugin will collect
-		final String dbUrl = api.getParams().get("dbUrl");
-		final String dbUser = api.getParams().get("dbUser");
-		final String dbPassword = api.getParams().get("dbPassword");
-
-		// the types(comma separated, empty value for all) that the plugin will collect,
-		// the types should be expressed in the format of the normalized types of BASE (for example 1,121,...)
-		final String acceptedNormTypesString = api.getParams().get("acceptedNormTypes");
-
-		log.info("baseUrl: {}", filePath);
-		log.info("dbUrl: {}", dbUrl);
-		log.info("dbUser: {}", dbUser);
-		log.info("dbPassword: {}", "***");
-		log.info("acceptedNormTypes: {}", acceptedNormTypesString);
-
-		try {
-			if (!this.fs.exists(filePath)) {
-				throw new CollectorException("path does not exist: " + filePath);
-			}
-		} catch (final Throwable e) {
-			throw new CollectorException(e);
-		}
-
-		final Set<String> acceptedOpendoarIds = findAcceptedOpendoarIds(dbUrl, dbUser, dbPassword);
-
-		final Set<String> acceptedNormTypes = new HashSet<>();
-		if (StringUtils.isNotBlank(acceptedNormTypesString)) {
-			for (final String s : StringUtils.split(acceptedNormTypesString, ",")) {
-				if (StringUtils.isNotBlank(s)) {
-					acceptedNormTypes.add(s.trim());
-				}
-			}
-		}
-
-		final Iterator<String> iterator = new BaseCollectorIterator(this.fs, filePath, report);
-		final Spliterator<String> spliterator = Spliterators.spliteratorUnknownSize(iterator, Spliterator.ORDERED);
-		return StreamSupport
-			.stream(spliterator, false)
-			.filter(doc -> filterXml(doc, acceptedOpendoarIds, acceptedNormTypes));
-	}
-
-	private Set<String> findAcceptedOpendoarIds(final String dbUrl, final String dbUser, final String dbPassword)
-		throws CollectorException {
-		final Set<String> accepted = new HashSet<>();
-
-		try (final DbClient dbClient = new DbClient(dbUrl, dbUser, dbPassword)) {
-
-			final String sql = IOUtils
-				.toString(
-					getClass().getResourceAsStream("/eu/dnetlib/dhp/collection/plugin/base/sql/opendoar-accepted.sql"));
-
-			dbClient.processResults(sql, row -> {
-				try {
-					final String dsId = row.getString("id");
-					log.info("Accepted Datasource: " + dsId);
-					accepted.add(dsId);
-				} catch (final SQLException e) {
-					log.error("Error in SQL", e);
-					throw new RuntimeException("Error in SQL", e);
-				}
-			});
-
-		} catch (final IOException e) {
-			log.error("Error accessong SQL", e);
-			throw new CollectorException("Error accessong SQL", e);
-		}
-
-		log.info("Accepted Datasources (TOTAL): " + accepted.size());
-
-		return accepted;
-	}
-
-	protected static boolean filterXml(final String xml,
-		final Set<String> acceptedOpendoarIds,
-		final Set<String> acceptedNormTypes) {
-		try {
-
-			final Document doc = DocumentHelper.parseText(xml);
-
-			final String id = doc.valueOf("//*[local-name()='collection']/@opendoar_id").trim();
-
-			if (StringUtils.isBlank(id) || !acceptedOpendoarIds.contains("opendoar____::" + id)) {
-				return false;
-			}
-
-			if (acceptedNormTypes.isEmpty()) {
-				return true;
-			}
-
-			for (final Object s : doc.selectNodes("//*[local-name()='typenorm']")) {
-				if (acceptedNormTypes.contains(((Node) s).getText().trim())) {
-					return true;
-				}
-			}
-
-			return false;
-		} catch (final DocumentException e) {
-			log.error("Error parsing document", e);
-			throw new RuntimeException("Error parsing document", e);
-		}
-	}
-
-}

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/collection/plugin/base/sql/base.sql

@@ -1,114 +0,0 @@
-BEGIN;
-
-INSERT INTO dsm_services(
-	_dnet_resource_identifier_, 
-	id,
-	officialname,
-	englishname,
-	namespaceprefix,
-	websiteurl,
-	logourl,
-	platform,
-	contactemail,
-	collectedfrom,
-	provenanceaction,
-	_typology_to_remove_,
-	eosc_type,
-	eosc_datasource_type,
-	research_entity_types,
-	thematic
-) VALUES (
-	'openaire____::base_search',
-	'openaire____::base_search',
-	'Bielefeld Academic Search Engine (BASE)',
-	'Bielefeld Academic Search Engine (BASE)',
-	'base_search_',
-	'https://www.base-search.net',
-	'https://www.base-search.net/about/download/logo_224x57_white.gif',
-	'BASE',
-	'openaire-helpdesk@uni-bielefeld.de',
-	'infrastruct_::openaire',
-	'user:insert',
-	'aggregator::pubsrepository::unknown',
-	'Data Source',
-	'Aggregator',
-	ARRAY['Research Products'],
-	false
-);
-
-INSERT INTO dsm_service_organization(
-	_dnet_resource_identifier_,
-	organization,
-	service
-) VALUES (
-	'fairsharing_::org::214@@openaire____::base_search',
-	'fairsharing_::org::214',
-	'openaire____::base_search'
-);
-
-INSERT INTO dsm_api(
-	_dnet_resource_identifier_,
-	id,
-	service,
-	protocol,
-	baseurl,
-	metadata_identifier_path
-) VALUES (
-	'api_________::openaire____::base_search::dump',
-	'api_________::openaire____::base_search::dump',
-	'openaire____::base_search',
-	'baseDump',
-	'/user/michele.artini/base-import/base_oaipmh_dump-current.tar',
-	'//*[local-name()=''header'']/*[local-name()=''identifier'']'
-);
-
-
-INSERT INTO dsm_apiparams(
-	_dnet_resource_identifier_, 
-	api, 
-	param, 
-	value
-) VALUES (
-	'api_________::openaire____::base_search::dump@@dbUrl',
-	'api_________::openaire____::base_search::dump',
-	'dbUrl',
-	'jdbc:postgresql://postgresql.services.openaire.eu:5432/dnet_openaireplus'
-);
-
-INSERT INTO dsm_apiparams(
-	_dnet_resource_identifier_, 
-	api, 
-	param, 
-	value
-) VALUES (
-	'api_________::openaire____::base_search::dump@@dbUser',
-	'api_________::openaire____::base_search::dump',
-	'dbUser',
-	'dnet'
-);
-
-INSERT INTO dsm_apiparams(
-	_dnet_resource_identifier_, 
-	api, 
-	param, 
-	value
-) VALUES (
-	'api_________::openaire____::base_search::dump@@dbPassword',
-	'api_________::openaire____::base_search::dump',
-	'dbPassword',
-	'***'
-);
-
-INSERT INTO dsm_apiparams(
-	_dnet_resource_identifier_, 
-	api, 
-	param, 
-	value
-) VALUES (
-	'api_________::openaire____::base_search::dump@@acceptedNormTypes',
-	'api_________::openaire____::base_search::dump',
-	'acceptedNormTypes',
-	'1,11,111,121,13,14,15,18,181,182,183,1A,6,7'
-);
-
-COMMIT;

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/collection/plugin/base/sql/opendoar-accepted.sql

@@ -1,9 +0,0 @@
-select s.id as id 
-from dsm_services s 
-where collectedfrom = 'openaire____::opendoar' 
-and jurisdiction = 'Institutional'
-and s.id in (
-	select service from dsm_api where coalesce(compatibility_override, compatibility) = 'driver' or coalesce(compatibility_override, compatibility) = 'UNKNOWN'
-) and s.id not in (
-	select service from dsm_api where coalesce(compatibility_override, compatibility) like '%openaire%'
-);

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/collection/plugin/base/sql/opendoar-aggregation-status.sql

@@ -1,11 +0,0 @@
-select 
-	s.id           as id, 
-	s.jurisdiction as jurisdiction, 
-	array_remove(array_agg(a.id || ' (compliance: ' || coalesce(a.compatibility_override, a.compatibility, 'UNKNOWN') || ')@@@' || coalesce(a.last_collection_total, 0)), NULL) as aggregations
-from 
-	dsm_services s 
-	join dsm_api a on (s.id = a.service) 
-where 
-	collectedfrom = 'openaire____::opendoar'
-group by 
-	s.id;

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/collection/plugin/base/xml/base-types.vocabulary.xml

@@ -1,180 +0,0 @@
-<RESOURCE_PROFILE>
-	<HEADER>
-		<RESOURCE_IDENTIFIER value="c67911d6-9988-4a3b-b965-7d39bdd4a31d_Vm9jYWJ1bGFyeURTUmVzb3VyY2VzL1ZvY2FidWxhcnlEU1Jlc291cmNlVHlwZQ==" />
-		<RESOURCE_TYPE value="VocabularyDSResourceType" />
-		<RESOURCE_KIND value="VocabularyDSResources" />
-		<RESOURCE_URI value="" />
-		<DATE_OF_CREATION value="2024-02-13T11:15:48+00:00" />
-	</HEADER>
-	<BODY>
-		<CONFIGURATION>
-			<VOCABULARY_NAME code="base:normalized_types">base:normalized_types</VOCABULARY_NAME>
-			<VOCABULARY_DESCRIPTION>base:normalized_types</VOCABULARY_DESCRIPTION>
-			<TERMS>
-				<TERM native_name="Text" code="Text" english_name="Text" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="1" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Book" code="Book" english_name="Book" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="11" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Book part" code="Book part" english_name="Book part" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="111" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Journal/Newspaper" code="Journal/Newspaper" english_name="Journal/Newspaper" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="12" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Article contribution" code="Article contribution" english_name="Article contribution" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="121" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Other non-article" code="Other non-article" english_name="Other non-article" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="122" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Conference object" code="Conference object" english_name="Conference object" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="13" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Report" code="Report" english_name="Report" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="14" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Review" code="Review" english_name="Review" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="15" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Course material" code="Course material" english_name="Course material" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="16" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Lecture" code="Lecture" english_name="Lecture" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="17" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Thesis" code="Thesis" english_name="Thesis" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="18" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Bachelor's thesis" code="Bachelor's thesis" english_name="Bachelor's thesis" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="181" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Master's thesis" code="Master's thesis" english_name="Master's thesis" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="182" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Doctoral and postdoctoral thesis" code="Doctoral and postdoctoral thesis" english_name="Doctoral and postdoctoral thesis" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="183" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Manuscript" code="Manuscript" english_name="Manuscript" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="19" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Patent" code="Patent" english_name="Patent" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="1A" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Musical notation" code="Musical notation" english_name="Musical notation" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="2" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Map" code="Map" english_name="Map" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="3" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Audio" code="Audio" english_name="Audio" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="4" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Image/Video" code="Image/Video" english_name="Image/Video" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="5" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Still image" code="Still image" english_name="Still image" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="51" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Moving image/Video" code="Moving image/Video" english_name="Moving image/Video" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="52" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Software" code="Software" english_name="Software" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="6" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Dataset" code="Dataset" english_name="Dataset" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="7" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-				<TERM native_name="Unknown" code="Unknown" english_name="Unknown" encoding="BASE">
-					<SYNONYMS>
-						<SYNONYM term="F" encoding="BASE" />
-					</SYNONYMS>
-					<RELATIONS />
-				</TERM>
-
-			</TERMS>
-		</CONFIGURATION>
-		<STATUS>
-			<LAST_UPDATE value="2013-11-18T10:46:36Z" />
-		</STATUS>
-		<SECURITY_PARAMETERS>String</SECURITY_PARAMETERS>
-	</BODY>
-</RESOURCE_PROFILE>
-
-                

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/collection/plugin/base/xml/base2oaf.transformationRule.xml

@@ -1,302 +0,0 @@
-<RESOURCE_PROFILE>
-	<HEADER>
-		<RESOURCE_IDENTIFIER value="" />
-		<RESOURCE_TYPE value="TransformationRuleDSResourceType" />
-		<RESOURCE_KIND value="TransformationRuleDSResources" />
-		<RESOURCE_URI value="" />
-		<DATE_OF_CREATION value="2024-03-05T11:23:00+00:00" />
-	</HEADER>
-	<BODY>
-		<CONFIGURATION>
-			<SOURCE_METADATA_FORMAT interpretation="cleaned" layout="store" name="dc" />
-			<SINK_METADATA_FORMAT name="oaf_hbase" />
-			<IMPORTED />
-			<SCRIPT>
-				<TITLE>xslt_base2oaf_hadoop</TITLE>
-				<CODE>
-					<xsl:stylesheet xmlns:oaire="http://namespace.openaire.eu/schema/oaire/" xmlns:dateCleaner="http://eu/dnetlib/transform/dateISO"
-						xmlns:base_dc="http://oai.base-search.net/base_dc/"
-						xmlns:datacite="http://datacite.org/schema/kernel-4" xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-						xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:vocabulary="http://eu/dnetlib/transform/clean" xmlns:oaf="http://namespace.openaire.eu/oaf"
-						xmlns:oai="http://www.openarchives.org/OAI/2.0/" xmlns:dri="http://www.driver-repository.eu/namespace/dri" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dc="http://purl.org/dc/elements/1.1/"
-						exclude-result-prefixes="xsl vocabulary dateCleaner base_dc" version="2.0">
-						<xsl:param name="varOfficialName" />
-						<xsl:param name="varDataSourceId" />
-						<xsl:param name="varFP7" select="'corda_______::'" />
-						<xsl:param name="varH2020" select="'corda__h2020::'" />
-						<xsl:param name="repoCode" select="substring-before(//*[local-name() = 'header']/*[local-name()='recordIdentifier'], ':')" />
-						<xsl:param name="index" select="0" />
-						<xsl:param name="transDate" select="current-dateTime()" />
-
-						<xsl:template name="terminate">
-							<xsl:message terminate="yes">
-								record is not compliant, transformation is interrupted.
-							</xsl:message>
-						</xsl:template>
-
-						<xsl:template match="/">
-							<record>
-								<xsl:apply-templates select="//*[local-name() = 'header']" />
-
-
-<!-- TO EVALUATE
-base_dc:authod_id
-base_dc:authod_id/base_dc:creator_id
-base_dc:authod_id/base_dc:creator_name
-
-example:
-
-<dc:creator>ALBU, Svetlana</dc:creator>
-
-<base_dc:authod_id>
-	<base_dc:creator_name>ALBU, Svetlana</base_dc:creator_name>
-    <base_dc:creator_id>https://orcid.org/0000-0002-8648-950X</base_dc:creator_id>
-</base_dc:authod_id>
--->
-
-<!-- NOT USED 
-base_dc:global_id (I used oai:identifier)
-base_dc:collection/text()
-
-base_dc:continent
-base_dc:year (I used dc:date)
-dc:coverage
-dc:language (I used base_dc:lang)
-base_dc:link (I used dc:identifier)
--->
-
-
-								<xsl:variable name="varBaseNormType" select="vocabulary:clean(//base_dc:typenorm, 'base:normalized_types')" />
-								
-
-								<metadata>
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:title" />
-										<xsl:with-param name="targetElement" select="'dc:title'" />
-									</xsl:call-template>
-
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:creator/replace(., '^(.*)\|.*$', '$1')" />
-										<xsl:with-param name="targetElement" select="'dc:creator'" />
-									</xsl:call-template>
-
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:contributor" />
-										<xsl:with-param name="targetElement" select="'dc:contributor'" />
-									</xsl:call-template>
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:description" />
-										<xsl:with-param name="targetElement" select="'dc:description'" />
-									</xsl:call-template>
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:subject" />
-										<xsl:with-param name="targetElement" select="'dc:subject'" />
-									</xsl:call-template>
-									
-									<!-- TODO: I'm not sure if this is the correct encoding -->
-									<xsl:for-each select="//base_dc:classcode|//base_dc:autoclasscode">
-										<dc:subject><xsl:value-of select="concat(@type, ':', .)" /></dc:subject>
-									</xsl:for-each>
-									<!-- END TODO -->
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:publisher" />
-										<xsl:with-param name="targetElement" select="'dc:publisher'" />
-									</xsl:call-template>
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:format" />
-										<xsl:with-param name="targetElement" select="'dc:format'" />
-									</xsl:call-template>
-									
-									<dc:type>
-										<xsl:value-of select="$varBaseNormType" />
-									</dc:type>
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:type" />
-										<xsl:with-param name="targetElement" select="'dc:type'" />
-									</xsl:call-template>
-									
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:source" />
-										<xsl:with-param name="targetElement" select="'dc:source'" />
-									</xsl:call-template>
-									
-									<dc:language>
-										<xsl:value-of select="vocabulary:clean( //base_dc:lang, 'dnet:languages')" />
-									</dc:language>
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:rights" />
-										<xsl:with-param name="targetElement" select="'dc:rights'" />
-									</xsl:call-template>
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:relation" />
-										<xsl:with-param name="targetElement" select="'dc:relation'" />
-									</xsl:call-template>
-									
-									<xsl:if test="not(//dc:identifier[starts-with(., 'http')])">
-										<xsl:call-template name="terminate" />
-									</xsl:if>
-									
-									<xsl:call-template name="allElements">
-										<xsl:with-param name="sourceElement" select="//dc:identifier[starts-with(., 'http')]" />
-										<xsl:with-param name="targetElement" select="'dc:identifier'" />
-									</xsl:call-template>
-									
-									<xsl:for-each select="//dc:relation">
-										<xsl:if test="matches(normalize-space(.), '(info:eu-repo/grantagreement/ec/fp7/)(\d\d\d\d\d\d)(.*)', 'i')">
-											<oaf:projectid>
-												<xsl:value-of select="concat($varFP7, replace(normalize-space(.), '(info:eu-repo/grantagreement/ec/fp7/)(\d\d\d\d\d\d)(.*)', '$2', 'i'))" />
-											</oaf:projectid>
-										</xsl:if>
-										<xsl:if test="matches(normalize-space(.), '(info:eu-repo/grantagreement/ec/h2020/)(\d\d\d\d\d\d)(.*)', 'i')">
-											<oaf:projectid>
-												<xsl:value-of select="concat($varH2020, replace(normalize-space(.), '(info:eu-repo/grantagreement/ec/h2020/)(\d\d\d\d\d\d)(.*)', '$2', 'i'))" />
-											</oaf:projectid>
-										</xsl:if>
-									</xsl:for-each>
-
-									<dr:CobjCategory>
-										<xsl:variable name="varCobjCategory" select="vocabulary:clean($varBaseNormType, 'dnet:publication_resource')" />
-										<xsl:variable name="varSuperType" select="vocabulary:clean($varCobjCategory, 'dnet:result_typologies')" />
-										<xsl:attribute name="type" select="$varSuperType" />
-										<xsl:value-of select="$varCobjCategory" />
-									</dr:CobjCategory>
-									
-									<oaf:accessrights>
-										<xsl:choose>
-											<xsl:when test="//base_dc:oa[.='1']">OPEN</xsl:when>
-											<xsl:when test="//base_dc:rightsnorm">
-												<xsl:value-of select="vocabulary:clean(//base_dc:rightsnorm, 'dnet:access_modes')" />
-											</xsl:when>
-											<xsl:when test="//dc:rights">
-												<xsl:value-of select="vocabulary:clean( //dc:rights, 'dnet:access_modes')" />
-											</xsl:when>
-											<xsl:otherwise>UNKNOWN</xsl:otherwise>
-										</xsl:choose>
-									</oaf:accessrights>
-									
-									<xsl:for-each select="//base_dc:doi">
-										<oaf:identifier identifierType="doi">
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-
-									<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'http') and (not(contains(., '://dx.doi.org/') or contains(., '://doi.org/') or contains(., '://hdl.handle.net/')))])">
-										<oaf:identifier identifierType="url">
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-
-									<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'http') and contains(., '://hdl.handle.net/')]/substring-after(., 'hdl.handle.net/'))">
-										<oaf:identifier identifierType="handle">
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>									
-
-									<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'urn:nbn:nl:') or starts-with(., 'URN:NBN:NL:')])">
-										<oaf:identifier identifierType='urn'>
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-									
-									<oaf:identifier identifierType="oai-original">
-										<xsl:value-of
-											select="//*[local-name() = 'about']/*[local-name() = 'provenance']//*[local-name() = 'originDescription' and not(./*[local-name() = 'originDescription'])]/*[local-name() = 'identifier']" />
-									</oaf:identifier>
-									
-									<oaf:hostedBy>
-										<xsl:attribute name="name">
-											<xsl:value-of select="//base_dc:collname" />
-										</xsl:attribute>
-										<xsl:attribute name="id">
-											<xsl:value-of select="concat('opendoar____::', //base_dc:collection/@opendoar_id)" />
-										</xsl:attribute>
-									</oaf:hostedBy>
-									
-									<oaf:collectedFrom>
-										<xsl:attribute name="name">
-											<xsl:value-of select="$varOfficialName" />
-										</xsl:attribute>
-										<xsl:attribute name="id">
-											<xsl:value-of select="$varDataSourceId" />
-										</xsl:attribute>
-									</oaf:collectedFrom>
-									
-									<oaf:dateAccepted>
-										<xsl:value-of select="dateCleaner:dateISO( //dc:date[1] )" />
-									</oaf:dateAccepted>
-									
-									<xsl:if test="//base_dc:oa[.='1']">
-										<xsl:for-each select="//dc:relation[starts-with(., 'http')]">
-											<oaf:fulltext>
-												<xsl:value-of select="normalize-space(.)" />
-											</oaf:fulltext>
-										</xsl:for-each>
-									</xsl:if>
-									
-									<xsl:for-each select="//base_dc:collection/@ror_id">
-										<oaf:relation relType="resultOrganization"
-										 	subRelType="affiliation"
-										 	relClass="hasAuthorInstitution"
-											targetType="organization">
-											<xsl:choose>
-												<xsl:when test="contains(.,'https://ror.org/')">
-													<xsl:value-of select="concat('ror_________::', normalize-space(.))" />
-												</xsl:when>
-												<xsl:otherwise>
-													<xsl:value-of select="concat('ror_________::https://ror.org/', normalize-space(.))" />
-												</xsl:otherwise>
-											</xsl:choose>
-										</oaf:relation>										
-									</xsl:for-each>
-
-									<xsl:for-each select="//base_dc:country">
-										<oaf:country><xsl:value-of select="vocabulary:clean(., 'dnet:countries')" /></oaf:country>
-									</xsl:for-each>
-									
-								</metadata>
-								<xsl:copy-of select="//*[local-name() = 'about']" />
-							</record>
-						</xsl:template>
-
-						<xsl:template name="allElements">
-							<xsl:param name="sourceElement" />
-							<xsl:param name="targetElement" />
-							<xsl:for-each select="$sourceElement">
-								<xsl:element name="{$targetElement}">
-									<xsl:value-of select="normalize-space(.)" />
-								</xsl:element>
-							</xsl:for-each>
-						</xsl:template>
-
-						<xsl:template match="//*[local-name() = 'header']">
-							<xsl:if test="//oai:header/@status='deleted'">
-								<xsl:call-template name="terminate" />
-							</xsl:if>
-							<xsl:copy>
-								<xsl:apply-templates select="node()|@*" />
-								<xsl:element name="dr:dateOfTransformation">
-									<xsl:value-of select="$transDate" />
-								</xsl:element>
-							</xsl:copy>
-						</xsl:template>
-
-						<xsl:template match="node()|@*">
-							<xsl:copy>
-								<xsl:apply-templates select="node()|@*" />
-							</xsl:copy>
-						</xsl:template>
-					</xsl:stylesheet>
-				</CODE>
-			</SCRIPT>
-		</CONFIGURATION>
-		<STATUS />
-		<SECURITY_PARAMETERS />
-	</BODY>
-</RESOURCE_PROFILE>

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/collection/plugin/base/xml/base2odf.transformationRule.xml

@@ -1,326 +0,0 @@
-<RESOURCE_PROFILE>
-	<HEADER>
-		<RESOURCE_IDENTIFIER value="2ad0cdd9-c96c-484c-8b0e-ed56d86891fe_VHJhbnNmb3JtYXRpb25SdWxlRFNSZXNvdXJjZXMvVHJhbnNmb3JtYXRpb25SdWxlRFNSZXNvdXJjZVR5cGU=" />
-		<RESOURCE_TYPE value="TransformationRuleDSResourceType" />
-		<RESOURCE_KIND value="TransformationRuleDSResources" />
-		<RESOURCE_URI value="" />
-		<DATE_OF_CREATION value="2024-03-05T11:23:00+00:00" />
-	</HEADER>
-	<BODY>
-		<CONFIGURATION>
-			<SOURCE_METADATA_FORMAT interpretation="cleaned" layout="store" name="dc" />
-			<SINK_METADATA_FORMAT name="odf_hbase" />
-			<IMPORTED />
-			<SCRIPT>
-				<TITLE>xslt_base2odf_hadoop</TITLE>
-				<CODE>
-					<xsl:stylesheet xmlns:oaire="http://namespace.openaire.eu/schema/oaire/" xmlns:dateCleaner="http://eu/dnetlib/transform/dateISO" xmlns:base_dc="http://oai.base-search.net/base_dc/"
-						xmlns:datacite="http://datacite.org/schema/kernel-4" xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-						xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:vocabulary="http://eu/dnetlib/transform/clean" xmlns:oaf="http://namespace.openaire.eu/oaf"
-						xmlns:oai="http://www.openarchives.org/OAI/2.0/" xmlns:dri="http://www.driver-repository.eu/namespace/dri" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dc="http://purl.org/dc/elements/1.1/"
-						exclude-result-prefixes="xsl vocabulary dateCleaner base_dc" version="2.0">
-						<xsl:param name="varOfficialName" />
-						<xsl:param name="varDataSourceId" />
-						<xsl:param name="varFP7" select="'corda_______::'" />
-						<xsl:param name="varH2020" select="'corda__h2020::'" />
-						<xsl:param name="repoCode" select="substring-before(//*[local-name() = 'header']/*[local-name()='recordIdentifier'], ':')" />
-						<xsl:param name="index" select="0" />
-						<xsl:param name="transDate" select="current-dateTime()" />
-
-						<xsl:template name="terminate">
-							<xsl:message terminate="yes">
-								record is not compliant, transformation is interrupted.
-							</xsl:message>
-						</xsl:template>
-
-						<xsl:template match="/">
-							<record>
-								<xsl:apply-templates select="//*[local-name() = 'header']" />
-
-
-								<!-- NOT USED 
-									base_dc:global_id (I used oai:identifier) 
-									base_dc:collection/text() 
-									base_dc:continent 
-									dc:coverage
-									dc:source
-									dc:relation
-									dc:type (I used //base_dc:typenorm)
-									dc:language (I used base_dc:lang) 
-									base_dc:link (I used dc:identifier)
-								 -->
-
-
-								<xsl:variable name="varBaseNormType" select="vocabulary:clean(//base_dc:typenorm, 'base:normalized_types')" />
-
-
-								<metadata>
-									<datacite:resource>
-
-										<xsl:for-each select="//base_dc:doi">
-											<datacite:identifier identifierType="DOI">
-												<xsl:value-of select="." />
-											</datacite:identifier>
-										</xsl:for-each>										
-
-										<datacite:alternateIdentifiers>
-											<xsl:for-each
-												select="distinct-values(//dc:identifier[starts-with(., 'http') and (not(contains(., '://dx.doi.org/') or contains(., '://doi.org/') or contains(., '://hdl.handle.net/')))])">
-												<datacite:identifier alternateIdentifierType="url">
-													<xsl:value-of select="." />
-												</datacite:identifier>
-											</xsl:for-each>
-
-											<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'http') and contains(., '://hdl.handle.net/')]/substring-after(., 'hdl.handle.net/'))">
-												<datacite:identifier alternateIdentifierType="handle">
-													<xsl:value-of select="." />
-												</datacite:identifier>
-											</xsl:for-each>
-
-											<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'urn:nbn:nl:') or starts-with(., 'URN:NBN:NL:')])">
-												<datacite:identifier alternateIdentifierType='urn'>
-													<xsl:value-of select="." />
-												</datacite:identifier>
-											</xsl:for-each>
-
-											<datacite:identifier alternateIdentifierType="oai-original">
-												<xsl:value-of
-													select="//*[local-name() = 'about']/*[local-name() = 'provenance']//*[local-name() = 'originDescription' and not(./*[local-name() = 'originDescription'])]/*[local-name() = 'identifier']" />
-											</datacite:identifier>
-										</datacite:alternateIdentifiers>
-
-										<datacite:relatedIdentifiers />
-
-
-										<datacite:resourceType><xsl:value-of select="$varBaseNormType" /></datacite:resourceType>
-
-										<datacite:titles>
-											<xsl:for-each select="//dc:title">
-												<datacite:title>
-													<xsl:value-of select="normalize-space(.)" />
-												</datacite:title>
-											</xsl:for-each>
-										</datacite:titles>
-
-										<datacite:creators>
-											<xsl:for-each select="//dc:creator">
-												<xsl:variable name="author" select="normalize-space(.)" />
-												<datacite:creator>
-													<datacite:creatorName>
-														<xsl:value-of select="$author" />
-													</datacite:creatorName>
-													<xsl:for-each select="//base_dc:authod_id[normalize-space(./base_dc:creator_name) = $author]/base_dc:creator_id ">
-														<xsl:if test="contains(.,'https://orcid.org/')">
-															<nameIdentifier schemeURI="https://orcid.org/" nameIdentifierScheme="ORCID">
-																<xsl:value-of select="substring-after(., 'https://orcid.org/')" />
-															</nameIdentifier>
-														</xsl:if>
-													</xsl:for-each>
-												</datacite:creator>
-											</xsl:for-each>
-										</datacite:creators>
-
-										<datacite:contributors>
-											<xsl:for-each select="//dc:contributor">
-												<datacite:contributor>
-													<datacite:contributorName>
-														<xsl:value-of select="normalize-space(.)" />
-													</datacite:contributorName>
-												</datacite:contributor>
-											</xsl:for-each>
-										</datacite:contributors>
-
-										<datacite:descriptions>
-											<xsl:for-each select="//dc:description">
-												<datacite:description descriptionType="Abstract">
-													<xsl:value-of select="normalize-space(.)" />
-												</datacite:description>
-											</xsl:for-each>
-										</datacite:descriptions>
-
-										<datacite:subjects>
-											<xsl:for-each select="//dc:subject">
-												<datacite:subject>
-													<xsl:value-of select="normalize-space(.)" />
-												</datacite:subject>
-											</xsl:for-each>
-											
-											<xsl:for-each select="//base_dc:classcode|//base_dc:autoclasscode">
-												<datacite:subject subjectScheme="{@type}" classificationCode="{normalize-space(.)}">
-													<!-- TODO the value should be obtained by the Code -->
-													<xsl:value-of select="normalize-space(.)" />
-												</datacite:subject>
-											</xsl:for-each>
-										</datacite:subjects>
-										
-										<datacite:publisher>
-											<xsl:value-of select="normalize-space(//dc:publisher)" />
-										</datacite:publisher>
-										
-										<datacite:publicationYear>
-											<xsl:value-of select="normalize-space(//base_dc:year)" />
-										</datacite:publicationYear>
-										
-										<datacite:formats>
-											<xsl:for-each select="//dc:format">
-												<datacite:format>
-													<xsl:value-of select="normalize-space(.)" />
-												</datacite:format>
-											</xsl:for-each>
-										</datacite:formats>
-										
-										<datacite:language>
-											<xsl:value-of select="vocabulary:clean( //base_dc:lang, 'dnet:languages')" />
-										</datacite:language>
-
-										<oaf:accessrights>
-											<xsl:if test="//base_dc:oa[.='1']">
-												<datacite:rights rightsURI="http://purl.org/coar/access_right/c_abf2">open access</datacite:rights>
-											</xsl:if>
-											<xsl:for-each select="//dc:rights|//base_dc:rightsnorm">
-												<datacite:rights><xsl:value-of select="vocabulary:clean(., 'dnet:access_modes')" /></datacite:rights>	
-											</xsl:for-each>
-										</oaf:accessrights>
-
-									</datacite:resource>
-
-										<xsl:for-each select="//dc:relation">
-											<xsl:if test="matches(normalize-space(.), '(info:eu-repo/grantagreement/ec/fp7/)(\d\d\d\d\d\d)(.*)', 'i')">
-												<oaf:projectid>
-													<xsl:value-of select="concat($varFP7, replace(normalize-space(.), '(info:eu-repo/grantagreement/ec/fp7/)(\d\d\d\d\d\d)(.*)', '$2', 'i'))" />
-												</oaf:projectid>
-											</xsl:if>
-											<xsl:if test="matches(normalize-space(.), '(info:eu-repo/grantagreement/ec/h2020/)(\d\d\d\d\d\d)(.*)', 'i')">
-												<oaf:projectid>
-													<xsl:value-of select="concat($varH2020, replace(normalize-space(.), '(info:eu-repo/grantagreement/ec/h2020/)(\d\d\d\d\d\d)(.*)', '$2', 'i'))" />
-												</oaf:projectid>
-											</xsl:if>
-										</xsl:for-each>
-
-									<dr:CobjCategory>
-										<xsl:variable name="varCobjCategory" select="vocabulary:clean($varBaseNormType, 'dnet:publication_resource')" />
-										<xsl:variable name="varSuperType" select="vocabulary:clean($varCobjCategory, 'dnet:result_typologies')" />
-										<xsl:attribute name="type" select="$varSuperType" />
-										<xsl:value-of select="$varCobjCategory" />
-									</dr:CobjCategory>
-
-									<oaf:accessrights>
-										<xsl:choose>
-											<xsl:when test="//base_dc:oa[.='1']">OPEN</xsl:when>
-											<xsl:when test="//base_dc:rightsnorm">
-												<xsl:value-of select="vocabulary:clean(//base_dc:rightsnorm, 'dnet:access_modes')" />
-											</xsl:when>
-											<xsl:when test="//dc:rights">
-												<xsl:value-of select="vocabulary:clean( //dc:rights, 'dnet:access_modes')" />
-											</xsl:when>
-											<xsl:otherwise>UNKNOWN</xsl:otherwise>
-										</xsl:choose>
-									</oaf:accessrights>
-
-									<xsl:for-each select="//base_dc:doi">
-										<oaf:identifier identifierType="doi">
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-
-									<xsl:for-each
-										select="distinct-values(//dc:identifier[starts-with(., 'http') and ( not(contains(., '://dx.doi.org/') or contains(., '://doi.org/') or contains(., '://hdl.handle.net/')))])">
-										<oaf:identifier identifierType="url">
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-
-									<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'http') and contains(., '://hdl.handle.net/')]/substring-after(., 'hdl.handle.net/'))">
-										<oaf:identifier identifierType="handle">
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-
-									<xsl:for-each select="distinct-values(//dc:identifier[starts-with(., 'urn:nbn:nl:') or starts-with(., 'URN:NBN:NL:')])">
-										<oaf:identifier identifierType='urn'>
-											<xsl:value-of select="." />
-										</oaf:identifier>
-									</xsl:for-each>
-
-									<oaf:identifier identifierType="oai-original">
-										<xsl:value-of
-											select="//*[local-name() = 'about']/*[local-name() = 'provenance']//*[local-name() = 'originDescription' and not(./*[local-name() = 'originDescription'])]/*[local-name() = 'identifier']" />
-									</oaf:identifier>
-
-									<oaf:hostedBy>
-										<xsl:attribute name="name">
-											<xsl:value-of select="//base_dc:collname" />
-										</xsl:attribute>
-										<xsl:attribute name="id">
-											<xsl:value-of select="concat('opendoar____::', //base_dc:collection/@opendoar_id)" />
-										</xsl:attribute>
-									</oaf:hostedBy>
-
-									<oaf:collectedFrom>
-										<xsl:attribute name="name">
-											<xsl:value-of select="$varOfficialName" />
-										</xsl:attribute>
-										<xsl:attribute name="id">
-											<xsl:value-of select="$varDataSourceId" />
-										</xsl:attribute>
-									</oaf:collectedFrom>
-
-									<oaf:dateAccepted>
-										<xsl:value-of select="dateCleaner:dateISO( //dc:date[1] )" />
-									</oaf:dateAccepted>
-
-									<xsl:if test="//base_dc:oa[.='1']">
-										<xsl:for-each select="//dc:relation[starts-with(., 'http')]">
-											<oaf:fulltext>
-												<xsl:value-of select="normalize-space(.)" />
-											</oaf:fulltext>
-										</xsl:for-each>
-									</xsl:if>
-
-									<xsl:for-each select="//base_dc:collection/@ror_id">
-										<oaf:relation relType="resultOrganization" subRelType="affiliation" relClass="hasAuthorInstitution" targetType="organization">
-											<xsl:choose>
-												<xsl:when test="contains(.,'https://ror.org/')">
-													<xsl:value-of select="concat('ror_________::', normalize-space(.))" />
-												</xsl:when>
-												<xsl:otherwise>
-													<xsl:value-of select="concat('ror_________::https://ror.org/', normalize-space(.))" />
-												</xsl:otherwise>
-											</xsl:choose>
-										</oaf:relation>
-									</xsl:for-each>
-									
-									<xsl:for-each select="//base_dc:country">
-										<oaf:country><xsl:value-of select="vocabulary:clean(., 'dnet:countries')" /></oaf:country>
-									</xsl:for-each>
-									
-								</metadata>
-								<xsl:copy-of select="//*[local-name() = 'about']" />
-							</record>
-						</xsl:template>
-
-						<xsl:template match="//*[local-name() = 'header']">
-							<xsl:if test="//oai:header/@status='deleted'">
-								<xsl:call-template name="terminate" />
-							</xsl:if>
-							<xsl:copy>
-								<xsl:apply-templates select="node()|@*" />
-								<xsl:element name="dr:dateOfTransformation">
-									<xsl:value-of select="$transDate" />
-								</xsl:element>
-							</xsl:copy>
-						</xsl:template>
-
-						<xsl:template match="node()|@*">
-							<xsl:copy>
-								<xsl:apply-templates select="node()|@*" />
-							</xsl:copy>
-						</xsl:template>
-					</xsl:stylesheet>
-				</CODE>
-			</SCRIPT>
-		</CONFIGURATION>
-		<STATUS />
-		<SECURITY_PARAMETERS />
-	</BODY>
-</RESOURCE_PROFILE>

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/datacite/hostedBy_map.json

@@ -1048,10 +1048,5 @@
   "openaire_id": "re3data_____::r3d100010399",
   "datacite_name": "ZEW Forschungsdatenzentrum",
   "official_name": "ZEW Forschungsdatenzentrum"
- },
- "HBP.NEUROINF": {
-  "openaire_id": "fairsharing_::2975",
-  "datacite_name": "EBRAINS",
-  "official_name": "EBRAINS"
  }
 }

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/sx/bio/db/oozie_app/workflow.xml

@@ -1,4 +1,4 @@
-<workflow-app name="Transform_BioEntity_Workflow" xmlns="uri:oozie:workflow:0.5">
+    <workflow-app name="Transform_BioEntity_Workflow" xmlns="uri:oozie:workflow:0.5">
     <parameters>
         <property>
             <name>sourcePath</name>
@@ -8,40 +8,19 @@
             <name>database</name>
             <description>the PDB Database Working Path</description>
         </property>
+
         <property>
-            <name>mdStoreOutputId</name>
-            <description>the identifier of the cleaned MDStore</description>
-        </property>
-        <property>
-            <name>mdStoreManagerURI</name>
-            <description>the path of the cleaned mdstore</description>
+            <name>targetPath</name>
+            <description>the Target Working dir path</description>
         </property>
     </parameters>
 
-    <start to="StartTransaction"/>
-
+    <start to="ConvertDB"/>
 
     <kill name="Kill">
         <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
     </kill>
 
-    <action name="StartTransaction">
-        <java>
-            <configuration>
-                <property>
-                    <name>oozie.launcher.mapreduce.user.classpath.first</name>
-                    <value>true</value>
-                </property>
-            </configuration>
-            <main-class>eu.dnetlib.dhp.aggregation.mdstore.MDStoreActionNode</main-class>
-            <arg>--action</arg><arg>NEW_VERSION</arg>
-            <arg>--mdStoreID</arg><arg>${mdStoreOutputId}</arg>
-            <arg>--mdStoreManagerURI</arg><arg>${mdStoreManagerURI}</arg>
-            <capture-output/>
-        </java>
-        <ok to="ConvertDB"/>
-        <error to="RollBack"/>
-    </action>
     <action name="ConvertDB">
         <spark xmlns="uri:oozie:spark-action:0.2">
             <master>yarn</master>
@@ -62,48 +41,11 @@
             <arg>--master</arg><arg>yarn</arg>
             <arg>--dbPath</arg><arg>${sourcePath}</arg>
             <arg>--database</arg><arg>${database}</arg>
-            <arg>--mdstoreOutputVersion</arg><arg>${wf:actionData('StartTransaction')['mdStoreVersion']}</arg>
+            <arg>--targetPath</arg><arg>${targetPath}</arg>
         </spark>
-        <ok to="CommitVersion"/>
-        <error to="RollBack"/>
-
+        <ok to="End"/>
+        <error to="Kill"/>
     </action>
-        <action name="CommitVersion">
-            <java>
-                <configuration>
-                    <property>
-                        <name>oozie.launcher.mapreduce.user.classpath.first</name>
-                        <value>true</value>
-                    </property>
-                </configuration>
-                <main-class>eu.dnetlib.dhp.aggregation.mdstore.MDStoreActionNode</main-class>
-                <arg>--action</arg><arg>COMMIT</arg>
-                <arg>--namenode</arg><arg>${nameNode}</arg>
-                <arg>--mdStoreVersion</arg><arg>${wf:actionData('StartTransaction')['mdStoreVersion']}</arg>
-                <arg>--mdStoreManagerURI</arg><arg>${mdStoreManagerURI}</arg>
-            </java>
-            <ok to="End"/>
-            <error to="Kill"/>
-        </action>
-
-        <action name="RollBack">
-            <java>
-                <configuration>
-                    <property>
-                        <name>oozie.launcher.mapreduce.user.classpath.first</name>
-                        <value>true</value>
-                    </property>
-                </configuration>
-                <main-class>eu.dnetlib.dhp.aggregation.mdstore.MDStoreActionNode</main-class>
-                <arg>--action</arg><arg>ROLLBACK</arg>
-                <arg>--mdStoreVersion</arg><arg>${wf:actionData('StartTransaction')['mdStoreVersion']}</arg>
-                <arg>--mdStoreManagerURI</arg><arg>${mdStoreManagerURI}</arg>
-            </java>
-            <ok to="Kill"/>
-            <error to="Kill"/>
-        </action>
-
-
-        <end name="End"/>
+    <end name="End"/>
 
 </workflow-app>

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/sx/bio/ebi/bio_to_oaf_params.json

@@ -2,5 +2,5 @@
   {"paramName":"mt",  "paramLongName":"master",       "paramDescription": "should be local or yarn",                  "paramRequired": true},
   {"paramName":"db",  "paramLongName":"database",     "paramDescription": "should be PDB or UNIPROT",                 "paramRequired": true},
   {"paramName":"p",   "paramLongName":"dbPath",       "paramDescription": "the path of the database to transform",    "paramRequired": true},
-  {"paramName":"mo",   "paramLongName":"mdstoreOutputVersion",     "paramDescription": "the oaf path ",                "paramRequired": true}
+  {"paramName":"t",   "paramLongName":"targetPath",   "paramDescription": "the OAF target path ",                     "paramRequired": true}
 ]

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/sx/bio/ebi/ebi_to_df_params.json

@@ -1,20 +1,5 @@
 [
-  {
-    "paramName": "mt",
-    "paramLongName": "master",
-    "paramDescription": "should be local or yarn",
-    "paramRequired": true
-  },
-  {
-    "paramName": "s",
-    "paramLongName": "sourcePath",
-    "paramDescription": "the source Path",
-    "paramRequired": true
-  },
-  {
-    "paramName": "mo",
-    "paramLongName": "mdstoreOutputVersion",
-    "paramDescription": "the oaf path ",
-    "paramRequired": true
-  }
+  {"paramName":"mt",  "paramLongName":"master",     "paramDescription": "should be local or yarn",                  "paramRequired": true},
+  {"paramName":"s",   "paramLongName":"sourcePath","paramDescription": "the source Path",                              "paramRequired": true},
+  {"paramName":"t",   "paramLongName":"targetPath","paramDescription": "the  oaf path ",  "paramRequired": true}
 ]

dhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/sx/bio/ebi/oozie_app/workflow.xml

@@ -9,26 +9,34 @@
             <description>the Working Path</description>
         </property>
         <property>
-            <name>mdStoreOutputId</name>
-            <description>the identifier of the cleaned MDStore</description>
+            <name>targetPath</name>
+            <description>the OAF MDStore Path</description>
         </property>
         <property>
-            <name>mdStoreManagerURI</name>
-            <description>the path of the cleaned mdstore</description>
+            <name>sparkDriverMemory</name>
+            <description>memory for driver process</description>
+        </property>
+        <property>
+            <name>sparkExecutorMemory</name>
+            <description>memory for individual executor</description>
+        </property>
+        <property>
+            <name>sparkExecutorCores</name>
+            <description>number of cores used by single executor</description>
         </property>
         <property>
             <name>resumeFrom</name>
-            <value>CreateEBIDataSet</value>
+            <value>DownloadEBILinks</value>
             <description>node to start</description>
         </property>
     </parameters>
 
-    <start to="StartTransaction"/>
+    <start to="resume_from"/>
 
     <decision name="resume_from">
         <switch>
             <case to="DownloadEBILinks">${wf:conf('resumeFrom') eq 'DownloadEBILinks'}</case>
-            <case to="StartTransaction">${wf:conf('resumeFrom') eq 'CreateEBIDataSet'}</case>
+            <case to="CreateEBIDataSet">${wf:conf('resumeFrom') eq 'CreateEBIDataSet'}</case>
             <default to="DownloadEBILinks"/>
         </switch>
     </decision>
@@ -69,29 +77,9 @@
             <move source="${sourcePath}/ebi_links_dataset" target="${sourcePath}/ebi_links_dataset_old"/>
             <move source="${workingPath}/links_final" target="${sourcePath}/ebi_links_dataset"/>
         </fs>
-        <ok to="StartTransaction"/>
+        <ok to="CreateEBIDataSet"/>
         <error to="Kill"/>
     </action>
-
-    <action name="StartTransaction">
-        <java>
-            <configuration>
-                <property>
-                    <name>oozie.launcher.mapreduce.user.classpath.first</name>
-                    <value>true</value>
-                </property>
-            </configuration>
-            <main-class>eu.dnetlib.dhp.aggregation.mdstore.MDStoreActionNode</main-class>
-            <arg>--action</arg><arg>NEW_VERSION</arg>
-            <arg>--mdStoreID</arg><arg>${mdStoreOutputId}</arg>
-            <arg>--mdStoreManagerURI</arg><arg>${mdStoreManagerURI}</arg>
-            <capture-output/>
-        </java>
-        <ok to="CreateEBIDataSet"/>
-        <error to="RollBack"/>
-    </action>
-
-
     <action name="CreateEBIDataSet">
         <spark xmlns="uri:oozie:spark-action:0.2">
             <master>yarn-cluster</master>
@@ -107,49 +95,11 @@
                 ${sparkExtraOPT}
             </spark-opts>
             <arg>--sourcePath</arg><arg>${sourcePath}/ebi_links_dataset</arg>
-            <arg>--mdstoreOutputVersion</arg><arg>${wf:actionData('StartTransaction')['mdStoreVersion']}</arg>
+            <arg>--targetPath</arg><arg>${targetPath}</arg>
             <arg>--master</arg><arg>yarn</arg>
         </spark>
         <ok to="End"/>
         <error to="Kill"/>
     </action>
-
-
-    <action name="CommitVersion">
-        <java>
-            <configuration>
-                <property>
-                    <name>oozie.launcher.mapreduce.user.classpath.first</name>
-                    <value>true</value>
-                </property>
-            </configuration>
-            <main-class>eu.dnetlib.dhp.aggregation.mdstore.MDStoreActionNode</main-class>
-            <arg>--action</arg><arg>COMMIT</arg>
-            <arg>--namenode</arg><arg>${nameNode}</arg>
-            <arg>--mdStoreVersion</arg><arg>${wf:actionData('StartTransaction')['mdStoreVersion']}</arg>
-            <arg>--mdStoreManagerURI</arg><arg>${mdStoreManagerURI}</arg>
-        </java>
-        <ok to="End"/>
-        <error to="Kill"/>
-    </action>
-
-    <action name="RollBack">
-        <java>
-            <configuration>
-                <property>
-                    <name>oozie.launcher.mapreduce.user.classpath.first</name>
-                    <value>true</value>
-                </property>
-            </configuration>
-            <main-class>eu.dnetlib.dhp.aggregation.mdstore.MDStoreActionNode</main-class>
-            <arg>--action</arg><arg>ROLLBACK</arg>
-            <arg>--mdStoreVersion</arg><arg>${wf:actionData('StartTransaction')['mdStoreVersion']}</arg>
-            <arg>--mdStoreManagerURI</arg><arg>${mdStoreManagerURI}</arg>
-        </java>
-        <ok to="Kill"/>
-        <error to="Kill"/>
-    </action>
-
     <end name="End"/>
-
 </workflow-app>

dhp-workflows/dhp-aggregation/src/main/scala/eu/dnetlib/dhp/sx/bio/BioDBToOAF.scala

@@ -231,7 +231,7 @@ object BioDBToOAF {
   def uniprotToOAF(input: String): List[Oaf] = {
     implicit lazy val formats: DefaultFormats.type = org.json4s.DefaultFormats
     lazy val json = parse(input)
-    val pid = (json \ "pid").extract[String].trim()
+    val pid = (json \ "pid").extract[String]
 
     val d = new Dataset
 

dhp-workflows/dhp-aggregation/src/main/scala/eu/dnetlib/dhp/sx/bio/SparkTransformBioDatabaseToOAF.scala

@@ -2,15 +2,12 @@ package eu.dnetlib.dhp.sx.bio
 
 import eu.dnetlib.dhp.application.ArgumentApplicationParser
 import eu.dnetlib.dhp.collection.CollectionUtils
-import eu.dnetlib.dhp.common.Constants.{MDSTORE_DATA_PATH, MDSTORE_SIZE_PATH}
-import eu.dnetlib.dhp.schema.mdstore.MDStoreVersion
 import eu.dnetlib.dhp.schema.oaf.Oaf
 import eu.dnetlib.dhp.sx.bio.BioDBToOAF.ScholixResolved
 import org.apache.commons.io.IOUtils
 import org.apache.spark.SparkConf
 import org.apache.spark.sql.{Encoder, Encoders, SparkSession}
 import org.slf4j.{Logger, LoggerFactory}
-import eu.dnetlib.dhp.utils.DHPUtils.{MAPPER, writeHdfsFile}
 
 object SparkTransformBioDatabaseToOAF {
 
@@ -28,13 +25,8 @@ object SparkTransformBioDatabaseToOAF {
 
     val dbPath: String = parser.get("dbPath")
     log.info("dbPath: {}", database)
-
-    val mdstoreOutputVersion = parser.get("mdstoreOutputVersion")
-    log.info("mdstoreOutputVersion: {}", mdstoreOutputVersion)
-
-    val cleanedMdStoreVersion = MAPPER.readValue(mdstoreOutputVersion, classOf[MDStoreVersion])
-    val outputBasePath = cleanedMdStoreVersion.getHdfsPath
-    log.info("outputBasePath: {}", outputBasePath)
+    val targetPath: String = parser.get("targetPath")
+    log.info("targetPath: {}", database)
 
     val spark: SparkSession =
       SparkSession
@@ -51,28 +43,24 @@ object SparkTransformBioDatabaseToOAF {
       case "UNIPROT" =>
         CollectionUtils.saveDataset(
           spark.createDataset(sc.textFile(dbPath).flatMap(i => BioDBToOAF.uniprotToOAF(i))),
-          s"$outputBasePath/$MDSTORE_DATA_PATH"
+          targetPath
         )
       case "PDB" =>
         CollectionUtils.saveDataset(
           spark.createDataset(sc.textFile(dbPath).flatMap(i => BioDBToOAF.pdbTOOaf(i))),
-          s"$outputBasePath/$MDSTORE_DATA_PATH"
+          targetPath
         )
       case "SCHOLIX" =>
         CollectionUtils.saveDataset(
           spark.read.load(dbPath).as[ScholixResolved].map(i => BioDBToOAF.scholixResolvedToOAF(i)),
-          s"$outputBasePath/$MDSTORE_DATA_PATH"
+          targetPath
         )
       case "CROSSREF_LINKS" =>
         CollectionUtils.saveDataset(
           spark.createDataset(sc.textFile(dbPath).map(i => BioDBToOAF.crossrefLinksToOaf(i))),
-          s"$outputBasePath/$MDSTORE_DATA_PATH"
+          targetPath
         )
     }
-
-    val df = spark.read.text(s"$outputBasePath/$MDSTORE_DATA_PATH")
-    val mdStoreSize = df.count
-    writeHdfsFile(spark.sparkContext.hadoopConfiguration, s"$mdStoreSize", s"$outputBasePath/$MDSTORE_SIZE_PATH")
   }
 
 }

dhp-workflows/dhp-aggregation/src/main/scala/eu/dnetlib/dhp/sx/bio/ebi/SparkEBILinksToOaf.scala

@@ -9,9 +9,6 @@ import org.apache.commons.io.IOUtils
 import org.apache.spark.SparkConf
 import org.apache.spark.sql._
 import org.slf4j.{Logger, LoggerFactory}
-import eu.dnetlib.dhp.common.Constants.{MDSTORE_DATA_PATH, MDSTORE_SIZE_PATH}
-import eu.dnetlib.dhp.schema.mdstore.MDStoreVersion
-import eu.dnetlib.dhp.utils.DHPUtils.{MAPPER, writeHdfsFile}
 
 object SparkEBILinksToOaf {
 
@@ -35,13 +32,8 @@ object SparkEBILinksToOaf {
     import spark.implicits._
     val sourcePath = parser.get("sourcePath")
     log.info(s"sourcePath  -> $sourcePath")
-    val mdstoreOutputVersion = parser.get("mdstoreOutputVersion")
-    log.info("mdstoreOutputVersion: {}", mdstoreOutputVersion)
-
-    val cleanedMdStoreVersion = MAPPER.readValue(mdstoreOutputVersion, classOf[MDStoreVersion])
-    val outputBasePath = cleanedMdStoreVersion.getHdfsPath
-    log.info("outputBasePath: {}", outputBasePath)
-
+    val targetPath = parser.get("targetPath")
+    log.info(s"targetPath  -> $targetPath")
     implicit val PMEncoder: Encoder[Oaf] = Encoders.kryo(classOf[Oaf])
 
     val ebLinks: Dataset[EBILinkItem] = spark.read
@@ -54,10 +46,7 @@ object SparkEBILinksToOaf {
         .flatMap(j => BioDBToOAF.parse_ebi_links(j.links))
         .filter(p => BioDBToOAF.EBITargetLinksFilter(p))
         .flatMap(p => BioDBToOAF.convertEBILinksToOaf(p)),
-      s"$outputBasePath/$MDSTORE_DATA_PATH"
+      targetPath
     )
-    val df = spark.read.text(s"$outputBasePath/$MDSTORE_DATA_PATH")
-    val mdStoreSize = df.count
-    writeHdfsFile(spark.sparkContext.hadoopConfiguration, s"$mdStoreSize", s"$outputBasePath/$MDSTORE_SIZE_PATH")
   }
 }

dhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/collection/plugin/base/BaseCollectionInfo.java

@@ -1,38 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import java.io.Serializable;
-
-public class BaseCollectionInfo implements Serializable {
-
-	private static final long serialVersionUID = 5766333937429419647L;
-
-	private String id;
-	private String opendoarId;
-	private String rorId;
-
-	public String getId() {
-		return this.id;
-	}
-
-	public void setId(final String id) {
-		this.id = id;
-	}
-
-	public String getOpendoarId() {
-		return this.opendoarId;
-	}
-
-	public void setOpendoarId(final String opendoarId) {
-		this.opendoarId = opendoarId;
-	}
-
-	public String getRorId() {
-		return this.rorId;
-	}
-
-	public void setRorId(final String rorId) {
-		this.rorId = rorId;
-	}
-
-}

dhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/collection/plugin/base/BaseCollectorIteratorTest.java

@@ -1,184 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedHashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Set;
-import java.util.concurrent.atomic.AtomicInteger;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.spark.api.java.JavaRDD;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.SparkSession;
-import org.dom4j.Attribute;
-import org.dom4j.Document;
-import org.dom4j.DocumentException;
-import org.dom4j.DocumentHelper;
-import org.dom4j.Element;
-import org.dom4j.Node;
-import org.junit.jupiter.api.Disabled;
-import org.junit.jupiter.api.Test;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-
-import eu.dnetlib.dhp.common.aggregation.AggregatorReport;
-
-@Disabled
-public class BaseCollectorIteratorTest {
-
-	@Test
-	void testImportFile() throws Exception {
-
-		long count = 0;
-
-		final BaseCollectorIterator iterator = new BaseCollectorIterator("base-sample.tar", new AggregatorReport());
-
-		final Map<String, Map<String, String>> collections = new HashMap<>();
-		final Map<String, AtomicInteger> fields = new HashMap<>();
-		final Set<String> types = new HashSet<>();
-
-		while (iterator.hasNext()) {
-
-			final Document record = DocumentHelper.parseText(iterator.next());
-
-			count++;
-
-			if ((count % 1000) == 0) {
-				System.out.println("# Read records: " + count);
-			}
-
-			// System.out.println(record.asXML());
-
-			for (final Object o : record.selectNodes("//*|//@*")) {
-				final String path = ((Node) o).getPath();
-
-				if (fields.containsKey(path)) {
-					fields.get(path).incrementAndGet();
-				} else {
-					fields.put(path, new AtomicInteger(1));
-				}
-
-				if (o instanceof Element) {
-					final Element n = (Element) o;
-
-					if ("collection".equals(n.getName())) {
-						final String collName = n.getText().trim();
-						if (StringUtils.isNotBlank(collName) && !collections.containsKey(collName)) {
-							final Map<String, String> collAttrs = new HashMap<>();
-							for (final Object ao : n.attributes()) {
-								collAttrs.put(((Attribute) ao).getName(), ((Attribute) ao).getValue());
-							}
-							collections.put(collName, collAttrs);
-						}
-					} else if ("type".equals(n.getName())) {
-						types.add(n.getText().trim());
-					}
-
-				}
-			}
-
-		}
-
-		final ObjectMapper mapper = new ObjectMapper();
-		for (final Entry<String, Map<String, String>> e : collections.entrySet()) {
-			System.out.println(e.getKey() + ": " + mapper.writeValueAsString(e.getValue()));
-
-		}
-
-		for (final Entry<String, AtomicInteger> e : fields.entrySet()) {
-			System.out.println(e.getKey() + ": " + e.getValue().get());
-
-		}
-
-		System.out.println("TYPES: ");
-		for (final String s : types) {
-			System.out.println(s);
-
-		}
-
-		assertEquals(30000, count);
-	}
-
-	@Test
-	public void testParquet() throws Exception {
-
-		final String xml = IOUtils.toString(getClass().getResourceAsStream("record.xml"));
-
-		final SparkSession spark = SparkSession.builder().master("local[*]").getOrCreate();
-
-		final List<BaseRecordInfo> ls = new ArrayList<>();
-
-		for (int i = 0; i < 10; i++) {
-			ls.add(extractInfo(xml));
-		}
-
-		final JavaRDD<BaseRecordInfo> rdd = JavaSparkContext
-			.fromSparkContext(spark.sparkContext())
-			.parallelize(ls);
-
-		final Dataset<BaseRecordInfo> df = spark
-			.createDataset(rdd.rdd(), Encoders.bean(BaseRecordInfo.class));
-
-		df.printSchema();
-
-		df.show(false);
-	}
-
-	private BaseRecordInfo extractInfo(final String s) {
-		try {
-			final Document record = DocumentHelper.parseText(s);
-
-			final BaseRecordInfo info = new BaseRecordInfo();
-
-			final Set<String> paths = new LinkedHashSet<>();
-			final Set<String> types = new LinkedHashSet<>();
-			final List<BaseCollectionInfo> colls = new ArrayList<>();
-
-			for (final Object o : record.selectNodes("//*|//@*")) {
-				paths.add(((Node) o).getPath());
-
-				if (o instanceof Element) {
-					final Element n = (Element) o;
-
-					final String nodeName = n.getName();
-
-					if ("collection".equals(nodeName)) {
-						final String collName = n.getText().trim();
-
-						if (StringUtils.isNotBlank(collName)) {
-							final BaseCollectionInfo coll = new BaseCollectionInfo();
-							coll.setId(collName);
-							coll.setOpendoarId(n.valueOf("@opendoar_id").trim());
-							coll.setRorId(n.valueOf("@ror_id").trim());
-							colls.add(coll);
-						}
-					} else if ("type".equals(nodeName)) {
-						types.add("TYPE: " + n.getText().trim());
-					} else if ("typenorm".equals(nodeName)) {
-						types.add("TYPE_NORM: " + n.getText().trim());
-					}
-				}
-			}
-
-			info.setId(record.valueOf("//*[local-name() = 'header']/*[local-name() = 'identifier']").trim());
-			info.getTypes().addAll(types);
-			info.getPaths().addAll(paths);
-			info.setCollections(colls);
-
-			return info;
-		} catch (final DocumentException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-}

dhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/collection/plugin/base/BaseCollectorPluginTest.java

@@ -1,32 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.Set;
-
-import org.apache.commons.io.IOUtils;
-import org.junit.jupiter.api.Test;
-
-class BaseCollectorPluginTest {
-
-	@Test
-	void testFilterXml() throws Exception {
-		final String xml = IOUtils.toString(getClass().getResourceAsStream("record.xml"));
-
-		final Set<String> validIds = new HashSet<>(Arrays.asList("opendoar____::1234", "opendoar____::4567"));
-		final Set<String> validTypes = new HashSet<>(Arrays.asList("1", "121"));
-		final Set<String> validTypes2 = new HashSet<>(Arrays.asList("1", "11"));
-
-		assertTrue(BaseCollectorPlugin.filterXml(xml, validIds, validTypes));
-		assertTrue(BaseCollectorPlugin.filterXml(xml, validIds, new HashSet<>()));
-
-		assertFalse(BaseCollectorPlugin.filterXml(xml, new HashSet<>(), validTypes));
-		assertFalse(BaseCollectorPlugin.filterXml(xml, validIds, validTypes2));
-
-	}
-
-}

dhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/collection/plugin/base/BaseRecordInfo.java

@@ -1,49 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import java.io.Serializable;
-import java.util.ArrayList;
-import java.util.List;
-
-public class BaseRecordInfo implements Serializable {
-
-	private static final long serialVersionUID = -8848232018350074593L;
-
-	private String id;
-	private List<BaseCollectionInfo> collections = new ArrayList<>();
-	private List<String> paths = new ArrayList<>();
-	private List<String> types = new ArrayList<>();
-
-	public String getId() {
-		return this.id;
-	}
-
-	public void setId(final String id) {
-		this.id = id;
-	}
-
-	public List<String> getPaths() {
-		return this.paths;
-	}
-
-	public void setPaths(final List<String> paths) {
-		this.paths = paths;
-	}
-
-	public List<String> getTypes() {
-		return this.types;
-	}
-
-	public void setTypes(final List<String> types) {
-		this.types = types;
-	}
-
-	public List<BaseCollectionInfo> getCollections() {
-		return this.collections;
-	}
-
-	public void setCollections(final List<BaseCollectionInfo> collections) {
-		this.collections = collections;
-	}
-
-}

dhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/collection/plugin/base/BaseTransfomationTest.java

@@ -1,77 +0,0 @@
-
-package eu.dnetlib.dhp.collection.plugin.base;
-
-import java.io.IOException;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.spark.SparkConf;
-import org.apache.spark.util.LongAccumulator;
-import org.dom4j.io.SAXReader;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import org.junit.jupiter.api.extension.ExtendWith;
-import org.mockito.junit.jupiter.MockitoExtension;
-
-import eu.dnetlib.dhp.aggregation.AbstractVocabularyTest;
-import eu.dnetlib.dhp.aggregation.common.AggregationCounter;
-import eu.dnetlib.dhp.schema.mdstore.MetadataRecord;
-import eu.dnetlib.dhp.schema.mdstore.Provenance;
-import eu.dnetlib.dhp.transformation.xslt.XSLTTransformationFunction;
-import eu.dnetlib.enabling.is.lookup.rmi.ISLookUpException;
-
-// @Disabled
-@ExtendWith(MockitoExtension.class)
-public class BaseTransfomationTest extends AbstractVocabularyTest {
-
-	private SparkConf sparkConf;
-
-	@BeforeEach
-	public void setUp() throws IOException, ISLookUpException {
-		setUpVocabulary();
-
-		this.sparkConf = new SparkConf();
-		this.sparkConf.setMaster("local[*]");
-		this.sparkConf.set("spark.driver.host", "localhost");
-		this.sparkConf.set("spark.ui.enabled", "false");
-	}
-
-	@Test
-	void testBase2ODF() throws Exception {
-
-		final MetadataRecord mr = new MetadataRecord();
-		mr.setProvenance(new Provenance("DSID", "DSNAME", "PREFIX"));
-		mr.setBody(IOUtils.toString(getClass().getResourceAsStream("record.xml")));
-
-		final XSLTTransformationFunction tr = loadTransformationRule("xml/base2odf.transformationRule.xml");
-
-		final MetadataRecord result = tr.call(mr);
-
-		System.out.println(result.getBody());
-	}
-
-	@Test
-	void testBase2OAF() throws Exception {
-
-		final MetadataRecord mr = new MetadataRecord();
-		mr.setProvenance(new Provenance("DSID", "DSNAME", "PREFIX"));
-		mr.setBody(IOUtils.toString(getClass().getResourceAsStream("record.xml")));
-
-		final XSLTTransformationFunction tr = loadTransformationRule("xml/base2oaf.transformationRule.xml");
-
-		final MetadataRecord result = tr.call(mr);
-
-		System.out.println(result.getBody());
-	}
-
-	private XSLTTransformationFunction loadTransformationRule(final String path) throws Exception {
-		final String xslt = new SAXReader()
-				.read(this.getClass().getResourceAsStream(path))
-				.selectSingleNode("//CODE/*")
-				.asXML();
-
-		final LongAccumulator la = new LongAccumulator();
-
-		return new XSLTTransformationFunction(new AggregationCounter(la, la, la), xslt, 0, this.vocabularies);
-	}
-
-}

dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/collection/plugin/base/record.xml

@@ -1,58 +0,0 @@
-<record>
-  <header xmlns="http://www.openarchives.org/OAI/2.0/">
-    <identifier>ftdoajarticles:oai:doaj.org/article:e2d5b5126b2d4e479933cc7f9a9ae0c1</identifier>
-    <datestamp>2022-12-31T11:48:55Z</datestamp>
-  </header>
-  <metadata xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:base_dc="http://oai.base-search.net/base_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dc="http://purl.org/dc/elements/1.1/">
-    <base_dc:dc xsi:schemaLocation="http://oai.base-search.net/base_dc/ http://oai.base-search.net/base_dc/base_dc.xsd">
-      <base_dc:global_id>ftdoajarticles:oai:doaj.org/article:e2d5b5126b2d4e479933cc7f9a9ae0c1</base_dc:global_id>
-      <base_dc:continent>cww</base_dc:continent>
-      <base_dc:country>org</base_dc:country>
-      <base_dc:collection opendoar_id="1234" ror_id="ror1234">ftdoajarticles</base_dc:collection>
-      <base_dc:collname>TEST REPO</base_dc:collname>
-      <dc:title>Assessment of cultural heritage: the legislative and methodological framework of Russian Federation</dc:title>
-      <dc:creator>ALBU, Svetlana</dc:creator>
-      <dc:creator>LEȘAN, Anna</dc:creator>
-      <dc:subject>architectural heritage</dc:subject>
-      <dc:subject>evaluation of architectural heritage</dc:subject>
-      <dc:subject>types of values</dc:subject>
-      <dc:subject>experience of russian federation</dc:subject>
-      <dc:subject>Social Sciences</dc:subject>
-      <dc:subject>H</dc:subject>
-      <dc:description>Architectural heritage is the real estate inheritance by population of a country becoming an extremely valuable and specific category, preserving and capitalizing on those assets requires considerable effort. The state does not have sufficient means to maintain and preserve cultural heritage, as a result it is included in the civil circuit. The transfer of property right or of some partial rights over the architectural patrimony is accompanied by the necessity to estimate the value of goods. In this article, the authors examine the experience of Russian Federation (one of the largest countries with a huge architectural heritage) on the legislative framework of architectural and methodological heritage of architectural heritage assessment. The particularities of cultural assets valuation compared to other categories of real estate are examined, as well as the methodological aspects (types of values, methods applied in valuation, approaches according to the purpose of valuation) regarding the valuation of real estate with architectural value in Russian Federation.</dc:description>
-      <dc:publisher>Technical University of Moldova</dc:publisher>
-      <dc:date>2020-09-01T00:00:00Z</dc:date>
-      <base_dc:year>2020</base_dc:year>
-      <dc:type>article</dc:type>
-      <base_dc:typenorm>121</base_dc:typenorm>
-      <dc:identifier>https://doi.org/10.5281/zenodo.3971988</dc:identifier>
-      <dc:identifier>https://doaj.org/article/e2d5b5126b2d4e479933cc7f9a9ae0c1</dc:identifier>
-      <base_dc:link>https://doi.org/10.5281/zenodo.3971988</base_dc:link>
-      <dc:source>Journal of Social Sciences, Vol 3, Iss 3, Pp 134-143 (2020)</dc:source>
-      <dc:language>EN</dc:language>
-      <dc:language>FR</dc:language>
-      <dc:language>RO</dc:language>
-      <dc:relation>http://ibn.idsi.md/sites/default/files/imag_file/JSS-3-2020_134-143.pdf</dc:relation>
-      <dc:relation>https://doaj.org/toc/2587-3490</dc:relation>
-      <dc:relation>https://doaj.org/toc/2587-3504</dc:relation>
-      <dc:relation>doi:10.5281/zenodo.3971988</dc:relation>
-      <dc:relation>2587-3490</dc:relation>
-      <dc:relation>2587-3504</dc:relation>
-      <dc:relation>https://doaj.org/article/e2d5b5126b2d4e479933cc7f9a9ae0c1</dc:relation>
-      <base_dc:autoclasscode type="ddc">720</base_dc:autoclasscode>
-      <base_dc:authod_id>
-        <base_dc:creator_name>ALBU, Svetlana</base_dc:creator_name>
-        <base_dc:creator_id>https://orcid.org/0000-0002-8648-950X</base_dc:creator_id>
-      </base_dc:authod_id>
-      <base_dc:authod_id>
-        <base_dc:creator_name>LEȘAN, Anna</base_dc:creator_name>
-        <base_dc:creator_id>https://orcid.org/0000-0003-3284-0525</base_dc:creator_id>
-      </base_dc:authod_id>
-      <base_dc:doi>https://doi.org/10.5281/zenodo.3971988</base_dc:doi>
-      <base_dc:oa>1</base_dc:oa>
-      <base_dc:lang>eng</base_dc:lang>
-      <base_dc:lang>fre</base_dc:lang>
-      <base_dc:lang>rum</base_dc:lang>
-    </base_dc:dc>
-  </metadata>
-</record>

dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/sx/graph/bio/pdb_dump

@@ -1,44 +1,15 @@
-{"classification": "Signaling protein", "pdb": "5NM4", "deposition_date": "2017-04-05", "title": "A2a adenosine receptor room-temperature structure determined by serial Femtosecond crystallography", "Keywords": ["Oom-temperature", " serial crystallography", " signaling protein"], "authors": ["T.weinert", "R.cheng", "D.james", "D.gashi", "P.nogly", "K.jaeger", "M.hennig", "", "J.standfuss"], "pmid": "28912485", "doi": "10.1038/S41467-017-00630-4"}
-{"classification": "Oxidoreductase/oxidoreductase inhibitor", "pdb": "4KN3", "deposition_date": "2013-05-08", "title": "Structure of the y34ns91g double mutant of dehaloperoxidase from Amphitrite ornata with 2,4,6-trichlorophenol", "Keywords": ["Lobin", " oxygen storage", " peroxidase", " oxidoreductase", " oxidoreductase-", "Oxidoreductase inhibitor complex"], "authors": ["C.wang", "L.lovelace", "L.lebioda"], "pmid": "23952341", "doi": "10.1021/BI400627W"}
-{"classification": "Transport protein", "pdb": "8HKM", "deposition_date": "2022-11-27", "title": "Ion channel", "Keywords": ["On channel", " transport protein"], "authors": ["D.h.jiang", "J.t.zhang"], "pmid": "37494189", "doi": "10.1016/J.CELREP.2023.112858"}
-{"classification": "Signaling protein", "pdb": "6JT1", "deposition_date": "2019-04-08", "title": "Structure of human soluble guanylate cyclase in the heme oxidised State", "Keywords": ["Oluble guanylate cyclase", " signaling protein"], "authors": ["L.chen", "Y.kang", "R.liu", "J.-x.wu"], "pmid": "31514202", "doi": "10.1038/S41586-019-1584-6"}
-{"classification": "Immune system", "pdb": "7OW6", "deposition_date": "2021-06-16", "title": "Crystal structure of a tcr in complex with hla-a*11:01 bound to kras G12d peptide (vvvgadgvgk)", "Keywords": ["La", " kras", " tcr", " immune system"], "authors": ["V.karuppiah", "R.a.robinson"], "doi": "10.1038/S41467-022-32811-1"}
-{"classification": "Biosynthetic protein", "pdb": "5EQ8", "deposition_date": "2015-11-12", "title": "Crystal structure of medicago truncatula histidinol-phosphate Phosphatase (mthpp) in complex with l-histidinol", "Keywords": ["Istidine biosynthesis", " metabolic pathways", " dimer", " plant", "", "Biosynthetic protein"], "authors": ["M.ruszkowski", "Z.dauter"], "pmid": "26994138", "doi": "10.1074/JBC.M115.708727"}
-{"classification": "De novo protein", "pdb": "8CWA", "deposition_date": "2022-05-18", "title": "Solution nmr structure of 8-residue rosetta-designed cyclic peptide D8.21 in cdcl3 with cis/trans switching (tc conformation, 53%)", "Keywords": ["Yclic peptide", " non natural amino acids", " cis/trans", " switch peptides", "", "De novo design", "Membrane permeability", "De novo protein"], "authors": ["T.a.ramelot", "R.tejero", "G.t.montelione"], "pmid": "36041435", "doi": "10.1016/J.CELL.2022.07.019"}
-{"classification": "Hydrolase", "pdb": "3R6M", "deposition_date": "2011-03-21", "title": "Crystal structure of vibrio parahaemolyticus yeaz", "Keywords": ["Ctin/hsp70 nucleotide-binding fold", " bacterial resuscitation", " viable", "But non-culturable state", "Resuscitation promoting factor", "Ygjd", "", "Yjee", "Vibrio parahaemolyticus", "Hydrolase"], "authors": ["A.roujeinikova", "I.aydin"], "pmid": "21858042", "doi": "10.1371/JOURNAL.PONE.0023245"}
-{"classification": "Hydrolase", "pdb": "2W5J", "deposition_date": "2008-12-10", "title": "Structure of the c14-rotor ring of the proton translocating Chloroplast atp synthase", "Keywords": ["Ydrolase", " chloroplast", " atp synthase", " lipid-binding", " cf(0)", " membrane", "", "Transport", "Formylation", "Energy transduction", "Hydrogen ion transport", "", "Ion transport", "Transmembrane", "Membrane protein"], "authors": ["M.vollmar", "D.schlieper", "M.winn", "C.buechner", "G.groth"], "pmid": "19423706", "doi": "10.1074/JBC.M109.006916"}
-{"classification": "De novo protein", "pdb": "4GLU", "deposition_date": "2012-08-14", "title": "Crystal structure of the mirror image form of vegf-a", "Keywords": ["-protein", " covalent dimer", " cysteine knot protein", " growth factor", " de", "Novo protein"], "authors": ["K.mandal", "M.uppalapati", "D.ault-riche", "J.kenney", "J.lowitz", "S.sidhu", "", "S.b.h.kent"], "pmid": "22927390", "doi": "10.1073/PNAS.1210483109"}
-{"classification": "Hydrolase/hydrolase inhibitor", "pdb": "3WYL", "deposition_date": "2014-09-01", "title": "Crystal structure of the catalytic domain of pde10a complexed with 5- Methoxy-3-(1-phenyl-1h-pyrazol-5-yl)-1-(3-(trifluoromethyl)phenyl) Pyridazin-4(1h)-one", "Keywords": ["Ydrolase-hydrolase inhibitor complex"], "authors": ["H.oki", "Y.hayano"], "pmid": "25384088", "doi": "10.1021/JM5013648"}
-{"classification": "Isomerase", "pdb": "5BOR", "deposition_date": "2015-05-27", "title": "Structure of acetobacter aceti pure-s57c, sulfonate form", "Keywords": ["Cidophile", " pure", " purine biosynthesis", " isomerase"], "authors": ["K.l.sullivan", "T.j.kappock"]}
-{"classification": "Hydrolase", "pdb": "1X0C", "deposition_date": "2005-03-17", "title": "Improved crystal structure of isopullulanase from aspergillus niger Atcc 9642", "Keywords": ["Ullulan", " glycoside hydrolase family 49", " glycoprotein", " hydrolase"], "authors": ["M.mizuno", "T.tonozuka", "A.yamamura", "Y.miyasaka", "H.akeboshi", "S.kamitori", "", "A.nishikawa", "Y.sakano"], "pmid": "18155243", "doi": "10.1016/J.JMB.2007.11.098"}
-{"classification": "Oxidoreductase", "pdb": "7CUP", "deposition_date": "2020-08-23", "title": "Structure of 2,5-dihydroxypridine dioxygenase from pseudomonas putida Kt2440", "Keywords": ["On-heme dioxygenase", " oxidoreductase"], "authors": ["G.q.liu", "H.z.tang"]}
-{"classification": "Ligase", "pdb": "1VCN", "deposition_date": "2004-03-10", "title": "Crystal structure of t.th. hb8 ctp synthetase complex with sulfate Anion", "Keywords": ["Etramer", " riken structural genomics/proteomics initiative", " rsgi", "", "Structural genomics", "Ligase"], "authors": ["M.goto", "Riken structural genomics/proteomics initiative (rsgi)"], "pmid": "15296735", "doi": "10.1016/J.STR.2004.05.013"}
-{"classification": "Transferase/transferase inhibitor", "pdb": "6C9V", "deposition_date": "2018-01-28", "title": "Mycobacterium tuberculosis adenosine kinase bound to (2r,3s,4r,5r)-2- (hydroxymethyl)-5-(6-(4-phenylpiperazin-1-yl)-9h-purin-9-yl) Tetrahydrofuran-3,4-diol", "Keywords": ["Ucleoside analog", " complex", " inhibitor", " structural genomics", " psi-2", "", "Protein structure initiative", "Tb structural genomics consortium", "", "Tbsgc", "Transferase-transferase inhibitor complex"], "authors": ["R.a.crespo", "Tb structural genomics consortium (tbsgc)"], "pmid": "31002508", "doi": "10.1021/ACS.JMEDCHEM.9B00020"}
-{"classification": "De novo protein", "pdb": "4LPY", "deposition_date": "2013-07-16", "title": "Crystal structure of tencon variant g10", "Keywords": ["Ibronectin type iii fold", " alternate scaffold", " de novo protein"], "authors": ["A.teplyakov", "G.obmolova", "G.l.gilliland"], "pmid": "24375666", "doi": "10.1002/PROT.24502"}
-{"classification": "Isomerase", "pdb": "2Y88", "deposition_date": "2011-02-03", "title": "Crystal structure of mycobacterium tuberculosis phosphoribosyl Isomerase (variant d11n) with bound prfar", "Keywords": ["Romatic amino acid biosynthesis", " isomerase", " tim-barrel", " histidine", "Biosynthesis", "Tryptophan biosynthesis"], "authors": ["J.kuper", "A.v.due", "A.geerlof", "M.wilmanns"], "pmid": "21321225", "doi": "10.1073/PNAS.1015996108"}
-{"classification": "Unknown function", "pdb": "1SR0", "deposition_date": "2004-03-22", "title": "Crystal structure of signalling protein from sheep(sps-40) at 3.0a Resolution using crystal grown in the presence of polysaccharides", "Keywords": ["Ignalling protein", " involution", " unknown function"], "authors": ["D.b.srivastava", "A.s.ethayathulla", "N.singh", "J.kumar", "S.sharma", "T.p.singh"]}
-{"classification": "Dna binding protein", "pdb": "3RH2", "deposition_date": "2011-04-11", "title": "Crystal structure of a tetr-like transcriptional regulator (sama_0099) From shewanella amazonensis sb2b at 2.42 a resolution", "Keywords": ["Na/rna-binding 3-helical bundle", " structural genomics", " joint center", "For structural genomics", "Jcsg", "Protein structure initiative", "Psi-", "Biology", "Dna binding protein"], "authors": ["Joint center for structural genomics (jcsg)"]}
-{"classification": "Transferase", "pdb": "2WK5", "deposition_date": "2009-06-05", "title": "Structural features of native human thymidine phosphorylase And in complex with 5-iodouracil", "Keywords": ["Lycosyltransferase", " developmental protein", " angiogenesis", "", "5-iodouracil", "Growth factor", "Enzyme kinetics", "", "Differentiation", "Disease mutation", "Thymidine", "Phosphorylase", "Chemotaxis", "Transferase", "Mutagenesis", "", "Polymorphism"], "authors": ["E.mitsiki", "A.c.papageorgiou", "S.iyer", "N.thiyagarajan", "S.h.prior", "", "D.sleep", "C.finnis", "K.r.acharya"], "pmid": "19555658", "doi": "10.1016/J.BBRC.2009.06.104"}
-{"classification": "Hydrolase", "pdb": "3P9Y", "deposition_date": "2010-10-18", "title": "Crystal structure of the drosophila melanogaster ssu72-pctd complex", "Keywords": ["Hosphatase", " cis proline", " lmw ptp-like fold", " rna polymerase ii ctd", "", "Hydrolase"], "authors": ["J.w.werner-allen", "P.zhou"], "pmid": "21159777", "doi": "10.1074/JBC.M110.197129"}
-{"classification": "Recombination/dna", "pdb": "6OEO", "deposition_date": "2019-03-27", "title": "Cryo-em structure of mouse rag1/2 nfc complex (dna1)", "Keywords": ["(d)j recombination", " dna transposition", " rag", " scid", " recombination", "", "Recombination-dna complex"], "authors": ["X.chen", "Y.cui", "Z.h.zhou", "W.yang", "M.gellert"], "pmid": "32015552", "doi": "10.1038/S41594-019-0363-2"}
-{"classification": "Hydrolase", "pdb": "4ECA", "deposition_date": "1997-02-21", "title": "Asparaginase from e. coli, mutant t89v with covalently bound aspartate", "Keywords": ["Ydrolase", " acyl-enzyme intermediate", " threonine amidohydrolase"], "authors": ["G.j.palm", "J.lubkowski", "A.wlodawer"], "pmid": "8706862", "doi": "10.1016/0014-5793(96)00660-6"}
-{"classification": "Transcription/protein binding", "pdb": "3UVX", "deposition_date": "2011-11-30", "title": "Crystal structure of the first bromodomain of human brd4 in complex With a diacetylated histone 4 peptide (h4k12ack16ac)", "Keywords": ["Romodomain", " bromodomain containing protein 4", " cap", " hunk1", " mcap", "", "Mitotic chromosome associated protein", "Peptide complex", "Structural", "Genomics consortium", "Sgc", "Transcription-protein binding complex"], "authors": ["P.filippakopoulos", "S.picaud", "T.keates", "E.ugochukwu", "F.von delft", "", "C.h.arrowsmith", "A.m.edwards", "J.weigelt", "C.bountra", "S.knapp", "Structural", "Genomics consortium (sgc)"], "pmid": "22464331", "doi": "10.1016/J.CELL.2012.02.013"}
-{"classification": "Membrane protein", "pdb": "1TLZ", "deposition_date": "2004-06-10", "title": "Tsx structure complexed with uridine", "Keywords": ["Ucleoside transporter", " beta barrel", " uridine", " membrane", "Protein"], "authors": ["J.ye", "B.van den berg"], "pmid": "15272310", "doi": "10.1038/SJ.EMBOJ.7600330"}
-{"classification": "Dna binding protein", "pdb": "7AZD", "deposition_date": "2020-11-16", "title": "Dna polymerase sliding clamp from escherichia coli with peptide 20 Bound", "Keywords": ["Ntibacterial drug", " dna binding protein"], "authors": ["C.monsarrat", "G.compain", "C.andre", "I.martiel", "S.engilberge", "V.olieric", "", "P.wolff", "K.brillet", "M.landolfo", "C.silva da veiga", "J.wagner", "G.guichard", "", "D.y.burnouf"], "pmid": "34806883", "doi": "10.1021/ACS.JMEDCHEM.1C00918"}
-{"classification": "Transferase", "pdb": "5N3K", "deposition_date": "2017-02-08", "title": "Camp-dependent protein kinase a from cricetulus griseus in complex With fragment like molecule o-guanidino-l-homoserine", "Keywords": ["Ragment", " complex", " transferase", " serine threonine kinase", " camp", "", "Kinase", "Pka"], "authors": ["C.siefker", "A.heine", "G.klebe"]}
-{"classification": "Biosynthetic protein", "pdb": "8H52", "deposition_date": "2022-10-11", "title": "Crystal structure of helicobacter pylori carboxyspermidine Dehydrogenase in complex with nadp", "Keywords": ["Arboxyspermidine dehydrogenase", " biosynthetic protein"], "authors": ["K.y.ko", "S.c.park", "S.y.cho", "S.i.yoon"], "pmid": "36283333", "doi": "10.1016/J.BBRC.2022.10.049"}
-{"classification": "Metal binding protein", "pdb": "6DYC", "deposition_date": "2018-07-01", "title": "Co(ii)-bound structure of the engineered cyt cb562 variant, ch3", "Keywords": ["Esigned protein", " 4-helix bundle", " electron transport", " metal binding", "Protein"], "authors": ["F.a.tezcan", "J.rittle"], "pmid": "30778140", "doi": "10.1038/S41557-019-0218-9"}
-{"classification": "Protein fibril", "pdb": "6A6B", "deposition_date": "2018-06-27", "title": "Cryo-em structure of alpha-synuclein fiber", "Keywords": ["Lpha-syn fiber", " parkinson disease", " protein fibril"], "authors": ["Y.w.li", "C.y.zhao", "F.luo", "Z.liu", "X.gui", "Z.luo", "X.zhang", "D.li", "C.liu", "X.li"], "pmid": "30065316", "doi": "10.1038/S41422-018-0075-X"}
-{"classification": "Dna", "pdb": "7D5E", "deposition_date": "2020-09-25", "title": "Left-handed g-quadruplex containing two bulges", "Keywords": ["-quadruplex", " bulge", " dna", " left-handed"], "authors": ["P.das", "A.maity", "K.h.ngo", "F.r.winnerdy", "B.bakalar", "Y.mechulam", "E.schmitt", "", "A.t.phan"], "pmid": "33503265", "doi": "10.1093/NAR/GKAA1259"}
-{"classification": "Transferase", "pdb": "3RSY", "deposition_date": "2011-05-02", "title": "Cellobiose phosphorylase from cellulomonas uda in complex with sulfate And glycerol", "Keywords": ["H94", " alpha barrel", " cellobiose phosphorylase", " disaccharide", "Phosphorylase", "Transferase"], "authors": ["A.van hoorebeke", "J.stout", "W.soetaert", "J.van beeumen", "T.desmet", "S.savvides"]}
-{"classification": "Oxidoreductase", "pdb": "7MCI", "deposition_date": "2021-04-02", "title": "Mofe protein from azotobacter vinelandii with a sulfur-replenished Cofactor", "Keywords": ["Zotobacter vinelandii", " mofe-protein", " nitrogenase", " oxidoreductase"], "authors": ["W.kang", "C.lee", "Y.hu", "M.w.ribbe"], "doi": "10.1038/S41929-022-00782-7"}
-{"classification": "Dna", "pdb": "1XUW", "deposition_date": "2004-10-26", "title": "Structural rationalization of a large difference in rna affinity Despite a small difference in chemistry between two 2'-o-modified Nucleic acid analogs", "Keywords": ["Na mimetic methylcarbamate amide analog", " dna"], "authors": ["R.pattanayek", "L.sethaphong", "C.pan", "M.prhavc", "T.p.prakash", "M.manoharan", "", "M.egli"], "pmid": "15547979", "doi": "10.1021/JA044637K"}
-{"classification": "Lyase", "pdb": "7C0D", "deposition_date": "2020-05-01", "title": "Crystal structure of azospirillum brasilense l-2-keto-3-deoxyarabonate Dehydratase (hydroxypyruvate-bound form)", "Keywords": ["-2-keto-3-deoxyarabonate dehydratase", " lyase"], "authors": ["Y.watanabe", "S.watanabe"], "pmid": "32697085", "doi": "10.1021/ACS.BIOCHEM.0C00515"}
-{"classification": "Signaling protein", "pdb": "5LYK", "deposition_date": "2016-09-28", "title": "Crystal structure of intracellular b30.2 domain of btn3a1 bound to Citrate", "Keywords": ["30.2", " butyrophilin", " signaling protein"], "authors": ["F.mohammed", "A.t.baker", "M.salim", "B.e.willcox"], "pmid": "28862425", "doi": "10.1021/ACSCHEMBIO.7B00694"}
-{"classification": "Toxin", "pdb": "4IZL", "deposition_date": "2013-01-30", "title": "Structure of the n248a mutant of the panton-valentine leucocidin s Component from staphylococcus aureus", "Keywords": ["I-component leucotoxin", " staphylococcus aureus", " s component", "Leucocidin", "Beta-barrel pore forming toxin", "Toxin"], "authors": ["L.maveyraud", "B.j.laventie", "G.prevost", "L.mourey"], "pmid": "24643034", "doi": "10.1371/JOURNAL.PONE.0092094"}
-{"classification": "Dna", "pdb": "6F3C", "deposition_date": "2017-11-28", "title": "The cytotoxic [pt(h2bapbpy)] platinum complex interacting with the Cgtacg hexamer", "Keywords": ["Rug-dna complex", " four-way junction", " dna"], "authors": ["M.ferraroni", "C.bazzicalupi", "P.gratteri", "F.papi"], "pmid": "31046177", "doi": "10.1002/ANIE.201814532"}
-{"classification": "Signaling protein/inhibitor", "pdb": "4L5M", "deposition_date": "2013-06-11", "title": "Complexe of arno sec7 domain with the protein-protein interaction Inhibitor n-(4-hydroxy-2,6-dimethylphenyl)benzenesulfonamide at ph6.5", "Keywords": ["Ec-7domain", " signaling protein-inhibitor complex"], "authors": ["F.hoh", "J.rouhana"], "pmid": "24112024", "doi": "10.1021/JM4009357"}
-{"classification": "Signaling protein", "pdb": "5I6J", "deposition_date": "2016-02-16", "title": "Crystal structure of srgap2 f-barx", "Keywords": ["Rgap2", " f-bar", " fx", " signaling protein"], "authors": ["M.sporny", "J.guez-haddad", "M.n.isupov", "Y.opatowsky"], "pmid": "28333212", "doi": "10.1093/MOLBEV/MSX094"}
-{"classification": "Metal binding protein", "pdb": "1Q80", "deposition_date": "2003-08-20", "title": "Solution structure and dynamics of nereis sarcoplasmic calcium binding Protein", "Keywords": ["Ll-alpha", " metal binding protein"], "authors": ["G.rabah", "R.popescu", "J.a.cox", "Y.engelborghs", "C.t.craescu"], "pmid": "15819893", "doi": "10.1111/J.1742-4658.2005.04629.X"}
-{"classification": "Transferase", "pdb": "1TW1", "deposition_date": "2004-06-30", "title": "Beta-1,4-galactosyltransferase mutant met344his (m344h-gal-t1) complex With udp-galactose and magnesium", "Keywords": ["Et344his mutation; closed conformation; mn binding", " transferase"], "authors": ["B.ramakrishnan", "E.boeggeman", "P.k.qasba"], "pmid": "15449940", "doi": "10.1021/BI049007+"}
-{"classification": "Rna", "pdb": "2PN4", "deposition_date": "2007-04-23", "title": "Crystal structure of hepatitis c virus ires subdomain iia", "Keywords": ["Cv", " ires", " subdoamin iia", " rna", " strontium", " hepatitis"], "authors": ["Q.zhao", "Q.han", "C.r.kissinger", "P.a.thompson"], "pmid": "18391410", "doi": "10.1107/S0907444908002011"}
+{"pdb": "1CW0", "title": "crystal structure analysis of very short patch repair (vsr) endonuclease in complex with a duplex dna", "authors": ["S.E.Tsutakawa", "H.Jingami", "K.Morikawa"], "doi": "10.1016/S0092-8674(00)81550-0", "pmid": "10612397"}
+{"pdb": "2CWW", "title": "crystal structure of thermus thermophilus ttha1280, a putative sam- dependent rna methyltransferase, in complex with s-adenosyl-l- homocysteine", "authors": ["A.A.Pioszak", "K.Murayama", "N.Nakagawa", "A.Ebihara", "S.Kuramitsu", "M.Shirouzu", "S.Yokoyama", "Riken Structural Genomics/proteomics Initiative (Rsgi)"], "doi": "10.1107/S1744309105029842", "pmid": "16511182"}
+{"pdb": "6CWE", "title": "structure of alpha-gsa[8,6p] bound by cd1d and in complex with the va14vb8.2 tcr", "authors": ["J.Wang", "D.Zajonc"], "doi": null, "pmid": null}
+{"pdb": "5CWS", "title": "crystal structure of the intact chaetomium thermophilum nsp1-nup49- nup57 channel nucleoporin heterotrimer bound to its nic96 nuclear pore complex attachment site", "authors": ["C.J.Bley", "S.Petrovic", "M.Paduch", "V.Lu", "A.A.Kossiakoff", "A.Hoelz"], "doi": "10.1126/SCIENCE.AAC9176", "pmid": "26316600"}
+{"pdb": "5CWE", "title": "structure of cyp107l2 from streptomyces avermitilis with lauric acid", "authors": ["T.-V.Pham", "S.-H.Han", "J.-H.Kim", "D.-H.Kim", "L.-W.Kang"], "doi": null, "pmid": null}
+{"pdb": "7CW4", "title": "acetyl-coa acetyltransferase from bacillus cereus atcc 14579", "authors": ["J.Hong", "K.J.Kim"], "doi": "10.1016/J.BBRC.2020.09.048", "pmid": "32972748"}
+{"pdb": "2CWP", "title": "crystal structure of metrs related protein from pyrococcus horikoshii", "authors": ["K.Murayama", "M.Kato-Murayama", "M.Shirouzu", "S.Yokoyama", "Riken StructuralGenomics/proteomics Initiative (Rsgi)"], "doi": null, "pmid": null}
+{"pdb": "2CW7", "title": "crystal structure of intein homing endonuclease ii", "authors": ["H.Matsumura", "H.Takahashi", "T.Inoue", "H.Hashimoto", "M.Nishioka", "S.Fujiwara", "M.Takagi", "T.Imanaka", "Y.Kai"], "doi": "10.1002/PROT.20858", "pmid": "16493661"}
+{"pdb": "1CWU", "title": "brassica napus enoyl acp reductase a138g mutant complexed with nad+ and thienodiazaborine", "authors": ["A.Roujeinikova", "J.B.Rafferty", "D.W.Rice"], "doi": "10.1074/JBC.274.43.30811", "pmid": "10521472"}
+{"pdb": "3CWN", "title": "escherichia coli transaldolase b mutant f178y", "authors": ["T.Sandalova", "G.Schneider", "A.Samland"], "doi": "10.1074/JBC.M803184200", "pmid": "18687684"}
+{"pdb": "1CWL", "title": "human cyclophilin a complexed with 4 4-hydroxy-meleu cyclosporin", "authors": ["V.Mikol", "J.Kallen", "P.Taylor", "M.D.Walkinshaw"], "doi": "10.1006/JMBI.1998.2108", "pmid": "9769216"}
+{"pdb": "3CW2", "title": "crystal structure of the intact archaeal translation initiation factor 2 from sulfolobus solfataricus .", "authors": ["E.A.Stolboushkina", "S.V.Nikonov", "A.D.Nikulin", "U.Blaesi", "D.J.Manstein", "R.V.Fedorov", "M.B.Garber", "O.S.Nikonov"], "doi": "10.1016/J.JMB.2008.07.039", "pmid": "18675278"}
+{"pdb": "3CW9", "title": "4-chlorobenzoyl-coa ligase/synthetase in the thioester-forming conformation, bound to 4-chlorophenacyl-coa", "authors": ["A.S.Reger", "J.Cao", "R.Wu", "D.Dunaway-Mariano", "A.M.Gulick"], "doi": "10.1021/BI800696Y", "pmid": "18620418"}
+{"pdb": "3CWU", "title": "crystal structure of an alka host/guest complex 2'-fluoro-2'-deoxy-1, n6-ethenoadenine:thymine base pair", "authors": ["B.R.Bowman", "S.Lee", "S.Wang", "G.L.Verdine"], "doi": "10.1016/J.STR.2008.04.012", "pmid": "18682218"}
+{"pdb": "5CWF", "title": "crystal structure of de novo designed helical repeat protein dhr8", "authors": ["G.Bhabha", "D.C.Ekiert"], "doi": "10.1038/NATURE16162", "pmid": "26675729"}

dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/sx/graph/bio/uniprot_dump

@@ -1,36 +1,6 @@
-{"pid": " Q6GZX4", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 43"}], "title": "Putative transcription factor 001R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZX3", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 45"}], "title": "Uncharacterized protein 002L", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197F8", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2022-02-23", "date_info": "entry version 29"}], "title": "Uncharacterized protein 002R", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q197F7", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2020-08-12", "date_info": "entry version 23"}], "title": "Uncharacterized protein 003L", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZX2", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 37"}], "title": "Uncharacterized protein 3R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZX1", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 38"}], "title": "Uncharacterized protein 004R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197F5", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2022-10-12", "date_info": "entry version 32"}], "title": "Uncharacterized protein 005L", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZX0", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 47"}], "title": "Uncharacterized protein 005R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q91G88", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2001-12-01", "date_info": "sequence version 1"}, {"date": "2023-06-28", "date_info": "entry version 53"}], "title": "Putative KilA-N domain-containing protein 006L", "organism_species": "Invertebrate iridescent virus 6 (IIV-6) (Chilo iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Iridovirus"], "references": [{"PubMed": "17239238"}, {"DOI": "10.1186/1743-422x-4-11"}]}
-{"pid": " Q6GZW9", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 34"}], "title": "Uncharacterized protein 006R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZW8", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 32"}], "title": "Uncharacterized protein 007R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197F3", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2023-02-22", "date_info": "entry version 28"}], "title": "Uncharacterized protein 007R", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q197F2", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2022-02-23", "date_info": "entry version 22"}], "title": "Uncharacterized protein 008L", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZW6", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 67"}], "title": "Putative helicase 009L", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q91G85", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2001-12-01", "date_info": "sequence version 1"}, {"date": "2023-02-22", "date_info": "entry version 38"}], "title": "Uncharacterized protein 009R", "organism_species": "Invertebrate iridescent virus 6 (IIV-6) (Chilo iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Iridovirus"], "references": [{"PubMed": "17239238"}, {"DOI": "10.1186/1743-422x-4-11"}]}
-{"pid": " Q6GZW5", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 37"}], "title": "Uncharacterized protein 010R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197E9", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2023-02-22", "date_info": "entry version 28"}], "title": "Uncharacterized protein 011L", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZW4", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 37"}], "title": "Uncharacterized protein 011R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZW3", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 35"}], "title": "Uncharacterized protein 012L", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197E7", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2023-02-22", "date_info": "entry version 37"}], "title": "Uncharacterized protein IIV3-013L", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZW2", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 30"}], "title": "Uncharacterized protein 013R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZW1", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 35"}], "title": "Uncharacterized protein 014R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZW0", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 50"}], "title": "Uncharacterized protein 015R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZV8", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 35"}], "title": "Uncharacterized protein 017L", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZV7", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 33"}], "title": "Uncharacterized protein 018L", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZV6", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 87"}], "title": "Putative serine/threonine-protein kinase 019R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZV5", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 40"}], "title": "Uncharacterized protein 020R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZV4", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 35"}], "title": "Uncharacterized protein 021L", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197D8", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2022-12-14", "date_info": "entry version 35"}], "title": "Transmembrane protein 022L", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZV2", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 33"}], "title": "Uncharacterized protein 023R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197D7", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2023-02-22", "date_info": "entry version 25"}], "title": "Uncharacterized protein 023R", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q6GZV1", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 37"}], "title": "Uncharacterized protein 024R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q197D5", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2006-07-11", "date_info": "sequence version 1"}, {"date": "2022-10-12", "date_info": "entry version 24"}], "title": "Uncharacterized protein 025R", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus"], "references": [{"PubMed": "16912294"}, {"DOI": "10.1128/jvi.00464-06"}]}
-{"pid": " Q91G70", "dates": [{"date": "2009-06-16", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2001-12-01", "date_info": "sequence version 1"}, {"date": "2020-08-12", "date_info": "entry version 32"}], "title": "Uncharacterized protein 026R", "organism_species": "Invertebrate iridescent virus 6 (IIV-6) (Chilo iridescent virus)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Iridovirus"], "references": [{"PubMed": "17239238"}, {"DOI": "10.1186/1743-422x-4-11"}]}
-{"pid": " Q6GZU9", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 49"}], "title": "Uncharacterized protein 027R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
-{"pid": " Q6GZU8", "dates": [{"date": "2011-06-28", "date_info": "integrated into UniProtKB/Swiss-Prot"}, {"date": "2004-07-19", "date_info": "sequence version 1"}, {"date": "2023-09-13", "date_info": "entry version 55"}], "title": "Uncharacterized protein 028R", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3)", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus"], "references": [{"PubMed": "15165820"}, {"DOI": "10.1016/j.virol.2004.02.019"}]}
+{"pid": "Q6GZX4", "dates": [{"date": "28-JUN-2011", "date_info": " integrated into UniProtKB/Swiss-Prot."}, {"date": "19-JUL-2004", "date_info": " sequence version 1."}, {"date": "12-AUG-2020", "date_info": " entry version 41."}], "title": "Putative transcription factor 001R;", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3).", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus."], "references": [{"PubMed": "15165820"}, {" DOI": "10.1016/j.virol.2004.02.019"}]}
+{"pid": "Q6GZX3", "dates": [{"date": "28-JUN-2011", "date_info": " integrated into UniProtKB/Swiss-Prot."}, {"date": "19-JUL-2004", "date_info": " sequence version 1."}, {"date": "12-AUG-2020", "date_info": " entry version 42."}], "title": "Uncharacterized protein 002L;", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3).", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus."], "references": [{"PubMed": "15165820"}, {" DOI": "10.1016/j.virol.2004.02.019"}]}
+{"pid": "Q197F8", "dates": [{"date": "16-JUN-2009", "date_info": " integrated into UniProtKB/Swiss-Prot."}, {"date": "11-JUL-2006", "date_info": " sequence version 1."}, {"date": "12-AUG-2020", "date_info": " entry version 27."}], "title": "Uncharacterized protein 002R;", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus).", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus."], "references": [{"PubMed": "16912294"}, {" DOI": "10.1128/jvi.00464-06"}]}
+{"pid": "Q197F7", "dates": [{"date": "16-JUN-2009", "date_info": " integrated into UniProtKB/Swiss-Prot."}, {"date": "11-JUL-2006", "date_info": " sequence version 1."}, {"date": "12-AUG-2020", "date_info": " entry version 23."}], "title": "Uncharacterized protein 003L;", "organism_species": "Invertebrate iridescent virus 3 (IIV-3) (Mosquito iridescent virus).", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Betairidovirinae", "Chloriridovirus."], "references": [{"PubMed": "16912294"}, {" DOI": "10.1128/jvi.00464-06"}]}
+{"pid": "Q6GZX2", "dates": [{"date": "28-JUN-2011", "date_info": " integrated into UniProtKB/Swiss-Prot."}, {"date": "19-JUL-2004", "date_info": " sequence version 1."}, {"date": "12-AUG-2020", "date_info": " entry version 36."}], "title": "Uncharacterized protein 3R;", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3).", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus."], "references": [{"PubMed": "15165820"}, {" DOI": "10.1016/j.virol.2004.02.019"}]}
+{"pid": "Q6GZX1", "dates": [{"date": "28-JUN-2011", "date_info": " integrated into UniProtKB/Swiss-Prot."}, {"date": "19-JUL-2004", "date_info": " sequence version 1."}, {"date": "12-AUG-2020", "date_info": " entry version 34."}], "title": "Uncharacterized protein 004R;", "organism_species": "Frog virus 3 (isolate Goorha) (FV-3).", "subjects": ["Viruses", "Varidnaviria", "Bamfordvirae", "Nucleocytoviricota", "Megaviricetes", "Pimascovirales", "Iridoviridae", "Alphairidovirinae", "Ranavirus."], "references": [{"PubMed": "15165820"}, {" DOI": "10.1016/j.virol.2004.02.019"}]}

dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/transform/synonyms.txt

@@ -1496,30 +1496,4 @@ cnr:institutes @=@ __CDS131__ @=@ IBE - Istituto per la BioEconomia
 cnr:institutes @=@ https://ror.org/0263zy895 @=@ CDS132
 cnr:institutes @=@ https://ror.org/0263zy895 @=@ SCITEC - Istituto di Scienze e Tecnologie Chimiche \"Giulio Natta\"
 cnr:institutes @=@ __CDS133__ @=@ CDS133
-cnr:institutes @=@ __CDS133__ @=@ STEMS - Istituto di Scienze e Tecnologie per l'Energia e la Mobilità Sostenibili
-base:normalized_types @=@ Text @=@ 1
-base:normalized_types @=@ Book @=@ 11
-base:normalized_types @=@ Book part @=@ 111
-base:normalized_types @=@ Journal/Newspaper @=@ 12
-base:normalized_types @=@ Article contribution @=@ 121
-base:normalized_types @=@ Other non-article @=@ 122
-base:normalized_types @=@ Conference object @=@ 13
-base:normalized_types @=@ Report @=@ 14
-base:normalized_types @=@ Review @=@ 15
-base:normalized_types @=@ Course material @=@ 16
-base:normalized_types @=@ Lecture @=@ 17
-base:normalized_types @=@ Thesis @=@ 18
-base:normalized_types @=@ Bachelor's thesis @=@ 181
-base:normalized_types @=@ Master's thesis @=@ 182
-base:normalized_types @=@ Doctoral and postdoctoral thesis @=@ 183
-base:normalized_types @=@ Manuscript @=@ 19
-base:normalized_types @=@ Patent @=@ 1A
-base:normalized_types @=@ Musical notation @=@ 2
-base:normalized_types @=@ Map @=@ 3
-base:normalized_types @=@ Audio @=@ 4
-base:normalized_types @=@ Image/Video @=@ 5
-base:normalized_types @=@ Still image @=@ 51
-base:normalized_types @=@ Moving image/Video @=@ 52
-base:normalized_types @=@ Software @=@ 6
-base:normalized_types @=@ Dataset @=@ 7
-base:normalized_types @=@ Unknown @=@ F
+cnr:institutes @=@ __CDS133__ @=@ STEMS - Istituto di Scienze e Tecnologie per l'Energia e la Mobilità Sostenibili

dhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/transform/terms.txt

@@ -1210,29 +1210,4 @@ cnr:institutes @=@ cnr:institutes @=@ __CDS130__ @=@ __CDS130__
 cnr:institutes @=@ cnr:institutes @=@ __CDS131__ @=@ __CDS131__
 cnr:institutes @=@ cnr:institutes @=@ https://ror.org/0263zy895 @=@ https://ror.org/0263zy895
 cnr:institutes @=@ cnr:institutes @=@ __CDS133__ @=@ __CDS133__
-base:normalized_types @=@ base:normalized_types @=@ Text @=@ Text
-base:normalized_types @=@ base:normalized_types @=@ Book @=@ Book
-base:normalized_types @=@ base:normalized_types @=@ Book part @=@ Book part
-base:normalized_types @=@ base:normalized_types @=@ Journal/Newspaper @=@ Journal/Newspaper
-base:normalized_types @=@ base:normalized_types @=@ Article contribution @=@ Article contribution
-base:normalized_types @=@ base:normalized_types @=@ Other non-article @=@ Other non-article
-base:normalized_types @=@ base:normalized_types @=@ Conference object @=@ Conference object
-base:normalized_types @=@ base:normalized_types @=@ Report @=@ Report
-base:normalized_types @=@ base:normalized_types @=@ Review @=@ Review
-base:normalized_types @=@ base:normalized_types @=@ Course material @=@ Course material
-base:normalized_types @=@ base:normalized_types @=@ Lecture @=@ Lecture
-base:normalized_types @=@ base:normalized_types @=@ Thesis @=@ Thesis
-base:normalized_types @=@ base:normalized_types @=@ Bachelor's thesis @=@ Bachelor's thesis
-base:normalized_types @=@ base:normalized_types @=@ Master's thesis @=@ Master's thesis
-base:normalized_types @=@ base:normalized_types @=@ Doctoral and postdoctoral thesis @=@ Doctoral and postdoctoral thesis
-base:normalized_types @=@ base:normalized_types @=@ Manuscript @=@ Manuscript
-base:normalized_types @=@ base:normalized_types @=@ Patent @=@ Patent
-base:normalized_types @=@ base:normalized_types @=@ Musical notation @=@ Musical notation
-base:normalized_types @=@ base:normalized_types @=@ Map @=@ Map
-base:normalized_types @=@ base:normalized_types @=@ Audio @=@ Audio
-base:normalized_types @=@ base:normalized_types @=@ Image/Video @=@ Image/Video
-base:normalized_types @=@ base:normalized_types @=@ Still image @=@ Still image
-base:normalized_types @=@ base:normalized_types @=@ Moving image/Video @=@ Moving image/Video
-base:normalized_types @=@ base:normalized_types @=@ Software @=@ Software
-base:normalized_types @=@ base:normalized_types @=@ Dataset @=@ Dataset
-base:normalized_types @=@ base:normalized_types @=@ Unknown @=@ Unknown
+

dhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/DedupRecordFactory.java

@@ -1,16 +1,6 @@
 
 package eu.dnetlib.dhp.oa.dedup;
 
-import java.util.*;
-import java.util.stream.Stream;
-
-import org.apache.commons.beanutils.BeanUtils;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.spark.api.java.function.FlatMapFunction;
-import org.apache.spark.api.java.function.MapFunction;
-import org.apache.spark.api.java.function.ReduceFunction;
-import org.apache.spark.sql.*;
-
 import eu.dnetlib.dhp.oa.dedup.model.Identifier;
 import eu.dnetlib.dhp.oa.merge.AuthorMerger;
 import eu.dnetlib.dhp.schema.common.ModelSupport;
@@ -18,195 +8,180 @@ import eu.dnetlib.dhp.schema.oaf.Author;
 import eu.dnetlib.dhp.schema.oaf.DataInfo;
 import eu.dnetlib.dhp.schema.oaf.OafEntity;
 import eu.dnetlib.dhp.schema.oaf.Result;
+import org.apache.commons.beanutils.BeanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.MapFunction;
+import org.apache.spark.api.java.function.ReduceFunction;
+import org.apache.spark.sql.*;
 import scala.Tuple2;
 import scala.Tuple3;
 import scala.collection.JavaConversions;
 
+import java.util.*;
+import java.util.stream.Stream;
+
 public class DedupRecordFactory {
-	public static final class DedupRecordReduceState {
-		public final String dedupId;
+    public static final class DedupRecordReduceState {
+        public final String dedupId;
 
-		public final ArrayList<String> aliases = new ArrayList<>();
+        public final ArrayList<String> aliases = new ArrayList<>();
 
-		public final HashSet<String> acceptanceDate = new HashSet<>();
+        public final HashSet<String> acceptanceDate = new HashSet<>();
 
-		public OafEntity entity;
+        public OafEntity entity;
 
-		public DedupRecordReduceState(String dedupId, String id, OafEntity entity) {
-			this.dedupId = dedupId;
-			this.entity = entity;
-			if (entity == null) {
-				aliases.add(id);
-			} else {
-				if (Result.class.isAssignableFrom(entity.getClass())) {
-					Result result = (Result) entity;
-					if (result.getDateofacceptance() != null
-						&& StringUtils.isNotBlank(result.getDateofacceptance().getValue())) {
-						acceptanceDate.add(result.getDateofacceptance().getValue());
-					}
-				}
-			}
-		}
+        public DedupRecordReduceState(String dedupId, String id, OafEntity entity) {
+            this.dedupId = dedupId;
+            this.entity = entity;
+            if (entity == null) {
+                aliases.add(id);
+            } else {
+                if (Result.class.isAssignableFrom(entity.getClass())) {
+                    Result result = (Result) entity;
+                    if (result.getDateofacceptance() != null && StringUtils.isNotBlank(result.getDateofacceptance().getValue())) {
+                        acceptanceDate.add(result.getDateofacceptance().getValue());
+                    }
+                }
+            }
+        }
 
-		public String getDedupId() {
-			return dedupId;
-		}
-	}
+        public String getDedupId() {
+            return dedupId;
+        }
+    }
+    private static final int MAX_ACCEPTANCE_DATE = 20;
 
-	private static final int MAX_ACCEPTANCE_DATE = 20;
+    private DedupRecordFactory() {
+    }
 
-	private DedupRecordFactory() {
-	}
+    public static Dataset<OafEntity> createDedupRecord(
+            final SparkSession spark,
+            final DataInfo dataInfo,
+            final String mergeRelsInputPath,
+            final String entitiesInputPath,
+            final Class<OafEntity> clazz) {
 
-	public static Dataset<OafEntity> createDedupRecord(
-		final SparkSession spark,
-		final DataInfo dataInfo,
-		final String mergeRelsInputPath,
-		final String entitiesInputPath,
-		final Class<OafEntity> clazz) {
+        final long ts = System.currentTimeMillis();
+        final Encoder<OafEntity> beanEncoder = Encoders.bean(clazz);
+        final Encoder<OafEntity> kryoEncoder = Encoders.kryo(clazz);
 
-		final long ts = System.currentTimeMillis();
-		final Encoder<OafEntity> beanEncoder = Encoders.bean(clazz);
-		final Encoder<OafEntity> kryoEncoder = Encoders.kryo(clazz);
+        // <id, json_entity>
+        Dataset<Row> entities = spark
+                .read()
+                .schema(Encoders.bean(clazz).schema())
+                .json(entitiesInputPath)
+                .as(beanEncoder)
+                .map(
+                        (MapFunction<OafEntity, Tuple2<String, OafEntity>>) entity -> {
+                            return new Tuple2<>(entity.getId(), entity);
+                        },
+                        Encoders.tuple(Encoders.STRING(), kryoEncoder))
+                .selectExpr("_1 AS id", "_2 AS kryoObject");
 
-		// <id, json_entity>
-		Dataset<Row> entities = spark
-			.read()
-			.schema(Encoders.bean(clazz).schema())
-			.json(entitiesInputPath)
-			.as(beanEncoder)
-			.map(
-				(MapFunction<OafEntity, Tuple2<String, OafEntity>>) entity -> {
-					return new Tuple2<>(entity.getId(), entity);
-				},
-				Encoders.tuple(Encoders.STRING(), kryoEncoder))
-			.selectExpr("_1 AS id", "_2 AS kryoObject");
+        // <source, target>: source is the dedup_id, target is the id of the mergedIn
+        Dataset<Row> mergeRels = spark
+                .read()
+                .load(mergeRelsInputPath)
+                .where("relClass == 'merges'")
+                .selectExpr("source as dedupId", "target as id");
 
-		// <source, target>: source is the dedup_id, target is the id of the mergedIn
-		Dataset<Row> mergeRels = spark
-			.read()
-			.load(mergeRelsInputPath)
-			.where("relClass == 'merges'")
-			.selectExpr("source as dedupId", "target as id");
+        return mergeRels
+                .join(entities, JavaConversions.asScalaBuffer(Collections.singletonList("id")), "left")
+                .select("dedupId", "id", "kryoObject")
+                .as(Encoders.tuple(Encoders.STRING(), Encoders.STRING(), kryoEncoder))
+                .map((MapFunction<Tuple3<String, String, OafEntity>, DedupRecordReduceState>) t -> new DedupRecordReduceState(t._1(), t._2(), t._3()), Encoders.kryo(DedupRecordReduceState.class))
+                .groupByKey((MapFunction<DedupRecordReduceState, String>) DedupRecordReduceState::getDedupId, Encoders.STRING())
+                .reduceGroups(
+                        (ReduceFunction<DedupRecordReduceState>) (t1, t2) -> {
+                            if (t1.entity == null) {
+                                t2.aliases.addAll(t1.aliases);
+                                return t2;
+                            }
+                            if (t1.acceptanceDate.size() < MAX_ACCEPTANCE_DATE) {
+                                t1.acceptanceDate.addAll(t2.acceptanceDate);
+                            }
+                            t1.aliases.addAll(t2.aliases);
+                            t1.entity = reduceEntity(t1.entity, t2.entity);
 
-		return mergeRels
-			.join(entities, JavaConversions.asScalaBuffer(Collections.singletonList("id")), "left")
-			.select("dedupId", "id", "kryoObject")
-			.as(Encoders.tuple(Encoders.STRING(), Encoders.STRING(), kryoEncoder))
-			.map(
-				(MapFunction<Tuple3<String, String, OafEntity>, DedupRecordReduceState>) t -> new DedupRecordReduceState(
-					t._1(), t._2(), t._3()),
-				Encoders.kryo(DedupRecordReduceState.class))
-			.groupByKey(
-				(MapFunction<DedupRecordReduceState, String>) DedupRecordReduceState::getDedupId, Encoders.STRING())
-			.reduceGroups(
-				(ReduceFunction<DedupRecordReduceState>) (t1, t2) -> {
-					if (t1.entity == null) {
-						t2.aliases.addAll(t1.aliases);
-						return t2;
-					}
-					if (t1.acceptanceDate.size() < MAX_ACCEPTANCE_DATE) {
-						t1.acceptanceDate.addAll(t2.acceptanceDate);
-					}
-					t1.aliases.addAll(t2.aliases);
-					t1.entity = reduceEntity(t1.entity, t2.entity);
+                            return t1;
+                        }
+                )
+                .flatMap
+                        ((FlatMapFunction<Tuple2<String, DedupRecordReduceState>, OafEntity>) t -> {
+                            String dedupId = t._1();
+                            DedupRecordReduceState agg = t._2();
 
-					return t1;
-				})
-			.flatMap((FlatMapFunction<Tuple2<String, DedupRecordReduceState>, OafEntity>) t -> {
-				String dedupId = t._1();
-				DedupRecordReduceState agg = t._2();
+                            if (agg.acceptanceDate.size() >= MAX_ACCEPTANCE_DATE) {
+                                return Collections.emptyIterator();
+                            }
 
-				if (agg.acceptanceDate.size() >= MAX_ACCEPTANCE_DATE) {
-					return Collections.emptyIterator();
-				}
+                            return Stream.concat(Stream.of(agg.getDedupId()), agg.aliases.stream())
+                                    .map(id -> {
+                                        try {
+                                            OafEntity res = (OafEntity) BeanUtils.cloneBean(agg.entity);
+                                            res.setId(id);
+                                            res.setDataInfo(dataInfo);
+                                            res.setLastupdatetimestamp(ts);
+                                            return res;
+                                        } catch (Exception e) {
+                                            throw new RuntimeException(e);
+                                        }
+                                    }).iterator();
+                        }, beanEncoder);
+    }
 
-				return Stream
-					.concat(
-						Stream
-							.of(agg.getDedupId())
-							.map(id -> createDedupOafEntity(id, agg.entity, dataInfo, ts)),
-						agg.aliases
-							.stream()
-							.map(id -> createMergedDedupAliasOafEntity(id, agg.entity, dataInfo, ts)))
-					.iterator();
-			}, beanEncoder);
-	}
-
-	private static OafEntity createDedupOafEntity(String id, OafEntity base, DataInfo dataInfo, long ts) {
-		try {
-			OafEntity res = (OafEntity) BeanUtils.cloneBean(base);
-			res.setId(id);
-			res.setDataInfo(dataInfo);
-			res.setLastupdatetimestamp(ts);
-			return res;
-		} catch (Exception e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	private static OafEntity createMergedDedupAliasOafEntity(String id, OafEntity base, DataInfo dataInfo, long ts) {
-		try {
-			OafEntity res = createDedupOafEntity(id, base, dataInfo, ts);
-			DataInfo ds = (DataInfo) BeanUtils.cloneBean(dataInfo);
-			ds.setDeletedbyinference(true);
-			res.setDataInfo(ds);
-			return res;
-		} catch (Exception e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	private static OafEntity reduceEntity(OafEntity entity, OafEntity duplicate) {
+    private static OafEntity reduceEntity(OafEntity entity, OafEntity duplicate) {
 
 		if (duplicate == null) {
 			return entity;
 		}
 
-		int compare = new IdentifierComparator<>()
-			.compare(Identifier.newInstance(entity), Identifier.newInstance(duplicate));
 
-		if (compare > 0) {
+        int compare = new IdentifierComparator<>()
+                .compare(Identifier.newInstance(entity), Identifier.newInstance(duplicate));
+
+        if (compare > 0) {
 			OafEntity swap = duplicate;
-			duplicate = entity;
-			entity = swap;
-		}
+            duplicate = entity;
+            entity = swap;
+        }
 
-		entity.mergeFrom(duplicate);
+        entity.mergeFrom(duplicate);
 
-		if (ModelSupport.isSubClass(duplicate, Result.class)) {
-			Result re = (Result) entity;
-			Result rd = (Result) duplicate;
+        if (ModelSupport.isSubClass(duplicate, Result.class)) {
+            Result re = (Result) entity;
+            Result rd = (Result) duplicate;
 
-			List<List<Author>> authors = new ArrayList<>();
-			if (re.getAuthor() != null) {
-				authors.add(re.getAuthor());
-			}
-			if (rd.getAuthor() != null) {
-				authors.add(rd.getAuthor());
-			}
+            List<List<Author>> authors = new ArrayList<>();
+            if (re.getAuthor() != null) {
+                authors.add(re.getAuthor());
+            }
+            if (rd.getAuthor() != null) {
+                authors.add(rd.getAuthor());
+            }
 
-			re.setAuthor(AuthorMerger.merge(authors));
-		}
+            re.setAuthor(AuthorMerger.merge(authors));
+        }
 
-		return entity;
-	}
+        return entity;
+    }
 
-	public static <T extends OafEntity> T entityMerger(
-		String id, Iterator<Tuple2<String, T>> entities, long ts, DataInfo dataInfo, Class<T> clazz) {
-		T base = entities.next()._2();
+    public static <T extends OafEntity> T entityMerger(
+            String id, Iterator<Tuple2<String, T>> entities, long ts, DataInfo dataInfo, Class<T> clazz) {
+        T base = entities.next()._2();
 
-		while (entities.hasNext()) {
-			T duplicate = entities.next()._2();
-			if (duplicate != null)
-				base = (T) reduceEntity(base, duplicate);
-		}
+        while (entities.hasNext()) {
+            T duplicate = entities.next()._2();
+            if (duplicate != null)
+                base = (T) reduceEntity(base, duplicate);
+        }
 
-		base.setId(id);
-		base.setDataInfo(dataInfo);
-		base.setLastupdatetimestamp(ts);
+        base.setId(id);
+        base.setDataInfo(dataInfo);
+        base.setLastupdatetimestamp(ts);
 
-		return base;
-	}
+        return base;
+    }
 
 }

dhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/SparkCreateMergeRels.java

@@ -242,14 +242,13 @@ public class SparkCreateMergeRels extends AbstractSparkAction {
 
 						// this was a pivot in a previous graph but it has been merged into a new group with different
 						// pivot
-						if (!r.isNullAt(r.fieldIndex("lastUsage")) && !pivot.equals(id)
-							&& !dedupId.equals(pivotDedupId)) {
+						if (!r.isNullAt(r.fieldIndex("lastUsage")) && !pivot.equals(id) && !dedupId.equals(pivotDedupId)) {
 							// materialize the previous dedup record as a merge relation with the new one
 							res.add(new Tuple3<>(dedupId, pivotDedupId, null));
 						}
 
 						// add merge relations
-						if (cut <= 0 || r.<Integer> getAs("position") <= cut) {
+						if (cut <=0 || r.<Integer>getAs("position") <= cut) {
 							res.add(new Tuple3<>(id, pivotDedupId, pivot));
 						}
 

dhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/openorgs/oozie_app/config-default.xml

@@ -15,12 +15,4 @@
         <name>oozie.action.sharelib.for.spark</name>
         <value>spark2</value>
     </property>
-    <property>
-        <name>hiveMetastoreUris</name>
-        <value>thrift://iis-cdh5-test-m3.ocean.icm.edu.pl:9083</value>
-    </property>
-    <property>
-        <name>pivotHistoryDatabase</name>
-        <value>&#x200B;</value>
-    </property>
 </configuration>

dhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/openorgs/oozie_app/workflow.xml

@@ -198,8 +198,6 @@
             <arg>--isLookUpUrl</arg><arg>${isLookUpUrl}</arg>
             <arg>--actionSetId</arg><arg>${actionSetId}</arg>
             <arg>--cutConnectedComponent</arg><arg>${cutConnectedComponent}</arg>
-            <arg>--hiveMetastoreUris</arg><arg>${hiveMetastoreUris}</arg>
-            <arg>--pivotHistoryDatabase</arg><arg>${pivotHistoryDatabase}</arg>
         </spark>
         <ok to="PrepareOrgRels"/>
         <error to="Kill"/>

dhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/pivothistory/oozie_app/config-default.xml

@@ -1,26 +0,0 @@
-<configuration>
-    <property>
-        <name>jobTracker</name>
-        <value>yarnRM</value>
-    </property>
-    <property>
-        <name>nameNode</name>
-        <value>hdfs://nameservice1</value>
-    </property>
-    <property>
-        <name>oozie.use.system.libpath</name>
-        <value>true</value>
-    </property>
-    <property>
-        <name>oozie.action.sharelib.for.spark</name>
-        <value>spark2</value>
-    </property>
-    <property>
-        <name>hiveMetastoreUris</name>
-        <value>thrift://iis-cdh5-test-m3.ocean.icm.edu.pl:9083</value>
-    </property>
-    <property>
-        <name>sparkSqlWarehouseDir</name>
-        <value>/user/hive/warehouse</value>
-    </property>
-</configuration>

dhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/pivothistory/oozie_app/sql.sql

@@ -1,62 +0,0 @@
-
-CREATE TABLE `${pivot_history_db}`.`dataset_new` STORED AS PARQUET AS
-WITH pivots (
-    SELECT property.value AS id, '${new_graph_date}' AS usedIn  FROM `${new_graph_db}`.`relation`
-    LEFT SEMI JOIN `${new_graph_db}`.`dataset` ON relation.source = dataset.id
-    LATERAL VIEW EXPLODE(properties) AS property WHERE relClass = 'isMergedIn' AND property.key = 'pivot'
-UNION
-    SELECT id, usedIn FROM `${pivot_history_db}`.`dataset` LATERAL VIEW EXPLODE(usages) AS usedIn
-)
-SELECT id, min(usedIn) as firstUsage, max(usedIn) as lastUsage, collect_set(usedIn) as usages
-    FROM pivots
-    GROUP BY id; /*EOS*/
-CREATE TABLE `${pivot_history_db}`.`publication_new` STORED AS PARQUET AS
-WITH pivots (
-    SELECT property.value AS id, '${new_graph_date}' AS usedIn  FROM `${new_graph_db}`.`relation`
-    LEFT SEMI JOIN `${new_graph_db}`.`publication` ON relation.source = publication.id
-    LATERAL VIEW EXPLODE(properties) AS property WHERE relClass = 'isMergedIn' AND property.key = 'pivot'
-UNION
-    SELECT id, usedIn FROM `${pivot_history_db}`.`publication` LATERAL VIEW EXPLODE(usages) AS usedIn
-)
-SELECT id, min(usedIn) as firstUsage, max(usedIn) as lastUsage, collect_set(usedIn) as usages
-    FROM pivots
-    GROUP BY id; /*EOS*/
-CREATE TABLE `${pivot_history_db}`.`software_new` STORED AS PARQUET AS
-WITH pivots (
-    SELECT property.value AS id, '${new_graph_date}' AS usedIn  FROM `${new_graph_db}`.`relation`
-    LEFT SEMI JOIN `${new_graph_db}`.`software` ON relation.source = software.id
-    LATERAL VIEW EXPLODE(properties) AS property WHERE relClass = 'isMergedIn' AND property.key = 'pivot'
-UNION
-    SELECT id, usedIn FROM `${pivot_history_db}`.`software` LATERAL VIEW EXPLODE(usages) AS usedIn
-)
-SELECT id, min(usedIn) as firstUsage, max(usedIn) as lastUsage, collect_set(usedIn) as usages
-    FROM pivots
-    GROUP BY id; /*EOS*/
-CREATE TABLE `${pivot_history_db}`.`otherresearchproduct_new` STORED AS PARQUET AS
-WITH pivots (
-    SELECT property.value AS id, '${new_graph_date}' AS usedIn  FROM `${new_graph_db}`.`relation`
-     LEFT SEMI JOIN `${new_graph_db}`.`otherresearchproduct` ON relation.source = otherresearchproduct.id
-     LATERAL VIEW EXPLODE(properties) AS property WHERE relClass = 'isMergedIn' AND property.key = 'pivot'
-UNION
-    SELECT id, usedIn FROM `${pivot_history_db}`.`otherresearchproduct` LATERAL VIEW EXPLODE(usages) AS usedIn
-)
-SELECT id, min(usedIn) as firstUsage, max(usedIn) as lastUsage, collect_set(usedIn) as usages
-    FROM pivots
-    GROUP BY id; /*EOS*/
-
-
-DROP TABLE IF EXISTS `${pivot_history_db}`.`dataset_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`dataset` RENAME TO `${pivot_history_db}`.`dataset_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`dataset_new` RENAME TO `${pivot_history_db}`.`dataset`; /*EOS*/
-
-DROP TABLE IF EXISTS `${pivot_history_db}`.`publication_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`publication` RENAME TO `${pivot_history_db}`.`publication_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`publication_new` RENAME TO `${pivot_history_db}`.`publication`; /*EOS*/
-
-DROP TABLE IF EXISTS `${pivot_history_db}`.`software_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`software` RENAME TO `${pivot_history_db}`.`software_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`software_new` RENAME TO `${pivot_history_db}`.`software`; /*EOS*/
-
-DROP TABLE IF EXISTS `${pivot_history_db}`.`otherresearchproduct_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`otherresearchproduct` RENAME TO `${pivot_history_db}`.`otherresearchproduct_old`; /*EOS*/
-ALTER TABLE `${pivot_history_db}`.`otherresearchproduct_new` RENAME TO `${pivot_history_db}`.`otherresearchproduct`; /*EOS*/

dhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/pivothistory/oozie_app/workflow.xml

@@ -1,95 +0,0 @@
-<workflow-app name="Update pivot history" xmlns="uri:oozie:workflow:0.5">
-    <parameters>
-        <!-- properties used in SQL -->
-        <property>
-            <name>pivot_history_db</name>
-            <!-- <value>openaire_beta_pivots_test</value> -->
-            <description>Pivot history DB on hive</description>
-        </property>
-        <property>
-            <name>new_graph_db</name>
-            <!--<value>openaire_beta_20231208</value> -->
-            <description>New graph DB on hive</description>
-    </property>
-    <property>
-        <name>new_graph_date</name>
-        <!-- <value>20231208</value> -->
-        <description>Creation date of new graph db</description>
-    </property>
-
-    <!-- RunSQLSparkJob properties -->
-        <property>
-            <name>hiveMetastoreUris</name>
-            <description>hive server metastore URIs</description>
-        </property>
-        <property>
-            <name>sparkSqlWarehouseDir</name>
-        </property>
-        <!-- General oozie workflow properties -->
-        <property>
-            <name>sparkClusterOpts</name>
-            <value>--conf spark.network.timeout=600 --conf spark.extraListeners= --conf spark.sql.queryExecutionListeners= --conf spark.yarn.historyServer.address=http://iis-cdh5-test-m3.ocean.icm.edu.pl:18088 --conf spark.eventLog.dir=hdfs://nameservice1/user/spark/applicationHistory</value>
-            <description>spark cluster-wide options</description>
-        </property>
-        <property>
-            <name>sparkResourceOpts</name>
-            <value>--executor-memory=3G --conf spark.executor.memoryOverhead=3G --executor-cores=6 --driver-memory=8G --driver-cores=4</value>
-            <description>spark resource options</description>
-        </property>
-        <property>
-            <name>sparkApplicationOpts</name>
-            <value>--conf spark.sql.shuffle.partitions=3840</value>
-            <description>spark resource options</description>
-        </property>
-    </parameters>
-
-    <global>
-        <job-tracker>${jobTracker}</job-tracker>
-        <name-node>${nameNode}</name-node>
-        <configuration>
-            <property>
-                <name>mapreduce.job.queuename</name>
-                <value>${queueName}</value>
-            </property>
-            <property>
-                <name>oozie.launcher.mapred.job.queue.name</name>
-                <value>${oozieLauncherQueueName}</value>
-            </property>
-            <property>
-                <name>oozie.action.sharelib.for.spark</name>
-                <value>${oozieActionShareLibForSpark2}</value>
-            </property>
-        </configuration>
-    </global>
-
-    <start to="UpgradePivotHistory"/>
-
-    <kill name="Kill">
-        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
-    </kill>
-
-    <action name="UpgradePivotHistory">
-        <spark xmlns="uri:oozie:spark-action:0.2">
-            <master>yarn</master>
-            <mode>cluster</mode>
-            <name>Upgrade Pivot History</name>
-            <class>eu.dnetlib.dhp.oozie.RunSQLSparkJob</class>
-            <jar>dhp-dedup-openaire-${projectVersion}.jar</jar>
-            <spark-opts>
-                --conf spark.sql.warehouse.dir=${sparkSqlWarehouseDir}
-                ${sparkClusterOpts}
-                ${sparkResourceOpts}
-                ${sparkApplicationOpts}
-            </spark-opts>
-            <arg>--hiveMetastoreUris</arg><arg>${hiveMetastoreUris}</arg>
-            <arg>--sql</arg><arg>eu/dnetlib/dhp/oa/dedup/pivothistory/oozie_app/sql.sql</arg>
-            <arg>--pivot_history_db</arg><arg>${pivot_history_db}</arg>
-            <arg>--new_graph_db</arg><arg>${new_graph_db}</arg>
-            <arg>--new_graph_date</arg><arg>${new_graph_date}</arg>
-        </spark>
-        <ok to="End"/>
-        <error to="Kill"/>
-    </action>
-
-    <end name="End"/>
-</workflow-app>

dhp-workflows/dhp-doiboost/src/main/resources/eu/dnetlib/dhp/doiboost/crossref/irish_funder.json

@@ -73,6 +73,12 @@
     "name": "Irish Nephrology Society",
     "synonym": []
   },
+  {
+    "id": "100011062",
+    "uri": "http://dx.doi.org/10.13039/100011062",
+    "name": "Asian Spinal Cord Network",
+    "synonym": []
+  },
   {
     "id": "100011096",
     "uri": "http://dx.doi.org/10.13039/100011096",
@@ -217,6 +223,12 @@
     "name": "Global Brain Health Institute",
     "synonym": []
   },
+  {
+    "id": "100015776",
+    "uri": "http://dx.doi.org/10.13039/100015776",
+    "name": "Health and Social Care Board",
+    "synonym": []
+  },
   {
     "id": "100015992",
     "uri": "http://dx.doi.org/10.13039/100015992",
@@ -391,6 +403,18 @@
     "name": "Irish Hospice Foundation",
     "synonym": []
   },
+  {
+    "id": "501100001596",
+    "uri": "http://dx.doi.org/10.13039/501100001596",
+    "name": "Irish Research Council for Science, Engineering and Technology",
+    "synonym": []
+  },
+  {
+    "id": "501100001597",
+    "uri": "http://dx.doi.org/10.13039/501100001597",
+    "name": "Irish Research Council for the Humanities and Social Sciences",
+    "synonym": []
+  },
   {
     "id": "501100001598",
     "uri": "http://dx.doi.org/10.13039/501100001598",
@@ -491,7 +515,7 @@
     "id": "501100002081",
     "uri": "http://dx.doi.org/10.13039/501100002081",
     "name": "Irish Research Council",
-    "synonym": ["501100001596", "501100001597"]
+    "synonym": []
   },
   {
     "id": "501100002736",

dhp-workflows/dhp-doiboost/src/main/scala/eu/dnetlib/doiboost/crossref/Crossref2Oaf.scala

@@ -25,7 +25,7 @@ case class mappingAffiliation(name: String) {}
 
 case class mappingAuthor(
   given: Option[String],
-  family: Option[String],
+  family: String,
   sequence: Option[String],
   ORCID: Option[String],
   affiliation: Option[mappingAffiliation]
@@ -226,14 +226,14 @@ case object Crossref2Oaf {
 
     //Mapping Author
     val authorList: List[mappingAuthor] =
-      (json \ "author").extract[List[mappingAuthor]].filter(a => a.family.isDefined)
+      (json \ "author").extractOrElse[List[mappingAuthor]](List())
 
     val sorted_list = authorList.sortWith((a: mappingAuthor, b: mappingAuthor) =>
       a.sequence.isDefined && a.sequence.get.equalsIgnoreCase("first")
     )
 
     result.setAuthor(sorted_list.zipWithIndex.map { case (a, index) =>
-      generateAuhtor(a.given.orNull, a.family.get, a.ORCID.orNull, index)
+      generateAuhtor(a.given.orNull, a.family, a.ORCID.orNull, index)
     }.asJava)
 
     // Mapping instance
@@ -587,15 +587,7 @@ case object Crossref2Oaf {
                 "10.13039/501100000266" | "10.13039/501100006041" | "10.13039/501100000265" | "10.13039/501100000270" |
                 "10.13039/501100013589" | "10.13039/501100000271" =>
               generateSimpleRelationFromAward(funder, "ukri________", a => a)
-            //HFRI
-            case "10.13039/501100013209" =>
-              generateSimpleRelationFromAward(funder, "hfri________", a => a)
-              val targetId = getProjectId("hfri________", "1e5e62235d094afd01cd56e65112fc63")
-              queue += generateRelation(sourceId, targetId, ModelConstants.IS_PRODUCED_BY)
-              queue += generateRelation(targetId, sourceId, ModelConstants.PRODUCES)
-            //ERASMUS+
-            case "10.13039/501100010790" =>
-              generateSimpleRelationFromAward(funder, "erasmusplus_", a => a)
+
             case _ => logger.debug("no match for " + funder.DOI.get)
 
           }

dhp-workflows/dhp-doiboost/src/test/scala/eu/dnetlib/dhp/doiboost/crossref/CrossrefMappingTest.scala

@@ -22,13 +22,6 @@ class CrossrefMappingTest {
   val logger: Logger = LoggerFactory.getLogger(Crossref2Oaf.getClass)
   val mapper = new ObjectMapper()
 
-  @Test
-  def testMissingAuthorParser():Unit = {
-    val json: String = Source.fromInputStream(getClass.getResourceAsStream("/eu/dnetlib/doiboost/crossref/s41567-022-01757-y.json")).mkString
-    val result = Crossref2Oaf.convert(json)
-    result.filter(o => o.isInstanceOf[Publication]).map(p=> p.asInstanceOf[Publication]).foreach(p =>assertTrue(p.getAuthor.size()>0))
-  }
-
   @Test
   def testFunderRelationshipsMapping(): Unit = {
     val template = Source

dhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/MoveResult.java

@@ -30,7 +30,7 @@ public class MoveResult implements Serializable {
 	public static void main(String[] args) throws Exception {
 		String jsonConfiguration = IOUtils
 			.toString(
-				MoveResult.class
+				SparkResultToCommunityFromOrganizationJob.class
 					.getResourceAsStream(
 						"/eu/dnetlib/dhp/wf/subworkflows/input_moveresult_parameters.json"));
 

dhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/bulktag/community/Constraints.java

@@ -53,8 +53,6 @@ public class Constraints implements Serializable {
 
 		for (Constraint sc : constraint) {
 			boolean verified = false;
-			if(!param.containsKey(sc.getField()))
-				return false;
 			for (String value : param.get(sc.getField())) {
 				if (sc.verifyCriteria(value.trim())) {
 					verified = true;

dhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/SparkCountryPropagationJob.java

@@ -14,7 +14,6 @@ import org.apache.spark.SparkConf;
 import org.apache.spark.api.java.function.MapFunction;
 import org.apache.spark.sql.Dataset;
 import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.Row;
 import org.apache.spark.sql.SaveMode;
 import org.apache.spark.sql.SparkSession;
 import org.slf4j.Logger;
@@ -85,26 +84,19 @@ public class SparkCountryPropagationJob {
 		Dataset<R> res = readPath(spark, sourcePath, resultClazz);
 
 		log.info("Reading prepared info: {}", preparedInfoPath);
-		final Dataset<Row> preparedInfoRaw = spark
+		Dataset<ResultCountrySet> prepared = spark
 			.read()
-			.json(preparedInfoPath);
+			.json(preparedInfoPath)
+			.as(Encoders.bean(ResultCountrySet.class));
+
+		res
+			.joinWith(prepared, res.col("id").equalTo(prepared.col("resultId")), "left_outer")
+			.map(getCountryMergeFn(), Encoders.bean(resultClazz))
+			.write()
+			.option("compression", "gzip")
+			.mode(SaveMode.Overwrite)
+			.json(outputPath);
 
-		if (!preparedInfoRaw.isEmpty()) {
-			final Dataset<ResultCountrySet> prepared = preparedInfoRaw.as(Encoders.bean(ResultCountrySet.class));
-			res
-				.joinWith(prepared, res.col("id").equalTo(prepared.col("resultId")), "left_outer")
-				.map(getCountryMergeFn(), Encoders.bean(resultClazz))
-				.write()
-				.option("compression", "gzip")
-				.mode(SaveMode.Overwrite)
-				.json(outputPath);
-		} else {
-			res
-				.write()
-				.option("compression", "gzip")
-				.mode(SaveMode.Overwrite)
-				.json(outputPath);
-		}
 	}
 
 	private static <R extends Result> MapFunction<Tuple2<R, ResultCountrySet>, R> getCountryMergeFn() {

dhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/oozie_app/workflow.xml

@@ -233,7 +233,7 @@
                 --conf spark.speculation=false
                 --conf spark.hadoop.mapreduce.map.speculative=false
                 --conf spark.hadoop.mapreduce.reduce.speculative=false
-                --conf spark.sql.shuffle.partitions=7680
+                --conf spark.sql.shuffle.partitions=3840
             </spark-opts>
             <arg>--sourcePath</arg><arg>${sourcePath}/publication</arg>
             <arg>--preparedInfoPath</arg><arg>${workingDir}/country/publication</arg>

dhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/oozie_app/workflow.xml

@@ -114,7 +114,7 @@
             <arg>--sourcePath</arg><arg>${sourcePath}</arg>
             <arg>--hive_metastore_uris</arg><arg>${hive_metastore_uris}</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.Publication</arg>
-            <arg>--outputPath</arg><arg>${workingDir}/orcid/targetOrcidAssoc</arg>
+            <arg>--outputPath</arg><arg>${workingDir}/orcid/preparedInfo/targetOrcidAssoc</arg>
             <arg>--allowedsemrels</arg><arg>${allowedsemrels}</arg>
         </spark>
         <ok to="wait"/>
@@ -142,7 +142,7 @@
             <arg>--sourcePath</arg><arg>${sourcePath}</arg>
             <arg>--hive_metastore_uris</arg><arg>${hive_metastore_uris}</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.Dataset</arg>
-            <arg>--outputPath</arg><arg>${workingDir}/orcid/targetOrcidAssoc</arg>
+            <arg>--outputPath</arg><arg>${workingDir}/orcid/preparedInfo/targetOrcidAssoc</arg>
             <arg>--allowedsemrels</arg><arg>${allowedsemrels}</arg>
         </spark>
         <ok to="wait"/>
@@ -170,7 +170,7 @@
             <arg>--sourcePath</arg><arg>${sourcePath}</arg>
             <arg>--hive_metastore_uris</arg><arg>${hive_metastore_uris}</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.OtherResearchProduct</arg>
-            <arg>--outputPath</arg><arg>${workingDir}/orcid/targetOrcidAssoc</arg>
+            <arg>--outputPath</arg><arg>${workingDir}/orcid/preparedInfo/targetOrcidAssoc</arg>
             <arg>--allowedsemrels</arg><arg>${allowedsemrels}</arg>
         </spark>
         <ok to="wait"/>
@@ -198,7 +198,7 @@
             <arg>--sourcePath</arg><arg>${sourcePath}</arg>
             <arg>--hive_metastore_uris</arg><arg>${hive_metastore_uris}</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.Software</arg>
-            <arg>--outputPath</arg><arg>${workingDir}/orcid/targetOrcidAssoc</arg>
+            <arg>--outputPath</arg><arg>${workingDir}/orcid/preparedInfo/targetOrcidAssoc</arg>
             <arg>--allowedsemrels</arg><arg>${allowedsemrels}</arg>
         </spark>
         <ok to="wait"/>
@@ -225,8 +225,8 @@
                 --conf spark.dynamicAllocation.enabled=true
                 --conf spark.dynamicAllocation.maxExecutors=${spark2MaxExecutors}
             </spark-opts>
-            <arg>--sourcePath</arg><arg>${workingDir}/orcid/targetOrcidAssoc</arg>
-            <arg>--outputPath</arg><arg>${workingDir}/orcid/mergedOrcidAssoc</arg>
+            <arg>--sourcePath</arg><arg>${workingDir}/orcid/orcidprop</arg>
+            <arg>--outputPath</arg><arg>${workingDir}/orcid/orcidprop/mergedOrcidAssoc</arg>
         </spark>
         <ok to="fork-join-exec-propagation"/>
         <error to="Kill"/>
@@ -247,10 +247,9 @@
             <class>eu.dnetlib.dhp.orcidtoresultfromsemrel.SparkOrcidToResultFromSemRelJob</class>
             <jar>dhp-enrichment-${projectVersion}.jar</jar>
             <spark-opts>
-                --executor-cores=4
-                --executor-memory=4G
+                --executor-cores=${sparkExecutorCores}
+                --executor-memory=${sparkExecutorMemory}
                 --driver-memory=${sparkDriverMemory}
-                --conf spark.executor.memoryOverhead=5G
                 --conf spark.extraListeners=${spark2ExtraListeners}
                 --conf spark.sql.queryExecutionListeners=${spark2SqlQueryExecutionListeners}
                 --conf spark.yarn.historyServer.address=${spark2YarnHistoryServerAddress}
@@ -260,9 +259,9 @@
                 --conf spark.speculation=false
                 --conf spark.hadoop.mapreduce.map.speculative=false
                 --conf spark.hadoop.mapreduce.reduce.speculative=false
-                --conf spark.sql.shuffle.partitions=15000
+                --conf spark.sql.shuffle.partitions=3840
             </spark-opts>
-            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/mergedOrcidAssoc</arg>
+            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/orcidprop/mergedOrcidAssoc</arg>
             <arg>--sourcePath</arg><arg>${sourcePath}/publication</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.Publication</arg>
             <arg>--outputPath</arg><arg>${outputPath}/publication</arg>
@@ -292,7 +291,7 @@
                 --conf spark.hadoop.mapreduce.map.speculative=false
                 --conf spark.hadoop.mapreduce.reduce.speculative=false
             </spark-opts>
-            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/mergedOrcidAssoc</arg>
+            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/orcidprop/mergedOrcidAssoc</arg>
             <arg>--sourcePath</arg><arg>${sourcePath}/dataset</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.Dataset</arg>
             <arg>--outputPath</arg><arg>${outputPath}/dataset</arg>
@@ -322,7 +321,7 @@
                 --conf spark.hadoop.mapreduce.map.speculative=false
                 --conf spark.hadoop.mapreduce.reduce.speculative=false
             </spark-opts>
-            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/mergedOrcidAssoc</arg>
+            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/orcidprop/mergedOrcidAssoc</arg>
             <arg>--sourcePath</arg><arg>${sourcePath}/otherresearchproduct</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.OtherResearchProduct</arg>
             <arg>--outputPath</arg><arg>${outputPath}/otherresearchproduct</arg>
@@ -352,7 +351,7 @@
                 --conf spark.hadoop.mapreduce.map.speculative=false
                 --conf spark.hadoop.mapreduce.reduce.speculative=false
             </spark-opts>
-            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/mergedOrcidAssoc</arg>
+            <arg>--possibleUpdatesPath</arg><arg>${workingDir}/orcid/orcidprop/mergedOrcidAssoc</arg>
             <arg>--sourcePath</arg><arg>${sourcePath}/software</arg>
             <arg>--resultTableName</arg><arg>eu.dnetlib.dhp.schema.oaf.Software</arg>
             <arg>--outputPath</arg><arg>${outputPath}/software</arg>

dhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/projecttoresult/oozie_app/workflow.xml

@@ -30,6 +30,7 @@
     <decision name="startFrom">
         <switch>
             <case to="prepare_project_results_association">${wf:conf('startFrom') eq 'undelete'}</case>
+
             <default to="reset_outputpath"/>
         </switch>
     </decision>
@@ -100,5 +101,4 @@
         <error to="Kill"/>
     </action>
 
-    <end name="End"/>
 </workflow-app>

dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/raw/MigrateDbEntitiesApplication.java

@@ -317,7 +317,7 @@ public class MigrateDbEntitiesApplication extends AbstractMigrationApplication i
 					listKeyValues(
 						createOpenaireId(10, rs.getString("collectedfromid"), true),
 						rs.getString("collectedfromname")));
-			p.setPid(prepareListOfStructProps(rs.getArray("pid"), info));
+			p.setPid(new ArrayList<>());
 			p.setDateofcollection(asString(rs.getDate("dateofcollection")));
 			p.setDateoftransformation(asString(rs.getDate("dateoftransformation")));
 			p.setExtraInfo(new ArrayList<>()); // Values not present in the DB

dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/raw/OdfToOafMapper.java

@@ -238,23 +238,11 @@ public class OdfToOafMapper extends AbstractMdRecordToOafMapper {
 				(Element) doc
 					.selectSingleNode(
 						"//*[local-name()='metadata']/*[local-name() = 'resource']/*[local-name() = 'resourceType']"))
-			.map(e -> {
-				final String resourceTypeURI = Optional
-					.ofNullable(e.attributeValue("uri"))
-					.filter(StringUtils::isNotBlank)
-					.orElse(null);
-				final String resourceTypeAnyURI = Optional
-					.ofNullable(e.attributeValue("anyURI"))
-					.filter(StringUtils::isNotBlank)
-					.orElse(null);
-				final String resourceTypeTxt = Optional
-					.ofNullable(e.getText())
-					.filter(StringUtils::isNotBlank)
-					.orElse(null);
-				final String resourceTypeGeneral = Optional
-					.ofNullable(e.attributeValue("resourceTypeGeneral"))
-					.filter(StringUtils::isNotBlank)
-					.orElse(null);
+			.map(element -> {
+				final String resourceTypeURI = element.attributeValue("uri");
+				final String resourceTypeAnyURI = element.attributeValue("anyURI");
+				final String resourceTypeTxt = element.getText();
+				final String resourceTypeGeneral = element.attributeValue("resourceTypeGeneral");
 
 				return ObjectUtils
 					.firstNonNull(resourceTypeURI, resourceTypeAnyURI, resourceTypeTxt, resourceTypeGeneral);

dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/enrich/orcid/oozie_app/workflow.xml

@@ -43,17 +43,6 @@
             <arg>--graphPath</arg><arg>${graphPath}</arg>
             <arg>--master</arg><arg>yarn</arg>
         </spark>
-        <ok to="reset_outputpath"/>
-        <error to="Kill"/>
-    </action>
-
-    <action name="reset_outputpath">
-        <fs>
-            <delete path="${targetPath}/datasource"/>
-            <delete path="${targetPath}/organization"/>
-            <delete path="${targetPath}/project"/>
-            <delete path="${targetPath}/relation"/>
-        </fs>
         <ok to="copy_datasource"/>
         <error to="Kill"/>
     </action>

dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/sql/queryProjects.sql

@@ -33,7 +33,7 @@ SELECT
                 dc.officialname                                                                                            AS collectedfromname,
                 p.contracttype || '@@@' || p.contracttypescheme                                                            AS contracttype,
                 p.provenanceactionclass || '@@@' || p.provenanceactionscheme                                             AS provenanceaction,
-                array_remove(array_agg(DISTINCT i.pid || '###' || i.issuertype || '@@@' || i.issuertype), NULL)            AS pid,
+                array_agg(DISTINCT i.pid || '###' || i.issuertype)                                                                  AS pid,
                 array_agg(DISTINCT s.name || '###' || s.semanticclass || '@@@' || s.semanticscheme)          AS subjects,
                 array_agg(DISTINCT fp.path)                                                                                         AS fundingtree
 

dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/sql/queryProjects_production.sql

@@ -33,7 +33,7 @@ SELECT
                 dc.officialname                                                                                            AS collectedfromname,
                 p.contracttypeclass || '@@@' || p.contracttypescheme                                                       AS contracttype,
                 p.provenanceactionclass || '@@@' || p.provenanceactionscheme                                               AS provenanceaction,
-                array_remove(array_agg(DISTINCT i.pid || '###' || i.issuertype || '@@@' || i.issuertype), NULL)            AS pid,
+                array_agg(DISTINCT i.pid || '###' || i.issuertype)                                                         AS pid,
                 array_agg(DISTINCT s.name || '###' || s.semanticclass || '@@@' || s.semanticscheme) AS subjects,
                 array_agg(DISTINCT fp.path)                                                                                AS fundingtree
         FROM projects p

dhp-workflows/dhp-graph-mapper/src/main/scala/eu/dnetlib/dhp/oa/graph/raw/CopyHdfsOafSparkApplication.scala

@@ -93,8 +93,8 @@ object CopyHdfsOafSparkApplication {
       hasSource != null && hasTarget != null
     } else {
       val hasId = (json \ "id").extractOrElse[String](null)
-      val resultType = (json \ "resulttype" \ "classid").extractOrElse[String]("")
-      hasId != null && oafType.startsWith(resultType)
+      val resultType = (json \ "resulttype" \ "classid").extractOrElse[String](null)
+      hasId != null && oafType.equalsIgnoreCase(resultType)
     }
 
   }