Merge branch 'beta' of https://code-repo.d4science.org/andreas.czerniak/BrBETA_dnet-hadoop into beta

.gitignore

@@ -26,4 +26,3 @@ spark-warehouse
 /**/*.log
 /**/.factorypath
 /**/.scalafmt.conf
-/.java-version

.scalafmt.conf

@@ -1,21 +0,0 @@
-style = defaultWithAlign
-
-align.openParenCallSite = false
-align.openParenDefnSite = false
-align.tokens = [{code = "->"}, {code = "<-"}, {code = "=>", owner = "Case"}]
-continuationIndent.callSite = 2
-continuationIndent.defnSite = 2
-danglingParentheses = true
-indentOperator = spray
-maxColumn = 120
-newlines.alwaysBeforeTopLevelStatements = true
-project.excludeFilters = [".*\\.sbt"]
-rewrite.rules = [AvoidInfix]
-rewrite.rules = [ExpandImportSelectors]
-rewrite.rules = [RedundantBraces]
-rewrite.rules = [RedundantParens]
-rewrite.rules = [SortImports]
-rewrite.rules = [SortModifiers]
-rewrite.rules = [PreferCurlyFors]
-spaces.inImportCurlyBraces = false
-unindentTopLevelOperators = true

CODE_OF_CONDUCT.md

@@ -1,43 +0,0 @@
-# Contributor Code of Conduct
-
-Openness, transparency and our community-driven participatory approach guide us in our day-to-day interactions and decision-making. Our open source projects are no exception. Trust, respect, collaboration and transparency are core values we believe should live and breathe within our projects. Our community welcomes participants from around the world with different experiences, unique perspectives, and great ideas to share.
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment include:
-
-- Using welcoming and inclusive language
-- Being respectful of differing viewpoints and experiences
-- Gracefully accepting constructive criticism
-- Attempting collaboration before conflict
-- Focusing on what is best for the community
-- Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-- Violence, threats of violence, or inciting others to commit self-harm
-- The use of sexualized language or imagery and unwelcome sexual attention or advances
-- Trolling, intentionally spreading misinformation, insulting/derogatory comments, and personal or political attacks
-- Public or private harassment
-- Publishing others' private information, such as a physical or electronic address, without explicit permission
-- Abuse of the reporting process to intentionally harass or exclude others
-- Advocating for, or encouraging, any of the above behavior
-- Other conduct which could reasonably be considered inappropriate in a professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), [version 1.4](https://www.contributor-covenant.org/version/1/4/code-of-conduct.html).

CONTRIBUTING.md

@@ -1,10 +0,0 @@
-# Contributing to D-Net Hadoop
-
-:+1::tada: First off, thanks for taking the time to contribute! :tada::+1:
-
-This project and everyone participating in it is governed by our [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to [dnet-team@isti.cnr.it](mailto:dnet-team@isti.cnr.it).
-
-The following is a set of guidelines for contributing to this project and its packages. These are mostly guidelines, not rules, which applies to this project as a while, including all its sub-modules. 
-Use your best judgment, and feel free to propose changes to this document in a pull request.
-
-All contributions are welcome, all contributions will be considered to be contributed under the [project license](LICENSE.md).

README.md

@@ -1,133 +1,2 @@
 # dnet-hadoop
-
-Dnet-hadoop is the project that defined all the [OOZIE workflows](https://oozie.apache.org/) for the OpenAIRE Graph construction, processing, provisioning.
-
-This project adheres to the Contributor Covenant [code of conduct](CODE_OF_CONDUCT.md).
-By participating, you are expected to uphold this code. Please report unacceptable behavior to [dnet-team@isti.cnr.it](mailto:dnet-team@isti.cnr.it).
-
-This project is licensed under the [AGPL v3 or later version](#LICENSE.md).
-
-How to build, package and run oozie workflows
-====================
-
-Oozie-installer is a utility allowing building, uploading and running oozie workflows. In practice, it creates a `*.tar.gz` 
-package that contains resources that define a workflow and some helper scripts.
-
-This module is automatically executed when running:
-
-`mvn package -Poozie-package -Dworkflow.source.dir=classpath/to/parent/directory/of/oozie_app`
-
-on module having set:
-
-```
-<parent>
-    <groupId>eu.dnetlib.dhp</groupId>
-    <artifactId>dhp-workflows</artifactId>
-</parent>
-```
-
-in `pom.xml` file. `oozie-package` profile initializes oozie workflow packaging, `workflow.source.dir` property points to 
-a workflow (notice: this is not a relative path but a classpath to directory usually holding `oozie_app` subdirectory).
-
-The outcome of this packaging is `oozie-package.tar.gz` file containing inside all the resources required to run Oozie workflow:
-
-- jar packages
-- workflow definitions
-- job properties
-- maintenance scripts
-
-Required properties
-====================
-
-In order to include proper workflow within package, `workflow.source.dir` property has to be set. It could be provided 
-by setting `-Dworkflow.source.dir=some/job/dir` maven parameter.
-
-In oder to define full set of cluster environment properties one should create `~/.dhp/application.properties` file with 
-the following properties:
-
-- `dhp.hadoop.frontend.user.name` - your user name on hadoop cluster and frontend machine
-- `dhp.hadoop.frontend.host.name` - frontend host name
-- `dhp.hadoop.frontend.temp.dir` - frontend directory for temporary files
-- `dhp.hadoop.frontend.port.ssh` - frontend machine ssh port
-- `oozieServiceLoc` - oozie service location required by run_workflow.sh script executing oozie job
-- `nameNode` - name node address
-- `jobTracker` - job tracker address
-- `oozie.execution.log.file.location` - location of file that will be created when executing oozie job, it contains output 
-produced by `run_workflow.sh` script (needed to obtain oozie job id)
-- `maven.executable` - mvn command location, requires parameterization due to a different setup of CI cluster
-- `sparkDriverMemory` - amount of memory assigned to spark jobs driver
-- `sparkExecutorMemory` - amount of memory assigned to spark jobs executors
-- `sparkExecutorCores` - number of cores assigned to spark jobs executors
-
-All values will be overriden with the ones from `job.properties` and eventually `job-override.properties` stored in module's 
-main folder.
-
-When overriding properties from `job.properties`, `job-override.properties` file can be created in main module directory 
-(the one containing `pom.xml` file) and define all new properties which will override existing properties. 
-One can provide those properties one by one as command line `-D` arguments.
-
-Properties overriding order is the following:
-
-1. `pom.xml` defined properties (located in the project root dir)
-2. `~/.dhp/application.properties` defined properties
-3. `${workflow.source.dir}/job.properties`
-4. `job-override.properties` (located in the project root dir)
-5. `maven -Dparam=value`
-
-where the maven `-Dparam` property is overriding all the other ones.
-
-Workflow definition requirements
-====================
-
-`workflow.source.dir` property should point to the following directory structure:
-
-	[${workflow.source.dir}]
-		|
-		|-job.properties (optional)
-		|
-		\-[oozie_app]
-			|
-			\-workflow.xml
-
-This property can be set using maven `-D` switch.
-
-`[oozie_app]` is the default directory name however it can be set to any value as soon as `oozieAppDir` property is 
-provided with directory name as value.
-
-Sub-workflows are supported as well and sub-workflow directories should be nested within `[oozie_app]` directory.
-
-Creating oozie installer step-by-step
-=====================================
-
-Automated oozie-installer steps are the following:
-
-1. creating jar packages:  `*.jar` and `*tests.jar` along with copying all dependencies in `target/dependencies`
-2. reading properties from maven, `~/.dhp/application.properties`, `job.properties`, `job-override.properties`
-3. invoking priming mechanism linking resources from import.txt file (currently resolving subworkflow resources)
-4. assembling shell scripts for preparing Hadoop filesystem, uploading Oozie application and starting workflow
-5. copying whole `${workflow.source.dir}` content to `target/${oozie.package.file.name}`
-6. generating updated `job.properties` file in `target/${oozie.package.file.name}` based on maven, 
-`~/.dhp/application.properties`, `job.properties` and `job-override.properties`
-7. creating `lib` directory (or multiple directories for sub-workflows for each nested directory) and copying jar packages 
-created at step (1) to each one of them
-8. bundling whole `${oozie.package.file.name}` directory into single tar.gz package
-
-Uploading oozie package and running workflow on cluster
-=======================================================
-
-In order to simplify deployment and execution process two dedicated profiles were introduced:
-
-- `deploy`
-- `run`
-
-to be used along with `oozie-package` profile e.g. by providing `-Poozie-package,deploy,run` maven parameters.
-
-The `deploy` profile supplements packaging process with:
-1) uploading oozie-package via scp to `/home/${user.name}/oozie-packages` directory on `${dhp.hadoop.frontend.host.name}` machine
-2) extracting uploaded package
-3) uploading oozie content to hadoop cluster HDFS location defined in `oozie.wf.application.path` property (generated dynamically by maven build process, based on `${dhp.hadoop.frontend.user.name}` and `workflow.source.dir` properties)
-
-The `run` profile introduces:
-1) executing oozie application uploaded to HDFS cluster using `deploy` command. Triggers `run_workflow.sh` script providing runtime properties defined in `job.properties` file.
-
-Notice: ssh access to frontend machine has to be configured on system level and it is preferable to set key-based authentication in order to simplify remote operations.
+Dnet-hadoop is the project that defined all the OOZIE workflows for the OpenAIRE Graph construction, processing, provisioning.

dhp-build/dhp-build-assembly-resources/pom.xml

@@ -6,7 +6,7 @@
     <parent>
         <groupId>eu.dnetlib.dhp</groupId>
         <artifactId>dhp-build</artifactId>
-        <version>1.2.5-SNAPSHOT</version>
+        <version>1.2.4-SNAPSHOT</version>
     </parent>
 
     <artifactId>dhp-build-assembly-resources</artifactId>

dhp-build/dhp-build-properties-maven-plugin/pom.xml

@@ -6,7 +6,7 @@
     <parent>
         <groupId>eu.dnetlib.dhp</groupId>
         <artifactId>dhp-build</artifactId>
-        <version>1.2.5-SNAPSHOT</version>
+        <version>1.2.4-SNAPSHOT</version>
     </parent>
 
     <artifactId>dhp-build-properties-maven-plugin</artifactId>

dhp-build/dhp-code-style/pom.xml

@@ -5,7 +5,7 @@
 
     <groupId>eu.dnetlib.dhp</groupId>
     <artifactId>dhp-code-style</artifactId>
-    <version>1.2.5-SNAPSHOT</version>
+    <version>1.2.4-SNAPSHOT</version>
 
     <packaging>jar</packaging>
 
@@ -47,16 +47,12 @@
                     <groupId>org.apache.maven.plugins</groupId>
                     <artifactId>maven-site-plugin</artifactId>
                     <version>3.9.1</version>
-                    <configuration>
-                        <skip>true</skip>
-                    </configuration>
                 </plugin>
             </plugins>
         </pluginManagement>
     </build>
 
     <properties>
-
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <dhp.site.stage.path>sftp://dnet-hadoop@static-web.d4science.org/dnet-hadoop</dhp.site.stage.path>
     </properties>

dhp-build/pom.xml

@@ -4,7 +4,7 @@
 	<parent>
 		<groupId>eu.dnetlib.dhp</groupId>
 		<artifactId>dhp</artifactId>
-		<version>1.2.5-SNAPSHOT</version>
+		<version>1.2.4-SNAPSHOT</version>
 	</parent>
 	<artifactId>dhp-build</artifactId>
 	<packaging>pom</packaging>

dhp-common/pom.xml

@@ -5,7 +5,7 @@
 	<parent>
 		<groupId>eu.dnetlib.dhp</groupId>
 		<artifactId>dhp</artifactId>
-		<version>1.2.5-SNAPSHOT</version>
+		<version>1.2.4-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 
 	</parent>
@@ -52,8 +52,6 @@
 					</execution>
 				</executions>
 				<configuration>
-					<failOnMultipleScalaVersions>true</failOnMultipleScalaVersions>
-					<scalaCompatVersion>${scala.binary.version}</scalaCompatVersion>
 					<scalaVersion>${scala.version}</scalaVersion>
 				</configuration>
 			</plugin>
@@ -62,11 +60,6 @@
 	</build>
 
 	<dependencies>
-		<dependency>
-			<groupId>eu.dnetlib.dhp</groupId>
-			<artifactId>dhp-pace-core</artifactId>
-			<version>${project.version}</version>
-		</dependency>
 
 		<dependency>
 			<groupId>org.apache.hadoop</groupId>
@@ -83,11 +76,11 @@
 
 		<dependency>
 			<groupId>org.apache.spark</groupId>
-			<artifactId>spark-core_${scala.binary.version}</artifactId>
+			<artifactId>spark-core_2.11</artifactId>
 		</dependency>
 		<dependency>
 			<groupId>org.apache.spark</groupId>
-			<artifactId>spark-sql_${scala.binary.version}</artifactId>
+			<artifactId>spark-sql_2.11</artifactId>
 		</dependency>
 
 		<dependency>
@@ -149,6 +142,11 @@
 			<artifactId>okhttp</artifactId>
 		</dependency>
 
+		<dependency>
+			<groupId>eu.dnetlib</groupId>
+			<artifactId>dnet-pace-core</artifactId>
+		</dependency>
+
 		<dependency>
 			<groupId>org.apache.httpcomponents</groupId>
 			<artifactId>httpclient</artifactId>
@@ -161,7 +159,7 @@
 
 		<dependency>
 			<groupId>eu.dnetlib.dhp</groupId>
-			<artifactId>${dhp-schemas.artifact}</artifactId>
+			<artifactId>dhp-schemas</artifactId>
 		</dependency>
 
 		<dependency>

dhp-common/src/main/java/eu/dnetlib/dhp/common/Constants.java

@@ -10,12 +10,6 @@ public class Constants {
 	public static final Map<String, String> accessRightsCoarMap = Maps.newHashMap();
 	public static final Map<String, String> coarCodeLabelMap = Maps.newHashMap();
 
-	public static final String ROR_NS_PREFIX = "ror_________";
-
-	public static final String ROR_OPENAIRE_ID = "10|openaire____::993a7ae7a863813cf95028b50708e222";
-
-	public static final String ROR_DATASOURCE_NAME = "Research Organization Registry (ROR)";
-
 	public static String COAR_ACCESS_RIGHT_SCHEMA = "http://vocabularies.coar-repositories.org/documentation/access_rights/";
 
 	private Constants() {
@@ -51,7 +45,6 @@ public class Constants {
 	public static final String RETRY_DELAY = "retryDelay";
 	public static final String CONNECT_TIMEOUT = "connectTimeOut";
 	public static final String READ_TIMEOUT = "readTimeOut";
-	public static final String REQUEST_METHOD = "requestMethod";
 	public static final String FROM_DATE_OVERRIDE = "fromDateOverride";
 	public static final String UNTIL_DATE_OVERRIDE = "untilDateOverride";
 

dhp-common/src/main/java/eu/dnetlib/dhp/common/MDStoreInfo.java

@@ -1,100 +0,0 @@
-
-package eu.dnetlib.dhp.common;
-
-/**
- * This utility represent the Metadata Store information
- * needed during the migration from mongo to HDFS to store
-  */
-public class MDStoreInfo {
-	private String mdstore;
-	private String currentId;
-	private Long latestTimestamp;
-
-	/**
-	 * Instantiates a new Md store info.
-	 */
-	public MDStoreInfo() {
-	}
-
-	/**
-	 * Instantiates a new Md store info.
-	 *
-	 * @param mdstore         the mdstore
-	 * @param currentId       the current id
-	 * @param latestTimestamp the latest timestamp
-	 */
-	public MDStoreInfo(String mdstore, String currentId, Long latestTimestamp) {
-		this.mdstore = mdstore;
-		this.currentId = currentId;
-		this.latestTimestamp = latestTimestamp;
-	}
-
-	/**
-	 * Gets mdstore.
-	 *
-	 * @return the mdstore
-	 */
-	public String getMdstore() {
-		return mdstore;
-	}
-
-	/**
-	 * Sets mdstore.
-	 *
-	 * @param mdstore the mdstore
-	 * @return the mdstore
-	 */
-	public MDStoreInfo setMdstore(String mdstore) {
-		this.mdstore = mdstore;
-		return this;
-	}
-
-	/**
-	 * Gets current id.
-	 *
-	 * @return the current id
-	 */
-	public String getCurrentId() {
-		return currentId;
-	}
-
-	/**
-	 * Sets current id.
-	 *
-	 * @param currentId the current id
-	 * @return the current id
-	 */
-	public MDStoreInfo setCurrentId(String currentId) {
-		this.currentId = currentId;
-		return this;
-	}
-
-	/**
-	 * Gets latest timestamp.
-	 *
-	 * @return the latest timestamp
-	 */
-	public Long getLatestTimestamp() {
-		return latestTimestamp;
-	}
-
-	/**
-	 * Sets latest timestamp.
-	 *
-	 * @param latestTimestamp the latest timestamp
-	 * @return the latest timestamp
-	 */
-	public MDStoreInfo setLatestTimestamp(Long latestTimestamp) {
-		this.latestTimestamp = latestTimestamp;
-		return this;
-	}
-
-	@Override
-	public String toString() {
-		return "MDStoreInfo{" +
-			"mdstore='" + mdstore + '\'' +
-			", currentId='" + currentId + '\'' +
-			", latestTimestamp=" + latestTimestamp +
-			'}';
-	}
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/MakeTarArchive.java

@@ -5,71 +5,13 @@ import java.io.BufferedInputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.Serializable;
-import java.util.Optional;
 
 import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
 import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
-import org.apache.commons.io.IOUtils;
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.*;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import eu.dnetlib.dhp.application.ArgumentApplicationParser;
 
 public class MakeTarArchive implements Serializable {
 
-	private static final Logger log = LoggerFactory.getLogger(MakeTarArchive.class);
-
-	public static void main(String[] args) throws Exception {
-		String jsonConfiguration = IOUtils
-			.toString(
-				MakeTarArchive.class
-					.getResourceAsStream(
-						"/eu/dnetlib/dhp/common/input_maketar_parameters.json"));
-
-		final ArgumentApplicationParser parser = new ArgumentApplicationParser(jsonConfiguration);
-		parser.parseArgument(args);
-
-		final String outputPath = parser.get("hdfsPath");
-		log.info("hdfsPath: {}", outputPath);
-
-		final String hdfsNameNode = parser.get("nameNode");
-		log.info("nameNode: {}", hdfsNameNode);
-
-		final String inputPath = parser.get("sourcePath");
-		log.info("input path : {}", inputPath);
-
-		final int gBperSplit = Optional
-			.ofNullable(parser.get("splitSize"))
-			.map(Integer::valueOf)
-			.orElse(10);
-
-		Configuration conf = new Configuration();
-		conf.set("fs.defaultFS", hdfsNameNode);
-
-		FileSystem fileSystem = FileSystem.get(conf);
-
-		makeTArArchive(fileSystem, inputPath, outputPath, gBperSplit);
-
-	}
-
-	public static void makeTArArchive(FileSystem fileSystem, String inputPath, String outputPath, int gBperSplit)
-		throws IOException {
-
-		RemoteIterator<LocatedFileStatus> dirIterator = fileSystem.listLocatedStatus(new Path(inputPath));
-
-		while (dirIterator.hasNext()) {
-			LocatedFileStatus fileStatus = dirIterator.next();
-
-			Path p = fileStatus.getPath();
-			String pathString = p.toString();
-			String entity = pathString.substring(pathString.lastIndexOf("/") + 1);
-
-			MakeTarArchive.tarMaxSize(fileSystem, pathString, outputPath + "/" + entity, entity, gBperSplit);
-		}
-	}
-
 	private static TarArchiveOutputStream getTar(FileSystem fileSystem, String outputPath) throws IOException {
 		Path hdfsWritePath = new Path(outputPath);
 		if (fileSystem.exists(hdfsWritePath)) {
@@ -79,7 +21,7 @@ public class MakeTarArchive implements Serializable {
 		return new TarArchiveOutputStream(fileSystem.create(hdfsWritePath).getWrappedStream());
 	}
 
-	private static void write(FileSystem fileSystem, String inputPath, String outputPath, String dirName)
+	private static void write(FileSystem fileSystem, String inputPath, String outputPath, String dir_name)
 		throws IOException {
 
 		Path hdfsWritePath = new Path(outputPath);
@@ -95,7 +37,7 @@ public class MakeTarArchive implements Serializable {
 					new Path(inputPath), true);
 
 			while (iterator.hasNext()) {
-				writeCurrentFile(fileSystem, dirName, iterator, ar, 0);
+				writeCurrentFile(fileSystem, dir_name, iterator, ar, 0);
 			}
 
 		}
@@ -117,30 +59,32 @@ public class MakeTarArchive implements Serializable {
 					new Path(inputPath), true);
 			boolean next = fileStatusListIterator.hasNext();
 			while (next) {
-				try (TarArchiveOutputStream ar = getTar(fileSystem, outputPath + "_" + (partNum + 1) + ".tar")) {
+				TarArchiveOutputStream ar = getTar(fileSystem, outputPath + "_" + (partNum + 1) + ".tar");
 
-					long currentSize = 0;
-					while (next && currentSize < bytesPerSplit) {
-						currentSize = writeCurrentFile(fileSystem, dir_name, fileStatusListIterator, ar, currentSize);
-						next = fileStatusListIterator.hasNext();
+				long current_size = 0;
+				while (next && current_size < bytesPerSplit) {
+					current_size = writeCurrentFile(fileSystem, dir_name, fileStatusListIterator, ar, current_size);
+					next = fileStatusListIterator.hasNext();
 
-					}
-
-					partNum += 1;
 				}
+
+				partNum += 1;
+				ar.close();
 			}
+
 		}
+
 	}
 
-	private static long writeCurrentFile(FileSystem fileSystem, String dirName,
+	private static long writeCurrentFile(FileSystem fileSystem, String dir_name,
 		RemoteIterator<LocatedFileStatus> fileStatusListIterator,
-		TarArchiveOutputStream ar, long currentSize) throws IOException {
+		TarArchiveOutputStream ar, long current_size) throws IOException {
 		LocatedFileStatus fileStatus = fileStatusListIterator.next();
 
 		Path p = fileStatus.getPath();
-		String pString = p.toString();
-		if (!pString.endsWith("_SUCCESS")) {
-			String name = pString.substring(pString.lastIndexOf("/") + 1);
+		String p_string = p.toString();
+		if (!p_string.endsWith("_SUCCESS")) {
+			String name = p_string.substring(p_string.lastIndexOf("/") + 1);
 			if (name.startsWith("part-") & name.length() > 10) {
 				String tmp = name.substring(0, 10);
 				if (name.contains(".")) {
@@ -148,9 +92,9 @@ public class MakeTarArchive implements Serializable {
 				}
 				name = tmp;
 			}
-			TarArchiveEntry entry = new TarArchiveEntry(dirName + "/" + name);
+			TarArchiveEntry entry = new TarArchiveEntry(dir_name + "/" + name);
 			entry.setSize(fileStatus.getLen());
-			currentSize += fileStatus.getLen();
+			current_size += fileStatus.getLen();
 			ar.putArchiveEntry(entry);
 
 			InputStream is = fileSystem.open(fileStatus.getPath());
@@ -166,7 +110,7 @@ public class MakeTarArchive implements Serializable {
 			ar.closeArchiveEntry();
 
 		}
-		return currentSize;
+		return current_size;
 	}
 
 }

dhp-common/src/main/java/eu/dnetlib/dhp/common/MdstoreClient.java

@@ -1,12 +1,12 @@
 
 package eu.dnetlib.dhp.common;
 
-import static com.mongodb.client.model.Sorts.descending;
-
 import java.io.Closeable;
 import java.io.IOException;
-import java.util.*;
-import java.util.stream.Collectors;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
 import java.util.stream.StreamSupport;
 
 import org.apache.commons.lang3.StringUtils;
@@ -38,26 +38,6 @@ public class MdstoreClient implements Closeable {
 		this.db = getDb(client, dbName);
 	}
 
-	private Long parseTimestamp(Document f) {
-		if (f == null || !f.containsKey("timestamp"))
-			return null;
-
-		Object ts = f.get("timestamp");
-
-		return Long.parseLong(ts.toString());
-	}
-
-	public Long getLatestTimestamp(final String collectionId) {
-		MongoCollection<Document> collection = db.getCollection(collectionId);
-		FindIterable<Document> result = collection.find().sort(descending("timestamp")).limit(1);
-		if (result == null) {
-			return null;
-		}
-
-		Document f = result.first();
-		return parseTimestamp(f);
-	}
-
 	public MongoCollection<Document> mdStore(final String mdId) {
 		BasicDBObject query = (BasicDBObject) QueryBuilder.start("mdId").is(mdId).get();
 
@@ -74,16 +54,6 @@ public class MdstoreClient implements Closeable {
 		return getColl(db, currentId, true);
 	}
 
-	public List<MDStoreInfo> mdStoreWithTimestamp(final String mdFormat, final String mdLayout,
-		final String mdInterpretation) {
-		Map<String, String> res = validCollections(mdFormat, mdLayout, mdInterpretation);
-		return res
-			.entrySet()
-			.stream()
-			.map(e -> new MDStoreInfo(e.getKey(), e.getValue(), getLatestTimestamp(e.getValue())))
-			.collect(Collectors.toList());
-	}
-
 	public Map<String, String> validCollections(
 		final String mdFormat, final String mdLayout, final String mdInterpretation) {
 

dhp-common/src/main/java/eu/dnetlib/dhp/common/PacePerson.java

@@ -1,18 +1,18 @@
 
 package eu.dnetlib.dhp.common;
 
-import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.text.Normalizer;
-import java.util.*;
-import java.util.stream.Collectors;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
 
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang3.text.WordUtils;
 
-import com.ctc.wstx.dtd.LargePrefixedNameSet;
 import com.google.common.base.Joiner;
 import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
 import com.google.common.hash.Hashing;
 
@@ -29,19 +29,7 @@ public class PacePerson {
 	private List<String> fullname = Lists.newArrayList();
 	private final String original;
 
-	private static Set<String> particles;
-
-	static {
-		try {
-			particles = new HashSet<>(IOUtils
-				.readLines(
-					PacePerson.class
-						.getResourceAsStream(
-							"/eu/dnetlib/dhp/common/name_particles.txt")));
-		} catch (IOException e) {
-			throw new RuntimeException(e);
-		}
-	}
+	private static Set<String> particles = null;
 
 	/**
 	 * Capitalizes a string
@@ -49,20 +37,29 @@ public class PacePerson {
 	 * @param s the string to capitalize
 	 * @return the input string with capital letter
 	 */
-	public static String capitalize(final String s) {
-		if (particles.contains(s)) {
-			return s;
-		}
+	public static final String capitalize(final String s) {
 		return WordUtils.capitalize(s.toLowerCase(), ' ', '-');
 	}
 
 	/**
 	 * Adds a dot to a string with length equals to 1
 	 */
-	public static String dotAbbreviations(final String s) {
+	public static final String dotAbbreviations(final String s) {
 		return s.length() == 1 ? s + "." : s;
 	}
 
+	public static Set<String> loadFromClasspath(final String classpath) {
+		final Set<String> h = new HashSet<>();
+		try {
+			for (final String s : IOUtils.readLines(PacePerson.class.getResourceAsStream(classpath))) {
+				h.add(s);
+			}
+		} catch (final Throwable e) {
+			return new HashSet<>();
+		}
+		return h;
+	}
+
 	/**
 	 * The constructor of the class. It fills the fields of the class basing on the input fullname.
 	 *
@@ -131,6 +128,10 @@ public class PacePerson {
 	}
 
 	private List<String> splitTerms(final String s) {
+		if (particles == null) {
+			particles = loadFromClasspath("/eu/dnetlib/dhp/oa/graph/pace/name_particles.txt");
+		}
+
 		final List<String> list = Lists.newArrayList();
 		for (final String part : Splitter.on(" ").omitEmptyStrings().split(s)) {
 			if (!particles.contains(part.toLowerCase())) {
@@ -186,36 +187,17 @@ public class PacePerson {
 	}
 
 	public List<String> getCapitalFirstnames() {
-		return Optional
-			.ofNullable(getNameWithAbbreviations())
-			.map(
-				name -> name
-					.stream()
-					.map(PacePerson::capitalize)
-					.collect(Collectors.toList()))
-			.orElse(new ArrayList<>());
+		return Lists
+			.newArrayList(
+				Iterables.transform(getNameWithAbbreviations(), PacePerson::capitalize));
 	}
 
 	public List<String> getCapitalSurname() {
-		return Optional
-			.ofNullable(getSurname())
-			.map(
-				surname -> surname
-					.stream()
-					.map(PacePerson::capitalize)
-					.collect(Collectors.toList()))
-			.orElse(new ArrayList<>());
+		return Lists.newArrayList(Iterables.transform(surname, PacePerson::capitalize));
 	}
 
 	public List<String> getNameWithAbbreviations() {
-		return Optional
-			.ofNullable(getName())
-			.map(
-				name -> name
-					.stream()
-					.map(PacePerson::dotAbbreviations)
-					.collect(Collectors.toList()))
-			.orElse(new ArrayList<>());
+		return Lists.newArrayList(Iterables.transform(name, PacePerson::dotAbbreviations));
 	}
 
 	public boolean isAccurate() {

dhp-common/src/main/java/eu/dnetlib/dhp/common/action/ReadDatasourceMasterDuplicateFromDB.java

@@ -1,81 +0,0 @@
-
-package eu.dnetlib.dhp.common.action;
-
-import java.io.BufferedWriter;
-import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.nio.charset.StandardCharsets;
-import java.sql.ResultSet;
-import java.sql.SQLException;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-
-import eu.dnetlib.dhp.common.DbClient;
-import eu.dnetlib.dhp.common.action.model.MasterDuplicate;
-import eu.dnetlib.dhp.schema.oaf.utils.OafMapperUtils;
-
-public class ReadDatasourceMasterDuplicateFromDB {
-
-	private static final Logger log = LoggerFactory.getLogger(ReadDatasourceMasterDuplicateFromDB.class);
-
-	private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
-
-	private static final String QUERY = "SELECT distinct dd.id as masterId, d.officialname as masterName, dd.duplicate as duplicateId "
-		+
-		"FROM dsm_dedup_services dd join dsm_services d on (dd.id = d.id);";
-
-	public static int execute(String dbUrl, String dbUser, String dbPassword, String hdfsPath, String hdfsNameNode)
-		throws IOException {
-		int count = 0;
-		try (DbClient dbClient = new DbClient(dbUrl, dbUser, dbPassword)) {
-			Configuration conf = new Configuration();
-			conf.set("fs.defaultFS", hdfsNameNode);
-			FileSystem fileSystem = FileSystem.get(conf);
-			FSDataOutputStream fos = fileSystem.create(new Path(hdfsPath));
-
-			log.info("running query: {}", QUERY);
-			log.info("storing results in: {}", hdfsPath);
-
-			try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(fos, StandardCharsets.UTF_8))) {
-				dbClient.processResults(QUERY, rs -> writeMap(datasourceMasterMap(rs), writer));
-				count++;
-			}
-		}
-		return count;
-	}
-
-	private static MasterDuplicate datasourceMasterMap(ResultSet rs) {
-		try {
-			final MasterDuplicate md = new MasterDuplicate();
-
-			final String duplicateId = rs.getString("duplicateId");
-			final String masterId = rs.getString("masterId");
-			final String masterName = rs.getString("masterName");
-
-			md.setDuplicateId(OafMapperUtils.createOpenaireId(10, duplicateId, true));
-			md.setMasterId(OafMapperUtils.createOpenaireId(10, masterId, true));
-			md.setMasterName(masterName);
-
-			return md;
-		} catch (final SQLException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	private static void writeMap(final MasterDuplicate dm, final BufferedWriter writer) {
-		try {
-			writer.write(OBJECT_MAPPER.writeValueAsString(dm));
-			writer.newLine();
-		} catch (final IOException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/action/model/MasterDuplicate.java

@@ -1,38 +0,0 @@
-
-package eu.dnetlib.dhp.common.action.model;
-
-import java.io.Serializable;
-
-/**
- * @author miriam.baglioni
- * @Date 21/07/22
- */
-public class MasterDuplicate implements Serializable {
-	private String duplicateId;
-	private String masterId;
-	private String masterName;
-
-	public String getDuplicateId() {
-		return duplicateId;
-	}
-
-	public void setDuplicateId(String duplicateId) {
-		this.duplicateId = duplicateId;
-	}
-
-	public String getMasterId() {
-		return masterId;
-	}
-
-	public void setMasterId(String masterId) {
-		this.masterId = masterId;
-	}
-
-	public String getMasterName() {
-		return masterName;
-	}
-
-	public void setMasterName(String masterName) {
-		this.masterName = masterName;
-	}
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/InputStreamRequestBody.java

@@ -0,0 +1,53 @@
+
+package eu.dnetlib.dhp.common.api;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import okhttp3.MediaType;
+import okhttp3.RequestBody;
+import okhttp3.internal.Util;
+import okio.BufferedSink;
+import okio.Okio;
+import okio.Source;
+
+public class InputStreamRequestBody extends RequestBody {
+
+	private final InputStream inputStream;
+	private final MediaType mediaType;
+	private final long lenght;
+
+	public static RequestBody create(final MediaType mediaType, final InputStream inputStream, final long len) {
+
+		return new InputStreamRequestBody(inputStream, mediaType, len);
+	}
+
+	private InputStreamRequestBody(InputStream inputStream, MediaType mediaType, long len) {
+		this.inputStream = inputStream;
+		this.mediaType = mediaType;
+		this.lenght = len;
+	}
+
+	@Override
+	public MediaType contentType() {
+		return mediaType;
+	}
+
+	@Override
+	public long contentLength() {
+
+		return lenght;
+
+	}
+
+	@Override
+	public void writeTo(BufferedSink sink) throws IOException {
+		Source source = null;
+		try {
+			source = Okio.source(inputStream);
+			sink.writeAll(source);
+		} finally {
+			Util.closeQuietly(source);
+		}
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/MissingConceptDoiException.java

@@ -0,0 +1,8 @@
+
+package eu.dnetlib.dhp.common.api;
+
+public class MissingConceptDoiException extends Throwable {
+	public MissingConceptDoiException(String message) {
+		super(message);
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/ZenodoAPIClient.java

@@ -0,0 +1,318 @@
+
+package eu.dnetlib.dhp.common.api;
+
+import java.io.*;
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.http.HttpHeaders;
+import org.apache.http.entity.ContentType;
+
+import com.google.gson.Gson;
+
+import eu.dnetlib.dhp.common.api.zenodo.ZenodoModel;
+import eu.dnetlib.dhp.common.api.zenodo.ZenodoModelList;
+import okhttp3.*;
+
+public class ZenodoAPIClient implements Serializable {
+
+	String urlString;
+	String bucket;
+
+	String deposition_id;
+	String access_token;
+
+	public static final MediaType MEDIA_TYPE_JSON = MediaType.parse("application/json; charset=utf-8");
+
+	private static final MediaType MEDIA_TYPE_ZIP = MediaType.parse("application/zip");
+
+	public String getUrlString() {
+		return urlString;
+	}
+
+	public void setUrlString(String urlString) {
+		this.urlString = urlString;
+	}
+
+	public String getBucket() {
+		return bucket;
+	}
+
+	public void setBucket(String bucket) {
+		this.bucket = bucket;
+	}
+
+	public void setDeposition_id(String deposition_id) {
+		this.deposition_id = deposition_id;
+	}
+
+	public ZenodoAPIClient(String urlString, String access_token) {
+
+		this.urlString = urlString;
+		this.access_token = access_token;
+	}
+
+	/**
+	 * Brand new deposition in Zenodo. It sets the deposition_id and the bucket where to store the files to upload
+	 *
+	 * @return response code
+	 * @throws IOException
+	 */
+	public int newDeposition() throws IOException {
+		String json = "{}";
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		RequestBody body = RequestBody.create(json, MEDIA_TYPE_JSON);
+
+		Request request = new Request.Builder()
+			.url(urlString)
+			.addHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString()) // add request headers
+			.addHeader(HttpHeaders.AUTHORIZATION, "Bearer " + access_token)
+			.post(body)
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			// Get response body
+			json = response.body().string();
+
+			ZenodoModel newSubmission = new Gson().fromJson(json, ZenodoModel.class);
+			this.bucket = newSubmission.getLinks().getBucket();
+			this.deposition_id = newSubmission.getId();
+
+			return response.code();
+
+		}
+
+	}
+
+	/**
+	 * Upload files in Zenodo.
+	 *
+	 * @param is the inputStream for the file to upload
+	 * @param file_name the name of the file as it will appear on Zenodo
+	 * @param len the size of the file
+	 * @return the response code
+	 */
+	public int uploadIS(InputStream is, String file_name, long len) throws IOException {
+		OkHttpClient httpClient = new OkHttpClient.Builder()
+			.writeTimeout(600, TimeUnit.SECONDS)
+			.readTimeout(600, TimeUnit.SECONDS)
+			.connectTimeout(600, TimeUnit.SECONDS)
+			.build();
+
+		Request request = new Request.Builder()
+			.url(bucket + "/" + file_name)
+			.addHeader(HttpHeaders.CONTENT_TYPE, "application/zip") // add request headers
+			.addHeader(HttpHeaders.AUTHORIZATION, "Bearer " + access_token)
+			.put(InputStreamRequestBody.create(MEDIA_TYPE_ZIP, is, len))
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+			return response.code();
+		}
+	}
+
+	/**
+	 * Associates metadata information to the current deposition
+	 *
+	 * @param metadata the metadata
+	 * @return response code
+	 * @throws IOException
+	 */
+	public int sendMretadata(String metadata) throws IOException {
+
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		RequestBody body = RequestBody.create(metadata, MEDIA_TYPE_JSON);
+
+		Request request = new Request.Builder()
+			.url(urlString + "/" + deposition_id)
+			.addHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString()) // add request headers
+			.addHeader(HttpHeaders.AUTHORIZATION, "Bearer " + access_token)
+			.put(body)
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			return response.code();
+
+		}
+
+	}
+
+	/**
+	 * To publish the current deposition. It works for both new deposition or new version of an old deposition
+	 *
+	 * @return response code
+	 * @throws IOException
+	 */
+	public int publish() throws IOException {
+
+		String json = "{}";
+
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		RequestBody body = RequestBody.create(json, MEDIA_TYPE_JSON);
+
+		Request request = new Request.Builder()
+			.url(urlString + "/" + deposition_id + "/actions/publish")
+			.addHeader("Authorization", "Bearer " + access_token)
+			.post(body)
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			return response.code();
+
+		}
+	}
+
+	/**
+	 * To create a new version of an already published deposition. It sets the deposition_id and the bucket to be used
+	 * for the new version.
+	 *
+	 * @param concept_rec_id the concept record id of the deposition for which to create a new version. It is the last
+	 *            part of the url for the DOI Zenodo suggests to use to cite all versions: DOI: 10.xxx/zenodo.656930
+	 *            concept_rec_id = 656930
+	 * @return response code
+	 * @throws IOException
+	 * @throws MissingConceptDoiException
+	 */
+	public int newVersion(String concept_rec_id) throws IOException, MissingConceptDoiException {
+		setDepositionId(concept_rec_id);
+		String json = "{}";
+
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		RequestBody body = RequestBody.create(json, MEDIA_TYPE_JSON);
+
+		Request request = new Request.Builder()
+			.url(urlString + "/" + deposition_id + "/actions/newversion")
+			.addHeader(HttpHeaders.AUTHORIZATION, "Bearer " + access_token)
+			.post(body)
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			ZenodoModel zenodoModel = new Gson().fromJson(response.body().string(), ZenodoModel.class);
+			String latest_draft = zenodoModel.getLinks().getLatest_draft();
+			deposition_id = latest_draft.substring(latest_draft.lastIndexOf("/") + 1);
+			bucket = getBucket(latest_draft);
+			return response.code();
+
+		}
+	}
+
+	/**
+	 * To finish uploading a version or new deposition not published
+	 * It sets the deposition_id and the bucket to be used
+	 *
+	 *
+	 * @param deposition_id the deposition id of the not yet published upload
+	 *            concept_rec_id = 656930
+	 * @return response code
+	 * @throws IOException
+	 * @throws MissingConceptDoiException
+	 */
+	public int uploadOpenDeposition(String deposition_id) throws IOException, MissingConceptDoiException {
+
+		this.deposition_id = deposition_id;
+
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		Request request = new Request.Builder()
+			.url(urlString + "/" + deposition_id)
+			.addHeader("Authorization", "Bearer " + access_token)
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			ZenodoModel zenodoModel = new Gson().fromJson(response.body().string(), ZenodoModel.class);
+			bucket = zenodoModel.getLinks().getBucket();
+			return response.code();
+
+		}
+
+	}
+
+	private void setDepositionId(String concept_rec_id) throws IOException, MissingConceptDoiException {
+
+		ZenodoModelList zenodoModelList = new Gson().fromJson(getPrevDepositions(), ZenodoModelList.class);
+
+		for (ZenodoModel zm : zenodoModelList) {
+			if (zm.getConceptrecid().equals(concept_rec_id)) {
+				deposition_id = zm.getId();
+				return;
+			}
+		}
+
+		throw new MissingConceptDoiException("The concept record id specified was missing in the list of depositions");
+
+	}
+
+	private String getPrevDepositions() throws IOException {
+		OkHttpClient httpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).build();
+
+		Request request = new Request.Builder()
+			.url(urlString)
+			.addHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString()) // add request headers
+			.addHeader(HttpHeaders.AUTHORIZATION, "Bearer " + access_token)
+			.get()
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			return response.body().string();
+
+		}
+
+	}
+
+	private String getBucket(String url) throws IOException {
+		OkHttpClient httpClient = new OkHttpClient.Builder()
+			.connectTimeout(600, TimeUnit.SECONDS)
+			.build();
+
+		Request request = new Request.Builder()
+			.url(url)
+			.addHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString()) // add request headers
+			.addHeader(HttpHeaders.AUTHORIZATION, "Bearer " + access_token)
+			.get()
+			.build();
+
+		try (Response response = httpClient.newCall(request).execute()) {
+
+			if (!response.isSuccessful())
+				throw new IOException("Unexpected code " + response + response.body().string());
+
+			// Get response body
+			ZenodoModel zenodoModel = new Gson().fromJson(response.body().string(), ZenodoModel.class);
+
+			return zenodoModel.getLinks().getBucket();
+
+		}
+
+	}
+
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/CategorySummary.java

@@ -1,39 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-public class CategorySummary {
-
-	private String id;
-
-	private String label;
-
-	private boolean hasConcept;
-
-	public String getId() {
-		return id;
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public boolean isHasConcept() {
-		return hasConcept;
-	}
-
-	public CategorySummary setId(final String id) {
-		this.id = id;
-		return this;
-	}
-
-	public CategorySummary setLabel(final String label) {
-		this.label = label;
-		return this;
-	}
-
-	public CategorySummary setHasConcept(final boolean hasConcept) {
-		this.hasConcept = hasConcept;
-		return this;
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/CategorySummaryList.java

@@ -1,7 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.ArrayList;
-
-public class CategorySummaryList extends ArrayList<CategorySummary> {
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ConceptSummary.java

@@ -1,52 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.List;
-
-public class ConceptSummary {
-
-	private String id;
-
-	private String label;
-
-	public boolean hasSubConcept;
-
-	private List<ConceptSummary> concepts;
-
-	public String getId() {
-		return id;
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public List<ConceptSummary> getConcepts() {
-		return concepts;
-	}
-
-	public ConceptSummary setId(final String id) {
-		this.id = id;
-		return this;
-	}
-
-	public ConceptSummary setLabel(final String label) {
-		this.label = label;
-		return this;
-	}
-
-	public boolean isHasSubConcept() {
-		return hasSubConcept;
-	}
-
-	public ConceptSummary setHasSubConcept(final boolean hasSubConcept) {
-		this.hasSubConcept = hasSubConcept;
-		return this;
-	}
-
-	public ConceptSummary setConcept(final List<ConceptSummary> concepts) {
-		this.concepts = concepts;
-		return this;
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ConceptSummaryList.java

@@ -1,7 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.ArrayList;
-
-public class ConceptSummaryList extends ArrayList<ConceptSummary> {
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ContextSummary.java

@@ -1,50 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-public class ContextSummary {
-
-	private String id;
-
-	private String label;
-
-	private String type;
-
-	private String status;
-
-	public String getId() {
-		return id;
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public String getType() {
-		return type;
-	}
-
-	public String getStatus() {
-		return status;
-	}
-
-	public ContextSummary setId(final String id) {
-		this.id = id;
-		return this;
-	}
-
-	public ContextSummary setLabel(final String label) {
-		this.label = label;
-		return this;
-	}
-
-	public ContextSummary setType(final String type) {
-		this.type = type;
-		return this;
-	}
-
-	public ContextSummary setStatus(final String status) {
-		this.status = status;
-		return this;
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/context/ContextSummaryList.java

@@ -1,7 +0,0 @@
-
-package eu.dnetlib.dhp.common.api.context;
-
-import java.util.ArrayList;
-
-public class ContextSummaryList extends ArrayList<ContextSummary> {
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Community.java

@@ -0,0 +1,14 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+public class Community {
+	private String identifier;
+
+	public String getIdentifier() {
+		return identifier;
+	}
+
+	public void setIdentifier(String identifier) {
+		this.identifier = identifier;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Creator.java

@@ -0,0 +1,47 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+public class Creator {
+	private String affiliation;
+	private String name;
+	private String orcid;
+
+	public String getAffiliation() {
+		return affiliation;
+	}
+
+	public void setAffiliation(String affiliation) {
+		this.affiliation = affiliation;
+	}
+
+	public String getName() {
+		return name;
+	}
+
+	public void setName(String name) {
+		this.name = name;
+	}
+
+	public String getOrcid() {
+		return orcid;
+	}
+
+	public void setOrcid(String orcid) {
+		this.orcid = orcid;
+	}
+
+	public static Creator newInstance(String name, String affiliation, String orcid) {
+		Creator c = new Creator();
+		if (name != null) {
+			c.name = name;
+		}
+		if (affiliation != null) {
+			c.affiliation = affiliation;
+		}
+		if (orcid != null) {
+			c.orcid = orcid;
+		}
+
+		return c;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/File.java

@@ -0,0 +1,44 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class File implements Serializable {
+	private String checksum;
+	private String filename;
+	private long filesize;
+	private String id;
+
+	public String getChecksum() {
+		return checksum;
+	}
+
+	public void setChecksum(String checksum) {
+		this.checksum = checksum;
+	}
+
+	public String getFilename() {
+		return filename;
+	}
+
+	public void setFilename(String filename) {
+		this.filename = filename;
+	}
+
+	public long getFilesize() {
+		return filesize;
+	}
+
+	public void setFilesize(long filesize) {
+		this.filesize = filesize;
+	}
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(String id) {
+		this.id = id;
+	}
+
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Grant.java

@@ -0,0 +1,23 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class Grant implements Serializable {
+	private String id;
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(String id) {
+		this.id = id;
+	}
+
+	public static Grant newInstance(String id) {
+		Grant g = new Grant();
+		g.id = id;
+
+		return g;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Links.java

@@ -0,0 +1,92 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class Links implements Serializable {
+
+	private String bucket;
+
+	private String discard;
+
+	private String edit;
+	private String files;
+	private String html;
+	private String latest_draft;
+	private String latest_draft_html;
+	private String publish;
+
+	private String self;
+
+	public String getBucket() {
+		return bucket;
+	}
+
+	public void setBucket(String bucket) {
+		this.bucket = bucket;
+	}
+
+	public String getDiscard() {
+		return discard;
+	}
+
+	public void setDiscard(String discard) {
+		this.discard = discard;
+	}
+
+	public String getEdit() {
+		return edit;
+	}
+
+	public void setEdit(String edit) {
+		this.edit = edit;
+	}
+
+	public String getFiles() {
+		return files;
+	}
+
+	public void setFiles(String files) {
+		this.files = files;
+	}
+
+	public String getHtml() {
+		return html;
+	}
+
+	public void setHtml(String html) {
+		this.html = html;
+	}
+
+	public String getLatest_draft() {
+		return latest_draft;
+	}
+
+	public void setLatest_draft(String latest_draft) {
+		this.latest_draft = latest_draft;
+	}
+
+	public String getLatest_draft_html() {
+		return latest_draft_html;
+	}
+
+	public void setLatest_draft_html(String latest_draft_html) {
+		this.latest_draft_html = latest_draft_html;
+	}
+
+	public String getPublish() {
+		return publish;
+	}
+
+	public void setPublish(String publish) {
+		this.publish = publish;
+	}
+
+	public String getSelf() {
+		return self;
+	}
+
+	public void setSelf(String self) {
+		this.self = self;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/Metadata.java

@@ -0,0 +1,153 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+import java.util.List;
+
+public class Metadata implements Serializable {
+
+	private String access_right;
+	private List<Community> communities;
+	private List<Creator> creators;
+	private String description;
+	private String doi;
+	private List<Grant> grants;
+	private List<String> keywords;
+	private String language;
+	private String license;
+	private PrereserveDoi prereserve_doi;
+	private String publication_date;
+	private List<String> references;
+	private List<RelatedIdentifier> related_identifiers;
+	private String title;
+	private String upload_type;
+	private String version;
+
+	public String getUpload_type() {
+		return upload_type;
+	}
+
+	public void setUpload_type(String upload_type) {
+		this.upload_type = upload_type;
+	}
+
+	public String getVersion() {
+		return version;
+	}
+
+	public void setVersion(String version) {
+		this.version = version;
+	}
+
+	public String getAccess_right() {
+		return access_right;
+	}
+
+	public void setAccess_right(String access_right) {
+		this.access_right = access_right;
+	}
+
+	public List<Community> getCommunities() {
+		return communities;
+	}
+
+	public void setCommunities(List<Community> communities) {
+		this.communities = communities;
+	}
+
+	public List<Creator> getCreators() {
+		return creators;
+	}
+
+	public void setCreators(List<Creator> creators) {
+		this.creators = creators;
+	}
+
+	public String getDescription() {
+		return description;
+	}
+
+	public void setDescription(String description) {
+		this.description = description;
+	}
+
+	public String getDoi() {
+		return doi;
+	}
+
+	public void setDoi(String doi) {
+		this.doi = doi;
+	}
+
+	public List<Grant> getGrants() {
+		return grants;
+	}
+
+	public void setGrants(List<Grant> grants) {
+		this.grants = grants;
+	}
+
+	public List<String> getKeywords() {
+		return keywords;
+	}
+
+	public void setKeywords(List<String> keywords) {
+		this.keywords = keywords;
+	}
+
+	public String getLanguage() {
+		return language;
+	}
+
+	public void setLanguage(String language) {
+		this.language = language;
+	}
+
+	public String getLicense() {
+		return license;
+	}
+
+	public void setLicense(String license) {
+		this.license = license;
+	}
+
+	public PrereserveDoi getPrereserve_doi() {
+		return prereserve_doi;
+	}
+
+	public void setPrereserve_doi(PrereserveDoi prereserve_doi) {
+		this.prereserve_doi = prereserve_doi;
+	}
+
+	public String getPublication_date() {
+		return publication_date;
+	}
+
+	public void setPublication_date(String publication_date) {
+		this.publication_date = publication_date;
+	}
+
+	public List<String> getReferences() {
+		return references;
+	}
+
+	public void setReferences(List<String> references) {
+		this.references = references;
+	}
+
+	public List<RelatedIdentifier> getRelated_identifiers() {
+		return related_identifiers;
+	}
+
+	public void setRelated_identifiers(List<RelatedIdentifier> related_identifiers) {
+		this.related_identifiers = related_identifiers;
+	}
+
+	public String getTitle() {
+		return title;
+	}
+
+	public void setTitle(String title) {
+		this.title = title;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/PrereserveDoi.java

@@ -0,0 +1,25 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class PrereserveDoi implements Serializable {
+	private String doi;
+	private String recid;
+
+	public String getDoi() {
+		return doi;
+	}
+
+	public void setDoi(String doi) {
+		this.doi = doi;
+	}
+
+	public String getRecid() {
+		return recid;
+	}
+
+	public void setRecid(String recid) {
+		this.recid = recid;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/RelatedIdentifier.java

@@ -0,0 +1,43 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+
+public class RelatedIdentifier implements Serializable {
+	private String identifier;
+	private String relation;
+	private String resource_type;
+	private String scheme;
+
+	public String getIdentifier() {
+		return identifier;
+	}
+
+	public void setIdentifier(String identifier) {
+		this.identifier = identifier;
+	}
+
+	public String getRelation() {
+		return relation;
+	}
+
+	public void setRelation(String relation) {
+		this.relation = relation;
+	}
+
+	public String getResource_type() {
+		return resource_type;
+	}
+
+	public void setResource_type(String resource_type) {
+		this.resource_type = resource_type;
+	}
+
+	public String getScheme() {
+		return scheme;
+	}
+
+	public void setScheme(String scheme) {
+		this.scheme = scheme;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/ZenodoModel.java

@@ -0,0 +1,118 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.io.Serializable;
+import java.util.List;
+
+public class ZenodoModel implements Serializable {
+
+	private String conceptrecid;
+	private String created;
+
+	private List<File> files;
+	private String id;
+	private Links links;
+	private Metadata metadata;
+	private String modified;
+	private String owner;
+	private String record_id;
+	private String state;
+	private boolean submitted;
+	private String title;
+
+	public String getConceptrecid() {
+		return conceptrecid;
+	}
+
+	public void setConceptrecid(String conceptrecid) {
+		this.conceptrecid = conceptrecid;
+	}
+
+	public String getCreated() {
+		return created;
+	}
+
+	public void setCreated(String created) {
+		this.created = created;
+	}
+
+	public List<File> getFiles() {
+		return files;
+	}
+
+	public void setFiles(List<File> files) {
+		this.files = files;
+	}
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(String id) {
+		this.id = id;
+	}
+
+	public Links getLinks() {
+		return links;
+	}
+
+	public void setLinks(Links links) {
+		this.links = links;
+	}
+
+	public Metadata getMetadata() {
+		return metadata;
+	}
+
+	public void setMetadata(Metadata metadata) {
+		this.metadata = metadata;
+	}
+
+	public String getModified() {
+		return modified;
+	}
+
+	public void setModified(String modified) {
+		this.modified = modified;
+	}
+
+	public String getOwner() {
+		return owner;
+	}
+
+	public void setOwner(String owner) {
+		this.owner = owner;
+	}
+
+	public String getRecord_id() {
+		return record_id;
+	}
+
+	public void setRecord_id(String record_id) {
+		this.record_id = record_id;
+	}
+
+	public String getState() {
+		return state;
+	}
+
+	public void setState(String state) {
+		this.state = state;
+	}
+
+	public boolean isSubmitted() {
+		return submitted;
+	}
+
+	public void setSubmitted(boolean submitted) {
+		this.submitted = submitted;
+	}
+
+	public String getTitle() {
+		return title;
+	}
+
+	public void setTitle(String title) {
+		this.title = title;
+	}
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/api/zenodo/ZenodoModelList.java

@@ -0,0 +1,7 @@
+
+package eu.dnetlib.dhp.common.api.zenodo;
+
+import java.util.ArrayList;
+
+public class ZenodoModelList extends ArrayList<ZenodoModel> {
+}

dhp-common/src/main/java/eu/dnetlib/dhp/common/collection/DecompressTarGz.java

@@ -1,40 +0,0 @@
-
-package eu.dnetlib.dhp.common.collection;
-
-import java.io.BufferedOutputStream;
-import java.io.IOException;
-import java.util.zip.GZIPOutputStream;
-
-import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
-import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
-import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;
-import org.apache.commons.io.IOUtils;
-import org.apache.hadoop.fs.FSDataInputStream;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-public class DecompressTarGz {
-
-	public static void doExtract(FileSystem fs, String outputPath, String tarGzPath) throws IOException {
-
-		FSDataInputStream inputFileStream = fs.open(new Path(tarGzPath));
-		try (TarArchiveInputStream tais = new TarArchiveInputStream(
-			new GzipCompressorInputStream(inputFileStream))) {
-			TarArchiveEntry entry = null;
-			while ((entry = tais.getNextTarEntry()) != null) {
-				if (!entry.isDirectory()) {
-					try (
-						FSDataOutputStream out = fs
-							.create(new Path(outputPath.concat(entry.getName()).concat(".gz")));
-						GZIPOutputStream gzipOs = new GZIPOutputStream(new BufferedOutputStream(out))) {
-
-						IOUtils.copy(tais, gzipOs);
-
-					}
-
-				}
-			}
-		}
-	}
-}

dhp-common/src/main/java/eu/dnetlib/dhp/common/collection/HttpClientParams.java

@@ -1,9 +1,6 @@
 
 package eu.dnetlib.dhp.common.collection;
 
-import java.util.HashMap;
-import java.util.Map;
-
 /**
  * Bundles the http connection parameters driving the client behaviour.
  */
@@ -16,8 +13,6 @@ public class HttpClientParams {
 	public static int _connectTimeOut = 10; // seconds
 	public static int _readTimeOut = 30; // seconds
 
-	public static String _requestMethod = "GET";
-
 	/**
 	 * Maximum number of allowed retires before failing
 	 */
@@ -43,30 +38,17 @@ public class HttpClientParams {
 	 */
 	private int readTimeOut;
 
-	/**
-	 * Custom http headers
-	 */
-	private Map<String, String> headers;
-
-	/**
-	 * Request method (i.e., GET, POST etc)
-	 */
-	private String requestMethod;
-
 	public HttpClientParams() {
-		this(_maxNumberOfRetry, _requestDelay, _retryDelay, _connectTimeOut, _readTimeOut, new HashMap<>(),
-			_requestMethod);
+		this(_maxNumberOfRetry, _requestDelay, _retryDelay, _connectTimeOut, _readTimeOut);
 	}
 
 	public HttpClientParams(int maxNumberOfRetry, int requestDelay, int retryDelay, int connectTimeOut,
-		int readTimeOut, Map<String, String> headers, String requestMethod) {
+		int readTimeOut) {
 		this.maxNumberOfRetry = maxNumberOfRetry;
 		this.requestDelay = requestDelay;
 		this.retryDelay = retryDelay;
 		this.connectTimeOut = connectTimeOut;
 		this.readTimeOut = readTimeOut;
-		this.headers = headers;
-		this.requestMethod = requestMethod;
 	}
 
 	public int getMaxNumberOfRetry() {
@@ -109,19 +91,4 @@ public class HttpClientParams {
 		this.readTimeOut = readTimeOut;
 	}
 
-	public Map<String, String> getHeaders() {
-		return headers;
-	}
-
-	public void setHeaders(Map<String, String> headers) {
-		this.headers = headers;
-	}
-
-	public String getRequestMethod() {
-		return requestMethod;
-	}
-
-	public void setRequestMethod(String requestMethod) {
-		this.requestMethod = requestMethod;
-	}
 }

dhp-common/src/main/java/eu/dnetlib/dhp/common/collection/HttpConnector2.java

@@ -8,13 +8,10 @@ import java.io.InputStream;
 import java.net.*;
 import java.util.List;
 import java.util.Map;
-import java.util.concurrent.TimeUnit;
 
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang3.math.NumberUtils;
-import org.apache.commons.lang3.time.DateUtils;
 import org.apache.http.HttpHeaders;
-import org.joda.time.Instant;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -97,37 +94,29 @@ public class HttpConnector2 {
 			throw new CollectorException(msg);
 		}
 
+		log.info("Request attempt {} [{}]", retryNumber, requestUrl);
+
 		InputStream input = null;
 
-		long start = System.currentTimeMillis();
 		try {
 			if (getClientParams().getRequestDelay() > 0) {
 				backoffAndSleep(getClientParams().getRequestDelay());
 			}
-
-			log.info("Request attempt {} [{}]", retryNumber, requestUrl);
-
 			final HttpURLConnection urlConn = (HttpURLConnection) new URL(requestUrl).openConnection();
 			urlConn.setInstanceFollowRedirects(false);
 			urlConn.setReadTimeout(getClientParams().getReadTimeOut() * 1000);
 			urlConn.setConnectTimeout(getClientParams().getConnectTimeOut() * 1000);
 			urlConn.addRequestProperty(HttpHeaders.USER_AGENT, userAgent);
-			urlConn.setRequestMethod(getClientParams().getRequestMethod());
 
-			// if provided, add custom headers
-			if (!getClientParams().getHeaders().isEmpty()) {
-				for (Map.Entry<String, String> headerEntry : getClientParams().getHeaders().entrySet()) {
-					urlConn.addRequestProperty(headerEntry.getKey(), headerEntry.getValue());
-				}
+			if (log.isDebugEnabled()) {
+				logHeaderFields(urlConn);
 			}
 
-			logHeaderFields(urlConn);
-
 			int retryAfter = obtainRetryAfter(urlConn.getHeaderFields());
 			String rateLimit = urlConn.getHeaderField(Constants.HTTPHEADER_IETF_DRAFT_RATELIMIT_LIMIT);
 			String rateRemaining = urlConn.getHeaderField(Constants.HTTPHEADER_IETF_DRAFT_RATELIMIT_REMAINING);
 
-			if ((rateLimit != null) && (rateRemaining != null) && (Integer.parseInt(rateRemaining) < 2)) {
+			if ((rateLimit != null) && (rateRemaining != null) && (Integer.parseInt(rateRemaining) < 9)) {
 				if (retryAfter > 0) {
 					backoffAndSleep(retryAfter);
 				} else {
@@ -136,7 +125,9 @@ public class HttpConnector2 {
 			}
 
 			if (is2xx(urlConn.getResponseCode())) {
-				return getInputStream(urlConn, start);
+				input = urlConn.getInputStream();
+				responseType = urlConn.getContentType();
+				return input;
 			}
 			if (is3xx(urlConn.getResponseCode())) {
 				// REDIRECTS
@@ -146,7 +137,6 @@ public class HttpConnector2 {
 					.put(
 						REPORT_PREFIX + urlConn.getResponseCode(),
 						String.format("Moved to: %s", newUrl));
-				logRequestTime(start);
 				urlConn.disconnect();
 				if (retryAfter > 0) {
 					backoffAndSleep(retryAfter);
@@ -162,50 +152,26 @@ public class HttpConnector2 {
 						if (retryAfter > 0) {
 							log
 								.warn(
-									"waiting and repeating request after suggested retry-after {} sec for URL {}",
-									retryAfter, requestUrl);
+									"{} - waiting and repeating request after suggested retry-after {} sec.",
+									requestUrl, retryAfter);
 							backoffAndSleep(retryAfter * 1000);
 						} else {
 							log
 								.warn(
-									"waiting and repeating request after default delay of {} sec for URL {}",
-									getClientParams().getRetryDelay(), requestUrl);
-							backoffAndSleep(retryNumber * getClientParams().getRetryDelay());
+									"{} - waiting and repeating request after default delay of {} sec.",
+									requestUrl, getClientParams().getRetryDelay());
+							backoffAndSleep(retryNumber * getClientParams().getRetryDelay() * 1000);
 						}
 						report.put(REPORT_PREFIX + urlConn.getResponseCode(), requestUrl);
-
-						logRequestTime(start);
-
 						urlConn.disconnect();
-
 						return attemptDownload(requestUrl, retryNumber + 1, report);
-					case 422: // UNPROCESSABLE ENTITY
-						report.put(REPORT_PREFIX + urlConn.getResponseCode(), requestUrl);
-						log.warn("waiting and repeating request after 10 sec for URL {}", requestUrl);
-						backoffAndSleep(10000);
-						urlConn.disconnect();
-						logRequestTime(start);
-						try {
-							return getInputStream(urlConn, start);
-						} catch (IOException e) {
-							log
-								.error(
-									"server returned 422 and got IOException accessing the response body from URL {}",
-									requestUrl);
-							log.error("IOException:", e);
-							return attemptDownload(requestUrl, retryNumber + 1, report);
-						}
 					default:
-						log.error("gor error {} from URL: {}", urlConn.getResponseCode(), urlConn.getURL());
-						log.error("response message: {}", urlConn.getResponseMessage());
 						report
 							.put(
 								REPORT_PREFIX + urlConn.getResponseCode(),
 								String
 									.format(
 										"%s Error: %s", requestUrl, urlConn.getResponseMessage()));
-						logRequestTime(start);
-						urlConn.disconnect();
 						throw new CollectorException(urlConn.getResponseCode() + " error " + report);
 				}
 			}
@@ -226,27 +192,13 @@ public class HttpConnector2 {
 		}
 	}
 
-	private InputStream getInputStream(HttpURLConnection urlConn, long start) throws IOException {
-		InputStream input = urlConn.getInputStream();
-		responseType = urlConn.getContentType();
-		logRequestTime(start);
-		return input;
-	}
-
-	private static void logRequestTime(long start) {
-		log
-			.info(
-				"request time elapsed: {}sec",
-				TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - start));
-	}
-
 	private void logHeaderFields(final HttpURLConnection urlConn) throws IOException {
-		log.info("Response: {} - {}", urlConn.getResponseCode(), urlConn.getResponseMessage());
+		log.debug("StatusCode: {}", urlConn.getResponseMessage());
 
 		for (Map.Entry<String, List<String>> e : urlConn.getHeaderFields().entrySet()) {
 			if (e.getKey() != null) {
 				for (String v : e.getValue()) {
-					log.info("  key: {} - value: {}", e.getKey(), v);
+					log.debug("  key: {} - value: {}", e.getKey(), v);
 				}
 			}
 		}
@@ -266,7 +218,7 @@ public class HttpConnector2 {
 		for (String key : headerMap.keySet()) {
 			if ((key != null) && key.equalsIgnoreCase(HttpHeaders.RETRY_AFTER) && (!headerMap.get(key).isEmpty())
 				&& NumberUtils.isCreatable(headerMap.get(key).get(0))) {
-				return Integer.parseInt(headerMap.get(key).get(0));
+				return Integer.parseInt(headerMap.get(key).get(0)) + 10;
 			}
 		}
 		return -1;

dhp-common/src/main/java/eu/dnetlib/dhp/common/vocabulary/Vocabulary.java

@@ -4,7 +4,6 @@ package eu.dnetlib.dhp.common.vocabulary;
 import java.io.Serializable;
 import java.util.HashMap;
 import java.util.Map;
-import java.util.Objects;
 import java.util.Optional;
 
 import org.apache.commons.lang3.StringUtils;
@@ -63,46 +62,25 @@ public class Vocabulary implements Serializable {
 	}
 
 	public VocabularyTerm getTermBySynonym(final String syn) {
-		return Optional
-			.ofNullable(syn)
-			.map(s -> getTerm(synonyms.get(s.toLowerCase())))
-			.orElse(null);
+		return getTerm(synonyms.get(syn.toLowerCase()));
 	}
 
 	public Qualifier getTermAsQualifier(final String termId) {
-		return getTermAsQualifier(termId, false);
-	}
-
-	public Qualifier getTermAsQualifier(final String termId, boolean strict) {
-		final VocabularyTerm term = getTerm(termId);
-		if (Objects.nonNull(term)) {
-			return OafMapperUtils.qualifier(term.getId(), term.getName(), getId(), getName());
-		} else if (Objects.isNull(term) && strict) {
+		if (StringUtils.isBlank(termId)) {
 			return OafMapperUtils.unknown(getId(), getName());
+		} else if (termExists(termId)) {
+			final VocabularyTerm t = getTerm(termId);
+			return OafMapperUtils.qualifier(t.getId(), t.getName(), getId(), getName());
 		} else {
 			return OafMapperUtils.qualifier(termId, termId, getId(), getName());
 		}
 	}
 
 	public Qualifier getSynonymAsQualifier(final String syn) {
-		return getSynonymAsQualifier(syn, false);
-	}
-
-	public Qualifier getSynonymAsQualifier(final String syn, boolean strict) {
 		return Optional
 			.ofNullable(getTermBySynonym(syn))
-			.map(term -> getTermAsQualifier(term.getId(), strict))
+			.map(term -> getTermAsQualifier(term.getId()))
 			.orElse(null);
 	}
 
-	public Qualifier lookup(String id) {
-		return lookup(id, false);
-	}
-
-	public Qualifier lookup(String id, boolean strict) {
-		return Optional
-			.ofNullable(getSynonymAsQualifier(id, strict))
-			.orElse(getTermAsQualifier(id, strict));
-	}
-
 }

dhp-common/src/main/java/eu/dnetlib/dhp/common/vocabulary/VocabularyGroup.java

@@ -81,13 +81,6 @@ public class VocabularyGroup implements Serializable {
 		vocs.put(id.toLowerCase(), new Vocabulary(id, name));
 	}
 
-	public Optional<Vocabulary> find(final String vocId) {
-		return Optional
-			.ofNullable(vocId)
-			.map(String::toLowerCase)
-			.map(vocs::get);
-	}
-
 	public void addTerm(final String vocId, final String id, final String name) {
 		if (vocabularyExists(vocId)) {
 			vocs.get(vocId.toLowerCase()).addTerm(id, name);
@@ -135,24 +128,6 @@ public class VocabularyGroup implements Serializable {
 		return vocs.get(vocId.toLowerCase()).getSynonymAsQualifier(syn);
 	}
 
-	public Qualifier lookupTermBySynonym(final String vocId, final String syn) {
-		return find(vocId)
-			.map(
-				vocabulary -> Optional
-					.ofNullable(vocabulary.getTerm(syn))
-					.map(
-						term -> OafMapperUtils
-							.qualifier(term.getId(), term.getName(), vocabulary.getId(), vocabulary.getName()))
-					.orElse(
-						Optional
-							.ofNullable(vocabulary.getTermBySynonym(syn))
-							.map(
-								term -> OafMapperUtils
-									.qualifier(term.getId(), term.getName(), vocabulary.getId(), vocabulary.getName()))
-							.orElse(null)))
-			.orElse(null);
-	}
-
 	/**
 	 * getSynonymAsQualifierCaseSensitive
 	 *

dhp-common/src/main/java/eu/dnetlib/dhp/oa/merge/AuthorMerger.java

@@ -119,131 +119,6 @@ public class AuthorMerger {
 				});
 	}
 
-	public static String normalizeFullName(final String fullname) {
-		return nfd(fullname)
-			.toLowerCase()
-			// do not compact the regexes in a single expression, would cause StackOverflowError
-			// in case
-			// of large input strings
-			.replaceAll("(\\W)+", " ")
-			.replaceAll("(\\p{InCombiningDiacriticalMarks})+", " ")
-			.replaceAll("(\\p{Punct})+", " ")
-			.replaceAll("(\\d)+", " ")
-			.replaceAll("(\\n)+", " ")
-
-			.trim();
-	}
-
-	private static String authorFieldToBeCompared(Author author) {
-		if (StringUtils.isNotBlank(author.getSurname())) {
-			return author.getSurname();
-
-		}
-		if (StringUtils.isNotBlank(author.getFullname())) {
-			return author.getFullname();
-		}
-		return null;
-	}
-
-	/**
-	 * This method tries to figure out when two author are the same in the contest
-	 * of ORCID enrichment
-	 *
-	 * @param left  Author in the OAF entity
-	 * @param right Author ORCID
-	 * @return based on a heuristic on the names of the authors if they are the same.
-	 */
-	public static boolean checkORCIDSimilarity(final Author left, final Author right) {
-		final Person pl = parse(left);
-		final Person pr = parse(right);
-
-		// If one of them didn't have a surname we verify if they have the fullName not empty
-		// and verify if the normalized version is equal
-		if (!(pl.getSurname() != null && pl.getSurname().stream().anyMatch(StringUtils::isNotBlank) &&
-			pr.getSurname() != null && pr.getSurname().stream().anyMatch(StringUtils::isNotBlank))) {
-
-			if (pl.getFullname() != null && !pl.getFullname().isEmpty() && pr.getFullname() != null
-				&& !pr.getFullname().isEmpty()) {
-				return pl
-					.getFullname()
-					.stream()
-					.anyMatch(
-						fl -> pr.getFullname().stream().anyMatch(fr -> normalize(fl).equalsIgnoreCase(normalize(fr))));
-			} else {
-				return false;
-			}
-		}
-		// The Authors have one surname in common
-		if (pl.getSurname().stream().anyMatch(sl -> pr.getSurname().stream().anyMatch(sr -> sr.equalsIgnoreCase(sl)))) {
-
-			// If one of them has only a surname and is the same we can say that they are the same author
-			if ((pl.getName() == null || pl.getName().stream().allMatch(StringUtils::isBlank)) ||
-				(pr.getName() == null || pr.getName().stream().allMatch(StringUtils::isBlank)))
-				return true;
-			// The authors have the same initials of Name in common
-			if (pl
-				.getName()
-				.stream()
-				.anyMatch(
-					nl -> pr
-						.getName()
-						.stream()
-						.anyMatch(nr -> nr.equalsIgnoreCase(nl))))
-				return true;
-		}
-
-		// Sometimes we noticed that publication have author wrote in inverse order Surname, Name
-		// We verify if we have an exact match between name and surname
-		if (pl.getSurname().stream().anyMatch(sl -> pr.getName().stream().anyMatch(nr -> nr.equalsIgnoreCase(sl))) &&
-			pl.getName().stream().anyMatch(nl -> pr.getSurname().stream().anyMatch(sr -> sr.equalsIgnoreCase(nl))))
-			return true;
-		else
-			return false;
-	}
-	//
-
-	/**
-	 * Method to enrich ORCID information in one list of authors based on another list
-	 *
-	 * @param baseAuthor  the Author List in the OAF Entity
-	 * @param orcidAuthor The list of ORCID Author intersected
-	 * @return The Author List of the OAF Entity enriched with the orcid Author
-	 */
-	public static List<Author> enrichOrcid(List<Author> baseAuthor, List<Author> orcidAuthor) {
-
-		if (baseAuthor == null || baseAuthor.isEmpty())
-			return orcidAuthor;
-
-		if (orcidAuthor == null || orcidAuthor.isEmpty())
-			return baseAuthor;
-
-		if (baseAuthor.size() == 1 && orcidAuthor.size() > 10)
-			return baseAuthor;
-
-		final List<Author> oAuthor = new ArrayList<>();
-		oAuthor.addAll(orcidAuthor);
-
-		baseAuthor.forEach(ba -> {
-			Optional<Author> aMatch = oAuthor.stream().filter(oa -> checkORCIDSimilarity(ba, oa)).findFirst();
-			if (aMatch.isPresent()) {
-				final Author sameAuthor = aMatch.get();
-				addPid(ba, sameAuthor.getPid());
-				oAuthor.remove(sameAuthor);
-			}
-		});
-		return baseAuthor;
-	}
-
-	private static void addPid(final Author a, final List<StructuredProperty> pids) {
-
-		if (a.getPid() == null) {
-			a.setPid(new ArrayList<>());
-		}
-
-		a.getPid().addAll(pids);
-
-	}
-
 	public static String pidToComparableString(StructuredProperty pid) {
 		final String classid = pid.getQualifier().getClassid() != null ? pid.getQualifier().getClassid().toLowerCase()
 			: "";
@@ -296,7 +171,7 @@ public class AuthorMerger {
 		}
 	}
 
-	public static String normalize(final String s) {
+	private static String normalize(final String s) {
 		String[] normalized = nfd(s)
 			.toLowerCase()
 			// do not compact the regexes in a single expression, would cause StackOverflowError

dhp-common/src/main/java/eu/dnetlib/dhp/oa/merge/GroupEntitiesSparkJob.java

@@ -1,194 +0,0 @@
-
-package eu.dnetlib.dhp.oa.merge;
-
-import static eu.dnetlib.dhp.common.SparkSessionSupport.runWithSparkSession;
-import static org.apache.spark.sql.functions.col;
-import static org.apache.spark.sql.functions.when;
-
-import java.util.Map;
-import java.util.Optional;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ForkJoinPool;
-import java.util.stream.Collectors;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.spark.SparkConf;
-import org.apache.spark.api.java.function.MapFunction;
-import org.apache.spark.api.java.function.ReduceFunction;
-import org.apache.spark.sql.*;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import eu.dnetlib.dhp.application.ArgumentApplicationParser;
-import eu.dnetlib.dhp.common.HdfsSupport;
-import eu.dnetlib.dhp.common.vocabulary.VocabularyGroup;
-import eu.dnetlib.dhp.schema.common.EntityType;
-import eu.dnetlib.dhp.schema.common.ModelSupport;
-import eu.dnetlib.dhp.schema.oaf.OafEntity;
-import eu.dnetlib.dhp.schema.oaf.utils.GraphCleaningFunctions;
-import eu.dnetlib.dhp.schema.oaf.utils.OafMapperUtils;
-import eu.dnetlib.dhp.utils.ISLookupClientFactory;
-import eu.dnetlib.enabling.is.lookup.rmi.ISLookUpException;
-import eu.dnetlib.enabling.is.lookup.rmi.ISLookUpService;
-import scala.Tuple2;
-
-/**
- * Groups the graph content by entity identifier to ensure ID uniqueness
- */
-public class GroupEntitiesSparkJob {
-	private static final Logger log = LoggerFactory.getLogger(GroupEntitiesSparkJob.class);
-
-	private static final Encoder<OafEntity> OAFENTITY_KRYO_ENC = Encoders.kryo(OafEntity.class);
-
-	private ArgumentApplicationParser parser;
-
-	public GroupEntitiesSparkJob(ArgumentApplicationParser parser) {
-		this.parser = parser;
-	}
-
-	public static void main(String[] args) throws Exception {
-
-		String jsonConfiguration = IOUtils
-			.toString(
-				GroupEntitiesSparkJob.class
-					.getResourceAsStream(
-						"/eu/dnetlib/dhp/oa/merge/group_graph_entities_parameters.json"));
-		final ArgumentApplicationParser parser = new ArgumentApplicationParser(jsonConfiguration);
-		parser.parseArgument(args);
-
-		Boolean isSparkSessionManaged = Optional
-			.ofNullable(parser.get("isSparkSessionManaged"))
-			.map(Boolean::valueOf)
-			.orElse(Boolean.TRUE);
-		log.info("isSparkSessionManaged: {}", isSparkSessionManaged);
-
-		final String isLookupUrl = parser.get("isLookupUrl");
-		log.info("isLookupUrl: {}", isLookupUrl);
-
-		final ISLookUpService isLookupService = ISLookupClientFactory.getLookUpService(isLookupUrl);
-
-		new GroupEntitiesSparkJob(parser).run(isSparkSessionManaged, isLookupService);
-	}
-
-	public void run(Boolean isSparkSessionManaged, ISLookUpService isLookUpService)
-		throws ISLookUpException {
-
-		String graphInputPath = parser.get("graphInputPath");
-		log.info("graphInputPath: {}", graphInputPath);
-
-		String checkpointPath = parser.get("checkpointPath");
-		log.info("checkpointPath: {}", checkpointPath);
-
-		String outputPath = parser.get("outputPath");
-		log.info("outputPath: {}", outputPath);
-
-		boolean filterInvisible = Boolean.parseBoolean(parser.get("filterInvisible"));
-		log.info("filterInvisible: {}", filterInvisible);
-
-		SparkConf conf = new SparkConf();
-		conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
-		conf.registerKryoClasses(ModelSupport.getOafModelClasses());
-
-		final VocabularyGroup vocs = VocabularyGroup.loadVocsFromIS(isLookUpService);
-
-		runWithSparkSession(
-			conf,
-			isSparkSessionManaged,
-			spark -> {
-				HdfsSupport.remove(checkpointPath, spark.sparkContext().hadoopConfiguration());
-				groupEntities(spark, graphInputPath, checkpointPath, outputPath, filterInvisible, vocs);
-			});
-	}
-
-	private static void groupEntities(
-		SparkSession spark,
-		String inputPath,
-		String checkpointPath,
-		String outputPath,
-		boolean filterInvisible, VocabularyGroup vocs) {
-
-		Dataset<OafEntity> allEntities = spark.emptyDataset(OAFENTITY_KRYO_ENC);
-
-		for (Map.Entry<EntityType, Class> e : ModelSupport.entityTypes.entrySet()) {
-			String entity = e.getKey().name();
-			Class<? extends OafEntity> entityClass = e.getValue();
-			String entityInputPath = inputPath + "/" + entity;
-
-			if (!HdfsSupport.exists(entityInputPath, spark.sparkContext().hadoopConfiguration())) {
-				continue;
-			}
-
-			allEntities = allEntities
-				.union(
-					((Dataset<OafEntity>) spark
-						.read()
-						.schema(Encoders.bean(entityClass).schema())
-						.json(entityInputPath)
-						.filter("length(id) > 0")
-						.as(Encoders.bean(entityClass)))
-							.map((MapFunction<OafEntity, OafEntity>) r -> r, OAFENTITY_KRYO_ENC));
-		}
-
-		Dataset<?> groupedEntities = allEntities
-			.map(
-				(MapFunction<OafEntity, OafEntity>) entity -> GraphCleaningFunctions
-					.applyCoarVocabularies(entity, vocs),
-				OAFENTITY_KRYO_ENC)
-			.groupByKey((MapFunction<OafEntity, String>) OafEntity::getId, Encoders.STRING())
-			.reduceGroups((ReduceFunction<OafEntity>) OafMapperUtils::mergeEntities)
-			.map(
-				(MapFunction<Tuple2<String, OafEntity>, Tuple2<String, OafEntity>>) t -> new Tuple2<>(
-					t._2().getClass().getName(), t._2()),
-				Encoders.tuple(Encoders.STRING(), OAFENTITY_KRYO_ENC));
-
-		// pivot on "_1" (classname of the entity)
-		// created columns containing only entities of the same class
-		for (Map.Entry<EntityType, Class> e : ModelSupport.entityTypes.entrySet()) {
-			String entity = e.getKey().name();
-			Class<? extends OafEntity> entityClass = e.getValue();
-
-			groupedEntities = groupedEntities
-				.withColumn(
-					entity,
-					when(col("_1").equalTo(entityClass.getName()), col("_2")));
-		}
-
-		groupedEntities
-			.drop("_1", "_2")
-			.write()
-			.mode(SaveMode.Overwrite)
-			.option("compression", "gzip")
-			.save(checkpointPath);
-
-		ForkJoinPool parPool = new ForkJoinPool(ModelSupport.entityTypes.size());
-
-		ModelSupport.entityTypes
-			.entrySet()
-			.stream()
-			.map(e -> parPool.submit(() -> {
-				String entity = e.getKey().name();
-				Class<? extends OafEntity> entityClass = e.getValue();
-
-				spark
-					.read()
-					.load(checkpointPath)
-					.select(col(entity).as("value"))
-					.filter("value IS NOT NULL")
-					.as(OAFENTITY_KRYO_ENC)
-					.map((MapFunction<OafEntity, OafEntity>) r -> r, (Encoder<OafEntity>) Encoders.bean(entityClass))
-					.filter(filterInvisible ? "dataInfo.invisible != TRUE" : "TRUE")
-					.write()
-					.mode(SaveMode.Overwrite)
-					.option("compression", "gzip")
-					.json(outputPath + "/" + entity);
-			}))
-			.collect(Collectors.toList())
-			.forEach(t -> {
-				try {
-					t.get();
-				} catch (InterruptedException | ExecutionException e) {
-					throw new RuntimeException(e);
-				}
-			});
-	}
-}

dhp-common/src/main/java/eu/dnetlib/dhp/oozie/RunSQLSparkJob.java

@@ -1,77 +0,0 @@
-
-package eu.dnetlib.dhp.oozie;
-
-import static eu.dnetlib.dhp.common.SparkSessionSupport.runWithSparkHiveSession;
-
-import java.net.URL;
-import java.nio.charset.StandardCharsets;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Optional;
-
-import org.apache.commons.lang3.time.DurationFormatUtils;
-import org.apache.commons.text.StringSubstitutor;
-import org.apache.spark.SparkConf;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.io.Resources;
-
-import eu.dnetlib.dhp.application.ArgumentApplicationParser;
-
-public class RunSQLSparkJob {
-	private static final Logger log = LoggerFactory.getLogger(RunSQLSparkJob.class);
-
-	private final ArgumentApplicationParser parser;
-
-	public RunSQLSparkJob(ArgumentApplicationParser parser) {
-		this.parser = parser;
-	}
-
-	public static void main(String[] args) throws Exception {
-
-		Map<String, String> params = new HashMap<>();
-		for (int i = 0; i < args.length - 1; i++) {
-			if (args[i].startsWith("--")) {
-				params.put(args[i].substring(2), args[++i]);
-			}
-		}
-
-		/*
-		 * String jsonConfiguration = IOUtils .toString( Objects .requireNonNull( RunSQLSparkJob.class
-		 * .getResourceAsStream( "/eu/dnetlib/dhp/oozie/run_sql_parameters.json"))); final ArgumentApplicationParser
-		 * parser = new ArgumentApplicationParser(jsonConfiguration); parser.parseArgument(args);
-		 */
-
-		Boolean isSparkSessionManaged = Optional
-			.ofNullable(params.get("isSparkSessionManaged"))
-			.map(Boolean::valueOf)
-			.orElse(Boolean.TRUE);
-		log.info("isSparkSessionManaged: {}", isSparkSessionManaged);
-
-		URL url = com.google.common.io.Resources.getResource(params.get("sql"));
-		String raw_sql = Resources.toString(url, StandardCharsets.UTF_8);
-
-		String sql = StringSubstitutor.replace(raw_sql, params);
-		log.info("sql: {}", sql);
-
-		SparkConf conf = new SparkConf();
-		conf.set("hive.metastore.uris", params.get("hiveMetastoreUris"));
-
-		runWithSparkHiveSession(
-			conf,
-			isSparkSessionManaged,
-			spark -> {
-				for (String statement : sql.split(";\\s*/\\*\\s*EOS\\s*\\*/\\s*")) {
-					log.info("executing: {}", statement);
-					long startTime = System.currentTimeMillis();
-					spark.sql(statement).show();
-					log
-						.info(
-							"executed in {}",
-							DurationFormatUtils.formatDuration(System.currentTimeMillis() - startTime, "HH:mm:ss.S"));
-				}
-			});
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/DoiCleaningRule.java

@@ -1,14 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-public class DoiCleaningRule {
-
-	public static String clean(final String doi) {
-		return doi
-			.toLowerCase()
-			.replaceAll("\\s", "")
-			.replaceAll("^doi:", "")
-			.replaceFirst(CleaningFunctions.DOI_PREFIX_REGEX, CleaningFunctions.DOI_PREFIX);
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/FundRefCleaningRule.java

@@ -1,25 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-public class FundRefCleaningRule {
-
-	public static final Pattern PATTERN = Pattern.compile("\\d+");
-
-	public static String clean(final String fundRefId) {
-
-		String s = fundRefId
-			.toLowerCase()
-			.replaceAll("\\s", "");
-
-		Matcher m = PATTERN.matcher(s);
-		if (m.find()) {
-			return m.group();
-		} else {
-			return "";
-		}
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GridCleaningRule.java

@@ -1,24 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-public class GridCleaningRule {
-
-	public static final Pattern PATTERN = Pattern.compile("(?<grid>\\d{4,6}\\.[0-9a-z]{1,2})");
-
-	public static String clean(String grid) {
-		String s = grid
-			.replaceAll("\\s", "")
-			.toLowerCase();
-
-		Matcher m = PATTERN.matcher(s);
-		if (m.find()) {
-			return "grid." + m.group("grid");
-		}
-
-		return "";
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/ISNICleaningRule.java

@@ -1,21 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-// https://www.wikidata.org/wiki/Property:P213
-public class ISNICleaningRule {
-
-	public static final Pattern PATTERN = Pattern.compile("([0]{4}) ?([0-9]{4}) ?([0-9]{4}) ?([0-9]{3}[0-9X])");
-
-	public static String clean(final String isni) {
-
-		Matcher m = PATTERN.matcher(isni);
-		if (m.find()) {
-			return String.join("", m.group(1), m.group(2), m.group(3), m.group(4));
-		} else {
-			return "";
-		}
-	}
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/OafMapperUtils.java

@@ -3,8 +3,6 @@ package eu.dnetlib.dhp.schema.oaf.utils;
 
 import static eu.dnetlib.dhp.schema.common.ModelConstants.*;
 
-import java.sql.Array;
-import java.sql.SQLException;
 import java.util.*;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
@@ -49,17 +47,6 @@ public class OafMapperUtils {
 	}
 
 	public static Result mergeResults(Result left, Result right) {
-
-		final boolean leftFromDelegatedAuthority = isFromDelegatedAuthority(left);
-		final boolean rightFromDelegatedAuthority = isFromDelegatedAuthority(right);
-
-		if (leftFromDelegatedAuthority && !rightFromDelegatedAuthority) {
-			return left;
-		}
-		if (!leftFromDelegatedAuthority && rightFromDelegatedAuthority) {
-			return right;
-		}
-
 		if (new ResultTypeComparator().compare(left, right) < 0) {
 			left.mergeFrom(right);
 			return left;
@@ -69,18 +56,6 @@ public class OafMapperUtils {
 		}
 	}
 
-	private static boolean isFromDelegatedAuthority(Result r) {
-		return Optional
-			.ofNullable(r.getInstance())
-			.map(
-				instance -> instance
-					.stream()
-					.filter(i -> Objects.nonNull(i.getCollectedfrom()))
-					.map(i -> i.getCollectedfrom().getKey())
-					.anyMatch(cfId -> IdentifierFactory.delegatedAuthorityDatasourceIds().contains(cfId)))
-			.orElse(false);
-	}
-
 	public static KeyValue keyValue(final String k, final String v) {
 		final KeyValue kv = new KeyValue();
 		kv.setKey(k);
@@ -120,17 +95,6 @@ public class OafMapperUtils {
 			.collect(Collectors.toList());
 	}
 
-	public static <T> List<T> listValues(Array values) throws SQLException {
-		if (Objects.isNull(values)) {
-			return null;
-		}
-		return Arrays
-			.stream((T[]) values.getArray())
-			.filter(Objects::nonNull)
-			.distinct()
-			.collect(Collectors.toList());
-	}
-
 	public static List<Field<String>> listFields(final DataInfo info, final List<String> values) {
 		return values
 			.stream()
@@ -140,30 +104,8 @@ public class OafMapperUtils {
 			.collect(Collectors.toList());
 	}
 
-	public static InstanceTypeMapping instanceTypeMapping(String originalType, String code, String label,
-		String vocabularyName) {
-		final InstanceTypeMapping m = new InstanceTypeMapping();
-		m.setVocabularyName(vocabularyName);
-		m.setOriginalType(originalType);
-		m.setTypeCode(code);
-		m.setTypeLabel(label);
-		return m;
-	}
-
-	public static InstanceTypeMapping instanceTypeMapping(String originalType, Qualifier term) {
-		return instanceTypeMapping(originalType, term.getClassid(), term.getClassname(), term.getSchemeid());
-	}
-
-	public static InstanceTypeMapping instanceTypeMapping(String originalType) {
-		return instanceTypeMapping(originalType, null, null, null);
-	}
-
-	public static InstanceTypeMapping instanceTypeMapping(String originalType, String vocabularyName) {
-		return instanceTypeMapping(originalType, null, null, vocabularyName);
-	}
-
 	public static Qualifier unknown(final String schemeid, final String schemename) {
-		return qualifier(UNKNOWN, "Unknown", schemeid, schemename);
+		return qualifier("UNKNOWN", "Unknown", schemeid, schemename);
 	}
 
 	public static AccessRight accessRight(
@@ -211,17 +153,6 @@ public class OafMapperUtils {
 		return q;
 	}
 
-	public static Subject subject(
-		final String value,
-		final String classid,
-		final String classname,
-		final String schemeid,
-		final String schemename,
-		final DataInfo dataInfo) {
-
-		return subject(value, qualifier(classid, classname, schemeid, schemename), dataInfo);
-	}
-
 	public static StructuredProperty structuredProperty(
 		final String value,
 		final String classid,
@@ -233,20 +164,6 @@ public class OafMapperUtils {
 		return structuredProperty(value, qualifier(classid, classname, schemeid, schemename), dataInfo);
 	}
 
-	public static Subject subject(
-		final String value,
-		final Qualifier qualifier,
-		final DataInfo dataInfo) {
-		if (value == null) {
-			return null;
-		}
-		final Subject s = new Subject();
-		s.setValue(value);
-		s.setQualifier(qualifier);
-		s.setDataInfo(dataInfo);
-		return s;
-	}
-
 	public static StructuredProperty structuredProperty(
 		final String value,
 		final Qualifier qualifier,
@@ -451,88 +368,4 @@ public class OafMapperUtils {
 		}
 		return null;
 	}
-
-	public static KeyValue newKeyValueInstance(String key, String value, DataInfo dataInfo) {
-		KeyValue kv = new KeyValue();
-		kv.setDataInfo(dataInfo);
-		kv.setKey(key);
-		kv.setValue(value);
-		return kv;
-	}
-
-	public static Measure newMeasureInstance(String id, String value, String key, DataInfo dataInfo) {
-		Measure m = new Measure();
-		m.setId(id);
-		m.setUnit(Arrays.asList(newKeyValueInstance(key, value, dataInfo)));
-		return m;
-	}
-
-	public static Relation getRelation(final String source,
-		final String target,
-		final String relType,
-		final String subRelType,
-		final String relClass,
-		final OafEntity entity) {
-		return getRelation(source, target, relType, subRelType, relClass, entity, null);
-	}
-
-	public static Relation getRelation(final String source,
-		final String target,
-		final String relType,
-		final String subRelType,
-		final String relClass,
-		final OafEntity entity,
-		final String validationDate) {
-		return getRelation(
-			source, target, relType, subRelType, relClass, entity.getCollectedfrom(), entity.getDataInfo(),
-			entity.getLastupdatetimestamp(), validationDate, null);
-	}
-
-	public static Relation getRelation(final String source,
-		final String target,
-		final String relType,
-		final String subRelType,
-		final String relClass,
-		final List<KeyValue> collectedfrom,
-		final DataInfo dataInfo,
-		final Long lastupdatetimestamp) {
-		return getRelation(
-			source, target, relType, subRelType, relClass, collectedfrom, dataInfo, lastupdatetimestamp, null, null);
-	}
-
-	public static Relation getRelation(final String source,
-		final String target,
-		final String relType,
-		final String subRelType,
-		final String relClass,
-		final List<KeyValue> collectedfrom,
-		final DataInfo dataInfo,
-		final Long lastupdatetimestamp,
-		final String validationDate,
-		final List<KeyValue> properties) {
-		final Relation rel = new Relation();
-		rel.setRelType(relType);
-		rel.setSubRelType(subRelType);
-		rel.setRelClass(relClass);
-		rel.setSource(source);
-		rel.setTarget(target);
-		rel.setCollectedfrom(collectedfrom);
-		rel.setDataInfo(dataInfo);
-		rel.setLastupdatetimestamp(lastupdatetimestamp);
-		rel.setValidated(StringUtils.isNotBlank(validationDate));
-		rel.setValidationDate(StringUtils.isNotBlank(validationDate) ? validationDate : null);
-		rel.setProperties(properties);
-		return rel;
-	}
-
-	public static String getProvenance(DataInfo dataInfo) {
-		return Optional
-			.ofNullable(dataInfo)
-			.map(
-				d -> Optional
-					.ofNullable(d.getProvenanceaction())
-					.map(Qualifier::getClassid)
-					.orElse(""))
-			.orElse("");
-	}
 }

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/PICCleaningRule.java

@@ -1,21 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-public class PICCleaningRule {
-
-	public static final Pattern PATTERN = Pattern.compile("\\d{9}");
-
-	public static String clean(final String pic) {
-
-		Matcher m = PATTERN.matcher(pic);
-		if (m.find()) {
-			return m.group();
-		} else {
-			return "";
-		}
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/PidCleaner.java

@@ -1,62 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.Optional;
-
-import eu.dnetlib.dhp.schema.oaf.StructuredProperty;
-
-public class PidCleaner {
-
-	/**
-	 * Utility method that normalises PID values on a per-type basis.
-	 * @param pid the PID whose value will be normalised.
-	 * @return the PID containing the normalised value.
-	 */
-	public static StructuredProperty normalizePidValue(StructuredProperty pid) {
-		pid
-			.setValue(
-				normalizePidValue(
-					pid.getQualifier().getClassid(),
-					pid.getValue()));
-
-		return pid;
-	}
-
-	public static String normalizePidValue(String pidType, String pidValue) {
-		String value = Optional
-			.ofNullable(pidValue)
-			.map(String::trim)
-			.orElseThrow(() -> new IllegalArgumentException("PID value cannot be empty"));
-
-		switch (pidType) {
-
-			// TODO add cleaning for more PID types as needed
-
-			// Result
-			case "doi":
-				return DoiCleaningRule.clean(value);
-			case "pmid":
-				return PmidCleaningRule.clean(value);
-			case "pmc":
-				return PmcCleaningRule.clean(value);
-			case "handle":
-			case "arXiv":
-				return value;
-
-			// Organization
-			case "GRID":
-				return GridCleaningRule.clean(value);
-			case "ISNI":
-				return ISNICleaningRule.clean(value);
-			case "ROR":
-				return RorCleaningRule.clean(value);
-			case "PIC":
-				return PICCleaningRule.clean(value);
-			case "FundRef":
-				return FundRefCleaningRule.clean(value);
-			default:
-				return value;
-		}
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/PmcCleaningRule.java

@@ -1,24 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-public class PmcCleaningRule {
-
-	public static final Pattern PATTERN = Pattern.compile("PMC\\d{1,8}");
-
-	public static String clean(String pmc) {
-		String s = pmc
-			.replaceAll("\\s", "")
-			.toUpperCase();
-
-		final Matcher m = PATTERN.matcher(s);
-
-		if (m.find()) {
-			return m.group();
-		}
-		return "";
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/PmidCleaningRule.java

@@ -1,25 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-// https://researchguides.stevens.edu/c.php?g=442331&p=6577176
-public class PmidCleaningRule {
-
-	public static final Pattern PATTERN = Pattern.compile("0*(\\d{1,8})");
-
-	public static String clean(String pmid) {
-		String s = pmid
-			.toLowerCase()
-			.replaceAll("\\s", "");
-
-		final Matcher m = PATTERN.matcher(s);
-
-		if (m.find()) {
-			return m.group(1);
-		}
-		return "";
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/RorCleaningRule.java

@@ -1,27 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-// https://ror.readme.io/docs/ror-identifier-pattern
-public class RorCleaningRule {
-
-	public static final String ROR_PREFIX = "https://ror.org/";
-
-	private static final Pattern PATTERN = Pattern.compile("(?<ror>0[a-hj-km-np-tv-z|0-9]{6}[0-9]{2})");
-
-	public static String clean(String ror) {
-		String s = ror
-			.replaceAll("\\s", "")
-			.toLowerCase();
-
-		Matcher m = PATTERN.matcher(s);
-
-		if (m.find()) {
-			return ROR_PREFIX + m.group("ror");
-		}
-		return "";
-	}
-
-}

dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/SubjectProvenanceComparator.java

@@ -1,46 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static eu.dnetlib.dhp.schema.oaf.utils.OafMapperUtils.getProvenance;
-import static org.apache.commons.lang3.StringUtils.isBlank;
-
-import java.util.Comparator;
-
-import eu.dnetlib.dhp.schema.oaf.Subject;
-
-public class SubjectProvenanceComparator implements Comparator<Subject> {
-
-	@Override
-	public int compare(Subject left, Subject right) {
-
-		String lProv = getProvenance(left.getDataInfo());
-		String rProv = getProvenance(right.getDataInfo());
-
-		if (isBlank(lProv) && isBlank(rProv))
-			return 0;
-		if (isBlank(lProv))
-			return 1;
-		if (isBlank(rProv))
-			return -1;
-		if (lProv.equals(rProv))
-			return 0;
-		if (lProv.toLowerCase().contains("crosswalk"))
-			return -1;
-		if (rProv.toLowerCase().contains("crosswalk"))
-			return 1;
-		if (lProv.toLowerCase().contains("user"))
-			return -1;
-		if (rProv.toLowerCase().contains("user"))
-			return 1;
-		if (lProv.toLowerCase().contains("propagation"))
-			return -1;
-		if (rProv.toLowerCase().contains("propagation"))
-			return 1;
-		if (lProv.toLowerCase().contains("iis"))
-			return -1;
-		if (rProv.toLowerCase().contains("iis"))
-			return 1;
-
-		return 0;
-	}
-}

dhp-common/src/main/java/eu/dnetlib/dhp/utils/DHPUtils.java

@@ -75,14 +75,9 @@ public class DHPUtils {
 
 		final HttpGet req = new HttpGet(url);
 
-		log.info("MDStoreManager request: {}", req);
-
 		try (final CloseableHttpClient client = HttpClients.createDefault()) {
 			try (final CloseableHttpResponse response = client.execute(req)) {
 				final String json = IOUtils.toString(response.getEntity().getContent());
-
-				log.info("MDStoreManager response: {}", json);
-
 				final MDStoreWithInfo[] mdstores = objectMapper.readValue(json, MDStoreWithInfo[].class);
 				return Arrays
 					.stream(mdstores)

dhp-common/src/main/resources/eu/dnetlib/dhp/common/name_particles.txt

@@ -1,8 +0,0 @@
-van
-von
-der
-de
-dell
-sig
-mr
-mrs

dhp-common/src/main/resources/eu/dnetlib/dhp/oa/merge/group_graph_entities_parameters.json

@@ -1,38 +0,0 @@
-[
-  {
-    "paramName": "issm",
-    "paramLongName": "isSparkSessionManaged",
-    "paramDescription": "when true will stop SparkSession after job execution",
-    "paramRequired": false
-  },
-  {
-    "paramName": "gin",
-    "paramLongName": "graphInputPath",
-    "paramDescription": "the input graph root path",
-    "paramRequired": true
-  },
-  {
-    "paramName": "cp",
-    "paramLongName": "checkpointPath",
-    "paramDescription": "checkpoint directory",
-    "paramRequired": true
-  },
-  {
-    "paramName": "out",
-    "paramLongName": "outputPath",
-    "paramDescription": "the output graph root path",
-    "paramRequired": true
-  },
-  {
-    "paramName": "fi",
-    "paramLongName": "filterInvisible",
-    "paramDescription": "if true filters out invisible entities",
-    "paramRequired": true
-  },
-  {
-    "paramName": "isu",
-    "paramLongName": "isLookupUrl",
-    "paramDescription": "url to the ISLookup Service",
-    "paramRequired": true
-  }
-]

dhp-common/src/main/resources/eu/dnetlib/dhp/oozie/run_sql_parameters.json

@@ -1,20 +0,0 @@
-[
-  {
-    "paramName": "issm",
-    "paramLongName": "isSparkSessionManaged",
-    "paramDescription": "when true will stop SparkSession after job execution",
-    "paramRequired": false
-  },
-  {
-    "paramName": "hmu",
-    "paramLongName": "hiveMetastoreUris",
-    "paramDescription": "the hive metastore uris",
-    "paramRequired": true
-  },
-  {
-    "paramName": "sql",
-    "paramLongName": "sql",
-    "paramDescription": "sql script to execute",
-    "paramRequired": true
-  }
-]

dhp-common/src/main/scala/eu/dnetlib/dhp/application/dedup/log/DedupLogModel.scala

@@ -1,10 +0,0 @@
-package eu.dnetlib.dhp.application.dedup.log
-
-case class DedupLogModel(
-  tag: String,
-  configuration: String,
-  entity: String,
-  startTS: Long,
-  endTS: Long,
-  totalMs: Long
-) {}

dhp-common/src/main/scala/eu/dnetlib/dhp/application/dedup/log/DedupLogWriter.scala

@@ -1,14 +0,0 @@
-package eu.dnetlib.dhp.application.dedup.log
-
-import org.apache.spark.sql.{SaveMode, SparkSession}
-
-class DedupLogWriter(path: String) {
-
-  def appendLog(dedupLogModel: DedupLogModel, spark: SparkSession): Unit = {
-    import spark.implicits._
-    val df = spark.createDataset[DedupLogModel](data = List(dedupLogModel))
-    df.write.mode(SaveMode.Append).save(path)
-
-  }
-
-}

dhp-common/src/test/java/eu/dnetlib/dhp/common/MdStoreClientTest.java

@@ -1,36 +0,0 @@
-
-package eu.dnetlib.dhp.common;
-
-import java.io.IOException;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.nio.file.Paths;
-import java.util.List;
-
-import org.junit.jupiter.api.Test;
-
-import com.fasterxml.jackson.core.JsonProcessingException;
-import com.fasterxml.jackson.databind.ObjectMapper;
-
-public class MdStoreClientTest {
-
-	// @Test
-	public void testMongoCollection() throws IOException {
-		final MdstoreClient client = new MdstoreClient("mongodb://localhost:27017", "mdstore");
-
-		final ObjectMapper mapper = new ObjectMapper();
-
-		final List<MDStoreInfo> infos = client.mdStoreWithTimestamp("ODF", "store", "cleaned");
-
-		infos.forEach(System.out::println);
-
-		final String s = mapper.writeValueAsString(infos);
-
-		Path fileName = Paths.get("/Users/sandro/mdstore_info.json");
-
-		// Writing into the file
-		Files.write(fileName, s.getBytes(StandardCharsets.UTF_8));
-
-	}
-}

dhp-common/src/test/java/eu/dnetlib/dhp/common/api/ZenodoAPIClientTest.java

@@ -0,0 +1,109 @@
+
+package eu.dnetlib.dhp.common.api;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+@Disabled
+class ZenodoAPIClientTest {
+
+	private final String URL_STRING = "https://sandbox.zenodo.org/api/deposit/depositions";
+	private final String ACCESS_TOKEN = "";
+
+	private final String CONCEPT_REC_ID = "657113";
+
+	private final String depositionId = "674915";
+
+	@Test
+	void testUploadOldDeposition() throws IOException, MissingConceptDoiException {
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+		Assertions.assertEquals(200, client.uploadOpenDeposition(depositionId));
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/COVID-19.json.gz")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "COVID-19.json.gz", file.length()));
+
+		String metadata = IOUtils.toString(getClass().getResourceAsStream("/eu/dnetlib/dhp/common/api/metadata.json"));
+
+		Assertions.assertEquals(200, client.sendMretadata(metadata));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+	@Test
+	void testNewDeposition() throws IOException {
+
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+		Assertions.assertEquals(201, client.newDeposition());
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/COVID-19.json.gz")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "COVID-19.json.gz", file.length()));
+
+		String metadata = IOUtils.toString(getClass().getResourceAsStream("/eu/dnetlib/dhp/common/api/metadata.json"));
+
+		Assertions.assertEquals(200, client.sendMretadata(metadata));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+	@Test
+	void testNewVersionNewName() throws IOException, MissingConceptDoiException {
+
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+
+		Assertions.assertEquals(201, client.newVersion(CONCEPT_REC_ID));
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/newVersion")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "newVersion_deposition", file.length()));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+	@Test
+	void testNewVersionOldName() throws IOException, MissingConceptDoiException {
+
+		ZenodoAPIClient client = new ZenodoAPIClient(URL_STRING,
+			ACCESS_TOKEN);
+
+		Assertions.assertEquals(201, client.newVersion(CONCEPT_REC_ID));
+
+		File file = new File(getClass()
+			.getResource("/eu/dnetlib/dhp/common/api/newVersion2")
+			.getPath());
+
+		InputStream is = new FileInputStream(file);
+
+		Assertions.assertEquals(200, client.uploadIS(is, "newVersion_deposition", file.length()));
+
+		Assertions.assertEquals(202, client.publish());
+
+	}
+
+}

dhp-common/src/test/java/eu/dnetlib/dhp/oa/merge/AuthorMergerTest.java

@@ -0,0 +1,100 @@
+
+package eu.dnetlib.dhp.oa.merge;
+
+import java.io.BufferedReader;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import eu.dnetlib.dhp.schema.oaf.Author;
+import eu.dnetlib.dhp.schema.oaf.Publication;
+import eu.dnetlib.dhp.schema.oaf.StructuredProperty;
+import eu.dnetlib.pace.util.MapDocumentUtil;
+import scala.Tuple2;
+
+class AuthorMergerTest {
+
+	private String publicationsBasePath;
+
+	private List<List<Author>> authors;
+
+	@BeforeEach
+	public void setUp() throws Exception {
+
+		publicationsBasePath = Paths
+			.get(AuthorMergerTest.class.getResource("/eu/dnetlib/dhp/oa/merge").toURI())
+			.toFile()
+			.getAbsolutePath();
+
+		authors = readSample(publicationsBasePath + "/publications_with_authors.json", Publication.class)
+			.stream()
+			.map(p -> p._2().getAuthor())
+			.collect(Collectors.toList());
+
+	}
+
+	@Test
+	void mergeTest() { // used in the dedup: threshold set to 0.95
+
+		for (List<Author> authors1 : authors) {
+			System.out.println("List " + (authors.indexOf(authors1) + 1));
+			for (Author author : authors1) {
+				System.out.println(authorToString(author));
+			}
+		}
+
+		List<Author> merge = AuthorMerger.merge(authors);
+
+		System.out.println("Merge ");
+		for (Author author : merge) {
+			System.out.println(authorToString(author));
+		}
+
+		Assertions.assertEquals(7, merge.size());
+
+	}
+
+	public <T> List<Tuple2<String, T>> readSample(String path, Class<T> clazz) {
+		List<Tuple2<String, T>> res = new ArrayList<>();
+		BufferedReader reader;
+		try {
+			reader = new BufferedReader(new FileReader(path));
+			String line = reader.readLine();
+			while (line != null) {
+				res
+					.add(
+						new Tuple2<>(
+							MapDocumentUtil.getJPathString("$.id", line),
+							new ObjectMapper().readValue(line, clazz)));
+				// read next line
+				line = reader.readLine();
+			}
+			reader.close();
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+
+		return res;
+	}
+
+	public String authorToString(Author a) {
+
+		String print = "Fullname = ";
+		print += a.getFullname() + " pid = [";
+		if (a.getPid() != null)
+			for (StructuredProperty sp : a.getPid()) {
+				print += sp.toComparableString() + " ";
+			}
+		print += "]";
+		return print;
+	}
+}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/GridCleaningRuleTest.java

@@ -1,18 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import org.junit.jupiter.api.Test;
-
-class GridCleaningRuleTest {
-
-	@Test
-	void testCleaning() {
-		assertEquals("grid.493784.5", GridCleaningRule.clean("grid.493784.5"));
-		assertEquals("grid.493784.5x", GridCleaningRule.clean("grid.493784.5x"));
-		assertEquals("grid.493784.5x", GridCleaningRule.clean("493784.5x"));
-		assertEquals("", GridCleaningRule.clean("493x784.5x"));
-	}
-
-}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/ISNICleaningRuleTest.java

@@ -1,19 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import org.junit.jupiter.api.Test;
-
-class ISNICleaningRuleTest {
-
-	@Test
-	void testCleaning() {
-		assertEquals("0000000463436020", ISNICleaningRule.clean("0000 0004 6343 6020"));
-		assertEquals("0000000463436020", ISNICleaningRule.clean("0000000463436020"));
-		assertEquals("", ISNICleaningRule.clean("Q30256598"));
-		assertEquals("0000000493403529", ISNICleaningRule.clean("ISNI:0000000493403529"));
-		assertEquals("000000008614884X", ISNICleaningRule.clean("0000 0000 8614 884X"));
-	}
-
-}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/OafMapperUtilsTest.java

@@ -44,104 +44,105 @@ class OafMapperUtilsTest {
 	@Test
 	void testDateValidation() {
 
-		assertNotNull(GraphCleaningFunctions.cleanDate("2016-05-07T12:41:19.202Z  "));
-		assertNotNull(GraphCleaningFunctions.cleanDate("2020-09-10 11:08:52 "));
-		assertNotNull(GraphCleaningFunctions.cleanDate("  2016-04-05"));
+		assertTrue(GraphCleaningFunctions.doCleanDate("2016-05-07T12:41:19.202Z  ").isPresent());
+		assertTrue(GraphCleaningFunctions.doCleanDate("2020-09-10 11:08:52 ").isPresent());
+		assertTrue(GraphCleaningFunctions.doCleanDate("  2016-04-05").isPresent());
 
-		assertEquals("2016-04-05", GraphCleaningFunctions.cleanDate("2016 Apr 05"));
+		assertEquals("2016-04-05", GraphCleaningFunctions.doCleanDate("2016 Apr 05").get());
 
-		assertEquals("2009-05-08", GraphCleaningFunctions.cleanDate("May 8, 2009 5:57:51 PM"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("oct 7, 1970"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("oct 7, '70"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("oct. 7, 1970"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("oct. 7, 70"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("Mon Jan  2 15:04:05 2006"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("Mon Jan  2 15:04:05 MST 2006"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("Mon Jan 02 15:04:05 -0700 2006"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("Monday, 02-Jan-06 15:04:05 MST"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("Mon, 02 Jan 2006 15:04:05 MST"));
-		assertEquals("2017-07-11", GraphCleaningFunctions.cleanDate("Tue, 11 Jul 2017 16:28:13 +0200 (CEST)"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("Mon, 02 Jan 2006 15:04:05 -0700"));
-		assertEquals("2018-01-04", GraphCleaningFunctions.cleanDate("Thu, 4 Jan 2018 17:53:36 +0000"));
-		assertEquals("2015-08-10", GraphCleaningFunctions.cleanDate("Mon Aug 10 15:44:11 UTC+0100 2015"));
+		assertEquals("2009-05-08", GraphCleaningFunctions.doCleanDate("May 8, 2009 5:57:51 PM").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("oct 7, 1970").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("oct 7, '70").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("oct. 7, 1970").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("oct. 7, 70").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("Mon Jan  2 15:04:05 2006").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("Mon Jan  2 15:04:05 MST 2006").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("Mon Jan 02 15:04:05 -0700 2006").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("Monday, 02-Jan-06 15:04:05 MST").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("Mon, 02 Jan 2006 15:04:05 MST").get());
+		assertEquals("2017-07-11", GraphCleaningFunctions.doCleanDate("Tue, 11 Jul 2017 16:28:13 +0200 (CEST)").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("Mon, 02 Jan 2006 15:04:05 -0700").get());
+		assertEquals("2018-01-04", GraphCleaningFunctions.doCleanDate("Thu, 4 Jan 2018 17:53:36 +0000").get());
+		assertEquals("2015-08-10", GraphCleaningFunctions.doCleanDate("Mon Aug 10 15:44:11 UTC+0100 2015").get());
 		assertEquals(
 			"2015-07-03",
-			GraphCleaningFunctions.cleanDate("Fri Jul 03 2015 18:04:07 GMT+0100 (GMT Daylight Time)"));
-		assertEquals("2012-09-17", GraphCleaningFunctions.cleanDate("September 17, 2012 10:09am"));
-		assertEquals("2012-09-17", GraphCleaningFunctions.cleanDate("September 17, 2012 at 10:09am PST-08"));
-		assertEquals("2012-09-17", GraphCleaningFunctions.cleanDate("September 17, 2012, 10:10:09"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("October 7, 1970"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("October 7th, 1970"));
-		assertEquals("2006-02-12", GraphCleaningFunctions.cleanDate("12 Feb 2006, 19:17"));
-		assertEquals("2006-02-12", GraphCleaningFunctions.cleanDate("12 Feb 2006 19:17"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("7 oct 70"));
-		assertEquals("1970-10-07", GraphCleaningFunctions.cleanDate("7 oct 1970"));
-		assertEquals("2013-02-03", GraphCleaningFunctions.cleanDate("03 February 2013"));
-		assertEquals("2013-07-01", GraphCleaningFunctions.cleanDate("1 July 2013"));
-		assertEquals("2013-02-03", GraphCleaningFunctions.cleanDate("2013-Feb-03"));
-		assertEquals("2014-03-31", GraphCleaningFunctions.cleanDate("3/31/2014"));
-		assertEquals("2014-03-31", GraphCleaningFunctions.cleanDate("03/31/2014"));
-		assertEquals("1971-08-21", GraphCleaningFunctions.cleanDate("08/21/71"));
-		assertEquals("1971-01-08", GraphCleaningFunctions.cleanDate("8/1/71"));
-		assertEquals("2014-08-04", GraphCleaningFunctions.cleanDate("4/8/2014 22:05"));
-		assertEquals("2014-08-04", GraphCleaningFunctions.cleanDate("04/08/2014 22:05"));
-		assertEquals("2014-08-04", GraphCleaningFunctions.cleanDate("4/8/14 22:05"));
-		assertEquals("2014-02-04", GraphCleaningFunctions.cleanDate("04/2/2014 03:00:51"));
-		assertEquals("1965-08-08", GraphCleaningFunctions.cleanDate("8/8/1965 12:00:00 AM"));
-		assertEquals("1965-08-08", GraphCleaningFunctions.cleanDate("8/8/1965 01:00:01 PM"));
-		assertEquals("1965-08-08", GraphCleaningFunctions.cleanDate("8/8/1965 01:00 PM"));
-		assertEquals("1965-08-08", GraphCleaningFunctions.cleanDate("8/8/1965 1:00 PM"));
-		assertEquals("1965-08-08", GraphCleaningFunctions.cleanDate("8/8/1965 12:00 AM"));
-		assertEquals("2014-02-04", GraphCleaningFunctions.cleanDate("4/02/2014 03:00:51"));
-		assertEquals("2012-03-19", GraphCleaningFunctions.cleanDate("03/19/2012 10:11:59"));
-		assertEquals("2012-03-19", GraphCleaningFunctions.cleanDate("03/19/2012 10:11:59.3186369"));
-		assertEquals("2014-03-31", GraphCleaningFunctions.cleanDate("2014/3/31"));
-		assertEquals("2014-03-31", GraphCleaningFunctions.cleanDate("2014/03/31"));
-		assertEquals("2014-04-08", GraphCleaningFunctions.cleanDate("2014/4/8 22:05"));
-		assertEquals("2014-04-08", GraphCleaningFunctions.cleanDate("2014/04/08 22:05"));
-		assertEquals("2014-04-02", GraphCleaningFunctions.cleanDate("2014/04/2 03:00:51"));
-		assertEquals("2014-04-02", GraphCleaningFunctions.cleanDate("2014/4/02 03:00:51"));
-		assertEquals("2012-03-19", GraphCleaningFunctions.cleanDate("2012/03/19 10:11:59"));
-		assertEquals("2012-03-19", GraphCleaningFunctions.cleanDate("2012/03/19 10:11:59.3186369"));
-		assertEquals("2014-04-08", GraphCleaningFunctions.cleanDate("2014年04月08日"));
-		assertEquals("2006-01-02", GraphCleaningFunctions.cleanDate("2006-01-02T15:04:05+0000"));
-		assertEquals("2009-08-13", GraphCleaningFunctions.cleanDate("2009-08-12T22:15:09-07:00"));
-		assertEquals("2009-08-12", GraphCleaningFunctions.cleanDate("2009-08-12T22:15:09"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26 17:24:37.3186369"));
-		assertEquals("2012-08-03", GraphCleaningFunctions.cleanDate("2012-08-03 18:31:59.257000000"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26 17:24:37.123"));
-		assertEquals("2013-04-01", GraphCleaningFunctions.cleanDate("2013-04-01 22:43"));
-		assertEquals("2013-04-01", GraphCleaningFunctions.cleanDate("2013-04-01 22:43:22"));
-		assertEquals("2014-12-16", GraphCleaningFunctions.cleanDate("2014-12-16 06:20:00 UTC"));
-		assertEquals("2014-12-16", GraphCleaningFunctions.cleanDate("2014-12-16 06:20:00 GMT"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26 05:24:37 PM"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26 13:13:43 +0800"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26 13:13:43 +0800 +08"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26 13:13:44 +09:00"));
-		assertEquals("2012-08-03", GraphCleaningFunctions.cleanDate("2012-08-03 18:31:59.257000000 +0000 UTC"));
-		assertEquals("2015-09-30", GraphCleaningFunctions.cleanDate("2015-09-30 18:48:56.35272715 +0000 UTC"));
-		assertEquals("2015-02-18", GraphCleaningFunctions.cleanDate("2015-02-18 00:12:00 +0000 GMT"));
-		assertEquals("2015-02-18", GraphCleaningFunctions.cleanDate("2015-02-18 00:12:00 +0000 UTC"));
+			GraphCleaningFunctions.doCleanDate("Fri Jul 03 2015 18:04:07 GMT+0100 (GMT Daylight Time)").get());
+		assertEquals("2012-09-17", GraphCleaningFunctions.doCleanDate("September 17, 2012 10:09am").get());
+		assertEquals("2012-09-17", GraphCleaningFunctions.doCleanDate("September 17, 2012 at 10:09am PST-08").get());
+		assertEquals("2012-09-17", GraphCleaningFunctions.doCleanDate("September 17, 2012, 10:10:09").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("October 7, 1970").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("October 7th, 1970").get());
+		assertEquals("2006-02-12", GraphCleaningFunctions.doCleanDate("12 Feb 2006, 19:17").get());
+		assertEquals("2006-02-12", GraphCleaningFunctions.doCleanDate("12 Feb 2006 19:17").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("7 oct 70").get());
+		assertEquals("1970-10-07", GraphCleaningFunctions.doCleanDate("7 oct 1970").get());
+		assertEquals("2013-02-03", GraphCleaningFunctions.doCleanDate("03 February 2013").get());
+		assertEquals("2013-07-01", GraphCleaningFunctions.doCleanDate("1 July 2013").get());
+		assertEquals("2013-02-03", GraphCleaningFunctions.doCleanDate("2013-Feb-03").get());
+		assertEquals("2014-03-31", GraphCleaningFunctions.doCleanDate("3/31/2014").get());
+		assertEquals("2014-03-31", GraphCleaningFunctions.doCleanDate("03/31/2014").get());
+		assertEquals("1971-08-21", GraphCleaningFunctions.doCleanDate("08/21/71").get());
+		assertEquals("1971-01-08", GraphCleaningFunctions.doCleanDate("8/1/71").get());
+		assertEquals("2014-08-04", GraphCleaningFunctions.doCleanDate("4/8/2014 22:05").get());
+		assertEquals("2014-08-04", GraphCleaningFunctions.doCleanDate("04/08/2014 22:05").get());
+		assertEquals("2014-08-04", GraphCleaningFunctions.doCleanDate("4/8/14 22:05").get());
+		assertEquals("2014-02-04", GraphCleaningFunctions.doCleanDate("04/2/2014 03:00:51").get());
+		assertEquals("1965-08-08", GraphCleaningFunctions.doCleanDate("8/8/1965 12:00:00 AM").get());
+		assertEquals("1965-08-08", GraphCleaningFunctions.doCleanDate("8/8/1965 01:00:01 PM").get());
+		assertEquals("1965-08-08", GraphCleaningFunctions.doCleanDate("8/8/1965 01:00 PM").get());
+		assertEquals("1965-08-08", GraphCleaningFunctions.doCleanDate("8/8/1965 1:00 PM").get());
+		assertEquals("1965-08-08", GraphCleaningFunctions.doCleanDate("8/8/1965 12:00 AM").get());
+		assertEquals("2014-02-04", GraphCleaningFunctions.doCleanDate("4/02/2014 03:00:51").get());
+		assertEquals("2012-03-19", GraphCleaningFunctions.doCleanDate("03/19/2012 10:11:59").get());
+		assertEquals("2012-03-19", GraphCleaningFunctions.doCleanDate("03/19/2012 10:11:59.3186369").get());
+		assertEquals("2014-03-31", GraphCleaningFunctions.doCleanDate("2014/3/31").get());
+		assertEquals("2014-03-31", GraphCleaningFunctions.doCleanDate("2014/03/31").get());
+		assertEquals("2014-04-08", GraphCleaningFunctions.doCleanDate("2014/4/8 22:05").get());
+		assertEquals("2014-04-08", GraphCleaningFunctions.doCleanDate("2014/04/08 22:05").get());
+		assertEquals("2014-04-02", GraphCleaningFunctions.doCleanDate("2014/04/2 03:00:51").get());
+		assertEquals("2014-04-02", GraphCleaningFunctions.doCleanDate("2014/4/02 03:00:51").get());
+		assertEquals("2012-03-19", GraphCleaningFunctions.doCleanDate("2012/03/19 10:11:59").get());
+		assertEquals("2012-03-19", GraphCleaningFunctions.doCleanDate("2012/03/19 10:11:59.3186369").get());
+		assertEquals("2014-04-08", GraphCleaningFunctions.doCleanDate("2014年04月08日").get());
+		assertEquals("2006-01-02", GraphCleaningFunctions.doCleanDate("2006-01-02T15:04:05+0000").get());
+		assertEquals("2009-08-13", GraphCleaningFunctions.doCleanDate("2009-08-12T22:15:09-07:00").get());
+		assertEquals("2009-08-12", GraphCleaningFunctions.doCleanDate("2009-08-12T22:15:09").get());
+		assertEquals("2009-08-13", GraphCleaningFunctions.doCleanDate("2009-08-12T22:15:09Z").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26 17:24:37.3186369").get());
+		assertEquals("2012-08-03", GraphCleaningFunctions.doCleanDate("2012-08-03 18:31:59.257000000").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26 17:24:37.123").get());
+		assertEquals("2013-04-01", GraphCleaningFunctions.doCleanDate("2013-04-01 22:43").get());
+		assertEquals("2013-04-01", GraphCleaningFunctions.doCleanDate("2013-04-01 22:43:22").get());
+		assertEquals("2014-12-16", GraphCleaningFunctions.doCleanDate("2014-12-16 06:20:00 UTC").get());
+		assertEquals("2014-12-16", GraphCleaningFunctions.doCleanDate("2014-12-16 06:20:00 GMT").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26 05:24:37 PM").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26 13:13:43 +0800").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26 13:13:43 +0800 +08").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26 13:13:44 +09:00").get());
+		assertEquals("2012-08-03", GraphCleaningFunctions.doCleanDate("2012-08-03 18:31:59.257000000 +0000 UTC").get());
+		assertEquals("2015-09-30", GraphCleaningFunctions.doCleanDate("2015-09-30 18:48:56.35272715 +0000 UTC").get());
+		assertEquals("2015-02-18", GraphCleaningFunctions.doCleanDate("2015-02-18 00:12:00 +0000 GMT").get());
+		assertEquals("2015-02-18", GraphCleaningFunctions.doCleanDate("2015-02-18 00:12:00 +0000 UTC").get());
 		assertEquals(
-			"2015-02-08", GraphCleaningFunctions.cleanDate("2015-02-08 03:02:00 +0300 MSK m=+0.000000001"));
+			"2015-02-08", GraphCleaningFunctions.doCleanDate("2015-02-08 03:02:00 +0300 MSK m=+0.000000001").get());
 		assertEquals(
-			"2015-02-08", GraphCleaningFunctions.cleanDate("2015-02-08 03:02:00.001 +0300 MSK m=+0.000000001"));
-		assertEquals("2017-07-19", GraphCleaningFunctions.cleanDate("2017-07-19 03:21:51+00:00"));
-		assertEquals("2014-04-26", GraphCleaningFunctions.cleanDate("2014-04-26"));
-		assertEquals("2014-04-01", GraphCleaningFunctions.cleanDate("2014-04"));
-		assertEquals("2014-01-01", GraphCleaningFunctions.cleanDate("2014"));
-		assertEquals("2014-05-11", GraphCleaningFunctions.cleanDate("2014-05-11 08:20:13,787"));
-		assertEquals("2014-03-31", GraphCleaningFunctions.cleanDate("3.31.2014"));
-		assertEquals("2014-03-31", GraphCleaningFunctions.cleanDate("03.31.2014"));
-		assertEquals("1971-08-21", GraphCleaningFunctions.cleanDate("08.21.71"));
-		assertEquals("2014-03-01", GraphCleaningFunctions.cleanDate("2014.03"));
-		assertEquals("2014-03-30", GraphCleaningFunctions.cleanDate("2014.03.30"));
-		assertEquals("2014-06-01", GraphCleaningFunctions.cleanDate("20140601"));
-		assertEquals("2014-07-22", GraphCleaningFunctions.cleanDate("20140722105203"));
-		assertEquals("2012-03-19", GraphCleaningFunctions.cleanDate("1332151919"));
-		assertEquals("2013-11-12", GraphCleaningFunctions.cleanDate("1384216367189"));
-		assertEquals("2013-11-12", GraphCleaningFunctions.cleanDate("1384216367111222"));
-		assertEquals("2013-11-12", GraphCleaningFunctions.cleanDate("1384216367111222333"));
+			"2015-02-08", GraphCleaningFunctions.doCleanDate("2015-02-08 03:02:00.001 +0300 MSK m=+0.000000001").get());
+		assertEquals("2017-07-19", GraphCleaningFunctions.doCleanDate("2017-07-19 03:21:51+00:00").get());
+		assertEquals("2014-04-26", GraphCleaningFunctions.doCleanDate("2014-04-26").get());
+		assertEquals("2014-04-01", GraphCleaningFunctions.doCleanDate("2014-04").get());
+		assertEquals("2014-01-01", GraphCleaningFunctions.doCleanDate("2014").get());
+		assertEquals("2014-05-11", GraphCleaningFunctions.doCleanDate("2014-05-11 08:20:13,787").get());
+		assertEquals("2014-03-31", GraphCleaningFunctions.doCleanDate("3.31.2014").get());
+		assertEquals("2014-03-31", GraphCleaningFunctions.doCleanDate("03.31.2014").get());
+		assertEquals("1971-08-21", GraphCleaningFunctions.doCleanDate("08.21.71").get());
+		assertEquals("2014-03-01", GraphCleaningFunctions.doCleanDate("2014.03").get());
+		assertEquals("2014-03-30", GraphCleaningFunctions.doCleanDate("2014.03.30").get());
+		assertEquals("2014-06-01", GraphCleaningFunctions.doCleanDate("20140601").get());
+		assertEquals("2014-07-22", GraphCleaningFunctions.doCleanDate("20140722105203").get());
+		assertEquals("2012-03-19", GraphCleaningFunctions.doCleanDate("1332151919").get());
+		assertEquals("2013-11-12", GraphCleaningFunctions.doCleanDate("1384216367189").get());
+		assertEquals("2013-11-12", GraphCleaningFunctions.doCleanDate("1384216367111222").get());
+		assertEquals("2013-11-12", GraphCleaningFunctions.doCleanDate("1384216367111222333").get());
 
 	}
 
@@ -184,22 +185,6 @@ class OafMapperUtilsTest {
 				.getClassid());
 	}
 
-	@Test
-	void testDelegatedAuthority() throws IOException {
-		Dataset d1 = read("dataset_2.json", Dataset.class);
-		Dataset d2 = read("dataset_delegated.json", Dataset.class);
-
-		assertEquals(1, d2.getCollectedfrom().size());
-		assertTrue(cfId(d2.getCollectedfrom()).contains(ModelConstants.ZENODO_OD_ID));
-
-		Result res = OafMapperUtils.mergeResults(d1, d2);
-
-		assertEquals(d2, res);
-
-		System.out.println(OBJECT_MAPPER.writeValueAsString(res));
-
-	}
-
 	protected HashSet<String> cfId(List<KeyValue> collectedfrom) {
 		return collectedfrom.stream().map(KeyValue::getKey).collect(Collectors.toCollection(HashSet::new));
 	}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/PICCleaningRuleTest.java

@@ -1,19 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import org.junit.jupiter.api.Test;
-
-class PICCleaningRuleTest {
-
-	@Test
-	void testCleaning() {
-		assertEquals("887624982", PICCleaningRule.clean("887624982"));
-		assertEquals("", PICCleaningRule.clean("887 624982"));
-		assertEquals("887624982", PICCleaningRule.clean(" 887624982 "));
-		assertEquals("887624982", PICCleaningRule.clean(" 887624982x "));
-		assertEquals("887624982", PICCleaningRule.clean(" 88762498200 "));
-	}
-
-}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/PmcCleaningRuleTest.java

@@ -1,19 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import org.junit.jupiter.api.Test;
-
-class PmcCleaningRuleTest {
-
-	@Test
-	void testCleaning() {
-		assertEquals("PMC1234", PmcCleaningRule.clean("PMC1234"));
-		assertEquals("PMC1234", PmcCleaningRule.clean(" PMC1234"));
-		assertEquals("PMC12345678", PmcCleaningRule.clean("PMC12345678"));
-		assertEquals("PMC12345678", PmcCleaningRule.clean("PMC123456789"));
-		assertEquals("PMC12345678", PmcCleaningRule.clean("PMC 12345678"));
-	}
-
-}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/PmidCleaningRuleTest.java

@@ -1,24 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import org.junit.jupiter.api.Test;
-
-class PmidCleaningRuleTest {
-
-	@Test
-	void testCleaning() {
-		// leading zeros are removed
-		assertEquals("1234", PmidCleaningRule.clean("01234"));
-		// tolerant to spaces in the middle
-		assertEquals("1234567", PmidCleaningRule.clean("0123 4567"));
-		// stop parsing at first not numerical char
-		assertEquals("123", PmidCleaningRule.clean("0123x4567"));
-		// invalid id leading to empty result
-		assertEquals("", PmidCleaningRule.clean("abc"));
-		// valid id with zeroes in the number
-		assertEquals("20794075", PmidCleaningRule.clean("20794075"));
-	}
-
-}

dhp-common/src/test/java/eu/dnetlib/dhp/schema/oaf/utils/RorCleaningRuleTest.java

@@ -1,17 +0,0 @@
-
-package eu.dnetlib.dhp.schema.oaf.utils;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import org.junit.jupiter.api.Test;
-
-class RorCleaningRuleTest {
-
-	@Test
-	void testCleaning() {
-		assertEquals("https://ror.org/05rpz9w55", RorCleaningRule.clean("https://ror.org/05rpz9w55"));
-		assertEquals("https://ror.org/05rpz9w55", RorCleaningRule.clean("05rpz9w55"));
-		assertEquals("", RorCleaningRule.clean("05rpz9w_55"));
-	}
-
-}

dhp-common/src/test/java/eu/dnetlib/oa/merge/AuthorMergerTest.java

@@ -1,114 +0,0 @@
-
-package eu.dnetlib.oa.merge;
-
-import static org.junit.jupiter.api.Assertions.*;
-
-import java.io.BufferedReader;
-import java.io.InputStreamReader;
-import java.util.List;
-import java.util.Objects;
-
-import org.junit.jupiter.api.Test;
-import org.junit.platform.commons.util.StringUtils;
-
-import com.fasterxml.jackson.core.type.TypeReference;
-import com.fasterxml.jackson.databind.ObjectMapper;
-
-import eu.dnetlib.dhp.oa.merge.AuthorMerger;
-import eu.dnetlib.dhp.schema.oaf.Author;
-
-public class AuthorMergerTest {
-
-	@Test
-	public void testEnrcichAuthor() throws Exception {
-		final ObjectMapper mapper = new ObjectMapper();
-
-		BufferedReader pr = new BufferedReader(new InputStreamReader(
-			Objects
-				.requireNonNull(
-					AuthorMergerTest.class
-						.getResourceAsStream("/eu/dnetlib/dhp/oa/merge/authors_publication_sample.json"))));
-		BufferedReader or = new BufferedReader(new InputStreamReader(
-			Objects
-				.requireNonNull(
-					AuthorMergerTest.class.getResourceAsStream("/eu/dnetlib/dhp/oa/merge/authors_orcid_sample.json"))));
-
-		TypeReference<List<Author>> aclass = new TypeReference<List<Author>>() {
-		};
-		String pubLine;
-
-		int i = 0;
-		while ((pubLine = pr.readLine()) != null) {
-			final String pubId = pubLine;
-			final String MatchPidOrcid = or.readLine();
-			final String pubOrcid = or.readLine();
-
-			final String data = pr.readLine();
-
-			if (StringUtils.isNotBlank(data)) {
-				List<Author> publicationAuthors = mapper.readValue(data, aclass);
-				List<Author> orcidAuthors = mapper.readValue(or.readLine(), aclass);
-				System.out.printf("OAF ID = %s \n", pubId);
-				System.out.printf("ORCID Intersected ID = %s \n", pubOrcid);
-				System.out.printf("OAF Author Size = %d \n", publicationAuthors.size());
-				System.out.printf("Oricd Author Size = %d \n", orcidAuthors.size());
-				System.out.printf("Oricd Matched PID = %s \n", MatchPidOrcid);
-
-				long originalAuthorWithPiD = publicationAuthors
-					.stream()
-					.filter(
-						a -> a.getPid() != null && a
-							.getPid()
-							.stream()
-							.anyMatch(
-								p -> p.getQualifier() != null
-									&& p.getQualifier().getClassid().toLowerCase().contains("orcid")))
-					.count();
-				long start = System.currentTimeMillis();
-
-//                final List<Author> enrichedList = AuthorMerger.enrichOrcid(publicationAuthors, orcidAuthors);
-				final List<Author> enrichedList = AuthorMerger.enrichOrcid(publicationAuthors, orcidAuthors);
-
-				long enrichedAuthorWithPid = enrichedList
-					.stream()
-					.filter(
-						a -> a.getPid() != null && a
-							.getPid()
-							.stream()
-							.anyMatch(
-								p -> p.getQualifier() != null
-									&& p.getQualifier().getClassid().toLowerCase().contains("orcid")))
-					.count();
-
-				long totalTime = (System.currentTimeMillis() - start) / 1000;
-				System.out
-					.printf(
-						"Enriched authors in %d seconds from %d pid to %d pid \n", totalTime, originalAuthorWithPiD,
-						enrichedAuthorWithPid);
-
-				System.out.println("=================");
-			}
-		}
-	}
-
-	@Test
-	public void checkSimilarityTest() {
-		final Author left = new Author();
-		left.setName("Anand");
-		left.setSurname("Rachna");
-		left.setFullname("Anand, Rachna");
-
-		System.out.println(AuthorMerger.normalizeFullName(left.getFullname()));
-
-		final Author right = new Author();
-		right.setName("Rachna");
-		right.setSurname("Anand");
-		right.setFullname("Rachna, Anand");
-//        System.out.println(AuthorMerger.normalize(right.getFullname()));
-		boolean same = AuthorMerger.checkORCIDSimilarity(left, right);
-
-		assertTrue(same);
-
-	}
-
-}

dhp-common/src/test/resources/eu/dnetlib/dhp/schema/oaf/utils/dataset_2.json

@@ -1,140 +1 @@
-{
-  "id": "50|DansKnawCris::0829b5191605bdbea36d6502b8c1ce1g",
-  "resuttype": {"classid": "dataset"},
-  "pid": [
-    {
-      "qualifier": {"classid": "doi"},
-      "value": "10.1016/j.cmet.2011.03.013"
-    },
-    {
-      "qualifier": {"classid": "urn"},
-      "value": "urn:nbn:nl:ui:29-f3ed5f9e-edf6-457e-8848-61b58a4075e2"
-    },
-    {
-      "qualifier": {"classid": "scp-number"},
-      "value": "79953761260"
-    },
-    {
-      "qualifier": {"classid": "pmc"},
-      "value": "21459329"
-    }
-  ],
-  "collectedfrom": [
-    {
-      "key": "10|openaire____::081b82f96300b6a6e3d282bad31cb6e3",
-      "value": "Repository B"
-    }
-  ],
-  "instance": [
-    {
-      "refereed": {
-        "classid": "0000",
-        "classname": "UNKNOWN",
-        "schemeid": "dnet:review_levels",
-        "schemename": "dnet:review_levels"
-      },
-      "hostedby": {
-        "key": "10|opendoar____::358aee4cc897452c00244351e4d91f69",
-        "value": "Zenodo"
-      },
-      "accessright": {
-        "classid": "OPEN",
-        "classname": "Open Access",
-        "schemeid": "dnet:access_modes",
-        "schemename": "dnet:access_modes"
-      },
-      "processingchargecurrency": {
-        "dataInfo": {
-          "provenanceaction": {
-            "classid": "sysimport:crosswalk:datasetarchive",
-            "classname": "Harvested",
-            "schemeid": "dnet:provenanceActions",
-            "schemename": "dnet:provenanceActions"
-          },
-          "deletedbyinference": false,
-          "inferred": false,
-          "inferenceprovenance": "",
-          "invisible": true,
-          "trust": "0.9"
-        },
-        "value": "EUR"
-      },
-      "pid": [
-        {
-          "dataInfo": {
-            "provenanceaction": {
-              "classid": "sysimport:crosswalk:datasetarchive",
-              "classname": "Harvested",
-              "schemeid": "dnet:provenanceActions",
-              "schemename": "dnet:provenanceActions"
-            },
-            "deletedbyinference": false,
-            "inferred": false,
-            "inferenceprovenance": "",
-            "invisible": true,
-            "trust": "0.9"
-          },
-          "qualifier": {
-            "classid": "doi",
-            "classname": "Digital Object Identifier",
-            "schemeid": "dnet:pid_types",
-            "schemename": "dnet:pid_types"
-          },
-          "value": "10.1371/journal.pone.0085605"
-        }
-      ],
-      "distributionlocation": "",
-      "url": ["https://doi.org/10.1371/journal.pone.0085605"],
-      "alternateIdentifier": [
-        {
-          "dataInfo": {
-            "provenanceaction": {
-              "classid": "sysimport:crosswalk:datasetarchive",
-              "classname": "Harvested",
-              "schemeid": "dnet:provenanceActions",
-              "schemename": "dnet:provenanceActions"
-            },
-            "deletedbyinference": false,
-            "inferred": false,
-            "inferenceprovenance": "",
-            "invisible": true,
-            "trust": "0.9"
-          },
-          "qualifier": {
-            "classid": "pmid",
-            "classname": "PubMed ID",
-            "schemeid": "dnet:pid_types",
-            "schemename": "dnet:pid_types"
-          },
-          "value": "24454899.0"
-        }
-      ],
-      "collectedfrom": {
-        "key": "10|openaire____::081b82f96300b6a6e3d282bad31cb6e3",
-        "value": "Repository B"
-      },
-      "processingchargeamount": {
-        "dataInfo": {
-          "provenanceaction": {
-            "classid": "sysimport:crosswalk:datasetarchive",
-            "classname": "Harvested",
-            "schemeid": "dnet:provenanceActions",
-            "schemename": "dnet:provenanceActions"
-          },
-          "deletedbyinference": false,
-          "inferred": false,
-          "inferenceprovenance": "",
-          "invisible": true,
-          "trust": "0.9"
-        },
-        "value": "1022.02"
-      },
-      "instancetype": {
-        "classid": "0004",
-        "classname": "Conference object",
-        "schemeid": "dnet:publication_resource",
-        "schemename": "dnet:publication_resource"
-      }
-    }
-  ]
-}
+{"id":"50|DansKnawCris::0829b5191605bdbea36d6502b8c1ce1g", "resuttype" : { "classid" : "dataset" }, "pid":[{"qualifier":{"classid":"doi"},"value":"10.1016/j.cmet.2011.03.013"},{"qualifier":{"classid":"urn"},"value":"urn:nbn:nl:ui:29-f3ed5f9e-edf6-457e-8848-61b58a4075e2"},{"qualifier":{"classid":"scp-number"},"value":"79953761260"},{"qualifier":{"classid":"pmc"},"value":"21459329"}], "collectedfrom" : [ { "key" : "10|openaire____::081b82f96300b6a6e3d282bad31cb6e3", "value" : "Repository B"} ]}

dhp-common/src/test/resources/eu/dnetlib/dhp/schema/oaf/utils/dataset_delegated.json

@@ -1,140 +0,0 @@
-{
-  "id": "50|DansKnawCris::0829b5191605bdbea36d6502b8c1ce1g",
-  "resuttype": {"classid": "dataset"},
-  "pid": [
-    {
-      "qualifier": {"classid": "doi"},
-      "value": "10.1016/j.cmet.2011.03.013"
-    },
-    {
-      "qualifier": {"classid": "urn"},
-      "value": "urn:nbn:nl:ui:29-f3ed5f9e-edf6-457e-8848-61b58a4075e2"
-    },
-    {
-      "qualifier": {"classid": "scp-number"},
-      "value": "79953761260"
-    },
-    {
-      "qualifier": {"classid": "pmc"},
-      "value": "21459329"
-    }
-  ],
-  "collectedfrom": [
-    {
-      "key": "10|opendoar____::358aee4cc897452c00244351e4d91f69",
-      "value": "Zenodo"
-    }
-  ],
-  "instance": [
-    {
-      "refereed": {
-        "classid": "0000",
-        "classname": "UNKNOWN",
-        "schemeid": "dnet:review_levels",
-        "schemename": "dnet:review_levels"
-      },
-      "hostedby": {
-        "key": "10|opendoar____::358aee4cc897452c00244351e4d91f69",
-        "value": "Zenodo"
-      },
-      "accessright": {
-        "classid": "OPEN",
-        "classname": "Open Access",
-        "schemeid": "dnet:access_modes",
-        "schemename": "dnet:access_modes"
-      },
-      "processingchargecurrency": {
-        "dataInfo": {
-          "provenanceaction": {
-            "classid": "sysimport:crosswalk:datasetarchive",
-            "classname": "Harvested",
-            "schemeid": "dnet:provenanceActions",
-            "schemename": "dnet:provenanceActions"
-          },
-          "deletedbyinference": false,
-          "inferred": false,
-          "inferenceprovenance": "",
-          "invisible": true,
-          "trust": "0.9"
-        },
-        "value": "EUR"
-      },
-      "pid": [
-        {
-          "dataInfo": {
-            "provenanceaction": {
-              "classid": "sysimport:crosswalk:datasetarchive",
-              "classname": "Harvested",
-              "schemeid": "dnet:provenanceActions",
-              "schemename": "dnet:provenanceActions"
-            },
-            "deletedbyinference": false,
-            "inferred": false,
-            "inferenceprovenance": "",
-            "invisible": true,
-            "trust": "0.9"
-          },
-          "qualifier": {
-            "classid": "doi",
-            "classname": "Digital Object Identifier",
-            "schemeid": "dnet:pid_types",
-            "schemename": "dnet:pid_types"
-          },
-          "value": "10.1371/journal.pone.0085605"
-        }
-      ],
-      "distributionlocation": "",
-      "url": ["https://doi.org/10.1371/journal.pone.0085605"],
-      "alternateIdentifier": [
-        {
-          "dataInfo": {
-            "provenanceaction": {
-              "classid": "sysimport:crosswalk:datasetarchive",
-              "classname": "Harvested",
-              "schemeid": "dnet:provenanceActions",
-              "schemename": "dnet:provenanceActions"
-            },
-            "deletedbyinference": false,
-            "inferred": false,
-            "inferenceprovenance": "",
-            "invisible": true,
-            "trust": "0.9"
-          },
-          "qualifier": {
-            "classid": "pmid",
-            "classname": "PubMed ID",
-            "schemeid": "dnet:pid_types",
-            "schemename": "dnet:pid_types"
-          },
-          "value": "24454899.0"
-        }
-      ],
-      "collectedfrom": {
-        "key": "10|opendoar____::358aee4cc897452c00244351e4d91f69",
-        "value": "Zenodo"
-      },
-      "processingchargeamount": {
-        "dataInfo": {
-          "provenanceaction": {
-            "classid": "sysimport:crosswalk:datasetarchive",
-            "classname": "Harvested",
-            "schemeid": "dnet:provenanceActions",
-            "schemename": "dnet:provenanceActions"
-          },
-          "deletedbyinference": false,
-          "inferred": false,
-          "inferenceprovenance": "",
-          "invisible": true,
-          "trust": "0.9"
-        },
-        "value": "1022.02"
-      },
-      "instancetype": {
-        "classid": "0004",
-        "classname": "Conference object",
-        "schemeid": "dnet:publication_resource",
-        "schemename": "dnet:publication_resource"
-      }
-    }
-  ]
-}

dhp-pace-core/pom.xml

@@ -1,110 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
-
-    <modelVersion>4.0.0</modelVersion>
-
-	<parent>
-		<groupId>eu.dnetlib.dhp</groupId>
-		<artifactId>dhp</artifactId>
-		<version>1.2.5-SNAPSHOT</version>
-        <relativePath>../pom.xml</relativePath>
-	</parent>
-
-	<groupId>eu.dnetlib.dhp</groupId>
-	<artifactId>dhp-pace-core</artifactId>
-	<version>1.2.5-SNAPSHOT</version>
-    <packaging>jar</packaging>
-
-	<build>
-		<plugins>
-			<plugin>
-				<groupId>net.alchim31.maven</groupId>
-				<artifactId>scala-maven-plugin</artifactId>
-				<version>${net.alchim31.maven.version}</version>
-				<executions>
-					<execution>
-						<id>scala-compile-first</id>
-						<phase>initialize</phase>
-						<goals>
-							<goal>add-source</goal>
-							<goal>compile</goal>
-						</goals>
-					</execution>
-					<execution>
-						<id>scala-test-compile</id>
-						<phase>process-test-resources</phase>
-						<goals>
-							<goal>testCompile</goal>
-						</goals>
-					</execution>
-				</executions>
-				<configuration>
-					<failOnMultipleScalaVersions>true</failOnMultipleScalaVersions>
-					<scalaCompatVersion>${scala.binary.version}</scalaCompatVersion>
-					<scalaVersion>${scala.version}</scalaVersion>
-				</configuration>
-			</plugin>
-		</plugins>
-
-	</build>
-
-	<dependencies>
-		<dependency>
-			<groupId>edu.cmu</groupId>
-			<artifactId>secondstring</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>com.google.guava</groupId>
-			<artifactId>guava</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>com.google.code.gson</groupId>
-			<artifactId>gson</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.commons</groupId>
-			<artifactId>commons-lang3</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>commons-io</groupId>
-			<artifactId>commons-io</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>org.antlr</groupId>
-			<artifactId>stringtemplate</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>commons-logging</groupId>
-			<artifactId>commons-logging</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>org.reflections</groupId>
-			<artifactId>reflections</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>com.fasterxml.jackson.core</groupId>
-			<artifactId>jackson-databind</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.commons</groupId>
-			<artifactId>commons-math3</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>com.jayway.jsonpath</groupId>
-			<artifactId>json-path</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>com.ibm.icu</groupId>
-			<artifactId>icu4j</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.spark</groupId>
-			<artifactId>spark-core_${scala.binary.version}</artifactId>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.spark</groupId>
-			<artifactId>spark-sql_${scala.binary.version}</artifactId>
-		</dependency>
-	</dependencies>
-
-</project>

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/AbstractClusteringFunction.java

@@ -1,59 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-
-import org.apache.commons.lang3.StringUtils;
-
-import eu.dnetlib.pace.common.AbstractPaceFunctions;
-import eu.dnetlib.pace.config.Config;
-
-public abstract class AbstractClusteringFunction extends AbstractPaceFunctions implements ClusteringFunction {
-
-	protected Map<String, Object> params;
-
-	public AbstractClusteringFunction(final Map<String, Object> params) {
-		this.params = params;
-	}
-
-	protected abstract Collection<String> doApply(Config conf, String s);
-
-	@Override
-	public Collection<String> apply(Config conf, List<String> fields) {
-		return fields
-			.stream()
-			.filter(f -> !f.isEmpty())
-			.map(s -> normalize(s))
-			.map(s -> filterAllStopWords(s))
-			.map(s -> doApply(conf, s))
-			.map(c -> filterBlacklisted(c, ngramBlacklist))
-			.flatMap(c -> c.stream())
-			.filter(StringUtils::isNotBlank)
-			.collect(Collectors.toCollection(HashSet::new));
-	}
-
-	public Map<String, Object> getParams() {
-		return params;
-	}
-
-	protected Integer param(String name) {
-		Object val = params.get(name);
-		if (val == null)
-			return null;
-		if (val instanceof Number) {
-			return ((Number) val).intValue();
-		}
-		return Integer.parseInt(val.toString());
-	}
-
-	protected int paramOrDefault(String name, int i) {
-		Integer res = param(name);
-		if (res == null)
-			res = i;
-		return res;
-	}
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/Acronyms.java

@@ -1,51 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.Map;
-import java.util.Set;
-import java.util.StringTokenizer;
-
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("acronyms")
-public class Acronyms extends AbstractClusteringFunction {
-
-	public Acronyms(Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(Config conf, String s) {
-		return extractAcronyms(s, param("max"), param("minLen"), param("maxLen"));
-	}
-
-	private Set<String> extractAcronyms(final String s, int maxAcronyms, int minLen, int maxLen) {
-
-		final Set<String> acronyms = Sets.newLinkedHashSet();
-
-		for (int i = 0; i < maxAcronyms; i++) {
-
-			final StringTokenizer st = new StringTokenizer(s);
-			final StringBuilder sb = new StringBuilder();
-
-			while (st.hasMoreTokens()) {
-				final String token = st.nextToken();
-				if (sb.length() > maxLen) {
-					break;
-				}
-				if (token.length() > 1 && i < token.length()) {
-					sb.append(token.charAt(i));
-				}
-			}
-			String acronym = sb.toString();
-			if (acronym.length() > minLen) {
-				acronyms.add(acronym);
-			}
-		}
-		return acronyms;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/ClusteringClass.java

@@ -1,14 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.lang.annotation.ElementType;
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
-import java.lang.annotation.Target;
-
-@Retention(RetentionPolicy.RUNTIME)
-@Target(ElementType.TYPE)
-public @interface ClusteringClass {
-
-	public String value();
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/ClusteringFunction.java

@@ -1,16 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import eu.dnetlib.pace.config.Config;
-
-public interface ClusteringFunction {
-
-	public Collection<String> apply(Config config, List<String> fields);
-
-	public Map<String, Object> getParams();
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/ImmutableFieldValue.java

@@ -1,28 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("immutablefieldvalue")
-public class ImmutableFieldValue extends AbstractClusteringFunction {
-
-	public ImmutableFieldValue(final Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, final String s) {
-		final List<String> res = Lists.newArrayList();
-
-		res.add(s);
-
-		return res;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/JSONListClustering.java

@@ -1,69 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-
-import org.apache.commons.lang3.StringUtils;
-
-import com.jayway.jsonpath.Configuration;
-import com.jayway.jsonpath.DocumentContext;
-import com.jayway.jsonpath.JsonPath;
-import com.jayway.jsonpath.Option;
-
-import eu.dnetlib.pace.common.AbstractPaceFunctions;
-import eu.dnetlib.pace.config.Config;
-import eu.dnetlib.pace.util.MapDocumentUtil;
-
-@ClusteringClass("jsonlistclustering")
-public class JSONListClustering extends AbstractPaceFunctions implements ClusteringFunction {
-
-	private Map<String, Object> params;
-
-	public JSONListClustering(Map<String, Object> params) {
-		this.params = params;
-	}
-
-	@Override
-	public Map<String, Object> getParams() {
-		return params;
-	}
-
-	@Override
-	public Collection<String> apply(Config conf, List<String> fields) {
-		return fields
-			.stream()
-			.filter(f -> !f.isEmpty())
-			.map(s -> doApply(conf, s))
-			.filter(StringUtils::isNotBlank)
-			.collect(Collectors.toCollection(HashSet::new));
-	}
-
-	private String doApply(Config conf, String json) {
-		StringBuilder st = new StringBuilder(); // to build the string used for comparisons basing on the jpath into
-		// parameters
-		final DocumentContext documentContext = JsonPath
-			.using(Configuration.defaultConfiguration().addOptions(Option.SUPPRESS_EXCEPTIONS))
-			.parse(json);
-
-		// for each path in the param list
-		for (String key : params.keySet().stream().filter(k -> k.contains("jpath")).collect(Collectors.toList())) {
-			String path = params.get(key).toString();
-			String value = MapDocumentUtil.getJPathString(path, documentContext);
-			if (value == null || value.isEmpty())
-				value = "";
-			st.append(value);
-			st.append(" ");
-		}
-
-		st.setLength(st.length() - 1);
-
-		if (StringUtils.isBlank(st)) {
-			return "1";
-		}
-		return st.toString();
-	}
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/KeywordsClustering.java

@@ -1,54 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.*;
-import java.util.stream.Collectors;
-
-import org.apache.commons.lang3.StringUtils;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("keywordsclustering")
-public class KeywordsClustering extends AbstractClusteringFunction {
-
-	public KeywordsClustering(Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, String s) {
-
-		// takes city codes and keywords codes without duplicates
-		Set<String> keywords = getKeywords(s, conf.translationMap(), paramOrDefault("windowSize", 4));
-		Set<String> cities = getCities(s, paramOrDefault("windowSize", 4));
-
-		// list of combination to return as result
-		final Collection<String> combinations = new LinkedHashSet<String>();
-
-		for (String keyword : keywordsToCodes(keywords, conf.translationMap())) {
-			for (String city : citiesToCodes(cities)) {
-				combinations.add(keyword + "-" + city);
-				if (combinations.size() >= paramOrDefault("max", 2)) {
-					return combinations;
-				}
-			}
-		}
-
-		return combinations;
-	}
-
-	@Override
-	public Collection<String> apply(final Config conf, List<String> fields) {
-		return fields
-			.stream()
-			.filter(f -> !f.isEmpty())
-			.map(KeywordsClustering::cleanup)
-			.map(KeywordsClustering::normalize)
-			.map(s -> filterAllStopWords(s))
-			.map(s -> doApply(conf, s))
-			.map(c -> filterBlacklisted(c, ngramBlacklist))
-			.flatMap(c -> c.stream())
-			.filter(StringUtils::isNotBlank)
-			.collect(Collectors.toCollection(HashSet::new));
-	}
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/LastNameFirstInitial.java

@@ -1,78 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.*;
-import java.util.stream.Collectors;
-
-import org.apache.commons.lang3.StringUtils;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.pace.config.Config;
-import eu.dnetlib.pace.model.Person;
-
-@ClusteringClass("lnfi")
-public class LastNameFirstInitial extends AbstractClusteringFunction {
-
-	private boolean DEFAULT_AGGRESSIVE = true;
-
-	public LastNameFirstInitial(final Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	public Collection<String> apply(Config conf, List<String> fields) {
-		return fields
-			.stream()
-			.filter(f -> !f.isEmpty())
-			.map(LastNameFirstInitial::normalize)
-			.map(s -> doApply(conf, s))
-			.map(c -> filterBlacklisted(c, ngramBlacklist))
-			.flatMap(c -> c.stream())
-			.filter(StringUtils::isNotBlank)
-			.collect(Collectors.toCollection(HashSet::new));
-	}
-
-	public static String normalize(final String s) {
-		return fixAliases(transliterate(nfd(unicodeNormalization(s))))
-			// do not compact the regexes in a single expression, would cause StackOverflowError in case of large input
-			// strings
-			.replaceAll("[^ \\w]+", "")
-			.replaceAll("(\\p{InCombiningDiacriticalMarks})+", "")
-			.replaceAll("(\\p{Punct})+", " ")
-			.replaceAll("(\\d)+", " ")
-			.replaceAll("(\\n)+", " ")
-			.trim();
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, final String s) {
-
-		final List<String> res = Lists.newArrayList();
-
-		final boolean aggressive = (Boolean) (getParams().containsKey("aggressive") ? getParams().get("aggressive")
-			: DEFAULT_AGGRESSIVE);
-
-		Person p = new Person(s, aggressive);
-
-		if (p.isAccurate()) {
-			String lastName = p.getNormalisedSurname().toLowerCase();
-			String firstInitial = p.getNormalisedFirstName().toLowerCase().substring(0, 1);
-
-			res.add(firstInitial.concat(lastName));
-		} else { // is not accurate, meaning it has no defined name and surname
-			List<String> fullname = Arrays.asList(p.getNormalisedFullname().split(" "));
-			if (fullname.size() == 1) {
-				res.add(p.getNormalisedFullname().toLowerCase());
-			} else if (fullname.size() == 2) {
-				res.add(fullname.get(0).substring(0, 1).concat(fullname.get(1)).toLowerCase());
-				res.add(fullname.get(1).substring(0, 1).concat(fullname.get(0)).toLowerCase());
-			} else {
-				res.add(fullname.get(0).substring(0, 1).concat(fullname.get(fullname.size() - 1)).toLowerCase());
-				res.add(fullname.get(fullname.size() - 1).substring(0, 1).concat(fullname.get(0)).toLowerCase());
-			}
-		}
-
-		return res;
-	}
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/LowercaseClustering.java

@@ -1,38 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import org.apache.commons.lang3.StringUtils;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("lowercase")
-public class LowercaseClustering extends AbstractClusteringFunction {
-
-	public LowercaseClustering(final Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	public Collection<String> apply(Config conf, List<String> fields) {
-		Collection<String> c = Sets.newLinkedHashSet();
-		for (String f : fields) {
-			c.addAll(doApply(conf, f));
-		}
-		return c;
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, final String s) {
-		if (StringUtils.isBlank(s)) {
-			return Lists.newArrayList();
-		}
-		return Lists.newArrayList(s.toLowerCase().trim());
-	}
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/NGramUtils.java

@@ -1,24 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Set;
-
-import org.apache.commons.lang3.StringUtils;
-
-import eu.dnetlib.pace.common.AbstractPaceFunctions;
-
-public class NGramUtils extends AbstractPaceFunctions {
-	static private final NGramUtils NGRAMUTILS = new NGramUtils();
-
-	private static final int SIZE = 100;
-
-	private static final Set<String> stopwords = AbstractPaceFunctions
-		.loadFromClasspath("/eu/dnetlib/pace/config/stopwords_en.txt");
-
-	public static String cleanupForOrdering(String s) {
-		return (NGRAMUTILS.filterStopWords(NGRAMUTILS.normalize(s), stopwords) + StringUtils.repeat(" ", SIZE))
-			.substring(0, SIZE)
-			.replaceAll(" ", "");
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/NgramPairs.java

@@ -1,41 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("ngrampairs")
-public class NgramPairs extends Ngrams {
-
-	public NgramPairs(Map<String, Object> params) {
-		super(params, false);
-	}
-
-	public NgramPairs(Map<String, Object> params, boolean sorted) {
-		super(params, sorted);
-	}
-
-	@Override
-	protected Collection<String> doApply(Config conf, String s) {
-		return ngramPairs(Lists.newArrayList(getNgrams(s, param("ngramLen"), param("max") * 2, 1, 2)), param("max"));
-	}
-
-	protected Collection<String> ngramPairs(final List<String> ngrams, int maxNgrams) {
-		Collection<String> res = Lists.newArrayList();
-		int j = 0;
-		for (int i = 0; i < ngrams.size() && res.size() < maxNgrams; i++) {
-			if (++j >= ngrams.size()) {
-				break;
-			}
-			res.add(ngrams.get(i) + ngrams.get(j));
-			// System.out.println("-- " + concatNgrams);
-		}
-		return res;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/Ngrams.java

@@ -1,52 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.*;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("ngrams")
-public class Ngrams extends AbstractClusteringFunction {
-
-	private final boolean sorted;
-
-	public Ngrams(Map<String, Object> params) {
-		this(params, false);
-	}
-
-	public Ngrams(Map<String, Object> params, boolean sorted) {
-		super(params);
-		this.sorted = sorted;
-	}
-
-	@Override
-	protected Collection<String> doApply(Config conf, String s) {
-		return getNgrams(s, param("ngramLen"), param("max"), param("maxPerToken"), param("minNgramLen"));
-	}
-
-	protected Collection<String> getNgrams(String s, int ngramLen, int max, int maxPerToken, int minNgramLen) {
-
-		final Collection<String> ngrams = sorted ? new TreeSet<>() : new LinkedHashSet<String>();
-		final StringTokenizer st = new StringTokenizer(s);
-
-		while (st.hasMoreTokens()) {
-			final String token = st.nextToken();
-			if (!token.isEmpty()) {
-				for (int i = 0; i < maxPerToken && ngramLen + i <= token.length(); i++) {
-					String ngram = token.substring(i, Math.min(ngramLen + i, token.length())).trim();
-
-					if (ngram.length() >= minNgramLen) {
-						ngrams.add(ngram);
-
-						if (ngrams.size() >= max) {
-							return ngrams;
-						}
-					}
-				}
-			}
-		}
-		// System.out.println(ngrams + " n: " + ngrams.size());
-		return ngrams;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/NumAuthorsTitleSuffixPrefixChain.java

@@ -1,113 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.*;
-import java.util.stream.Collectors;
-import java.util.stream.StreamSupport;
-
-import com.google.common.base.Splitter;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("numAuthorsTitleSuffixPrefixChain")
-public class NumAuthorsTitleSuffixPrefixChain extends AbstractClusteringFunction {
-
-	public NumAuthorsTitleSuffixPrefixChain(Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	public Collection<String> apply(Config conf, List<String> fields) {
-
-		try {
-			int num_authors = Math.min(Integer.parseInt(fields.get(0)), 21); // SIZE threshold is 20, +1
-
-			if (num_authors > 0) {
-				return super.apply(conf, fields.subList(1, fields.size()))
-					.stream()
-					.map(s -> num_authors + "-" + s)
-					.collect(Collectors.toList());
-			}
-		} catch (NumberFormatException e) {
-			// missing or null authors array
-		}
-
-		return Collections.emptyList();
-	}
-
-	@Override
-	protected Collection<String> doApply(Config conf, String s) {
-		return suffixPrefixChain(cleanup(s), param("mod"));
-	}
-
-	private Collection<String> suffixPrefixChain(String s, int mod) {
-		// create the list of words from the string (remove short words)
-		List<String> wordsList = Arrays
-			.stream(s.split(" "))
-			.filter(si -> si.length() > 3)
-			.collect(Collectors.toList());
-
-		final int words = wordsList.size();
-		final int letters = s.length();
-
-		// create the prefix: number of words + number of letters/mod
-		String prefix = words / mod + "-";
-
-		return doSuffixPrefixChain(wordsList, prefix);
-
-	}
-
-	private Collection<String> doSuffixPrefixChain(List<String> wordsList, String prefix) {
-
-		Set<String> set = Sets.newLinkedHashSet();
-		switch (wordsList.size()) {
-			case 0:
-				break;
-			case 1:
-				set.add(wordsList.get(0));
-				break;
-			case 2:
-				set
-					.add(
-						prefix +
-							suffix(wordsList.get(0), 3) +
-							prefix(wordsList.get(1), 3));
-
-				set
-					.add(
-						prefix +
-							prefix(wordsList.get(0), 3) +
-							suffix(wordsList.get(1), 3));
-
-				break;
-			default:
-				set
-					.add(
-						prefix +
-							suffix(wordsList.get(0), 3) +
-							prefix(wordsList.get(1), 3) +
-							suffix(wordsList.get(2), 3));
-
-				set
-					.add(
-						prefix +
-							prefix(wordsList.get(0), 3) +
-							suffix(wordsList.get(1), 3) +
-							prefix(wordsList.get(2), 3));
-				break;
-		}
-
-		return set;
-
-	}
-
-	private String suffix(String s, int len) {
-		return s.substring(s.length() - len);
-	}
-
-	private String prefix(String s, int len) {
-		return s.substring(0, len);
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/PersonClustering.java

@@ -1,84 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import org.apache.commons.lang3.StringUtils;
-
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.pace.common.AbstractPaceFunctions;
-import eu.dnetlib.pace.config.Config;
-import eu.dnetlib.pace.model.Person;
-
-@ClusteringClass("personClustering")
-public class PersonClustering extends AbstractPaceFunctions implements ClusteringFunction {
-
-	private Map<String, Object> params;
-
-	private static final int MAX_TOKENS = 5;
-
-	public PersonClustering(final Map<String, Object> params) {
-		this.params = params;
-	}
-
-	@Override
-	public Collection<String> apply(final Config conf, final List<String> fields) {
-		final Set<String> hashes = Sets.newHashSet();
-
-		for (final String f : fields) {
-
-			final Person person = new Person(f, false);
-
-			if (StringUtils.isNotBlank(person.getNormalisedFirstName())
-				&& StringUtils.isNotBlank(person.getNormalisedSurname())) {
-				hashes.add(firstLC(person.getNormalisedFirstName()) + person.getNormalisedSurname().toLowerCase());
-			} else {
-				for (final String token1 : tokens(f, MAX_TOKENS)) {
-					for (final String token2 : tokens(f, MAX_TOKENS)) {
-						if (!token1.equals(token2)) {
-							hashes.add(firstLC(token1) + token2);
-						}
-					}
-				}
-			}
-		}
-
-		return hashes;
-	}
-
-//	@Override
-//	public Collection<String> apply(final List<Field> fields) {
-//		final Set<String> hashes = Sets.newHashSet();
-//
-//		for (final Field f : fields) {
-//
-//			final GTAuthor gta = GTAuthor.fromOafJson(f.stringValue());
-//
-//			final Author a = gta.getAuthor();
-//
-//			if (StringUtils.isNotBlank(a.getFirstname()) && StringUtils.isNotBlank(a.getSecondnames())) {
-//				hashes.add(firstLC(a.getFirstname()) + a.getSecondnames().toLowerCase());
-//			} else {
-//				for (final String token1 : tokens(f.stringValue(), MAX_TOKENS)) {
-//					for (final String token2 : tokens(f.stringValue(), MAX_TOKENS)) {
-//						if (!token1.equals(token2)) {
-//							hashes.add(firstLC(token1) + token2);
-//						}
-//					}
-//				}
-//			}
-//		}
-//
-//		return hashes;
-//	}
-
-	@Override
-	public Map<String, Object> getParams() {
-		return params;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/PersonHash.java

@@ -1,34 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.pace.config.Config;
-import eu.dnetlib.pace.model.Person;
-
-@ClusteringClass("personHash")
-public class PersonHash extends AbstractClusteringFunction {
-
-	private boolean DEFAULT_AGGRESSIVE = false;
-
-	public PersonHash(final Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, final String s) {
-		final List<String> res = Lists.newArrayList();
-
-		final boolean aggressive = (Boolean) (getParams().containsKey("aggressive") ? getParams().get("aggressive")
-			: DEFAULT_AGGRESSIVE);
-
-		res.add(new Person(s, aggressive).hash());
-
-		return res;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/RandomClusteringFunction.java

@@ -1,20 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.Map;
-
-import eu.dnetlib.pace.config.Config;
-
-public class RandomClusteringFunction extends AbstractClusteringFunction {
-
-	public RandomClusteringFunction(Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, String s) {
-		return null;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/SortedNgramPairs.java

@@ -1,34 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-
-import com.google.common.base.Joiner;
-import com.google.common.base.Splitter;
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("sortedngrampairs")
-public class SortedNgramPairs extends NgramPairs {
-
-	public SortedNgramPairs(Map<String, Object> params) {
-		super(params, false);
-	}
-
-	@Override
-	protected Collection<String> doApply(Config conf, String s) {
-
-		final List<String> tokens = Lists.newArrayList(Splitter.on(" ").omitEmptyStrings().trimResults().split(s));
-
-		Collections.sort(tokens);
-
-		return ngramPairs(
-			Lists.newArrayList(getNgrams(Joiner.on(" ").join(tokens), param("ngramLen"), param("max") * 2, 1, 2)),
-			param("max"));
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/SpaceTrimmingFieldValue.java

@@ -1,34 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import org.apache.commons.lang3.RandomStringUtils;
-import org.apache.commons.lang3.StringUtils;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("spacetrimmingfieldvalue")
-public class SpaceTrimmingFieldValue extends AbstractClusteringFunction {
-
-	public SpaceTrimmingFieldValue(final Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(final Config conf, final String s) {
-		final List<String> res = Lists.newArrayList();
-
-		res
-			.add(
-				StringUtils.isBlank(s) ? RandomStringUtils.random(param("randomLength"))
-					: s.toLowerCase().replaceAll("\\s+", ""));
-
-		return res;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/SuffixPrefix.java

@@ -1,42 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.util.Collection;
-import java.util.Map;
-import java.util.Set;
-
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("suffixprefix")
-public class SuffixPrefix extends AbstractClusteringFunction {
-
-	public SuffixPrefix(Map<String, Object> params) {
-		super(params);
-	}
-
-	@Override
-	protected Collection<String> doApply(Config conf, String s) {
-		return suffixPrefix(s, param("len"), param("max"));
-	}
-
-	private Collection<String> suffixPrefix(String s, int len, int max) {
-		final Set<String> bigrams = Sets.newLinkedHashSet();
-		int i = 0;
-		while (++i < s.length() && bigrams.size() < max) {
-			int j = s.indexOf(" ", i);
-
-			int offset = j + len + 1 < s.length() ? j + len + 1 : s.length();
-
-			if (j - len > 0) {
-				String bigram = s.substring(j - len, offset).replaceAll(" ", "").trim();
-				if (bigram.length() >= 4) {
-					bigrams.add(bigram);
-				}
-			}
-		}
-		return bigrams;
-	}
-
-}

dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/UrlClustering.java

@@ -1,52 +0,0 @@
-
-package eu.dnetlib.pace.clustering;
-
-import java.net.MalformedURLException;
-import java.net.URL;
-import java.util.Collection;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-
-import eu.dnetlib.pace.common.AbstractPaceFunctions;
-import eu.dnetlib.pace.config.Config;
-
-@ClusteringClass("urlclustering")
-public class UrlClustering extends AbstractPaceFunctions implements ClusteringFunction {
-
-	protected Map<String, Object> params;
-
-	public UrlClustering(final Map<String, Object> params) {
-		this.params = params;
-	}
-
-	@Override
-	public Map<String, Object> getParams() {
-		return params;
-	}
-
-	@Override
-	public Collection<String> apply(final Config conf, List<String> fields) {
-		try {
-			return fields
-				.stream()
-				.filter(f -> !f.isEmpty())
-				.map(this::asUrl)
-				.map(URL::getHost)
-				.collect(Collectors.toCollection(HashSet::new));
-		} catch (IllegalStateException e) {
-			return new HashSet<>();
-		}
-	}
-
-	private URL asUrl(String value) {
-		try {
-			return new URL(value);
-		} catch (MalformedURLException e) {
-			// should not happen as checked by pace typing
-			throw new IllegalStateException("invalid URL: " + value);
-		}
-	}
-
-}