dnet-hadoop/dhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/ClusteringFunction.java


package eu.dnetlib.pace.clustering;

import java.util.Collection;
import java.util.List;
import java.util.Map;

import eu.dnetlib.pace.config.Config;

public interface ClusteringFunction {

	public Collection<String> apply(Config config, List<String> fields);

	public Map<String, Object> getParams();

}
New sources formatted by maven plugin 2023-07-06 10:28:53 +02:00
added d-net pace core module and ignored target folder 2018-10-02 10:37:54 +02:00			`package eu.dnetlib.pace.clustering;`

			`import java.util.Collection;`
			`import java.util.List;`
			`import java.util.Map;`

translation map moved in json configuration, support for synonyms added in the configuration, now the configuration is argument of conditions, distancealgos and clusteringfunctions 2019-10-08 14:53:52 +02:00			`import eu.dnetlib.pace.config.Config;`
added d-net pace core module and ignored target folder 2018-10-02 10:37:54 +02:00
			`public interface ClusteringFunction {`
New sources formatted by maven plugin 2023-07-06 10:28:53 +02:00
Refactor Dedup process to use Spark Dataframe API and intermediate representation with Row interface JsonPath cache contention fixed by using a ConcurrentHashMap Blacklist filtering performance improvement Minor performance improvements when evaluating similarity Sorting in clustered elements is deterministic (by ordering and identity field, instead of ordering field only) 2023-07-18 11:38:56 +02:00			`public Collection<String> apply(Config config, List<String> fields);`
New sources formatted by maven plugin 2023-07-06 10:28:53 +02:00
SparkCreateSimRels: - Create dedup blocks from the complete queue of records matching cluster key instead of truncating the results - Clean titles once before clustering and similarity comparisons - Added support for filtered fields in model - Added support for sorting List fields in model - Added new JSONListClustering and numAuthorsTitleSuffixPrefixChain clustering functions - Added new maxLengthMatch comparator function - Use reduced complexity Levenshtein with threshold in levensteinTitle - Use reduced complexity AuthorsMatch with threshold early-quit - Use incremental Connected Component to decrease comparisons in similarity match in BlockProcessor - Use new clusterings configuration in Dedup tests SparkWhitelistSimRels: use left semi join for clarity and performance SparkCreateMergeRels: - Use new connected component algorithm that converge faster than Spark GraphX provided algorithm - Refactored to use Windowing sorting rather than groupBy to reduce memory pressure - Use historical pivot table to generate singleton rels, merged rels and keep continuity with dedupIds used in the past - Comparator for pivot record selection now uses "tomorrow" as filler for missing or incorrect date instead of "2000-01-01" - Changed generation of ids of type dedup_wf_001 to avoid collisions DedupRecordFactory: use reduceGroups instead of mapGroups to decrease memory pressure 2023-10-02 09:25:12 +02:00			`public Map<String, Object> getParams();`
added d-net pace core module and ignored target folder 2018-10-02 10:37:54 +02:00
			`}`