Improvements and refactoring in Dedup #367

SparkCreateSimRels:

Create dedup blocks from the complete queue of records matching cluster key instead of truncating the results
Clean titles once before clustering and similarity comparisons
Added support for filtered fields in model
Added support for sorting List fields in model
Added new JSONListClustering and numAuthorsTitleSuffixPrefixChain clustering functions
Added new maxLengthMatch comparator function
Use reduced complexity Levenshtein with threshold in levensteinTitle
Use reduced complexity AuthorsMatch with threshold for early-quit
Use incremental Connected Component to decrease comparisons in similarity match in BlockProcessor
Use new clusterings configuration in Dedup tests

SparkWhitelistSimRels:

use left semi join for clarity and performance

SparkCreateMergeRels:

Use new connected component algorithm that converge faster than Spark GraphX provided algorithm
Refactored to use Windowing sorting rather than groupBy to reduce memory pressure
Use historical pivot table to generate singleton rels, merged rels and keep continuity with dedup records used in the past
Comparator for pivot record selection now uses "next week" as filler for missing or incorrect date instead of "2000-01-01"
Changed algorithm for generation of ids of type dedup_wf_001 to avoid collisions

DedupRecordFactory:

use reduceGroups instead of mapGroups to decrease memory pressure

SparkCreateSimRels: - Create dedup blocks from the complete queue of records matching cluster key instead of truncating the results - Clean titles once before clustering and similarity comparisons - Added support for filtered fields in model - Added support for sorting List fields in model - Added new JSONListClustering and numAuthorsTitleSuffixPrefixChain clustering functions - Added new maxLengthMatch comparator function - Use reduced complexity Levenshtein with threshold in levensteinTitle - Use reduced complexity AuthorsMatch with threshold for early-quit - Use incremental Connected Component to decrease comparisons in similarity match in BlockProcessor - Use new clusterings configuration in Dedup tests SparkWhitelistSimRels: - use left semi join for clarity and performance SparkCreateMergeRels: - Use new connected component algorithm that converge faster than Spark GraphX provided algorithm - Refactored to use Windowing sorting rather than groupBy to reduce memory pressure - Use historical pivot table to generate singleton rels, merged rels and keep continuity with dedup records used in the past - Comparator for pivot record selection now uses "next week" as filler for missing or incorrect date instead of "2000-01-01" - Changed algorithm for generation of ids of type dedup_wf_001 to avoid collisions DedupRecordFactory: - use reduceGroups instead of mapGroups to decrease memory pressure

Labels Milestones

Improvements and refactoring in Dedup #367

Step 1:

Step 2: