Giambattista Bloisi
02636e802c
- Create dedup blocks from the complete queue of records matching cluster key instead of truncating the results - Clean titles once before clustering and similarity comparisons - Added support for filtered fields in model - Added support for sorting List fields in model - Added new JSONListClustering and numAuthorsTitleSuffixPrefixChain clustering functions - Added new maxLengthMatch comparator function - Use reduced complexity Levenshtein with threshold in levensteinTitle - Use reduced complexity AuthorsMatch with threshold early-quit - Use incremental Connected Component to decrease comparisons in similarity match in BlockProcessor - Use new clusterings configuration in Dedup tests SparkWhitelistSimRels: use left semi join for clarity and performance SparkCreateMergeRels: - Use new connected component algorithm that converge faster than Spark GraphX provided algorithm - Refactored to use Windowing sorting rather than groupBy to reduce memory pressure - Use historical pivot table to generate singleton rels, merged rels and keep continuity with dedupIds used in the past - Comparator for pivot record selection now uses "tomorrow" as filler for missing or incorrect date instead of "2000-01-01" - Changed generation of ids of type dedup_wf_001 to avoid collisions DedupRecordFactory: use reduceGroups instead of mapGroups to decrease memory pressure |
||
|---|---|---|
| .. | ||
| consistency/oozie_app | ||
| openorgs/oozie_app | ||
| scan/oozie_app | ||
| statistics/oozie_app | ||
| copyOpenorgsMergeRels_parameters.json | ||
| copyOpenorgs_parameters.json | ||
| createBlockStats_parameters.json | ||
| createCC_parameters.json | ||
| createDedupRecord_parameters.json | ||
| createSimRels_parameters.json | ||
| prepareNewOrgs_parameters.json | ||
| prepareOrgRels_parameters.json | ||
| propagateRelation_parameters.json | ||
| updateEntity_parameters.json | ||
| updateOpenorgsJob_parameters.json | ||
| whitelistSimRels_parameters.json | ||