Optimizations for the Openorgs Dedup: normalization and inference of strings and implementation of new general-purpose comparators #455

michele.debonis · 2024-07-10T09:46:10+02:00

michele.debonis commented

2024-07-10 09:46:10 +02:00

This PR implements various changes in the Openorgs Deduplication:

inference of attributes starting from a field (e.g. country inference when UNKNOWN, city and keyword inference)
normalization moved from the comparator to the pre-processing to save computation time
addition of new dedup flags to drive the inference and the normalization
update of the configuration for the openorgs dedup

This PR implements various changes in the Openorgs Deduplication: - inference of attributes starting from a field (e.g. country inference when UNKNOWN, city and keyword inference) - normalization moved from the comparator to the pre-processing to save computation time - addition of new dedup flags to drive the inference and the normalization - update of the configuration for the openorgs dedup

michele.debonis added 1 commit 2024-07-10 09:46:11 +02:00

2a36ccb997 optimization of normalization stage in openorgs workflow, implementation of new comparators replacing older versions, openorgs configuration update, addition of inference flag in model definition, new test classes

claudio.atzori merged commit 6665976604 into beta

2024-07-17 10:25:21 +02:00

claudio.atzori referenced this issue from a commit

2024-07-17 10:25:22 +02:00

Merge pull request 'Optimizations for the Openorgs Dedup: normalization and inference of strings and implementation of new general-purpose comparators' (#455) from openorgs_optimization into beta

Sign in to join this conversation.

No reviewers