Import dnet-pace-core module in this project and use it after renaming to dhp-pace-core #319

giambattista.bloisi · 2023-07-06T10:33:47+02:00

2023-07-06 10:33:47 +02:00

Full commit history has been imported
pom.xml files have been changed to use the new module and to build it
new code has been reformatted by the maven

Full commit history has been imported pom.xml files have been changed to use the new module and to build it new code has been reformatted by the maven

giambattista.bloisi added 198 commits 2023-07-06 10:33:48 +02:00

476c3d7b07 added d-net pace core module and ignored target folder

d1c73bcf90 Added First Implementation of Spark Test

1bb5c26e6d Added FSpark Implementation of dedup

72ebf7c0f3 update of the spark test

1b46966383 updated maven project structure

5aec80345f [maven-release-plugin] prepare release dnet-dedup-3.0.0

5a48937ae1 [maven-release-plugin] prepare for next development iteration

4f29124607 [maven-release-plugin] rollback the release of dnet-dedup-3.0.0

82d5dce114 [maven-release-plugin] prepare release dnet-dedup-3.0.0

ed14476b06 [maven-release-plugin] rollback the release of dnet-dedup-3.0.0

63815be2d6 [maven-release-plugin] prepare release dnet-dedup-3.0.0

3197f26691 [maven-release-plugin] prepare for next development iteration

4d379c2227 revised PidMatch implementation, cleanup

1cbbc3f15a update in the discovery of clustering, conditions and distance functions (annotated with custom annotations)

3cf3dc1934 modification in the initialization of clustering functions, distance algos and conditions.

8b4762bf54 implementation of the toString methonds changed: from Gson to Jackson

ea36007d1f DedupConf parsed using Jackson library

dc41b76643 serialization test added. useless getter methods ignored by json serialization

e296f7a81c added DiffPatchMatch utility. Resumed commented tests!

748189af10 implementation of JaroWinklerNormalizedName, addition of various stopwords in different languages and configuration test

4c8485d0bb deleted useless imports

b247a86e69 configuration files changed: dedupRun instead of run, assertion updated in tests

e1c69ad24e [maven-release-plugin] prepare release dnet-dedup-3.0.1

46cee13596 [maven-release-plugin] prepare for next development iteration

6d09041288 [maven-release-plugin] rollback the release of dnet-dedup-3.0.1

f283d58a6e [maven-release-plugin] prepare release dnet-dedup-3.0.1

851ae5eec3 [maven-release-plugin] rollback the release of dnet-dedup-3.0.1

8cc2517f5d [maven-release-plugin] prepare release dnet-dedup-3.0.1

3dacba37ea [maven-release-plugin] prepare for next development iteration

b7bc7f0401 getting rid of spark libs from dnet-pace-core

e247218987 [maven-release-plugin] prepare release dnet-dedup-3.0.2

2a509b18fa [maven-release-plugin] prepare for next development iteration

b5062f5429 configuration file updated, addition of condition on domain

72a9b3139e Merge branch 'master' of https://github.com/dnet-team/dnet-dedup

df4b871c8b [maven-release-plugin] prepare release dnet-dedup-3.0.3

f7616c7a8a [maven-release-plugin] prepare for next development iteration

478ad72cb8 propagate exceptions in case of serialization errors, removed configuration pretty printing, removed unused class ScoredResult

59bab8dba4 less verbose logging

399e4bc80f default (empty) configuration should be aligned with the updated model

35ab630493 [maven-release-plugin] prepare release dnet-dedup-3.0.4

55a9b4f501 [maven-release-plugin] prepare for next development iteration

3d4372ced9 addition of cities check

0dfb2ea600 added distance function fot software titles

397554130c [maven-release-plugin] prepare release dnet-dedup-3.0.5

394fcafd41 [maven-release-plugin] prepare for next development iteration

c2d4cb3ba6 added new properties to FieldDef (size, length) to limit the information mapped onto each MapDocument

9869dff1d2 [maven-release-plugin] prepare release dnet-dedup-3.0.6

c5979ffe18 [maven-release-plugin] prepare for next development iteration

f1c68d8ba3 apply limits (length, size) to pace Fields

39613dbbd6 implementation of the decisional tree, addition of the dnet-openaire-data-protos module, definition of the person proto, blockprocessor and paceconfig modified with addition of support for the tree processing

7a8d28991f implementation of the decision tree for the deduplication of the authors, implementation of multiple comparators to be used in a tree node and definition of the proto for person entity

0735f3a822 implementation of the test classes and minor changes

968cd47436 replace existing attributes when loading default configuration

7d8e62d4cc [maven-release-plugin] prepare release dnet-dedup-3.0.7

ccb7e83196 [maven-release-plugin] prepare for next development iteration

470c4b0f20 default configuration includes configurationId

722431dde1 [maven-release-plugin] prepare release dnet-dedup-3.0.8

f2394fcd9f [maven-release-plugin] prepare for next development iteration

662448e584 update of the comparator for legalnames of organizations

4bd5a9beee minor changes

7e7018c51f addition of a sparktester test, implementation of 2 different classes for testing in dnet-dedup-test module, addition of new terms in the vocabulary and change in the implementation of the JaroWinklerNormalizedName comparator

1d29bae47c branch cities merged into master

a1ff4daa7f [maven-release-plugin] prepare release dnet-dedup-3.0.9

15fb9eb883 [maven-release-plugin] prepare for next development iteration

3e61a90c8f [maven-release-plugin] rollback the release of dnet-dedup-3.0.9

98c179c8fb [maven-release-plugin] prepare release dnet-dedup-3.0.9

f7a3bdf3f8 [maven-release-plugin] prepare for next development iteration

e8db8f2abb implementation of the integration test, addition of document blocks to group entities after clustering

54e4d0af04 exact match condition gives undefined if a field is missing, ignoremissing semantics changed: now performs the comparison in any case if =true, if false gives -1 in case of missing

e3f86b92c8 cleanup

f2bc665403 avoid to divide by zero: in case of missing values, return undefined response

eb5ce312a3 [maven-release-plugin] prepare release dnet-dedup-3.0.10

ff4956def9 [maven-release-plugin] prepare for next development iteration

15d7b584f3 optimized classpath resolvers

85126c59f7 [maven-release-plugin] prepare release dnet-dedup-3.0.11

2dcffb965f [maven-release-plugin] prepare for next development iteration

15bec5e876 addition of doi normalization in PidMatch comparator, addition of keywordsclustering (clustering based on terms in the translation maps for the organizations), minor changes

7065110a21 [maven-release-plugin] prepare release dnet-dedup-3.0.12

ba50aa8654 [maven-release-plugin] prepare for next development iteration

f5de20a508 [maven-release-plugin] rollback the release of dnet-dedup-3.0.12

c6baeb93d4 [maven-release-plugin] prepare release dnet-dedup-3.0.12

a69022617d [maven-release-plugin] prepare for next development iteration

3c6f8d1e44 bug fixing in the keywordsclustering class

fb5e38db26 Merge branch 'master' of https://github.com/dnet-team/dnet-dedup

c04d2232c2 [maven-release-plugin] prepare release dnet-dedup-3.0.13

6cb846331a [maven-release-plugin] prepare for next development iteration

a85576c27e restyling of the JaroWinklerNormalizedName comparator, now it is optimized. Addition of some translations in the translation maps, addition of a clustering based on keywords in organizations legalnames

cffb712a99 Merge branch 'master' of https://github.com/dnet-team/dnet-dedup

4502b44337 addition of the BlockUtils class for meta-blocking, implementation of a new local test with edge filtering example

8c867101ef addition of a fixSpecial function to address the problem with special character in organization names, addition of new terms in translation maps

f2136e1024 code refactoring: useless module removed

a5c5d2f01b implementation of the decision tree. It takes place of the distance algos, necessaryConditions and sufficientConditions are still there. The model contains only path, type and name of the field. ignoreMissing is still in the model because it is used by the conditions.

d71dae5fd2 implementation of the conditions in tree nodes. get rid of the conditions part of the configuration

afd2b398d5 optimize imports

aed81e4cfa translation map updated

74c6462b49 updated translation map and some tests

254eb46809 [maven-release-plugin] prepare release dnet-dedup-3.0.14

07355d2811 [maven-release-plugin] prepare for next development iteration

26b383fea2 translation map moved in json configuration, support for synonyms added in the configuration, now the configuration is argument of conditions, distancealgos and clusteringfunctions

50b7a12b3f normalization of the term in the translation map added

5499ca17c3 minor changes

c9863debfa minor changes and configuration updates (synonym field added)

5f249fd56c minor changes

285ec3ca17 release rollback

cf93515d94 [maven-release-plugin] prepare release dnet-dedup-3.0.15

13f93e6055 Revert "[maven-release-plugin] prepare release dnet-dedup-3.0.15"

This reverts commit cf93515d94.

ced4bcdd59 [maven-release-plugin] prepare release dnet-dedup-3.0.15

075f741d28 [maven-release-plugin] prepare for next development iteration

1beb776691 minor changes

30a873265f put the last modification of the master branch into the tree2. Addition of the configuration as parameter of the comparator. This is to allow the comparator to access it

0973899865 code cleaning, distribution of the classes in packages and implementation of the new configuration

c687956371 code cleaning and implementation of the TreeDedup + minor changes

ddd40540aa jarowinklernormalizedname splitted in 3 different comparators: citymatch, keywordmatch and jarowinkler. Implementation of the TreeStatistic support functions

8c0d346005 the param map has been updated: now it accepts string parameters

d2278fe358 minor change in the citymatch

f791730330 addition of one term to the translation maps in the configurations

49f9beb4a8 implementation of romansmatch and re-implementation of the getNumber function. New terms in the translation map and update of the configuration

16c670a5d5 Improved deduplication

35008fdbf9 fix stuff

5c01ae4c92 merged JqMapping branch into tree2

84aaa65501 implementation of new json comparator and update of the publication configuration

d924f28b93 fixed wrong use of jspath

20fcfe6328 implementation of new aggregation in the tree node processing

b21b1b8f61 implementation of new aggregation in the tree node processing

b3748b8d77 minor changes

4dce785375 update in the implementation of the tree: addition of new logic aggregations and statistics

5c8f6febee minor changes in comparators

46727f5c76 upgraded maven version of commons-lang

24e2ab9092 [maven-release-plugin] prepare release dnet-dedup-4.0.0

8e4211708e [maven-release-plugin] prepare for next development iteration

6e9b27f37d implementation of the mechanism to truncate the string and the lists

4365ba41c9 [maven-release-plugin] prepare release dnet-dedup-4.0.1

48c094f599 [maven-release-plugin] prepare for next development iteration

411d1cc24f implementation of the test for the dedup and addition of new support classes

f933fd33e0 implemented new function for clustering

c72055f543 [maven-release-plugin] prepare release dnet-dedup-4.0.2

7aa2001a8b [maven-release-plugin] prepare for next development iteration

244a037a90 implementation of a class to test the clustering functions

d2861950ac wordssuffixprefix: adjust the token length according to the number of words; removed maven release temporary files

92aadbfc7b [maven-release-plugin] prepare release dnet-dedup-4.0.3

b79ea97107 Revert "wordssuffixprefix: adjust the token length according to the number of words; removed maven release temporary files"

This reverts commit d2861950ac.

51d91fa520 wordssuffixprefix: adjust the token length according to the number of words; removed maven release temporary files

ec97983ce1 reverted to 4.0.3-SNAPSHOT

6c98d4c436 [maven-release-plugin] prepare release dnet-dedup-4.0.3

ba493f9ab8 [maven-release-plugin] rollback the release of dnet-dedup-4.0.3

9efcd8e245 Revert "reverted to 4.0.3-SNAPSHOT"

This reverts commit ec97983ce1.

9f20f23870 Revert "wordssuffixprefix: adjust the token length according to the number of words; removed maven release temporary files"

This reverts commit 51d91fa520.

14611ea450 reverted to 4.0.3-SNAPSHOT

7cc3742a26 removed maven release.property

ff2c8eba12 [maven-release-plugin] prepare release dnet-dedup-4.0.3

eaf7defe0c [maven-release-plugin] prepare for next development iteration

07ab904d60 implementation of the clustering function for the suffixprefix chain

840fe8f4d3 [maven-release-plugin] prepare release dnet-dedup-4.0.4

a5a3ea24f8 [maven-release-plugin] prepare for next development iteration

a109ebe287 fixed NPE

19c3c90d7b fixed error in the block processor: entities with orderField=null were not considered

dd34e371d7 fixed error in the treeprocessor. it used th=-1 as default value, now it use th=1

2a01022712 minor changes

05a03d97cd [maven-release-plugin] prepare release dnet-dedup-4.0.5

72116446ec [maven-release-plugin] prepare for next development iteration

ed0d5d3e1d implementation of the wf to dedup entities, addition of the module to run the wf on the cluster

3a7336157b [maven-release-plugin] prepare release dnet-dedup-4.0.6

58d013e24f [maven-release-plugin] prepare for next development iteration

4988e9f80d implementation of cross comparison for different fields, addition of clustering mechanism to collapse keys from different clustering functions on the same cluster

c6266242e3 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-dedup

4bce4f2e8e minor change: version updated

f33a18ca9d [maven-release-plugin] prepare release dnet-dedup-4.1.7

1144d50a11 [maven-release-plugin] prepare for next development iteration

fbb1b66bfb dedup test implementation & graph drawing tools

8f1db32921 implementation of the instance type comparator and its tests

a224bf70a4 implementation of new comparators for publication dedup configuration update

eb48d31ea6 [maven-release-plugin] prepare release dnet-dedup-4.1.8

cea8440153 [maven-release-plugin] prepare for next development iteration

2f1ba56f61 bug fix in the authormatch comparator, implementation of tests

003bcf1699 [maven-release-plugin] prepare release dnet-dedup-4.1.9

1ff6a3dc11 [maven-release-plugin] prepare for next development iteration

2ff97781d2 minor change

620e35db28 [maven-release-plugin] prepare release dnet-dedup-4.1.10

8f07f0c537 [maven-release-plugin] prepare for next development iteration

699612dd17 implementation of the size threshold on authors list match

3bc07c5881 bug fix in the AuthorMatch, implementation of the concat function in the model creation with jpath query

a5a6054039 [maven-release-plugin] prepare release dnet-dedup-4.1.11

ac9708e31b [maven-release-plugin] prepare for next development iteration

a965233dd0 bug fix in the normalization of a legalname, city map updated and transliteration support added

bd919ac98d [maven-release-plugin] prepare release dnet-dedup-4.1.12

10172553ab [maven-release-plugin] prepare for next development iteration

97a32faf9b test implementation for the new fdup version

9ddd24ba36 implementation of comparators and clustering function for the author deduplication

9fee2ed611 minor changes

14f6346676 implementation of the new software configuration

6a6c266dde implementation of author dedup configuration and lnfi clustering function

297eb207a5 minor change in the author match which now can compute count and percentage

cb595c87bb implementation of the support for authors deduplication: cosinesimilarity comparator and double array json parser

b0ade43608 Precompile blacklists patterns before evaluating clustering criteria

Enable Junit 5 tests in maven builds
Make path comparisons platform-independent
Read String resource files assuming they are encoded in UTF-8
Fix a few test conditions

6210f6ee48 Merge pull request 'Precompile blacklists patterns before evaluating clustering criteria' (#1 ) from optimized-clustering into master