D-Net
/
dnet-dedup
5
1
Fork 0
Code Issues Pull Requests 1 Releases Wiki Activity

modification of the README

This commit is contained in:
Michele De Bonis 2018-12-20 10:50:05 +01:00
parent 6e9bf11e2d
commit 2be03ecce9
1 changed files with 15 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
README.md
View File

@ -7,24 +7,26 @@ The decision tree has to be defined into the json configuration. The field decis
<String nodeName, TreeNodeDef treeNodeDef>: the nodeName is the key, the treeNodeDef contains the definition of the node.
In particular the TreeNodeDef contains:
- List<FieldConf> : list of fields processed by the node. Each field is associated to:
- field: name of the field
- comparator: name of the comparator to use for that particular field, it produces a similarity score, -1 if the comparison is not possible (missing field or few informations)
*Each FieldConf contains a comparator name which has to be defined. It is sufficient to implement the Comparator interface that exposes a "compare" method returning the similarity score. The new comparator must be annotated with @ComparatorClass("name") specifying the name used by the FieldConf to access to the right comparator.*
- weight: weight to assign to the similarity score of that comparator when aggregating
- params: list of parameters for the comparator
- threshold: this threshold is applied to the resulting similarity score of the particular treeNode.
if score>=th --- positive result
if score==-1 --- undefined result
if score<\th --- negative result
- aggregation: defines the type of aggregation to apply to the similarity scores of the fields in the list of fields
- List<FieldConf> : list of fields processed by the node. Each field is associated to:
- field: name of the field
- comparator: name of the comparator to use for that particular field, it produces a similarity score, -1 if the comparison is not possible (missing field or few informations)
*Each FieldConf contains a comparator name which has to be defined. It is sufficient to implement the Comparator interface that exposes a "compare" method returning the similarity score. The new comparator must be annotated with @ComparatorClass("name") specifying the name used by the FieldConf to access to the right comparator.*
- weight: weight to assign to the similarity score of that comparator when aggregating
- params: list of parameters for the comparator
- threshold: this threshold is applied to the resulting similarity score of the particular treeNode.
```
if score>=th --- positive result
if score==-1 --- undefined result
if score<\th --- negative result
```
- aggregation: defines the type of aggregation to apply to the similarity scores of the fields in the list of fields
AVG(average), MAX, MIN, SUM
e.g. the similarity scores are multiplied with the weight and then the defined aggregation is applied
- arcs: define the next node of the tree depending on the result
- arcs: define the next node of the tree depending on the result
positive: specifies the key of the next node in case of positive result
negative: specifies the key of the next node in case of negative result
undefined: specifies the key of the next node in case of undefined result
- ignoreMissing: defines the behavior of the treeNode in case of a missing field
- ignoreMissing: defines the behavior of the treeNode in case of a missing field
e.g. if a comparator on a particular field produces an undefined result (-1), if ignoreMissing=true that field is simply ignored, otherwise the entire treeNode score is considered to be -1
In order to make the decision tree work, the BlockProcessor has been modified with the following changes: