miconis
|
5dfb66b0fa
|
minor changes
|
2021-03-25 10:29:34 +01:00 |
|
miconis
|
348b0ef921
|
bug fix, implementation of the workflow for the creation of raw_organizations (openorgs dedup), addition of the pid lists to the openorgs postgres db
|
2021-03-24 15:51:27 +01:00 |
|
miconis
|
98854b0124
|
minor changes
|
2021-03-19 16:57:40 +01:00 |
|
miconis
|
1a85020572
|
bug fix in graph-mapper, changes in the implementation of the openorgs wf to create relations and populate openorgs db
|
2021-02-26 10:19:28 +01:00 |
|
miconis
|
4b2124a18e
|
implementation of the openorgs wfs, implementation of the raw_all wf to migrate openorgs db entities
|
2021-02-10 11:51:50 +01:00 |
|
miconis
|
8fea29177c
|
refactoring, minor changes and implementation of the wf for openorgs with integration of organization phases into the scan wf
|
2021-01-18 16:48:08 +01:00 |
|
miconis
|
1e1aab83e3
|
implementation of the raw wf for openorgs: still not complete, some functionalities are missing
|
2020-12-21 11:58:21 +01:00 |
 Sandro La Bruzzo
|
734934e2eb
|
fixed error on empty intersection with publication and relation on export to OAF
|
2020-10-08 17:29:29 +02:00 |
|
Sandro La Bruzzo
|
eec418cd26
|
moved AuthoreMerger into dhp-common
|
2020-10-08 10:33:55 +02:00 |
 Claudio Atzori
|
de72b1c859
|
cleanup
|
2020-07-20 09:59:11 +02:00 |
|
Claudio Atzori
|
805de4eca1
|
fix: filter the blocks with size = 1
|
2020-07-16 10:11:32 +02:00 |
|
Claudio Atzori
|
b90389bac4
|
code formatting
|
2020-07-15 11:24:48 +02:00 |
|
Claudio Atzori
|
4e6f46e8fa
|
filter blocks with one record only
|
2020-07-15 11:22:20 +02:00 |
|
Claudio Atzori
|
06def0c0cb
|
SparkBlockStats allows to repartition the input rdd via the numPartitions workflow parameter
|
2020-07-13 20:09:06 +02:00 |
|
miconis
|
b52c246aed
|
merge done
|
2020-07-13 19:57:02 +02:00 |
|
miconis
|
b8a45041fd
|
minor changes
|
2020-07-13 19:53:18 +02:00 |
|
Claudio Atzori
|
66f9f6d323
|
adjusted parameters for the dedup stats workflow
|
2020-07-13 19:26:46 +02:00 |
|
miconis
|
03ecfa5ebd
|
implementation of the test class for the new block stats spark action
|
2020-07-13 18:48:23 +02:00 |
|
miconis
|
10e08ccf45
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-13 18:22:45 +02:00 |
|
miconis
|
9258e4f095
|
implementation of a new workflow to compute statistics on the blocks
|
2020-07-13 18:22:34 +02:00 |
|
Claudio Atzori
|
c6f6fb0f28
|
code formatting
|
2020-07-13 16:46:13 +02:00 |
|
Claudio Atzori
|
1143f426aa
|
WIP SparkCreateMergeRels distinct relations
|
2020-07-13 16:13:36 +02:00 |
|
Claudio Atzori
|
8c67938ad0
|
configurable number of partitions used in the SparkCreateSimRels phase
|
2020-07-13 16:07:07 +02:00 |
|
Claudio Atzori
|
c8284bab06
|
WIP SparkCreateMergeRels distinct relations
|
2020-07-13 15:54:51 +02:00 |
|
Claudio Atzori
|
7dd91edf43
|
parsing of optional parameter
|
2020-07-13 15:40:41 +02:00 |
|
Claudio Atzori
|
4c101a9d66
|
WIP SparkCreateMergeRels distinct relations
|
2020-07-13 15:31:38 +02:00 |
|
Claudio Atzori
|
8a612d861a
|
WIP SparkCreateMergeRels distinct relations
|
2020-07-13 15:30:57 +02:00 |
|
Sandro La Bruzzo
|
9ef2385022
|
implemented test for cut of connected component
|
2020-07-13 15:28:17 +02:00 |
|
Sandro La Bruzzo
|
d561b2dd21
|
implemented cut of connected component
|
2020-07-13 14:18:42 +02:00 |
|
Claudio Atzori
|
7a3fd9f54c
|
dedup relation aggregator moved into dedicated class
|
2020-07-13 10:11:36 +02:00 |
|
Claudio Atzori
|
770adc26e9
|
WIP aggregator to make relationships unique
|
2020-07-10 19:35:10 +02:00 |
|
Claudio Atzori
|
752d28f8eb
|
make the relations produced by the dedup SparkPropagateRelation jon unique
|
2020-07-10 15:09:50 +02:00 |
|
Claudio Atzori
|
3c728aaa0c
|
trying to overcome OOM errors during duplicate scan phase
|
2020-07-08 22:39:51 +02:00 |
|
Claudio Atzori
|
4365cf41d7
|
trying to overcome OOM errors during duplicate scan phase
|
2020-07-08 22:31:46 +02:00 |
|
Claudio Atzori
|
0f77cac4b5
|
fix: deduper must use queueMaxSize instead of groupMaxSize for the block definition
|
2020-07-02 12:43:51 +02:00 |
|
Claudio Atzori
|
7181807e64
|
code formatting
|
2020-05-23 09:51:48 +02:00 |
|
miconis
|
0fd0c7d725
|
reimplementation of the sim between two authors. now it takes into account both name and surname. threshold incremented to 1.0 if the name is too short
|
2020-05-22 17:24:57 +02:00 |
|
Claudio Atzori
|
3cf2796ac6
|
code formatting
|
2020-05-22 12:34:00 +02:00 |
|
miconis
|
8bbd1d0501
|
reimplementation of the author merging in deduprecord creation. implementation of the test class.
|
2020-05-21 11:52:14 +02:00 |
|
Claudio Atzori
|
fd519df616
|
new rels produced by dedup workflow must be unique
|
2020-05-08 19:00:38 +02:00 |
|
Claudio Atzori
|
5b28bb4131
|
code formatting
|
2020-05-08 16:49:47 +02:00 |
|
miconis
|
3420998bb4
|
reltype set in mergerels
|
2020-05-08 15:43:30 +02:00 |
|
miconis
|
3df703f67d
|
mergerels added to propagate relations
|
2020-05-04 12:08:12 +02:00 |
|
Claudio Atzori
|
439c6255a2
|
cleanup
|
2020-04-29 19:09:07 +02:00 |
|
Claudio Atzori
|
6f5b899038
|
reformatted code according to the updated style descriptor
|
2020-04-28 11:23:29 +02:00 |
|
Claudio Atzori
|
a0bdbacdae
|
switched automatic code formatting plugin to net.revelc.code.formatter:formatter-maven-plugin
|
2020-04-27 14:52:31 +02:00 |
|
Claudio Atzori
|
7a3f8085f7
|
switched automatic code formatting plugin to net.revelc.code.formatter:formatter-maven-plugin
|
2020-04-27 14:45:40 +02:00 |
|
Claudio Atzori
|
9ddafd46ca
|
fixed dedup record id prefix, set the correct dataInfo in the DedupRecordFactory
|
2020-04-23 07:50:18 +02:00 |
|
Claudio Atzori
|
91e72a6944
|
Dataset based implementation for SparkCreateDedupRecord phase, fixed datasource entity dump supplementing dedup unit tests
|
2020-04-21 12:06:08 +02:00 |
|
Claudio Atzori
|
eb8a020859
|
fixed behaviour of DedupRecordFactory
|
2020-04-20 18:44:06 +02:00 |