 Claudio Atzori
|
7dd91edf43
|
parsing of optional parameter
|
2020-07-13 15:40:41 +02:00 |
|
Claudio Atzori
|
4c101a9d66
|
WIP SparkCreateMergeRels distinct relations
|
2020-07-13 15:31:38 +02:00 |
|
Claudio Atzori
|
8a612d861a
|
WIP SparkCreateMergeRels distinct relations
|
2020-07-13 15:30:57 +02:00 |
 Sandro La Bruzzo
|
9ef2385022
|
implemented test for cut of connected component
|
2020-07-13 15:28:17 +02:00 |
|
Sandro La Bruzzo
|
d561b2dd21
|
implemented cut of connected component
|
2020-07-13 14:18:42 +02:00 |
 Miriam Baglioni
|
8e0e090d7a
|
merge upstream
|
2020-07-13 12:46:55 +02:00 |
|
Claudio Atzori
|
e2093e42db
|
Merge branch 'master' into deduptesting
|
2020-07-13 10:57:49 +02:00 |
 Michele Artini
|
2c4ed9a043
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-13 10:55:39 +02:00 |
|
Michele Artini
|
ccbe5c5658
|
fixed import of eu.dnetlib.dhp:dnet-openaire-broker-common
|
2020-07-13 10:55:27 +02:00 |
|
Claudio Atzori
|
7a3fd9f54c
|
dedup relation aggregator moved into dedicated class
|
2020-07-13 10:11:36 +02:00 |
 Alessia Bardi
|
7e96105947
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop
|
2020-07-12 19:29:12 +02:00 |
|
Alessia Bardi
|
b7a39731a6
|
assert, not print
|
2020-07-12 19:28:56 +02:00 |
|
Claudio Atzori
|
770adc26e9
|
WIP aggregator to make relationships unique
|
2020-07-10 19:35:10 +02:00 |
|
Claudio Atzori
|
ecf119f37a
|
Merge branch 'master' into deduptesting
|
2020-07-10 19:04:16 +02:00 |
|
Claudio Atzori
|
31071e363f
|
Merge branch 'provision_indexing'
|
2020-07-10 19:03:57 +02:00 |
|
Claudio Atzori
|
06c1913062
|
added different limits for grouping by source and by target, incremented spark.sql.shuffle.partitions for the join operations
|
2020-07-10 19:03:33 +02:00 |
|
Claudio Atzori
|
cc77446dc4
|
added dbSchema parameter to the raw_db workflow
|
2020-07-10 19:01:50 +02:00 |
|
Claudio Atzori
|
4c3836f62e
|
materialize the related entities before joining them
|
2020-07-10 19:00:44 +02:00 |
|
Michele Artini
|
e1ae964bc4
|
stats
|
2020-07-10 16:12:08 +02:00 |
|
Claudio Atzori
|
752d28f8eb
|
make the relations produced by the dedup SparkPropagateRelation jon unique
|
2020-07-10 15:09:50 +02:00 |
|
Sandro La Bruzzo
|
c01efed79b
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-10 14:44:57 +02:00 |
|
Sandro La Bruzzo
|
a7d3977481
|
added generation of EBI Dataset
|
2020-07-10 14:44:50 +02:00 |
|
Claudio Atzori
|
b21866a2da
|
allow to set different to relations cut points by source and by target; adjusted weight assigned to relationship types
|
2020-07-10 13:59:48 +02:00 |
|
Claudio Atzori
|
ff4d6214f1
|
experimenting with pruning of relations
|
2020-07-10 10:06:41 +02:00 |
|
Miriam Baglioni
|
faea30cda0
|
-
|
2020-07-09 14:05:21 +02:00 |
|
Michele Artini
|
2d742a84ae
|
DedupConfig as json file
|
2020-07-09 12:53:46 +02:00 |
|
Miriam Baglioni
|
a634794242
|
merge upstream
|
2020-07-09 11:46:51 +02:00 |
|
Michele Artini
|
a44b9b36b9
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-09 11:02:31 +02:00 |
|
Michele Artini
|
1c6a171633
|
updated pom
|
2020-07-09 11:02:09 +02:00 |
|
Claudio Atzori
|
3c728aaa0c
|
trying to overcome OOM errors during duplicate scan phase
|
2020-07-08 22:39:51 +02:00 |
|
Claudio Atzori
|
18c555cd79
|
Merge branch 'master' into deduptesting
|
2020-07-08 22:32:01 +02:00 |
|
Claudio Atzori
|
4365cf41d7
|
trying to overcome OOM errors during duplicate scan phase
|
2020-07-08 22:31:46 +02:00 |
|
Claudio Atzori
|
67e1d222b6
|
bulk cleaning when found null or empty, sets bestaccessrights evaluating the result instances
|
2020-07-08 17:53:35 +02:00 |
|
Alessia Bardi
|
853e8d7987
|
test for software merge
|
2020-07-08 17:03:53 +02:00 |
|
Claudio Atzori
|
610d377d57
|
first implementation of the BETA & PROD graphs merge procedure
|
2020-07-08 16:54:26 +02:00 |
|
Claudio Atzori
|
e2ea30f89d
|
updated graph construction workflow definition: cleaning wf moved at the bottom to include cleaning of the information produced by the enrichment workflows
|
2020-07-08 12:16:24 +02:00 |
|
Michele Artini
|
dffa0b01a2
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-07 15:37:29 +02:00 |
|
Michele Artini
|
efadbdb2bc
|
fixed a bug with duplicated events
|
2020-07-07 15:37:13 +02:00 |
|
Claudio Atzori
|
8af8e7481a
|
code formatting
|
2020-07-07 14:23:34 +02:00 |
|
Claudio Atzori
|
b383ed42fa
|
pass optional parameter relationFilter to the PrepareRelationJob implementation
|
2020-07-07 14:21:28 +02:00 |
|
Claudio Atzori
|
911894a987
|
Merge branch 'deduptesting'
|
2020-07-07 14:20:43 +02:00 |
|
Miriam Baglioni
|
d22240c0ba
|
merge upstream
|
2020-07-06 13:58:02 +02:00 |
|
Michele Artini
|
edf6c6c4dc
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-03 11:48:24 +02:00 |
|
Michele Artini
|
04bebb708c
|
some fixes
|
2020-07-03 11:48:12 +02:00 |
|
Claudio Atzori
|
c3d67f709a
|
adjusted dedup configuration for result entities: using new wordssuffixprefix clustering function, removed ngrampairs, adjusted queueMaxSize (800) and slidingWindowSize (80)
|
2020-07-02 17:35:22 +02:00 |
|
Miriam Baglioni
|
c133a23cf0
|
merge upstream
|
2020-07-02 14:24:57 +02:00 |
|
Claudio Atzori
|
1d39f7901c
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop
|
2020-07-02 12:45:01 +02:00 |
|
Claudio Atzori
|
0f77cac4b5
|
fix: deduper must use queueMaxSize instead of groupMaxSize for the block definition
|
2020-07-02 12:43:51 +02:00 |
|
Sandro La Bruzzo
|
18b9330312
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-07-02 12:43:19 +02:00 |
|
Michele Artini
|
b413db0bff
|
white/blacklists
|
2020-07-02 12:43:03 +02:00 |