Claudio Atzori
03670bb9ce
[dedup] use common saveParquet and save methods to ensure outputs are compressed
2023-10-16 10:55:47 +02:00
Giambattista Bloisi
e64c2854a3
Refactor Dedup process to use Spark Dataframe API and intermediate representation with Row interface
...
JsonPath cache contention fixed by using a ConcurrentHashMap
Blacklist filtering performance improvement
Minor performance improvements when evaluating similarity
Sorting in clustered elements is deterministic (by ordering and identity field, instead of ordering field only)
2023-07-24 15:36:24 +02:00
Sandro La Bruzzo
9963fd6d29
updated log to add subentity
2023-06-28 13:36:05 +02:00
Sandro La Bruzzo
9910ce06ae
added to CreateSimRel the feature to write time log
2023-06-28 11:38:16 +02:00
Sandro La Bruzzo
bd17c3edc8
added to CreateSimRel the feature to write time log
2023-06-28 11:20:58 +02:00
Claudio Atzori
a24b9f8268
[dedup] trivial refactoring
2021-11-18 17:12:02 +01:00
Claudio Atzori
2ee21da43b
suggestions from SonarLint
2021-08-11 12:13:22 +02:00
Claudio Atzori
815b9f4d56
[openorgs dedup] fixed workflow parameter declarations. Introduced support for resuming the execution from intermediate steps
2021-04-20 17:24:45 +02:00
miconis
2355cc4e9b
minor changes and bug fix
2021-03-29 10:07:12 +02:00
miconis
28c1cdd132
merged stable_ids into openorgswf
2021-03-25 10:44:49 +01:00
miconis
1a85020572
bug fix in graph-mapper, changes in the implementation of the openorgs wf to create relations and populate openorgs db
2021-02-26 10:19:28 +01:00
miconis
e3f7798d1b
minor changes in dedup tests, bug fix in the idgenerator and pace-core version update
2020-09-29 15:31:46 +02:00
miconis
259362ef47
implementation of the job to collect simrels from postgres db
2020-09-22 09:43:27 +02:00
Claudio Atzori
06def0c0cb
SparkBlockStats allows to repartition the input rdd via the numPartitions workflow parameter
2020-07-13 20:09:06 +02:00
Claudio Atzori
c6f6fb0f28
code formatting
2020-07-13 16:46:13 +02:00
Claudio Atzori
8c67938ad0
configurable number of partitions used in the SparkCreateSimRels phase
2020-07-13 16:07:07 +02:00
Claudio Atzori
8a612d861a
WIP SparkCreateMergeRels distinct relations
2020-07-13 15:30:57 +02:00
Claudio Atzori
3c728aaa0c
trying to overcome OOM errors during duplicate scan phase
2020-07-08 22:39:51 +02:00
Claudio Atzori
4365cf41d7
trying to overcome OOM errors during duplicate scan phase
2020-07-08 22:31:46 +02:00
Claudio Atzori
6f5b899038
reformatted code according to the updated style descriptor
2020-04-28 11:23:29 +02:00
Claudio Atzori
a0bdbacdae
switched automatic code formatting plugin to net.revelc.code.formatter:formatter-maven-plugin
2020-04-27 14:52:31 +02:00
Claudio Atzori
7a3f8085f7
switched automatic code formatting plugin to net.revelc.code.formatter:formatter-maven-plugin
2020-04-27 14:45:40 +02:00
miconis
4da13e4570
Revert "Merge branch 'master' into deduptesting"
...
This reverts commit 772f75d167
, reversing
changes made to 5f45f2c77f
.
2020-04-20 16:04:49 +02:00
Claudio Atzori
d714bfb4d4
collectedfrom field moved in common parent class Oaf.java
2020-04-20 12:25:19 +02:00
Claudio Atzori
ad7a131b18
introduced common project code formatting plugin, works on the commit hook, based on https://github.com/Cosium/git-code-format-maven-plugin , applied to each java class in the project
2020-04-18 12:42:58 +02:00
Claudio Atzori
71813795f6
various refactorings on the dnet-dedup-openaire workflow
2020-04-18 12:06:23 +02:00
Claudio Atzori
038ac7afd7
relation consistency workflow separated from dedup scan and creation of CCs
2020-04-17 13:12:44 +02:00
miconis
0eccbc318b
Deduper class (utilities for dedup) cleaned. Useless methods removed
2020-04-16 12:36:37 +02:00
miconis
5689d49689
minor changes
2020-04-15 16:34:06 +02:00
miconis
53fd624c34
implemented test for sparkcreatesimrels
2020-04-03 18:32:25 +02:00
miconis
a61763d149
structure for sparksimrel changed to be compliant with mockito testing
2020-04-02 18:37:53 +02:00
miconis
9802bcb9fe
dedup testing
2020-04-01 18:48:31 +02:00
Claudio Atzori
673e744649
moved openaire specific implementations under dedicated package eu.dnetlib.dhp.oa
2020-03-27 10:42:17 +01:00