Pull Requests - dnet-hadoop - Gitea

New Pull Request

Import dnet-pace-core module in this project and use it after renaming to dhp-pace-core

#319 by giambattista.bloisi was merged 2023-07-11 14:03:15 +02:00

beta_with_pace_core

fix_beta_tests

#323 by giambattista.bloisi was merged 2023-07-24 10:47:36 +02:00

Refactor Dedup using Spark Dataframe API, initial support for scala 2.12 and Spark 3.4

#324 by giambattista.bloisi was merged 2023-07-25 10:17:18 +02:00

dedup-with-dataframe-2

Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleted by inference or that are pointing to dangling entities

#328 by giambattista.bloisi was merged 2023-08-08 09:49:13 +02:00

cleanup_relations_after_dedup

DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag

#329 by giambattista.bloisi was merged 2023-08-10 12:56:19 +02:00

dispatch_filter_invisible_entities

Rewrite SparkPropagateRelation exploiting Dataframe API

#330 by giambattista.bloisi was merged 2023-08-29 10:47:15 +02:00

propagate_relation_rewrite

Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb

#331 by giambattista.bloisi was merged 2023-08-29 16:31:37 +02:00

consistencywf_memoryoverhead_conf

use saveParquet method to ensure mergerel output is compressed

#348 by giambattista.bloisi was closed 2023-10-16 10:59:39 +02:00

fix_mergerels_notcompressed

OpenAIRE - DNet

SparkPropagateRelation relations do not propagate deletedByInference and invisible

#333 by giambattista.bloisi was merged 2023-10-02 11:27:58 +02:00

consistency_keep_mergerels

GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob

#334 by giambattista.bloisi was merged 2023-10-02 11:25:28 +02:00

merge_entities_job

Fix bug in conversion from dedup json model to Spark Dataset of Rows (instanceTypeMatch no longer working)

#339 by giambattista.bloisi was merged 2023-10-02 11:34:20 +02:00

fix_dedupfailsonmatchinginstances

OpenAIRE - DNet

Fix cleaning of Pmid where parsing of numbers stopped at first not leading 0 (zero) character

#345 by giambattista.bloisi was merged 2023-10-06 14:19:49 +02:00

fix_truncated_pmid

OpenAIRE - DNet

FIX: GroupEntitiesSparkJob deletes whole graph outputPath instead of its temporary folder

#351 by giambattista.bloisi was merged 2023-10-17 08:40:24 +02:00

fix_consistency_missing_rels

OpenAIRE - DNet

Implements pivots table update oozie workflow

#376 by giambattista.bloisi was merged 2024-01-22 16:37:30 +01:00

update_pivots_table

Improvements and refactoring in Dedup

#367 by giambattista.bloisi was merged 2024-01-11 11:24:07 +01:00

dedup_increasenumofblocks

Use SparkSQL in place of Hive for executing step16-createIndicatorsTables.sql of stats update wf

#386 by giambattista.bloisi was merged 2024-01-29 09:12:00 +01:00

stats_with_spark_sql

Set deletedbyinference =true to dedup aliases, created when a dedup in a previous build has been merged in a new dedup

#392 by giambattista.bloisi was merged 2024-02-08 15:29:30 +01:00

fix_dedupaliases_deletedbyinference

Revised instance type comparisons in dedup phase

#393 by giambattista.bloisi was merged 2024-02-15 12:15:38 +01:00

revisedInstanceType

Revised procedure when converting json data into xml

#395 by giambattista.bloisi was merged 2024-02-28 10:38:55 +01:00

restiterator_xmlcleanup

Unify merge logic of entities in MergeUtils.class

#370 by giambattista.bloisi was merged 2024-03-22 10:53:15 +01:00

1 2