Pull Requests - dnet-hadoop - Gitea

New Pull Request

Fixes in Graph Provision

#434 by giambattista.bloisi was merged 2024-05-09 14:15:05 +02:00

beta_provision_relation

1 waiting review

Miscellaneous related to changes in MergeUtils

#429 by giambattista.bloisi was merged 2024-04-24 08:55:38 +02:00

misc_fixes_merge_entities

Refinements to PR #404: refactoring the Oaf records merge utilities into dhp-common

#422 by giambattista.bloisi was merged 2024-04-17 11:58:27 +02:00

revised_merge_logic

Enhance Dedup authors matching with algorithms used for ORCID enhancements (task 9690)

#419 by giambattista.bloisi was merged 2024-04-16 10:24:12 +02:00

dedup_authorsmatch_bytoken

Enrich authors with ORCID info using new matching algorithm

#398 by giambattista.bloisi was merged 2024-03-22 17:29:20 +01:00

new_orcid_enhancement

Revised procedure when converting json data into xml

#395 by giambattista.bloisi was merged 2024-02-28 10:38:55 +01:00

restiterator_xmlcleanup

Revised instance type comparisons in dedup phase

#393 by giambattista.bloisi was merged 2024-02-15 12:15:38 +01:00

revisedInstanceType

Set deletedbyinference =true to dedup aliases, created when a dedup in a previous build has been merged in a new dedup

#392 by giambattista.bloisi was merged 2024-02-08 15:29:30 +01:00

fix_dedupaliases_deletedbyinference

Use SparkSQL in place of Hive for executing step16-createIndicatorsTables.sql of stats update wf

#386 by giambattista.bloisi was merged 2024-01-29 09:12:00 +01:00

stats_with_spark_sql

Implements pivots table update oozie workflow

#376 by giambattista.bloisi was merged 2024-01-22 16:37:30 +01:00

update_pivots_table

Unify merge logic of entities in MergeUtils.class

#370 by giambattista.bloisi was merged 2024-03-22 10:53:15 +01:00

Improvements and refactoring in Dedup

#367 by giambattista.bloisi was merged 2024-01-11 11:24:07 +01:00

dedup_increasenumofblocks

GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob

#334 by giambattista.bloisi was merged 2023-10-02 11:25:28 +02:00

merge_entities_job

SparkPropagateRelation relations do not propagate deletedByInference and invisible

#333 by giambattista.bloisi was merged 2023-10-02 11:27:58 +02:00

consistency_keep_mergerels

Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb

#331 by giambattista.bloisi was merged 2023-08-29 16:31:37 +02:00

consistencywf_memoryoverhead_conf

Rewrite SparkPropagateRelation exploiting Dataframe API

#330 by giambattista.bloisi was merged 2023-08-29 10:47:15 +02:00

propagate_relation_rewrite

DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag

#329 by giambattista.bloisi was merged 2023-08-10 12:56:19 +02:00

dispatch_filter_invisible_entities

Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleted by inference or that are pointing to dangling entities

#328 by giambattista.bloisi was merged 2023-08-08 09:49:13 +02:00

cleanup_relations_after_dedup

Refactor Dedup using Spark Dataframe API, initial support for scala 2.12 and Spark 3.4

#324 by giambattista.bloisi was merged 2023-07-25 10:17:18 +02:00

dedup-with-dataframe-2

fix_beta_tests

#323 by giambattista.bloisi was merged 2023-07-24 10:47:36 +02:00

1 2