BrBETA_dnet-hadoop

Commit Graph

Select branches

Hide Pull Requests

beta

beta_tf_hal

broker_relations_upgrade

crossref_dw_wf

datasource_model_eosc_beta

dedup_whitelist

deduptesting

dhp_oaf_model

doiboosi_accessright

doiboost_wf

dump

enrichment

fct_project_id_replacement

forceOrginalId_claims

graph_cleaning

graph_db

hostedbymap

master

organization_pids

refactoring_download_csv

serializeLicense

stable_ids

unique_field_in_lists

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

archive/master_pre_stable_ids

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

e84f5b5e64 extended existing codo to accomodate import of POCI from open citation Miriam Baglioni 2023-10-02 09:25:16 +0200
ab0d70691c Add step for archiving repoUrls to SWH Serafeim Chatzopoulos 2023-09-28 20:56:18 +0300
775c3f704a Fix bug in conversion from dedup json model to Spark Dataset of Rows: list of strings contained the json escaped representation of the value instead of the plain value, this caused instanceTypeMatch failures because of the leading and trailing double quotes Giambattista Bloisi 2023-09-27 22:30:47 +0200
ed9c81a0b7 Add steps to collect last visit data && archive not found repository URLs Serafeim Chatzopoulos 2023-09-27 19:00:54 +0300
9c3ab11d5b Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop Sandro La Bruzzo 2023-09-25 15:29:19 +0200
423ef30676 minor fix on the aggregation of uniprot and pdb Sandro La Bruzzo 2023-09-25 15:28:58 +0200
7152d47f84 Use asScala to convert java List to Scala Sequence Giambattista Bloisi 2023-09-20 16:14:01 +0200
4853c19b5e code formatting Claudio Atzori 2023-09-20 15:53:21 +0200
1f226d1dce Fix defect #8997: GenerateEventsJob is generating huge amounts of logs because broker entity similarity calculation consistently failed Giambattista Bloisi 2023-09-20 15:42:00 +0200
0935d7757c Use v5 of the UNIBI Gold ISSN list in test Alessia Bardi 2023-09-19 14:47:01 +0200
cc7204a089 tests for d4science catalog Alessia Bardi 2023-09-19 13:38:25 +0200
76476cdfb6 Added maven repo for dependencies that are not in maven central Sandro La Bruzzo 2023-09-20 10:33:14 +0200
6186cdc2cc Use v5 of the UNIBI Gold ISSN list in test Alessia Bardi 2023-09-19 14:47:01 +0200
d94b9bebf7 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Alessia Bardi 2023-09-19 13:38:45 +0200
19abba8fa7 tests for d4science catalog Alessia Bardi 2023-09-19 13:38:25 +0200
9ef971a146 Update step16-createIndicatorsTables.sql dimitrispie 2023-09-19 14:25:42 +0300
9d44418d38 Add collecting software code repository URLs Serafeim Chatzopoulos 2023-09-14 18:43:25 +0300
395a4af020 Run CC and RAM sequentieally in dhp-impact-indicators WF Serafeim Chatzopoulos 2023-09-12 22:31:50 +0300
c2f179800c Merge pull request 'Run CC and RAM sequentieally in dhp-impact-indicators WF' (#338) from run_cc_and_ram_sequentially into master Claudio Atzori 2023-09-13 08:52:53 +0200
2aed5a74be Run CC and RAM sequentieally in dhp-impact-indicators WF Serafeim Chatzopoulos 2023-09-12 22:31:50 +0300
8a6892cc63 [graph dedup] consistency wf should not remove the relations while dispatching the entities Claudio Atzori 2023-09-12 14:34:28 +0200
4dc4862011 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Claudio Atzori 2023-09-12 14:34:34 +0200
dc80ab14d3 [graph dedup] consistency wf should not remove the relations while dispatching the entities Claudio Atzori 2023-09-12 14:34:28 +0200
77a2199837 updated test for EOSC comunity Alessia Bardi 2023-09-08 11:05:49 +0200
4786aa0e09 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map Claudio Atzori 2023-09-07 11:20:35 +0200
265180bfd2 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map Claudio Atzori 2023-09-07 11:20:35 +0200
5f90cc11e9 Update step16-createIndicatorsTables.sql dimitrispie 2023-09-06 14:14:38 +0300
da0e9828f7 resolved conflicts for PR#337 Claudio Atzori 2023-09-06 11:28:46 +0200
9f5d16624c Merge pull request '[graph raw] datainfo.invisible set as true only for entities' (#336) from invisible_relations into beta Claudio Atzori 2023-09-04 16:14:47 +0200
adec6692ca Merge branch 'beta' into invisible_relations Claudio Atzori 2023-09-04 16:13:06 +0200
15666e86a8 added collectedfrom to the affiliation relations imported from Crossref Claudio Atzori 2023-09-04 15:56:06 +0200
7d6bd4f20b Merge pull request 'Fix import of affiliations relations from Crossref' (#335) from 8876_fix_crossref_affiliation_relations_import into beta Claudio Atzori 2023-09-04 15:19:58 +0200
5b06c9d06f [graph raw] datainfo.invisible set as true only for entities Claudio Atzori 2023-09-04 15:15:24 +0200
7de0164c26 Fix import of affiliations relations from Crossref Serafeim Chatzopoulos 2023-09-04 16:04:41 +0300
2caaaec42d Include SparkCleanRelation logic in SparkPropagateRelation SparkPropagateRelation includes merge relations Revised tests for SparkPropagateRelation Giambattista Bloisi 2023-09-01 09:32:57 +0200
964c2f553e Changes in indicators step, monitor step dimitrispie 2023-09-01 10:57:02 +0300
6cc7d8ca7b GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob Giambattista Bloisi 2023-08-24 21:48:07 +0200
488d9a1cea Merge pull request 'Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb' (#331) from consistencywf_memoryoverhead_conf into beta Claudio Atzori 2023-08-29 16:31:36 +0200
6b1c05d118 Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb Giambattista Bloisi 2023-08-29 16:04:19 +0200
bf35280ea6 code formatting Claudio Atzori 2023-08-29 11:11:00 +0200
0515d81c7c Merge pull request 'Rewrite SparkPropagateRelation exploiting Dataframe API' (#330) from propagate_relation_rewrite into beta Claudio Atzori 2023-08-29 10:47:14 +0200
58665a246c Merge branch 'beta' into propagate_relation_rewrite Claudio Atzori 2023-08-29 10:47:02 +0200
f437be80ad [impact indicators] adjusted paths in the bip ranker wf parameters Claudio Atzori 2023-08-29 09:03:03 +0200
d012aec0b3 Revert PropagateRelation's argument name from outputPath to graphOutputPath in consistency workflow (#8964) Giambattista Bloisi 2023-08-28 22:44:54 +0200
a860e19423 Fix ensure all relations are written out, not only those managed by dedup Giambattista Bloisi 2023-08-28 15:36:02 +0200
0d7b2bf83d Rewrite SparkPropagateRelation exploiting Dataframe API Giambattista Bloisi 2023-08-28 10:34:54 +0200
9c8b41475a Merge pull request '8172_impact_indicators_workflow' (#284) from 8172_impact_indicators_workflow into beta Miriam Baglioni 2023-08-14 15:50:48 +0200
97c1ba8918 Merge actionsets of results and projects Serafeim Chatzopoulos 2023-08-11 15:56:53 +0300
35b8deb2c6 Merge pull request 'DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag' (#329) from dispatch_filter_invisible_entities into beta Miriam Baglioni 2023-08-10 12:56:18 +0200
95cd2b9b1e Make filterInvisible a mandatory parameter of DispathEntitiesSparkJob Make filterInvisible a mandatory parameter of both dedup/consistency and graph/group oozie workflows Giambattista Bloisi 2023-08-10 11:53:48 +0200
fab9920271 DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag Giambattista Bloisi 2023-08-08 15:52:20 +0200
599828ce35 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Miriam Baglioni 2023-08-09 13:07:13 +0200
c25ac21e5e Merge pull request 'graph cleaning, suggestions from ticket 8898' (#325) from cleaning_8898 into beta Miriam Baglioni 2023-08-08 11:14:19 +0200
c334fe2438 Merge pull request 'Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleted by inference or that are pointing to dangling entities' (#328) from cleanup_relations_after_dedup into beta Miriam Baglioni 2023-08-08 09:49:12 +0200
0e2f855807 Merge pull request 'Updates Promotion DBs' (#321) from antonis.lempesis/dnet-hadoop:beta into beta Miriam Baglioni 2023-08-07 12:09:16 +0200
18fbe52b20 Merge pull request 'Import affiliation relations from Crossref' (#320) from 8876 into beta Miriam Baglioni 2023-08-07 10:45:30 +0200
97b6d1dc45 Filter ids by dataInfo.deletedbyinference and DataInfo.invisible flags Filter relations also by dataInfo.invisible flag Giambattista Bloisi 2023-08-07 10:24:11 +0200
af49424b59 Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleyted by inference or that are pointing to dangling entities Giambattista Bloisi 2023-08-04 14:27:39 +0200
0bc74e2000 code formatting 1693265288148170348/tmp_refs/heads/master 1693265288148170348/master 1693263292328622369/tmp_refs/heads/master 1693263292328622369/master 1693145428336165040/tmp_refs/heads/master 1693145428336165040/master Claudio Atzori 2023-08-02 11:52:10 +0200
7180911ded [graph cleaning] fixed regex behaviour for cleaning ROR and GRID identifiers, added tests Claudio Atzori 2023-06-23 16:10:49 +0200
b9dddbfe54 rule out records with NULL dataInfo, except for Relations Claudio Atzori 2023-07-31 17:52:56 +0200
da1727f93f rule out records with NULL dataInfo, except for Relations Claudio Atzori 2023-07-31 17:52:56 +0200
11ffb9bd68 rule out records with NULL dataInfo Claudio Atzori 2023-07-31 12:35:05 +0200
ccac6a7f75 rule out records with NULL dataInfo Claudio Atzori 2023-07-31 12:35:05 +0200
7cefe2665b Remove unnecessary classes Serafeim Chatzopoulos 2023-07-28 19:14:39 +0300
26a92ce762 Merge branch '8876' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8876 Serafeim Chatzopoulos 2023-07-28 19:03:57 +0300
ebfba38ab6 Add changes from code review Serafeim Chatzopoulos 2023-07-28 19:03:47 +0300
eb8684a8cf Merge branch 'beta' into 8876 Serafeim Chatzopoulos 2023-07-28 13:39:33 +0200
1275a07d45 Merge pull request '[graph indexing] expand the instance level fulltext in the XML records' (#326) from instance_fulltext_xml into beta Claudio Atzori 2023-07-27 15:02:07 +0200
a72b9e96ac expand the instance level fulltext in the XML records Claudio Atzori 2023-07-27 14:57:38 +0200
d512df8612 code formatting Claudio Atzori 2023-07-26 09:14:08 +0200
d8435a6512 inverted condition Claudio Atzori 2023-07-25 17:39:57 +0200
59764145bb cherry picked & fixed commit 270df939c4 Claudio Atzori 2023-07-25 17:39:00 +0200
270df939c4 partial implementation of the suggestions from https://support.openaire.eu/issues/8898 Claudio Atzori 2023-07-25 17:29:50 +0200
8c63e4a864 Merge pull request 'Refactor Dedup using Spark Dataframe API, initial support for scala 2.12 and Spark 3.4' (#324) from dedup-with-dataframe-2 into beta Claudio Atzori 2023-07-25 10:17:17 +0200
e64c2854a3 Refactor Dedup process to use Spark Dataframe API and intermediate representation with Row interface JsonPath cache contention fixed by using a ConcurrentHashMap Blacklist filtering performance improvement Minor performance improvements when evaluating similarity Sorting in clustered elements is deterministic (by ordering and identity field, instead of ordering field only) Giambattista Bloisi 2023-07-18 11:38:56 +0200
bb5b845e3c Use scala.binary.version property to resolve scala maven dependencies Ensure consistent usage of maven properties Profile for compiling with scala 2.12 and Spark 3.4 Giambattista Bloisi 2023-07-17 17:18:46 +0200
002b24e06f Merge pull request '[graph cleaning] fixed regex behaviour for cleaning ROR and GRID identifiers, added tests' (#315) from pid_cleaning into beta Claudio Atzori 2023-07-24 10:49:44 +0200
c754397a19 Merge branch 'beta' into pid_cleaning Claudio Atzori 2023-07-24 10:49:31 +0200
f0678cda09 Merge pull request 'fix_beta_tests' (#323) from fix_beta_tests into beta Claudio Atzori 2023-07-24 10:47:35 +0200
3a0f09774a Add script to find score limits Serafeim Chatzopoulos 2023-07-21 17:55:41 +0300
06b9b71c4e Merge branch '8172_impact_indicators_workflow' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8172_impact_indicators_workflow Ilias Kanellos 2023-07-21 17:42:49 +0300
2374f445a9 Produce additional bip update specific files Ilias Kanellos 2023-07-21 17:42:46 +0300
cb0f3c50f6 Format workflow.xml Serafeim Chatzopoulos 2023-07-21 16:07:10 +0300
c64e5e588f Merge branch '8172_impact_indicators_workflow' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8172_impact_indicators_workflow Serafeim Chatzopoulos 2023-07-21 15:27:02 +0300
2cc5b1a39b Fixes in workflow.xml Serafeim Chatzopoulos 2023-07-21 15:26:50 +0300
0f96af5d56 Merge branch '8172_impact_indicators_workflow' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8172_impact_indicators_workflow Ilias Kanellos 2023-07-21 13:42:35 +0300
03da965162 Format bip-score based file without doi references Ilias Kanellos 2023-07-21 13:42:30 +0300
f03153823a Update testCitationRelations number of expected citations according to changes made in 0559d8b4 (monodirectional citations) Giambattista Bloisi 2023-07-21 10:48:28 +0200
54c1eacef1 SparkJobTest was failing because testing workingdir was not cleaned up after eact test Giambattista Bloisi 2023-07-21 10:42:24 +0200
5e15f20e6e Fix entityMerger that was excluding the authors of the first entity in the list to merge Giambattista Bloisi 2023-07-21 00:46:54 +0200
0210a14e43 Ignore timestamp differences in PromoteActionPayloadForGraphTableJobTest Giambattista Bloisi 2023-07-20 23:45:57 +0200
dba34505de Fix SparkStatsTest bug where parquet tables were incorrectly read as text files leading to unpredictable count() values Giambattista Bloisi 2023-07-19 14:24:52 +0200
e47ed1fdb2 Use DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES in json mapper to avoid that tests fail if they encounter unmapped properties Giambattista Bloisi 2023-07-19 14:21:40 +0200
38dfebfbe6 Disable MdStoreClientTest test as it requires a local mongodb running and it does not perform any assertions Giambattista Bloisi 2023-07-19 14:18:56 +0200
9e8e39f78a - Miriam Baglioni 2023-07-19 11:35:58 +0200
373a5f2c83 Merge pull request 'Master branch updates from beta July 2023' (#317) from master_july23 into master Claudio Atzori 2023-07-18 18:22:04 +0200
db4ca43ee8 Resolve conflict Serafeim Chatzopoulos 2023-07-18 18:38:26 +0300
be320ba3c1 Indentation fixes Serafeim Chatzopoulos 2023-07-17 16:04:21 +0300
be4856ef35 Update step15.sql dimitrispie 2023-07-17 15:33:58 +0300