dnet-hadoop

Commit Graph

Select branches

Hide Pull Requests

dedupTest

deduptesting

doiboost

hive_timeout

impala_changes_by_alexiou

junit5

master

migration_wfs_regular_all_steps

provision_dataset

provision_indexing

scholexplorer

stats_wf_extensions_and_corrections

usage-stats-export-wf

#1

#2

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

d94b9bebf7 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Alessia Bardi 2023-09-19 13:38:45 +0200
19abba8fa7 tests for d4science catalog Alessia Bardi 2023-09-19 13:38:25 +0200
9ef971a146 Update step16-createIndicatorsTables.sql dimitrispie 2023-09-19 14:25:42 +0300
9d44418d38 Add collecting software code repository URLs Serafeim Chatzopoulos 2023-09-14 18:43:25 +0300
395a4af020 Run CC and RAM sequentieally in dhp-impact-indicators WF Serafeim Chatzopoulos 2023-09-12 22:31:50 +0300
c2f179800c Merge pull request 'Run CC and RAM sequentieally in dhp-impact-indicators WF' (#338) from run_cc_and_ram_sequentially into master Claudio Atzori 2023-09-13 08:52:53 +0200
2aed5a74be Run CC and RAM sequentieally in dhp-impact-indicators WF Serafeim Chatzopoulos 2023-09-12 22:31:50 +0300
8a6892cc63 [graph dedup] consistency wf should not remove the relations while dispatching the entities Claudio Atzori 2023-09-12 14:34:28 +0200
4dc4862011 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Claudio Atzori 2023-09-12 14:34:34 +0200
dc80ab14d3 [graph dedup] consistency wf should not remove the relations while dispatching the entities Claudio Atzori 2023-09-12 14:34:28 +0200
77a2199837 updated test for EOSC comunity Alessia Bardi 2023-09-08 11:05:49 +0200
4786aa0e09 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map Claudio Atzori 2023-09-07 11:20:35 +0200
265180bfd2 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map Claudio Atzori 2023-09-07 11:20:35 +0200
5f90cc11e9 Update step16-createIndicatorsTables.sql dimitrispie 2023-09-06 14:14:38 +0300
da0e9828f7 resolved conflicts for PR#337 Claudio Atzori 2023-09-06 11:28:46 +0200
9f5d16624c Merge pull request '[graph raw] datainfo.invisible set as true only for entities' (#336) from invisible_relations into beta Claudio Atzori 2023-09-04 16:14:47 +0200
adec6692ca Merge branch 'beta' into invisible_relations Claudio Atzori 2023-09-04 16:13:06 +0200
15666e86a8 added collectedfrom to the affiliation relations imported from Crossref Claudio Atzori 2023-09-04 15:56:06 +0200
7d6bd4f20b Merge pull request 'Fix import of affiliations relations from Crossref' (#335) from 8876_fix_crossref_affiliation_relations_import into beta Claudio Atzori 2023-09-04 15:19:58 +0200
5b06c9d06f [graph raw] datainfo.invisible set as true only for entities Claudio Atzori 2023-09-04 15:15:24 +0200
7de0164c26 Fix import of affiliations relations from Crossref Serafeim Chatzopoulos 2023-09-04 16:04:41 +0300
2caaaec42d Include SparkCleanRelation logic in SparkPropagateRelation SparkPropagateRelation includes merge relations Revised tests for SparkPropagateRelation Giambattista Bloisi 2023-09-01 09:32:57 +0200
964c2f553e Changes in indicators step, monitor step dimitrispie 2023-09-01 10:57:02 +0300
6cc7d8ca7b GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob Giambattista Bloisi 2023-08-24 21:48:07 +0200
488d9a1cea Merge pull request 'Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb' (#331) from consistencywf_memoryoverhead_conf into beta Claudio Atzori 2023-08-29 16:31:36 +0200
6b1c05d118 Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb Giambattista Bloisi 2023-08-29 16:04:19 +0200
bf35280ea6 code formatting Claudio Atzori 2023-08-29 11:11:00 +0200
0515d81c7c Merge pull request 'Rewrite SparkPropagateRelation exploiting Dataframe API' (#330) from propagate_relation_rewrite into beta Claudio Atzori 2023-08-29 10:47:14 +0200
58665a246c Merge branch 'beta' into propagate_relation_rewrite Claudio Atzori 2023-08-29 10:47:02 +0200
f437be80ad [impact indicators] adjusted paths in the bip ranker wf parameters Claudio Atzori 2023-08-29 09:03:03 +0200
d012aec0b3 Revert PropagateRelation's argument name from outputPath to graphOutputPath in consistency workflow (#8964) Giambattista Bloisi 2023-08-28 22:44:54 +0200
a860e19423 Fix ensure all relations are written out, not only those managed by dedup Giambattista Bloisi 2023-08-28 15:36:02 +0200
0d7b2bf83d Rewrite SparkPropagateRelation exploiting Dataframe API Giambattista Bloisi 2023-08-28 10:34:54 +0200
9c8b41475a Merge pull request '8172_impact_indicators_workflow' (#284) from 8172_impact_indicators_workflow into beta Miriam Baglioni 2023-08-14 15:50:48 +0200
97c1ba8918 Merge actionsets of results and projects Serafeim Chatzopoulos 2023-08-11 15:56:53 +0300
35b8deb2c6 Merge pull request 'DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag' (#329) from dispatch_filter_invisible_entities into beta Miriam Baglioni 2023-08-10 12:56:18 +0200
95cd2b9b1e Make filterInvisible a mandatory parameter of DispathEntitiesSparkJob Make filterInvisible a mandatory parameter of both dedup/consistency and graph/group oozie workflows Giambattista Bloisi 2023-08-10 11:53:48 +0200
fab9920271 DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag Giambattista Bloisi 2023-08-08 15:52:20 +0200
599828ce35 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Miriam Baglioni 2023-08-09 13:07:13 +0200
c25ac21e5e Merge pull request 'graph cleaning, suggestions from ticket 8898' (#325) from cleaning_8898 into beta Miriam Baglioni 2023-08-08 11:14:19 +0200
c334fe2438 Merge pull request 'Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleted by inference or that are pointing to dangling entities' (#328) from cleanup_relations_after_dedup into beta Miriam Baglioni 2023-08-08 09:49:12 +0200
0e2f855807 Merge pull request 'Updates Promotion DBs' (#321) from antonis.lempesis/dnet-hadoop:beta into beta Miriam Baglioni 2023-08-07 12:09:16 +0200
18fbe52b20 Merge pull request 'Import affiliation relations from Crossref' (#320) from 8876 into beta Miriam Baglioni 2023-08-07 10:45:30 +0200
97b6d1dc45 Filter ids by dataInfo.deletedbyinference and DataInfo.invisible flags Filter relations also by dataInfo.invisible flag Giambattista Bloisi 2023-08-07 10:24:11 +0200
af49424b59 Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleyted by inference or that are pointing to dangling entities Giambattista Bloisi 2023-08-04 14:27:39 +0200
0bc74e2000 code formatting Claudio Atzori 2023-08-02 11:52:10 +0200
7180911ded [graph cleaning] fixed regex behaviour for cleaning ROR and GRID identifiers, added tests Claudio Atzori 2023-06-23 16:10:49 +0200
b9dddbfe54 rule out records with NULL dataInfo, except for Relations Claudio Atzori 2023-07-31 17:52:56 +0200
da1727f93f rule out records with NULL dataInfo, except for Relations Claudio Atzori 2023-07-31 17:52:56 +0200
11ffb9bd68 rule out records with NULL dataInfo Claudio Atzori 2023-07-31 12:35:05 +0200
ccac6a7f75 rule out records with NULL dataInfo Claudio Atzori 2023-07-31 12:35:05 +0200
7cefe2665b Remove unnecessary classes Serafeim Chatzopoulos 2023-07-28 19:14:39 +0300
26a92ce762 Merge branch '8876' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8876 Serafeim Chatzopoulos 2023-07-28 19:03:57 +0300
ebfba38ab6 Add changes from code review Serafeim Chatzopoulos 2023-07-28 19:03:47 +0300
eb8684a8cf Merge branch 'beta' into 8876 Serafeim Chatzopoulos 2023-07-28 13:39:33 +0200
1275a07d45 Merge pull request '[graph indexing] expand the instance level fulltext in the XML records' (#326) from instance_fulltext_xml into beta Claudio Atzori 2023-07-27 15:02:07 +0200
a72b9e96ac expand the instance level fulltext in the XML records Claudio Atzori 2023-07-27 14:57:38 +0200
d512df8612 code formatting Claudio Atzori 2023-07-26 09:14:08 +0200
d8435a6512 inverted condition Claudio Atzori 2023-07-25 17:39:57 +0200
59764145bb cherry picked & fixed commit 270df939c4 Claudio Atzori 2023-07-25 17:39:00 +0200
270df939c4 partial implementation of the suggestions from https://support.openaire.eu/issues/8898 Claudio Atzori 2023-07-25 17:29:50 +0200
8c63e4a864 Merge pull request 'Refactor Dedup using Spark Dataframe API, initial support for scala 2.12 and Spark 3.4' (#324) from dedup-with-dataframe-2 into beta Claudio Atzori 2023-07-25 10:17:17 +0200
e64c2854a3 Refactor Dedup process to use Spark Dataframe API and intermediate representation with Row interface JsonPath cache contention fixed by using a ConcurrentHashMap Blacklist filtering performance improvement Minor performance improvements when evaluating similarity Sorting in clustered elements is deterministic (by ordering and identity field, instead of ordering field only) Giambattista Bloisi 2023-07-18 11:38:56 +0200
bb5b845e3c Use scala.binary.version property to resolve scala maven dependencies Ensure consistent usage of maven properties Profile for compiling with scala 2.12 and Spark 3.4 Giambattista Bloisi 2023-07-17 17:18:46 +0200
002b24e06f Merge pull request '[graph cleaning] fixed regex behaviour for cleaning ROR and GRID identifiers, added tests' (#315) from pid_cleaning into beta Claudio Atzori 2023-07-24 10:49:44 +0200
c754397a19 Merge branch 'beta' into pid_cleaning Claudio Atzori 2023-07-24 10:49:31 +0200
f0678cda09 Merge pull request 'fix_beta_tests' (#323) from fix_beta_tests into beta Claudio Atzori 2023-07-24 10:47:35 +0200
3a0f09774a Add script to find score limits Serafeim Chatzopoulos 2023-07-21 17:55:41 +0300
06b9b71c4e Merge branch '8172_impact_indicators_workflow' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8172_impact_indicators_workflow Ilias Kanellos 2023-07-21 17:42:49 +0300
2374f445a9 Produce additional bip update specific files Ilias Kanellos 2023-07-21 17:42:46 +0300
cb0f3c50f6 Format workflow.xml Serafeim Chatzopoulos 2023-07-21 16:07:10 +0300
c64e5e588f Merge branch '8172_impact_indicators_workflow' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8172_impact_indicators_workflow Serafeim Chatzopoulos 2023-07-21 15:27:02 +0300
2cc5b1a39b Fixes in workflow.xml Serafeim Chatzopoulos 2023-07-21 15:26:50 +0300
0f96af5d56 Merge branch '8172_impact_indicators_workflow' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8172_impact_indicators_workflow Ilias Kanellos 2023-07-21 13:42:35 +0300
03da965162 Format bip-score based file without doi references Ilias Kanellos 2023-07-21 13:42:30 +0300
f03153823a Update testCitationRelations number of expected citations according to changes made in 0559d8b4 (monodirectional citations) Giambattista Bloisi 2023-07-21 10:48:28 +0200
54c1eacef1 SparkJobTest was failing because testing workingdir was not cleaned up after eact test Giambattista Bloisi 2023-07-21 10:42:24 +0200
5e15f20e6e Fix entityMerger that was excluding the authors of the first entity in the list to merge Giambattista Bloisi 2023-07-21 00:46:54 +0200
0210a14e43 Ignore timestamp differences in PromoteActionPayloadForGraphTableJobTest Giambattista Bloisi 2023-07-20 23:45:57 +0200
dba34505de Fix SparkStatsTest bug where parquet tables were incorrectly read as text files leading to unpredictable count() values Giambattista Bloisi 2023-07-19 14:24:52 +0200
e47ed1fdb2 Use DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES in json mapper to avoid that tests fail if they encounter unmapped properties Giambattista Bloisi 2023-07-19 14:21:40 +0200
38dfebfbe6 Disable MdStoreClientTest test as it requires a local mongodb running and it does not perform any assertions Giambattista Bloisi 2023-07-19 14:18:56 +0200
9e8e39f78a - Miriam Baglioni 2023-07-19 11:35:58 +0200
373a5f2c83 Merge pull request 'Master branch updates from beta July 2023' (#317) from master_july23 into master Claudio Atzori 2023-07-18 18:22:04 +0200
db4ca43ee8 Resolve conflict Serafeim Chatzopoulos 2023-07-18 18:38:26 +0300
be320ba3c1 Indentation fixes Serafeim Chatzopoulos 2023-07-17 16:04:21 +0300
be4856ef35 Update step15.sql dimitrispie 2023-07-17 15:33:58 +0300
bc1a4611aa Minor changes Serafeim Chatzopoulos 2023-07-17 11:17:53 +0300
8af129b0c7 merged stats promotion step from antonis/promotion-prod-only Claudio Atzori 2023-07-13 15:03:28 +0200
706092bc19 Update updateProductionViews.sh dimitrispie 2023-07-13 15:48:12 +0300
aedd279f78 Updates Promotion DBs dimitrispie 2023-07-12 22:49:08 +0300
163b2ee2a8 Changes dimitrispie 2023-07-13 15:25:00 +0300
76901a25f9 Updates Promotion DBs dimitrispie 2023-07-12 22:49:08 +0300
ef493681d9 Merge pull request 'Import dnet-pace-core module in this project and use it after renaming to dhp-pace-core' (#319) from beta_with_pace_core into beta Giambattista Bloisi 2023-07-11 14:03:15 +0200
4eba14a80e Add oozie workflow Serafeim Chatzopoulos 2023-07-06 21:07:50 +0300
c2998a14e8 Add basic tests for affiliation relations Serafeim Chatzopoulos 2023-07-06 20:28:16 +0300
bc7b00bcd1 Add bi-directional affiliation relations Serafeim Chatzopoulos 2023-07-06 18:29:15 +0300
12528ed2ef Refactor PrepareAffiliationRelations.java to use OafMapperUtils common functions Serafeim Chatzopoulos 2023-07-06 18:08:33 +0300
bbc245696e Prepare actionsets for BIP affiliations Serafeim Chatzopoulos 2023-07-06 15:56:12 +0300
0c433eccdd Fix scores & Workflow Ilias Kanellos 2023-07-06 15:06:28 +0300