dnet-hadoop

lsmyrnaios

dnet-hadoop

forked from D-Net/dnet-hadoop

Fork 0

Commit Graph

Select branches

Hide Pull Requests

7096-fileGZip-collector-plugin

8172_impact_indicators_workflow

8232-mdstore-synch-improve

8876

9078_xml_records_irish_tender

SWH_integration

UsageStatsRecordDS

advConstraintsInBeta

apc_affiliation

beta

beta_dedup_configuration

bulkTag

bulkTagRefactor

citations_monodirectional

cleanCountryOnMaster

clean_context_master

clean_license_publisher

compatibility_order

continuous-validation

continuous_validation2

dedup-with-dataframe-spark34

dedup_increasenumofblocks

deduptesting

discard-non-wellformed

distinct_pids_from_openorgs_beta

doiboostFunderExtention

doiboostMappingExtention

doiboost_authormerger

doiboost_refactor

dump

dump_delta_projects

dump_new_funded_products

enrichment

eosc_context_tagging

fc4e-rsac

fulltext_url_validation

graph_cleaning

graph_cleaning_refactoring

h2020classification

horizontalConstraints

hostedByMap_oastartdate

hostedByMap_update

importpoci

instance_group_by_url

mag_citation_relation

master

master_july23

monitoring

native_records_migration

oaf_relation_mapping

orcid_import

organizationToRepresentative

propagationProjectThroughParentChils

pubmed_to_production

pubmed_update

removeDump

removeTaggingCondition

scholix_data_type_openaire

scholix_flat_indexing

scholix_to_solr

spark34-integration

subjectPropagation

sygma_indexing

ticket_8369

transformation_wf

tweaking_spark_params

updateCopyScripts

validation

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

archive/master_pre_stable_ids

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

september-2023

42a2dad975 implemented relation to irish funder from a Json list Sandro La Bruzzo 2023-10-06 11:52:33 +0200
13f332ce77 ignored jenv prop Sandro La Bruzzo 2023-10-06 10:40:05 +0200
1bb83b9188 Add prefix in SWH ID Serafeim Chatzopoulos 2023-10-04 20:31:45 +0300
ee8a39e7d2 cleanup and refinements importpoci Claudio Atzori 2023-10-04 12:32:05 +0200
e9f24df21c Move SWH API Key from constants to workflow param Serafeim Chatzopoulos 2023-10-03 20:57:57 +0300
cae75fc75d Add SWH in the collectedFrom field Serafeim Chatzopoulos 2023-10-03 16:55:10 +0300
b49a3ac9b2 Add actionsetsPath as a global WF param Serafeim Chatzopoulos 2023-10-03 15:43:38 +0300
24c43e0c60 Restructure workflow parameters Serafeim Chatzopoulos 2023-10-03 15:11:58 +0300
9f73d93e62 Add param for limiting repo Urls Serafeim Chatzopoulos 2023-10-03 14:39:08 +0300
b446a9ed98 Merge branch 'beta' into peer_reviewed Claudio Atzori 2023-10-03 10:52:23 +0200
f344ad76d0 Merge pull request 'extended existing code to import of POCI from open citation' (#340) from importpoci into beta Claudio Atzori 2023-10-03 10:52:11 +0200
5919e488dd Merge branch 'beta' into importpoci Claudio Atzori 2023-10-03 10:43:53 +0200
839a8524e7 Add action for creating actionsets Serafeim Chatzopoulos 2023-10-02 23:50:38 +0300
c9a5ad6a02 extending the coverage of the peer non-unknown refereed instances Claudio Atzori 2023-10-02 16:28:42 +0200
d7fccdc64b fixed paths in wf to match the req of the pathname Miriam Baglioni 2023-10-02 14:10:57 +0200
9898470b0e Addressing comments in D-Net/dnet-hadoop#340\#issuecomment-10592 Miriam Baglioni 2023-10-02 12:54:16 +0200
c412dc162b Fix bug in conversion from dedup json model to Spark Dataset of Rows: list of strings contained the json escaped representation of the value instead of the plain value, this caused instanceTypeMatch failures because of the leading and trailing double quotes Giambattista Bloisi 2023-09-27 22:30:47 +0200
4ac06c9e37 Merge pull request 'Fix bug in conversion from dedup json model to Spark Dataset of Rows (instanceTypeMatch no longer working)' (#339) from fix_dedupfailsonmatchinginstances into master Claudio Atzori 2023-10-02 11:34:20 +0200
fa692b3629 Merge branch 'master' into fix_dedupfailsonmatchinginstances Claudio Atzori 2023-10-02 11:28:16 +0200
5d09b7db8b Merge pull request 'SparkPropagateRelation relations do not propagate deletedByInference and invisible' (#333) from consistency_keep_mergerels into beta Claudio Atzori 2023-10-02 11:27:57 +0200
7b403a920f Merge branch 'beta' into consistency_keep_mergerels Claudio Atzori 2023-10-02 11:26:00 +0200
dc86018a5f Merge branch 'merge_entities_job' into beta Claudio Atzori 2023-10-02 11:24:48 +0200
3c47920c78 Use asScala to convert java List to Scala Sequence Giambattista Bloisi 2023-09-20 16:14:01 +0200
7f244d9a7a code formatting Claudio Atzori 2023-09-20 15:53:21 +0200
e239b81740 Fix defect #8997: GenerateEventsJob is generating huge amounts of logs because broker entity similarity calculation consistently failed Giambattista Bloisi 2023-09-20 15:42:00 +0200
ef02648399 Merge pull request 'fixed dedup configuration management in the Broker workflow' (#341) from fix_8997 into master Claudio Atzori 2023-10-02 11:03:50 +0200
d13bb534f0 Merge branch 'master' into fix_8997 Claudio Atzori 2023-10-02 11:03:18 +0200
e84f5b5e64 extended existing codo to accomodate import of POCI from open citation Miriam Baglioni 2023-10-02 09:25:16 +0200
ab0d70691c Add step for archiving repoUrls to SWH Serafeim Chatzopoulos 2023-09-28 20:56:18 +0300
775c3f704a Fix bug in conversion from dedup json model to Spark Dataset of Rows: list of strings contained the json escaped representation of the value instead of the plain value, this caused instanceTypeMatch failures because of the leading and trailing double quotes Giambattista Bloisi 2023-09-27 22:30:47 +0200
ed9c81a0b7 Add steps to collect last visit data && archive not found repository URLs Serafeim Chatzopoulos 2023-09-27 19:00:54 +0300
9c3ab11d5b Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop Sandro La Bruzzo 2023-09-25 15:29:19 +0200
423ef30676 minor fix on the aggregation of uniprot and pdb Sandro La Bruzzo 2023-09-25 15:28:58 +0200
7152d47f84 Use asScala to convert java List to Scala Sequence Giambattista Bloisi 2023-09-20 16:14:01 +0200
4853c19b5e code formatting Claudio Atzori 2023-09-20 15:53:21 +0200
1f226d1dce Fix defect #8997: GenerateEventsJob is generating huge amounts of logs because broker entity similarity calculation consistently failed Giambattista Bloisi 2023-09-20 15:42:00 +0200
0935d7757c Use v5 of the UNIBI Gold ISSN list in test Alessia Bardi 2023-09-19 14:47:01 +0200
cc7204a089 tests for d4science catalog Alessia Bardi 2023-09-19 13:38:25 +0200
76476cdfb6 Added maven repo for dependencies that are not in maven central Sandro La Bruzzo 2023-09-20 10:33:14 +0200
6186cdc2cc Use v5 of the UNIBI Gold ISSN list in test Alessia Bardi 2023-09-19 14:47:01 +0200
d94b9bebf7 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Alessia Bardi 2023-09-19 13:38:45 +0200
19abba8fa7 tests for d4science catalog Alessia Bardi 2023-09-19 13:38:25 +0200
9ef971a146 Update step16-createIndicatorsTables.sql dimitrispie 2023-09-19 14:25:42 +0300
9d44418d38 Add collecting software code repository URLs Serafeim Chatzopoulos 2023-09-14 18:43:25 +0300
395a4af020 Run CC and RAM sequentieally in dhp-impact-indicators WF Serafeim Chatzopoulos 2023-09-12 22:31:50 +0300
c2f179800c Merge pull request 'Run CC and RAM sequentieally in dhp-impact-indicators WF' (#338) from run_cc_and_ram_sequentially into master Claudio Atzori 2023-09-13 08:52:53 +0200
2aed5a74be Run CC and RAM sequentieally in dhp-impact-indicators WF Serafeim Chatzopoulos 2023-09-12 22:31:50 +0300
8a6892cc63 [graph dedup] consistency wf should not remove the relations while dispatching the entities Claudio Atzori 2023-09-12 14:34:28 +0200
4dc4862011 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Claudio Atzori 2023-09-12 14:34:34 +0200
dc80ab14d3 [graph dedup] consistency wf should not remove the relations while dispatching the entities Claudio Atzori 2023-09-12 14:34:28 +0200
77a2199837 updated test for EOSC comunity september-2023 Alessia Bardi 2023-09-08 11:05:49 +0200
4786aa0e09 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map Claudio Atzori 2023-09-07 11:20:35 +0200
265180bfd2 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map Claudio Atzori 2023-09-07 11:20:35 +0200
5f90cc11e9 Update step16-createIndicatorsTables.sql dimitrispie 2023-09-06 14:14:38 +0300
da0e9828f7 resolved conflicts for PR#337 Claudio Atzori 2023-09-06 11:28:46 +0200
9f5d16624c Merge pull request '[graph raw] datainfo.invisible set as true only for entities' (#336) from invisible_relations into beta Claudio Atzori 2023-09-04 16:14:47 +0200
adec6692ca Merge branch 'beta' into invisible_relations Claudio Atzori 2023-09-04 16:13:06 +0200
15666e86a8 added collectedfrom to the affiliation relations imported from Crossref Claudio Atzori 2023-09-04 15:56:06 +0200
7d6bd4f20b Merge pull request 'Fix import of affiliations relations from Crossref' (#335) from 8876_fix_crossref_affiliation_relations_import into beta Claudio Atzori 2023-09-04 15:19:58 +0200
5b06c9d06f [graph raw] datainfo.invisible set as true only for entities Claudio Atzori 2023-09-04 15:15:24 +0200
7de0164c26 Fix import of affiliations relations from Crossref Serafeim Chatzopoulos 2023-09-04 16:04:41 +0300
2caaaec42d Include SparkCleanRelation logic in SparkPropagateRelation SparkPropagateRelation includes merge relations Revised tests for SparkPropagateRelation Giambattista Bloisi 2023-09-01 09:32:57 +0200
964c2f553e Changes in indicators step, monitor step dimitrispie 2023-09-01 10:57:02 +0300
6cc7d8ca7b GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob Giambattista Bloisi 2023-08-24 21:48:07 +0200
488d9a1cea Merge pull request 'Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb' (#331) from consistencywf_memoryoverhead_conf into beta Claudio Atzori 2023-08-29 16:31:36 +0200
6b1c05d118 Add sparkExecutorMemoryOverhead workflow config to set off-heap memory for Spark actions. If not explicitly set it is defaulted to 1Gb Giambattista Bloisi 2023-08-29 16:04:19 +0200
bf35280ea6 code formatting Claudio Atzori 2023-08-29 11:11:00 +0200
0515d81c7c Merge pull request 'Rewrite SparkPropagateRelation exploiting Dataframe API' (#330) from propagate_relation_rewrite into beta Claudio Atzori 2023-08-29 10:47:14 +0200
58665a246c Merge branch 'beta' into propagate_relation_rewrite Claudio Atzori 2023-08-29 10:47:02 +0200
f437be80ad [impact indicators] adjusted paths in the bip ranker wf parameters Claudio Atzori 2023-08-29 09:03:03 +0200
d012aec0b3 Revert PropagateRelation's argument name from outputPath to graphOutputPath in consistency workflow (#8964) Giambattista Bloisi 2023-08-28 22:44:54 +0200
a860e19423 Fix ensure all relations are written out, not only those managed by dedup Giambattista Bloisi 2023-08-28 15:36:02 +0200
0d7b2bf83d Rewrite SparkPropagateRelation exploiting Dataframe API Giambattista Bloisi 2023-08-28 10:34:54 +0200
9c8b41475a Merge pull request '8172_impact_indicators_workflow' (#284) from 8172_impact_indicators_workflow into beta Miriam Baglioni 2023-08-14 15:50:48 +0200
97c1ba8918 Merge actionsets of results and projects 8172_impact_indicators_workflow Serafeim Chatzopoulos 2023-08-11 15:56:53 +0300
35b8deb2c6 Merge pull request 'DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag' (#329) from dispatch_filter_invisible_entities into beta Miriam Baglioni 2023-08-10 12:56:18 +0200
95cd2b9b1e Make filterInvisible a mandatory parameter of DispathEntitiesSparkJob Make filterInvisible a mandatory parameter of both dedup/consistency and graph/group oozie workflows Giambattista Bloisi 2023-08-10 11:53:48 +0200
fab9920271 DispatchEntitiesSparkJob: manage all entity types together, support filtering by dataInfo.invisible flag Giambattista Bloisi 2023-08-08 15:52:20 +0200
599828ce35 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop Miriam Baglioni 2023-08-09 13:07:13 +0200
c25ac21e5e Merge pull request 'graph cleaning, suggestions from ticket 8898' (#325) from cleaning_8898 into beta Miriam Baglioni 2023-08-08 11:14:19 +0200
c334fe2438 Merge pull request 'Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleted by inference or that are pointing to dangling entities' (#328) from cleanup_relations_after_dedup into beta Miriam Baglioni 2023-08-08 09:49:12 +0200
0e2f855807 Merge pull request 'Updates Promotion DBs' (#321) from antonis.lempesis/dnet-hadoop:beta into beta Miriam Baglioni 2023-08-07 12:09:16 +0200
18fbe52b20 Merge pull request 'Import affiliation relations from Crossref' (#320) from 8876 into beta Miriam Baglioni 2023-08-07 10:45:30 +0200
97b6d1dc45 Filter ids by dataInfo.deletedbyinference and DataInfo.invisible flags Filter relations also by dataInfo.invisible flag Giambattista Bloisi 2023-08-07 10:24:11 +0200
af49424b59 Add a "CleanRelation" action after the PropagateRelation to filter out all relations that have been deleyted by inference or that are pointing to dangling entities Giambattista Bloisi 2023-08-04 14:27:39 +0200
edf5a780b8 minor fix dedup-with-dataframe-spark34 Sandro La Bruzzo 2023-08-02 12:12:20 +0200
0bc74e2000 code formatting Claudio Atzori 2023-08-02 11:52:10 +0200
7180911ded [graph cleaning] fixed regex behaviour for cleaning ROR and GRID identifiers, added tests Claudio Atzori 2023-06-23 16:10:49 +0200
b9dddbfe54 rule out records with NULL dataInfo, except for Relations Claudio Atzori 2023-07-31 17:52:56 +0200
da1727f93f rule out records with NULL dataInfo, except for Relations Claudio Atzori 2023-07-31 17:52:56 +0200
11ffb9bd68 rule out records with NULL dataInfo Claudio Atzori 2023-07-31 12:35:05 +0200
ccac6a7f75 rule out records with NULL dataInfo Claudio Atzori 2023-07-31 12:35:05 +0200
7cefe2665b Remove unnecessary classes 8876 Serafeim Chatzopoulos 2023-07-28 19:14:39 +0300
26a92ce762 Merge branch '8876' of https://code-repo.d4science.org/D-Net/dnet-hadoop into 8876 Serafeim Chatzopoulos 2023-07-28 19:03:57 +0300
ebfba38ab6 Add changes from code review Serafeim Chatzopoulos 2023-07-28 19:03:47 +0300
eb8684a8cf Merge branch 'beta' into 8876 Serafeim Chatzopoulos 2023-07-28 13:39:33 +0200
1275a07d45 Merge pull request '[graph indexing] expand the instance level fulltext in the XML records' (#326) from instance_fulltext_xml into beta Claudio Atzori 2023-07-27 15:02:07 +0200
a72b9e96ac expand the instance level fulltext in the XML records Claudio Atzori 2023-07-27 14:57:38 +0200
d512df8612 code formatting Claudio Atzori 2023-07-26 09:14:08 +0200
d8435a6512 inverted condition Claudio Atzori 2023-07-25 17:39:57 +0200