dnet-hadoop

lsmyrnaios

dnet-hadoop

forked from D-Net/dnet-hadoop

Fork 0

Commit Graph

Select branches

Hide Pull Requests

7096-fileGZip-collector-plugin

8172_impact_indicators_workflow

8232-mdstore-synch-improve

8876

9078_xml_records_irish_tender

SWH_integration

UsageStatsRecordDS

advConstraintsInBeta

apc_affiliation

beta

beta_dedup_configuration

bulkTag

bulkTagRefactor

citations_monodirectional

cleanCountryOnMaster

clean_context_master

clean_license_publisher

compatibility_order

continuous-validation

continuous_validation2

dedup-with-dataframe-spark34

dedup_increasenumofblocks

deduptesting

discard-non-wellformed

distinct_pids_from_openorgs_beta

doiboostFunderExtention

doiboostMappingExtention

doiboost_authormerger

doiboost_refactor

dump

dump_delta_projects

dump_new_funded_products

enrichment

eosc_context_tagging

fc4e-rsac

fulltext_url_validation

graph_cleaning

graph_cleaning_refactoring

h2020classification

horizontalConstraints

hostedByMap_oastartdate

hostedByMap_update

importpoci

instance_group_by_url

mag_citation_relation

master

master_july23

monitoring

native_records_migration

oaf_relation_mapping

orcid_import

organizationToRepresentative

propagationProjectThroughParentChils

pubmed_to_production

pubmed_update

removeDump

removeTaggingCondition

scholix_data_type_openaire

scholix_flat_indexing

scholix_to_solr

spark34-integration

subjectPropagation

sygma_indexing

ticket_8369

transformation_wf

tweaking_spark_params

updateCopyScripts

validation

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

archive/master_pre_stable_ids

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

september-2023

0139f23d66 Merge pull request 'organization type from OpenOrgs' (#445) from import_openorg_type into beta Claudio Atzori 2024-06-07 12:17:31 +0200
c726572418 changed some parameters in OSF test Michele Artini 2024-06-07 12:03:26 +0200
ec79405cc9 [graph raw] set organization type from openorgs Claudio Atzori 2024-06-07 11:30:31 +0200
1477406ecc [bulkTag] fixed issue that made project disappear in graph_10_enriched Miriam Baglioni 2024-06-05 16:20:40 +0200
92c3abd5a4 [graph cleaning] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-05-29 14:36:49 +0200
ce2364743a applying changes from PR#442: Fix for missing collectedfrom after dedup Claudio Atzori 2024-06-06 10:43:43 +0200
f70dc76b61 minor Claudio Atzori 2024-06-06 10:43:10 +0200
73bd1938a5 [graph2hive] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-06-05 12:17:35 +0200
da5c1e73a4 Merge pull request 'Irish oaipmh exporter' (#443) from irish-oaipmh-exporter into beta Claudio Atzori 2024-06-05 10:55:09 +0200
81090ad593 [IE OAIPHM] added oozie workflow, minor changes, code formatting Claudio Atzori 2024-06-05 10:03:33 +0200
a02f3f0d2b code formatting Claudio Atzori 2024-05-30 10:21:18 +0200
eadfd8d71d Merge pull request 'Updated XMLIterator for splitting on different nodes' (#436) from dblp_collection_plugin into beta Alessia Bardi 2024-05-29 16:05:06 +0200
05ee783c07 Merge branch 'beta' into dblp_collection_plugin Alessia Bardi 2024-05-29 16:04:39 +0200
fe9fb59c90 Merge pull request 'Rest collector plugin on hadoop supports a new param to pass request headers' (#441) from rest-collector-request-header-map into beta Alessia Bardi 2024-05-29 15:54:39 +0200
c272c4ad68 code formatting Claudio Atzori 2024-05-29 15:50:07 +0200
c5f4da16a4 Merge branch 'beta' into rest-collector-request-header-map Alessia Bardi 2024-05-29 15:46:23 +0200
1b165a14a0 Rest collector plugin on hadoop supports a new param to pass request headers Alessia 2024-05-29 15:41:36 +0200
e996787be2 OSF test Michele Artini 2024-05-29 15:05:17 +0200
62716141c5 Merge pull request 'Miscellaneous updates to the copying operation to Impala Cluster' (#440) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-05-29 14:34:51 +0200
5d85b70e1f [NOAMI] removed Ireland funder id 501100011103. ticket 9635 Miriam Baglioni 2024-05-29 11:55:00 +0200
a644a6f4fe Catch Spark-sql errors and show a log with the statement that failed. Lampros Smyrnaios 2024-05-29 12:10:11 +0300
e3f28338c1 Miscellaneous updates to the copying operation to Impala Cluster: - Assign the WRITE and EXECUTE permissions to the DBs' HDFS-directories, in order to be able to create tables on top of them, in the Impala Cluster. - Make sure the "copydb" function returns early, when it encounters a fatal error, while respecting the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" config. Lampros Smyrnaios 2024-05-28 17:51:45 +0300
73316d8c83 Add jaxb and jaxws dependencies when compiling with spark-34 profile as they are required to run with jdk > 8 Giambattista Bloisi 2024-05-28 14:14:51 +0200
75d5ddb999 Update to include a blackList that filters out the results we know are wrongly associated to IE - update workflow definition - the blacklist parameter Miriam Baglioni 2024-05-24 16:01:19 +0200
87c9c61b41 Update to include a blackList that filters out the results we know are wrongly associated to IE - refactoring Miriam Baglioni 2024-05-24 15:23:42 +0200
b55fed09f8 Update to include a blackList that filters out the results we know are wrongly associated to IE Miriam Baglioni 2024-05-24 12:28:24 +0200
107d958b89 [org dedup] avoid NPEs in SparkPrepareNewOrgs Claudio Atzori 2024-05-26 21:23:30 +0200
3a7a6ecc32 [org dedup] avoid NPEs in SparkPrepareOrgRels Claudio Atzori 2024-05-26 16:48:11 +0200
1af4224d3d [org dedup] avoid NPEs in SparkPrepareOrgRels Claudio Atzori 2024-05-26 15:43:24 +0200
0d5bdb2db0 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Claudio Atzori 2024-05-27 11:59:02 +0200
66548e6a83 Merge pull request 'changes in copy script' (#438) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-05-27 11:54:03 +0200
888637773c Add missing "/*EOS*/" comments. Lampros Smyrnaios 2024-05-27 12:34:49 +0300
e0ac494859 Merge branch 'beta' into convert_hive_to_spark_actions Lampros Smyrnaios 2024-05-27 12:27:40 +0300
15b54a345a added fos lvl4 Antonis Lempesis 2024-05-24 13:21:28 +0300
b48ed6e617 Change configuration in the copy-operation to Impala Cluster: Set the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" parameter to "false". Lampros Smyrnaios 2024-05-23 16:58:12 +0300
68322843e2 Small updates to the copy-operation to Impala Cluster: - Add a configuration-"switch" to control whether the script exits upon an error or not. - Allow the script to exit when a table could not be created. - Show the elapsed time for processing each database. Lampros Smyrnaios 2024-05-23 15:07:49 +0300
c7b32bbacc Update CopyDataToImpalaCluster: Update the code of acquiring the entities from Ocean cluster, through hive, in order to optimize the process and account for additional reserved keywords in Impala. Lampros Smyrnaios 2024-05-23 13:00:19 +0300
1b2357e10a Merge pull request 'Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12' (#327) from spark34-integration into beta Giambattista Bloisi 2024-05-23 09:20:28 +0200
f1fe363b19 merged again from beta (I hope for the last time) Sandro La Bruzzo 2024-05-22 11:08:52 +0200
66c1ffc866 merged again from beta (I hope for the last time) Sandro La Bruzzo 2024-05-22 11:02:46 +0200
1ea67eba82 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Claudio Atzori 2024-05-21 13:48:48 +0200
f9fb2fef6e Merge pull request 'Modification of Microsoft Academic Graph Mapping' (#435) from mag_only_doi into beta Claudio Atzori 2024-05-21 13:48:42 +0200
834461ba26 [graph provision]fixed wf definition, revised serialization of the usage counts measures Claudio Atzori 2024-05-21 13:47:05 +0200
e8a61d5dd5 removed plugin, use only FileGZip plugin Sandro La Bruzzo 2024-05-21 13:45:29 +0200
ca9414b737 Implement multiple node name splitter on GZipCollectorPlugin and all nodes that use XMLIterator. If the splitter name contains is a comma separated values it splits for all the values Sandro La Bruzzo 2024-05-21 09:11:13 +0200
032bcc8279 since last beta workflow we decide to introduce in the graph only MAG item with DOI and set them invisible ( this should be the same behaviour of the previous DOIBoost mapping). This commit apply this type of mapping Sandro La Bruzzo 2024-05-20 09:24:15 +0200
103e2652b3 merged beta Sandro La Bruzzo 2024-05-17 14:43:07 +0200
a87f9ea643 fixed scholexplorer bug Sandro La Bruzzo 2024-05-17 14:16:43 +0200
6efab4d88e fixed scholexplorer bug Sandro La Bruzzo 2024-05-16 16:19:18 +0200
92f018d196 [graph provision] fixed path pointing to an intermediate data store in the working directory Claudio Atzori 2024-05-15 15:39:18 +0200
0611c81a2f [graph provision] using Qualifier.classNames to populate the correponsing fields in the JSON payload Claudio Atzori 2024-05-15 15:33:10 +0200
2b3b5fe9a1 oai finalization and test Michele Artini 2024-05-15 14:13:16 +0200
1efe7f7e39 [graph provision] upgrade to dhp-schema:6.1.2, included project.oamandatepublications in the JSON payload mapping, fixed serialisation of the usageCounts measures Claudio Atzori 2024-05-14 12:39:31 +0200
53e7bb4336 Merge pull request 'rest-collector-plugin-with-retry' (#432) from rest-collector-plugin-with-retry into beta Claudio Atzori 2024-05-10 09:02:33 +0200
f7d56e2ef2 Merge branch 'beta' into rest-collector-plugin-with-retry Claudio Atzori 2024-05-10 09:02:21 +0200
c1237ab39e Merge pull request 'Fixes in Graph Provision' (#434) from beta_provision_relation into beta Claudio Atzori 2024-05-09 14:15:05 +0200
dc3a5858f7 Merge branch 'beta' into beta_provision_relation Claudio Atzori 2024-05-09 14:14:43 +0200
55f39f7850 [graph provision] adds the possibility to validate the XML records before storing them via the validateXML parameter Claudio Atzori 2024-05-09 14:06:04 +0200
39a2afe8b5 [graph provision] fixed XML serialization of the usage counts measures, renamed workflow actions to better reflect their role Claudio Atzori 2024-05-09 13:54:42 +0200
908ed9da7a Merge pull request 'Various fixes in the stats wf' (#430) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-05-08 13:41:02 +0200
0cada3cc8f every step is run in the analytics queue. Hardcoded for now, will make a parameter later Antonis Lempesis 2024-05-08 13:42:53 +0300
90a4fb3547 fixed typos Antonis Lempesis 2024-05-08 13:17:58 +0300
18aa323ee9 cleanup unused classes, adjustments in the oozie wf definition Claudio Atzori 2024-05-08 11:36:46 +0200
c9a327bc50 refactoring of gzip method Michele Artini 2024-05-08 11:34:08 +0200
e234848af8 oaf record: xpath for root Michele Artini 2024-05-08 10:00:53 +0200
b4e3389432 fixed property mapping creating the RelatedEntity transient objects. spark cores & memory adjustments. Code formatting Claudio Atzori 2024-05-07 16:25:17 +0200
711048ceed PrepareRelationsJob rewritten to use Spark Dataframe API and Windowing functions Giambattista Bloisi 2024-05-07 15:44:33 +0200
70bf6ac415 oai exporter tests Michele Artini 2024-05-07 09:36:26 +0200
aa40e53c19 oai exporter parameters Michele Artini 2024-05-07 08:01:19 +0200
ed052a3476 job for the population of the oai database Michele Artini 2024-05-06 16:08:33 +0200
26363060ed fixed id prefix creation for the fosnodoi records, again Claudio Atzori 2024-05-03 15:53:52 +0200
0486227185 [cleaning] deactivating the cleaning of FOS subjects found in the metadata provided by repositories Claudio Atzori 2024-05-03 14:31:12 +0200
a5d13d5d27 code formatting Claudio Atzori 2024-05-03 14:14:34 +0200
e1a0fb8933 fixed id prefix creation for the fosnodoi records Claudio Atzori 2024-05-03 14:14:18 +0200
69c5efbd8b Fix: when applying enrichments with no instance information the resulting merge entity was generated with no instance instead of keeping the original information Giambattista Bloisi 2024-05-03 13:57:56 +0200
db358ad0d2 code formatted Sandro La Bruzzo 2024-05-02 15:25:57 +0200
26bf8e763a merged from beta Sandro La Bruzzo 2024-05-02 15:20:23 +0200
a860c57bbc updated .gitignore Sandro La Bruzzo 2024-05-02 15:16:00 +0200
0646d0d064 Updated main sparkApplication to avoid to require master variable Sandro La Bruzzo 2024-05-02 15:15:03 +0200
00ad21d814 Merge pull request 'preparations for dhp-common beta release 1.2.5' (#433) from beta-release-1.2.5 into beta Claudio Atzori 2024-05-02 11:28:19 +0200
4355f64810 reverted to version 1.2.5-SNAPSHOT Claudio Atzori 2024-05-02 11:23:53 +0200
66680b8b9a refactoring of common utilities Claudio Atzori 2024-05-02 11:16:58 +0200
dcf23b3d06 Merge branch 'beta' into beta-release-1.2.5 Claudio Atzori 2024-05-02 10:01:49 +0200
f4068de298 code reindent + tests Michele Artini 2024-05-02 09:51:33 +0200
11bd89e132 [enrichment] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-05-01 08:32:59 +0200
e96c2c1606 [ranking wf] set spark.executor.memoryOverhead to fine tune the resource consumption Claudio Atzori 2024-04-30 16:23:25 +0200
50c18f7a0b [dedup wf] revised memory settings to address the increased volume of input contents Claudio Atzori 2024-04-30 12:34:16 +0200
2615136efc added a retry mechanism Michele Artini 2024-04-30 11:58:42 +0200
133ead1e3e updated new version of scholexplorer Generation Sandro La Bruzzo 2024-04-29 09:00:30 +0200
052c6aac9d formatted code Sandro La Bruzzo 2024-04-26 16:03:04 +0200
9cd3bc0f10 Added a new generation of the dump for scholexplorer tested with last version of spark, and strongly refactored Sandro La Bruzzo 2024-04-26 16:02:07 +0200
c08a58bba8 Merge pull request 'Miscellaneous related to changes in MergeUtils' (#429) from misc_fixes_merge_entities into beta Claudio Atzori 2024-04-24 08:55:37 +0200
e2937db385 Merge branch 'beta' into misc_fixes_merge_entities Claudio Atzori 2024-04-24 08:55:28 +0200
1878199dae Miscellaneous fixes: - in Merge By ID pick by preference those records coming from delegated Authorities - fix various tests - close spark session in SparkCreateSimRels Giambattista Bloisi 2024-04-24 08:12:45 +0200
0d628cd62b merged again from beta Sandro La Bruzzo 2024-04-23 17:34:55 +0200
3c17183d10 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into convert_hive_to_spark_actions Lampros Smyrnaios 2024-04-23 17:18:16 +0300
49af2e5740 Miscellaneous updates to the copying operation to Impala Cluster: - Update the algorithm for creating views that depend on other views; overcome some bash-instabilities. - Upon any error, fail the whole process, not just the current DB-creation, as those errors usually indicate a bug in the initial DB-creation, that should be fixed immediately. - Enhance parallel-copy of large files by "hadoop distcp" command. - Reduce the "invalidate metadata" commands to just the current DB's tables, in order to eliminate the general overhead on Impala. - Show the number of tables and views in the logs. - Fix some log-messages. Lampros Smyrnaios 2024-04-23 17:15:04 +0300
d2649a1429 increased the jvm ram Antonis Lempesis 2024-04-23 16:03:16 +0300
c3053ef34d using version 1.2.5-beta for the release Claudio Atzori 2024-04-23 14:52:32 +0200
b5bcab13ec using version 1.2.5-beta for the release Claudio Atzori 2024-04-23 14:36:39 +0200