Labels Milestones

Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12 #327

Open

giambattista.bloisi wants to merge 12 commits from spark34-integration into beta

pull from: spark34-integration

merge into: D-Net:beta

D-Net:beta

D-Net:misc_fixes_merge_entities

D-Net:beta-release-1.2.5

D-Net:WebCrowlBeta

D-Net:WebCrowl

D-Net:master

D-Net:provision_memoryOverhead

D-Net:stats_step16_fix

D-Net:doidoost_dismiss

D-Net:CrossrefFundersMap

D-Net:taggingProjects

D-Net:9647_datacite_affiliations

D-Net:UsageStatsRecordDS

D-Net:mergeutils

D-Net:oaf_country_beta

D-Net:index_records

D-Net:ocnew

D-Net:FOSNew

D-Net:bulkTaggingPathMapExtention

D-Net:transformativeagreement

D-Net:new_orcid_enhancement

D-Net:9559_DBLP_data

D-Net:base_stats_job_deprecated

D-Net:SWH_issue_377

D-Net:import_orps_fix

D-Net:spark_join_param_tuning

D-Net:crossref_mapping_vocabularies

D-Net:promote_actions_join_type_master

D-Net:promote_actions_join_type

D-Net:provision_community_api

D-Net:enrichmentSingleStepFixed

D-Net:fosPreparationBeta

D-Net:resource_types

D-Net:enrichmentSingleStep

D-Net:oldPropagationOrganizationCommunity

D-Net:beta_to_master_dicember2023

D-Net:orcid_import

D-Net:9078_xml_records_irish_tender

D-Net:clean_license_publisher

D-Net:bulkTag

D-Net:SWH_integration

D-Net:importpoci

D-Net:8172_impact_indicators_workflow

D-Net:dedup-with-dataframe-spark34

D-Net:8876

D-Net:master_july23

D-Net:distinct_pids_from_openorgs_beta

D-Net:propagationProjectThroughParentChils

D-Net:fulltext_url_validation

D-Net:removeTaggingCondition

D-Net:ticket_8369

D-Net:tweaking_spark_params

D-Net:fc4e-rsac

D-Net:doiboost_authormerger

D-Net:beta_dedup_configuration

D-Net:apc_affiliation

D-Net:bulkTagRefactor

D-Net:organizationToRepresentative

D-Net:graph_cleaning_refactoring

D-Net:scholix_flat_indexing

D-Net:scholix_data_type_openaire

D-Net:advConstraintsInBeta

D-Net:doiboostMappingExtention

D-Net:mag_citation_relation

D-Net:h2020classification

D-Net:doiboostFunderExtention

D-Net:citations_monodirectional

D-Net:compatibility_order

D-Net:8232-mdstore-synch-improve

D-Net:subjectPropagation

D-Net:pubmed_to_production

D-Net:cleanCountryOnMaster

D-Net:graph_cleaning

D-Net:deduptesting

D-Net:horizontalConstraints

D-Net:enrichment

D-Net:scholix_to_solr

D-Net:transformation_wf

D-Net:discard-non-wellformed

D-Net:removeDump

D-Net:eosc_context_tagging

D-Net:pubmed_update

D-Net:doiboost_refactor

D-Net:clean_context_master

D-Net:monitoring

D-Net:dump_new_funded_products

D-Net:dump_delta_projects

D-Net:dump

D-Net:7096-fileGZip-collector-plugin

D-Net:oaf_relation_mapping

D-Net:validation

D-Net:native_records_migration

D-Net:instance_group_by_url

D-Net:hostedByMap_update

D-Net:hostedByMap_oastartdate

D-Net:sygma_indexing

Conversation 0 Commits 12 Files Changed 45

giambattista.bloisi commented 9 months ago

Collaborator

This is a work in progress to upgrade this project in order to use Spark 3.4.x ands scala 2.12
At the moment the project builds normally using Spark 2.4.x and Scala 2.11, but still require a few workarounds (Patched Spark 3.4, code changes for spark-solr library usage) to complete successfully builds and tests of all modules.

This is a work in progress to upgrade this project in order to use Spark 3.4.x ands scala 2.12 At the moment the project builds normally using Spark 2.4.x and Scala 2.11, but still require a few workarounds (Patched Spark 3.4, code changes for spark-solr library usage) to complete successfully builds and tests of all modules.

miriam.baglioni changed title from WIP Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12 to WIP: Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12 9 months ago

giambattista.bloisi force-pushed spark34-integration from c13df9d6c3 to 81ab6a3991 8 months ago

sandro.labruzzo added 1 commit 7 months ago

ada06d1176 added repository openaire-third-parties

sandro.labruzzo added 1 commit 7 months ago

8064abf86c formatted code

sandro.labruzzo added 1 commit 7 months ago

e21b65cabb used javax.xml.stream.XMLEventReader instead of deprecated scala.xml.pull.XMLEventReader

giambattista.bloisi added 1 commit 7 months ago

727ccbe575 Add profiles for different spark versions: spark-24, spark-34, spark-35

giambattista.bloisi added 1 commit 7 months ago

e826591897 Format scala code

claudio.atzori added this to the OpenAIRE project 7 months ago

claudio.atzori modified the project from OpenAIRE to OpenAIRE - DNet 6 months ago

giambattista.bloisi force-pushed spark34-integration from e826591897 to 613ec5ffce 5 months ago

giambattista.bloisi changed title from WIP: Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12 to Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12 5 months ago

sandro.labruzzo added 1 commit 1 week ago

342cb6189b fixed problem on changed signature on RowEncoder ...

removed property dhp.schema.artifact

sandro.labruzzo added 1 commit 1 week ago

8dd9cf84e2 code formatted

sandro.labruzzo added 1 commit 1 week ago

b84ad0c06e merged beta

sandro.labruzzo added 1 commit 6 days ago

073f320c6a Added module containing all the dependencies, useful for spark deploy on k8.

sandro.labruzzo added 1 commit 5 days ago

0d628cd62b merged again from beta

sandro.labruzzo added 1 commit 2 days ago

9cd3bc0f10 Added a new generation of the dump for scholexplorer tested with last version of spark, and strongly refactored

sandro.labruzzo added 1 commit 2 days ago

052c6aac9d formatted code

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.

git checkout -b spark34-integration beta

git pull origin spark34-integration

Step 2:

Merge the changes and update on Gitea.

git checkout beta

git merge --no-ff spark34-integration

git push origin beta

Sign in to join this conversation.

Reviewers

Request review

No reviewers

Labels

Apply labels

Clear labels   

bug

Something is not working   

duplicate

This issue or pull request already exists   

enhancement

New feature / refactoring   

help wanted

Need some help   

invalid

Something is wrong   

question

More information is needed   

RDGraph

EOSC Research Discovery Graph   

RSAC

EOSC Research Software APIs and Connectors   

wontfix

This won't be fixed

No Label

bug

duplicate

enhancement

help wanted

invalid

question

RDGraph

RSAC

wontfix

Milestone

Set milestone

Clear milestone

No items

No Milestone

Projects

Set Project

Clear projects

No project

OpenAIRE - DNet

Assignees

Assign users

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#327

Write Preview

Loading…

Cancel

Save

There is no content yet.