Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12 #327

giambattista.bloisi · 2023-08-02T18:12:11+02:00

2023-08-02 18:12:11 +02:00

This is a work in progress to upgrade this project in order to use Spark 3.4.x ands scala 2.12
At the moment the project builds normally using Spark 2.4.x and Scala 2.11, but still require a few workarounds (Patched Spark 3.4, code changes for spark-solr library usage) to complete successfully builds and tests of all modules.

This is a work in progress to upgrade this project in order to use Spark 3.4.x ands scala 2.12 At the moment the project builds normally using Spark 2.4.x and Scala 2.11, but still require a few workarounds (Patched Spark 3.4, code changes for spark-solr library usage) to complete successfully builds and tests of all modules.

miriam.baglioni changed title from ~~WIP Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12~~ to WIP: Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12

2023-08-08 10:44:10 +02:00

giambattista.bloisi force-pushed spark34-integration from c13df9d6c3 to 81ab6a3991

2023-09-07 11:59:36 +02:00

Compare

sandro.labruzzo added 1 commit 2023-09-18 12:51:24 +02:00

ada06d1176 added repository openaire-third-parties

sandro.labruzzo added 1 commit 2023-09-18 12:57:48 +02:00

8064abf86c formatted code

sandro.labruzzo added 1 commit 2023-09-18 13:58:27 +02:00

e21b65cabb used javax.xml.stream.XMLEventReader instead of deprecated scala.xml.pull.XMLEventReader

giambattista.bloisi added 1 commit 2023-09-21 14:26:11 +02:00

727ccbe575 Add profiles for different spark versions: spark-24, spark-34, spark-35

giambattista.bloisi added 1 commit 2023-09-22 15:26:50 +02:00

e826591897 Format scala code

claudio.atzori added this to the OpenAIRE project 2023-10-16 11:26:49 +02:00

claudio.atzori modified the project from OpenAIRE to OpenAIRE - DNet

2023-10-26 09:58:54 +02:00

giambattista.bloisi force-pushed spark34-integration from e826591897 to 613ec5ffce

2023-12-05 19:12:24 +01:00

Compare

giambattista.bloisi changed title from ~~WIP: Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12~~ to Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12

2023-12-05 19:14:17 +01:00

sandro.labruzzo added 1 commit 2024-04-19 12:13:34 +02:00

342cb6189b fixed problem on changed signature on RowEncoder

removed property dhp.schema.artifact

sandro.labruzzo added 1 commit 2024-04-19 12:31:07 +02:00

8dd9cf84e2 code formatted

sandro.labruzzo added 1 commit 2024-04-19 14:40:06 +02:00

b84ad0c06e merged beta

sandro.labruzzo added 1 commit 2024-04-22 11:32:54 +02:00

073f320c6a Added module containing all the dependencies, useful for spark deploy on k8.

sandro.labruzzo added 1 commit 2024-04-23 17:35:04 +02:00

0d628cd62b merged again from beta

sandro.labruzzo added 1 commit 2024-04-26 16:02:50 +02:00

9cd3bc0f10 Added a new generation of the dump for scholexplorer tested with last version of spark, and strongly refactored

sandro.labruzzo added 1 commit 2024-04-26 16:03:15 +02:00

052c6aac9d formatted code

sandro.labruzzo added 1 commit 2024-04-29 09:00:36 +02:00

133ead1e3e updated new version of scholexplorer Generation

sandro.labruzzo added 2 commits 2024-05-02 15:16:15 +02:00

0646d0d064 Updated main sparkApplication to avoid to require master variable

a860c57bbc updated .gitignore

sandro.labruzzo added 2 commits 2024-05-02 15:26:03 +02:00

26bf8e763a merged from beta

db358ad0d2 code formatted

sandro.labruzzo added 1 commit 2024-05-16 16:19:36 +02:00

6efab4d88e fixed scholexplorer bug

sandro.labruzzo added 1 commit 2024-05-17 14:16:54 +02:00

a87f9ea643 fixed scholexplorer bug

sandro.labruzzo added 1 commit 2024-05-17 14:43:19 +02:00

103e2652b3 merged beta

sandro.labruzzo added 2 commits 2024-05-22 11:09:06 +02:00

66c1ffc866 merged again from beta (I hope for the last time)

f1fe363b19 merged again from beta (I hope for the last time)

sandro.labruzzo commented

2024-05-22 12:39:33 +02:00

@giambattista.bloisi, I've compiled it on all profiles, and it seems to be working. I give you the honor, but above all, the responsibility to apply the PR.

giambattista.bloisi merged commit 1b2357e10a into beta

2024-05-23 09:20:29 +02:00

giambattista.bloisi referenced this issue from a commit

2024-05-23 09:20:32 +02:00

Merge pull request 'Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12' (#327) from spark34-integration into beta

giambattista.bloisi referenced this pull request

2024-05-23 09:33:43 +02:00

WIP: playing with dependencies to compile also with macos arm64 and openjdk 11 or 17 #431

Sign in to join this conversation.

No reviewers