D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 13 Projects 2 Releases 2 Wiki Activity

WIP: playing with dependencies to compile also with macos arm64 and openjdk 11 or 17 #431

Draft
alessia.bardi wants to merge 5 commits from dependency-revision into beta
pull from: dependency-revision
merge into: D-Net:beta
Conversation 2 Commits 5 Files Changed 9 +318 -87
alessia.bardi commented 2024-04-30 14:13:25 +02:00
Owner
Copy Link

Compilation problems have been recently reported.
I also have compilation problems on my new Macos arm64 and latest OpenJDK (21).
With this branch we can play with pom and dependencies.
My today's attempt revealed that we cannot use an OpenJDK version > 11 due to incompatibiliy with the Scala version we use (see https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html)
I also had compilation problems due to:

  • org.antipathy plugin (shall we find another plugin for formatting scala code?)
  • impsort-maven-plugin (for sorting import)

dhp-common has an indirect dependency to lombok and e only way I found so far to avoid the compilation error

Fatal error compiling: java.lang.IllegalAccessError: class lombok.javac.apt.LombokProcessor (in unnamed module @xxxxxxxx) cannot access class com.sun.tools.javac.processing.JavacProcessingEnvironment (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.processing to unnamed module @xxxxxxxxx

was to add a provided dependency to lombok, which I do not like.

Results of compilation on

alessia@nb-bardi dnet-hadoop % mvn -version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec
Java version: 11.0.23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk@11/11.0.23/libexec/openjdk.jdk/Contents/Home
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "14.4.1", arch: "aarch64", family: "mac"

with the current commit is:

[INFO] Reactor Summary for dhp 1.2.5-SNAPSHOT:
[INFO] 
[INFO] dhp-code-style ..................................... SUCCESS [  0.121 s]
[INFO] dhp ................................................ SUCCESS [  0.237 s]
[INFO] dhp-build .......................................... SUCCESS [  0.003 s]
[INFO] dhp-build-assembly-resources ....................... SUCCESS [  0.066 s]
[INFO] dhp-build-properties-maven-plugin .................. SUCCESS [  0.505 s]
[INFO] dhp-pace-core ...................................... SUCCESS [  1.349 s]
[INFO] dhp-common ......................................... FAILURE [  0.818 s]
[INFO] dhp-workflows ...................................... SKIPPED
[INFO] dhp-workflow-profiles .............................. SKIPPED
[INFO] dhp-aggregation .................................... SKIPPED
[INFO] dhp-actionmanager .................................. SKIPPED
[INFO] dhp-graph-mapper ................................... SKIPPED
[INFO] dhp-dedup-openaire ................................. SKIPPED
[INFO] dhp-enrichment ..................................... SKIPPED
[INFO] dhp-graph-provision ................................ SKIPPED
[INFO] dhp-blacklist ...................................... SKIPPED
[INFO] dhp-stats-actionsets ............................... SKIPPED
[INFO] dhp-stats-hist-snaps ............................... SKIPPED
[INFO] dhp-stats-monitor-irish ............................ SKIPPED
[INFO] dhp-stats-monitor-update ........................... SKIPPED
[INFO] dhp-stats-update ................................... SKIPPED
[INFO] dhp-stats-promote .................................. SKIPPED
[INFO] dhp-usage-stats-build .............................. SKIPPED
[INFO] dhp-usage-raw-data-update .......................... SKIPPED
[INFO] dhp-broker-events .................................. SKIPPED
[INFO] dhp-doiboost ....................................... SKIPPED
[INFO] dhp-impact-indicators .............................. SKIPPED
[INFO] dhp-swh ............................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.266 s
[INFO] Finished at: 2024-04-30T14:10:42+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.0.1:doc (scala-doc) on project dhp-common: wrap: org.apache.maven.reporting.MavenReportException: wrap: Process exited with an error: 240 (Exit value: 240)
Compilation problems have been recently reported. I also have compilation problems on my new Macos arm64 and latest OpenJDK (21). With this branch we can play with pom and dependencies. My today's attempt revealed that we cannot use an OpenJDK version > 11 due to incompatibiliy with the Scala version we use (see https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html) I also had compilation problems due to: - org.antipathy plugin (shall we find another plugin for formatting scala code?) - impsort-maven-plugin (for sorting import) dhp-common has an indirect dependency to lombok and e only way I found so far to avoid the compilation error `Fatal error compiling: java.lang.IllegalAccessError: class lombok.javac.apt.LombokProcessor (in unnamed module @xxxxxxxx) cannot access class com.sun.tools.javac.processing.JavacProcessingEnvironment (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.processing to unnamed module @xxxxxxxxx` was to add a provided dependency to lombok, which I do not like. Results of compilation on ``` alessia@nb-bardi dnet-hadoop % mvn -version Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec Java version: 11.0.23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk@11/11.0.23/libexec/openjdk.jdk/Contents/Home Default locale: en_GB, platform encoding: UTF-8 OS name: "mac os x", version: "14.4.1", arch: "aarch64", family: "mac" ``` with the current commit is: ``` [INFO] Reactor Summary for dhp 1.2.5-SNAPSHOT: [INFO] [INFO] dhp-code-style ..................................... SUCCESS [ 0.121 s] [INFO] dhp ................................................ SUCCESS [ 0.237 s] [INFO] dhp-build .......................................... SUCCESS [ 0.003 s] [INFO] dhp-build-assembly-resources ....................... SUCCESS [ 0.066 s] [INFO] dhp-build-properties-maven-plugin .................. SUCCESS [ 0.505 s] [INFO] dhp-pace-core ...................................... SUCCESS [ 1.349 s] [INFO] dhp-common ......................................... FAILURE [ 0.818 s] [INFO] dhp-workflows ...................................... SKIPPED [INFO] dhp-workflow-profiles .............................. SKIPPED [INFO] dhp-aggregation .................................... SKIPPED [INFO] dhp-actionmanager .................................. SKIPPED [INFO] dhp-graph-mapper ................................... SKIPPED [INFO] dhp-dedup-openaire ................................. SKIPPED [INFO] dhp-enrichment ..................................... SKIPPED [INFO] dhp-graph-provision ................................ SKIPPED [INFO] dhp-blacklist ...................................... SKIPPED [INFO] dhp-stats-actionsets ............................... SKIPPED [INFO] dhp-stats-hist-snaps ............................... SKIPPED [INFO] dhp-stats-monitor-irish ............................ SKIPPED [INFO] dhp-stats-monitor-update ........................... SKIPPED [INFO] dhp-stats-update ................................... SKIPPED [INFO] dhp-stats-promote .................................. SKIPPED [INFO] dhp-usage-stats-build .............................. SKIPPED [INFO] dhp-usage-raw-data-update .......................... SKIPPED [INFO] dhp-broker-events .................................. SKIPPED [INFO] dhp-doiboost ....................................... SKIPPED [INFO] dhp-impact-indicators .............................. SKIPPED [INFO] dhp-swh ............................................ SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.266 s [INFO] Finished at: 2024-04-30T14:10:42+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.0.1:doc (scala-doc) on project dhp-common: wrap: org.apache.maven.reporting.MavenReportException: wrap: Process exited with an error: 240 (Exit value: 240) ```
alessia.bardi added 1 commit 2024-04-30 14:13:26 +02:00
sandro.labruzzo was assigned by alessia.bardi 2024-04-30 14:14:41 +02:00
claudio.atzori was assigned by alessia.bardi 2024-04-30 14:14:41 +02:00
alessia.bardi self-assigned this 2024-04-30 14:14:41 +02:00
antonis.lempesis was assigned by alessia.bardi 2024-04-30 14:14:41 +02:00
ant.lebesis was assigned by alessia.bardi 2024-04-30 14:14:42 +02:00
giambattista.bloisi was assigned by alessia.bardi 2024-04-30 14:14:42 +02:00
alessia.bardi changed title from playing with dependencies to compile also with macos arm64 and openjdk 11 or 17 to WIP: playing with dependencies to compile also with macos arm64 and openjdk 11 or 17 2024-04-30 14:14:54 +02:00
claudio.atzori added 1 commit 2024-05-02 11:55:01 +02:00
claudio.atzori added 2 commits 2024-05-02 15:07:12 +02:00
claudio.atzori added 1 commit 2024-05-02 15:08:32 +02:00
alessia.bardi commented 2024-05-02 16:32:11 +02:00
Author
Owner
Copy Link

Instructions to install Java8 on MacOS ARM chips available in the wiki: https://support.openaire.eu/projects/openaire/wiki/Wiki#Compile-dnet-hadoop-with-ARM-chips-eg-MacOS-M3

Instructions to install Java8 on MacOS ARM chips available in the wiki: https://support.openaire.eu/projects/openaire/wiki/Wiki#Compile-dnet-hadoop-with-ARM-chips-eg-MacOS-M3
giambattista.bloisi commented 2024-05-23 09:33:43 +02:00
Member
Copy Link

@alessia.bardi I just merged in beta PR #327 that solves part of the problems. A new profile, -P spark-34, can be used to activate building against spark 3.4.2.openaire that has recently been installed on ICM ocean cluster. With that profile it is also possible to compile and run tests using an updated jdk (indeed with spark-34 profile only).

Regarding Silicon macs I found two main problems:

  • When using spark-2.4 snappy library needs to be updated to get Silicon processor support (this is required to run tests):
    <dependency> <groupId>org.xerial.snappy</groupId> <artifactId>snappy-java</artifactId> <version>1.1.8.4</version> </dependency>
  • In both versions (2.4, 3.4) I was not able to find or build the hadoop 2.6 native library for Silicon cpu (hadoop version is so old it makes more sense to update it than adding support to Silicon, but we are stuck to it because of our hadoop clusters). As a consequence it is not possible to use the hadoop zlib codec we use to compress sequential file. A work-around could be to use bzip2 codec instead.
@alessia.bardi I just merged in beta PR #327 that solves part of the problems. A new profile, -P spark-34, can be used to activate building against spark 3.4.2.openaire that has recently been installed on ICM ocean cluster. With that profile it is also possible to compile and run tests using an updated jdk (indeed with spark-34 profile only). Regarding Silicon macs I found two main problems: - When using spark-2.4 snappy library needs to be updated to get Silicon processor support (this is required to run tests): `<dependency> <groupId>org.xerial.snappy</groupId> <artifactId>snappy-java</artifactId> <version>1.1.8.4</version> </dependency>` - In both versions (2.4, 3.4) I was not able to find or build the hadoop 2.6 native library for Silicon cpu (hadoop version is so old it makes more sense to update it than adding support to Silicon, but we are stuck to it because of our hadoop clusters). As a consequence it is not possible to use the hadoop zlib codec we use to compress sequential file. A work-around could be to use bzip2 codec instead.
This pull request has changes conflicting with the target branch.
  • dhp-common/pom.xml
  • dhp-pace-core/pom.xml
  • dhp-workflows/dhp-actionmanager/pom.xml
  • pom.xml
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b dependency-revision beta
git pull origin dependency-revision

Step 2:

Merge the changes and update on Gitea.
git checkout beta
git merge --no-ff dependency-revision
git push origin beta
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
giambattista.bloisi
ant.lebesis
antonis.lempesis
alessia.bardi
claudio.atzori
sandro.labruzzo
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#431
No description provided.
Delete Branch "dependency-revision"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?