WIP: playing with dependencies to compile also with macos arm64 and openjdk 11 or 17 #431

alessia.bardi · 2024-04-30T14:13:25+02:00

alessia.bardi commented

2024-04-30 14:13:25 +02:00

Compilation problems have been recently reported.
I also have compilation problems on my new Macos arm64 and latest OpenJDK (21).
With this branch we can play with pom and dependencies.
My today's attempt revealed that we cannot use an OpenJDK version > 11 due to incompatibiliy with the Scala version we use (see https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html)
I also had compilation problems due to:

org.antipathy plugin (shall we find another plugin for formatting scala code?)
impsort-maven-plugin (for sorting import)

dhp-common has an indirect dependency to lombok and e only way I found so far to avoid the compilation error

Fatal error compiling: java.lang.IllegalAccessError: class lombok.javac.apt.LombokProcessor (in unnamed module @xxxxxxxx) cannot access class com.sun.tools.javac.processing.JavacProcessingEnvironment (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.processing to unnamed module @xxxxxxxxx

was to add a provided dependency to lombok, which I do not like.

Results of compilation on

alessia@nb-bardi dnet-hadoop % mvn -version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec
Java version: 11.0.23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk@11/11.0.23/libexec/openjdk.jdk/Contents/Home
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "14.4.1", arch: "aarch64", family: "mac"

with the current commit is:

[INFO] Reactor Summary for dhp 1.2.5-SNAPSHOT:
[INFO] 
[INFO] dhp-code-style ..................................... SUCCESS [  0.121 s]
[INFO] dhp ................................................ SUCCESS [  0.237 s]
[INFO] dhp-build .......................................... SUCCESS [  0.003 s]
[INFO] dhp-build-assembly-resources ....................... SUCCESS [  0.066 s]
[INFO] dhp-build-properties-maven-plugin .................. SUCCESS [  0.505 s]
[INFO] dhp-pace-core ...................................... SUCCESS [  1.349 s]
[INFO] dhp-common ......................................... FAILURE [  0.818 s]
[INFO] dhp-workflows ...................................... SKIPPED
[INFO] dhp-workflow-profiles .............................. SKIPPED
[INFO] dhp-aggregation .................................... SKIPPED
[INFO] dhp-actionmanager .................................. SKIPPED
[INFO] dhp-graph-mapper ................................... SKIPPED
[INFO] dhp-dedup-openaire ................................. SKIPPED
[INFO] dhp-enrichment ..................................... SKIPPED
[INFO] dhp-graph-provision ................................ SKIPPED
[INFO] dhp-blacklist ...................................... SKIPPED
[INFO] dhp-stats-actionsets ............................... SKIPPED
[INFO] dhp-stats-hist-snaps ............................... SKIPPED
[INFO] dhp-stats-monitor-irish ............................ SKIPPED
[INFO] dhp-stats-monitor-update ........................... SKIPPED
[INFO] dhp-stats-update ................................... SKIPPED
[INFO] dhp-stats-promote .................................. SKIPPED
[INFO] dhp-usage-stats-build .............................. SKIPPED
[INFO] dhp-usage-raw-data-update .......................... SKIPPED
[INFO] dhp-broker-events .................................. SKIPPED
[INFO] dhp-doiboost ....................................... SKIPPED
[INFO] dhp-impact-indicators .............................. SKIPPED
[INFO] dhp-swh ............................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.266 s
[INFO] Finished at: 2024-04-30T14:10:42+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.0.1:doc (scala-doc) on project dhp-common: wrap: org.apache.maven.reporting.MavenReportException: wrap: Process exited with an error: 240 (Exit value: 240)

Compilation problems have been recently reported. I also have compilation problems on my new Macos arm64 and latest OpenJDK (21). With this branch we can play with pom and dependencies. My today's attempt revealed that we cannot use an OpenJDK version > 11 due to incompatibiliy with the Scala version we use (see https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html) I also had compilation problems due to: - org.antipathy plugin (shall we find another plugin for formatting scala code?) - impsort-maven-plugin (for sorting import) dhp-common has an indirect dependency to lombok and e only way I found so far to avoid the compilation error `Fatal error compiling: java.lang.IllegalAccessError: class lombok.javac.apt.LombokProcessor (in unnamed module @xxxxxxxx) cannot access class com.sun.tools.javac.processing.JavacProcessingEnvironment (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.processing to unnamed module @xxxxxxxxx` was to add a provided dependency to lombok, which I do not like. Results of compilation on ``` alessia@nb-bardi dnet-hadoop % mvn -version Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec Java version: 11.0.23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk@11/11.0.23/libexec/openjdk.jdk/Contents/Home Default locale: en_GB, platform encoding: UTF-8 OS name: "mac os x", version: "14.4.1", arch: "aarch64", family: "mac" ``` with the current commit is: ``` [INFO] Reactor Summary for dhp 1.2.5-SNAPSHOT: [INFO] [INFO] dhp-code-style ..................................... SUCCESS [ 0.121 s] [INFO] dhp ................................................ SUCCESS [ 0.237 s] [INFO] dhp-build .......................................... SUCCESS [ 0.003 s] [INFO] dhp-build-assembly-resources ....................... SUCCESS [ 0.066 s] [INFO] dhp-build-properties-maven-plugin .................. SUCCESS [ 0.505 s] [INFO] dhp-pace-core ...................................... SUCCESS [ 1.349 s] [INFO] dhp-common ......................................... FAILURE [ 0.818 s] [INFO] dhp-workflows ...................................... SKIPPED [INFO] dhp-workflow-profiles .............................. SKIPPED [INFO] dhp-aggregation .................................... SKIPPED [INFO] dhp-actionmanager .................................. SKIPPED [INFO] dhp-graph-mapper ................................... SKIPPED [INFO] dhp-dedup-openaire ................................. SKIPPED [INFO] dhp-enrichment ..................................... SKIPPED [INFO] dhp-graph-provision ................................ SKIPPED [INFO] dhp-blacklist ...................................... SKIPPED [INFO] dhp-stats-actionsets ............................... SKIPPED [INFO] dhp-stats-hist-snaps ............................... SKIPPED [INFO] dhp-stats-monitor-irish ............................ SKIPPED [INFO] dhp-stats-monitor-update ........................... SKIPPED [INFO] dhp-stats-update ................................... SKIPPED [INFO] dhp-stats-promote .................................. SKIPPED [INFO] dhp-usage-stats-build .............................. SKIPPED [INFO] dhp-usage-raw-data-update .......................... SKIPPED [INFO] dhp-broker-events .................................. SKIPPED [INFO] dhp-doiboost ....................................... SKIPPED [INFO] dhp-impact-indicators .............................. SKIPPED [INFO] dhp-swh ............................................ SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.266 s [INFO] Finished at: 2024-04-30T14:10:42+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.0.1:doc (scala-doc) on project dhp-common: wrap: org.apache.maven.reporting.MavenReportException: wrap: Process exited with an error: 240 (Exit value: 240) ```

alessia.bardi added 1 commit 2024-04-30 14:13:26 +02:00

0173bf02f3 playing with dependencies to compile also with macos arm64 and openjdk 11 or 17

sandro.labruzzo was assigned by alessia.bardi

2024-04-30 14:14:41 +02:00

claudio.atzori was assigned by alessia.bardi

2024-04-30 14:14:41 +02:00

alessia.bardi self-assigned this 2024-04-30 14:14:41 +02:00

antonis.lempesis was assigned by alessia.bardi

2024-04-30 14:14:41 +02:00

ant.lebesis was assigned by alessia.bardi

2024-04-30 14:14:42 +02:00

giambattista.bloisi was assigned by alessia.bardi

2024-04-30 14:14:42 +02:00

alessia.bardi changed title from ~~playing with dependencies to compile also with macos arm64 and openjdk 11 or 17~~ to WIP: playing with dependencies to compile also with macos arm64 and openjdk 11 or 17

2024-04-30 14:14:54 +02:00

claudio.atzori added 1 commit 2024-05-02 11:55:01 +02:00

591b76047f Merge branch 'beta' into dependency-revision

claudio.atzori added 2 commits 2024-05-02 15:07:12 +02:00

564bdb2dcf applying suggestions from the maven dependency:analyze report

7033592469 applying suggestions from the maven dependency:analyze report

claudio.atzori added 1 commit 2024-05-02 15:08:32 +02:00

375b07a2e9 fixed pom

alessia.bardi commented

2024-05-02 16:32:11 +02:00

Instructions to install Java8 on MacOS ARM chips available in the wiki: https://support.openaire.eu/projects/openaire/wiki/Wiki#Compile-dnet-hadoop-with-ARM-chips-eg-MacOS-M3

giambattista.bloisi commented

2024-05-23 09:33:43 +02:00

@alessia.bardi I just merged in beta PR #327 that solves part of the problems. A new profile, -P spark-34, can be used to activate building against spark 3.4.2.openaire that has recently been installed on ICM ocean cluster. With that profile it is also possible to compile and run tests using an updated jdk (indeed with spark-34 profile only).

Regarding Silicon macs I found two main problems:

When using spark-2.4 snappy library needs to be updated to get Silicon processor support (this is required to run tests):
<dependency> <groupId>org.xerial.snappy</groupId> <artifactId>snappy-java</artifactId> <version>1.1.8.4</version> </dependency>
In both versions (2.4, 3.4) I was not able to find or build the hadoop 2.6 native library for Silicon cpu (hadoop version is so old it makes more sense to update it than adding support to Silicon, but we are stuck to it because of our hadoop clusters). As a consequence it is not possible to use the hadoop zlib codec we use to compress sequential file. A work-around could be to use bzip2 codec instead.

@alessia.bardi I just merged in beta PR #327 that solves part of the problems. A new profile, -P spark-34, can be used to activate building against spark 3.4.2.openaire that has recently been installed on ICM ocean cluster. With that profile it is also possible to compile and run tests using an updated jdk (indeed with spark-34 profile only). Regarding Silicon macs I found two main problems: - When using spark-2.4 snappy library needs to be updated to get Silicon processor support (this is required to run tests): `<dependency> <groupId>org.xerial.snappy</groupId> <artifactId>snappy-java</artifactId> <version>1.1.8.4</version> </dependency>` - In both versions (2.4, 3.4) I was not able to find or build the hadoop 2.6 native library for Silicon cpu (hadoop version is so old it makes more sense to update it than adding support to Silicon, but we are stuck to it because of our hadoop clusters). As a consequence it is not possible to use the hadoop zlib codec we use to compress sequential file. A work-around could be to use bzip2 codec instead.

This pull request has changes conflicting with the target branch.