- Update README.

- Update dependencies.
This commit is contained in:
Lampros Smyrnaios 2024-04-26 13:36:41 +03:00
parent 736d0f8526
commit 795d6e7c93
2 changed files with 6 additions and 9 deletions

View File

@ -1,4 +1,5 @@
# UrlsWorker [![Build Status](https://jenkins-dnet.d4science.org/buildStatus/icon?job=UrlsWorker)](https://jenkins-dnet.d4science.org/job/UrlsWorker/)
# UrlsWorker
### [![Jenkins build status](https://jenkins-dnet.d4science.org/buildStatus/icon?job=UrlsWorker)](https://jenkins-dnet.d4science.org/job/UrlsWorker/)
The Worker's Application, requests assignments from the [**Controller**](https://code-repo.d4science.org/lsmyrnaios/UrlsController) and processes them with the help of the [__PublicationsRetriever__](https://github.com/LSmyrnaios/PublicationsRetriever) software and downloads the available full-texts.<br>
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.<br>
@ -16,9 +17,5 @@ We use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression
<br>
**Notes**:
- The above script, installs the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, as a library and then compiles and runs the whole Application.<br>
- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, apart from the SpringBoot-app not re-building,
the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, will not be re-installed as well.<br>
- If you want to avoid re-installing the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, but still building the Worker's application, run the script with the argument "0", followed by the argument "1": ```./installAndRun.sh 0 1```, i.e. when using a development (non-published) version
or when nothing has changed and wanting to avoid the time-overhead.<br>
- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, the SpringBoot-app will not be re-built.<br>
<br>

View File

@ -44,15 +44,15 @@ dependencies {
exclude group: 'io.minio' // This is not used in the Worker, since it's the Controller which uploads the full-texts to S3. It also includes an older "commons-compress" version which causes problems.
}
implementation group: 'com.google.guava', name: 'guava', version: '33.0.0-jre'
implementation group: 'com.google.guava', name: 'guava', version: '33.1.0-jre'
// https://mvnrepository.com/artifact/com.google.code.gson/gson
implementation 'com.google.code.gson:gson:2.10.1'
implementation("org.apache.commons:commons-compress:1.25.0") {
implementation("org.apache.commons:commons-compress:1.26.1") {
exclude group: 'com.github.luben', module: 'zstd-jni'
}
implementation 'com.github.luben:zstd-jni:1.5.5-11' // Even though this is part of the above dependency, the Apache commons rarely updates it, while the zstd team makes improvements very often.
implementation 'com.github.luben:zstd-jni:1.5.6-3' // Even though this is part of the above dependency, the Apache commons rarely updates it, while the zstd team makes improvements very often.
// Also, for compressing, we strangely need it to be explicitly declared independently, otherwise it does not work.
testImplementation 'org.springframework.security:spring-security-test'