parent
736d0f8526
commit
795d6e7c93
|
@ -1,4 +1,5 @@
|
|||
# UrlsWorker [![Build Status](https://jenkins-dnet.d4science.org/buildStatus/icon?job=UrlsWorker)](https://jenkins-dnet.d4science.org/job/UrlsWorker/)
|
||||
# UrlsWorker
|
||||
### [![Jenkins build status](https://jenkins-dnet.d4science.org/buildStatus/icon?job=UrlsWorker)](https://jenkins-dnet.d4science.org/job/UrlsWorker/)
|
||||
|
||||
The Worker's Application, requests assignments from the [**Controller**](https://code-repo.d4science.org/lsmyrnaios/UrlsController) and processes them with the help of the [__PublicationsRetriever__](https://github.com/LSmyrnaios/PublicationsRetriever) software and downloads the available full-texts.<br>
|
||||
Then, it posts the results to the Controller, which in turn, requests from the Worker, the full-texts which are not already found by other workers, in batches.<br>
|
||||
|
@ -16,9 +17,5 @@ We use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression
|
|||
<br>
|
||||
|
||||
**Notes**:
|
||||
- The above script, installs the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, as a library and then compiles and runs the whole Application.<br>
|
||||
- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, apart from the SpringBoot-app not re-building,
|
||||
the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, will not be re-installed as well.<br>
|
||||
- If you want to avoid re-installing the [PublicationsRetriever](https://github.com/LSmyrnaios/PublicationsRetriever) software, but still building the Worker's application, run the script with the argument "0", followed by the argument "1": ```./installAndRun.sh 0 1```, i.e. when using a development (non-published) version
|
||||
or when nothing has changed and wanting to avoid the time-overhead.<br>
|
||||
- If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```. In this scenario, the SpringBoot-app will not be re-built.<br>
|
||||
<br>
|
||||
|
|
|
@ -44,15 +44,15 @@ dependencies {
|
|||
exclude group: 'io.minio' // This is not used in the Worker, since it's the Controller which uploads the full-texts to S3. It also includes an older "commons-compress" version which causes problems.
|
||||
}
|
||||
|
||||
implementation group: 'com.google.guava', name: 'guava', version: '33.0.0-jre'
|
||||
implementation group: 'com.google.guava', name: 'guava', version: '33.1.0-jre'
|
||||
|
||||
// https://mvnrepository.com/artifact/com.google.code.gson/gson
|
||||
implementation 'com.google.code.gson:gson:2.10.1'
|
||||
|
||||
implementation("org.apache.commons:commons-compress:1.25.0") {
|
||||
implementation("org.apache.commons:commons-compress:1.26.1") {
|
||||
exclude group: 'com.github.luben', module: 'zstd-jni'
|
||||
}
|
||||
implementation 'com.github.luben:zstd-jni:1.5.5-11' // Even though this is part of the above dependency, the Apache commons rarely updates it, while the zstd team makes improvements very often.
|
||||
implementation 'com.github.luben:zstd-jni:1.5.6-3' // Even though this is part of the above dependency, the Apache commons rarely updates it, while the zstd team makes improvements very often.
|
||||
// Also, for compressing, we strangely need it to be explicitly declared independently, otherwise it does not work.
|
||||
|
||||
testImplementation 'org.springframework.security:spring-security-test'
|
||||
|
|
Loading…
Reference in New Issue