lsmyrnaios
/
UrlsController
1
0
Fork 1
Code Issues Pull Requests Releases 10 Wiki Activity
The Controller app of the PDF Aggregation Service.
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
73 Commits
2 Branches
10 Tags
4.2 MiB
Java 98.2%
Shell 1.7%
 
 
master
springify_project
2.7.0
2.6.2
2.6.1
2.4
2.3
2.2
2.1
2.0
2.6
2.5
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '9b95eebb6c'
${ noResults }
Go to file
Lampros Smyrnaios 9b95eebb6c - Remove the obsolete "parenthesis" and "increasing duplicate-num" from the full-texts' names, before sending them to the S3-Object-Store. They now end with the "file-hash", so it is guaranteed that they will be unique. The Worker continues to produce the previous kind of names, without any disturbance. 
- Improve logging.
- Update MinIO dependency.
 		2 years ago
gradle/wrapper - Change the fileNames' structure in the S3-ObjectStore. 2 years ago
src/main - Remove the obsolete "parenthesis" and "increasing duplicate-num" from the full-texts' names, before sending them to the S3-Object-Store. They now end with the "file-hash", so it is guaranteed that they will be unique. The Worker continues to produce the previous kind of names, without any disturbance. 2 years ago
.gitignore springified project 2 years ago
Dockerfile - Allow the user to build, push and run the App in Docker, straight though the "installAndRun.sh" script. 2 years ago
README.md Update the README.md 2 years ago
build.gradle - Remove the obsolete "parenthesis" and "increasing duplicate-num" from the full-texts' names, before sending them to the S3-Object-Store. They now end with the "file-hash", so it is guaranteed that they will be unique. The Worker continues to produce the previous kind of names, without any disturbance. 2 years ago
installAndRun.sh - Change the fileNames' structure in the S3-ObjectStore. 2 years ago
settings.gradle - Add the "isControllerAlive"-endpoint. 3 years ago

README.md

UrlsController

The Controller's Application receives requests coming from the Workers , constructs an assignments-list with data received from a database and returns the list to the workers.
Then, it receives the "WorkerReports", it requests the full-texts from the workers, in batches, and uploads them on the S3-Object-Store. Finally, it writes the related reports, along with the updated file-locations into the database.
The database used is the Impala .

To install and run the application:

  • Run git clone and then cd UrlsController.
  • Provide the S3 Object Store related configurations, inside the src/main/resources/application.properties file.
  • Execute the installAndRun.sh script which builds and runs the app.
    If you want to just run the app, then run the script with the argument "1": ./installAndRun.sh 1.
    If you want to build and run the app on a docker container, then run the script with the argument "0" followed by the argument "1": ./installAndRun.sh 0 1.