UrlsController/README.md

# UrlsController

The Controller's Application receives requests coming from the [Workers](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker) , constructs an assignments-list with data received from a database and returns the list to the workers.<br>
Then, it receives the "WorkerReports", it requests the full-texts from the workers, in batches, and uploads them on the S3-Object-Store. Finally, it writes the related reports, along with the updated file-locations into the database.<br> 
<br>
It can also process **Bulk-Import** requests, from compatible data sources, in which case it receives the full-text files immediately, without offloading crawling jobs to Workers.<br>
<br>
For interacting with the database we use [Impala](https://impala.apache.org/).<br>
<br>

**BulkImport API**:
- "**bulkImportFullTexts**" endpoint: **http://\<IP\>:<PORT>/api/bulkImportFullTexts?provenance=\<provenance\>&bulkImportDir=\<bulkImportDir\>&shouldDeleteFilesOnFinish={true|false}** <br>
  This endpoint loads the right configuration with the help of the "provenance" param and then processes the full-texts files inside the given directory, in the following way: it generates the openAireIDs, uploads the files to the S3 ObjectStore, generates and stores the the "payload" records in the database. If it is specified, it removes the successfully imported full-texts from the directory.
- "**getBulkImportReport**" endpoint: **http://\<IP\>:<PORT>/api/getBulkImportReport?id=\<bulkImportReportId\>** <br>
  This endpoint returns the bulkImport report, which corresponds to the given ID, in JSON  format.
<br>
<br>

**Statistics API**:
- "**getNumberOfAllPayloads**" endpoint: **http://\<IP\>:<PORT>/api/stats/getNumberOfAllPayloads** <br>
  This endpoint returns the total number of payloads existing in the database, independently of the way they were aggregated. This includes the payloads created by other pieces of software, before the PDF-Aggregation-Service was created.
- "**getNumberOfPayloadsAggregatedByService**" endpoint: **http://\<IP\>:<PORT>/api/stats/getNumberOfPayloadsAggregatedByService** <br>
  This endpoint returns the number of payloads aggregated by the PDF-Aggregated-Service itself. It excludes the payloads aggregated by other methods, by applying a Date-filter for the records created in 2021 or later.
- "**getNumberOfPayloadsForDatasource**" endpoint:  **http://\<IP\>:<PORT>/api/stats/getNumberOfPayloadsForDatasource?datasourceId=\<givenDatasourceId\>** <br>
  This endpoint returns the number of payloads which belong to the datasource specified by the given datasourceID.
- "**getNumberOfRecordsInspected**" endpoint: **http://\<IP\>:<PORT>/api/stats/getNumberOfRecordsInspected** <br>
  This endpoint returns the number of records inspected by the PDF-Aggregation-Service.
<br>
<br>

**To install and run the application**:
- Run ```git clone``` and then ```cd UrlsController```.
- Set the preferable values inside the [__application.yml__](https://code-repo.d4science.org/lsmyrnaios/UrlsController/src/branch/master/src/main/resources/application.yml) file.
- Execute the ```installAndRun.sh``` script which builds and runs the app.<br>
If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```.<br>
If you want to build and run the app on a **Docker Container**, then run the script with the argument "0" followed by the argument "1": ```./installAndRun.sh 0 1```.<br>
<br>

Implementation notes:
- For transferring the full-text files, we use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits in compression rate and speed.
- The uploaded full-text files follow this naming-scheme: "**datasourceID/recordID::fileHash.pdf**"