UrlsController/README.md

# UrlsController

The Controller's Application receives requests coming from the [Workers](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker) , constructs an assignments-list with data received from a database and returns the list to the workers.<br>
Then, it receives the "WorkerReports", it requests the full-texts from the workers, in batches, and uploads them on the S3-Object-Store. Finally, it writes the related reports, along with the updated file-locations into the database.<br> 
The database used is the [Impala](https://impala.apache.org/).<br>
<br>

**Statistics API**:
- "**getNumberOfAllPayloads**" endpoint: **http://<IP>:<PORT>/api/stats/getNumberOfAllPayloads** <br>
  This endpoint returns the total number of payloads existing in the database, independently of the way they were aggregated. This includes the payloads created by other pieces of software, before the PDF-Aggregation-Service was created.
- "**getNumberOfPayloadsAggregatedByService**" endpoint: **http://<IP>:<PORT>/api/stats/getNumberOfPayloadsAggregatedByService** <br>
  This endpoint returns the number of payloads aggregated by the PDF-Aggregated-Service itself. It excludes the payloads aggregated by other methods, by applying a Date-filter for the records created in 2021 or later.
- "**getNumberOfRecordsInspected**" endpoint: **http://<IP>:<PORT>/api/stats/getNumberOfRecordsInspected** <br>
    This endpoint returns the number of records inspected by the PDF-Aggregation-Service.
<br>
<br>

**To install and run the application**:
- Run ```git clone``` and then ```cd UrlsController```.
- Set the preferable values inside the [__application.properties__](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker/src/branch/master/src/main/resources/application.properties) file.
- Execute the ```installAndRun.sh``` script which builds and runs the app.<br>
If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```.<br>
If you want to build and run the app on a **Docker Container**, then run the script with the argument "0" followed by the argument "1": ```./installAndRun.sh 0 1```.<br>
<br>

Implementation notes:
- For transferring the full-text files, we use Facebook's [**Zstandard**](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits in compression rate and speed.
- The uploaded full-text files follow this naming-scheme: "**datasourceID/recordId::fileHash.pdf**"
Initial commit of UrlsController. 2021-03-16 14:25:15 +01:00			`# UrlsController`

Update the README.md 2022-02-07 20:11:03 +01:00			`The Controller's Application receives requests coming from the [Workers](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker) , constructs an assignments-list with data received from a database and returns the list to the workers.<br>`
			`Then, it receives the "WorkerReports", it requests the full-texts from the workers, in batches, and uploads them on the S3-Object-Store. Finally, it writes the related reports, along with the updated file-locations into the database.<br>`
- Add info about the Stats API usage in "README.md". - Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()". - Handle the "EmptyResultDataAccessException" inside "StatsController". - Optimize gradle's performance. - Code polishing. 2022-12-15 13:04:22 +01:00			`The database used is the [Impala](https://impala.apache.org/).<br>`
			`<br>`
Improve documentation. 2023-01-27 13:31:07 +01:00
			`Statistics API:`
- Refactor the payloads-statistics-code and provide two endpoints: "getNumberOfPayloadsAggregatedByService", which returns the number of payloads aggregated only by the PDF-Aggregation-Service, and the "getNumberOfAllPayloads", which returns the number of all payloads existing in the database, even the ones aggregated in the past, by other pieces of software. - Update README.md. - Make sure the docker image is clean-built, by avoiding the use of cache. 2023-02-02 16:58:47 +01:00			`- "getNumberOfAllPayloads" endpoint: http://<IP>:<PORT>/api/stats/getNumberOfAllPayloads <br>`
			`This endpoint returns the total number of payloads existing in the database, independently of the way they were aggregated. This includes the payloads created by other pieces of software, before the PDF-Aggregation-Service was created.`
			`- "getNumberOfPayloadsAggregatedByService" endpoint: http://<IP>:<PORT>/api/stats/getNumberOfPayloadsAggregatedByService <br>`
			`This endpoint returns the number of payloads aggregated by the PDF-Aggregated-Service itself. It excludes the payloads aggregated by other methods, by applying a Date-filter for the records created in 2021 or later.`
			`- "getNumberOfRecordsInspected" endpoint: http://<IP>:<PORT>/api/stats/getNumberOfRecordsInspected <br>`
			`This endpoint returns the number of records inspected by the PDF-Aggregation-Service.`
- Add info about the Stats API usage in "README.md". - Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()". - Handle the "EmptyResultDataAccessException" inside "StatsController". - Optimize gradle's performance. - Code polishing. 2022-12-15 13:04:22 +01:00			`<br>`
- Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. - Move the "mergeParquetFiles" and "getCutBatchExceptionMessage" methods inside the "FileUtils" class. - Code cleanup. 2021-11-30 17:23:27 +01:00			`<br>`
Improve documentation. 2023-01-27 13:31:07 +01:00
			`To install and run the application:`
Update the README.md 2022-02-07 20:11:03 +01:00			- Run ```git clone``` and then ```cd UrlsController```.
Improve documentation. 2023-01-27 13:31:07 +01:00			`- Set the preferable values inside the [__application.properties__](https://code-repo.d4science.org/lsmyrnaios/UrlsWorker/src/branch/master/src/main/resources/application.properties) file.`
Update the README.md 2022-02-07 20:11:03 +01:00			- Execute the ```installAndRun.sh``` script which builds and runs the app.<br>
- Add the "installAndRun.sh" script. - Update the README. - Update the dependencies. 2021-09-09 14:56:37 +02:00			If you want to just run the app, then run the script with the argument "1": ```./installAndRun.sh 1```.<br>
- Fix creating the "assignment" table, always in the testDatabase. - Code polishing. 2022-12-07 13:58:38 +01:00			If you want to build and run the app on a Docker Container, then run the script with the argument "0" followed by the argument "1": ```./installAndRun.sh 0 1```.<br>
- Add the "installAndRun.sh" script. - Update the README. - Update the dependencies. 2021-09-09 14:56:37 +02:00			`<br>`
- Use Facebook's [Zstandard](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits on compression rate and speed. - Update the minIO dependency. - Code polishing. 2023-01-10 12:34:54 +01:00
			`Implementation notes:`
			`- For transferring the full-text files, we use Facebook's [Zstandard](https://facebook.github.io/zstd/) compression algorithm, which brings very big benefits in compression rate and speed.`
- Exclude empty and null urls in the assignments. - Update the "getFullTextsImproved"-call to "getFullTexts", now that the "improved" version is stable. - Update Gradle. - Code polishing. 2023-02-16 13:24:47 +01:00			`- The uploaded full-text files follow this naming-scheme: "datasourceID/recordId::fileHash.pdf"`