lsmyrnaios
/
UrlsController
1
0
Fork 1
Code Issues Pull Requests Releases 9 Wiki Activity
26 Commits
2 Branches
9 Tags
4.1 MiB
master
springify_project
2.6.2
2.6.1
2.4
2.3
2.2
2.1
2.0
2.6
2.5
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '48eed20dd8'
${ noResults }
Commit Graph

26 Commits (48eed20dd8783671fc42ecede87902ec86cfc854)
 

Author SHA1 Message Date
Lampros Smyrnaios 48eed20dd8 - Implement the "getAndUploadFullTexts" functionality. In order to access the S3-ObjectStore from one trusted place, the Controller will request the files from the workers and upload them on S3. Afterwards, the workers will delete those files from their local storage. Previously, each worker uploaded its own files. 
- Move the "mergeParquetFiles" and "getCutBatchExceptionMessage" methods inside the "FileUtils" class.
- Code cleanup.
 2 years ago
Lampros Smyrnaios 780ed15ce2 - Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction. 
- Fix the "null" representation of an "unknown" payload-size in the database.
- Remove the obsolete thread-locking for the "CreateDatabase" operation. This code is guaranteed to run BEFORE any other operation in the database.
- Implement the "handlePreparedStatementException" and "closeConnection" methods.
- Improve error-logs.
- Update dependencies.
- Code cleanup.
 2 years ago
Lampros Smyrnaios d100af35d0 - Implement the "getUrls" and "addWorkerReport" endpoints with full database-handling. 
- Add connectivity with an Impala-database and create a dedicated Controller for future statistics-requests.
- Optimize the "getTestUrls"-endpoint.
- Disable the "reportCurrentTime()" scheduled-task.
- Update dependencies and bump project's version to '1.0.0-SNAPSHOT'.
- Set the logging-appender to "File".
- Code cleanup.
 2 years ago
Lampros Smyrnaios 0d47c33a08 - Improve logging configurations. 
- Fix the "AssignmentResponse.toString()" method.
- Update SpringBoot to v.2.5.6
- Code cleanup.
 2 years ago
Lampros Smyrnaios 0540820817 -Update "installAndRun.sh": 
-- Check if a gradle installation with the given version already exists, before downloading and installing gradle.
-- Make sure the "unzip" package is installed, before trying unzipping the gradle package.
- Update the logs' fileName.
 3 years ago
Lampros Smyrnaios d931315ced - Add the "isControllerAlive"-endpoint. 
- Change the data-type of the "UrlReport.status" to be "enum StatusType", in order to increase consistency and comparability.
- Change the "Date" datatype in "Payload" to have the SQL's version.
- Fix the project's name inside "settings.gradle".
- Code cleanup.
 3 years ago
Lampros Smyrnaios 983b900da7 - Add the "installAndRun.sh" script. 
- Update the README.
- Update the dependencies.
 3 years ago
Lampros Smyrnaios d56e988518 - Process the Error of PDF-aggregation. Distinguish between "couldRetry" and "noRetry" cases. 
- Update the "RequestParam" for the getUrls endpoints.
- Fix the "assignmentCounter".
- Code cleanup.
 3 years ago
Lampros Smyrnaios 25c566bf68 - Change server's port. 
- Update dependencies.
 3 years ago
Lampros Smyrnaios 27375b9396 - Refactor the Assignment-creation. 
In order to match the database, now we have a list of Assignments sent through the AssignmentResponse, instead of a single Assignment having a list of tasks.
- Cleanup the members of the "Payload" model (also prepare for database integration).
 3 years ago
Lampros Smyrnaios 5e7ccbd8c6 Add the "addWorkerReport" endpoint. 3 years ago
Lampros Smyrnaios 40763ec146 Update the "WorkerReport" request and the "UrlReport" and "Payload" models. 3 years ago
Lampros Smyrnaios c194af167f Allow handling of concurrent requests to the "getTestUrls"-endpoint. 3 years ago
Lampros Smyrnaios 308cab5ecd - Return an HTTP-500-error when the server cannot find the resourceFile requested by the "getTestUrls"-endpoint. 
- Close the "inputScanner" after each use when retrieving the test-tasks.
- Show info-logs when sending an assignment to a worker.
- Code cleanup.
 3 years ago
Lampros Smyrnaios 6729f51b03 Add an "assignmentId" field in the "Assignment"-class. 3 years ago
Lampros Smyrnaios 87044574b5 Update dependencies and add the "gradle-wrapper.properties" file which defines the gradle version. 3 years ago
Lampros Smyrnaios 787299b5b7 Add the "Datasource" inside the "Task" class and include it in the Assignment. 3 years ago
Lampros Smyrnaios d20fcf9cce - Update the "getUrls" and "getTestUrls" endpoints to take the data from the workers, in http-request-parameters instead of the http-body. 
- Fix tasksLimit-check.
- Code cleanup.
 3 years ago
Lampros Smyrnaios e2cc320baf - Add the "getTestUrls"-endpoint which returns an "Assignment" with data retrieved from the added resource-file. 
- Update the "getUrls"-endpoint to be ready to retrieve data from the database, once it's added.
- Update the dependencies.
- Code cleanup.
 3 years ago
Lampros Smyrnaios d3588ea36b Add the "DownloadAttempt" class. 3 years ago
Lampros Smyrnaios a6ab810ad3 Update classes: "Publication" and "Payload". 3 years ago
Lampros Smyrnaios 85ecc4a36b Add classes: "AssignmentResponse", "WorkerReport", "WorkerRequest", "UrlReport". 3 years ago
Lampros Smyrnaios c2ea8a69de Update classes: "Assignment", "Task", "Error", "Payload", "UrlsRequest". 3 years ago
Lampros Smyrnaios 89c6a73a30 Add "Assignment", "Task" and "Error" classes. 3 years ago
Lampros Smyrnaios c6e12d3e95 - Update "addResults"-endpoint. 
- Add "UrlsRequest.java".
- Some minor updates in "build.gradle" and "application.properties".
 3 years ago
Lampros Smyrnaios 8a4376da9c Initial commit of UrlsController. 3 years ago