Commit Graph

28 Commits

Author SHA1 Message Date
Lampros Smyrnaios b6e8cd1889 New feature: BulkImport full-text files from compatible datasources. 2023-05-11 03:07:55 +03:00
Lampros Smyrnaios 55ea5118ac - Update the "testDatabaseName" property.
- Code polishing.
2023-04-26 19:33:28 +03:00
Lampros Smyrnaios c39fef2654 Upgrade payload-table to payload-view which consists of three separate payload tables: "payload_legacy", "payload_aggregated" and "payload_bulk_import". 2023-04-10 15:55:50 +03:00
Lampros Smyrnaios 882c6f447b Update the "testDatabaseName". 2023-03-21 23:10:21 +02:00
Lampros Smyrnaios 4280f89296 - Set the default value of the "isTestEnvironment" property to "true", in order to avoid undesired outcomes in the production db.
- Code polishing.
2023-03-21 17:04:28 +02:00
Lampros Smyrnaios e975bec911 - Add Prometheus and Grafana which help measuring various metrics for the Controller's health and performance.
- Fix Docker config still using the old (now removed) "application.properties" file.
- Simplify the process of building and running the docker image; Now we use docker compose to run the Controller, along with the Prometheus and Grafana. Also, now it is not requested from the user to login and push the image (this may change in the future).
2023-03-21 16:46:33 +02:00
Lampros Smyrnaios 003c0bf179 - Add support for excluding specific datasources from being crawled. These datasources may be aggregated through bulk-imports, by other pieces of software. Such a datasource is "arXiv.org".
- Fix an issue, where the "datasource-type" was retrieved instead of the "datasource-name".
- Polish the "findAssignmentsQuery".
2023-03-21 07:19:35 +02:00
Lampros Smyrnaios f835a752bf Transform the "application.properties" file to "application.yml" and optimize the property-trees. 2023-03-20 15:23:00 +02:00
Lampros Smyrnaios 9cdbbdea67 Refactor the files' storage location. 2022-12-15 18:29:51 +02:00
Lampros Smyrnaios c8baf5a5fc - Fix not finding the parquet-schema files when the app was run inside a Docker Container.
- Update the "namespaces" and the "names" inside the parquet schemas.
- Code polishing.
2022-12-08 12:16:05 +02:00
Lampros Smyrnaios 95c38c4a24 - Fix creating the "assignment" table, always in the testDatabase.
- Code polishing.
2022-12-07 14:58:38 +02:00
Lampros Smyrnaios 6226e2298d - Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.
One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it.
- Code polishing.
2022-11-10 17:18:21 +02:00
Lampros Smyrnaios 9096137008 Update documentation. 2022-04-14 14:42:36 +03:00
Lampros Smyrnaios a81ed3c60f - Add an "isTestEnvironment"-switch, which makes it easier to work with production and test databases.
- In case the Worker cannot be reached during a full-texts' batch request, abort the rest of the batches.
- Fix memory leaks when unzipping the batch-zip-file.
- Add explanatory comments for picking the database related to a full-text file.
2022-04-08 17:39:45 +03:00
Lampros Smyrnaios a23c918a42 - Fix a "@JsonProperty" annotation inside "Payload.java".
- Fix a "@Value" annotation inside "FileUtils.java".
- Add a new database and show its name along with the initial's name in the logs.
- Code cleanup and improvement.
2022-04-05 00:01:44 +03:00
Lampros Smyrnaios b206114144 - Allow the user to build, push and run the App in Docker, straight though the "installAndRun.sh" script.
- Re-add the logback-spring configuration.
- Change the docker-app name.
2022-02-04 15:49:56 +02:00
Antonis Lempesis 35966b6f6e finishing toucehs 2022-02-01 16:57:28 +02:00
Antonis Lempesis 6dde8c0faa finished merge 2022-01-31 04:17:16 +02:00
Antonis Lempesis bf26bf955f springified project 2022-01-30 22:14:52 +02:00
Lampros Smyrnaios d0ab42e4fa - Change the scheme of the file-location URI.
- Move the old and the current database names in the "application.properties" file.
- Improve logging.
2022-01-28 07:24:42 +02:00
Lampros Smyrnaios dea257b87f - Fix a bug, which caused the get-full-texts request to fail, because of the wrong "requestAssignmentsCounter".
- Fix a bug, which caused multiple workers to get assigned the same batch-counter, while the assignment-tasks where different.
- Set a max-size limit to the amount of space the logs can use. Over that size, the older logs will be deleted.
- Show the error-message returned from the Worker, when a getFullTexts-request fails.
- Improve some log-messages.
- Update dependencies.
- Code cleanup.
2021-12-06 20:18:30 +02:00
Lampros Smyrnaios d100af35d0 - Implement the "getUrls" and "addWorkerReport" endpoints with full database-handling.
- Add connectivity with an Impala-database and create a dedicated Controller for future statistics-requests.
- Optimize the "getTestUrls"-endpoint.
- Disable the "reportCurrentTime()" scheduled-task.
- Update dependencies and bump project's version to '1.0.0-SNAPSHOT'.
- Set the logging-appender to "File".
- Code cleanup.
2021-11-09 23:59:27 +02:00
Lampros Smyrnaios 0d47c33a08 - Improve logging configurations.
- Fix the "AssignmentResponse.toString()" method.
- Update SpringBoot to v.2.5.6
- Code cleanup.
2021-11-04 11:57:19 +02:00
Lampros Smyrnaios 0540820817 -Update "installAndRun.sh":
-- Check if a gradle installation with the given version already exists, before downloading and installing gradle.
-- Make sure the "unzip" package is installed, before trying unzipping the gradle package.
- Update the logs' fileName.
2021-10-14 02:46:33 +03:00
Lampros Smyrnaios 25c566bf68 - Change server's port.
- Update dependencies.
2021-07-29 08:44:36 +03:00
Lampros Smyrnaios e2cc320baf - Add the "getTestUrls"-endpoint which returns an "Assignment" with data retrieved from the added resource-file.
- Update the "getUrls"-endpoint to be ready to retrieve data from the database, once it's added.
- Update the dependencies.
- Code cleanup.
2021-05-18 17:23:20 +03:00
Lampros Smyrnaios c6e12d3e95 - Update "addResults"-endpoint.
- Add "UrlsRequest.java".
- Some minor updates in "build.gradle" and "application.properties".
2021-03-16 18:07:30 +02:00
Lampros Smyrnaios 8a4376da9c Initial commit of UrlsController. 2021-03-16 15:25:15 +02:00