Lampros Smyrnaios
8876089022
- Use Facebook's [**Zstandard**]( https://facebook.github.io/zstd/ ) compression algorithm, which brings very big benefits on compression rate and speed.
...
- Update the minIO dependency.
- Code polishing.
2023-01-10 13:34:54 +02:00
Lampros Smyrnaios
d1a4c84289
- Make sure the fullPath of the baseFilesLocation is available when the user specifies a non-root directory.
...
- Improve error-checking and exception-handling in some "S3ObjectStore"-methods.
- Make sure the "responseCode" is "200-OK", before trying to get the InputStream in "UriBuilder.getPublicIP()".
2023-01-09 15:44:53 +02:00
Lampros Smyrnaios
e51ee9dd27
- Add info about the Stats API usage in "README.md".
...
- Optimize performance in "ParquetFileUtils.createAndLoadParquetDataIntoAttemptTable()" and "ParquetFileUtils.createAndLoadParquetDataIntoPayloadTable()".
- Handle the "EmptyResultDataAccessException" inside "StatsController".
- Optimize gradle's performance.
- Code polishing.
2022-12-15 14:04:22 +02:00
Lampros Smyrnaios
bfdf06bd09
- Apply error-checking on individual CallableTasks and in tasks-batches related to the creation and upload of all the data related to the "attempt" and "payload" table. So, if no data could be uploaded for one or both tables, no "load"-queries will be executed for that/those tables.
...
- Catch the more general "Exception", inside "FileUtils.mergeParquetFiles()", in order to be certain that the "SQLException" can also be caught.
- Code polishing.
2022-12-09 12:46:06 +02:00
Lampros Smyrnaios
0209d24068
- Change the parquet compression from "Snappy" to "Gzip", as there is an unhandleable exception when the app is running inside a Docker Container and uses the "Snappy" compression.
...
- Code polishing.
2022-12-08 16:28:41 +02:00
Lampros Smyrnaios
c8baf5a5fc
- Fix not finding the parquet-schema files when the app was run inside a Docker Container.
...
- Update the "namespaces" and the "names" inside the parquet schemas.
- Code polishing.
2022-12-08 12:16:05 +02:00
Lampros Smyrnaios
b0c57d79a5
- When the Controller cannot retrieve any assignments from Impala (without an error), return an HTTP-"MULTI_STATUS" with an empty "AssignmentsResponse", instead of an "INTERNAL_SERVER_ERROR".
...
- Fix an error-message.
2022-12-05 16:44:00 +02:00
Lampros Smyrnaios
577ea983e8
- Improve some log-messages.
...
- Set some optimization settings for gradle.
- Fix error-handling in "installAndRun.sh".
- Update dependencies.
2022-11-30 16:28:39 +02:00
Lampros Smyrnaios
6226e2298d
- Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.
...
One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it.
- Code polishing.
2022-11-10 17:18:21 +02:00
Lampros Smyrnaios