Lampros Smyrnaios 6226e2298d - Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement. ... One side effect of using the parquet-files, is that the timestamps are now BIGDECIMAL numbers, instead of "Timestamp" objects, but, converting them to such objects is pretty easy, if we ever need to do it. - Code polishing.		2022-11-10 17:18:21 +02:00
..
components	- Avoid getting and uploading (to S3), full-texts which are already uploaded by previous assignments-batches.	2021-12-21 15:55:27 +02:00
configuration	- Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.	2022-11-10 17:18:21 +02:00
controllers	- Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.	2022-11-10 17:18:21 +02:00
models	- Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.	2022-11-10 17:18:21 +02:00
payloads	- Fix a "databaseLock" bug, which could cause both the payload and attempt inserts and the "mergeParquetFiles" to fail, as the inserts could be executed concurrently with tables-compaction.	2021-11-30 13:26:19 +02:00
security	- Update the Spring-Security-code to use the "SecurityFilterChain", as the previous code was deprecated.	2022-06-27 21:41:32 +03:00
util	- Upgrade the results-loading process: Instead of making thousands of sql-insert requests to Impala now we write the results to parquet files, upload them to HDFS and then import the data into the Impala tables with just 2 requests. This results in a huge performance improvement.	2022-11-10 17:18:21 +02:00
Application.java	Update the "UriBuilder.java" to be able to acquire the running port of the server, in case the port-number was initially set to "random" (0). Also make sure we get the "localHostAddress" and not the "localHostName", in case the public IP is not retrievable.	2022-09-12 17:04:05 +03:00