// TODO - Implement an endpoint in the Controller to request the Controller to shutdown everything.
// The controller will make sure that it has finished with requesting the full-texts and sent a "shutDownRequest" to each worker (as we will have its IP)
// (some shutdown may fail (for any reason), but that should not halt the process ?)
// after the shut-Down-request have been sent the endpoint return the message that the shutdown process is in progress.
// TODO - Make another endpoint in the Controller to take POST requests from the workers about their shutdown-process.
// This endpoint will assign to the worker-s hashmap the value of "hashShutdown=true", and check if all the workers have finished.
// So, if we have the info that the current shutdown worker in the last one, then show a log-message and shutdown the Controller.
// TODO - Will the "last one" be the "actual last one" ? What if we had 7 workers but one crashed and now we have 6 workers to shutdown properly but the 7th worker seems to be still working..?
// In that case, we can cross it out easily, as the Controller will get either a "Connection refused" or a "connection timeout", depending on the state of the worker.
// TODO - Make the Worker to sent a POST request to the Controller to notify it that is has finished all work and it is about to close.
// As the Impala-driver is buggy and struggles to support parameterized queries in some types of prepared-statements, we have to sanitize the "workerId" ourselves.
logger.info("Worker with id: \""+workerId+"\", requested "+workerAssignmentsLimit+" assignments. The assignments-limit of the controller is: "+assignmentLimit);
logger.warn("The given \"workerAssignmentsLimit\" ("+workerAssignmentsLimit+") was larger than the Controller's limit ("+assignmentLimit+"). Will use the Controller's limit.");
logger.info("The worker \""+workerId+"\" is requesting assignments for the first time. Going to store its IP and create the remote parquet subdirectories (in HDFS).");
// Create extra subdirectories in HDFS parquet-directories, so that the parquet directory does not become empty right before "loading" the data to the DB, in case another worker loaded multiple-worker's data to the DB.
// As the Impala-driver is buggy and struggles to support parameterized queries in some types of prepared-statements, we have to sanitize the "workerId" ourselves.
logger.info("Received the WorkerReport for batch-assignments_"+curReportAssignments+", from the worker with id: "+curWorkerId+". It contains "+sizeOUrlReports+" urlReports. Going to request the fullTexts from the Worker and insert the UrlReports into the database.");