michal.politowski
/
UrlsController
forked from lsmyrnaios/UrlsController
1
0
Fork 0
Code Pull Requests Activity

Avoid deleting the assignment-records in case of a "parquet-data creation, upload or insertion" problem, in order to avoid double-processing of the urls, until the report has been retried by the scheduler.

This commit is contained in:
Lampros Smyrnaios 2023-09-08 13:44:24 +03:00
parent 718f5cfefb
commit bd9245cc3d
1 changed files with 1 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
src/main/java/eu/openaire/urls_controller/services/UrlsServiceImpl.java
View File

@ -375,12 +375,7 @@ public class UrlsServiceImpl implements UrlsService {
logger.warn("The current thread was interrupted when waiting for the worker-threads to finish inserting into the tables: " + ie.getMessage());
// This is a very rare case. At the moment, we just move on with table-merging.
} catch (RuntimeException re) {
DatabaseConnector.databaseLock.lock();
String assignmentErrorMsg = deleteAssignmentsBatch(curReportAssignmentsCounter);
DatabaseConnector.databaseLock.unlock();
String errorMsg = re.getMessage();
if ( assignmentErrorMsg != null )
errorMsg += "\n" + assignmentErrorMsg;
logger.error(errorMsg);
postReportResultToWorker(curWorkerId, curReportAssignmentsCounter, errorMsg);
return false;
@ -404,6 +399,7 @@ public class UrlsServiceImpl implements UrlsService {
}
// Delete the assignments each time, as they are bound to the "current" assignmentsCounter. Otherwise, they will never be deleted!
// If this method exits sooner, due tio an error, then the assignments are not deleted in order to wait for the schedulers to retry them and not be given to workers, to avoid reprocessing the urls.
DatabaseConnector.databaseLock.lock();
String deleteErrorMsg = deleteAssignmentsBatch(curReportAssignmentsCounter);
if ( deleteErrorMsg != null ) {