Full stack trace is probably not needed as the Redis data corruption is probably happening somewhere else, the error log should make it easier to investigate it when it does happen.
I sometimes see "connection timed out" message which are reported as
sqlalchemy.exc.DatabaseError, so by catching the latter exception, it'd
avoid the harvester to be stuck in "limbo" state.
As DatabaseError is a super-class of OperationalError, the latter would
still be catched.
logging.exception() already logs an ERROR message with exception
information, so there's no need to call both log.exception() and
log.error().
Along the way, make messages uniform in fetch_callback() and
gather_callback().
Previously purging the queue on the Redis backend would clear the whole
database, making it hard to share the same database with other parts of
CKAN. With this commit, only the keys that belong to ckanext-harvest and
the current CKAN instance are purged.
The `ckan.site_id` config option (or `default` if missing) is used to
namespace the Redis keys: routing key and persistance key. Consumers
will only get the relevant keys for their instance.
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
At some point we may want to transform these to local time at the
dictization level. We will need a library like dateutil to handle it
properly though.
See issue for full details. Basically we don't want to catch any
exception at the queue.py level, as they prevent debugging. Harvesters
should deal with them and return a list of ids or an empty list if no
objects need to be fetched.
Also improved the debug messages.
Until now, harvest jobs were set to Finished just after sending all
objects to the fetch stage. Now every time the run command is run, jobs
are set to Running, and all previous Running jobs are checked to see if
all harvest objects have a state of Complete or Error. Only then the job
is flagged as Finished.
Ken Tsang
Edward Kerry
Jari Voutilainen
Gavin Cannizzaro
Denis Laxalde
Florian Brucker
David Read
amercader
Samuele Santi
kindly