1a6dca7c00
[ #148 ] Catch a more specific exception.
2015-10-01 12:30:40 +01:00
de17e0ae8c
Catch, record and recover from temporary db problems.
2015-07-22 10:25:11 +01:00
9f8aae3a18
Append site id to queue name
...
This allows multiple CKAN sites to share the same RabbitMQ exchange
(For the Redis backend this is handled via different Redis databases)
2015-06-01 17:54:22 +01:00
55d2b4e304
Fix purge command
2013-10-16 12:59:23 +01:00
f89f12203c
Merge branch 'fix/rename-ampq-to-amqp' of git://github.com/opendatatrentino/ckanext-harvest into opendatatrentino-fix/rename-ampq-to-amqp
2013-10-04 17:24:53 +01:00
611b9aab6d
Fixed typo: ampq -> amqp
2013-09-19 11:43:03 +02:00
cc3f3d3426
[ #50 ] Fix objects deletion on gather exceptions
2013-07-05 13:29:11 +01:00
e2696b98bb
[ #50 ] Save all dates as UTC in the database
...
At some point we may want to transform these to local time at the
dictization level. We will need a library like dateutil to handle it
properly though.
2013-07-04 14:59:27 +01:00
9041f3f3ad
Changes in Redis conusmer to make tests work
2013-04-22 18:08:19 +01:00
dcfd201cdd
[ #32 ] redis queue support
2013-04-21 17:04:57 +01:00
0ce59a29b6
delete insead of update harvest objects when error
2013-04-12 12:32:33 +01:00
7d7657f94a
make gather phase as finished if there is an error
2013-04-12 10:35:08 +01:00
0b5c3c608a
catch and raise gather exception, acking the message
2013-03-25 11:57:57 +00:00
634a0bbd30
return instead of continue
2013-03-19 01:21:20 +00:00
3adf38105e
readd code from old branch seperating the fetch and import logic
2013-03-19 01:16:43 +00:00
d77f16aba9
[ #21 ] Improve gather stage error handling
...
See issue for full details. Basically we don't want to catch any
exception at the queue.py level, as they prevent debugging. Harvesters
should deal with them and return a list of ids or an empty list if no
objects need to be fetched.
Also improved the debug messages.
2013-03-14 17:31:07 +00:00
5c17a525c1
Refresh session after each harvest stage
...
Otherwise the eg the source config got cached and you needed to restart
the consumers to refresh it.
2013-03-01 12:55:59 +00:00
ebe246fe99
make report emit added so shows up on front end
2013-02-22 17:32:33 +00:00
acb17ff3b0
capture errors more cleanly
2013-01-10 10:48:48 +00:00
36389e7ce0
make sure gather phase finishes job if there is a severe error
2012-12-24 12:21:21 +00:00
6b42d96fe0
add report_status field
2012-12-17 23:50:26 +00:00
0dde483992
Set job status to Finished when actually finishing it
...
Until now, harvest jobs were set to Finished just after sending all
objects to the fetch stage. Now every time the run command is run, jobs
are set to Running, and all previous Running jobs are checked to see if
all harvest objects have a state of Complete or Error. Only then the job
is flagged as Finished.
2012-12-13 18:19:22 +00:00
37efb3b978
Set harvest object state depending on the output of import_stage
...
Either to COMPLETE or ERROR, depending on whether it returns True or
False.
2012-12-13 14:30:13 +00:00
03fd1884f4
Implement retry times for harvest objects
2012-11-15 18:11:35 +00:00
c9c1eb4848
use generator to consume
2012-11-15 14:14:55 +00:00
33d5e09722
Change fetch_callback to proper acknowledge objects
2012-11-15 11:36:06 +00:00
5063626554
make sure state is changed to error on fetch error
2012-11-07 09:53:16 +00:00
28e5e9137a
add perge queues command
2012-11-07 09:51:25 +00:00
d7f8c9165c
Merge branch 'model_upgrade' into release-v2.0
2012-10-30 18:07:24 +00:00
82a498d9fc
Rename function to be implementation independent
2012-10-30 17:13:39 +00:00
bc079c6644
model upgrade with tests and migration
2012-10-25 19:01:54 +01:00
da125cdcc2
pika now used as queue library
2012-10-24 00:34:32 +01:00
ccf0cd3da2
Add copious logging to record what happens in harvesting.
2012-06-08 17:09:22 +01:00
718202d886
Logging is now assured.
2012-04-10 20:53:29 +01:00
00e911a70c
Fix name of the queue logger. Moved imports of ckanext.harvest until after _load_config so that the loggers do not start disabled.
2012-04-10 20:10:17 +01:00
2b98080266
[merge] from new-forms, as forms refactoring has been merged in core
2011-05-20 13:50:15 +01:00
565eaf3d0a
Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form
2011-05-13 18:39:36 +01:00
1edf984f1b
Log an error if no harvester could be found for the source
2011-04-15 16:12:47 +01:00
e2faa14b8d
[refactoring] Save gathering starting and finishing time
2011-04-11 16:29:47 +01:00
b38fc57ec5
[refactoring] Changes in the queue code and the IHarvester interface"
2011-04-07 16:59:11 +01:00
56d5acc867
[refactoring] Add code to handle queuing and the new IHarvester interface. Add
...
new commands in the CLI to start the queue consumers and fire the harvesting
process.
2011-04-06 12:45:00 +01:00
David Read
amercader
Samuele Santi
kindly
David Read