5c17a525c1
Refresh session after each harvest stage
...
Otherwise the eg the source config got cached and you needed to restart
the consumers to refresh it.
2013-03-01 12:55:59 +00:00
ebe246fe99
make report emit added so shows up on front end
2013-02-22 17:32:33 +00:00
acb17ff3b0
capture errors more cleanly
2013-01-10 10:48:48 +00:00
36389e7ce0
make sure gather phase finishes job if there is a severe error
2012-12-24 12:21:21 +00:00
6b42d96fe0
add report_status field
2012-12-17 23:50:26 +00:00
0dde483992
Set job status to Finished when actually finishing it
...
Until now, harvest jobs were set to Finished just after sending all
objects to the fetch stage. Now every time the run command is run, jobs
are set to Running, and all previous Running jobs are checked to see if
all harvest objects have a state of Complete or Error. Only then the job
is flagged as Finished.
2012-12-13 18:19:22 +00:00
37efb3b978
Set harvest object state depending on the output of import_stage
...
Either to COMPLETE or ERROR, depending on whether it returns True or
False.
2012-12-13 14:30:13 +00:00
03fd1884f4
Implement retry times for harvest objects
2012-11-15 18:11:35 +00:00
c9c1eb4848
use generator to consume
2012-11-15 14:14:55 +00:00
33d5e09722
Change fetch_callback to proper acknowledge objects
2012-11-15 11:36:06 +00:00
5063626554
make sure state is changed to error on fetch error
2012-11-07 09:53:16 +00:00
28e5e9137a
add perge queues command
2012-11-07 09:51:25 +00:00
d7f8c9165c
Merge branch 'model_upgrade' into release-v2.0
2012-10-30 18:07:24 +00:00
82a498d9fc
Rename function to be implementation independent
2012-10-30 17:13:39 +00:00
bc079c6644
model upgrade with tests and migration
2012-10-25 19:01:54 +01:00
da125cdcc2
pika now used as queue library
2012-10-24 00:34:32 +01:00
ccf0cd3da2
Add copious logging to record what happens in harvesting.
2012-06-08 17:09:22 +01:00
718202d886
Logging is now assured.
2012-04-10 20:53:29 +01:00
00e911a70c
Fix name of the queue logger. Moved imports of ckanext.harvest until after _load_config so that the loggers do not start disabled.
2012-04-10 20:10:17 +01:00
2b98080266
[merge] from new-forms, as forms refactoring has been merged in core
2011-05-20 13:50:15 +01:00
565eaf3d0a
Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form
2011-05-13 18:39:36 +01:00
1edf984f1b
Log an error if no harvester could be found for the source
2011-04-15 16:12:47 +01:00
e2faa14b8d
[refactoring] Save gathering starting and finishing time
2011-04-11 16:29:47 +01:00
b38fc57ec5
[refactoring] Changes in the queue code and the IHarvester interface"
2011-04-07 16:59:11 +01:00
56d5acc867
[refactoring] Add code to handle queuing and the new IHarvester interface. Add
...
new commands in the CLI to start the queue consumers and fire the harvesting
process.
2011-04-06 12:45:00 +01:00
amercader
kindly
David Read
David Read