421e6da660
Add run_test, job_abort, source commands
...
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
2015-10-28 17:51:58 +00:00
3c6cc55be0
Only flush keys on the current Redis database
2015-10-23 11:52:22 +01:00
d1f84295f8
purge_queues command now has warning about impact of Redis flushall, plus add some (log) output when you run a purge.
2015-10-21 16:12:40 +00:00
ab76830e85
[ #145 ] Throw + catch a custom exception if there are no jobs to run
...
If there are no harvesting jobs to run, there was always an ugly
exception message when using the paster command. This replaces the ugly
output with a proper message and uses a custom exception to allow others
to deal with this error differently.
2015-07-20 18:41:50 +02:00
88d9ba0397
[ #136 ] Fix broken RabbitMQ queue names
...
The harvester command was still using the old ones.
Use specific ones for testing.
2015-06-11 13:56:22 +01:00
64ff0f3a3a
Use single quotes to be consistent
2015-06-10 16:22:04 +02:00
2a2d85f60c
Wording changes for clearsource and rmsource
2015-06-10 16:19:23 +02:00
8ebb843052
Add documentation for clearsource command
2015-06-10 11:29:24 +02:00
61bc150ae6
Expose clear harvester source as a paster command
2015-06-10 11:19:10 +02:00
7459358fa1
Support for single import commands
...
We are now able to run `paster harvester import` for a single harvest
object or for a single dataset, providing ids or name.
2014-05-15 16:30:30 +01:00
380c14c22c
Fix CLI sources list output
2013-10-16 13:03:32 +01:00
1c36b33aaf
[ #59 ] Ignore auth when using site_user
2013-08-14 12:28:27 +01:00
ffea49ca62
[ #56 ] Update parameters on source create command
...
Add missing title and owner_org fields, remove deprecated user_id and
publisher_id
2013-08-14 11:54:51 +01:00
3494727d3f
[ #56 ] Increase max params number
2013-08-14 11:43:32 +01:00
8e33262026
[ #56 ] Fix syntax error and wrong type
2013-08-14 11:31:23 +01:00
7ae9d6e208
Made print method more robust against KeyErrors
...
This is especially needed if you create a new harvest source which does not have all the optional arguments. Before this lead to a KeyError after the creation of the source. Now this simply output 'None'.
2013-08-05 23:50:30 +02:00
1249564be5
Add additional name argument when creating new harvest source
2013-08-05 23:46:21 +02:00
ade5f83e38
Change key of data_dict from 'type' to 'source_type'
2013-08-05 23:07:25 +02:00
dcfd201cdd
[ #32 ] redis queue support
2013-04-21 17:04:57 +01:00
e64c8ead0f
fix print gather_errors
2013-03-05 12:49:20 +00:00
9432368bea
fix gather_stage if there is a previous job
...
change check on gather stage to check for changed packages since
last job instead of current harvest job's gather_start
fix attribute look up bug
fix print_job to print 0 gather_errors instead of key error
2013-02-28 19:06:21 +00:00
ba486a9482
add indexing of datasets whilst harvesting
2013-02-27 11:34:09 +00:00
daa9a385ff
Update job keys changed on 9ba6e8f
2013-01-24 17:36:58 +00:00
b2b89dfd61
Add command for reindex all harvest sources
2013-01-22 16:43:36 +00:00
202c9d9fcc
use correct queue for gather stage
2012-11-15 14:21:09 +00:00
c9c1eb4848
use generator to consume
2012-11-15 14:14:55 +00:00
13357893ad
Fix typo
2012-11-13 14:41:38 +00:00
28e5e9137a
add perge queues command
2012-11-07 09:51:25 +00:00
6db65b5826
made manual default not null
2012-11-05 13:17:32 +00:00
2529a17304
add jobs at certain frequencies
2012-10-29 17:15:02 +00:00
da125cdcc2
pika now used as queue library
2012-10-24 00:34:32 +01:00
4d2fdeac57
Allow defining segments of harvest objects to import
...
Useful when importing large number of objects, as it allows
parallelization
2012-08-02 18:41:59 +01:00
7011efe5dc
Allow not linking to datasets when importing records
...
With the -j flag, harvest objects are not linked to datasets when
importing. This is useful sometimes when importing records for the first
time.
2012-07-30 12:11:55 +01:00
00e911a70c
Fix name of the queue logger. Moved imports of ckanext.harvest until after _load_config so that the loggers do not start disabled.
2012-04-10 20:10:17 +01:00
e797f50a05
[cli] Fix create job command
2012-03-19 17:28:53 +00:00
763f07fcad
[logic,cli] Add session to the context in cli commands
2012-03-07 15:20:49 +00:00
6cccbb61c9
Bug fix, new job count property had not been updated
2012-03-07 12:10:32 +00:00
3b68298bba
[logic,auth] Use the site user for CLI commands auth checks
2012-03-01 12:46:42 +00:00
a35eb75440
[logic,auth] Add auth logic layer
...
The first version of the auth layer is based on the current policy, i.e.
you need to be sysadmin to perform any action.
TODO: the CLI is still not working.
2012-03-01 12:02:16 +00:00
c798013752
[logic] Refactor the rest of the logic functions (create,update,delete)
2012-02-29 15:20:35 +00:00
651474e9f1
[logic] Refactor logic layer to follow CKAN core conventions
...
To make maintenance easier and better support the upcoming auth checks,
the logic layer has been refactored to mimic the structure of the one on
CKAN core: separate actions and dictize functions and logic functions
receive a context.
Only get functions are included in this commit.
2012-02-29 10:59:02 +00:00
a5cf445fa6
[ #1727 ][lib] Use 'current' field in queries returning harvest objects
2012-02-02 13:20:03 +00:00
31dac7029a
fix to make import stage work on its own
2011-09-28 14:27:28 +01:00
c36d9bdd8e
Add new command to create new jobs for all active sources
2011-09-06 18:25:17 +01:00
ef04ce1774
Add support for config options in CLI
2011-06-13 15:56:19 +01:00
b3a88070e3
[forms] Adapt CLI commands to changes in lib
2011-05-13 16:00:36 +01:00
e320d0588f
Add command to reimport existing harvest objects
2011-05-10 16:06:57 +01:00
5fb6512d82
Do not output messages when running the 'run' command
2011-04-15 15:35:19 +01:00
040e4d6956
[refactoring] Add a command to create the necessary tables in the database
2011-04-13 12:39:53 +01:00
842bea8433
[refactoring] Remove all geo related functions as they now have it own extension ckanext-spatial
2011-04-11 19:19:09 +01:00
David Read
amercader
Stefan Oderbolz
kindly
joetsoi
David Read