7257258ca4
mark new harvest objects as current
...
When a new harvest_object for a new package was being created, it
was immediately being marked as false, as all objects were marked
as false, including the new object just created and newly marked
as current=true.
Fix so that old HarvestObjects are only marked as current=False
when updating an existing package.
2013-03-07 20:27:27 +00:00
9432368bea
fix gather_stage if there is a previous job
...
change check on gather stage to check for changed packages since
last job instead of current harvest job's gather_start
fix attribute look up bug
fix print_job to print 0 gather_errors instead of key error
2013-02-28 19:06:21 +00:00
ba486a9482
add indexing of datasets whilst harvesting
2013-02-27 11:34:09 +00:00
f97e3b4c6c
add return True to import stage of ckanharvester
...
Was causing queue.py to report that the import had errored.
2013-02-22 10:13:36 +00:00
177349fd76
Update HarvesterBase
...
This is a convenience class that other harvesters can extend. Updates
include a cleanup of old functions and porting of enhancements from the
spatial harvesters.
2013-02-12 16:10:13 +00:00
871eae94b6
[ckan harvester] Fix bug on force all check
2012-03-15 11:31:12 +00:00
f210455aef
[ckan harvester] Replace title on default extras
2012-03-13 12:38:14 +00:00
e0bef2ef9c
[base] Minor fix for harvesters without config
2012-03-12 14:46:28 +00:00
50537a6738
Merge branch 'master' into enh-1726-harvesting-model-update
2012-02-15 12:01:15 +00:00
9ed152cbea
[ckan harvester] Add support for forcing gathering of all remote packages
2012-02-03 17:54:34 +00:00
479750da09
[ #1726 ][base harvester] Set current field when importing
2012-02-02 13:18:43 +00:00
eb646b3385
[ckan harvester] Add support for defining default extras
2012-01-10 17:07:19 +00:00
ae51093213
[ckan harvester] Ignore __junk field, was causing imports to fail
2012-01-10 14:46:12 +00:00
da469ab08e
[base harvester] Custom tag munge function. TODO: check with flexible tags
2011-11-23 11:05:52 +00:00
cfaba6e1e8
[ckan harvester] Add support for sending an API key
2011-11-21 17:29:10 +00:00
0ab5c53b47
[ckan harvester] Fix typo
2011-11-18 17:53:01 +00:00
994590531e
[ckan harvester] Support for creating read-only packages
2011-11-18 14:30:10 +00:00
c939d90dbb
[ckan harvester] Support for defining a custom user to do the harvesting
2011-11-18 14:12:30 +00:00
2018d9e513
[ckan harvester] Support for default tags and groups
2011-11-18 13:20:41 +00:00
c04d80e27e
Use get_action function instead of directly calling the action functions
2011-10-26 17:26:18 +01:00
7927329536
Make harvesters work with latest ckan release
2011-07-29 11:31:03 +01:00
cabbb4922d
Use API version defined in config if present
2011-07-18 17:35:32 +01:00
c867660e7d
Add docs to base harvester functions
2011-07-18 17:35:03 +01:00
54de6759fe
Fix bug with empty config
2011-06-28 15:04:40 +01:00
c80e68a12f
Ensure the correct configuration is used on each stage
2011-06-14 15:59:13 +01:00
3125bb1514
Add a check to ensure sources with no packages are reharvested
2011-06-14 12:59:48 +01:00
98bfd50f47
Load config in the CKAN harvester
2011-06-07 13:35:11 +01:00
6e75d362e3
Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method.
2011-06-07 12:07:53 +01:00
89934b8538
[harvesters] factor out a base harvester for use in generic harvesting apps
2011-06-02 12:07:07 +02:00
joetsoi
amercader
Friedrich Lindenberg