05094090af
Change type of the API version to integer
...
The CKAN logic uses integers when dealing with the API version, e.g.
making checks which API version is in use. Currently, the harvester
uses strings to identify the API version. Instead of dealing with
type conversion the harvester could use integers directly.
This commit fixes okfn/ckanext-harvest#36 . When the API version is
parsed from the configuration it is passed through the int() function.
This way the harvesting will still work even if a harvest source was
configured with a string API version which makes this commit backward
compatible.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 12:51:48 +02:00
3d2867ca04
[ #17 ] Remove ckanclient dependency as it is not used
2013-05-24 17:55:37 +01:00
f1d11c1307
[ #17 ] Import remote groups in CKAN harvester
...
This is a cleaner commit of the great work done by @platzhirsch
implementing remote groups import on the CKAN harvester.
2013-05-24 16:55:05 +01:00
1efd7ab4cd
Ignore remote orgs in CKAN harvester
...
If #17 progresses we can do somethign similar for them, although it amy
be more complicated because of authorization issues.
2013-05-16 17:30:54 +01:00
c754479014
#29 make new idatasets form work with harvest source form
2013-03-25 17:38:07 +00:00
7257258ca4
mark new harvest objects as current
...
When a new harvest_object for a new package was being created, it
was immediately being marked as false, as all objects were marked
as false, including the new object just created and newly marked
as current=true.
Fix so that old HarvestObjects are only marked as current=False
when updating an existing package.
2013-03-07 20:27:27 +00:00
9432368bea
fix gather_stage if there is a previous job
...
change check on gather stage to check for changed packages since
last job instead of current harvest job's gather_start
fix attribute look up bug
fix print_job to print 0 gather_errors instead of key error
2013-02-28 19:06:21 +00:00
ba486a9482
add indexing of datasets whilst harvesting
2013-02-27 11:34:09 +00:00
f97e3b4c6c
add return True to import stage of ckanharvester
...
Was causing queue.py to report that the import had errored.
2013-02-22 10:13:36 +00:00
177349fd76
Update HarvesterBase
...
This is a convenience class that other harvesters can extend. Updates
include a cleanup of old functions and porting of enhancements from the
spatial harvesters.
2013-02-12 16:10:13 +00:00
871eae94b6
[ckan harvester] Fix bug on force all check
2012-03-15 11:31:12 +00:00
f210455aef
[ckan harvester] Replace title on default extras
2012-03-13 12:38:14 +00:00
e0bef2ef9c
[base] Minor fix for harvesters without config
2012-03-12 14:46:28 +00:00
50537a6738
Merge branch 'master' into enh-1726-harvesting-model-update
2012-02-15 12:01:15 +00:00
9ed152cbea
[ckan harvester] Add support for forcing gathering of all remote packages
2012-02-03 17:54:34 +00:00
479750da09
[ #1726 ][base harvester] Set current field when importing
2012-02-02 13:18:43 +00:00
eb646b3385
[ckan harvester] Add support for defining default extras
2012-01-10 17:07:19 +00:00
ae51093213
[ckan harvester] Ignore __junk field, was causing imports to fail
2012-01-10 14:46:12 +00:00
da469ab08e
[base harvester] Custom tag munge function. TODO: check with flexible tags
2011-11-23 11:05:52 +00:00
cfaba6e1e8
[ckan harvester] Add support for sending an API key
2011-11-21 17:29:10 +00:00
0ab5c53b47
[ckan harvester] Fix typo
2011-11-18 17:53:01 +00:00
994590531e
[ckan harvester] Support for creating read-only packages
2011-11-18 14:30:10 +00:00
c939d90dbb
[ckan harvester] Support for defining a custom user to do the harvesting
2011-11-18 14:12:30 +00:00
2018d9e513
[ckan harvester] Support for default tags and groups
2011-11-18 13:20:41 +00:00
c04d80e27e
Use get_action function instead of directly calling the action functions
2011-10-26 17:26:18 +01:00
7927329536
Make harvesters work with latest ckan release
2011-07-29 11:31:03 +01:00
cabbb4922d
Use API version defined in config if present
2011-07-18 17:35:32 +01:00
c867660e7d
Add docs to base harvester functions
2011-07-18 17:35:03 +01:00
54de6759fe
Fix bug with empty config
2011-06-28 15:04:40 +01:00
c80e68a12f
Ensure the correct configuration is used on each stage
2011-06-14 15:59:13 +01:00
3125bb1514
Add a check to ensure sources with no packages are reharvested
2011-06-14 12:59:48 +01:00
98bfd50f47
Load config in the CKAN harvester
2011-06-07 13:35:11 +01:00
6e75d362e3
Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method.
2011-06-07 12:07:53 +01:00
89934b8538
[harvesters] factor out a base harvester for use in generic harvesting apps
2011-06-02 12:07:07 +02:00
Konrad Reiche
amercader
kindly
joetsoi
Friedrich Lindenberg