18a506a112
[ #249 ] Add test for default_extras.
2016-06-10 09:51:17 +00:00
f1742fb51a
Fix default_groups. It accepted a list of package_name/ids and was trying to add this to the package, but the package needs a dict. Added test.
2016-06-10 09:16:32 +00:00
bfc9b8e0d9
[ #249 ] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE.
2016-06-09 22:11:03 +00:00
16a6e9fbf6
Add tests for group creation during harvesting
2016-05-23 10:20:52 +01:00
49faa0ae6c
Tests for CKANHarvester._last_error_free_job
2016-02-15 13:30:28 +00:00
52c071dbe9
Improved error handling. e.g. if the site it harvests just returns errors.
2016-02-15 12:10:44 +00:00
331ad84272
Deal with worry about datasets on the remote CKAN being added/removed during harvest.
2016-02-12 18:00:00 +00:00
7096b7ddf2
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
2016-02-12 16:51:26 +00:00
4cc39e5ef4
Merge branch 'test-guid-fix'
2015-12-10 13:15:54 +00:00
f0a2e9fb8e
CKAN revision API returns package ids not names (for v2 of the API). This ensures harvest guid is always the ID rather than the name.
2015-11-24 16:41:43 +00:00
52f7e0dd07
Use the ckan version of munge_tag if available, but provide a fallback for older ckans.
2015-11-23 12:48:05 +00:00
ede50aa3fb
Merge branch 'immediate-harvest'
2015-11-20 12:28:35 +00:00
3f42eb6ba0
Merge branch 'revision-id-fix'
...
Conflicts:
ckanext/harvest/tests/harvesters/test_ckanharvester.py
2015-11-20 12:28:17 +00:00
735ab3e286
[ #157 ] Try to fix test for ckan 2.2 - cf 91afc0e928
2015-11-04 11:37:03 +00:00
91afc0e928
[ #178 ] Fix test for ckan 2.2, which does not suffer problem #180 .
2015-11-03 10:01:47 +00:00
f4f124c181
[ #178 ] Fix resouce_revision_id_fkey error. Fixes #178 .
2015-11-03 07:38:01 +00:00
c7fac36c1c
[ #107 ] "unchanged" response tested and related fixes
...
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
2015-11-03 00:22:53 +00:00
24415844e0
[ #158 ] Fix revision_id problem in second harvest.
2015-11-02 18:13:29 +00:00
1a680f3fd3
[ #158 ] Fix spaces encoding broken in previous merge. Tested with data.gov.uk.
2015-10-29 17:31:04 +00:00
f1d2d5fdc4
[ #111 ] Run jobs straight away.
2015-10-28 21:58:36 +00:00
e2ab9e58e7
Merge remote-tracking branch 'origin/master' into 157-version-three-apify
...
Conflicts:
ckanext/harvest/harvesters/ckanharvester.py
2015-10-28 14:34:27 +00:00
3f74c29c99
Merge branch 'master' into 157-version-three-apify
2015-10-27 17:45:27 +00:00
2a79873855
[ #158 ] Use package search to get all datasets. Add paging search results. Store pkg_dict from search in the object rather than request it again in fetch_stage.
2015-10-27 17:33:22 +00:00
86630adab7
Merge branch 'include-exclude-org'
2015-10-27 15:52:55 +00:00
b56fae8aed
Fixes and tests
...
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
2015-10-23 17:30:28 +00:00
24574f485b
Setup harvest model in harvester tests
2015-10-23 15:43:01 +01:00
bc49149d5e
Merge branch 'master' into include-exclude-org
2015-10-23 14:36:53 +01:00
2f4adfb338
Merge branch 'tests'
2015-10-23 13:18:15 +01:00
eb9aa17862
Include/exclude orgs funcationality based on work by memaldi and ross.
2015-10-21 16:33:16 +00:00
f70c16bce7
Add framework for testing harvesters. Modernize existing tests.
2015-10-21 16:26:57 +00:00
82bdff2f34
Add tests
2015-10-01 17:59:17 +01:00
David Read
amercader