Reusing the same context on all calls can lead to hard to debug failures
like
Action function organization_show did not call its auth function
In this case that was caused because the first organization/group_show
raised a NotFound so the auth audit was still in the context. When
organization/group_show was called again at the end of
organization/group_create the auth audit exception was raised.
This commit makes sure that each call has its own context.
With this change, all harvesters that extend the base harvester have the
possibility to use the very useful create_or_update method, but still
define their own way of detecting what package is the existing one.
This is very useful for harvest sources that have no knowledge of the
CKAN internal id, but have another way of finding previous package.
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
* Harvesters that change the name when the title changes have had a
problem when the change is small and a number was unnecessarily
appended. e.g. "Trees "->"Trees" meant _gen_new_name("Trees") returned
"trees1". Now you can specify the existing value and it will return
that if it still holds.
* Maximum dataset name length is now adhered to.
* To make a name unique, a sequential number is now added, since for
users that is more understandable and pleasant. However hex digits are
still an option, for those that want to harvest concurrently.
etj
David Read
Adrià Mercader
Mark Gregson
seitenbau-govdata
Jardel Weyrich
Petar Efnushev
Stefan Oderbolz
Raphael Stolt