Some improvements on the endpoints that return the contents of the
harvest objects:
* Nicer URLs with redirects to the old ones
* Returning the raw harvest object content is available on the main
harvest extension, so just redirect there
* Support for showing the original document of a harvest object, if
present
* Suport for defining a custom XSLT for the HTML view, via
ckanext.spatial.harvest.xslt_html_content
ckanext.spatial.harvest.xslt_html_content_original
The import CLI reruns the import stage for the last current objects, so
when running it, the previous objects don't need to be changed. Any
date check is overridden to force the update of the package.
This can be set instance wide on the ini file with
ckanext.spatial.harvest.continue_on_validation_errors
or per source, adding continue_on_validation_errors=true to the source
config.
You don't need to create a 'harvest' sysadmin user any more.
By default this will be the internal site admin user. This is the
recommended setting, but if necessary it can be overridden by
the `ckanext.spatial.harvest.user_name` config option, eg to
support the old hardcoded 'harvest' user
This can be overridden by custom harvesters willing to support non ISO
formats (like FGDC). It is called whrn the original_document and
original_format harvest object extras are present. Custom harvesters are
responsible for transforming the original document to ISO.
Prior to the merging of the new spatial harvesters, the existing ones
based on Gemini and UKLP have been moved to their own namespace
(ckanext.spatial.harvesters.gemini). The plugin points have been updated
so users currently using these harvesters will still be able to use them
as normal.
The base harvester (SpatialHarvester) has been updated with new methods,
most significally '_get_package_dict' and 'import_stage'. Note that
SpatialHarvester now extends HarvesterBase on ckanext-harvest, which had
some of its methods updated.
TODO: still some geo.data.gov specific bits!
Changes in multiplicity to support the ISO 19115 spec rather than just
the Gemini 2 one. Thanks to @dread for his help on this.
Summary of the changes:
* dataset-reference-date: Set to 1..*
Note that there was a bug with mutliple values allowed per date.
Returned object should now be like:
"dataset-reference-date": [{"type": "creation", "value": "2004-02"},
{"type": "revision", "value": "2006-07-03"}]
* metadata-languge: Set to 0..1
* resource-type: Set to *. That means that a list is now returned
* bbox: Set to *. Note that bboxes are now returned as objects such as:
[{"north":xxx, "south":xxx,, "east":xxx, "west":xxx}, {"north":xxx,
"south":xxx,, "east":xxx, "west":xxx}]
The existing Gemini based harvesters and validators have been adapted,
all tests pass.
Add new mandatory fields when creating sources, status dict has new
keys, CKAN lower cases formats, take into account harvest source
datasets.
Added a local getcapabilities response to avoid remote 404s.
Note that the TestValidation tests need to be fixed, as 27c4ee81e
removed the validation from the gather stage.
- Added a config option ('ckanext.spatial.use_postgis_sorting') to
activate this as this behaviour will be deprecated in the future
in favour of Solr 4 spatial sorting capabilities.
Also fixed the tests
Conflicts:
ckanext/spatial/plugin.py
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit 91e547a622.
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit ecd6036efe.
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit 01536873b9.
Conflicts:
ckanext/spatial/harvesters.py
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit 3f627d9700.
To make easier to filter and display errors on the UI, the validators
have been modified to return the message and line number separately. The
return format for validators is now:
(is_valid, [(error_message_string, error_line_number)])
Also the XSD based validators were returning only the last error found on
the document, instead of iterating the whole error log. Harvesters should
create a harvest object error for each of this validation errors.
Tests have been adapted to these changes.
Sometimes, even when requesting 10 records, the CSW server returns less
of them (see eg http://goo.gl/b7Rdj, only 9 records returned). The
current check made the process stop on this case, missing other
identifiers.
The profiles used are decided as follows:
1. 'validator_profiles' property of the harvest source config
object
2. 'ckan.spatial.validator.profiles' configuration option in
the ini file
3. Default value as defined in DEFAULT_VALIDATOR_PROFILES
Tests are passing, apart from a couple which didn't work before:
* test_functional.py -> functional/test_package.py (3 failures in 4)
* functional/test_dataset_map.py (1 fail in 1)
There may be some code errors still untested.
Renamed Validator -> Validators to make more sense.
When checking whether the core tables have been alredy created it is
best to use package_table.exists(), as are_tables_created reflects the
tables, causing conflicts with other extensions.
This allows ckanext-harvest and ckanext-spatial to be used together on
ckan 1.8 onwards.
The new theme is based on Jinja2 templates. Extensions can define their
own snippets, which can used when creating custom themes. In this case
two snippets have been created, one showing the map on the primary area
and the other on the sidebar.
If filtering the results after the search, recalculating the facets
is too complicated, so we use the alternative approach of doing the
spatial search first and adding the ids as a filter in the Solr query.
Search results coming from the default CKAN search (which are received
via the `search` method of the `IPackageController` interface) are
filtered to only contain the ones intersecting the provided bbox.
The search form temporarily shows a text field, which will be replaced
by a nice map widget.
The spatial_query plugin now automatically checks for the existance of an
'spatial' extra when creating, updating or deleting a package and syncs its
value (in GeoJSON format) with the geometry column of the package_extent
table. The extension now uses GeoAlchemy, that makes performing spatial
operation extremely easy. The geometry field has been updated to support
multiple geometry types (Polygons, Points, MultiPolygons...) and the default
srid has been set as WGS 84 lat/lon (EPSG:4326).
amercader
David Read
kindly
David Read
james gardner