Because plugins are loaded for every paster command, regardless of which
paster command is being run, when lots of plugins are installed startup
time can be substantial.
To fix this, making sure that plugin.py in various extensions doesn't
import unnecessary (and potentially slow) modules, this PR moves the
imports of some function into the function where they are called.
This reduces paster startup time (for me) by up to .5 seconds, which
while not significant by itself, contributes to my current 8 second
startup time.
Set the L.TileLayer as a TMS. It's required if the base layer specified in custom.url is a TMS. The default value of tms is false in Leaflet.
To use the TMS option, ```ckanext.spatial.common_map.tms = true``` has to be set in the ckan .INI file.
Same as https://github.com/ckan/ckanext-geoview/pull/24
* Keep old entry points for preview plugins so people does not need to
do python setup.py develop
* Version checks should be done including the min_version parameter,
otherwise they will fail for 2.4, 2.5...
* No need to define different methods for different interfaces. If
running eg 2.3, the IResourcePreview interface methods will simply be
ignored.
* Simplified same domain checking
* Add note about these being eventually migrated to ckanext-geoviews
* Added docs
Completely reviewed and refactored tests to support both CKAN 2.2/2.3,
geoalchemy 1/2 and PostGIS 1/2 (TODO test on PostGIS 1.5)
Removed all references to legacy ckan tests, and removed old or
incomplete tests.
This allows code in lib, model etc to not worry about GeoAlchemy
versions. This assume the correct library installed, as the check has
been done when starting up.
If using CKAN>=2.3 and GeoAlchemy2 is not installed an ImportError
will be raised with details about how to proceed. Added a section
on the Troubleshooting.
Moved the call to get_site_user higher on base.py so it's available to
extensions. Also added the parsed XML etree so it does not need to be
parsed from the string again.
Two extension points: ``get_package_dict`` and ``transform_to_iso``,
with the same expected behaviour as the old hooks meant to be overriden.
For ``get_package_dict`` we now pass, apart from the generated
package_dict, the parsed iso_values and the harvest object.
Updated docs and added autodocs.
Keywords were extracted via 3 different elements
`keyword-inspire-theme`, `keyword-controlled-other` and
`keyword-free-text`. The latter didn't actually do anything and the
second duplicated xpaths from the first and added a non-standard one.
A new `keywords` key has been added which contains all keyword objects,
including type. This is not used to modify the `tags` key right now.
The common map function has been extended to allow configuring different
options for the base layer. There are default presets for MapQuest and
MapBox, but any XYZ layer can be added configuration options. The
options are passed to the templates via a helper function and to the js
modules via data-module attributes. Full docs included.
Based on the work by @kindly on https://github.com/kindly/ckan_pycsw_import,
adds a command to the extension that allows to export easily the spatial
datasets to pycsw. This will only work for harvested ISO datasets for the time
being.
The command can be run both via paster (in the context of a CKAN install) or
independently (eg a separate server), as it uses the API to get the datasets.
TODO docs
All the 'self' on the 'showError' function should be 'this', otherwise you
will refer to the window. Also 'this.i18n' assumes an 'i18n' key in the
module options. Looks like you are trying to pass some variables to the
string template, note the sprintf-like syntax.
Adds a new map widget to the 2.0 search templates. It is shown initially
in the sidebar but it is expanded when the user needs to draw an area.
It uses Leaflet and Leaflet.draw.
Otherwise you get this exception when the date on the document has time
zone information, as dates as stored without it on the database:
TypeError: can't compare offset-naive and offset-aware datetimes
This backend does not use the spatial field on Solr, but indexes the
area and coordinates of the dataset's extent. On query time, a boost
function is used to calculate a ratio between the query area and the
dataset extent, which will give a value of 0 for areas that don't
overlap and 1 for a perfect fit. The same function is used both for
sorting and filtering.
To return correct results on a spatial query, rectangle geometries must
be defined in counter-clockwise order [1]. This changeset adds a small
sanity check to before_index when we are dealing with a Polygon geometry
that has 5 coordinate pairs. Shapely is used to generate a LinearRing
from the polygon coordinates and check if they are ccw. If not, they are
reordered and a new polygon is generated so the WKT sent to Solr is
properly ordered.
The GeoJSON template used for extents in the base spatial harvester has
been also updated to define the coordinates counter-clockwise.
[1]
http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4#JTS_.2BAC8_WKT_.2BAC8_Polygon_notes
When the 'ckanext.spatial.search_backend' config option is set to
'solr', the extension will index geometries stored in the 'spatial'
extra on the spatial field of the Solr index (named 'spatial_geom').
This is done on the 'before_index' extension point.
Also, when doing a query, if the same config option is in place, the
necessary fq parameter will be set to pass the spatial query to Solr.
Some common problems:
* Whitespace, tabs, line feeds and plus signs: should be handled by
float()
* Text: log error and skip creation of spatial extra
* Same set of 2 coords for extent: create point instead of polygon
Note that the bbox values are stored as they are in the bbox-xx-yy
extras
The 'guess_resource_format' function looks for common patterns in popular
geospatial services and file extensions. It just looks at the provided URL,
it does not attempt to perform any remote check. By default, it will use the
mimetypes module if no match was found before to try to guess the format.
On the previous version, all resources in documents of type 'service' were
queried to see if they were actually WMS. This is no longer the case,
but services flagged as 'wms' can be verified if the following setting
is set to True: ckanext.spatial.harvest.validate_wms
Some improvements on the endpoints that return the contents of the
harvest objects:
* Nicer URLs with redirects to the old ones
* Returning the raw harvest object content is available on the main
harvest extension, so just redirect there
* Support for showing the original document of a harvest object, if
present
* Suport for defining a custom XSLT for the HTML view, via
ckanext.spatial.harvest.xslt_html_content
ckanext.spatial.harvest.xslt_html_content_original
The import CLI reruns the import stage for the last current objects, so
when running it, the previous objects don't need to be changed. Any
date check is overridden to force the update of the package.
This can be set instance wide on the ini file with
ckanext.spatial.harvest.continue_on_validation_errors
or per source, adding continue_on_validation_errors=true to the source
config.
You don't need to create a 'harvest' sysadmin user any more.
By default this will be the internal site admin user. This is the
recommended setting, but if necessary it can be overridden by
the `ckanext.spatial.harvest.user_name` config option, eg to
support the old hardcoded 'harvest' user
This can be overridden by custom harvesters willing to support non ISO
formats (like FGDC). It is called whrn the original_document and
original_format harvest object extras are present. Custom harvesters are
responsible for transforming the original document to ISO.
Prior to the merging of the new spatial harvesters, the existing ones
based on Gemini and UKLP have been moved to their own namespace
(ckanext.spatial.harvesters.gemini). The plugin points have been updated
so users currently using these harvesters will still be able to use them
as normal.
The base harvester (SpatialHarvester) has been updated with new methods,
most significally '_get_package_dict' and 'import_stage'. Note that
SpatialHarvester now extends HarvesterBase on ckanext-harvest, which had
some of its methods updated.
TODO: still some geo.data.gov specific bits!
Changes in multiplicity to support the ISO 19115 spec rather than just
the Gemini 2 one. Thanks to @dread for his help on this.
Summary of the changes:
* dataset-reference-date: Set to 1..*
Note that there was a bug with mutliple values allowed per date.
Returned object should now be like:
"dataset-reference-date": [{"type": "creation", "value": "2004-02"},
{"type": "revision", "value": "2006-07-03"}]
* metadata-languge: Set to 0..1
* resource-type: Set to *. That means that a list is now returned
* bbox: Set to *. Note that bboxes are now returned as objects such as:
[{"north":xxx, "south":xxx,, "east":xxx, "west":xxx}, {"north":xxx,
"south":xxx,, "east":xxx, "west":xxx}]
The existing Gemini based harvesters and validators have been adapted,
all tests pass.
Add new mandatory fields when creating sources, status dict has new
keys, CKAN lower cases formats, take into account harvest source
datasets.
Added a local getcapabilities response to avoid remote 404s.
Note that the TestValidation tests need to be fixed, as 27c4ee81e
removed the validation from the gather stage.
- Added a config option ('ckanext.spatial.use_postgis_sorting') to
activate this as this behaviour will be deprecated in the future
in favour of Solr 4 spatial sorting capabilities.
Also fixed the tests
Conflicts:
ckanext/spatial/plugin.py
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit 91e547a622.
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit ecd6036efe.
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit 01536873b9.
Conflicts:
ckanext/spatial/harvesters.py
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.
This reverts commit 3f627d9700.
To make easier to filter and display errors on the UI, the validators
have been modified to return the message and line number separately. The
return format for validators is now:
(is_valid, [(error_message_string, error_line_number)])
Also the XSD based validators were returning only the last error found on
the document, instead of iterating the whole error log. Harvesters should
create a harvest object error for each of this validation errors.
Tests have been adapted to these changes.
Sometimes, even when requesting 10 records, the CSW server returns less
of them (see eg http://goo.gl/b7Rdj, only 9 records returned). The
current check made the process stop on this case, missing other
identifiers.
The profiles used are decided as follows:
1. 'validator_profiles' property of the harvest source config
object
2. 'ckan.spatial.validator.profiles' configuration option in
the ini file
3. Default value as defined in DEFAULT_VALIDATOR_PROFILES
Tests are passing, apart from a couple which didn't work before:
* test_functional.py -> functional/test_package.py (3 failures in 4)
* functional/test_dataset_map.py (1 fail in 1)
There may be some code errors still untested.
Renamed Validator -> Validators to make more sense.