Commit Graph

509 Commits

Author SHA1 Message Date
Tom Kralidis a28baf4c7e remove cswinfo 2016-01-13 12:33:27 -05:00
Tobia Di Pisa 3a6342cd9d Metadata license ID check for package 2016-01-13 17:43:13 +01:00
Tom Kralidis c2b8f8ea82 update docs, remove code not used 2015-12-30 11:22:23 -05:00
Tom Kralidis e4ffbe611b remove legacy CSW server implementation 2015-12-26 08:40:49 -05:00
Tom Kralidis eb9cd6fc1c force all CSW GetRecords requests to set ogc:SortBy to ensure proper ordering 2015-12-25 14:09:37 -05:00
Ross Jones e6fa57055f Speed up non-spatial paster commands.
Because plugins are loaded for every paster command, regardless of which
paster command is being run, when lots of plugins are installed startup
time can be substantial.

To fix this, making sure that plugin.py in various extensions doesn't
import unnecessary (and potentially slow) modules, this PR moves the
imports of some function into the function where they are called.

This reduces paster startup time (for me) by up to .5 seconds, which
while not significant by itself, contributes to my current 8 second
startup time.
2015-12-02 10:42:28 +00:00
Nicolas Brisebois-Tétreault 3543ebba87 Set tms option for custom layer
Set the L.TileLayer as a TMS. It's required if the base layer specified in custom.url is a TMS. The default value of tms is false in Leaflet.
To use the TMS option, ```ckanext.spatial.common_map.tms = true``` has to be set in the ckan .INI file.

Same as https://github.com/ckan/ckanext-geoview/pull/24
2015-11-06 10:40:34 -05:00
Harald von Waldow aa68ac5bbf fix map attribution 2015-10-23 11:00:55 +02:00
Mark Gregson 259333c6fa [#114] Updates spatial search query for Solr 5 compatibility 2015-10-06 10:40:30 +11:00
amercader efc8b6c80c [#90] Adjust latitudes when doing spatial queries
If the minx falls below -180 or over 180 adjust minx and maxx to ensure
minx stays within bounds
2015-08-21 13:53:50 +01:00
amercader 4a69cbe9da Merge branch '87-attribution' of https://github.com/jqnatividad/ckanext-spatial into jqnatividad-87-attribution 2015-07-24 20:38:22 +01:00
amercader 8645cceed2 Merge branch 'master' of https://github.com/tomkralidis/ckanext-spatial into tomkralidis-master2 2015-07-24 20:36:31 +01:00
fxia b0b71a5438 [#112] bbox fix only kicks in with positive w-long and negative e-long 2015-07-24 11:41:11 -04:00
Tom Kralidis 36ab8d68fe support multiple GML namespaces 2015-06-17 21:24:23 -04:00
amercader 414255e34a [#102] MapBox tiles require access token 2015-05-07 17:59:21 +01:00
amercader 86c4fbdbf7 [#101] Change Mapquest tiles base URL under https 2015-05-07 17:43:03 +01:00
amercader 0886e15ed5 [#107] Remove geojson and wms views
The GeoJSON Leaflet plugin now lives on ckanext-geoview:

https://github.com/ckan/ckanext-geoview

The old WMS preview has been completely removed in favour of the
OpenLayers based geoview:

https://github.com/ckan/ckanext-geoview#openlayers-viewer
2015-05-07 13:43:52 +01:00
etj a5a703445b Fix delete in CSW. Closes #103. 2015-04-24 12:43:58 +02:00
fxia e530017a8a bbox fix, more 2015-04-17 14:11:41 +01:00
fxia cc0d0b14ce bbox fix. make sure w long value is less than e long value 2015-04-17 14:11:38 +01:00
amercader ba2656287d Remove debug call 2015-04-08 10:24:36 +01:00
amercader a96d134f1e [#95] Fix GeoJSON preview for features without properties 2015-04-08 10:01:55 +01:00
amercader 4a142e8377 [#98] Minor tweaks to @u10313335 PR
* Keep old entry points for preview plugins so people does not need to
  do python setup.py develop

* Version checks should be done including the min_version parameter,
  otherwise they will fail for 2.4, 2.5...

* No need to define different methods for different interfaces. If
  running eg 2.3, the IResourcePreview interface methods will simply be
  ignored.

* Simplified same domain checking

* Add note about these being eventually migrated to ckanext-geoviews

* Added docs
2015-04-02 11:41:50 +01:00
amercader c9c40606c2 Merge branch '2.3' of https://github.com/taijiang-tw/ckanext-spatial into taijiang-tw-2.3 2015-04-02 09:28:02 +01:00
amercader bcd7c0e5e6 [#97] Support PostGIS 1.5 when defining the Geometry column
We need to pass management=True when initializing it:

https://github.com/geoalchemy/geoalchemy2/issues/50
2015-04-01 12:49:24 +01:00
amercader 50df9130cf [#97] The tests that ckanext-spatial deserves
Completely reviewed and refactored tests to support both CKAN 2.2/2.3,
geoalchemy 1/2 and PostGIS 1/2 (TODO test on PostGIS 1.5)

Removed all references to legacy ckan tests, and removed old or
incomplete tests.
2015-04-01 11:36:00 +01:00
amercader a23ec96b2b [#97] Rename function to make it Postgis 2.0 compatible 2015-04-01 11:13:54 +01:00
amercader 8d1a731b0a [#97] Saner way of comparing geometries 2015-04-01 11:13:05 +01:00
Sol Lee c291e30a0e make wms and geojson viewer compatible with CKAN 2.3 2015-04-01 18:02:31 +08:00
amercader f41f20c0da [#97] Move all GeoAlchemy(2) related code to a common module
This allows code in lib, model etc to not worry about GeoAlchemy
versions. This assume the correct library installed, as the check has
been done when starting up.
2015-03-27 15:57:55 +00:00
amercader 39373c9144 [#97] Check GeoAlchemy version on startup
If using CKAN>=2.3 and GeoAlchemy2 is not installed an ImportError
will be raised with details about how to proceed. Added a section
on the Troubleshooting.
2015-03-27 15:57:11 +00:00
jqnatividad 89795c4b3d Display map widget custom attribution
Fixes #87
2014-11-03 16:51:46 -05:00
amercader 23a83fac29 Merge branch '57_default_tags_extras' of https://github.com/geosolutions-it/ckanext-spatial into geosolutions-it-57_default_tags_extras 2014-06-27 16:10:13 +01:00
Tom Mortimer-Jones d6896d68c2 Update csw.py
Fix copy paste error
2014-06-20 12:36:04 +01:00
amercader 10bb6a1edb [#69] Fix autodoc in RTD, minor doc tweaks 2014-06-11 12:22:13 +01:00
amercader b428c33ff6 [#69] Add config option to keep old behaviour (not reindex) 2014-06-10 18:08:38 +01:00
amercader dbf139e732 [#63] Added extension point for defining custom validators 2014-05-13 18:07:14 +01:00
amercader 6c55aad223 [#63] Add extra stuff to the get_package_dict extension point
Moved the call to get_site_user higher on base.py so it's available to
extensions. Also added the parsed XML etree so it does not need to be
parsed from the string again.
2014-05-13 18:03:12 +01:00
amercader 5461bebb62 Merge branch 'master' into 63-extend-spatial-harvesters 2014-05-13 17:54:44 +01:00
amercader fa4161af87 [#70] Fix deletion of harvested CSW records
The object id was pushed to the list returned by gather_stage before
being saved on the db, so None was added, causing an exception in the
Redis queue
2014-05-13 17:53:18 +01:00
amercader b25a01029a Merge branch 'master' into 63-extend-spatial-harvesters
Conflicts:
	ckanext/spatial/harvesters/base.py
2014-05-13 15:33:04 +01:00
amercader ab241d2530 Pass defer_commit in context on get_site_user calls
See ckan/ckan#1714. Until that is fixed properly, the `defer_commit`
flag avoids some `DetachedInstanceErrors` happening during the
harvesting.
2014-05-13 15:30:34 +01:00
amercader 57b7e51e5a Merge branch '69-reindex-dataset-no-object-change' into 63-extend-spatial-harvesters 2014-04-30 18:02:15 +01:00
amercader e979d08e77 [#69] Reindex dataset if harvest object did not change
We replace the old harvest object with the new one, and if we don't
reindex the reference to the old harvest object will remain in the
dataset dict
2014-04-30 18:01:42 +01:00
amercader e9be08d8f1 Merge branch 'master' into 63-extend-spatial-harvesters 2014-04-28 18:41:38 +01:00
amercader e19ed8100f [#62] Fix xpath for Alternate Title 2014-04-28 17:25:17 +01:00
amercader 211f3e4a95 [#63] Tweak docs 2014-03-19 13:02:03 +00:00
amercader f960733727 Merge branch 'master' into 63-extend-spatial-harvesters 2014-03-19 12:48:49 +00:00
amercader 0513e360e9 [#63] Add previous_object check
In rare cases (eg if there was a previous error of two objects sharing
a guid) we can have a "changed" state and no previous_object
2014-03-19 12:46:01 +00:00
amercader 119c0fd40c [#63] Add user to delete context to avoid exception 2014-03-19 12:45:49 +00:00
amercader bbe4125257 [#65] Add individualName to ResponsibleParty 2014-03-19 11:59:08 +00:00
amercader 3b1b857efd Merge branch 'master' into 63-extend-spatial-harvesters 2014-03-14 14:12:30 +00:00
amercader 26763218bc [#62] Fix spatial-resolution parsing 2014-03-14 14:12:05 +00:00
amercader 13f03878e2 Merge branch 'master' into 63-extend-spatial-harvesters 2014-03-14 12:36:36 +00:00
amercader 1fa72f4c16 [#64] Fix validation exception when resource-type was missing 2014-03-14 12:36:01 +00:00
amercader badd723259 [#63] Add new ISpatialHarvest interface
Two extension points: ``get_package_dict`` and ``transform_to_iso``,
with the same expected behaviour as the old hooks meant to be overriden.

For ``get_package_dict`` we now pass, apart from the generated
package_dict, the parsed iso_values and the harvest object.

Updated docs and added autodocs.
2014-03-14 11:30:26 +00:00
amercader 60828520ba Merge branch '55_harvest_cql_master' of https://github.com/geosolutions-it/ckanext-spatial into geosolutions-it-55_harvest_cql_master 2014-03-14 10:29:47 +00:00
amercader 4551efdddb [#62] Relax multiplicities 2014-03-14 10:19:57 +00:00
amercader 8b4d23a315 [#62] Add new fields to the ISO parser
* Metadata standard name / version
* Unique resource identifier (reenabled and fixed)
* Presentation form
* Purpose
* Maintenance note
* Access constraints
* Distributor
* Usage
* Aggregation info
2014-03-11 17:05:08 +00:00
amercader ee77800111 [#62] Clean up keywords parsing
Keywords were extracted via 3 different elements
`keyword-inspire-theme`, `keyword-controlled-other` and
`keyword-free-text`. The latter didn't actually do anything and the
second duplicated xpaths from the first and added a non-standard one.

A new `keywords` key has been added which contains all keyword objects,
including type. This is not used to modify the `tags` key right now.
2014-03-11 14:09:55 +00:00
amercader 239b8313e9 [#62] Add links to organization parties 2014-03-11 14:07:01 +00:00
amercader 93e07fe987 [#61] Add media types for GeoJSON and GML 2014-03-07 16:26:50 +00:00
etj 9116a6fd1f [#55] Allow CSW harvesters to define CQL filters (2nd try) 2014-03-02 23:12:54 +01:00
etj f544f41806 [#57] Add default_tags and default_extras handling 2014-02-25 14:55:20 +01:00
amercader e3c35e9927 Merge branch 'tomkralidis-master' 2014-01-24 18:15:52 +00:00
amercader 15461acff7 [#52] Add class for link removed on 2.2 2014-01-09 17:49:18 +00:00
amercader c352fbdadb [#52] Show dataset map correctly on all 2.x versions 2014-01-09 17:46:45 +00:00
Tom Kralidis 3f91a4411b Update csw.py 2013-12-26 18:18:15 -05:00
Tom Kralidis af695826e8 add functionality to set CSW keywords from CKAN tag counts 2013-11-18 16:32:23 -05:00
amercader dac18b69b1 Revert accidental deletion from ffc5121c 2013-10-23 13:02:22 +01:00
amercader ffc5121c6d Fix csw client to properly use getrecords2 2013-10-16 13:58:14 +01:00
Tom Kralidis 1fdf0cde4c s/owslib.csw.getrecords/owslib.csw.getrecords2/g 2013-10-14 09:18:42 -04:00
amercader 45ebb5a13e [#37] Update attributions 2013-10-10 17:56:23 +01:00
amercader 2f28c8c5c7 [#37] Add support for MapBox tiles and others
The common map function has been extended to allow configuring different
options for the base layer. There are default presets for MapQuest and
MapBox, but any XYZ layer can be added configuration options. The
options are passed to the templates via a helper function and to the js
modules via data-module attributes. Full docs included.
2013-10-06 23:22:04 +01:00
amercader 8a48ceb0e7 Merge branch 'master' of git://github.com/FuhuXia/ckanext-spatial into FuhuXia-master 2013-10-04 15:45:44 +01:00
fxia a19010d8e5 for progress, use gemini_values not iso_values 2013-09-10 10:57:15 -04:00
amercader f7f1a72df8 Remove progress field from tests 2013-09-10 11:08:06 +01:00
amercader 8ffc392f40 Merge branch 'master' of git://github.com/FuhuXia/ckanext-spatial into FuhuXia-master 2013-09-10 11:00:15 +01:00
fxia c1fe37647f change progress multiplicity to * 2013-09-09 14:47:14 -04:00
amercader 028d26e6e0 [#37] Common function for initializing maps
This avoids duplication, mkaes all map consistent and makes easier
future support for other base maps.
2013-08-30 17:10:11 +01:00
amercader b371aee381 [#36] Fix spatial query widget for CKAN 2.1 2013-08-30 16:26:02 +01:00
fxia a9414e755d add progress into the iso values 2013-08-29 00:05:24 -04:00
amercader 3f784de737 Merge branch 'master' into 24-pycsw-integration-command
Conflicts:
	README.rst
2013-08-20 14:35:53 +01:00
amercader c6e29ee25f [#35] Ignore auth when using site_user 2013-08-14 12:23:00 +01:00
amercader 5f56877eba Merge branch '29-new-preview-interface' 2013-08-08 17:52:05 +01:00
amercader 063a1b8530 [#26] Fix broken markers on debug mode 2013-08-08 17:33:56 +01:00
amercader 19ea9b1753 [#32] Do not set height to avoid scrollbars if container is less than 600px 2013-08-08 17:31:21 +01:00
amercader 28042d2b6f [#28] Fix function name 2013-07-29 17:25:04 +01:00
amercader 14048a153c [#31] Check if sort parameter exists 2013-07-29 17:23:21 +01:00
Dominik Moritz f0aa5076a0 [#29] Fix missing variable 2013-07-19 23:04:49 +02:00
Dominik Moritz f4ea4da3e0 [#29] Use new precedence feature for previews. 2013-07-19 12:35:35 +02:00
amercader 44e4ae380a [#24] Document CKAN-pycsw integration
Ad overview of how it works, how to set it up and deploy it in a
production server.
2013-06-21 13:21:30 +01:00
amercader 67b0941b3b [#24] Add clear command to empty the pycsw table 2013-06-20 16:45:37 +01:00
amercader 7b65813cba [#24] Add command for pycsw integration
Based on the work by @kindly on https://github.com/kindly/ckan_pycsw_import,
adds a command to the extension that allows to export easily the spatial
datasets to pycsw. This will only work for harvested ISO datasets for the time
being.

The command can be run both via paster (in the context of a CKAN install) or
independently (eg a separate server), as it uses the API to get the datasets.

TODO docs
2013-06-20 16:44:56 +01:00
amercader d8ed7c03cd Merge branch '21-geojson-preview' 2013-06-14 17:56:26 +01:00
amercader e30b662a36 [#21] Make popup table match the rest of CKAN tables 2013-06-14 17:53:38 +01:00
amercader f8dd4c02bd [#21] Fix i18n calls
All the 'self' on the 'showError' function should be 'this', otherwise you
will refer to the window. Also 'this.i18n' assumes an 'i18n' key in the
module options. Looks like you are trying to pass some variables to the
string template, note the sprintf-like syntax.
2013-06-14 17:39:58 +01:00
amercader a4b8ddb7db Merge branch 'master' of github.com:okfn/ckanext-spatial 2013-06-14 16:01:49 +01:00
amercader 9eec88b320 [#15] Check CKAN version when using the Solr Search backend 2013-06-14 15:59:45 +01:00
Dominik Moritz fe8c469c12 [#21] show object in popup, fix layout 2013-06-14 13:23:06 +02:00
Dominik Moritz 6eb180c02a [#21] Add properties information to popup, lighter style 2013-06-13 23:24:01 +02:00
Dominik Moritz 64d1846e90 [#21] Add basic GeoJSON preview and fix set up of wms preview 2013-06-13 22:51:01 +02:00
kindly add78d5931 allow csw to fetch differen output schema 2013-05-18 18:28:25 +01:00
amercader 27521221d6 [#20] Flag datasets created via the spatial harvesters via a generic extra 2013-05-15 16:58:12 +01:00
amercader 8e81d1bd69 [#19] Extract thumbnail from ISO documents 2013-05-15 16:41:36 +01:00
amercader b4a7cf2289 [#15] Reenable the Solr backend on master
It can be used against CKAN core master (eventually 2.1)
2013-05-14 14:34:10 +01:00
amercader ce8747198f Merge branch 'release-v2.0'
Conflicts:
	README.rst
	ckanext/spatial/commands/validation.py
	ckanext/spatial/controllers/api.py
	ckanext/spatial/harvesters/gemini.py
	ckanext/spatial/plugin.py
	ckanext/spatial/tests/lib/test_spatial.py
	ckanext/spatial/tests/model/test_harvested_metadata.py
	ckanext/spatial/tests/test_harvest.py
2013-05-14 14:02:28 +01:00
amercader 7919a19f7c Revert "[#15] Add suport for Solr spatial indexing and querying"
Reverting #15 as CKAN 2.0 does not include the necessary changes.

This reverts commit c7a9cc660f.
2013-05-13 18:55:32 +01:00
amercader 83d903f84f Revert "[#15] Ensure that bounding boxes are defined counter-clockwise"
Reverting #15 as CKAN 2.0 does not include the necessary changes.

This reverts commit fede0b0831.
2013-05-13 18:55:27 +01:00
amercader 5e35ac0fcb Revert "[#15] Read search backend config option once only"
Reverting #15 as CKAN 2.0 does not include the necessary changes.

This reverts commit d11693f076.

Conflicts:

	ckanext/spatial/plugin.py
2013-05-13 18:55:12 +01:00
amercader 17796c4ad0 Revert "[#15] Add a new Solr search backend based on boost function"
Reverting #15 as CKAN 2.0 does not include the necessary changes.

This reverts commit bddfdf6fba.

Conflicts:

	ckanext/spatial/plugin.py
2013-05-13 18:53:09 +01:00
amercader 72127fa151 Revert "[#15] Tweak doc to remove paper"
Reverting #15 as CKAN 2.0 does not include the necessary changes.

This reverts commit cf22017b7d.
2013-05-13 18:51:44 +01:00
amercader 6f3c699311 [#7] Remove old genshi spatial query widget 2013-05-13 17:13:19 +01:00
amercader 7e10eb9c16 [#7] Allow to customize the default extent for the spatial query widget 2013-05-13 16:59:49 +01:00
amercader af253d0f89 [#7] Spatial search widget for 2.0
Adds a new map widget to the 2.0 search templates. It is shown initially
in the sidebar but it is expanded when the user needs to draw an area.
It uses Leaflet and Leaflet.draw.
2013-05-13 16:14:48 +01:00
amercader 45f4f4da57 [#16] Ignore time zones when parsing harvest object modified date
Otherwise you get this exception when the date on the document has time
zone information, as dates as stored without it on the database:

TypeError: can't compare offset-naive and offset-aware datetimes
2013-04-25 17:13:03 +01:00
amercader a4c86d26e3 [#15] Better checks following 27d4d5 2013-04-12 16:44:52 +01:00
Ian Ward 27d4d5b2c9 make spatial field optional 2013-04-12 16:39:38 +01:00
amercader 51a2b20501 Merge branch '15-solr-based-spatial-search' into release-v2.0 2013-04-12 10:56:21 +01:00
amercader 822ddbb1b5 [#8] Don't add object id to error so it can be aggregated 2013-04-12 10:54:02 +01:00
amercader cf22017b7d [#15] Tweak doc to remove paper 2013-04-12 10:06:50 +01:00
amercader bddfdf6fba [#15] Add a new Solr search backend based on boost function
This backend does not use the spatial field on Solr, but indexes the
area and coordinates of the dataset's extent. On query time, a boost
function is used to calculate a ratio between the query area and the
dataset extent, which will give a value of 0 for areas that don't
overlap and 1 for a perfect fit. The same function is used both for
sorting and filtering.
2013-04-11 19:29:41 +01:00
amercader d11693f076 [#15] Read search backend config option once only 2013-04-11 17:13:53 +01:00
amercader 970dfd1b68 Merge branch 'release-v2.0' into 15-solr-based-spatial-search 2013-04-11 12:51:08 +01:00
kindly d1594b3790 do not use kwarg for unicode errors 2013-04-09 12:06:38 +01:00
amercader 65e056d519 Merge branch 'release-v2.0' into 15-solr-based-spatial-search 2013-04-03 12:14:20 +01:00
amercader ff25ff2f2b [#8] Abort import stage if get_package_dict returns nothing 2013-04-02 18:40:03 +01:00
amercader 9760546d13 [#8] Add extra xpath for getting resources
According to the ISO spec gmd:MD_DigitalTransferOptions can be accessed
via a gmd:MD_Distributer tag
2013-03-27 17:39:51 +00:00
amercader 0c98e6ec4c [#8] Minor fix in single doc harvester 2013-03-27 17:38:42 +00:00
amercader fede0b0831 [#15] Ensure that bounding boxes are defined counter-clockwise
To return correct results on a spatial query, rectangle geometries must
be defined in counter-clockwise order [1]. This changeset adds a small
sanity check to before_index when we are dealing with a Polygon geometry
that has 5 coordinate pairs. Shapely is used to generate a LinearRing
from the polygon coordinates and check if they are ccw. If not, they are
reordered and a new polygon is generated so the WKT sent to Solr is
properly ordered.

The GeoJSON template used for extents in the base spatial harvester has
been also updated to define the coordinates counter-clockwise.

[1]
http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4#JTS_.2BAC8_WKT_.2BAC8_Polygon_notes
2013-03-23 19:28:31 +00:00
amercader 40967385b0 [#8] Fix typo in WMS format detection 2013-03-21 17:53:38 +00:00
amercader 0e0b5a2cc2 [#8] Fix bug that prevented setting a default resource name 2013-03-21 14:34:42 +00:00
amercader 627c4c58e0 [#8] Fix bug that prevented setting a default resource name 2013-03-21 14:34:23 +00:00
amercader c7a9cc660f [#15] Add suport for Solr spatial indexing and querying
When the 'ckanext.spatial.search_backend' config option is set to
'solr', the extension will index geometries stored in the 'spatial'
extra on the spatial field of the Solr index (named 'spatial_geom').
This is done on the 'before_index' extension point.

Also, when doing a query, if the same config option is in place, the
necessary fq parameter will be set to pass the spatial query to Solr.
2013-03-20 16:54:55 +00:00
amercader a727aa815b Add helper functions useful to format extras coming from the spatial harvesters 2013-03-18 15:59:20 +00:00
amercader a7fc19768b [#8] Remove print commands from WAF harvester 2013-03-14 17:45:34 +00:00
amercader eb201e1759 [#8] Waf harvester: improve exception and return empty list if no records 2013-03-14 17:35:51 +00:00
amercader 0aafffc8dc [#8] Capture exceptions during request in WAF harvester 2013-03-14 14:56:42 +00:00
amercader d2723c3020 [#8] resource-type not always present 2013-03-14 14:30:16 +00:00
amercader a76a8d2ca7 [#8] Don't use object id so messages can be grouped 2013-03-14 12:37:35 +00:00
amercader 4638b3899f Revert "[#8] Don't use object id so messages can be grouped"
This reverts commit 032cc4d961.
2013-03-14 12:36:39 +00:00
amercader 032cc4d961 [#8] Don't use object id so messages can be grouped 2013-03-14 12:34:20 +00:00
amercader da1dc02c7e [#8] Improve fields returned in the package dict
Make them less uklp specific and more parse friendly. Helper functions
should be used in the UI to format them nicely.
2013-03-08 18:57:30 +00:00
amercader 724ef6ed7c [#8] Fix gemini harvester after change in spatial field 2013-03-08 18:56:03 +00:00
amercader e7f70c4f85 [#8] Fix KeyError in point template 2013-03-05 18:38:55 +00:00
amercader 7c5071bfc2 [#8] Sanitize bbox before creating spatial extra
Some common problems:
* Whitespace, tabs, line feeds and plus signs: should be handled by
  float()
* Text: log error and skip creation of spatial extra
* Same set of 2 coords for extent: create point instead of polygon

Note that the bbox values are stored as they are in the bbox-xx-yy
extras
2013-03-05 18:31:49 +00:00
amercader d158a6c684 [#8] Change 'Resource locator' string for unnamed resources
'Resource locator' was confusing, has been replaced by 'Unnamed
resource' and made translatable. Also don't set description if not
present, set name.
2013-03-04 17:55:34 +00:00
amercader d43cbb8800 [#8] Improve resource format detection
The 'guess_resource_format' function looks for common patterns in popular
geospatial services and file extensions. It just looks at the provided URL,
it does not attempt to perform any remote check. By default, it will use the
mimetypes module if no match was found before to try to guess the format.

On the previous version, all resources in documents of type 'service' were
queried to see if they were actually WMS. This is no longer the case,
but services flagged as 'wms' can be verified if the following setting
is set to True: ckanext.spatial.harvest.validate_wms
2013-03-04 17:44:18 +00:00
amercader da5b37bc45 [#8] Fix typo in GeoJSON template 2013-03-01 17:42:51 +00:00
amercader d5461477aa [#8] Do not add XML declaration when storing the content
Rather than store the XML declaration in the DB, we add it if not present
when outputing the contents (Also in ckanext-harvest's show_object)
2013-03-01 17:33:53 +00:00
amercader d7d49c37e5 [#8] Check if tags exist on get_package_dict 2013-02-25 16:03:36 +00:00
amercader 5aeaa209ed Fix bug in spatial query sorting 2013-02-20 12:09:50 +00:00
amercader cc60327d0b [#10] Improve harvested metadata API
Some improvements on the endpoints that return the contents of the
harvest objects:

* Nicer URLs with redirects to the old ones
* Returning the raw harvest object content is available on the main
 harvest extension, so just redirect there
* Support for showing the original document of a harvest object, if
 present
* Suport for defining a custom XSLT for the HTML view, via

ckanext.spatial.harvest.xslt_html_content
ckanext.spatial.harvest.xslt_html_content_original
2013-02-19 18:38:15 +00:00
amercader 8647f90cb6 [#8] Clean up base harvester, docstrings, some pep8 2013-02-18 18:17:36 +00:00
amercader 6783d58006 [#8] Add support for import command to new harvester
The import CLI reruns the import stage for the last current objects, so
when running it, the previous objects don't need to be changed. Any
date check is overridden to force the update of the package.
2013-02-18 17:19:14 +00:00
amercader 1ffe8fc902 [#8] Use plugins toolkit whenever possible 2013-02-15 15:03:03 +00:00
amercader 32a93b8ec8 [#8] Add get_extra to SpatialHarvester class 2013-02-15 14:56:00 +00:00
amercader 4964ce9ec3 [#8] Provide a config option to continue after validation errors
This can be set instance wide on the ini file with

ckanext.spatial.harvest.continue_on_validation_errors

or per source, adding continue_on_validation_errors=true to the source
config.
2013-02-15 14:47:25 +00:00
amercader 596bdbf5d0 [#8] Minor tweak to source_config 2013-02-15 12:50:09 +00:00
amercader 2ff5a11911 [#8] Use site user for harvesting actions
You don't need to create a 'harvest' sysadmin user any more.
By default this will be the internal site admin user. This is the
recommended setting, but if necessary it can be overridden by
the `ckanext.spatial.harvest.user_name` config option, eg to
support the old hardcoded 'harvest' user
2013-02-15 12:28:58 +00:00
amercader baf7b5da67 [#8] Rename harvested document model to ISODocument
GeminiDocument has been kept for backwards incompatibility.
2013-02-13 19:16:36 +00:00
amercader dadd174293 [#8] Make sure only list and dict extras are dumped as JSON 2013-02-13 18:48:53 +00:00
amercader 00a0b5946b [#8] Add transform_to_iso method
This can be overridden by custom harvesters willing to support non ISO
formats (like FGDC). It is called whrn the original_document and
original_format harvest object extras are present. Custom harvesters are
responsible for transforming the original document to ISO.
2013-02-13 18:33:58 +00:00
amercader 641ffad589 Remove silly debug message 2013-02-13 17:49:15 +00:00
amercader 305951aeb5 [#8] Make get_package_dict public and document
So it is more obvious that it can be overridden by custom harvesters.
2013-02-13 17:06:01 +00:00
amercader 1d8a4c17c4 [#8] Update harvesters for CsW, WAF and Doc sources
These are the new versions of the spatial harvesters with significant
improvement over previous ones.
2013-02-12 18:29:30 +00:00
amercader f153b0f4ba [#8] Minor fixes in base harvester 2013-02-12 18:26:03 +00:00
amercader c7d872af7e [#8] Rename source config object and method to avoid confusion 2013-02-12 18:07:05 +00:00
amercader 24270cb4cb [#8] Move Gemini harvesters and updated base harvester to own files
Prior to the merging of the new spatial harvesters, the existing ones
based on Gemini and UKLP have been moved to their own namespace
(ckanext.spatial.harvesters.gemini). The plugin points have been updated
so users currently using these harvesters will still be able to use them
as normal.

The base harvester (SpatialHarvester) has been updated with new methods,
most significally '_get_package_dict' and 'import_stage'. Note that
SpatialHarvester now extends HarvesterBase on ckanext-harvest, which had
some of its methods updated.

TODO: still some geo.data.gov specific bits!
2013-02-12 17:40:41 +00:00
amercader b7f486ce04 [#8] Adapt model parsing code to make it ISO 19115 friendly
Changes in multiplicity to support the ISO 19115 spec rather than just
the Gemini 2 one. Thanks to @dread for his help on this.

Summary of the changes:

* dataset-reference-date: Set to 1..*
Note that there was a bug with mutliple values allowed per date.
Returned object should now be like:
 "dataset-reference-date": [{"type": "creation", "value": "2004-02"},
{"type": "revision", "value": "2006-07-03"}]

* metadata-languge: Set to 0..1

* resource-type: Set to *. That means that a list is now returned

* bbox: Set to *. Note that bboxes are now returned as objects such as:
[{"north":xxx, "south":xxx,, "east":xxx, "west":xxx}, {"north":xxx,
  "south":xxx,, "east":xxx, "west":xxx}]

The existing Gemini based harvesters and validators have been adapted,
all tests pass.
2013-02-11 17:35:06 +00:00
amercader 672e168bfa [#8] Adapt harvest tests to CKAN 2.0
Add new mandatory fields when creating sources, status dict has new
keys, CKAN lower cases formats, take into account harvest source
datasets.

Added a local getcapabilities response to avoid remote 404s.

Note that the TestValidation tests need to be fixed, as 27c4ee81e
removed the validation from the gather stage.
2013-02-11 16:57:38 +00:00
David Read b0312ed3a5 Conflicts:
ckanext/spatial/harvesters.py
	ckanext/spatial/tests/test_harvest.py
2013-02-08 17:47:32 +00:00
David Read fe2ebe016f Conflicts:
ckanext/spatial/harvesters.py
2013-02-08 17:40:19 +00:00
David Read f7d23dd576 #154 Gemini schematron 1.3 has been accepted, so loses the "a" suffix. 2013-02-08 17:38:41 +00:00
David Read aa080e9f75 #noticket No functionality has changed! Factored out responsible_organisation stuff into a separate method to add tests to show what it does. 2013-02-08 17:38:05 +00:00
David Read 9daff6a5b2 #noticket Tests added to clarify license URL extraction. 2013-02-08 17:36:34 +00:00
David Read e20080e69d Latest schematron added. FCSC is a good test of it. 2013-02-08 17:35:31 +00:00
David Read 6e23ae55c8 Relaxed "Multiplicity Check" so that it does not raise Exceptions any more - just log errors. This is because they are simply duplicates of the Gemini Schematron. Adria agreed these will be deleted anyway in 2.0. 2013-02-08 17:34:43 +00:00
David Read 779e00cd75 [xs] Improve docstrings and error messages. 2013-02-08 17:33:42 +00:00
David Read 5bcffdf14b More debug logging added to WAF harvester. 2013-02-08 17:33:16 +00:00
David Read 44728f12f7 Add XSL for converting Gemini XML to nice HTML, used in controllers/api.py. 2013-02-08 17:31:34 +00:00
David Read bcdf360b01 Spatial query can now be ordered. Does not play nicely with SOLR options - just uses that to get the facets counts and return each result. Have added performance tests for two alternative queries.
- Added a config option ('ckanext.spatial.use_postgis_sorting') to
activate this as this behaviour will be deprecated in the future
in favour of Solr 4 spatial sorting capabilities.
Also fixed the tests

Conflicts:

	ckanext/spatial/plugin.py
2013-02-08 17:28:37 +00:00
David Read fb4b041b30 Adding Parslow constraints schema previously missed. 2013-02-08 16:41:44 +00:00
David Read 8e0f7c7148 Added lower level tests for bbox search (at the lib level), complementing the API level ones.
Conflicts:

	ckanext/spatial/plugin.py
2013-02-08 16:41:02 +00:00
David Read 46fb0030a5 Comments about on cardinality/multiplicity. 2013-02-08 16:38:14 +00:00
David Read d2c97fe3cc Added new Parslow Constraints Schematron to test. Added command to validate on the command-line. 2013-02-08 16:37:44 +00:00
David Read 9ea3295b46 Get the validation XML to be included in the distribution. 2013-02-08 16:36:36 +00:00
David Read 892b44a3b3 Added useful logging to the validation report. Useful to have the date (i.e. version) in the name of the Eden schema. 2013-02-08 16:35:59 +00:00
David Read c6ac9494a2 Revert "#276 Coupled resources - first iteration."
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.

This reverts commit 91e547a622.
2013-02-04 16:07:19 +00:00
David Read f4e3cfad00 Revert "#276 Coupled resources - second iteration. Just need to update harvester now."
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.

This reverts commit ecd6036efe.
2013-02-04 16:07:03 +00:00
David Read c771a76e3d Revert "#276 Coupled Resource table now gets updated during harvest."
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.

This reverts commit 01536873b9.

Conflicts:

	ckanext/spatial/harvesters.py
2013-02-04 16:03:50 +00:00
David Read 84b75ea759 Revert "#276 Coupled Resource - fix to not show withdrawn packages in list of coupled resources."
Reverting changes for "#276 Coupled Resource" on master as it is INSPIRE-specific. Moving to datagovuk/ckanext-spatial branch dgu.

This reverts commit 3f627d9700.
2013-02-04 16:00:34 +00:00
David Read 27c4ee81e2 #287 Avoid doing extra validation for WAF in the gather stage. 2013-02-02 00:06:33 +00:00
David Read 3f627d9700 #276 Coupled Resource - fix to not show withdrawn packages in list of coupled resources. 2013-02-01 23:10:58 +00:00
David Read 3eee0be135 #noticket Bugfix - diff (when harvest content changed without timestamp change) displayed raw html. 2013-02-01 18:09:12 +00:00
David Read cdabab12cd #154 Gemini schematron 1.3 has been accepted, so loses the "a" suffix. 2013-02-01 14:00:51 +00:00
David Read 73af616b81 #noticket No functionality has changed! Factored out responsible_organisation stuff into a separate method to add tests to show what it does. 2013-02-01 12:04:50 +00:00
David Read 01536873b9 #276 Coupled Resource table now gets updated during harvest. 2013-01-31 18:05:48 +00:00
David Read 2cef9d6dae #noticket Tests added to clarify license URL extraction. 2013-01-31 14:32:19 +00:00
David Read ecd6036efe #276 Coupled resources - second iteration. Just need to update harvester now. 2013-01-28 23:56:12 +00:00
David Read 91e547a622 #276 Coupled resources - first iteration. 2013-01-28 21:53:15 +00:00
David Read 775a57d1f5 Latest schematron added. FCSC is a good test of it. 2013-01-24 11:30:23 +00:00
David Read 70c3eccdf5 Relaxed "Multiplicity Check" so that it does not raise Exceptions any more - just log errors. This is because they are simply duplicates of the Gemini Schematron. Adria agreed these will be deleted anyway in 2.0. 2013-01-21 17:22:24 +00:00
amercader 461607f06f Merge branch 'release-v2.0' into 2.0-validation-changes 2013-01-21 16:30:59 +00:00
amercader fd1071959e [#6] Move to Leaflet for dataset map widget
For this particular use case Leaflet offered the best option. Also
solves the issue when showing extent covering the whole world.
2013-01-18 15:12:09 +00:00
amercader 66b72163d5 Remove stuff from html.py
The rest will eventually go when we migrate the spatial search snippet.
2013-01-18 15:11:06 +00:00
amercader 70f7f6144b Factor out DGU code from the dataset map
It now lives in the DGU extension:

https://github.com/datagovuk/ckanext-dgu/blob/master/ckanext/dgu/theme/public/scripts/dgu-dataset-map.js
2013-01-18 13:32:30 +00:00
amercader 3da5807eb4 [#6] Update dataset map to be a pure snippet
No need to load an extension.
2013-01-18 13:07:26 +00:00
David Read 1622223a38 [xs] Improve docstrings and error messages. 2013-01-16 22:22:39 +00:00
David Read 7841691fc3 More debug logging added to WAF harvester. 2013-01-16 13:10:32 +00:00
David Read 348c1c4dc1 Add XSL for converting Gemini XML to nice HTML, used in controllers/api.py. 2013-01-16 13:08:28 +00:00
amercader 21a85a6b3f [#4] Register resources only once for all plugins
spatial_metadata will load the resources (public, templates and
resources) for all plugins to use, as it needs to be loaded anyway.
2013-01-15 20:00:46 +00:00
amercader 7abfb4eb61 Use plugins toolkit whenver possible on plugin.py 2013-01-15 19:57:31 +00:00
amercader da7bb48eb5 Merge branch 'release-v2.0' into 2.0-validation-changes 2013-01-14 14:03:04 +00:00
amercader 711391e971 [#4] Rewrtie WMS preview plugin for ResourcePreview interface
Much simplified plugin for previewing WMS. It requires the
resource_proxy plugin to work.

Also clean up public and template dir to mimic core layout.
2013-01-14 13:59:15 +00:00
kindly 4e47141717 add extra resource locator 2012-12-24 10:43:44 +00:00
amercader c927d8b6ab Add method for adding custom validators
This probably needs to be done properly, adding them once on startup
somehow.
2012-12-20 18:26:40 +00:00
David Read 6fcbf878c2 Spatial query can now be ordered. Does not play nicely with SOLR options - just uses that to get the facets counts and return each result. Have added performance tests for two alternative queries. 2012-12-06 18:17:45 +00:00
David Read 36905eed17 Adding Parslow constraints schema previously missed. 2012-12-05 15:02:27 +00:00
David Read 68f22147bc Added lower level tests for bbox search (at the lib level), complementing the API level ones. 2012-12-05 15:01:16 +00:00
David Read df884bdd3d Comments about on cardinality/multiplicity. 2012-12-05 12:05:45 +00:00
David Read 20e1a59227 Added new Parslow Constraints Schematron to test. Added command to validate on the command-line. 2012-12-05 11:42:57 +00:00
amercader 615a58ce93 Reduce log level for harvest object errors 2012-11-21 16:22:55 +00:00
amercader 9ea721e256 Encode remote documents from CSW servers as unicode 2012-11-20 15:42:07 +00:00
amercader 6cf7f79942 Save line if present when storing object errors 2012-11-20 11:47:04 +00:00
amercader 7113466760 Update harvesters to new validator outputs 2012-11-19 18:12:40 +00:00
amercader e12e38cab0 Improvements on the validation code
To make easier to filter and display errors on the UI, the validators
have been modified to return the message and line number separately. The
return format for validators is now:

(is_valid, [(error_message_string, error_line_number)])

Also the XSD based validators were returning only the last error found on
the document, instead of iterating the whole error log. Harvesters should
create a harvest object error for each of this validation errors.

Tests have been adapted to these changes.
2012-11-19 17:15:16 +00:00
amercader 0379852fe0 Merge branch '2641-spatial-widgets-new-theme' into release-v2.0 2012-11-12 16:44:05 +00:00
amercader b82dd4a9c0 Merge branch 'csw-harvester-enhancements' into release-v2.0 2012-11-12 16:43:52 +00:00
David Read ccbac9c3c5 Get the validation XML to be included in the distribution. 2012-11-07 16:23:27 +00:00
amercader a84268abf6 Don't trust the number of records returned by the remote server
Sometimes, even when requesting 10 records, the CSW server returns less
of them (see eg http://goo.gl/b7Rdj, only 9 records returned). The
current check made the process stop on this case, missing other
identifiers.
2012-11-02 11:12:46 +00:00
amercader 0444c14da2 Better check for config object 2012-10-30 16:40:32 +00:00
amercader 99dc2a7c55 Allow to define the validation profiles via source config
The profiles used are decided as follows:

1. 'validator_profiles' property of the harvest source config
object
2. 'ckan.spatial.validator.profiles' configuration option in
the ini file
3. Default value as defined in DEFAULT_VALIDATOR_PROFILES
2012-10-30 14:18:01 +00:00
David Read a45ff0c75f Merge branch 'master' of github.com:okfn/ckanext-spatial 2012-10-30 09:34:48 +00:00
David Read ee543df158 Added useful logging to the validation report. Useful to have the date (i.e. version) in the name of the Eden schema. 2012-10-30 09:34:40 +00:00
amercader ac7947549e Remove unused imports 2012-10-29 16:35:52 +00:00
amercader 9e9048c272 Add validator for FGDC XSD schema 2012-10-29 14:34:29 +00:00
amercader 9488ecd5a9 Add validator for ISO 19139 NGDC XSD schema 2012-10-29 14:28:58 +00:00
amercader c1d2a479f2 Add traceback to exception when getting CSW identifiers 2012-10-23 18:57:11 +01:00
amercader 92b781d0f1 Minor model tweaks to support parsing generic ISO documents 2012-10-23 13:03:53 +01:00
amercader aeb7d27bab Fix failing tests
The WMS one is just skipped, until we have a more clear way of how the
previews will work.
2012-10-22 19:44:33 +01:00
amercader 7f58374ac7 Enable command line interface for validation, useful for debug 2012-10-22 19:39:07 +01:00
amercader d95602eaff Fix wrong resource paths on validation 2012-10-22 19:37:54 +01:00
amercader 019cb3b45f Fix wrong imports and docs 2012-10-22 19:36:03 +01:00
David Read d90114cf07 Added ability to produce report into validation errors, for when changing validation. Added report infrastructure. 2012-10-19 18:20:32 +01:00
David Read 0e8a62fe1e Reorganise XML test files into more sensible directory names. Add lower level validation tests. 2012-10-19 14:23:34 +01:00
David Read 2d6f497720 Missed off some files from the previous commit., 2012-10-19 12:14:09 +01:00
David Read 20e8f12615 Merged in ckanext-inspire.
Tests are passing, apart from a couple which didn't work before:
* test_functional.py -> functional/test_package.py (3 failures in 4)
* functional/test_dataset_map.py (1 fail in 1)
There may be some code errors still untested.
Renamed Validator -> Validators to make more sense.
2012-10-19 11:19:01 +01:00
David Read 58fa06051d Added EDEN ISO10139 schema that was missed off before. 2012-10-17 17:08:20 +01:00
David Read 8181b3d3bf Merged in ckanext-csw @44d5a04656dff084e6bca57dda7b63deec69778c. Not tested yet. 2012-10-17 16:59:02 +01:00