Commit Graph

603 Commits

Author SHA1 Message Date
amercader 7459358fa1 Support for single import commands
We are now able to run `paster harvester import` for a single harvest
object or for a single dataset, providing ids or name.
2014-05-15 16:30:30 +01:00
amercader d6998d8d04 Merge branch '96-harvest-object-encoding-errors' 2014-05-15 15:54:31 +01:00
amercader 2c6aaf5bb1 Merge branch 'master' into 96-harvest-object-encoding-errors 2014-05-15 15:52:13 +01:00
amercader 43f1d08255 [#97] Persitent endpoint for datasets harvest objects
Contrary to `/harvest/object/xxx`, this endpoint is passed the dataset
id, thus it not depends on a particular object but the most recent one.
2014-04-30 17:45:07 +01:00
amercader 1b458b1772 [#96] Handle encoding errors on harvest object endpoint
When parsing the harvest object content to see if it is an XML file,
etree.fromstring would fail id there are incorrect unicode errors.
2014-04-28 12:48:09 +01:00
amercader 040984d4ec Merge branch 'rikdc-89-extras-check' 2014-02-28 18:23:43 +00:00
Richard Claydon e3492b57e7 Update plugin.py
Updating plugin.py to check for the existence of the extras key in the data_dict.
2014-02-27 16:05:39 +00:00
amercader d3cf5e58d1 [#86] Fix duplicate extras 2014-02-11 18:16:49 +00:00
amercader fbde0b8dc1 [#87] Remove remote url_type from resources
Otherwise CKAN thinks they are uploads, datastore resources, etc, which
it can cause problems eg when displaying the URL of the resource. We
are just linking to the remote resource URL.
2014-02-11 17:27:19 +00:00
amercader 5739e541d7 [#80] Support for Python 2.6 when handling xml exceptions 2014-02-10 18:44:46 +00:00
amercader 2a07a144fc [#84] Fix auth audit exception when creating datasets
This was caused by a combination of the auth audit leaking and the
harvester reusing the context for the package_show and package_create
actions. If the package is not found, package_show does not call
check_access, and the auth audit does not pass. This is stored in the
context (`__auth_audit`) and is raised next time that we call
get_action (when we call package_create with the same context)

It could potentially be fixed on master, but it is probably quite rare.
2014-02-10 18:22:48 +00:00
amercader 5b677b6099 [#83] Fix key error when using default_groups 2014-02-10 13:16:58 +00:00
amercader 3693028009 Merge branch 'ogdch-79-optional-tag-munging' 2014-02-10 13:13:48 +00:00
Rachel Knowler bf11e4d330 Moved clean_tags check into _create_or_update_package method. 2014-02-10 09:29:01 +01:00
Rachel Knowler 2ba9908653 Config option to munge tags changed to be consistent with other config options in this extension, and noted in README. 2014-01-29 10:55:51 +01:00
Rachel Knowler 5e1aef1d08 Removed extra newline. 2014-01-29 10:06:32 +01:00
Rachel Knowler 7d71b0a00b Wrap tag munging code in config option, defaulting to False. 2014-01-29 10:02:16 +01:00
amercader 2b803a3f66 [#77] Use auth_allow_anonymous_access decorator
Starting from 2.2 you need to explicitly flag auth functions that
allow anonymous access with the p.toolkit.auth_allow_anonymous_access
decorator. A local version of the decorator is used to ensure we only
use it on CKAN>=2.2
2014-01-20 13:47:37 +00:00
amercader 4cc56f51ab [#76] Use harvest_source_show on reindex command 2014-01-14 17:04:34 +00:00
amercader 95d0ef0f01 [#76] Add extra fields to the source schema
Add 'private' and its core validators, and 'metadata_modified' and
'metadata_created'.

Also ignore '__extras'
2014-01-14 17:01:25 +00:00
amercader 467fb7bb8f Fix resource updating for harvested datasets
Starting from 2.2, resource_update calls package_show before updating
the resource via a package_update call. The dict passed had the harvest
extras (eg harvest_object_id) added which made the update call fails due
to duplicated extra keys. To fix it we now remove any harvest extras
on after_show if there is a 'for_edit' property on the context.
2014-01-13 10:30:52 +00:00
amercader 278a8e1ada Merge branch 'master' of github.com:okfn/ckanext-harvest 2014-01-10 13:49:38 +00:00
amercader 1e94a11255 [#70] Fix Add harvest source button not showing
Due to changes in the templates starting on 2.1 the add source button
was not showing. The whole search template has been simplified,
separating in a separate file the 2.0 only code.

Tested in 2.0, 2.1 and 2.2
2014-01-10 13:48:02 +00:00
Adrià Mercader 5ac2594fdf Merge pull request #74 from kata-csc/master
Put harvest_source_id in quotes to prevent Solr errors
2013-11-28 05:38:23 -08:00
Mikko Koho 51e842ee6e Add quotes to harvest_source_id in Solr query when clearing harvest sources 2013-11-22 11:01:26 +02:00
Mikko Koho c338452872 Put harvest_source_id in quotes to prevent Solr errors 2013-11-21 14:14:56 +02:00
amercader 928ea061aa Improve organizations dropdown on source form 2013-10-24 12:33:44 +01:00
Adrià Mercader 9be495d403 Merge pull request #72 from metaodi/61-harvest-source-error
[#61] Truly ignore harvest sources
2013-10-23 10:19:08 -07:00
Stefan Oderbolz c52085006a [#61] Truly ignore harvest sources
The currently implementation returns False when a harvest source is being harvested. This leads to an error on the harvesting job, which in turn tends to confuse users that have no idea of this special implementation. This fix ensures that harvest sources are still ignored, but silently.
2013-10-23 07:40:55 +02:00
amercader c18d9dc3af [#71] CKAN harvester: Add datasets to source organization
If the harvest source belongs to an organization, new datasets should be added
to it. This is already the case in the spatial harvesters.

The remote orgs logic has been kept, with the difference that if for
some reason the remote org can not be assigned, the local one is used.

If the source does not have an organization, none is added.
2013-10-22 16:24:43 +01:00
amercader 0a845b6308 Move factory boy to dev requirements 2013-10-16 17:29:21 +01:00
amercader 380c14c22c Fix CLI sources list output 2013-10-16 13:03:32 +01:00
amercader 55d2b4e304 Fix purge command 2013-10-16 12:59:23 +01:00
amercader bd62b62764 Merge branch 'metaodi-add-harvesting-of-organizations' 2013-10-15 17:50:04 +01:00
amercader 49999893e7 Merge branch 'add-harvesting-of-organizations' of git://github.com/metaodi/ckanext-harvest into metaodi-add-harvesting-of-organizations 2013-10-15 17:49:28 +01:00
Stefan Oderbolz 8b5d70c6fe Only try to create/match a organization if there is a remote_org 2013-10-11 18:08:32 +02:00
amercader 0f5624822c Use remote name if present when creating datasets on CKAN harvester 2013-10-11 16:50:25 +01:00
amercader 340e9eed63 Merge branch 'add-harvesting-of-organizations' of git://github.com/metaodi/ckanext-harvest into metaodi-add-harvesting-of-organizations 2013-10-11 16:14:18 +01:00
amercader e9dde3f48a Only show the 'Add Harvest Source' button if user is authorized 2013-10-11 11:55:14 +01:00
Stefan Oderbolz dd1acd0c6b Use remote_orgs for organizations 2013-10-07 11:22:19 +02:00
amercader 10b4dec490 Merge branch 'opendatatrentino-fix/rename-ampq-to-amqp' 2013-10-04 17:25:36 +01:00
amercader f89f12203c Merge branch 'fix/rename-ampq-to-amqp' of git://github.com/opendatatrentino/ckanext-harvest into opendatatrentino-fix/rename-ampq-to-amqp 2013-10-04 17:24:53 +01:00
amercader c5f4d6889b Merge branch 'harvest-object-create' 2013-10-04 17:22:58 +01:00
Stefan Oderbolz d50eb6fca8 Harvesting of remote organisations similar to remote groups 2013-10-04 16:37:52 +02:00
joetsoi da2fd45e80 [#65] make harvest_job_exists validator return model object
return the model in the validator instead of checking that it exists in
the validator, returning the id and then fetching it again in the action
function
2013-10-03 15:51:37 +01:00
Samuele Santi 611b9aab6d Fixed typo: ampq -> amqp 2013-09-19 11:43:03 +02:00
joetsoi 9b3199b41b [#65] remove unused code 2013-09-17 17:02:38 +01:00
joetsoi 5da153c6b6 [#65] harvest_object_create action
update to use schema and validators. Also accept more parameters to
data_dict.
2013-09-17 16:49:19 +01:00
John Martin 71aedf3fd4 Makes ede45bd work in both CKAN 2.0 and 2.1 2013-09-05 15:36:00 +01:00
John Martin ede45bd1be Fixes #66 by correcting the block name 2013-09-05 15:31:44 +01:00