Commit Graph

934 Commits

Author SHA1 Message Date
Stefan Oderbolz 191c39ce5c Catch the more general URLError instead of HTTPError
HTTPError is a subclass of URLError, so catch URLError is enough. I
think the HTTP error code is not as important in this situation, so
catching the more generic error seems like the best solution.
2015-01-15 10:57:24 +01:00
Stefan Oderbolz b978c26e70 Use ContentFetchError instead of generic Exception 2015-01-15 00:49:11 +01:00
Stefan Oderbolz 935b9dda01 Munge group name before fetching remote group
The API call /api/2/rest/package/<id> returns the display name of the
group instead of its ID. To properly match the group, munge the name
before calling /api/2/rest/group
2015-01-15 00:44:53 +01:00
Stefan Oderbolz ef35c21e2a Improve exception handling with custom exception
1. Try whenever possible to catch specific exceptions
2. Raise custom exception where appropriate
3. Fix the exception handling in _get_group and _get_organization
2015-01-15 00:44:45 +01:00
amercader a71cc495fb Merge branch 'metaodi-117-url-already-in-use' 2015-01-14 11:16:26 +00:00
Stefan Oderbolz 0fd38e0e54 Use _get_group as a fallback for remote orgs
First try to get a remote org from the remote Action API, if this fails
try to use the old rest api call, which works on older CKAN versions.

Only if both options fail, its currently not possible to get the remote
organization.
2015-01-14 00:10:27 +01:00
Stefan Oderbolz f214577872 Fetch remote organization via action api
Organizations used to be returned by /api/2/rest/group, this is what the
old implementation used to fetch the information to create the remote
organization on the local instance of CKAN.

With this commit the Action API is used to fetch the same information.
2015-01-13 14:46:53 +01:00
Stefan Oderbolz ea9debf714 Fix logic of conditional and make it more pythonic 2014-12-18 16:03:33 +01:00
Stefan Oderbolz 08930d01bf Make sure for new packages get a unique 'name' 2014-12-16 15:02:36 +01:00
clementmouchet 82c7988bf3 Removed ResourceGroup from query when using CKAN 2.3 or above 2014-12-12 13:10:40 +00:00
amercader a3affc9702 Fix validators on harvest_source_show schema
Remove validators on several keys so they don't get stripped during the
show validation.
2014-10-08 12:02:26 +01:00
amercader 623daa72c4 Merge branch 'waldvogel-clear-source-delete-related' 2014-09-29 13:50:59 +01:00
amercader 098b54f1e5 Merge branch 'clear-source-delete-related' of https://github.com/waldvogel/ckanext-harvest into waldvogel-clear-source-delete-related 2014-09-29 13:49:19 +01:00
amercader e60e2eee03 Fix output for harvest_source_create/update
They were using an incorrect schema, so not returning a harvest source
like dict.
2014-09-29 12:43:37 +01:00
amercader c9aff29613 Recommend Redis as the preferred backend 2014-09-19 11:38:38 +01:00
waldvogel c9b4e10506 delete records from related and related_dataset when clearing source 2014-09-12 10:56:37 +02:00
Jari Voutilainen 1e0376cff6 fix typo 2014-09-10 10:33:13 +03:00
Jari Voutilainen f6c1456abe fix job reporting to have job finished timestamp when there was zero datasets to gather 2014-09-10 09:22:55 +03:00
Jari Voutilainen 97f09913cf fix job reporting all datasets deleted when actually nothing changed during last two harvests 2014-09-10 09:22:44 +03:00
amercader f34c1e6d74 Merge branch 'morty-99-all-non-ascii-tags' 2014-08-29 14:40:54 +01:00
amercader 8cf254f112 Merge branch '99-all-non-ascii-tags' of https://github.com/morty/ckanext-harvest into morty-99-all-non-ascii-tags 2014-08-29 14:40:43 +01:00
amercader 786e9011aa Merge branch 'morty-101-modified-package-name' 2014-08-29 14:38:44 +01:00
amercader 546159744e Merge branch '101-modified-package-name' of https://github.com/morty/ckanext-harvest into morty-101-modified-package-name 2014-08-29 14:38:33 +01:00
amercader 039ac7c0ad Always remove harvest extras on after_show if there
Up until now we where relying on `for_edit` being present in the
context, but this is only added on the controllers. It's better to be
safe and remove them always. If needed (at index time) they will be
added afterwards.
2014-08-14 15:31:39 +01:00
Tom Mortimer-Jones 8a2c072d4e [101] Use name from database when reharvesting package 2014-08-12 11:18:48 +01:00
Tom Mortimer-Jones 65cfade420 [99] Remove empty tags produced by munging all non-ascii tags
I thought this way of filtering was easier to read than filter(None, tags)
2014-08-07 17:05:16 +01:00
amercader 13dbb1eea4 Fix variable not defined 2014-07-30 15:49:02 +01:00
amercader 33000dbfbc [#88] Add redis to requirements 2014-06-27 16:59:15 +01:00
amercader 4f7563f066 Merge branch '91-reindex-list-fields-in-config' 2014-06-27 16:56:08 +01:00
amercader 58a873ac7a [#91] Remove config fields from source dict before indexing
We don't need them and will avoid indexing errors
2014-06-27 16:54:39 +01:00
amercader a59ab4b5ff [#91] Consolidate all harvest source reindex code in a single action
Make it available to users with permissions on the harvest source
2014-06-27 16:48:14 +01:00
amercader 7459358fa1 Support for single import commands
We are now able to run `paster harvester import` for a single harvest
object or for a single dataset, providing ids or name.
2014-05-15 16:30:30 +01:00
amercader d6998d8d04 Merge branch '96-harvest-object-encoding-errors' 2014-05-15 15:54:31 +01:00
amercader 2c6aaf5bb1 Merge branch 'master' into 96-harvest-object-encoding-errors 2014-05-15 15:52:13 +01:00
amercader 43f1d08255 [#97] Persitent endpoint for datasets harvest objects
Contrary to `/harvest/object/xxx`, this endpoint is passed the dataset
id, thus it not depends on a particular object but the most recent one.
2014-04-30 17:45:07 +01:00
amercader 1b458b1772 [#96] Handle encoding errors on harvest object endpoint
When parsing the harvest object content to see if it is an XML file,
etree.fromstring would fail id there are incorrect unicode errors.
2014-04-28 12:48:09 +01:00
amercader 040984d4ec Merge branch 'rikdc-89-extras-check' 2014-02-28 18:23:43 +00:00
Richard Claydon e3492b57e7 Update plugin.py
Updating plugin.py to check for the existence of the extras key in the data_dict.
2014-02-27 16:05:39 +00:00
amercader d3cf5e58d1 [#86] Fix duplicate extras 2014-02-11 18:16:49 +00:00
amercader fbde0b8dc1 [#87] Remove remote url_type from resources
Otherwise CKAN thinks they are uploads, datastore resources, etc, which
it can cause problems eg when displaying the URL of the resource. We
are just linking to the remote resource URL.
2014-02-11 17:27:19 +00:00
amercader 5739e541d7 [#80] Support for Python 2.6 when handling xml exceptions 2014-02-10 18:44:46 +00:00
amercader 2a07a144fc [#84] Fix auth audit exception when creating datasets
This was caused by a combination of the auth audit leaking and the
harvester reusing the context for the package_show and package_create
actions. If the package is not found, package_show does not call
check_access, and the auth audit does not pass. This is stored in the
context (`__auth_audit`) and is raised next time that we call
get_action (when we call package_create with the same context)

It could potentially be fixed on master, but it is probably quite rare.
2014-02-10 18:22:48 +00:00
amercader 5b677b6099 [#83] Fix key error when using default_groups 2014-02-10 13:16:58 +00:00
amercader 3693028009 Merge branch 'ogdch-79-optional-tag-munging' 2014-02-10 13:13:48 +00:00
Rachel Knowler bf11e4d330 Moved clean_tags check into _create_or_update_package method. 2014-02-10 09:29:01 +01:00
Rachel Knowler 2ba9908653 Config option to munge tags changed to be consistent with other config options in this extension, and noted in README. 2014-01-29 10:55:51 +01:00
Rachel Knowler 5e1aef1d08 Removed extra newline. 2014-01-29 10:06:32 +01:00
Rachel Knowler 7d71b0a00b Wrap tag munging code in config option, defaulting to False. 2014-01-29 10:02:16 +01:00
amercader 2b803a3f66 [#77] Use auth_allow_anonymous_access decorator
Starting from 2.2 you need to explicitly flag auth functions that
allow anonymous access with the p.toolkit.auth_allow_anonymous_access
decorator. A local version of the decorator is used to ensure we only
use it on CKAN>=2.2
2014-01-20 13:47:37 +00:00
amercader 4cc56f51ab [#76] Use harvest_source_show on reindex command 2014-01-14 17:04:34 +00:00