f70c16bce7
Add framework for testing harvesters. Modernize existing tests.
2015-10-21 16:26:57 +00:00
6360681a8f
[ #105 ] Fix order of deletes, as agreed with @florianm.
2015-10-12 15:57:27 +01:00
a6cdda0a14
set max version to 2.4.99
2015-08-19 08:41:42 +00:00
1905caa961
upgrade harvest_source_clear to not delete from authz models removed in migration 078
2015-08-19 10:25:20 +08:00
ab76830e85
[ #145 ] Throw + catch a custom exception if there are no jobs to run
...
If there are no harvesting jobs to run, there was always an ugly
exception message when using the paster command. This replaces the ugly
output with a proper message and uses a custom exception to allow others
to deal with this error differently.
2015-07-20 18:41:50 +02:00
83dd0b4b68
[ #138 ] Add data attributes to support timezone conversion
2015-07-09 22:35:54 +02:00
4dc2f7367d
[ #139 ] Delete package relationships when clearing a harvest source
2015-06-26 17:20:23 +02:00
88d9ba0397
[ #136 ] Fix broken RabbitMQ queue names
...
The harvester command was still using the old ones.
Use specific ones for testing.
2015-06-11 13:56:22 +01:00
673dfc9882
[ #127 ] Use site user on the CKAN harvester
...
Add missing call
2015-06-11 10:38:33 +01:00
d3a3f09ad1
[ #127 ] Use site user on the CKAN harvester
...
To avoid having to create a 'harvest' sysadmin explicitly. It will still
be used if present, but if not the site user will be used. You can also
define to user to use via a config option.
2015-06-11 10:19:07 +01:00
b17c3269b5
Merge branch 'clear-command' of https://github.com/metaodi/ckanext-harvest into metaodi-clear-command
2015-06-10 15:32:37 +01:00
64ff0f3a3a
Use single quotes to be consistent
2015-06-10 16:22:04 +02:00
2a2d85f60c
Wording changes for clearsource and rmsource
2015-06-10 16:19:23 +02:00
92b93c53fc
add some translation strings
2015-06-10 12:14:20 +01:00
8ebb843052
Add documentation for clearsource command
2015-06-10 11:29:24 +02:00
61bc150ae6
Expose clear harvester source as a paster command
2015-06-10 11:19:10 +02:00
9f8aae3a18
Append site id to queue name
...
This allows multiple CKAN sites to share the same RabbitMQ exchange
(For the Redis backend this is handled via different Redis databases)
2015-06-01 17:54:22 +01:00
3e21ea4f82
Fix tests, set up Travis
...
TODO: sort out the tests properly, avoiding imports from the legacy ones
2015-04-07 13:31:45 +01:00
f72d6da521
Change toolkit import
...
Apparently on package installs this is not well supported
from ckan.plugins.toolkit import check_ckan_version
But this works:
from ckan.plugins import toolkit
toolkit.check_ckan_version(...
2015-03-19 12:48:46 +00:00
7a20e93716
Raise on startup import errors so we don't mask problems
...
Otherwise if there was eg an actual ImportError we jut got
2015-03-19 12:30:08,430 DEBUG [ckanext.harvest.plugin] No auth module
for action "update"
on the log
2015-03-19 12:48:15 +00:00
d6e9b80496
Merge pull request #118 from clementmouchet/114-remove_resource_groups
...
Removed ResourceGroup from query when using CKAN 2.3 or above
2015-02-24 09:56:44 +00:00
ead9e67a33
updated def harvest_source_clear() to delete resource views, resource revisions & resources in CKAN >= 2.3
2015-02-23 17:02:21 +00:00
b3ed6cae5a
Merge pull request #121 from metaodi/120-create-remote-orgs
...
Fetch remote organization via action api
2015-01-15 10:49:09 +00:00
c1bcee9684
Use str() to get the error message
2015-01-15 11:36:15 +01:00
191c39ce5c
Catch the more general URLError instead of HTTPError
...
HTTPError is a subclass of URLError, so catch URLError is enough. I
think the HTTP error code is not as important in this situation, so
catching the more generic error seems like the best solution.
2015-01-15 10:57:24 +01:00
b978c26e70
Use ContentFetchError instead of generic Exception
2015-01-15 00:49:11 +01:00
935b9dda01
Munge group name before fetching remote group
...
The API call /api/2/rest/package/<id> returns the display name of the
group instead of its ID. To properly match the group, munge the name
before calling /api/2/rest/group
2015-01-15 00:44:53 +01:00
ef35c21e2a
Improve exception handling with custom exception
...
1. Try whenever possible to catch specific exceptions
2. Raise custom exception where appropriate
3. Fix the exception handling in _get_group and _get_organization
2015-01-15 00:44:45 +01:00
0fd38e0e54
Use _get_group as a fallback for remote orgs
...
First try to get a remote org from the remote Action API, if this fails
try to use the old rest api call, which works on older CKAN versions.
Only if both options fail, its currently not possible to get the remote
organization.
2015-01-14 00:10:27 +01:00
f214577872
Fetch remote organization via action api
...
Organizations used to be returned by /api/2/rest/group, this is what the
old implementation used to fetch the information to create the remote
organization on the local instance of CKAN.
With this commit the Action API is used to fetch the same information.
2015-01-13 14:46:53 +01:00
ea9debf714
Fix logic of conditional and make it more pythonic
2014-12-18 16:03:33 +01:00
08930d01bf
Make sure for new packages get a unique 'name'
2014-12-16 15:02:36 +01:00
82c7988bf3
Removed ResourceGroup from query when using CKAN 2.3 or above
2014-12-12 13:10:40 +00:00
a3affc9702
Fix validators on harvest_source_show schema
...
Remove validators on several keys so they don't get stripped during the
show validation.
2014-10-08 12:02:26 +01:00
098b54f1e5
Merge branch 'clear-source-delete-related' of https://github.com/waldvogel/ckanext-harvest into waldvogel-clear-source-delete-related
2014-09-29 13:49:19 +01:00
e60e2eee03
Fix output for harvest_source_create/update
...
They were using an incorrect schema, so not returning a harvest source
like dict.
2014-09-29 12:43:37 +01:00
c9b4e10506
delete records from related and related_dataset when clearing source
2014-09-12 10:56:37 +02:00
8cf254f112
Merge branch '99-all-non-ascii-tags' of https://github.com/morty/ckanext-harvest into morty-99-all-non-ascii-tags
2014-08-29 14:40:43 +01:00
546159744e
Merge branch '101-modified-package-name' of https://github.com/morty/ckanext-harvest into morty-101-modified-package-name
2014-08-29 14:38:33 +01:00
039ac7c0ad
Always remove harvest extras on after_show if there
...
Up until now we where relying on `for_edit` being present in the
context, but this is only added on the controllers. It's better to be
safe and remove them always. If needed (at index time) they will be
added afterwards.
2014-08-14 15:31:39 +01:00
8a2c072d4e
[101] Use name from database when reharvesting package
2014-08-12 11:18:48 +01:00
65cfade420
[99] Remove empty tags produced by munging all non-ascii tags
...
I thought this way of filtering was easier to read than filter(None, tags)
2014-08-07 17:05:16 +01:00
13dbb1eea4
Fix variable not defined
2014-07-30 15:49:02 +01:00
58a873ac7a
[ #91 ] Remove config fields from source dict before indexing
...
We don't need them and will avoid indexing errors
2014-06-27 16:54:39 +01:00
a59ab4b5ff
[ #91 ] Consolidate all harvest source reindex code in a single action
...
Make it available to users with permissions on the harvest source
2014-06-27 16:48:14 +01:00
7459358fa1
Support for single import commands
...
We are now able to run `paster harvester import` for a single harvest
object or for a single dataset, providing ids or name.
2014-05-15 16:30:30 +01:00
2c6aaf5bb1
Merge branch 'master' into 96-harvest-object-encoding-errors
2014-05-15 15:52:13 +01:00
43f1d08255
[ #97 ] Persitent endpoint for datasets harvest objects
...
Contrary to `/harvest/object/xxx`, this endpoint is passed the dataset
id, thus it not depends on a particular object but the most recent one.
2014-04-30 17:45:07 +01:00
1b458b1772
[ #96 ] Handle encoding errors on harvest object endpoint
...
When parsing the harvest object content to see if it is an XML file,
etree.fromstring would fail id there are incorrect unicode errors.
2014-04-28 12:48:09 +01:00
e3492b57e7
Update plugin.py
...
Updating plugin.py to check for the existence of the extras key in the data_dict.
2014-02-27 16:05:39 +00:00
David Read
Florian Mayer
Stefan Oderbolz
amercader
joetsoi
clementmouchet
waldvogel
Tom Mortimer-Jones
Richard Claydon