3f42eb6ba0
Merge branch 'revision-id-fix'
...
Conflicts:
ckanext/harvest/tests/harvesters/test_ckanharvester.py
2015-11-20 12:28:17 +00:00
60c4371df4
Add "not modified" to the stats always returned.
2015-11-17 12:45:00 +00:00
ae7c500745
Merge branch 'master' into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-17 12:35:59 +00:00
8e02aedc65
Fix tests
2015-11-17 13:29:25 +01:00
f9b87fff0c
Make sure all possible status are always returned
...
It makes it harded to parse the API response if you always have to check
if 'deleted' is set and if what value it has. I think to simply return 0
for all status value is good practice.
2015-11-17 11:43:11 +01:00
084723abb7
Catch JSONDecodeError when no JSON content
2015-11-16 10:59:18 +01:00
c0a865e64e
Revert ok_ - makes it slightly less readable for little benefit.
2015-11-13 13:45:56 +00:00
42ab55cb6d
No longer need uuid since we clear db between tests now. Added ignore_missing because of occasional failures.
2015-11-13 13:32:55 +00:00
b150b50887
Move the SkipTest to include inherited tests too.
2015-11-13 12:44:27 +00:00
01a4bfd314
Patch test should skip if ckan version is wrong, rather than ignore all exceptions from posts of all tests. Remove FunctionalTestBaseWithoutClearBetweenTests now the tests are modernized.
2015-11-13 12:33:18 +00:00
1288a4d9e7
Reflow text to 79 char width. Warning not necessary with an exception I think.
2015-11-13 12:32:13 +00:00
0ce3748153
Do not use ActionError as this does not yet exist in CKAN 2.2
2015-11-13 12:01:19 +01:00
c33c6e8c13
Raise an error instead of falling back to harvest_source_update
...
As the behaviour of *_patch is clearly different from the *_update we
should raise an error if this action is called on a CKAN instance, where
the action is not available.
2015-11-13 11:41:53 +01:00
644fa49dd4
Make tests independent from cls.sysadmin
...
Generate unique harvest sources
2015-11-11 19:49:49 +01:00
ffca5cc3da
Use new factory style for fixtures
...
- Remove default_source_dict from tests
- Replace setup_class with constructor
- Create mixin for harvest source fixture
- Replace assert with assert_equal where appropriate
- Replace assert with ok_
- Remove dependency to global SOURCE_DICT
- General refactoring of tests
2015-11-11 19:07:59 +01:00
136fcb87d5
Make sure package_patch has a fallback for package_update on CKAN < 2.3
2015-11-11 11:37:23 +01:00
359da2eb69
Add test class for harvest_source_patch
2015-11-11 11:34:01 +01:00
3f09010039
Add harvest_source_patch to API
2015-11-11 05:39:29 +01:00
735ab3e286
[ #157 ] Try to fix test for ckan 2.2 - cf 91afc0e928
2015-11-04 11:37:03 +00:00
a0742d69b0
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
...
Conflicts:
ckanext/harvest/logic/action/update.py
2015-11-04 09:50:00 +00:00
679ed421e9
Merge branch 'master' of github.com:ckan/ckanext-harvest into immediate-harvest
...
Conflicts:
ckanext/harvest/logic/action/update.py
2015-11-04 09:44:05 +00:00
f0207ad38f
Merge branch 'master' of github.com:ckan/ckanext-harvest into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-04 09:36:55 +00:00
cbe9b40e66
Merge branch 'master' of github.com:ckan/ckanext-harvest into revision-id-fix
2015-11-04 09:36:08 +00:00
f9da3654f8
[ #184 ] Fix tests for older ckan versions.
2015-11-03 23:27:52 +00:00
5fba056c59
[ #184 ] Add tests
2015-11-03 23:19:05 +00:00
77e5b89a01
Blank line needed.
2015-11-03 22:23:04 +00:00
8c1f7619cb
Fix code style to be more ckan-like whilst still pep8.
2015-11-03 22:08:46 +00:00
20531c0dda
Merge branch 'master' of github.com:ckan/ckanext-harvest into LondonAppDev-master
...
Conflicts:
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/validators.py
2015-11-03 22:02:49 +00:00
10685badb5
PEP8 based on #174
...
Conflicts:
ckanext/harvest/logic/action/delete.py
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/validators.py
2015-11-03 21:56:06 +00:00
5a5260ff0b
Add test for harvest_source_clear since the PEP8 changes were quite a lot there.
2015-11-03 21:42:39 +00:00
4f71612002
PEP8 based on #174
2015-11-03 20:30:11 +00:00
208d1c4185
Setting back to master.
2015-11-03 17:31:00 +00:00
91afc0e928
[ #178 ] Fix test for ckan 2.2, which does not suffer problem #180 .
2015-11-03 10:01:47 +00:00
f4f124c181
[ #178 ] Fix resouce_revision_id_fkey error. Fixes #178 .
2015-11-03 07:38:01 +00:00
59be6e2c71
Merge branch 'master' into db-error
...
Conflicts:
ckanext/harvest/queue.py
2015-11-03 00:57:14 +00:00
8a7bc9e1d8
Merge remote-tracking branch 'origin/master' into immediate-harvest
...
Conflicts:
README.rst
ckanext/harvest/commands/harvester.py
ckanext/harvest/logic/action/create.py
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/auth/update.py
2015-11-03 00:40:25 +00:00
c7fac36c1c
[ #107 ] "unchanged" response tested and related fixes
...
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
2015-11-03 00:22:53 +00:00
e59760fefe
Merge branch 'job-reporting-fixes' of https://github.com/yhteentoimivuuspalvelut/ckanext-harvest into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-02 21:25:32 +00:00
24415844e0
[ #158 ] Fix revision_id problem in second harvest.
2015-11-02 18:13:29 +00:00
d495e269e7
[ #158 ] Fix tests
2015-11-02 17:29:45 +00:00
14f372aec6
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
...
Conflicts:
README.rst
2015-11-02 17:01:22 +00:00
7ffd6748f3
Corrected docstring params field, duplicate if statement and deleting keys
...
for blank values.
2015-11-02 16:59:43 +00:00
b7552ba700
[ #158 ] Try harder to use the "get datasets since time X" method of harvesting. Go back to the last completely successful harvest, rather than just consider the previous one. And that had a bug, because fetch errors were ignored, meaning one fetch error could mean that dataset never got harvested again.
2015-11-02 16:59:19 +00:00
443d690ac8
Fixed big typo error.
2015-11-02 16:45:16 +00:00
53f692b802
Merge remote-tracking branch 'remotes/upstream/master'
2015-11-02 16:00:14 +00:00
1702cf2f09
Remove ', None' on .get() calls because it's the default value.
2015-11-02 15:51:25 +00:00
0c19acba78
Changed double quotes to single quotes in docstrings.
2015-11-02 15:50:04 +00:00
a6069d93db
Fixed bug where the harvest source url validator would validate against
...
all harvest sources that were ever created instead of just sources that
were currently enabled.
2015-10-30 16:59:04 +00:00
3f37ae5f45
Corrected docstring.
2015-10-30 16:11:25 +00:00
02b81187df
Fixed bug with deleting harvest source's which have a custom
...
configuration. Added PEP-8 compliance.
2015-10-30 15:15:41 +00:00
55325f5940
Updated harvest source url validator to allow for duplicate URL's with
...
unique configs.
2015-10-30 11:59:24 +00:00
2c41293c9c
Updated the validator to check for unique sets as well as URL.
2015-10-29 18:30:51 +00:00
1a680f3fd3
[ #158 ] Fix spaces encoding broken in previous merge. Tested with data.gov.uk.
2015-10-29 17:31:04 +00:00
39ce744368
Modified to make PEP-8 compliant.
2015-10-29 17:18:51 +00:00
f1d2d5fdc4
[ #111 ] Run jobs straight away.
2015-10-28 21:58:36 +00:00
421e6da660
Add run_test, job_abort, source commands
...
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
2015-10-28 17:51:58 +00:00
e2ab9e58e7
Merge remote-tracking branch 'origin/master' into 157-version-three-apify
...
Conflicts:
ckanext/harvest/harvesters/ckanharvester.py
2015-10-28 14:34:27 +00:00
3f74c29c99
Merge branch 'master' into 157-version-three-apify
2015-10-27 17:45:27 +00:00
55245b5091
[ #158 ] PEP8/formatting.
2015-10-27 17:43:11 +00:00
2a79873855
[ #158 ] Use package search to get all datasets. Add paging search results. Store pkg_dict from search in the object rather than request it again in fetch_stage.
2015-10-27 17:33:22 +00:00
86630adab7
Merge branch 'include-exclude-org'
2015-10-27 15:52:55 +00:00
b56fae8aed
Fixes and tests
...
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
2015-10-23 17:30:28 +00:00
24574f485b
Setup harvest model in harvester tests
2015-10-23 15:43:01 +01:00
caeeace8dc
Merge branch 'master' into 157-version-three-apify
2015-10-23 14:39:48 +01:00
bc49149d5e
Merge branch 'master' into include-exclude-org
2015-10-23 14:36:53 +01:00
0c0a996b85
Merge branch 'master' into db-error
...
Conflicts:
ckanext/harvest/queue.py
2015-10-23 13:33:44 +01:00
2f4adfb338
Merge branch 'tests'
2015-10-23 13:18:15 +01:00
3c6cc55be0
Only flush keys on the current Redis database
2015-10-23 11:52:22 +01:00
fdbade465f
Merge branch 'master' into purge
2015-10-23 11:33:43 +01:00
d950b13400
Merge branch 'unique-names-improved'
2015-10-23 11:02:49 +01:00
501edffe2d
Merge branch 'master' into migration-states
2015-10-23 10:59:04 +01:00
3e4a9933ce
Remove prints.
2015-10-21 16:52:19 +00:00
dc7af5d150
Remove prints.
2015-10-21 16:38:03 +00:00
eb9aa17862
Include/exclude orgs funcationality based on work by memaldi and ross.
2015-10-21 16:33:16 +00:00
f70c16bce7
Add framework for testing harvesters. Modernize existing tests.
2015-10-21 16:26:57 +00:00
d1f84295f8
purge_queues command now has warning about impact of Redis flushall, plus add some (log) output when you run a purge.
2015-10-21 16:12:40 +00:00
6360681a8f
[ #105 ] Fix order of deletes, as agreed with @florianm.
2015-10-12 15:57:27 +01:00
82bdff2f34
Add tests
2015-10-01 17:59:17 +01:00
be3e88086a
Generating unique names improved
...
* Harvesters that change the name when the title changes have had a
problem when the change is small and a number was unnecessarily
appended. e.g. "Trees "->"Trees" meant _gen_new_name("Trees") returned
"trees1". Now you can specify the existing value and it will return
that if it still holds.
* Maximum dataset name length is now adhered to.
* To make a name unique, a sequential number is now added, since for
users that is more understandable and pleasant. However hex digits are
still an option, for those that want to harvest concurrently.
2015-10-01 17:53:03 +01:00
1a6dca7c00
[ #148 ] Catch a more specific exception.
2015-10-01 12:30:40 +01:00
6dd40bfcf9
Changes the gather state to use v3 API
...
Rather than using the revisions in v2 API this now uses the
package_search API so that we can extend it with proper filters in
future.
2015-09-10 18:53:16 +01:00
a6cdda0a14
set max version to 2.4.99
2015-08-19 08:41:42 +00:00
1905caa961
upgrade harvest_source_clear to not delete from authz models removed in migration 078
2015-08-19 10:25:20 +08:00
de17e0ae8c
Catch, record and recover from temporary db problems.
2015-07-22 10:25:11 +01:00
46f7b32b04
Merge branch 'master' of github.com:okfn/ckanext-harvest into migration-states
2015-07-22 10:13:55 +01:00
2da918c2e4
Fix migration for old harvests so that ones that errored are correctly marked. Added helpful comments in model.
2015-07-22 10:13:02 +01:00
ab76830e85
[ #145 ] Throw + catch a custom exception if there are no jobs to run
...
If there are no harvesting jobs to run, there was always an ugly
exception message when using the paster command. This replaces the ugly
output with a proper message and uses a custom exception to allow others
to deal with this error differently.
2015-07-20 18:41:50 +02:00
83dd0b4b68
[ #138 ] Add data attributes to support timezone conversion
2015-07-09 22:35:54 +02:00
4dc2f7367d
[ #139 ] Delete package relationships when clearing a harvest source
2015-06-26 17:20:23 +02:00
88d9ba0397
[ #136 ] Fix broken RabbitMQ queue names
...
The harvester command was still using the old ones.
Use specific ones for testing.
2015-06-11 13:56:22 +01:00
673dfc9882
[ #127 ] Use site user on the CKAN harvester
...
Add missing call
2015-06-11 10:38:33 +01:00
d3a3f09ad1
[ #127 ] Use site user on the CKAN harvester
...
To avoid having to create a 'harvest' sysadmin explicitly. It will still
be used if present, but if not the site user will be used. You can also
define to user to use via a config option.
2015-06-11 10:19:07 +01:00
b17c3269b5
Merge branch 'clear-command' of https://github.com/metaodi/ckanext-harvest into metaodi-clear-command
2015-06-10 15:32:37 +01:00
64ff0f3a3a
Use single quotes to be consistent
2015-06-10 16:22:04 +02:00
2a2d85f60c
Wording changes for clearsource and rmsource
2015-06-10 16:19:23 +02:00
92b93c53fc
add some translation strings
2015-06-10 12:14:20 +01:00
8ebb843052
Add documentation for clearsource command
2015-06-10 11:29:24 +02:00
61bc150ae6
Expose clear harvester source as a paster command
2015-06-10 11:19:10 +02:00
9f8aae3a18
Append site id to queue name
...
This allows multiple CKAN sites to share the same RabbitMQ exchange
(For the Redis backend this is handled via different Redis databases)
2015-06-01 17:54:22 +01:00
3e21ea4f82
Fix tests, set up Travis
...
TODO: sort out the tests properly, avoiding imports from the legacy ones
2015-04-07 13:31:45 +01:00
f72d6da521
Change toolkit import
...
Apparently on package installs this is not well supported
from ckan.plugins.toolkit import check_ckan_version
But this works:
from ckan.plugins import toolkit
toolkit.check_ckan_version(...
2015-03-19 12:48:46 +00:00
7a20e93716
Raise on startup import errors so we don't mask problems
...
Otherwise if there was eg an actual ImportError we jut got
2015-03-19 12:30:08,430 DEBUG [ckanext.harvest.plugin] No auth module
for action "update"
on the log
2015-03-19 12:48:15 +00:00
859133fe36
move detecting unchanged datasets to ckanharvester and queue.py
2015-03-10 14:48:41 +02:00
d6e9b80496
Merge pull request #118 from clementmouchet/114-remove_resource_groups
...
Removed ResourceGroup from query when using CKAN 2.3 or above
2015-02-24 09:56:44 +00:00
ead9e67a33
updated def harvest_source_clear() to delete resource views, resource revisions & resources in CKAN >= 2.3
2015-02-23 17:02:21 +00:00
b3ed6cae5a
Merge pull request #121 from metaodi/120-create-remote-orgs
...
Fetch remote organization via action api
2015-01-15 10:49:09 +00:00
c1bcee9684
Use str() to get the error message
2015-01-15 11:36:15 +01:00
191c39ce5c
Catch the more general URLError instead of HTTPError
...
HTTPError is a subclass of URLError, so catch URLError is enough. I
think the HTTP error code is not as important in this situation, so
catching the more generic error seems like the best solution.
2015-01-15 10:57:24 +01:00
b978c26e70
Use ContentFetchError instead of generic Exception
2015-01-15 00:49:11 +01:00
935b9dda01
Munge group name before fetching remote group
...
The API call /api/2/rest/package/<id> returns the display name of the
group instead of its ID. To properly match the group, munge the name
before calling /api/2/rest/group
2015-01-15 00:44:53 +01:00
ef35c21e2a
Improve exception handling with custom exception
...
1. Try whenever possible to catch specific exceptions
2. Raise custom exception where appropriate
3. Fix the exception handling in _get_group and _get_organization
2015-01-15 00:44:45 +01:00
0fd38e0e54
Use _get_group as a fallback for remote orgs
...
First try to get a remote org from the remote Action API, if this fails
try to use the old rest api call, which works on older CKAN versions.
Only if both options fail, its currently not possible to get the remote
organization.
2015-01-14 00:10:27 +01:00
f214577872
Fetch remote organization via action api
...
Organizations used to be returned by /api/2/rest/group, this is what the
old implementation used to fetch the information to create the remote
organization on the local instance of CKAN.
With this commit the Action API is used to fetch the same information.
2015-01-13 14:46:53 +01:00
ea9debf714
Fix logic of conditional and make it more pythonic
2014-12-18 16:03:33 +01:00
08930d01bf
Make sure for new packages get a unique 'name'
2014-12-16 15:02:36 +01:00
82c7988bf3
Removed ResourceGroup from query when using CKAN 2.3 or above
2014-12-12 13:10:40 +00:00
a3affc9702
Fix validators on harvest_source_show schema
...
Remove validators on several keys so they don't get stripped during the
show validation.
2014-10-08 12:02:26 +01:00
098b54f1e5
Merge branch 'clear-source-delete-related' of https://github.com/waldvogel/ckanext-harvest into waldvogel-clear-source-delete-related
2014-09-29 13:49:19 +01:00
e60e2eee03
Fix output for harvest_source_create/update
...
They were using an incorrect schema, so not returning a harvest source
like dict.
2014-09-29 12:43:37 +01:00
c9b4e10506
delete records from related and related_dataset when clearing source
2014-09-12 10:56:37 +02:00
1e0376cff6
fix typo
2014-09-10 10:33:13 +03:00
f6c1456abe
fix job reporting to have job finished timestamp when there was zero datasets to gather
2014-09-10 09:22:55 +03:00
97f09913cf
fix job reporting all datasets deleted when actually nothing changed during last two harvests
2014-09-10 09:22:44 +03:00
8cf254f112
Merge branch '99-all-non-ascii-tags' of https://github.com/morty/ckanext-harvest into morty-99-all-non-ascii-tags
2014-08-29 14:40:43 +01:00
546159744e
Merge branch '101-modified-package-name' of https://github.com/morty/ckanext-harvest into morty-101-modified-package-name
2014-08-29 14:38:33 +01:00
039ac7c0ad
Always remove harvest extras on after_show if there
...
Up until now we where relying on `for_edit` being present in the
context, but this is only added on the controllers. It's better to be
safe and remove them always. If needed (at index time) they will be
added afterwards.
2014-08-14 15:31:39 +01:00
8a2c072d4e
[101] Use name from database when reharvesting package
2014-08-12 11:18:48 +01:00
65cfade420
[99] Remove empty tags produced by munging all non-ascii tags
...
I thought this way of filtering was easier to read than filter(None, tags)
2014-08-07 17:05:16 +01:00
13dbb1eea4
Fix variable not defined
2014-07-30 15:49:02 +01:00
58a873ac7a
[ #91 ] Remove config fields from source dict before indexing
...
We don't need them and will avoid indexing errors
2014-06-27 16:54:39 +01:00
a59ab4b5ff
[ #91 ] Consolidate all harvest source reindex code in a single action
...
Make it available to users with permissions on the harvest source
2014-06-27 16:48:14 +01:00
7459358fa1
Support for single import commands
...
We are now able to run `paster harvester import` for a single harvest
object or for a single dataset, providing ids or name.
2014-05-15 16:30:30 +01:00
2c6aaf5bb1
Merge branch 'master' into 96-harvest-object-encoding-errors
2014-05-15 15:52:13 +01:00
43f1d08255
[ #97 ] Persitent endpoint for datasets harvest objects
...
Contrary to `/harvest/object/xxx`, this endpoint is passed the dataset
id, thus it not depends on a particular object but the most recent one.
2014-04-30 17:45:07 +01:00
1b458b1772
[ #96 ] Handle encoding errors on harvest object endpoint
...
When parsing the harvest object content to see if it is an XML file,
etree.fromstring would fail id there are incorrect unicode errors.
2014-04-28 12:48:09 +01:00
e3492b57e7
Update plugin.py
...
Updating plugin.py to check for the existence of the extras key in the data_dict.
2014-02-27 16:05:39 +00:00
d3cf5e58d1
[ #86 ] Fix duplicate extras
2014-02-11 18:16:49 +00:00
fbde0b8dc1
[ #87 ] Remove remote url_type from resources
...
Otherwise CKAN thinks they are uploads, datastore resources, etc, which
it can cause problems eg when displaying the URL of the resource. We
are just linking to the remote resource URL.
2014-02-11 17:27:19 +00:00
5739e541d7
[ #80 ] Support for Python 2.6 when handling xml exceptions
2014-02-10 18:44:46 +00:00
2a07a144fc
[ #84 ] Fix auth audit exception when creating datasets
...
This was caused by a combination of the auth audit leaking and the
harvester reusing the context for the package_show and package_create
actions. If the package is not found, package_show does not call
check_access, and the auth audit does not pass. This is stored in the
context (`__auth_audit`) and is raised next time that we call
get_action (when we call package_create with the same context)
It could potentially be fixed on master, but it is probably quite rare.
2014-02-10 18:22:48 +00:00
5b677b6099
[ #83 ] Fix key error when using default_groups
2014-02-10 13:16:58 +00:00
bf11e4d330
Moved clean_tags check into _create_or_update_package method.
2014-02-10 09:29:01 +01:00
2ba9908653
Config option to munge tags changed to be consistent with other config options in this extension, and noted in README.
2014-01-29 10:55:51 +01:00
5e1aef1d08
Removed extra newline.
2014-01-29 10:06:32 +01:00
7d71b0a00b
Wrap tag munging code in config option, defaulting to False.
2014-01-29 10:02:16 +01:00
2b803a3f66
[ #77 ] Use auth_allow_anonymous_access decorator
...
Starting from 2.2 you need to explicitly flag auth functions that
allow anonymous access with the p.toolkit.auth_allow_anonymous_access
decorator. A local version of the decorator is used to ensure we only
use it on CKAN>=2.2
2014-01-20 13:47:37 +00:00
4cc56f51ab
[ #76 ] Use harvest_source_show on reindex command
2014-01-14 17:04:34 +00:00
95d0ef0f01
[ #76 ] Add extra fields to the source schema
...
Add 'private' and its core validators, and 'metadata_modified' and
'metadata_created'.
Also ignore '__extras'
2014-01-14 17:01:25 +00:00
467fb7bb8f
Fix resource updating for harvested datasets
...
Starting from 2.2, resource_update calls package_show before updating
the resource via a package_update call. The dict passed had the harvest
extras (eg harvest_object_id) added which made the update call fails due
to duplicated extra keys. To fix it we now remove any harvest extras
on after_show if there is a 'for_edit' property on the context.
2014-01-13 10:30:52 +00:00
278a8e1ada
Merge branch 'master' of github.com:okfn/ckanext-harvest
2014-01-10 13:49:38 +00:00
1e94a11255
[ #70 ] Fix Add harvest source button not showing
...
Due to changes in the templates starting on 2.1 the add source button
was not showing. The whole search template has been simplified,
separating in a separate file the 2.0 only code.
Tested in 2.0, 2.1 and 2.2
2014-01-10 13:48:02 +00:00
51e842ee6e
Add quotes to harvest_source_id in Solr query when clearing harvest sources
2013-11-22 11:01:26 +02:00
c338452872
Put harvest_source_id in quotes to prevent Solr errors
2013-11-21 14:14:56 +02:00
928ea061aa
Improve organizations dropdown on source form
2013-10-24 12:33:44 +01:00
c52085006a
[ #61 ] Truly ignore harvest sources
...
The currently implementation returns False when a harvest source is being harvested. This leads to an error on the harvesting job, which in turn tends to confuse users that have no idea of this special implementation. This fix ensures that harvest sources are still ignored, but silently.
2013-10-23 07:40:55 +02:00
c18d9dc3af
[ #71 ] CKAN harvester: Add datasets to source organization
...
If the harvest source belongs to an organization, new datasets should be added
to it. This is already the case in the spatial harvesters.
The remote orgs logic has been kept, with the difference that if for
some reason the remote org can not be assigned, the local one is used.
If the source does not have an organization, none is added.
2013-10-22 16:24:43 +01:00
380c14c22c
Fix CLI sources list output
2013-10-16 13:03:32 +01:00
55d2b4e304
Fix purge command
2013-10-16 12:59:23 +01:00
bd62b62764
Merge branch 'metaodi-add-harvesting-of-organizations'
2013-10-15 17:50:04 +01:00
49999893e7
Merge branch 'add-harvesting-of-organizations' of git://github.com/metaodi/ckanext-harvest into metaodi-add-harvesting-of-organizations
2013-10-15 17:49:28 +01:00
8b5d70c6fe
Only try to create/match a organization if there is a remote_org
2013-10-11 18:08:32 +02:00
0f5624822c
Use remote name if present when creating datasets on CKAN harvester
2013-10-11 16:50:25 +01:00
340e9eed63
Merge branch 'add-harvesting-of-organizations' of git://github.com/metaodi/ckanext-harvest into metaodi-add-harvesting-of-organizations
2013-10-11 16:14:18 +01:00
e9dde3f48a
Only show the 'Add Harvest Source' button if user is authorized
2013-10-11 11:55:14 +01:00
dd1acd0c6b
Use remote_orgs for organizations
2013-10-07 11:22:19 +02:00
f89f12203c
Merge branch 'fix/rename-ampq-to-amqp' of git://github.com/opendatatrentino/ckanext-harvest into opendatatrentino-fix/rename-ampq-to-amqp
2013-10-04 17:24:53 +01:00
c5f4d6889b
Merge branch 'harvest-object-create'
2013-10-04 17:22:58 +01:00
d50eb6fca8
Harvesting of remote organisations similar to remote groups
2013-10-04 16:37:52 +02:00
da2fd45e80
[ #65 ] make harvest_job_exists validator return model object
...
return the model in the validator instead of checking that it exists in
the validator, returning the id and then fetching it again in the action
function
2013-10-03 15:51:37 +01:00
611b9aab6d
Fixed typo: ampq -> amqp
2013-09-19 11:43:03 +02:00
9b3199b41b
[ #65 ] remove unused code
2013-09-17 17:02:38 +01:00
5da153c6b6
[ #65 ] harvest_object_create action
...
update to use schema and validators. Also accept more parameters to
data_dict.
2013-09-17 16:49:19 +01:00
71aedf3fd4
Makes ede45bd work in both CKAN 2.0 and 2.1
2013-09-05 15:36:00 +01:00
ede45bd1be
Fixes #66 by correcting the block name
2013-09-05 15:31:44 +01:00
1b663bbff4
add harvest_object_create action
2013-09-04 14:17:01 +01:00
52956feab9
Merge branch '62-default-package-name'
2013-08-19 18:23:37 +01:00
f51b8e905a
[ #58 ] Check properly for version numbers (patch numbers)
2013-08-19 18:13:01 +01:00
f028375ad3
[ #62 ] Use current name when updating package, if the user haven't sent a new one
...
It's hard for someone outside CKAN to make sure they're sending it in the format
we expect. And they'll also have to keep track of our name format, to keep in
sync whenever we change.
To fix this, we simply do what we already do when creating packages: use a
default name. In this case, the current one.
2013-08-18 12:08:30 -03:00
86dcd933ea
Merged master
2013-08-15 18:47:16 +01:00
712e150b52
[ #58 ] Fix to make merge nice
2013-08-15 18:43:46 +01:00
575df637b4
[ #58 ] Fixes to make harvest templates to work with both CKAN 2.0 and 2.1
2013-08-15 16:45:02 +01:00
05e6362c38
Merge branch 'fix-jinja-status-exception' of git://github.com/metaodi/ckanext-harvest into metaodi-fix-jinja-status-exception
2013-08-15 14:39:20 +01:00
01ca5c0dfd
[ #61 ] Ignore harvest sources on the CKAN harvester
2013-08-15 14:38:33 +01:00
b25fffda93
[ #36 ] Fix bug on API version checking
2013-08-15 14:37:55 +01:00
39ad78d90a
[ #59 ] Ignore auth in the CKAN harvester
2013-08-15 14:37:12 +01:00
f26baf6c09
Hide both the label and the number of datasets when 'status' is not available
2013-08-15 13:25:16 +02:00
1c36b33aaf
[ #59 ] Ignore auth when using site_user
2013-08-14 12:28:27 +01:00
ffea49ca62
[ #56 ] Update parameters on source create command
...
Add missing title and owner_org fields, remove deprecated user_id and
publisher_id
2013-08-14 11:54:51 +01:00
3494727d3f
[ #56 ] Increase max params number
2013-08-14 11:43:32 +01:00
8e33262026
[ #56 ] Fix syntax error and wrong type
2013-08-14 11:31:23 +01:00
4dfd091aec
Make the /harvest page more robust if source.status is not set
...
This prevents exceptions from appearing in the log from Jinja:
[error] [client 1.2.3.4] Error - <class 'jinja2.exceptions.UndefinedError'>: 'dict object' has no attribute 'status'
2013-08-14 11:52:11 +02:00
7ae9d6e208
Made print method more robust against KeyErrors
...
This is especially needed if you create a new harvest source which does not have all the optional arguments. Before this lead to a KeyError after the creation of the source. Now this simply output 'None'.
2013-08-05 23:50:30 +02:00
1249564be5
Add additional name argument when creating new harvest source
2013-08-05 23:46:21 +02:00
ade5f83e38
Change key of data_dict from 'type' to 'source_type'
2013-08-05 23:07:25 +02:00
cb745c3c3e
Avoid importing unnecessary functions from the harvest logic
2013-08-05 18:39:44 +01:00
70e53a7833
Fix bug where source was being treated as an object, when it's a dict
2013-07-29 07:06:58 -03:00
cc3f3d3426
[ #50 ] Fix objects deletion on gather exceptions
2013-07-05 13:29:11 +01:00
e2696b98bb
[ #50 ] Save all dates as UTC in the database
...
At some point we may want to transform these to local time at the
dictization level. We will need a library like dateutil to handle it
properly though.
2013-07-04 14:59:27 +01:00
c2283e3fdb
only migrate harvest sources which are active
2013-06-28 02:32:45 +01:00
a42991b8c9
fix so that non sysadmins can edit harvest sources of organizations they
...
are admins or editors of.
2013-06-27 12:16:11 +01:00
6540726c47
use correct limit for paging harvest listing
2013-06-26 11:14:38 +01:00
584c340583
Merge branch '42-remove-non-string-extras'
2013-06-03 10:33:59 +01:00
01df3a1db4
[ #42 ] Dump non-string extras with json
...
Convert any non-string extra values to strings using json.dumps(),
instead of just deleting them.
2013-05-31 20:35:06 +02:00
3a31db59b6
[ #36 ] Move validation code to validate_config
...
This ensures it is checked whenever the source is edited or created.
2013-05-31 17:23:40 +01:00
a6a0196a4e
Merge branch 'api-version-fix' of git://github.com/fraunhoferfokus/ckanext-harvest into fraunhoferfokus-api-version-fix
2013-05-31 17:15:43 +01:00
85a013f2c9
[ #42 ] Remove non-string extras from packages
...
Remove extras whose values are not strings (e.g. dicts, lists..) from
packages before attempting to create or update the packages on the
target site.
In CKAN 1 it was possible for the values of extras to be other types,
but in CKAN 2 they must be strings, so when harvesting from a CKAN 1 site
into a CKAN 2 site SQLAlchemy would crash when trying to create packages
with non-string extras.
The fix in this commit is to simply remove any non-string extras from
the harvested package. (Alternatively, we could try to convert them to a
string using JSON.)
Fixes #42 .
2013-05-31 15:43:42 +02:00
361abcfc07
[ #17 ] Fix bug with remote groups handling
...
If neither 'only_local' or 'create' are used the remote groups property
needs to be removed, otherwise it causes an exception when the group is
not found.
2013-05-30 18:06:15 +01:00
87cae31c75
Fix api_version check in the group importer code
...
I have forgotten to update one check for the api_version 1 in the code
responsible for the remote group import feature. This commit fixes that.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 13:36:56 +02:00
c858b9fe9f
Add exception handling for the API version parsing
...
I have added try-except clauses in order to prevent the process from
crashing if a non-parsable integer is used for the api_version option.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 13:12:05 +02:00
05094090af
Change type of the API version to integer
...
The CKAN logic uses integers when dealing with the API version, e.g.
making checks which API version is in use. Currently, the harvester
uses strings to identify the API version. Instead of dealing with
type conversion the harvester could use integers directly.
This commit fixes okfn/ckanext-harvest#36 . When the API version is
parsed from the configuration it is passed through the int() function.
This way the harvesting will still work even if a harvest source was
configured with a string API version which makes this commit backward
compatible.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 12:51:48 +02:00
ff7287d4b4
[ #30 ] Remove lxml dependency
2013-05-24 18:12:02 +01:00
3d2867ca04
[ #17 ] Remove ckanclient dependency as it is not used
2013-05-24 17:55:37 +01:00
f1d11c1307
[ #17 ] Import remote groups in CKAN harvester
...
This is a cleaner commit of the great work done by @platzhirsch
implementing remote groups import on the CKAN harvester.
2013-05-24 16:55:05 +01:00
1792180e4f
Better harvest source dataset migration
...
Current implementation only checked for the first source to exist and
didn't allow to rerun the migration for other sources if there was an
error. With the new one, all non existing sources are migrated each
time.
2013-05-24 14:49:55 +01:00
1d54edfdaa
Fix bug in source datasets migration
...
Wrong dataset type was causing the default package schema to be used,
thus failing when providing an id.
2013-05-24 14:25:05 +01:00
751409ab7d
[ #34 ] Integrate clear command with delete source
...
When deleting a source, if clear_source equals true in the context,
harvest_source_clear will be called. Default is false. The UI shows a
select with the two options.
2013-05-20 14:30:22 +01:00
6d5d0fbaae
Add hover helper text to refresh and clear buttons
2013-05-20 12:09:14 +01:00
edfc49719b
Use page_heading helper consistently with the main CKAN templates.
2013-05-17 16:12:57 +01:00
d0bc52f2d8
[ #34 ] Fix typo in warning message
2013-05-16 18:07:32 +01:00
b9e2613458
[ #34 ] Allow all authorized users for a source to clear it
2013-05-16 17:57:59 +01:00
71349e658b
[ #34 ] Expose harvest source clear button
2013-05-16 17:51:48 +01:00
7b652542e7
[ #34 ] Fix harvest_source_clear action
...
Some typos in the SQL statements, and also the source needs to be
reindexed to update the status with the counts.
2013-05-16 17:33:39 +01:00
1efd7ab4cd
Ignore remote orgs in CKAN harvester
...
If #17 progresses we can do somethign similar for them, although it amy
be more complicated because of authorization issues.
2013-05-16 17:30:54 +01:00
1714e55110
simplify harvest_clear queries so they do not lock on big db
2013-04-30 13:59:23 +01:00
a2b8ab1994
make harvest source clear not create table
2013-04-30 12:40:46 +01:00
9041f3f3ad
Changes in Redis conusmer to make tests work
2013-04-22 18:08:19 +01:00
70dfee1a36
Update queue tests
2013-04-22 17:56:11 +01:00
dcfd201cdd
[ #32 ] redis queue support
2013-04-21 17:04:57 +01:00
0ce59a29b6
delete insead of update harvest objects when error
2013-04-12 12:32:33 +01:00
7d7657f94a
make gather phase as finished if there is an error
2013-04-12 10:35:08 +01:00
bd761498f0
make sure config dict is not jsonified if it contains an error
2013-04-08 18:52:36 +01:00
eaebeb4e6e
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-04-08 13:25:33 +01:00
5414b6c08d
Merge branch '29-new-idataset-form' into release-v2.0
2013-04-08 13:23:41 +01:00
66ff773f99
remove previous commit import
2013-03-29 12:47:14 +00:00
3ac065f0f0
fix package_schema import
2013-03-29 01:17:24 +00:00
cb8b808274
sanity check that harvest source id matches harvest dataset id
...
remove author_email, license_id, maintainer_email, maintainer and
author from pacakge_dict, these were not actually necessary
2013-03-29 00:59:20 +00:00
99bd17401c
Handle wrong JSON in harvest_source_extra_validator
2013-03-28 16:19:16 +00:00
a9b8be8f01
harvest source index clear
2013-03-28 15:36:44 +00:00
95ebb5bbf3
[ #29 ] Remove check_data_dict ✨
2013-03-28 15:01:21 +00:00
fbc8ecde97
[ #29 ] Fix some imports on actions and plugin
2013-03-28 15:00:44 +00:00
c754479014
#29 make new idatasets form work with harvest source form
2013-03-25 17:38:07 +00:00
548d3c1c2a
fix validation issue on db upgrade
2013-03-25 12:02:07 +00:00
b5a697ec87
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-25 11:58:31 +00:00
0b5c3c608a
catch and raise gather exception, acking the message
2013-03-25 11:57:57 +00:00
438ba672e2
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-03-25 11:44:37 +00:00
845c9927a8
add harvest source clear
2013-03-25 11:39:00 +00:00
d518b6709a
[ #27 ] fix package_list_for_source for unowned data sources
2013-03-21 15:59:22 +00:00
7bff041568
[ #25 ] Further tweaks on helpers texts
2013-03-21 13:47:23 +00:00
4d0dd9a4d3
[ #25 ] Small copy tweak to confirmation dialog
2013-03-21 12:14:33 +00:00
78bde974b9
[ #25 ] Adds confirmation dialog to reharvest button
2013-03-21 10:56:39 +00:00
amercader
David Read
Stefan Oderbolz
Raphael Stolt
Mark Winterbottom
Ross Jones
Florian Mayer
joetsoi
Jari Voutilainen
clementmouchet
waldvogel
Tom Mortimer-Jones
Richard Claydon
Rachel Knowler
Mikko Koho
Samuele Santi
John Martin
Vitor Baptista
kindly
Sean Hammond
Konrad Reiche
Tom Rees