4516bfe44e
PEP8 and lint, extracted from PR158
2016-02-15 13:50:18 +00:00
49faa0ae6c
Tests for CKANHarvester._last_error_free_job
2016-02-15 13:30:28 +00:00
385b369148
Error-free jobs now include ones where an object was not modified.
2016-02-15 13:16:23 +00:00
f63140354d
Fix logic error in previous commit
2016-02-15 12:28:46 +00:00
52c071dbe9
Improved error handling. e.g. if the site it harvests just returns errors.
2016-02-15 12:10:44 +00:00
331ad84272
Deal with worry about datasets on the remote CKAN being added/removed during harvest.
2016-02-12 18:00:00 +00:00
7096b7ddf2
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
2016-02-12 16:51:26 +00:00
69ea33647e
Update plugin.py
...
Prevent the 'harvest' datasets being shown in the group and dataset search results.
2016-02-12 13:15:09 +00:00
6354ad5656
Fix source clean command on CKAN > 2.5, as related don't exist any more
2016-02-04 13:40:02 +00:00
b58cd8b38f
Merge branch 'master' into 219-missing-form
...
Conflicts:
ckanext/harvest/templates_new/source/edit.html
ckanext/harvest/templates_new/source/new.html
test-core.ini
2016-01-14 11:43:12 +00:00
73196b942b
Merge branch '214-remove-genshi'
2016-01-14 11:28:55 +00:00
5d23fab03f
[ #219 ] Support CKAN < 2.3
2016-01-14 11:23:41 +00:00
5bf0ac9b86
[ #219 ] Fix tests auth
2016-01-14 11:23:17 +00:00
1665e86065
[ #219 ] Don't use c.form to render the form
2016-01-14 10:16:45 +00:00
497dfeea02
Add test for missing form
2016-01-14 10:15:46 +00:00
e0c3316531
Add legacy import for CreateTestData
2016-01-11 22:35:11 +01:00
c141cf44fa
Swap import statements to make sure `run_test` works on CKAN 2.3
2016-01-11 22:13:25 +01:00
82f48d5afa
[ #214 ] Move ckanext/harvest/templates/templates_new to ckanext/harvest/templates
2015-12-11 15:24:45 +00:00
6b23208b2a
Merge branch '212-module-import-error'
2015-12-11 13:38:32 +00:00
d08f72ad13
Fix tests for ckan 2.2 again with amercaders suggestion.
2015-12-11 11:53:51 +00:00
6ef58addbc
Fix tests for ckan 2.2 again with amercaders suggestion.
2015-12-11 11:52:00 +00:00
f0ba0c865c
Fix tests for ckan 2.2
2015-12-10 16:36:27 +00:00
77f98d5b18
Compatibility with ckan 2.2.
2015-12-10 16:33:09 +00:00
8f54421c99
[ #214 ] ckan.legacy_templates=True is really harmless, so just warn if it is set.
2015-12-10 16:11:56 +00:00
260cd1f2b7
Merge branch 'master' of github.com:ckan/ckanext-harvest into 214-remove-genshi
2015-12-10 16:02:50 +00:00
fa1ec64cc7
Rename templates_new dir to templates.
2015-12-10 15:52:46 +00:00
8f3006f494
[ #214 ] Config for legacy templates removed.
2015-12-10 15:46:57 +00:00
41975a93d8
[ #214 ] Removed genshi templates
2015-12-10 15:32:20 +00:00
468a195a25
Merge branch 'import-guid'
2015-12-10 13:37:23 +00:00
5ff3ef9d17
Merge branch '205-inconsistent-stats'
2015-12-10 13:36:15 +00:00
9d06820bcd
Merge branch 'error_creation_moved_to_model'
2015-12-10 13:25:05 +00:00
b7a0343763
Merge branch 'abort-by-job'
2015-12-10 13:22:40 +00:00
82fe2d0e53
Merge branch 'fetch_unchanged'
2015-12-10 13:19:45 +00:00
80b82ee968
Merge branch 'factory-job'
2015-12-10 13:18:19 +00:00
52fe7fb21d
Merge branch 'named-exceptions'
2015-12-10 13:17:18 +00:00
072698b4bb
Merge branch 'validator-raise'
2015-12-10 13:16:26 +00:00
4cc39e5ef4
Merge branch 'test-guid-fix'
2015-12-10 13:15:54 +00:00
04162ce9e4
Merge branch 'munge-tag'
2015-12-10 13:15:17 +00:00
4ca4b3a2f2
Merge pull request #213 from ckan/212-module-import-error
...
[#212 ] Fixes #212 - auth for harvest_job_create was broken.
2015-12-09 15:52:37 +00:00
8b8086fe48
[ #212 ] Fixes #212 - auth for harvest_job_create was broken.
2015-12-09 15:50:05 +00:00
84fb3e3325
Add support for the new ITranslation interface
...
Move the i18n files to the expected place, load the interface if CKAN >=
2.5.0
2015-12-09 14:46:12 +00:00
031e680b6c
Add option to re-import based on guid.
2015-12-08 16:17:39 +00:00
18169c5133
[ #205 ] "detailed" removed in a couple of other benign places.
2015-12-08 10:18:36 +00:00
05d4baf040
[ #205 ] "detailed" source status is removed. Now we have lost harvest_source_for_a_dataset it is not possible to call it. And it returned the wrong keys anyway.
2015-12-08 10:14:15 +00:00
67366093fe
[ #205 ] Remove harvest_source_for_a_dataset action. It has been deprecated since Mar 6 16:54:33 2013 and returns wrong harvest stats keys.
2015-12-08 10:06:42 +00:00
07c76b0cbf
Docs & pep8
2015-12-02 16:23:54 +00:00
121e8bd918
Merge pull request #198 from ckan/catch-exceptions
...
Catch exceptions from urllib2.urlopen more comprehensively
2015-12-02 16:22:53 +00:00
c7021933a0
Move creation of errors to the model as thats a more natural home. Provide backwards compatibility.
2015-12-02 08:15:13 +00:00
b53682f267
You can abort a job by specifying the ID of the job, rather than the source. This is helpful since the "harvest run" command returns a list of still running job ids.
2015-12-02 07:59:08 +00:00
f67029b993
Hint about rerunning the import_stage.
2015-12-01 17:59:11 +00:00
b8c7d39e1a
Document existing functionality for aborting during the gather stage.
2015-12-01 17:56:24 +00:00
25301e2152
Update README from interfaces.py. PEP8. Mention HarvestObject - Package relation in interface.
2015-12-01 17:51:35 +00:00
b0780b2062
Fetch stage can also return "unchanged", same as the import stage. Used by DGU. It is useful to skip an object like this, to avoid saving the fetched content in a HarvestObject (saves disk usage).
2015-12-01 17:38:57 +00:00
414c33ac6b
Easy fix to stop test_queue.py using MockHarvester in test_queue2.py by mistake.
2015-12-01 16:55:42 +00:00
673a64a820
Merge pull request #200 from ckan/fix-error-report-status
...
Fix job stats key
2015-11-30 17:49:14 +00:00
8163ec4d39
HarvestObject factory was creating a extra field containing the job, by mistake. PEP8
2015-11-27 14:21:16 +00:00
1f947471eb
Take advantage of named exceptions.
2015-11-27 11:57:40 +00:00
798866d872
Fix name of class to enable automatic conversion of datetimes
...
when ckan/ckan#2505 got finally merged, the class name to mark datetimes
with a time part that can be automatically converted to the users
timezone, changed.
This commit makes sure this change is reflected in ckanext-harvest.
2015-11-26 01:49:11 +01:00
4fbaec0986
Improve harvester type error message. Add docstring.
2015-11-25 20:58:50 +00:00
6c3efe53df
Fix job stats key - it is "errored" not "errors" - the keys are from report_status. And lots of PEP8.
2015-11-25 20:55:32 +00:00
f0a2e9fb8e
CKAN revision API returns package ids not names (for v2 of the API). This ensures harvest guid is always the ID rather than the name.
2015-11-24 16:41:43 +00:00
392c13d828
If not revisions then we get a 404, so deal with that better.
2015-11-23 21:36:45 +00:00
4405066fab
Catch exceptions from urllib2.urlopen more comprehensively. I think 400 errors were from CKAN 0.6 or something like that - ignore now.
2015-11-23 21:26:32 +00:00
4b5014d381
Fix test for older ckan.
2015-11-23 18:27:04 +00:00
3b4daf0609
fix typo
2015-11-23 17:40:35 +00:00
bc26159fb6
tag_munge from ckan 2.2 fails the test with dashes, so use the harvest one for this ckan version.
2015-11-23 17:31:20 +00:00
52f7e0dd07
Use the ckan version of munge_tag if available, but provide a fallback for older ckans.
2015-11-23 12:48:05 +00:00
129b1a0cf5
Enable custom solution to detect existing packages
...
With this change, all harvesters that extend the base harvester have the
possibility to use the very useful create_or_update method, but still
define their own way of detecting what package is the existing one.
This is very useful for harvest sources that have no knowledge of the
CKAN internal id, but have another way of finding previous package.
2015-11-20 16:31:47 +01:00
e71cf35504
Fix queue tests
2015-11-20 14:25:31 +00:00
f1ba2bcfb3
Namespace Redis keys to avoid conflicts between instances
...
The `ckan.site_id` config option (or `default` if missing) is used to
namespace the Redis keys: routing key and persistance key. Consumers
will only get the relevant keys for their instance.
2015-11-20 14:17:25 +00:00
920df684ae
Merge branch 'db-error'
2015-11-20 12:29:37 +00:00
ede50aa3fb
Merge branch 'immediate-harvest'
2015-11-20 12:28:35 +00:00
3f42eb6ba0
Merge branch 'revision-id-fix'
...
Conflicts:
ckanext/harvest/tests/harvesters/test_ckanharvester.py
2015-11-20 12:28:17 +00:00
60c4371df4
Add "not modified" to the stats always returned.
2015-11-17 12:45:00 +00:00
ae7c500745
Merge branch 'master' into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-17 12:35:59 +00:00
8e02aedc65
Fix tests
2015-11-17 13:29:25 +01:00
f9b87fff0c
Make sure all possible status are always returned
...
It makes it harded to parse the API response if you always have to check
if 'deleted' is set and if what value it has. I think to simply return 0
for all status value is good practice.
2015-11-17 11:43:11 +01:00
084723abb7
Catch JSONDecodeError when no JSON content
2015-11-16 10:59:18 +01:00
c0a865e64e
Revert ok_ - makes it slightly less readable for little benefit.
2015-11-13 13:45:56 +00:00
42ab55cb6d
No longer need uuid since we clear db between tests now. Added ignore_missing because of occasional failures.
2015-11-13 13:32:55 +00:00
b150b50887
Move the SkipTest to include inherited tests too.
2015-11-13 12:44:27 +00:00
01a4bfd314
Patch test should skip if ckan version is wrong, rather than ignore all exceptions from posts of all tests. Remove FunctionalTestBaseWithoutClearBetweenTests now the tests are modernized.
2015-11-13 12:33:18 +00:00
1288a4d9e7
Reflow text to 79 char width. Warning not necessary with an exception I think.
2015-11-13 12:32:13 +00:00
0ce3748153
Do not use ActionError as this does not yet exist in CKAN 2.2
2015-11-13 12:01:19 +01:00
c33c6e8c13
Raise an error instead of falling back to harvest_source_update
...
As the behaviour of *_patch is clearly different from the *_update we
should raise an error if this action is called on a CKAN instance, where
the action is not available.
2015-11-13 11:41:53 +01:00
644fa49dd4
Make tests independent from cls.sysadmin
...
Generate unique harvest sources
2015-11-11 19:49:49 +01:00
ffca5cc3da
Use new factory style for fixtures
...
- Remove default_source_dict from tests
- Replace setup_class with constructor
- Create mixin for harvest source fixture
- Replace assert with assert_equal where appropriate
- Replace assert with ok_
- Remove dependency to global SOURCE_DICT
- General refactoring of tests
2015-11-11 19:07:59 +01:00
136fcb87d5
Make sure package_patch has a fallback for package_update on CKAN < 2.3
2015-11-11 11:37:23 +01:00
359da2eb69
Add test class for harvest_source_patch
2015-11-11 11:34:01 +01:00
3f09010039
Add harvest_source_patch to API
2015-11-11 05:39:29 +01:00
735ab3e286
[ #157 ] Try to fix test for ckan 2.2 - cf 91afc0e928
2015-11-04 11:37:03 +00:00
a0742d69b0
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
...
Conflicts:
ckanext/harvest/logic/action/update.py
2015-11-04 09:50:00 +00:00
679ed421e9
Merge branch 'master' of github.com:ckan/ckanext-harvest into immediate-harvest
...
Conflicts:
ckanext/harvest/logic/action/update.py
2015-11-04 09:44:05 +00:00
f0207ad38f
Merge branch 'master' of github.com:ckan/ckanext-harvest into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-04 09:36:55 +00:00
cbe9b40e66
Merge branch 'master' of github.com:ckan/ckanext-harvest into revision-id-fix
2015-11-04 09:36:08 +00:00
f9da3654f8
[ #184 ] Fix tests for older ckan versions.
2015-11-03 23:27:52 +00:00
5fba056c59
[ #184 ] Add tests
2015-11-03 23:19:05 +00:00
77e5b89a01
Blank line needed.
2015-11-03 22:23:04 +00:00
8c1f7619cb
Fix code style to be more ckan-like whilst still pep8.
2015-11-03 22:08:46 +00:00
20531c0dda
Merge branch 'master' of github.com:ckan/ckanext-harvest into LondonAppDev-master
...
Conflicts:
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/validators.py
2015-11-03 22:02:49 +00:00
10685badb5
PEP8 based on #174
...
Conflicts:
ckanext/harvest/logic/action/delete.py
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/validators.py
2015-11-03 21:56:06 +00:00
5a5260ff0b
Add test for harvest_source_clear since the PEP8 changes were quite a lot there.
2015-11-03 21:42:39 +00:00
4f71612002
PEP8 based on #174
2015-11-03 20:30:11 +00:00
208d1c4185
Setting back to master.
2015-11-03 17:31:00 +00:00
91afc0e928
[ #178 ] Fix test for ckan 2.2, which does not suffer problem #180 .
2015-11-03 10:01:47 +00:00
f4f124c181
[ #178 ] Fix resouce_revision_id_fkey error. Fixes #178 .
2015-11-03 07:38:01 +00:00
59be6e2c71
Merge branch 'master' into db-error
...
Conflicts:
ckanext/harvest/queue.py
2015-11-03 00:57:14 +00:00
8a7bc9e1d8
Merge remote-tracking branch 'origin/master' into immediate-harvest
...
Conflicts:
README.rst
ckanext/harvest/commands/harvester.py
ckanext/harvest/logic/action/create.py
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/auth/update.py
2015-11-03 00:40:25 +00:00
c7fac36c1c
[ #107 ] "unchanged" response tested and related fixes
...
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
2015-11-03 00:22:53 +00:00
e59760fefe
Merge branch 'job-reporting-fixes' of https://github.com/yhteentoimivuuspalvelut/ckanext-harvest into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-02 21:25:32 +00:00
24415844e0
[ #158 ] Fix revision_id problem in second harvest.
2015-11-02 18:13:29 +00:00
d495e269e7
[ #158 ] Fix tests
2015-11-02 17:29:45 +00:00
14f372aec6
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
...
Conflicts:
README.rst
2015-11-02 17:01:22 +00:00
7ffd6748f3
Corrected docstring params field, duplicate if statement and deleting keys
...
for blank values.
2015-11-02 16:59:43 +00:00
b7552ba700
[ #158 ] Try harder to use the "get datasets since time X" method of harvesting. Go back to the last completely successful harvest, rather than just consider the previous one. And that had a bug, because fetch errors were ignored, meaning one fetch error could mean that dataset never got harvested again.
2015-11-02 16:59:19 +00:00
443d690ac8
Fixed big typo error.
2015-11-02 16:45:16 +00:00
53f692b802
Merge remote-tracking branch 'remotes/upstream/master'
2015-11-02 16:00:14 +00:00
1702cf2f09
Remove ', None' on .get() calls because it's the default value.
2015-11-02 15:51:25 +00:00
0c19acba78
Changed double quotes to single quotes in docstrings.
2015-11-02 15:50:04 +00:00
a6069d93db
Fixed bug where the harvest source url validator would validate against
...
all harvest sources that were ever created instead of just sources that
were currently enabled.
2015-10-30 16:59:04 +00:00
3f37ae5f45
Corrected docstring.
2015-10-30 16:11:25 +00:00
02b81187df
Fixed bug with deleting harvest source's which have a custom
...
configuration. Added PEP-8 compliance.
2015-10-30 15:15:41 +00:00
55325f5940
Updated harvest source url validator to allow for duplicate URL's with
...
unique configs.
2015-10-30 11:59:24 +00:00
2c41293c9c
Updated the validator to check for unique sets as well as URL.
2015-10-29 18:30:51 +00:00
1a680f3fd3
[ #158 ] Fix spaces encoding broken in previous merge. Tested with data.gov.uk.
2015-10-29 17:31:04 +00:00
39ce744368
Modified to make PEP-8 compliant.
2015-10-29 17:18:51 +00:00
f1d2d5fdc4
[ #111 ] Run jobs straight away.
2015-10-28 21:58:36 +00:00
421e6da660
Add run_test, job_abort, source commands
...
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
2015-10-28 17:51:58 +00:00
e2ab9e58e7
Merge remote-tracking branch 'origin/master' into 157-version-three-apify
...
Conflicts:
ckanext/harvest/harvesters/ckanharvester.py
2015-10-28 14:34:27 +00:00
3f74c29c99
Merge branch 'master' into 157-version-three-apify
2015-10-27 17:45:27 +00:00
55245b5091
[ #158 ] PEP8/formatting.
2015-10-27 17:43:11 +00:00
2a79873855
[ #158 ] Use package search to get all datasets. Add paging search results. Store pkg_dict from search in the object rather than request it again in fetch_stage.
2015-10-27 17:33:22 +00:00
86630adab7
Merge branch 'include-exclude-org'
2015-10-27 15:52:55 +00:00
b56fae8aed
Fixes and tests
...
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
2015-10-23 17:30:28 +00:00
24574f485b
Setup harvest model in harvester tests
2015-10-23 15:43:01 +01:00
caeeace8dc
Merge branch 'master' into 157-version-three-apify
2015-10-23 14:39:48 +01:00
bc49149d5e
Merge branch 'master' into include-exclude-org
2015-10-23 14:36:53 +01:00
0c0a996b85
Merge branch 'master' into db-error
...
Conflicts:
ckanext/harvest/queue.py
2015-10-23 13:33:44 +01:00
2f4adfb338
Merge branch 'tests'
2015-10-23 13:18:15 +01:00
3c6cc55be0
Only flush keys on the current Redis database
2015-10-23 11:52:22 +01:00
fdbade465f
Merge branch 'master' into purge
2015-10-23 11:33:43 +01:00
d950b13400
Merge branch 'unique-names-improved'
2015-10-23 11:02:49 +01:00
501edffe2d
Merge branch 'master' into migration-states
2015-10-23 10:59:04 +01:00
3e4a9933ce
Remove prints.
2015-10-21 16:52:19 +00:00
dc7af5d150
Remove prints.
2015-10-21 16:38:03 +00:00
eb9aa17862
Include/exclude orgs funcationality based on work by memaldi and ross.
2015-10-21 16:33:16 +00:00
f70c16bce7
Add framework for testing harvesters. Modernize existing tests.
2015-10-21 16:26:57 +00:00
d1f84295f8
purge_queues command now has warning about impact of Redis flushall, plus add some (log) output when you run a purge.
2015-10-21 16:12:40 +00:00
6360681a8f
[ #105 ] Fix order of deletes, as agreed with @florianm.
2015-10-12 15:57:27 +01:00
82bdff2f34
Add tests
2015-10-01 17:59:17 +01:00
David Read
London App Developer
amercader
Stefan Oderbolz
Raphael Stolt