b625f642dc
Add force-import to reharvest datasets
...
- datasets that were created incorrectly because of bugs in the system would not otherwise be reharvested unless the harvest source changed.
2019-07-24 09:49:43 +01:00
62d1843962
[ #369 ] Fix conditional imports
2019-07-02 12:27:19 +02:00
0cbb3579a9
Update Redis exception to drop stack trace
...
Full stack trace is probably not needed as the Redis data corruption is probably happening somewhere else, the error log should make it easier to investigate it when it does happen.
2019-05-08 11:26:05 +01:00
edcf80c944
Add try except around Redis set to handle corrupt redis data
...
- prevents bad data from stopping harvest processing
- log the error for investigation, full stack trace output as part of log
2019-05-03 13:13:31 +01:00
8be627a454
Use ckantoolkit to avoid imports from new_test
2019-03-18 17:48:25 +01:00
dfc147e6d7
Fixed codestyle in test_ckanharvester.py
2019-03-18 11:16:50 +00:00
95e6e934ac
Add tests for raised errors during _get_content
2019-03-18 10:53:57 +00:00
2fbac74c29
Fix importing assert_in
2019-03-06 14:12:27 +02:00
98364723d7
Apply flake8
2019-03-06 13:19:05 +02:00
204b1f301a
One more fix for 2.9
2018-10-26 14:11:20 +02:00
ed288dca63
Reduce usage of c vars, add more tests
2018-10-26 13:38:09 +02:00
7bad691415
Add tests and docs for include/exclude groups
2018-10-25 13:04:37 +02:00
cd6f75251f
test if error-mails are only sent to respective org’s admins
2018-06-14 14:45:26 +02:00
a6cb710127
Merge branch 'error-mail' of https://github.com/stefina/ckanext-harvest into stefina-error-mail
2018-06-13 14:17:42 +02:00
e1e6b44a44
extend TestHarvestErrorMail from FunctionalTestBase
...
This prevents failing tests for older versions.
2018-06-13 13:31:03 +02:00
ad9e513c0d
mock ckan.lib.mailer
2018-06-13 13:12:14 +02:00
00911a6924
test harvest-error-mail
2018-06-13 10:53:36 +02:00
fff9800266
Fix print statements to be Py3 friendly
...
Fixes the print statements where they were doung to use print as a
function, and also to switch to .format() rather than %.
Also contains some flake8 whitespace changes as I suspect making this
PEP8ish will take several PRs.
2018-06-08 09:53:13 +01:00
431d202d4e
Add tests and documentation for dataset name suffix config
2018-06-01 21:45:29 +02:00
717fdb35dd
move _last_error_free_job from CKANHarvester to HarvesterBase
2017-11-10 12:19:25 +01:00
d1dd4eb227
303: fix clean_tags with tags dict (fixes requested by review)
2017-11-08 13:46:44 +01:00
881a9581d5
Fix url_for depending tests
2017-07-27 16:09:00 +01:00
cc438786de
More explicit checks of the exception thrown when checking harvest config. Also the default_groups test was checking the wrong thing completely.
2017-05-02 20:39:49 +00:00
e7c03855ca
Avoid the "# dont use factory because it looks for the existing source" by copying the SOURCE_DICT each time, rather than letting tests edit the master copy.
2016-11-16 15:40:44 +01:00
7f76f60ec3
Fixed variable name
2016-11-16 00:34:07 +01:00
d511663038
Sort lists for assert
2016-11-16 00:25:19 +01:00
f68bf323f0
Using test class wide unique harvest source url
...
Using test class wide unique harvest source url, because in a test created objects are still present in following tests.
2016-11-15 22:28:37 +01:00
d01a86680e
Fix creating different harvest sources
...
Fix creating different harvest sources. Different harvest sources can't be created with factory.
2016-11-15 21:56:57 +01:00
096e746c81
Fixed HarvestSourceObj argument
2016-11-15 21:23:20 +01:00
8d5ff4b4ef
Fixed harvest_sources_job_history_clear test
...
Fixed harvest_sources_job_history_clear test by creating different harvest sources.
2016-11-15 21:09:42 +01:00
cf1cfcca48
Fixed using property of object
2016-11-15 15:50:03 +01:00
1acab98026
Added tests for clearsource history command
2016-11-15 15:37:26 +01:00
2602de9094
[ #257 ] Purge only our own Redis data.
...
Previously purging the queue on the Redis backend would clear the whole
database, making it hard to share the same database with other parts of
CKAN. With this commit, only the keys that belong to ckanext-harvest and
the current CKAN instance are purged.
2016-07-20 16:24:13 +02:00
18a506a112
[ #249 ] Add test for default_extras.
2016-06-10 09:51:17 +00:00
f1742fb51a
Fix default_groups. It accepted a list of package_name/ids and was trying to add this to the package, but the package needs a dict. Added test.
2016-06-10 09:16:32 +00:00
bfc9b8e0d9
[ #249 ] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE.
2016-06-09 22:11:03 +00:00
16a6e9fbf6
Add tests for group creation during harvesting
2016-05-23 10:20:52 +01:00
cc6cb3e389
Changed default config params for the database logger
...
Added database logger test case
README updates
2016-05-16 13:15:12 +02:00
3d519ce0b2
Partial fixes
2016-04-25 19:53:49 +02:00
a1968e4c63
Check if harvest_log table is populated on source creation
2016-04-12 19:28:43 +02:00
49faa0ae6c
Tests for CKANHarvester._last_error_free_job
2016-02-15 13:30:28 +00:00
385b369148
Error-free jobs now include ones where an object was not modified.
2016-02-15 13:16:23 +00:00
52c071dbe9
Improved error handling. e.g. if the site it harvests just returns errors.
2016-02-15 12:10:44 +00:00
331ad84272
Deal with worry about datasets on the remote CKAN being added/removed during harvest.
2016-02-12 18:00:00 +00:00
7096b7ddf2
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
2016-02-12 16:51:26 +00:00
5bf0ac9b86
[ #219 ] Fix tests auth
2016-01-14 11:23:17 +00:00
497dfeea02
Add test for missing form
2016-01-14 10:15:46 +00:00
e0c3316531
Add legacy import for CreateTestData
2016-01-11 22:35:11 +01:00
c141cf44fa
Swap import statements to make sure `run_test` works on CKAN 2.3
2016-01-11 22:13:25 +01:00
6b23208b2a
Merge branch '212-module-import-error'
2015-12-11 13:38:32 +00:00
6ef58addbc
Fix tests for ckan 2.2 again with amercaders suggestion.
2015-12-11 11:52:00 +00:00
f0ba0c865c
Fix tests for ckan 2.2
2015-12-10 16:36:27 +00:00
5ff3ef9d17
Merge branch '205-inconsistent-stats'
2015-12-10 13:36:15 +00:00
82fe2d0e53
Merge branch 'fetch_unchanged'
2015-12-10 13:19:45 +00:00
80b82ee968
Merge branch 'factory-job'
2015-12-10 13:18:19 +00:00
4cc39e5ef4
Merge branch 'test-guid-fix'
2015-12-10 13:15:54 +00:00
04162ce9e4
Merge branch 'munge-tag'
2015-12-10 13:15:17 +00:00
8b8086fe48
[ #212 ] Fixes #212 - auth for harvest_job_create was broken.
2015-12-09 15:50:05 +00:00
18169c5133
[ #205 ] "detailed" removed in a couple of other benign places.
2015-12-08 10:18:36 +00:00
b0780b2062
Fetch stage can also return "unchanged", same as the import stage. Used by DGU. It is useful to skip an object like this, to avoid saving the fetched content in a HarvestObject (saves disk usage).
2015-12-01 17:38:57 +00:00
414c33ac6b
Easy fix to stop test_queue.py using MockHarvester in test_queue2.py by mistake.
2015-12-01 16:55:42 +00:00
8163ec4d39
HarvestObject factory was creating a extra field containing the job, by mistake. PEP8
2015-11-27 14:21:16 +00:00
6c3efe53df
Fix job stats key - it is "errored" not "errors" - the keys are from report_status. And lots of PEP8.
2015-11-25 20:55:32 +00:00
f0a2e9fb8e
CKAN revision API returns package ids not names (for v2 of the API). This ensures harvest guid is always the ID rather than the name.
2015-11-24 16:41:43 +00:00
52f7e0dd07
Use the ckan version of munge_tag if available, but provide a fallback for older ckans.
2015-11-23 12:48:05 +00:00
e71cf35504
Fix queue tests
2015-11-20 14:25:31 +00:00
ede50aa3fb
Merge branch 'immediate-harvest'
2015-11-20 12:28:35 +00:00
3f42eb6ba0
Merge branch 'revision-id-fix'
...
Conflicts:
ckanext/harvest/tests/harvesters/test_ckanharvester.py
2015-11-20 12:28:17 +00:00
60c4371df4
Add "not modified" to the stats always returned.
2015-11-17 12:45:00 +00:00
ae7c500745
Merge branch 'master' into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-17 12:35:59 +00:00
8e02aedc65
Fix tests
2015-11-17 13:29:25 +01:00
c0a865e64e
Revert ok_ - makes it slightly less readable for little benefit.
2015-11-13 13:45:56 +00:00
42ab55cb6d
No longer need uuid since we clear db between tests now. Added ignore_missing because of occasional failures.
2015-11-13 13:32:55 +00:00
b150b50887
Move the SkipTest to include inherited tests too.
2015-11-13 12:44:27 +00:00
01a4bfd314
Patch test should skip if ckan version is wrong, rather than ignore all exceptions from posts of all tests. Remove FunctionalTestBaseWithoutClearBetweenTests now the tests are modernized.
2015-11-13 12:33:18 +00:00
0ce3748153
Do not use ActionError as this does not yet exist in CKAN 2.2
2015-11-13 12:01:19 +01:00
c33c6e8c13
Raise an error instead of falling back to harvest_source_update
...
As the behaviour of *_patch is clearly different from the *_update we
should raise an error if this action is called on a CKAN instance, where
the action is not available.
2015-11-13 11:41:53 +01:00
644fa49dd4
Make tests independent from cls.sysadmin
...
Generate unique harvest sources
2015-11-11 19:49:49 +01:00
ffca5cc3da
Use new factory style for fixtures
...
- Remove default_source_dict from tests
- Replace setup_class with constructor
- Create mixin for harvest source fixture
- Replace assert with assert_equal where appropriate
- Replace assert with ok_
- Remove dependency to global SOURCE_DICT
- General refactoring of tests
2015-11-11 19:07:59 +01:00
136fcb87d5
Make sure package_patch has a fallback for package_update on CKAN < 2.3
2015-11-11 11:37:23 +01:00
359da2eb69
Add test class for harvest_source_patch
2015-11-11 11:34:01 +01:00
735ab3e286
[ #157 ] Try to fix test for ckan 2.2 - cf 91afc0e928
2015-11-04 11:37:03 +00:00
a0742d69b0
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
...
Conflicts:
ckanext/harvest/logic/action/update.py
2015-11-04 09:50:00 +00:00
679ed421e9
Merge branch 'master' of github.com:ckan/ckanext-harvest into immediate-harvest
...
Conflicts:
ckanext/harvest/logic/action/update.py
2015-11-04 09:44:05 +00:00
f0207ad38f
Merge branch 'master' of github.com:ckan/ckanext-harvest into yhteentoimivuuspalvelut-job-reporting-fixes
2015-11-04 09:36:55 +00:00
cbe9b40e66
Merge branch 'master' of github.com:ckan/ckanext-harvest into revision-id-fix
2015-11-04 09:36:08 +00:00
f9da3654f8
[ #184 ] Fix tests for older ckan versions.
2015-11-03 23:27:52 +00:00
5fba056c59
[ #184 ] Add tests
2015-11-03 23:19:05 +00:00
5a5260ff0b
Add test for harvest_source_clear since the PEP8 changes were quite a lot there.
2015-11-03 21:42:39 +00:00
91afc0e928
[ #178 ] Fix test for ckan 2.2, which does not suffer problem #180 .
2015-11-03 10:01:47 +00:00
f4f124c181
[ #178 ] Fix resouce_revision_id_fkey error. Fixes #178 .
2015-11-03 07:38:01 +00:00
c7fac36c1c
[ #107 ] "unchanged" response tested and related fixes
...
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
2015-11-03 00:22:53 +00:00
24415844e0
[ #158 ] Fix revision_id problem in second harvest.
2015-11-02 18:13:29 +00:00
1a680f3fd3
[ #158 ] Fix spaces encoding broken in previous merge. Tested with data.gov.uk.
2015-10-29 17:31:04 +00:00
f1d2d5fdc4
[ #111 ] Run jobs straight away.
2015-10-28 21:58:36 +00:00
e2ab9e58e7
Merge remote-tracking branch 'origin/master' into 157-version-three-apify
...
Conflicts:
ckanext/harvest/harvesters/ckanharvester.py
2015-10-28 14:34:27 +00:00
3f74c29c99
Merge branch 'master' into 157-version-three-apify
2015-10-27 17:45:27 +00:00
2a79873855
[ #158 ] Use package search to get all datasets. Add paging search results. Store pkg_dict from search in the object rather than request it again in fetch_stage.
2015-10-27 17:33:22 +00:00
86630adab7
Merge branch 'include-exclude-org'
2015-10-27 15:52:55 +00:00
b56fae8aed
Fixes and tests
...
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
2015-10-23 17:30:28 +00:00
Ken Tsang
amercader
Stefan Oderbolz
Jari Voutilainen
Stefanie Taepke
Ross Jones
seitenbau-govdata
Knud Möller
etj
David Read
Florian Brucker
Petar Efnushev