CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
775 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

775 Commits

Author SHA1 Message Date
David Read 673a64a820 Merge pull request #200 from ckan/fix-error-report-status 
Fix job stats key
 2015-11-30 17:49:14 +00:00
David Read 154a3f42cd Merge pull request #202 from metaodi/local-time-display 
Fix name of class to enable automatic conversion of datetimes
 2015-11-27 10:22:54 +00:00
Stefan Oderbolz 798866d872 Fix name of class to enable automatic conversion of datetimes 
when ckan/ckan#2505 got finally merged, the class name to mark datetimes
with a time part that can be automatically converted to the users
timezone, changed.
This commit makes sure this change is reflected in ckanext-harvest.
 2015-11-26 01:49:11 +01:00
David Read 6c3efe53df Fix job stats key - it is "errored" not "errors" - the keys are from report_status. And lots of PEP8. 2015-11-25 20:55:32 +00:00
David Read d1476e6f7a Merge pull request #196 from metaodi/enable-custom-update-detection 
Provide extension point to detect existing packages
 2015-11-20 16:23:38 +00:00
Stefan Oderbolz 129b1a0cf5 Enable custom solution to detect existing packages 
With this change, all harvesters that extend the base harvester have the
possibility to use the very useful create_or_update method, but still
define their own way of detecting what package is the existing one.

This is very useful for harvest sources that have no knowledge of the
CKAN internal id, but have another way of finding previous package.
 2015-11-20 16:31:47 +01:00
amercader e71cf35504 Fix queue tests 2015-11-20 14:25:31 +00:00
amercader f1ba2bcfb3 Namespace Redis keys to avoid conflicts between instances 
The `ckan.site_id` config option (or `default` if missing) is used to
namespace the Redis keys: routing key and persistance key. Consumers
will only get the relevant keys for their instance.
 2015-11-20 14:17:25 +00:00
amercader 920df684ae Merge branch 'db-error' 2015-11-20 12:29:37 +00:00
amercader ede50aa3fb Merge branch 'immediate-harvest' 2015-11-20 12:28:35 +00:00
amercader 3f42eb6ba0 Merge branch 'revision-id-fix' 
Conflicts:
	ckanext/harvest/tests/harvesters/test_ckanharvester.py
 2015-11-20 12:28:17 +00:00
amercader 1b0bb910a5 Merge branch '180-fix-modified-date' 
Conflicts:
	ckanext/harvest/harvesters/ckanharvester.py
	ckanext/harvest/tests/harvesters/test_ckanharvester.py
 2015-11-20 12:27:11 +00:00
amercader 36babfb26a Merge branch 'yhteentoimivuuspalvelut-job-reporting-fixes' 2015-11-20 12:20:32 +00:00
David Read 60c4371df4 Add "not modified" to the stats always returned. 2015-11-17 12:45:00 +00:00
David Read ae7c500745 Merge branch 'master' into yhteentoimivuuspalvelut-job-reporting-fixes 2015-11-17 12:35:59 +00:00
David Read cc79eeb942 Merge pull request #192 from metaodi/consistent-api-response 
Make sure all status keys are always returned
 2015-11-17 12:34:27 +00:00
Stefan Oderbolz 8e02aedc65 Fix tests 2015-11-17 13:29:25 +01:00
Stefan Oderbolz f9b87fff0c Make sure all possible status are always returned 
It makes it harded to parse the API response if you always have to check
if 'deleted' is set and if what value it has. I think to simply return 0
for all status value is good practice.
 2015-11-17 11:43:11 +01:00
David Read 5f96346487 Merge pull request #191 from raphaelstolt/master 
Catch JSONDecodeError when no JSON content
 2015-11-16 11:42:32 +00:00
Raphael Stolt 084723abb7 Catch JSONDecodeError when no JSON content 2015-11-16 10:59:18 +01:00
David Read 6415cdd740 Merge pull request #190 from ckan/metaodi-harvest-source-patch 
Metaodi harvest source patch
 2015-11-13 13:52:09 +00:00
David Read c0a865e64e Revert ok_ - makes it slightly less readable for little benefit. 2015-11-13 13:45:56 +00:00
David Read 42ab55cb6d No longer need uuid since we clear db between tests now. Added ignore_missing because of occasional failures. 2015-11-13 13:32:55 +00:00
David Read b150b50887 Move the SkipTest to include inherited tests too. 2015-11-13 12:44:27 +00:00
David Read 01a4bfd314 Patch test should skip if ckan version is wrong, rather than ignore all exceptions from posts of all tests. Remove FunctionalTestBaseWithoutClearBetweenTests now the tests are modernized. 2015-11-13 12:33:18 +00:00
David Read 1288a4d9e7 Reflow text to 79 char width. Warning not necessary with an exception I think. 2015-11-13 12:32:13 +00:00
Stefan Oderbolz 0ce3748153 Do not use ActionError as this does not yet exist in CKAN 2.2 2015-11-13 12:01:19 +01:00
Stefan Oderbolz c33c6e8c13 Raise an error instead of falling back to harvest_source_update 
As the behaviour of *_patch is clearly different from the *_update we
should raise an error if this action is called on a CKAN instance, where
the action is not available.
 2015-11-13 11:41:53 +01:00
Stefan Oderbolz 73aecb7f12 Add datastore_default user due to PR #2234 from ckan 2015-11-11 20:06:12 +01:00
Stefan Oderbolz 644fa49dd4 Make tests independent from cls.sysadmin 
Generate unique harvest sources
 2015-11-11 19:49:49 +01:00
Stefan Oderbolz ffca5cc3da Use new factory style for fixtures 
- Remove default_source_dict from tests
- Replace setup_class with constructor
- Create mixin for harvest source fixture
- Replace assert with assert_equal where appropriate
- Replace assert with ok_
- Remove dependency to global SOURCE_DICT
- General refactoring of tests
 2015-11-11 19:07:59 +01:00
Stefan Oderbolz 136fcb87d5 Make sure package_patch has a fallback for package_update on CKAN < 2.3 2015-11-11 11:37:23 +01:00
Stefan Oderbolz 359da2eb69 Add test class for harvest_source_patch 2015-11-11 11:34:01 +01:00
Stefan Oderbolz 3f09010039 Add harvest_source_patch to API 2015-11-11 05:39:29 +01:00
David Read 679ed421e9 Merge branch 'master' of github.com:ckan/ckanext-harvest into immediate-harvest 
Conflicts:
	ckanext/harvest/logic/action/update.py
 2015-11-04 09:44:05 +00:00
David Read f0207ad38f Merge branch 'master' of github.com:ckan/ckanext-harvest into yhteentoimivuuspalvelut-job-reporting-fixes 2015-11-04 09:36:55 +00:00
David Read cbe9b40e66 Merge branch 'master' of github.com:ckan/ckanext-harvest into revision-id-fix 2015-11-04 09:36:08 +00:00
David Read 46e7e182f5 Merge branch 'master' of github.com:ckan/ckanext-harvest into 180-fix-modified-date 2015-11-04 09:35:23 +00:00
David Read 72ef3e4826 Merge pull request #184 from ckan/LondonAppDev-master 
Allow Harvest Sources with duplicate URLs if they have unique configs
 2015-11-03 23:41:47 +00:00
David Read f9da3654f8 [#184] Fix tests for older ckan versions. 2015-11-03 23:27:52 +00:00
David Read 5fba056c59 [#184] Add tests 2015-11-03 23:19:05 +00:00
David Read 6159d900b3 Merge branch 'master' into LondonAppDev-master 2015-11-03 22:26:07 +00:00
David Read 77e5b89a01 Blank line needed. 2015-11-03 22:23:04 +00:00
David Read 3d35814f3a Merge pull request #183 from ckan/pep8-from-174 
A few more PEP8 changes taken from #174
 2015-11-03 22:14:37 +00:00
David Read 1ccb0dae44 A few more PEP8 changes taken from #174. 2015-11-03 22:12:49 +00:00
David Read 8c1f7619cb Fix code style to be more ckan-like whilst still pep8. 2015-11-03 22:08:46 +00:00
David Read 20531c0dda Merge branch 'master' of github.com:ckan/ckanext-harvest into LondonAppDev-master 
Conflicts:
	ckanext/harvest/logic/action/update.py
	ckanext/harvest/logic/validators.py
 2015-11-03 22:02:49 +00:00
David Read 2ff017d9de Merge pull request #182 from ckan/pep8-from-174 
Pep8 from 174
 2015-11-03 21:59:29 +00:00
David Read 10685badb5 PEP8 based on #174 
Conflicts:
	ckanext/harvest/logic/action/delete.py
	ckanext/harvest/logic/action/update.py
	ckanext/harvest/logic/validators.py
 2015-11-03 21:56:06 +00:00
David Read 5a5260ff0b Add test for harvest_source_clear since the PEP8 changes were quite a lot there. 2015-11-03 21:42:39 +00:00