CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,274 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

1274 Commits

Author SHA1 Message Date
amercader 3742ea61de [#369] Test master 2019-10-18 14:41:50 +02:00
amercader d18079fe40 [#369] Verbose tests 2019-10-18 14:41:38 +02:00
amercader 12196268d8 Merge branch 'master' into 369-broken-tests 2019-10-18 14:40:12 +02:00
Adrià Mercader 30e8ec4395
Merge pull request #373 from wood-chris/patch-1 
Updated README.rst
 2019-10-18 14:25:44 +02:00
amercader 3be2d6acd0 Update CHANGELOG 2019-10-03 11:52:40 +02:00
amercader 3793480a91 Fix Flake8 warnings 2019-10-03 11:50:15 +02:00
amercader 57718973ce Add exit-zero to the flake8 command for pep8 2019-10-03 11:39:24 +02:00
amercader 5ffe6d4bfa Resend awaiting objects in the DB that are not present in Redis 
This happens when there is an ongoing harvest job and the keys on redis
are deleted for some reason (eg not persistent container)
 2019-10-03 11:32:48 +02:00
amercader 5ad6d866ed Mark early errored job as finished 
Otherwise it's displayed as Running forever
 2019-10-03 11:22:23 +02:00
amercader 8e862c8f04 Fix wrong toolkit reference 2019-10-03 11:21:44 +02:00
Adrià Mercader 5c9c093803
Merge pull request #376 from ckan/remove-old-migrations 
Remove old database migrations that targeted ancient versions
 2019-10-02 15:58:00 +02:00
amercader 72591e83ad Add changelog notice 2019-10-01 12:08:29 +02:00
amercader 7ac70cd58e Remove old database migrations that targeted ancient versions 
For this version it is safe to assume that is these migrations are no
longer needed. They could introduce hard to track errors as part of the
startup work they did.
 2019-10-01 12:04:16 +02:00
amercader 819706ae5d Add further notes on extending ckan harvester 2019-08-26 13:48:09 +02:00
amercader 30f19713fa Add hook to extend the package dict in CKAN harvester 2019-08-26 13:38:30 +02:00
amercader 51bf79a626 Revert back to StrictRedis client 2019-08-26 12:26:05 +02:00
amercader 8b2f2f9a24 Merge branch 'default-to-ckan.redis.url' 2019-08-23 15:46:37 +02:00
amercader 18e8f5ce13 Merge branch 'master' into 369-broken-tests 2019-08-23 14:40:50 +02:00
Adrià Mercader 25775ffa7b
Merge pull request #358 from ckan/ckantoolkit 
Use ckantoolkit to clean up imports
 2019-08-23 14:39:54 +02:00
amercader a8b1b211e1 [#369] [#369] Remove dodgy patching 2019-08-23 14:21:24 +02:00
amercader b1b284a810 Default to the ckan.redis.url setting 
99.9% of cases you probably want to use the same redis instance that
CKAN core uses so it makes sense to default to that setting. If it's not
set if will default to the hardcoded values (localhost:6379/0) which is
the same default as CKAN core. If there is an existing legacy setting
(ckan.harvest.mq.hostname) those will be used.
 2019-08-23 13:34:17 +02:00
Chris Wood 17868b28dc
Updated README.rst 
Updated README.rst to clarify that dev-requirements needs to be installed for ``run_test``
 2019-07-31 17:30:27 +01:00
kentsanggds 0ba5e11cc8
Merge pull request #7 from alphagov/add-force_import 
Add force import
 2019-07-24 09:56:17 +01:00
Ken Tsang 2d1d99486e Update README for `force-import` 2019-07-24 09:49:43 +01:00
Ken Tsang b625f642dc Add force-import to reharvest datasets 
- datasets that were created incorrectly because of bugs in the system would not otherwise be reharvested  unless the harvest source changed.
 2019-07-24 09:49:43 +01:00
Stefan Oderbolz b8cf47bb91
Merge pull request #365 from usingsky/fix-SAWarning 
Fix SAWarning when tracking update from ckan
 2019-07-09 07:58:49 +02:00
amercader 62d1843962 [#369] Fix conditional imports 2019-07-02 12:27:19 +02:00
Stefan Oderbolz 9eee08b034
Merge pull request #367 from GovDataOfficial/fix-broken-abort-job-in-ui 
Fix abort job from ui
 2019-07-01 17:16:20 +02:00
seitenbau-govdata b206c4cf36 Fix abort job from ui 
* Fix abort job from ui which was broken by commit
ed288dca63
 2019-07-01 12:19:36 +02:00
kentsanggds 5671bdf932
Merge pull request #6 from alphagov/comment-out-harvest_source-stats 
Comment out harvest_source statistics
 2019-06-14 15:14:15 +01:00
Ken Tsang 937e27d24c Comment out harvest_source statistics 2019-06-14 13:29:53 +01:00
Jinil Lee 7e006e8c9e Fix SAWarning when tracking update from ckan 
Warnings emitted when coercing full SQL fragments into text()
 2019-06-01 18:08:11 +09:00
kentsanggds 4071faf221
Merge pull request #5 from alphagov/handle-redis-errors 
Add try except around Redis set to handle corrupt redis data
 2019-05-09 10:08:45 +01:00
Ken Tsang 0cbb3579a9 Update Redis exception to drop stack trace 
Full stack trace is probably not needed as the Redis data corruption is probably happening somewhere else, the error log should make it easier to investigate it when it does happen.
 2019-05-08 11:26:05 +01:00
Ken Tsang edcf80c944 Add try except around Redis set to handle corrupt redis data 
- prevents bad data from stopping harvest processing
- log the error for investigation, full stack trace output as part of log
 2019-05-03 13:13:31 +01:00
Edward Kerry fb0cf32546
Merge pull request #4 from alphagov/add-not-modified-status 
Add `Not modified` statistic report for unchanged imports.
 2019-04-15 13:17:28 +01:00
Ken Tsang 4772bb08b7 Don't send the job to the gather queue when running the paster run_test command 2019-04-12 11:32:45 +01:00
Ken Tsang 93a839efce Use == rather than `is` for string comparison 2019-04-12 11:31:04 +01:00
Ken Tsang a9e32e755a Add not modified status to job details and list 2019-04-12 11:30:31 +01:00
David Read 3a72337f1e
Merge pull request #359 from ckan/fix-model-codestyle 
Code style changes to model and plugin
 2019-03-18 19:55:31 +00:00
Stefan Oderbolz 5335d66316 Changelog entry for ckantoolkit 2019-03-18 17:48:25 +01:00
Stefan Oderbolz 8be627a454 Use ckantoolkit to avoid imports from new_test 2019-03-18 17:48:25 +01:00
Stefan Oderbolz 2c8a909c66
Add #357 to CHANGELOG 2019-03-18 14:22:46 +01:00
Stefan Oderbolz ecf578648b
Merge pull request #357 from alphagov/add-test-error-raise 
Fix harvest request exceptions
 2019-03-18 14:21:20 +01:00
Ken Tsang dfc147e6d7 Fixed codestyle in test_ckanharvester.py 2019-03-18 11:16:50 +00:00
Ken Tsang 95e6e934ac Add tests for raised errors during _get_content 2019-03-18 10:53:57 +00:00
Ken Tsang 1b609eeb60 import HTTPError to ckanharvester.py 2019-03-18 10:53:57 +00:00
Bruce Bolt 730b476c08 Fix harvest requests exceptions 2019-03-18 10:53:57 +00:00
Stefan Oderbolz 487e06fc24 Add flake8 config to setup.cfg 
if the config is set, the output of `flake8` locally is correct without
    specifiying and further options.
 2019-03-17 22:59:53 +01:00
Stefan Oderbolz ef3dc4a46a Update code style in model and plugin 
This is based on the review feedback in #354
 2019-03-17 22:59:15 +01:00