cf1cfcca48
Fixed using property of object
2016-11-15 15:50:03 +01:00
1acab98026
Added tests for clearsource history command
2016-11-15 15:37:26 +01:00
af0e1712b9
Changed filter query for reading harvest sources
...
Changed filter query for reading harvest sources in according to the code in /ckanext/harvest/plugin.py.
2016-11-15 15:04:01 +01:00
e8570b9e50
Add clearsource history command
2016-11-15 15:04:01 +01:00
f2e1dc512c
Change 'redirect' calls to 'h.redirect_to'
...
ckan.lib.base.redirect was removed in [1], guidance is to always use h.redirect_to instead. Manifested as 503 errors in the harvest UI against an install of CKAN master branch.
[1] 34f3f18e88
2016-09-18 03:23:29 -07:00
4ee772f064
Use toolkit.asbool to parse given boolean parameter
2016-08-31 09:13:36 +02:00
8b081e2868
Add new parameter return_last_job_status to harvest_source_list
...
In order to get a quick overview over successful/failed harvesters, a
call to harvest_source_list with return_last_job_status=true can be used
to get this information.
By default return_last_job_status is False, and hence the extra
resources to grab this information is not wasted for every call, but
only if the client requests is explicitly.
The original 'status' field stays as-is, this PR introduces a new field
called 'last_job_status' to return this information.
The returned information is gather by a call to
harvest_source_status_show.
2016-08-25 17:56:51 +02:00
2602de9094
[ #257 ] Purge only our own Redis data.
...
Previously purging the queue on the Redis backend would clear the whole
database, making it hard to share the same database with other parts of
CKAN. With this commit, only the keys that belong to ckanext-harvest and
the current CKAN instance are purged.
2016-07-20 16:24:13 +02:00
78933fb775
[ #253 ] Fix default_groups by saving the dicts to the config object, since saving it to the harvester object doesnt work in the real world. This is a lot more efficient than doing group_show for every dataset imported.
2016-06-27 12:01:35 +01:00
e8f539a45e
Don't let the user specify mutually exclusive configuration options:
...
- organizations_filter_include
- organizations_filter_exclude
2016-06-14 11:35:38 -03:00
18a506a112
[ #249 ] Add test for default_extras.
2016-06-10 09:51:17 +00:00
f1742fb51a
Fix default_groups. It accepted a list of package_name/ids and was trying to add this to the package, but the package needs a dict. Added test.
2016-06-10 09:16:32 +00:00
bfc9b8e0d9
[ #249 ] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE.
2016-06-09 22:11:03 +00:00
aeab60ece6
Merge pull request #247 from keitaroinc/change-db-logger-pagination-param-name
...
Change db logger pagination param name
2016-06-06 09:45:57 +01:00
92120ca47e
Change db logger pagination limit param name
2016-06-06 10:23:57 +02:00
5e1512f717
Don't reuse contexts on ckan harvester
...
Reusing the same context on all calls can lead to hard to debug failures
like
Action function organization_show did not call its auth function
In this case that was caused because the first organization/group_show
raised a NotFound so the auth audit was still in the context. When
organization/group_show was called again at the end of
organization/group_create the auth audit exception was raised.
This commit makes sure that each call has its own context.
2016-05-23 12:20:08 +01:00
16a6e9fbf6
Add tests for group creation during harvesting
2016-05-23 10:20:52 +01:00
314be8bc33
Merge branch '240-fix-groups-import' of https://github.com/keitaroinc/ckanext-harvest into keitaroinc-240-fix-groups-import
2016-05-23 09:59:32 +01:00
c16ecea7f0
reverted change in default groups validation
2016-05-20 20:15:54 +02:00
c154365371
Fixed creation/import of groups and organizations when harvesting from remote ckan instance
2016-05-20 16:38:48 +02:00
1ec2af0590
Merge pull request #230 from keitaroinc/logging-module
...
Logging module
2016-05-17 14:12:03 +01:00
cc6cb3e389
Changed default config params for the database logger
...
Added database logger test case
README updates
2016-05-16 13:15:12 +02:00
0be2c868cb
README updates
...
DBLogHandler updates
Added harvest_log table migration for existing users
Implemented database log scoping
2016-05-11 13:29:53 +02:00
623fca5f80
New syntax for pysolr connection
2016-05-10 11:14:35 +00:00
41c329c6d4
Fix syntax error in template
2016-05-09 14:36:21 +03:00
009cc57e09
Added clean-up mechanism for the harvest log
2016-05-06 18:44:02 +02:00
d372f112f0
Convert deprecated helper.
2016-04-27 15:41:17 +00:00
633a32075e
create index to harvest_object
2016-04-27 09:27:12 +03:00
3d519ce0b2
Partial fixes
2016-04-25 19:53:49 +02:00
a1968e4c63
Check if harvest_log table is populated on source creation
2016-04-12 19:28:43 +02:00
97cd64b172
Added harvest_log_list get action
2016-04-05 23:53:14 +02:00
a79ad2e325
Added basic DBLogHandler
2016-04-05 20:21:04 +02:00
afbf0f0dfe
fix unicode encode error in facet filters with scandinavian alphabets
2016-03-23 10:38:04 +02:00
15592dab6c
Merge branch '227-get-rid-of-action-not-found-error' of https://github.com/smotornyuk/ckanext-harvest into smotornyuk-227-get-rid-of-action-not-found-error
2016-03-21 13:20:42 +00:00
e1ebde7030
[ #227 ] 'harvest_source_show_status' not found
...
Added try block and try to clear action's cache in case exception raised
2016-03-21 14:35:09 +02:00
a8732553c9
Replace deprecated nav_named_link by nav_link
2016-03-02 11:44:21 +01:00
7f506913f8
Merge branch '157-version-three-apify'
2016-02-17 10:08:27 +00:00
9dfeb154eb
[ #158 ] Tone down log message
2016-02-17 10:05:57 +00:00
d8fb2ed7f6
[ #220 ] Simplify check
2016-02-17 09:30:19 +00:00
566939a655
Merge branch 'master' of https://github.com/LondonAppDev/ckanext-harvest into LondonAppDev-master
2016-02-17 09:28:30 +00:00
84b0462979
No need to go back twice
2016-02-15 15:36:02 +00:00
794fc93230
Maintain compatibility with rest-style updates
2016-02-15 15:23:39 +00:00
f22100e6c2
Merge remote-tracking branch 'origin' into 157-version-three-apify
2016-02-15 15:20:33 +00:00
bf0d1fd779
Fix name error
2016-02-15 13:54:58 +00:00
4516bfe44e
PEP8 and lint, extracted from PR158
2016-02-15 13:50:18 +00:00
49faa0ae6c
Tests for CKANHarvester._last_error_free_job
2016-02-15 13:30:28 +00:00
385b369148
Error-free jobs now include ones where an object was not modified.
2016-02-15 13:16:23 +00:00
f63140354d
Fix logic error in previous commit
2016-02-15 12:28:46 +00:00
52c071dbe9
Improved error handling. e.g. if the site it harvests just returns errors.
2016-02-15 12:10:44 +00:00
331ad84272
Deal with worry about datasets on the remote CKAN being added/removed during harvest.
2016-02-12 18:00:00 +00:00
rnoerenberg
Raphael Stolt
Andy Gross
Stefan Oderbolz
Florian Brucker
David Read
Jardel Weyrich
Adrià Mercader
Petar Efnushev
Jari Voutilainen
Motornyuk Sergey
Denis Laxalde