bfc9b8e0d9
[ #249 ] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE.
2016-06-09 22:11:03 +00:00
aeab60ece6
Merge pull request #247 from keitaroinc/change-db-logger-pagination-param-name
...
Change db logger pagination param name
2016-06-06 09:45:57 +01:00
92120ca47e
Change db logger pagination limit param name
2016-06-06 10:23:57 +02:00
5e1512f717
Don't reuse contexts on ckan harvester
...
Reusing the same context on all calls can lead to hard to debug failures
like
Action function organization_show did not call its auth function
In this case that was caused because the first organization/group_show
raised a NotFound so the auth audit was still in the context. When
organization/group_show was called again at the end of
organization/group_create the auth audit exception was raised.
This commit makes sure that each call has its own context.
2016-05-23 12:20:08 +01:00
16a6e9fbf6
Add tests for group creation during harvesting
2016-05-23 10:20:52 +01:00
314be8bc33
Merge branch '240-fix-groups-import' of https://github.com/keitaroinc/ckanext-harvest into keitaroinc-240-fix-groups-import
2016-05-23 09:59:32 +01:00
c16ecea7f0
reverted change in default groups validation
2016-05-20 20:15:54 +02:00
c154365371
Fixed creation/import of groups and organizations when harvesting from remote ckan instance
2016-05-20 16:38:48 +02:00
1ec2af0590
Merge pull request #230 from keitaroinc/logging-module
...
Logging module
2016-05-17 14:12:03 +01:00
cc6cb3e389
Changed default config params for the database logger
...
Added database logger test case
README updates
2016-05-16 13:15:12 +02:00
0be2c868cb
README updates
...
DBLogHandler updates
Added harvest_log table migration for existing users
Implemented database log scoping
2016-05-11 13:29:53 +02:00
623fca5f80
New syntax for pysolr connection
2016-05-10 11:14:35 +00:00
41c329c6d4
Fix syntax error in template
2016-05-09 14:36:21 +03:00
009cc57e09
Added clean-up mechanism for the harvest log
2016-05-06 18:44:02 +02:00
d372f112f0
Convert deprecated helper.
2016-04-27 15:41:17 +00:00
633a32075e
create index to harvest_object
2016-04-27 09:27:12 +03:00
3d519ce0b2
Partial fixes
2016-04-25 19:53:49 +02:00
a1968e4c63
Check if harvest_log table is populated on source creation
2016-04-12 19:28:43 +02:00
97cd64b172
Added harvest_log_list get action
2016-04-05 23:53:14 +02:00
a79ad2e325
Added basic DBLogHandler
2016-04-05 20:21:04 +02:00
afbf0f0dfe
fix unicode encode error in facet filters with scandinavian alphabets
2016-03-23 10:38:04 +02:00
15592dab6c
Merge branch '227-get-rid-of-action-not-found-error' of https://github.com/smotornyuk/ckanext-harvest into smotornyuk-227-get-rid-of-action-not-found-error
2016-03-21 13:20:42 +00:00
e1ebde7030
[ #227 ] 'harvest_source_show_status' not found
...
Added try block and try to clear action's cache in case exception raised
2016-03-21 14:35:09 +02:00
a8732553c9
Replace deprecated nav_named_link by nav_link
2016-03-02 11:44:21 +01:00
7f506913f8
Merge branch '157-version-three-apify'
2016-02-17 10:08:27 +00:00
9dfeb154eb
[ #158 ] Tone down log message
2016-02-17 10:05:57 +00:00
d8fb2ed7f6
[ #220 ] Simplify check
2016-02-17 09:30:19 +00:00
566939a655
Merge branch 'master' of https://github.com/LondonAppDev/ckanext-harvest into LondonAppDev-master
2016-02-17 09:28:30 +00:00
84b0462979
No need to go back twice
2016-02-15 15:36:02 +00:00
794fc93230
Maintain compatibility with rest-style updates
2016-02-15 15:23:39 +00:00
f22100e6c2
Merge remote-tracking branch 'origin' into 157-version-three-apify
2016-02-15 15:20:33 +00:00
bf0d1fd779
Fix name error
2016-02-15 13:54:58 +00:00
4516bfe44e
PEP8 and lint, extracted from PR158
2016-02-15 13:50:18 +00:00
49faa0ae6c
Tests for CKANHarvester._last_error_free_job
2016-02-15 13:30:28 +00:00
385b369148
Error-free jobs now include ones where an object was not modified.
2016-02-15 13:16:23 +00:00
f63140354d
Fix logic error in previous commit
2016-02-15 12:28:46 +00:00
52c071dbe9
Improved error handling. e.g. if the site it harvests just returns errors.
2016-02-15 12:10:44 +00:00
331ad84272
Deal with worry about datasets on the remote CKAN being added/removed during harvest.
2016-02-12 18:00:00 +00:00
7096b7ddf2
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
2016-02-12 16:51:26 +00:00
69ea33647e
Update plugin.py
...
Prevent the 'harvest' datasets being shown in the group and dataset search results.
2016-02-12 13:15:09 +00:00
6354ad5656
Fix source clean command on CKAN > 2.5, as related don't exist any more
2016-02-04 13:40:02 +00:00
b58cd8b38f
Merge branch 'master' into 219-missing-form
...
Conflicts:
ckanext/harvest/templates_new/source/edit.html
ckanext/harvest/templates_new/source/new.html
test-core.ini
2016-01-14 11:43:12 +00:00
73196b942b
Merge branch '214-remove-genshi'
2016-01-14 11:28:55 +00:00
5d23fab03f
[ #219 ] Support CKAN < 2.3
2016-01-14 11:23:41 +00:00
5bf0ac9b86
[ #219 ] Fix tests auth
2016-01-14 11:23:17 +00:00
1665e86065
[ #219 ] Don't use c.form to render the form
2016-01-14 10:16:45 +00:00
497dfeea02
Add test for missing form
2016-01-14 10:15:46 +00:00
e0c3316531
Add legacy import for CreateTestData
2016-01-11 22:35:11 +01:00
c141cf44fa
Swap import statements to make sure `run_test` works on CKAN 2.3
2016-01-11 22:13:25 +01:00
82f48d5afa
[ #214 ] Move ckanext/harvest/templates/templates_new to ckanext/harvest/templates
2015-12-11 15:24:45 +00:00
6b23208b2a
Merge branch '212-module-import-error'
2015-12-11 13:38:32 +00:00
d08f72ad13
Fix tests for ckan 2.2 again with amercaders suggestion.
2015-12-11 11:53:51 +00:00
6ef58addbc
Fix tests for ckan 2.2 again with amercaders suggestion.
2015-12-11 11:52:00 +00:00
f0ba0c865c
Fix tests for ckan 2.2
2015-12-10 16:36:27 +00:00
77f98d5b18
Compatibility with ckan 2.2.
2015-12-10 16:33:09 +00:00
8f54421c99
[ #214 ] ckan.legacy_templates=True is really harmless, so just warn if it is set.
2015-12-10 16:11:56 +00:00
260cd1f2b7
Merge branch 'master' of github.com:ckan/ckanext-harvest into 214-remove-genshi
2015-12-10 16:02:50 +00:00
fa1ec64cc7
Rename templates_new dir to templates.
2015-12-10 15:52:46 +00:00
8f3006f494
[ #214 ] Config for legacy templates removed.
2015-12-10 15:46:57 +00:00
41975a93d8
[ #214 ] Removed genshi templates
2015-12-10 15:32:20 +00:00
468a195a25
Merge branch 'import-guid'
2015-12-10 13:37:23 +00:00
5ff3ef9d17
Merge branch '205-inconsistent-stats'
2015-12-10 13:36:15 +00:00
9d06820bcd
Merge branch 'error_creation_moved_to_model'
2015-12-10 13:25:05 +00:00
b7a0343763
Merge branch 'abort-by-job'
2015-12-10 13:22:40 +00:00
82fe2d0e53
Merge branch 'fetch_unchanged'
2015-12-10 13:19:45 +00:00
80b82ee968
Merge branch 'factory-job'
2015-12-10 13:18:19 +00:00
52fe7fb21d
Merge branch 'named-exceptions'
2015-12-10 13:17:18 +00:00
072698b4bb
Merge branch 'validator-raise'
2015-12-10 13:16:26 +00:00
4cc39e5ef4
Merge branch 'test-guid-fix'
2015-12-10 13:15:54 +00:00
04162ce9e4
Merge branch 'munge-tag'
2015-12-10 13:15:17 +00:00
4ca4b3a2f2
Merge pull request #213 from ckan/212-module-import-error
...
[#212 ] Fixes #212 - auth for harvest_job_create was broken.
2015-12-09 15:52:37 +00:00
8b8086fe48
[ #212 ] Fixes #212 - auth for harvest_job_create was broken.
2015-12-09 15:50:05 +00:00
84fb3e3325
Add support for the new ITranslation interface
...
Move the i18n files to the expected place, load the interface if CKAN >=
2.5.0
2015-12-09 14:46:12 +00:00
031e680b6c
Add option to re-import based on guid.
2015-12-08 16:17:39 +00:00
18169c5133
[ #205 ] "detailed" removed in a couple of other benign places.
2015-12-08 10:18:36 +00:00
05d4baf040
[ #205 ] "detailed" source status is removed. Now we have lost harvest_source_for_a_dataset it is not possible to call it. And it returned the wrong keys anyway.
2015-12-08 10:14:15 +00:00
67366093fe
[ #205 ] Remove harvest_source_for_a_dataset action. It has been deprecated since Mar 6 16:54:33 2013 and returns wrong harvest stats keys.
2015-12-08 10:06:42 +00:00
07c76b0cbf
Docs & pep8
2015-12-02 16:23:54 +00:00
121e8bd918
Merge pull request #198 from ckan/catch-exceptions
...
Catch exceptions from urllib2.urlopen more comprehensively
2015-12-02 16:22:53 +00:00
c7021933a0
Move creation of errors to the model as thats a more natural home. Provide backwards compatibility.
2015-12-02 08:15:13 +00:00
b53682f267
You can abort a job by specifying the ID of the job, rather than the source. This is helpful since the "harvest run" command returns a list of still running job ids.
2015-12-02 07:59:08 +00:00
f67029b993
Hint about rerunning the import_stage.
2015-12-01 17:59:11 +00:00
b8c7d39e1a
Document existing functionality for aborting during the gather stage.
2015-12-01 17:56:24 +00:00
25301e2152
Update README from interfaces.py. PEP8. Mention HarvestObject - Package relation in interface.
2015-12-01 17:51:35 +00:00
b0780b2062
Fetch stage can also return "unchanged", same as the import stage. Used by DGU. It is useful to skip an object like this, to avoid saving the fetched content in a HarvestObject (saves disk usage).
2015-12-01 17:38:57 +00:00
414c33ac6b
Easy fix to stop test_queue.py using MockHarvester in test_queue2.py by mistake.
2015-12-01 16:55:42 +00:00
673a64a820
Merge pull request #200 from ckan/fix-error-report-status
...
Fix job stats key
2015-11-30 17:49:14 +00:00
8163ec4d39
HarvestObject factory was creating a extra field containing the job, by mistake. PEP8
2015-11-27 14:21:16 +00:00
1f947471eb
Take advantage of named exceptions.
2015-11-27 11:57:40 +00:00
798866d872
Fix name of class to enable automatic conversion of datetimes
...
when ckan/ckan#2505 got finally merged, the class name to mark datetimes
with a time part that can be automatically converted to the users
timezone, changed.
This commit makes sure this change is reflected in ckanext-harvest.
2015-11-26 01:49:11 +01:00
4fbaec0986
Improve harvester type error message. Add docstring.
2015-11-25 20:58:50 +00:00
6c3efe53df
Fix job stats key - it is "errored" not "errors" - the keys are from report_status. And lots of PEP8.
2015-11-25 20:55:32 +00:00
f0a2e9fb8e
CKAN revision API returns package ids not names (for v2 of the API). This ensures harvest guid is always the ID rather than the name.
2015-11-24 16:41:43 +00:00
392c13d828
If not revisions then we get a 404, so deal with that better.
2015-11-23 21:36:45 +00:00
4405066fab
Catch exceptions from urllib2.urlopen more comprehensively. I think 400 errors were from CKAN 0.6 or something like that - ignore now.
2015-11-23 21:26:32 +00:00
4b5014d381
Fix test for older ckan.
2015-11-23 18:27:04 +00:00
3b4daf0609
fix typo
2015-11-23 17:40:35 +00:00
bc26159fb6
tag_munge from ckan 2.2 fails the test with dashes, so use the harvest one for this ckan version.
2015-11-23 17:31:20 +00:00
52f7e0dd07
Use the ckan version of munge_tag if available, but provide a fallback for older ckans.
2015-11-23 12:48:05 +00:00
129b1a0cf5
Enable custom solution to detect existing packages
...
With this change, all harvesters that extend the base harvester have the
possibility to use the very useful create_or_update method, but still
define their own way of detecting what package is the existing one.
This is very useful for harvest sources that have no knowledge of the
CKAN internal id, but have another way of finding previous package.
2015-11-20 16:31:47 +01:00
David Read
Adrià Mercader
Petar Efnushev
Jari Voutilainen
Motornyuk Sergey
Denis Laxalde
London App Developer
Stefan Oderbolz