1714e55110
simplify harvest_clear queries so they do not lock on big db
2013-04-30 13:59:23 +01:00
a2b8ab1994
make harvest source clear not create table
2013-04-30 12:40:46 +01:00
9041f3f3ad
Changes in Redis conusmer to make tests work
2013-04-22 18:08:19 +01:00
70dfee1a36
Update queue tests
2013-04-22 17:56:11 +01:00
dcfd201cdd
[ #32 ] redis queue support
2013-04-21 17:04:57 +01:00
0ce59a29b6
delete insead of update harvest objects when error
2013-04-12 12:32:33 +01:00
7d7657f94a
make gather phase as finished if there is an error
2013-04-12 10:35:08 +01:00
bd761498f0
make sure config dict is not jsonified if it contains an error
2013-04-08 18:52:36 +01:00
eaebeb4e6e
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-04-08 13:25:33 +01:00
5414b6c08d
Merge branch '29-new-idataset-form' into release-v2.0
2013-04-08 13:23:41 +01:00
66ff773f99
remove previous commit import
2013-03-29 12:47:14 +00:00
3ac065f0f0
fix package_schema import
2013-03-29 01:17:24 +00:00
cb8b808274
sanity check that harvest source id matches harvest dataset id
...
remove author_email, license_id, maintainer_email, maintainer and
author from pacakge_dict, these were not actually necessary
2013-03-29 00:59:20 +00:00
99bd17401c
Handle wrong JSON in harvest_source_extra_validator
2013-03-28 16:19:16 +00:00
a9b8be8f01
harvest source index clear
2013-03-28 15:36:44 +00:00
95ebb5bbf3
[ #29 ] Remove check_data_dict ✨
2013-03-28 15:01:21 +00:00
fbc8ecde97
[ #29 ] Fix some imports on actions and plugin
2013-03-28 15:00:44 +00:00
c754479014
#29 make new idatasets form work with harvest source form
2013-03-25 17:38:07 +00:00
548d3c1c2a
fix validation issue on db upgrade
2013-03-25 12:02:07 +00:00
b5a697ec87
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-25 11:58:31 +00:00
0b5c3c608a
catch and raise gather exception, acking the message
2013-03-25 11:57:57 +00:00
438ba672e2
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-03-25 11:44:37 +00:00
845c9927a8
add harvest source clear
2013-03-25 11:39:00 +00:00
d518b6709a
[ #27 ] fix package_list_for_source for unowned data sources
2013-03-21 15:59:22 +00:00
7bff041568
[ #25 ] Further tweaks on helpers texts
2013-03-21 13:47:23 +00:00
4d0dd9a4d3
[ #25 ] Small copy tweak to confirmation dialog
2013-03-21 12:14:33 +00:00
78bde974b9
[ #25 ] Adds confirmation dialog to reharvest button
2013-03-21 10:56:39 +00:00
3197162fe6
[ #25 ] Changed 'Refresh' to 'Reharvest' on button
2013-03-21 10:36:12 +00:00
a7583a7b8b
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-21 02:32:11 +00:00
b676fb02e1
only get out harvest items in interface and when indexing
2013-03-21 02:31:34 +00:00
15c44d9aa8
Merge branch '23-harvest-form-cleanup' into release-v2.0
2013-03-20 17:03:41 +00:00
4ba298fe58
[ #23 ] Make labels a little wider on harvest new form
2013-03-20 14:07:03 +00:00
02e90767f4
Fix source listing in organization page
...
It needed update after #515 in ckan core
2013-03-20 13:01:23 +00:00
86355fb9db
[ #23 ] Form cleanup after core bootstrap upgrade
2013-03-20 10:44:24 +00:00
634a0bbd30
return instead of continue
2013-03-19 01:21:20 +00:00
3adf38105e
readd code from old branch seperating the fetch and import logic
2013-03-19 01:16:43 +00:00
c2a6bd14eb
Add auth function for harvest_source_show_status
2013-03-18 16:48:27 +00:00
c76b7d95f3
Only count public datasets on the source status
...
This is more in line with what is done on the orgs/groups pages
2013-03-18 16:41:01 +00:00
cb80ac784e
Add logic to show private datasets to authorizaded users
2013-03-18 16:29:29 +00:00
341331ac53
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-03-14 17:33:21 +00:00
d77f16aba9
[ #21 ] Improve gather stage error handling
...
See issue for full details. Basically we don't want to catch any
exception at the queue.py level, as they prevent debugging. Harvesters
should deal with them and return a list of ids or an empty list if no
objects need to be fetched.
Also improved the debug messages.
2013-03-14 17:31:07 +00:00
b30cc54427
Fix for add harvest source button within org
2013-03-14 14:45:54 +00:00
91f18bffab
Fix pagination on org sources listing
2013-03-14 11:44:38 +00:00
8cac0977aa
Fix import on org sources listing
2013-03-14 11:44:22 +00:00
cd6c1b56a8
[ #18 ] Get package dict on after_delete to check type
...
No need for #615 in core then
2013-03-13 17:31:39 +00:00
1b11b00946
[ #18 ] Fix wrong logic for setting the source active field
2013-03-13 13:19:43 +00:00
cb5e06119e
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-12 23:31:58 +00:00
06355ee6c4
Make IFacets work for harvest source related searches
2013-03-12 23:31:06 +00:00
fab5b81c2c
Pass context to functions handling harvest sources
2013-03-12 17:30:31 +00:00
5e50a5c9ad
[ #8 ] Update how state is handled for source objects
2013-03-12 15:35:49 +00:00
2ee3f33f51
[ #18 ] Allow reactivation of sources
...
Due to #607 in CKAN core, once a source was deleted you could not
reactivate it again. As a workaround, if the source is deleted the
Delete button is not shown and the state select is, so you can set it to
'active'.
Also fixed wrong redirect after deletion.
2013-03-12 14:06:54 +00:00
23d1d5742c
[ #18 ] Update delete harvest source functionality
...
The harvest_source_delete logic function proxies to package delete,
which will delete the harvest source dataset. The harvest plugin then
hooks to the after_delete extension point in order to inactivate the
actual HarvestSource object and abort any pending jobs.
Also added the Delete button to the harvest source form.
2013-03-12 13:14:07 +00:00
c957fdf17c
Merge branch '14-template-tweaks' into release-v2.0
2013-03-08 14:49:43 +00:00
ecceff48ed
[ #14 ] Use source.organization again after fix in 949bb6f
2013-03-08 14:48:49 +00:00
949bb6fe6a
[ #16 ] Add organization to source dict
2013-03-08 14:47:11 +00:00
f25ef19985
[ #14 ] Fix for org breadcrumbs on sources
2013-03-08 12:48:11 +00:00
2a53e4a2e4
[ #14 ] Couple of minor template tweaks
2013-03-08 12:38:41 +00:00
7257258ca4
mark new harvest objects as current
...
When a new harvest_object for a new package was being created, it
was immediately being marked as false, as all objects were marked
as false, including the new object just created and newly marked
as current=true.
Fix so that old HarvestObjects are only marked as current=False
when updating an existing package.
2013-03-07 20:27:27 +00:00
14e51ec587
Fix for removed snippet from ckan core
2013-03-07 11:52:59 +00:00
2ee27164c3
[ #13 ] Remove or deprecate unused code
...
Mostly in controllers, dictization and plugin, either related to the old
templates pre-dataset type or old authorization.
2013-03-06 16:54:33 +00:00
6c02c87f8d
[ #13 ] Set routes to /harvest
...
Mostly painless as we (most of the time) were using DATASET_TYPE_NAME.
All old routes now point to the correct place in the new interface.
2013-03-06 16:33:46 +00:00
eda280f266
Merge branch '12-org-source-listing' into 2.0-dataset-sources
2013-03-06 15:45:45 +00:00
889325dd9c
[ #12 ] Clean up and rename organization controller
2013-03-06 15:43:10 +00:00
e9adaa7f91
[ #12 ] Change URL for org sources list
...
Use "/organization/harvest_source/{id}", which will turn into
"/organization/harvest/{id}" soon
2013-03-06 15:38:38 +00:00
74633d0803
Fix error count in job stats
...
We want to take into account objects with errors that where created or
updated anyway (eg bbox errors), so we bascially query for the number of
objects that have object errors.
Also add the number of gather errors to this count.
2013-03-06 13:44:04 +00:00
ef2defbcf9
[ #7 ] Refactor job report page to include all errors
2013-03-06 13:43:40 +00:00
bec31a611e
Fix empty job finished date
2013-03-06 13:42:35 +00:00
04710fd1c6
Revert removal of filter in job list action in 7544d5c
2013-03-06 12:19:20 +00:00
c2b552b980
[ #12 ] Better faceting for specifically harvest sources
2013-03-06 11:38:24 +00:00
246898049e
[ #12 ] When harvest source listing is within org links goto edit pages
2013-03-06 11:36:24 +00:00
9d149e4e5d
[ #12 ] Makes a harvest source admin page within org look a little nicer
2013-03-06 11:23:36 +00:00
ca2df234d2
[ #12 ] begin work on org harvest source controller
2013-03-06 04:11:31 +00:00
23aa45cc71
Merge branch '2.0-dataset-sources' into source_extra_config_validation
2013-03-06 01:10:48 +00:00
d9a71f7c59
[ #7 ] Fix wrong finish date on job listing
2013-03-05 18:56:30 +00:00
e566c96d62
[ #7 ] Adds new harvest source button
2013-03-05 16:06:04 +00:00
7544d5c5ef
[ #7 ] Removed faceted navigation for uneeded toggles in job reports
2013-03-05 15:23:42 +00:00
e64c8ead0f
fix print gather_errors
2013-03-05 12:49:20 +00:00
574c69fa9c
Merge branch '2.0-dataset-sources' into 7-harvest-source-templates
2013-03-01 17:55:16 +00:00
182fbf054a
Add XML declaration to contents if not present
2013-03-01 17:25:35 +00:00
5c17a525c1
Refresh session after each harvest stage
...
Otherwise the eg the source config got cached and you needed to restart
the consumers to refresh it.
2013-03-01 12:55:59 +00:00
bd128ab58b
Refresh session after each harvest stage
...
Otherwise the eg the source config got cached and you needed to restart
the consumers to refresh it.
2013-03-01 12:52:58 +00:00
3b6468b181
Merge branch '2.0-dataset-sources' of github.com:okfn/ckanext-harvest into 2.0-dataset-sources
2013-03-01 12:51:17 +00:00
9432368bea
fix gather_stage if there is a previous job
...
change check on gather stage to check for changed packages since
last job instead of current harvest job's gather_start
fix attribute look up bug
fix print_job to print 0 gather_errors instead of key error
2013-02-28 19:06:21 +00:00
ffce2c7915
Merge branch '2.0-dataset-sources' of github.com:okfn/ckanext-harvest into 2.0-dataset-sources
2013-02-28 18:11:12 +00:00
217d58d3a4
Merge branch 'source_extra_config_validation' of github.com:okfn/ckanext-harvest into source_extra_config_validation
2013-02-28 16:03:27 +00:00
f28dc97f79
Fix bug in harvest job reports
2013-02-28 15:47:56 +00:00
dab98112dc
Fix bug in harvest job reports
2013-02-28 15:47:35 +00:00
871576f89c
Merge remote-tracking branch 'remotes/origin/source_extra_config_validation' into source_extra_config_validation
2013-02-28 13:48:58 +00:00
9cef777e7b
make sure config is also on top level
2013-02-28 13:46:16 +00:00
e82410724a
Merge branch '7-harvest-source-templates' into source_extra_config_validation
2013-02-28 12:18:09 +00:00
f7cba69fe6
Merge branch '2.0-dataset-sources' into 7-harvest-source-templates
2013-02-28 12:17:47 +00:00
a86d91c3f0
[ #11 ] Make get actions side_effect_free
2013-02-28 12:17:15 +00:00
fe6952ed00
Merge branch '7-harvest-source-templates' into source_extra_config_validation
2013-02-27 15:45:33 +00:00
ba486a9482
add indexing of datasets whilst harvesting
2013-02-27 11:34:09 +00:00
d1b2b158b2
[ #7 ] Harvest listing page and HTML/CSS cleanup
...
* I'm happy with /harvest_source/ now
* Also I've removed a load of undeeded CSS that wasn't really needed
* Also templates are now using core styles instead of custom ones
2013-02-27 11:14:04 +00:00
e0a3eb7899
add javascript for source type
2013-02-25 18:12:47 +00:00
5b50126670
source extras field type
2013-02-25 18:07:34 +00:00
efe977512b
Include gather errors on job summaries and reports
2013-02-25 17:17:08 +00:00
d1b71308af
[ #7 ] Minor tweaks in job pages
2013-02-25 16:15:37 +00:00
c7bb897cdd
[ #7 ] Inactivate Refresh button if a new job alredy exists
2013-02-25 15:33:29 +00:00
kindly
amercader
joetsoi
John Martin