bec31a611e
Fix empty job finished date
2013-03-06 13:42:35 +00:00
04710fd1c6
Revert removal of filter in job list action in 7544d5c
2013-03-06 12:19:20 +00:00
c2b552b980
[ #12 ] Better faceting for specifically harvest sources
2013-03-06 11:38:24 +00:00
246898049e
[ #12 ] When harvest source listing is within org links goto edit pages
2013-03-06 11:36:24 +00:00
9d149e4e5d
[ #12 ] Makes a harvest source admin page within org look a little nicer
2013-03-06 11:23:36 +00:00
ca2df234d2
[ #12 ] begin work on org harvest source controller
2013-03-06 04:11:31 +00:00
23aa45cc71
Merge branch '2.0-dataset-sources' into source_extra_config_validation
2013-03-06 01:10:48 +00:00
d9a71f7c59
[ #7 ] Fix wrong finish date on job listing
2013-03-05 18:56:30 +00:00
e566c96d62
[ #7 ] Adds new harvest source button
2013-03-05 16:06:04 +00:00
7544d5c5ef
[ #7 ] Removed faceted navigation for uneeded toggles in job reports
2013-03-05 15:23:42 +00:00
e64c8ead0f
fix print gather_errors
2013-03-05 12:49:20 +00:00
574c69fa9c
Merge branch '2.0-dataset-sources' into 7-harvest-source-templates
2013-03-01 17:55:16 +00:00
182fbf054a
Add XML declaration to contents if not present
2013-03-01 17:25:35 +00:00
5c17a525c1
Refresh session after each harvest stage
...
Otherwise the eg the source config got cached and you needed to restart
the consumers to refresh it.
2013-03-01 12:55:59 +00:00
bd128ab58b
Refresh session after each harvest stage
...
Otherwise the eg the source config got cached and you needed to restart
the consumers to refresh it.
2013-03-01 12:52:58 +00:00
3b6468b181
Merge branch '2.0-dataset-sources' of github.com:okfn/ckanext-harvest into 2.0-dataset-sources
2013-03-01 12:51:17 +00:00
9432368bea
fix gather_stage if there is a previous job
...
change check on gather stage to check for changed packages since
last job instead of current harvest job's gather_start
fix attribute look up bug
fix print_job to print 0 gather_errors instead of key error
2013-02-28 19:06:21 +00:00
ffce2c7915
Merge branch '2.0-dataset-sources' of github.com:okfn/ckanext-harvest into 2.0-dataset-sources
2013-02-28 18:11:12 +00:00
217d58d3a4
Merge branch 'source_extra_config_validation' of github.com:okfn/ckanext-harvest into source_extra_config_validation
2013-02-28 16:03:27 +00:00
f28dc97f79
Fix bug in harvest job reports
2013-02-28 15:47:56 +00:00
dab98112dc
Fix bug in harvest job reports
2013-02-28 15:47:35 +00:00
871576f89c
Merge remote-tracking branch 'remotes/origin/source_extra_config_validation' into source_extra_config_validation
2013-02-28 13:48:58 +00:00
9cef777e7b
make sure config is also on top level
2013-02-28 13:46:16 +00:00
e82410724a
Merge branch '7-harvest-source-templates' into source_extra_config_validation
2013-02-28 12:18:09 +00:00
f7cba69fe6
Merge branch '2.0-dataset-sources' into 7-harvest-source-templates
2013-02-28 12:17:47 +00:00
a86d91c3f0
[ #11 ] Make get actions side_effect_free
2013-02-28 12:17:15 +00:00
fe6952ed00
Merge branch '7-harvest-source-templates' into source_extra_config_validation
2013-02-27 15:45:33 +00:00
ba486a9482
add indexing of datasets whilst harvesting
2013-02-27 11:34:09 +00:00
d1b2b158b2
[ #7 ] Harvest listing page and HTML/CSS cleanup
...
* I'm happy with /harvest_source/ now
* Also I've removed a load of undeeded CSS that wasn't really needed
* Also templates are now using core styles instead of custom ones
2013-02-27 11:14:04 +00:00
e0a3eb7899
add javascript for source type
2013-02-25 18:12:47 +00:00
5b50126670
source extras field type
2013-02-25 18:07:34 +00:00
efe977512b
Include gather errors on job summaries and reports
2013-02-25 17:17:08 +00:00
d1b71308af
[ #7 ] Minor tweaks in job pages
2013-02-25 16:15:37 +00:00
c7bb897cdd
[ #7 ] Inactivate Refresh button if a new job alredy exists
2013-02-25 15:33:29 +00:00
57b3739dd4
[ #7 ] Return most recent job on source status, not just finished
2013-02-25 15:32:39 +00:00
60f9360e84
[ #7 ] Don't show job snippet in dashboard if no jobs
2013-02-25 13:11:08 +00:00
93e15dc529
[ #7 ] Restrict access to source admin page
2013-02-25 13:10:30 +00:00
457b8d5988
[ #7 ] 404 on last job if no jobs yet
2013-02-25 12:49:14 +00:00
34ae6be689
[ #7 ] Fix dataset count on source page
2013-02-25 12:19:09 +00:00
b3819e8df4
[ #7 ] Use dict instead of domain object in templates
2013-02-25 12:18:30 +00:00
49a1c467cf
Merge branch '7-harvest-source-templates' of github.com:okfn/ckanext-harvest into 7-harvest-source-templates
2013-02-25 12:04:34 +00:00
e1d73c82f0
[ #7 ] Make new routes more custom
...
In case we change the root name
2013-02-25 12:03:34 +00:00
ebe246fe99
make report emit added so shows up on front end
2013-02-22 17:32:33 +00:00
57d6b3de74
[ #7 ] Fix auth check on new source form
...
Auth check failed because source was undefined
2013-02-22 17:32:05 +00:00
52c0a5cbd6
Merge branch '2.0-dataset-sources' into 7-harvest-source-templates
2013-02-22 17:26:34 +00:00
f97e3b4c6c
add return True to import stage of ckanharvester
...
Was causing queue.py to report that the import had errored.
2013-02-22 10:13:36 +00:00
83f8cf69a6
Remove unnecessary extra quotes (see #381 on CKAN core)
2013-02-19 11:51:22 +00:00
28e589ee92
[ #7 ] Updates to the edit/new harvest source form
2013-02-12 16:29:07 +00:00
177349fd76
Update HarvesterBase
...
This is a convenience class that other harvesters can extend. Updates
include a cleanup of old functions and porting of enhancements from the
spatial harvesters.
2013-02-12 16:10:13 +00:00
891f247181
[ #7 ] Small template tweaks to job pages
2013-02-12 15:49:06 +00:00
eaa8988440
[ #4 ] Changes in schema to accommodate organizations
...
Basically handle the 'owner_org' field in form_to_db and db_to_form.
Added 'owner_org', 'frequency' (has default) and 'config' to surplus
keys in check_data_dict.
Also remove schema tweaks to let package_show call the appropiate schema
function.
2013-02-11 16:34:52 +00:00
bdc8206e8b
[ #7 ] Harvest job pages UX are complete
2013-02-08 17:19:04 +00:00
7209723856
[ #7 ] Admin templates now are in the correct places
2013-02-08 13:52:48 +00:00
0aa1c1fcbc
[ #7 ] Re-jigged harvest source read pages
2013-02-08 12:15:14 +00:00
3c50a40a76
[ #5 ] Fix auth for harvest_job_list (should forward to harvest_source_update)
2013-02-05 16:41:29 +00:00
413ef8786c
[ #5 ] Fix counts on jobs listing
2013-02-05 16:40:22 +00:00
5956e5a9d5
Merge branch '4-new-auth-for-2.0' into 5-improve-job-errors-reporting
2013-02-05 12:36:26 +00:00
ca7819b885
Merge branch 'release-v2.0' into 2.0-dataset-sources
2013-02-05 12:35:14 +00:00
cca554c5ec
Fix typo and add missing column on v3 migration script
2013-02-05 12:33:56 +00:00
e1ce0b7267
[ #5 ] Allow not returning error summary on job dictize
2013-02-04 18:28:45 +00:00
8576ad6784
[ #5 ] Add job listing page
2013-02-04 18:20:58 +00:00
22389fc52a
[ #5 ] Update report templates
...
The job details page has been updated to show the full error report, and
the whole report page has been dropped. All job details are loaded via a
snippet, which is also loaded on the harvest source page.
The frontend is still completely provisional.
2013-02-01 18:32:41 +00:00
42bace3628
[ #5 ] Add new finished field for harvest job
...
When the run command flags a job as finished, it will query the most
recent harvest object for this job and use its import_finished value as
the job finishing time.
2013-01-28 17:19:28 +00:00
920f07cdf7
[ #5 ] Cleanup the job controller actions
2013-01-28 16:32:53 +00:00
c8e7086567
[ #5 ] Change default auth for showing and listing jobs
...
Forward auth checks to harvest_source_update instead of
harvest_source_show, as job reports should only be visible to users that
can manage sources.
2013-01-28 16:31:11 +00:00
ab78bf21b9
[ #5 ] Fix typo in delete auth function
2013-01-28 16:15:38 +00:00
8431182f01
Document method and cleanup the interface file
2013-01-24 18:39:19 +00:00
676c7d34b6
[ #5 ] Add method for returning the original URL for a document
...
Harvesters implementing IHarvester can define a `get_original_url`
method that should return a URL pointing to the original location of a
document in the remote server. If present, this URL will be used on the
job reports.
Examples:
* For a CKAN record: http://{ckan-instance}/api/rest/{guid}
* For a WAF record: http://{waf-root}/{file-name}
* For a CSW record: http://{csw-server}/?Request=GetElementById&Id={guid}& ...
2013-01-24 18:35:43 +00:00
d4b6dcb7f6
[ #5 ] Add helper function for generating a link to a harvest object
2013-01-24 18:21:05 +00:00
daa9a385ff
Update job keys changed on 9ba6e8f
2013-01-24 17:36:58 +00:00
30d58b2b7b
[ #5 ] Preliminary job report logic function and page (WIP)
2013-01-23 18:04:19 +00:00
234f9f4cc0
[ #5 ] Add job summary page
...
Shows dataset and error counts, job details and a summary of the more
frequent errors.
2013-01-23 17:33:44 +00:00
b2b89dfd61
Add command for reindex all harvest sources
2013-01-22 16:43:36 +00:00
0d79252a09
Add command for reindex all harvest sources
2013-01-22 16:43:25 +00:00
6c861afe39
Update template with new harvest source status
2013-01-22 16:37:31 +00:00
9ba6e8f3b3
[ #5 ] Add error summary to harvest_job_dictize
...
It will return the counts for the 20 most common errors for that
particular job. These will available when calling harvest_job_show.
Also refactor the harvest source status object to just call
harvest_job_dictize on the 'last_job' key, as it has all the
interesting fields anyway.
2013-01-22 13:13:24 +00:00
30c9eedf5f
Improve harvest source status creation
...
Use report_status field to improve speed, remove unnecessary fields.
2013-01-17 15:43:45 +00:00
bfce5185f0
[ #4 ] Add db_to_form_schema_options to harvest plugin to avoid validation on show
2013-01-16 17:45:33 +00:00
2ab10afcf9
[ #4 ] Fix typo in auth functions
2013-01-16 12:56:58 +00:00
2f4cd3a4b0
[ #4 ] Fix logic functions importer
2013-01-15 19:29:17 +00:00
2bb669af21
[ #4 ] Add owner_org field to schema and form
...
This should store the owner organization id.
Also added the errors box on the form.
2013-01-10 12:23:01 +00:00
acb17ff3b0
capture errors more cleanly
2013-01-10 10:48:48 +00:00
e49dd94b34
[ #4 ] Remove authorization functions for the publisher profile
...
The different profiles will be now configured via the harvest source
datasets on CKAN core, so they are no longer needed.
2013-01-09 17:35:47 +00:00
288e1429a6
[ #4 ] Remove the loading of different authorization profiles
...
The different profiles will be now configured via the harvest source
datasets on CKAN core, so it is no longer needed.
Also simplify IActions and IAuthFunction hook calls.
2013-01-09 17:32:05 +00:00
058dcad435
[ #4 ] Minor change on the state field to fix a bug on harvest_source_show
2013-01-09 17:31:30 +00:00
a866445023
[ #4 ] Refactor authorization functions
...
The authorization functions have been refactored to take into account
both the new organizaton based authorization on CKAN core and the
harvest source datasets.
Basically at the source level, authorization checks are forwarded to the
relevant package auth function (package_create, package_update, etc.)
wich will check for organizations membership, sysadmin, etc.
Also we only use functions available on the plugins toolkit whenever
possible.
2013-01-09 17:26:48 +00:00
1342463f8a
Merge branch '2.0-dataset-sources' into 4-new-auth-for-2.0
...
Conflicts:
ckanext/harvest/logic/action/get.py
2013-01-09 11:09:34 +00:00
6b23082010
Move logic from setup_template_variables to helper functions
2013-01-09 11:07:44 +00:00
7b6beb1470
fix wrong authorization logic
2012-12-24 22:34:37 +00:00
01dfda59b6
Merge branch 'release-v2.0' into 4-new-auth-for-2.0
2012-12-24 12:46:56 +00:00
36389e7ce0
make sure gather phase finishes job if there is a severe error
2012-12-24 12:21:21 +00:00
43950aa4ff
Merge branch 'release-v2.0' into 4-new-auth-for-2.0
...
Conflicts:
ckanext/harvest/logic/action/get.py
ckanext/harvest/tests/test_queue.py
2012-12-20 16:38:57 +00:00
fdac761fba
Merge branch 'release-v2.0' into 2.0-dataset-sources
...
Conflicts:
ckanext/harvest/logic/action/get.py
ckanext/harvest/tests/test_queue.py
2012-12-20 16:16:30 +00:00
19cd80b264
[ #4 ] Fixes on the auth layer against the new core auth
...
Thanks @locusf for the original patch
2012-12-20 16:09:26 +00:00
510e2d3725
Fix pager links in harvest source page
2012-12-19 17:27:05 +00:00
b940baacc0
make statistics use new report_field
2012-12-18 02:39:14 +00:00
6b42d96fe0
add report_status field
2012-12-17 23:50:26 +00:00
596b9bb475
fix auth to use new sysadmin flag
2012-12-17 23:46:43 +00:00
478326922b
Fix tests
...
* Adapt test_queue to harvest source datasets
* Don't use the same mock harvester on different datasets as it messes
the tests up
* Skip auth tests for the time being
2012-12-14 14:52:19 +00:00
6df525a377
Reindex the harvest source dataset after finishing jobs
...
This ensures that the status details shown on the harvest sources search
page is up to date (as it is loaded from the indexed data_dict)
2012-12-14 14:27:55 +00:00
amercader
John Martin
kindly
joetsoi