58a873ac7a
[ #91 ] Remove config fields from source dict before indexing
...
We don't need them and will avoid indexing errors
2014-06-27 16:54:39 +01:00
a59ab4b5ff
[ #91 ] Consolidate all harvest source reindex code in a single action
...
Make it available to users with permissions on the harvest source
2014-06-27 16:48:14 +01:00
7459358fa1
Support for single import commands
...
We are now able to run `paster harvester import` for a single harvest
object or for a single dataset, providing ids or name.
2014-05-15 16:30:30 +01:00
43f1d08255
[ #97 ] Persitent endpoint for datasets harvest objects
...
Contrary to `/harvest/object/xxx`, this endpoint is passed the dataset
id, thus it not depends on a particular object but the most recent one.
2014-04-30 17:45:07 +01:00
4cc56f51ab
[ #76 ] Use harvest_source_show on reindex command
2014-01-14 17:04:34 +00:00
51e842ee6e
Add quotes to harvest_source_id in Solr query when clearing harvest sources
2013-11-22 11:01:26 +02:00
da2fd45e80
[ #65 ] make harvest_job_exists validator return model object
...
return the model in the validator instead of checking that it exists in
the validator, returning the id and then fetching it again in the action
function
2013-10-03 15:51:37 +01:00
9b3199b41b
[ #65 ] remove unused code
2013-09-17 17:02:38 +01:00
5da153c6b6
[ #65 ] harvest_object_create action
...
update to use schema and validators. Also accept more parameters to
data_dict.
2013-09-17 16:49:19 +01:00
1b663bbff4
add harvest_object_create action
2013-09-04 14:17:01 +01:00
70e53a7833
Fix bug where source was being treated as an object, when it's a dict
2013-07-29 07:06:58 -03:00
751409ab7d
[ #34 ] Integrate clear command with delete source
...
When deleting a source, if clear_source equals true in the context,
harvest_source_clear will be called. Default is false. The UI shows a
select with the two options.
2013-05-20 14:30:22 +01:00
7b652542e7
[ #34 ] Fix harvest_source_clear action
...
Some typos in the SQL statements, and also the source needs to be
reindexed to update the status with the counts.
2013-05-16 17:33:39 +01:00
1714e55110
simplify harvest_clear queries so they do not lock on big db
2013-04-30 13:59:23 +01:00
a2b8ab1994
make harvest source clear not create table
2013-04-30 12:40:46 +01:00
dcfd201cdd
[ #32 ] redis queue support
2013-04-21 17:04:57 +01:00
5414b6c08d
Merge branch '29-new-idataset-form' into release-v2.0
2013-04-08 13:23:41 +01:00
a9b8be8f01
harvest source index clear
2013-03-28 15:36:44 +00:00
fbc8ecde97
[ #29 ] Fix some imports on actions and plugin
2013-03-28 15:00:44 +00:00
845c9927a8
add harvest source clear
2013-03-25 11:39:00 +00:00
c2a6bd14eb
Add auth function for harvest_source_show_status
2013-03-18 16:48:27 +00:00
c76b7d95f3
Only count public datasets on the source status
...
This is more in line with what is done on the orgs/groups pages
2013-03-18 16:41:01 +00:00
23d1d5742c
[ #18 ] Update delete harvest source functionality
...
The harvest_source_delete logic function proxies to package delete,
which will delete the harvest source dataset. The harvest plugin then
hooks to the after_delete extension point in order to inactivate the
actual HarvestSource object and abort any pending jobs.
Also added the Delete button to the harvest source form.
2013-03-12 13:14:07 +00:00
2ee27164c3
[ #13 ] Remove or deprecate unused code
...
Mostly in controllers, dictization and plugin, either related to the old
templates pre-dataset type or old authorization.
2013-03-06 16:54:33 +00:00
bec31a611e
Fix empty job finished date
2013-03-06 13:42:35 +00:00
04710fd1c6
Revert removal of filter in job list action in 7544d5c
2013-03-06 12:19:20 +00:00
7544d5c5ef
[ #7 ] Removed faceted navigation for uneeded toggles in job reports
2013-03-05 15:23:42 +00:00
5b0b6a4b8b
Fix bug in harvest job reports
2013-02-28 15:47:48 +00:00
f7cba69fe6
Merge branch '2.0-dataset-sources' into 7-harvest-source-templates
2013-02-28 12:17:47 +00:00
a86d91c3f0
[ #11 ] Make get actions side_effect_free
2013-02-28 12:17:15 +00:00
efe977512b
Include gather errors on job summaries and reports
2013-02-25 17:17:08 +00:00
57b3739dd4
[ #7 ] Return most recent job on source status, not just finished
2013-02-25 15:32:39 +00:00
eaa8988440
[ #4 ] Changes in schema to accommodate organizations
...
Basically handle the 'owner_org' field in form_to_db and db_to_form.
Added 'owner_org', 'frequency' (has default) and 'config' to surplus
keys in check_data_dict.
Also remove schema tweaks to let package_show call the appropiate schema
function.
2013-02-11 16:34:52 +00:00
e1ce0b7267
[ #5 ] Allow not returning error summary on job dictize
2013-02-04 18:28:45 +00:00
8576ad6784
[ #5 ] Add job listing page
2013-02-04 18:20:58 +00:00
42bace3628
[ #5 ] Add new finished field for harvest job
...
When the run command flags a job as finished, it will query the most
recent harvest object for this job and use its import_finished value as
the job finishing time.
2013-01-28 17:19:28 +00:00
676c7d34b6
[ #5 ] Add method for returning the original URL for a document
...
Harvesters implementing IHarvester can define a `get_original_url`
method that should return a URL pointing to the original location of a
document in the remote server. If present, this URL will be used on the
job reports.
Examples:
* For a CKAN record: http://{ckan-instance}/api/rest/{guid}
* For a WAF record: http://{waf-root}/{file-name}
* For a CSW record: http://{csw-server}/?Request=GetElementById&Id={guid}& ...
2013-01-24 18:35:43 +00:00
daa9a385ff
Update job keys changed on 9ba6e8f
2013-01-24 17:36:58 +00:00
30d58b2b7b
[ #5 ] Preliminary job report logic function and page (WIP)
2013-01-23 18:04:19 +00:00
b2b89dfd61
Add command for reindex all harvest sources
2013-01-22 16:43:36 +00:00
9ba6e8f3b3
[ #5 ] Add error summary to harvest_job_dictize
...
It will return the counts for the 20 most common errors for that
particular job. These will available when calling harvest_job_show.
Also refactor the harvest source status object to just call
harvest_job_dictize on the 'last_job' key, as it has all the
interesting fields anyway.
2013-01-22 13:13:24 +00:00
30c9eedf5f
Improve harvest source status creation
...
Use report_status field to improve speed, remove unnecessary fields.
2013-01-17 15:43:45 +00:00
7b6beb1470
fix wrong authorization logic
2012-12-24 22:34:37 +00:00
43950aa4ff
Merge branch 'release-v2.0' into 4-new-auth-for-2.0
...
Conflicts:
ckanext/harvest/logic/action/get.py
ckanext/harvest/tests/test_queue.py
2012-12-20 16:38:57 +00:00
19cd80b264
[ #4 ] Fixes on the auth layer against the new core auth
...
Thanks @locusf for the original patch
2012-12-20 16:09:26 +00:00
596b9bb475
fix auth to use new sysadmin flag
2012-12-17 23:46:43 +00:00
6df525a377
Reindex the harvest source dataset after finishing jobs
...
This ensures that the status details shown on the harvest sources search
page is up to date (as it is loaded from the indexed data_dict)
2012-12-14 14:27:55 +00:00
c1b0415cb6
Merge branch 'release-v2.0' into 2.0-dataset-sources
...
Conflicts:
ckanext/harvest/model/__init__.py
2012-12-13 18:33:59 +00:00
b424ba1cea
Add flag to avoid returning all objects when getting a job
2012-12-13 18:20:49 +00:00
0dde483992
Set job status to Finished when actually finishing it
...
Until now, harvest jobs were set to Finished just after sending all
objects to the fetch stage. Now every time the run command is run, jobs
are set to Running, and all previous Running jobs are checked to see if
all harvest objects have a state of Complete or Error. Only then the job
is flagged as Finished.
2012-12-13 18:19:22 +00:00
amercader
Mikko Koho
joetsoi
Vitor Baptista
kindly
John Martin