CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,026 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

119 Commits

Author SHA1 Message Date
Stefanie Taepke c29b52867c update readme for receivers of error-mails 2018-06-13 13:54:13 +02:00
Stefanie Taepke 5f414e2c99 add error-notifications in case of harvest-failure 
requires configuration-values:
ckan.harvest.status_mail.errored = true
email_to = %comma seperated list of emails to receive the email%
error_email_from = %sender of email%
ckan.site_title = %will be included in the email%
 2018-06-11 13:38:38 +02:00
Stefan Oderbolz 2619185bf9
Merge pull request #318 from ckan/readme-test-tip-ckan-parameter 
Readme test tip ckan parameter
 2018-06-07 18:57:09 +02:00
seitenbau-govdata 431d202d4e Add tests and documentation for dataset name suffix config 2018-06-01 21:45:29 +02:00
David Read 47bdd576fb Minor formatting improvements - unquote 2018-04-03 12:24:33 +01:00
David Read a11c2a6697 Minor formatting improvements 2018-04-03 12:20:18 +01:00
David Read c2000e3804 Added a tip, based on running related tests in ckanext-dcat. 2018-04-03 12:13:03 +01:00
Stefan Oderbolz 291d573870 Add a "Releases" section to README 2017-03-30 23:50:39 +02:00
Stefan Oderbolz a0562de442 Update README to link to CONTRIBUTING 
Add link to CKAN guidlines in CONTRIBUTING
 2017-03-30 23:25:25 +02:00
Davor Lozic 863305ece8 Fix JSON typo 2017-01-05 12:31:40 +01:00
rnoerenberg ff1b861f1b Update documentation 
Added note with the limit of 1000 harvest sources
 2016-11-16 16:02:12 +01:00
Raphael Stolt e8570b9e50 Add clearsource history command 2016-11-15 15:04:01 +01:00
David Read ab6a83c49a [#260] Fix the ckan ini filename 2016-08-05 16:12:59 +01:00
David Read d0a8cab479 [#260] Improve install docs 2016-07-29 12:05:12 +01:00
David Read 9d1c20fc44 [#249] Slight README improvement. 2016-06-13 10:28:18 +00:00
David Read f1742fb51a Fix default_groups. It accepted a list of package_name/ids and was trying to add this to the package, but the package needs a dict. Added test. 2016-06-10 09:16:32 +00:00
David Read bfc9b8e0d9 [#249] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE. 2016-06-09 22:11:03 +00:00
Petar Efnushev 1fda3c0255 Change db logger pagination param name 2016-06-06 10:27:05 +02:00
Petar Efnushev 34dedf3869 Add harvest log API usage documentation 2016-06-03 13:55:34 +02:00
Petar Efnushev 9641dc69c2 README updates 2016-05-17 11:31:38 +02:00
Petar Efnushev cc6cb3e389 Changed default config params for the database logger 
Added database logger test case
README updates
 2016-05-16 13:15:12 +02:00
Petar Efnushev a3026f915a README updates 2016-05-12 09:46:33 +02:00
Petar Efnushev ceb8d26aa0 Spacing fixes in README.rst 2016-05-11 17:26:40 +02:00
Petar Efnushev 0be2c868cb README updates 
DBLogHandler updates
Added harvest_log table migration for existing users
Implemented database log scoping
 2016-05-11 13:29:53 +02:00
Petar Efnushev cc86f4062a README updates 2016-05-06 19:07:03 +02:00
Petar Efnushev fe3e92bffe README updates 2016-05-06 18:59:16 +02:00
Petar Efnushev dc4abdcbd8 README updates 2016-05-06 18:54:57 +02:00
Petar Efnushev 30388f04e7 spacing fix 2016-04-11 21:03:40 +02:00
Petar Efnushev 17804b902b spacing fixes 2016-04-11 21:01:11 +02:00
Petar Efnushev 3665889f27 Updated README.rst with configuration options for the logger 2016-04-11 19:42:09 +02:00
David Read 4c1f27161e Fix merge 2016-02-12 18:22:36 +00:00
David Read 7096b7ddf2 Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify 2016-02-12 16:51:26 +00:00
David Read 260cd1f2b7 Merge branch 'master' of github.com:ckan/ckanext-harvest into 214-remove-genshi 2015-12-10 16:02:50 +00:00
David Read b4370f35b7 Note that CKAN 2 is the minimum version now genshi is removed. 2015-12-10 16:02:07 +00:00
David Read f67029b993 Hint about rerunning the import_stage. 2015-12-01 17:59:11 +00:00
David Read b8c7d39e1a Document existing functionality for aborting during the gather stage. 2015-12-01 17:56:24 +00:00
David Read 25301e2152 Update README from interfaces.py. PEP8. Mention HarvestObject - Package relation in interface. 2015-12-01 17:51:35 +00:00
amercader f1ba2bcfb3 Namespace Redis keys to avoid conflicts between instances 
The `ckan.site_id` config option (or `default` if missing) is used to
namespace the Redis keys: routing key and persistance key. Consumers
will only get the relevant keys for their instance.
 2015-11-20 14:17:25 +00:00
amercader ede50aa3fb Merge branch 'immediate-harvest' 2015-11-20 12:28:35 +00:00
David Read 8a7bc9e1d8 Merge remote-tracking branch 'origin/master' into immediate-harvest 
Conflicts:
	README.rst
	ckanext/harvest/commands/harvester.py
	ckanext/harvest/logic/action/create.py
	ckanext/harvest/logic/action/update.py
	ckanext/harvest/logic/auth/update.py
 2015-11-03 00:40:25 +00:00
David Read c7fac36c1c [#107] "unchanged" response tested and related fixes 
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
 2015-11-03 00:22:53 +00:00
David Read 14f372aec6 Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify 
Conflicts:
	README.rst
 2015-11-02 17:01:22 +00:00
amercader 3f856c3b8c Merge branch 'activate-pyenv' 2015-10-30 14:12:56 +00:00
David Read f1d2d5fdc4 [#111] Run jobs straight away. 2015-10-28 21:58:36 +00:00
David Read a617979338 [#141] Add install instruction to activate the pyenv. Fixes #141. 2015-10-28 20:05:43 +00:00
David Read 421e6da660 Add run_test, job_abort, source commands 
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
 2015-10-28 17:51:58 +00:00
David Read e2ab9e58e7 Merge remote-tracking branch 'origin/master' into 157-version-three-apify 
Conflicts:
	ckanext/harvest/harvesters/ckanharvester.py
 2015-10-28 14:34:27 +00:00
David Read 55245b5091 [#158] PEP8/formatting. 2015-10-27 17:43:11 +00:00
David Read b56fae8aed Fixes and tests 
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package

TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
 2015-10-23 17:30:28 +00:00
David Read bc49149d5e Merge branch 'master' into include-exclude-org 2015-10-23 14:36:53 +01:00