CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
977 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

111 Commits

Author SHA1 Message Date
Stefan Oderbolz a0562de442 Update README to link to CONTRIBUTING 
Add link to CKAN guidlines in CONTRIBUTING
 2017-03-30 23:25:25 +02:00
Davor Lozic 863305ece8 Fix JSON typo 2017-01-05 12:31:40 +01:00
rnoerenberg ff1b861f1b Update documentation 
Added note with the limit of 1000 harvest sources
 2016-11-16 16:02:12 +01:00
Raphael Stolt e8570b9e50 Add clearsource history command 2016-11-15 15:04:01 +01:00
David Read ab6a83c49a [#260] Fix the ckan ini filename 2016-08-05 16:12:59 +01:00
David Read d0a8cab479 [#260] Improve install docs 2016-07-29 12:05:12 +01:00
David Read 9d1c20fc44 [#249] Slight README improvement. 2016-06-13 10:28:18 +00:00
David Read f1742fb51a Fix default_groups. It accepted a list of package_name/ids and was trying to add this to the package, but the package needs a dict. Added test. 2016-06-10 09:16:32 +00:00
David Read bfc9b8e0d9 [#249] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE. 2016-06-09 22:11:03 +00:00
Petar Efnushev 1fda3c0255 Change db logger pagination param name 2016-06-06 10:27:05 +02:00
Petar Efnushev 34dedf3869 Add harvest log API usage documentation 2016-06-03 13:55:34 +02:00
Petar Efnushev 9641dc69c2 README updates 2016-05-17 11:31:38 +02:00
Petar Efnushev cc6cb3e389 Changed default config params for the database logger 
Added database logger test case
README updates
 2016-05-16 13:15:12 +02:00
Petar Efnushev a3026f915a README updates 2016-05-12 09:46:33 +02:00
Petar Efnushev ceb8d26aa0 Spacing fixes in README.rst 2016-05-11 17:26:40 +02:00
Petar Efnushev 0be2c868cb README updates 
DBLogHandler updates
Added harvest_log table migration for existing users
Implemented database log scoping
 2016-05-11 13:29:53 +02:00
Petar Efnushev cc86f4062a README updates 2016-05-06 19:07:03 +02:00
Petar Efnushev fe3e92bffe README updates 2016-05-06 18:59:16 +02:00
Petar Efnushev dc4abdcbd8 README updates 2016-05-06 18:54:57 +02:00
Petar Efnushev 30388f04e7 spacing fix 2016-04-11 21:03:40 +02:00
Petar Efnushev 17804b902b spacing fixes 2016-04-11 21:01:11 +02:00
Petar Efnushev 3665889f27 Updated README.rst with configuration options for the logger 2016-04-11 19:42:09 +02:00
David Read 4c1f27161e Fix merge 2016-02-12 18:22:36 +00:00
David Read 7096b7ddf2 Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify 2016-02-12 16:51:26 +00:00
David Read 260cd1f2b7 Merge branch 'master' of github.com:ckan/ckanext-harvest into 214-remove-genshi 2015-12-10 16:02:50 +00:00
David Read b4370f35b7 Note that CKAN 2 is the minimum version now genshi is removed. 2015-12-10 16:02:07 +00:00
David Read f67029b993 Hint about rerunning the import_stage. 2015-12-01 17:59:11 +00:00
David Read b8c7d39e1a Document existing functionality for aborting during the gather stage. 2015-12-01 17:56:24 +00:00
David Read 25301e2152 Update README from interfaces.py. PEP8. Mention HarvestObject - Package relation in interface. 2015-12-01 17:51:35 +00:00
amercader f1ba2bcfb3 Namespace Redis keys to avoid conflicts between instances 
The `ckan.site_id` config option (or `default` if missing) is used to
namespace the Redis keys: routing key and persistance key. Consumers
will only get the relevant keys for their instance.
 2015-11-20 14:17:25 +00:00
amercader ede50aa3fb Merge branch 'immediate-harvest' 2015-11-20 12:28:35 +00:00
David Read 8a7bc9e1d8 Merge remote-tracking branch 'origin/master' into immediate-harvest 
Conflicts:
	README.rst
	ckanext/harvest/commands/harvester.py
	ckanext/harvest/logic/action/create.py
	ckanext/harvest/logic/action/update.py
	ckanext/harvest/logic/auth/update.py
 2015-11-03 00:40:25 +00:00
David Read c7fac36c1c [#107] "unchanged" response tested and related fixes 
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
 2015-11-03 00:22:53 +00:00
David Read 14f372aec6 Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify 
Conflicts:
	README.rst
 2015-11-02 17:01:22 +00:00
amercader 3f856c3b8c Merge branch 'activate-pyenv' 2015-10-30 14:12:56 +00:00
David Read f1d2d5fdc4 [#111] Run jobs straight away. 2015-10-28 21:58:36 +00:00
David Read a617979338 [#141] Add install instruction to activate the pyenv. Fixes #141. 2015-10-28 20:05:43 +00:00
David Read 421e6da660 Add run_test, job_abort, source commands 
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
 2015-10-28 17:51:58 +00:00
David Read e2ab9e58e7 Merge remote-tracking branch 'origin/master' into 157-version-three-apify 
Conflicts:
	ckanext/harvest/harvesters/ckanharvester.py
 2015-10-28 14:34:27 +00:00
David Read 55245b5091 [#158] PEP8/formatting. 2015-10-27 17:43:11 +00:00
David Read b56fae8aed Fixes and tests 
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package

TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
 2015-10-23 17:30:28 +00:00
David Read bc49149d5e Merge branch 'master' into include-exclude-org 2015-10-23 14:36:53 +01:00
amercader 0f12c71aef Merge branch 'docs' 2015-10-23 10:56:23 +01:00
Mark Winterbottom 6fb5728d02 Fixed Typos. 2015-10-21 17:48:07 +01:00
David Read eb9aa17862 Include/exclude orgs funcationality based on work by memaldi and ross. 2015-10-21 16:33:16 +00:00
David Read 1f81fefcbb Correct doc error for configuring amqp. 2015-10-21 16:11:11 +00:00
amercader 84126bb821 Update README 
Simplify install instructions (no need to choose a branch on
ckanext-harvest, just use master). Updated examples and paths.

Added contributing and license info.
 2015-06-10 15:57:28 +01:00
Stefan Oderbolz 2a2d85f60c Wording changes for clearsource and rmsource 2015-06-10 16:19:23 +02:00
Stefan Oderbolz 8ebb843052 Add documentation for clearsource command 2015-06-10 11:29:24 +02:00
amercader fd93a69da9 Document extra connection parameters 2015-06-01 17:55:37 +01:00