53f692b802
Merge remote-tracking branch 'remotes/upstream/master'
2015-11-02 16:00:14 +00:00
1702cf2f09
Remove ', None' on .get() calls because it's the default value.
2015-11-02 15:51:25 +00:00
0c19acba78
Changed double quotes to single quotes in docstrings.
2015-11-02 15:50:04 +00:00
a6069d93db
Fixed bug where the harvest source url validator would validate against
...
all harvest sources that were ever created instead of just sources that
were currently enabled.
2015-10-30 16:59:04 +00:00
3f37ae5f45
Corrected docstring.
2015-10-30 16:11:25 +00:00
02b81187df
Fixed bug with deleting harvest source's which have a custom
...
configuration. Added PEP-8 compliance.
2015-10-30 15:15:41 +00:00
3f856c3b8c
Merge branch 'activate-pyenv'
2015-10-30 14:12:56 +00:00
975db5a6c9
Merge branch 'harvest-command'
2015-10-30 14:12:50 +00:00
55325f5940
Updated harvest source url validator to allow for duplicate URL's with
...
unique configs.
2015-10-30 11:59:24 +00:00
2c41293c9c
Updated the validator to check for unique sets as well as URL.
2015-10-29 18:30:51 +00:00
1a680f3fd3
[ #158 ] Fix spaces encoding broken in previous merge. Tested with data.gov.uk.
2015-10-29 17:31:04 +00:00
39ce744368
Modified to make PEP-8 compliant.
2015-10-29 17:18:51 +00:00
f1d2d5fdc4
[ #111 ] Run jobs straight away.
2015-10-28 21:58:36 +00:00
a617979338
[ #141 ] Add install instruction to activate the pyenv. Fixes #141 .
2015-10-28 20:05:43 +00:00
421e6da660
Add run_test, job_abort, source commands
...
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
2015-10-28 17:51:58 +00:00
e2ab9e58e7
Merge remote-tracking branch 'origin/master' into 157-version-three-apify
...
Conflicts:
ckanext/harvest/harvesters/ckanharvester.py
2015-10-28 14:34:27 +00:00
3f74c29c99
Merge branch 'master' into 157-version-three-apify
2015-10-27 17:45:27 +00:00
55245b5091
[ #158 ] PEP8/formatting.
2015-10-27 17:43:11 +00:00
2a79873855
[ #158 ] Use package search to get all datasets. Add paging search results. Store pkg_dict from search in the object rather than request it again in fetch_stage.
2015-10-27 17:33:22 +00:00
86630adab7
Merge branch 'include-exclude-org'
2015-10-27 15:52:55 +00:00
b56fae8aed
Fixes and tests
...
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
2015-10-23 17:30:28 +00:00
24574f485b
Setup harvest model in harvester tests
2015-10-23 15:43:01 +01:00
caeeace8dc
Merge branch 'master' into 157-version-three-apify
2015-10-23 14:39:48 +01:00
bc49149d5e
Merge branch 'master' into include-exclude-org
2015-10-23 14:36:53 +01:00
0c0a996b85
Merge branch 'master' into db-error
...
Conflicts:
ckanext/harvest/queue.py
2015-10-23 13:33:44 +01:00
2f4adfb338
Merge branch 'tests'
2015-10-23 13:18:15 +01:00
992cdc57d2
Test 2.4 and latest branches for all releases
2015-10-23 11:56:12 +01:00
1bad80e273
Merge branch 'purge'
2015-10-23 11:52:35 +01:00
3c6cc55be0
Only flush keys on the current Redis database
2015-10-23 11:52:22 +01:00
fdbade465f
Merge branch 'master' into purge
2015-10-23 11:33:43 +01:00
d950b13400
Merge branch 'unique-names-improved'
2015-10-23 11:02:49 +01:00
9b10f9e227
Merge branch 'migration-states'
2015-10-23 10:59:17 +01:00
501edffe2d
Merge branch 'master' into migration-states
2015-10-23 10:59:04 +01:00
0f12c71aef
Merge branch 'docs'
2015-10-23 10:56:23 +01:00
5ec47db63e
Merge pull request #166 from LondonAppDev/master
...
Fixed Typos.
2015-10-21 17:57:02 +01:00
3e4a9933ce
Remove prints.
2015-10-21 16:52:19 +00:00
6fb5728d02
Fixed Typos.
2015-10-21 17:48:07 +01:00
dc7af5d150
Remove prints.
2015-10-21 16:38:03 +00:00
eb9aa17862
Include/exclude orgs funcationality based on work by memaldi and ross.
2015-10-21 16:33:16 +00:00
f70c16bce7
Add framework for testing harvesters. Modernize existing tests.
2015-10-21 16:26:57 +00:00
d1f84295f8
purge_queues command now has warning about impact of Redis flushall, plus add some (log) output when you run a purge.
2015-10-21 16:12:40 +00:00
1f81fefcbb
Correct doc error for configuring amqp.
2015-10-21 16:11:11 +00:00
6360681a8f
[ #105 ] Fix order of deletes, as agreed with @florianm.
2015-10-12 15:57:27 +01:00
372fc56c89
Merge branch '105-clean-the-cleaner' of https://github.com/datawagovau/ckanext-harvest into datawagovau-105-clean-the-cleaner
2015-10-09 15:00:56 +01:00
82bdff2f34
Add tests
2015-10-01 17:59:17 +01:00
be3e88086a
Generating unique names improved
...
* Harvesters that change the name when the title changes have had a
problem when the change is small and a number was unnecessarily
appended. e.g. "Trees "->"Trees" meant _gen_new_name("Trees") returned
"trees1". Now you can specify the existing value and it will return
that if it still holds.
* Maximum dataset name length is now adhered to.
* To make a name unique, a sequential number is now added, since for
users that is more understandable and pleasant. However hex digits are
still an option, for those that want to harvest concurrently.
2015-10-01 17:53:03 +01:00
1a6dca7c00
[ #148 ] Catch a more specific exception.
2015-10-01 12:30:40 +01:00
6dd40bfcf9
Changes the gather state to use v3 API
...
Rather than using the revisions in v2 API this now uses the
package_search API so that we can extend it with proper filters in
future.
2015-09-10 18:53:16 +01:00
89b6ad2ce1
Merge pull request #150 from raphaelstolt/templates_new-fix
...
[#103 ] Include templates_new directory
2015-09-09 11:32:57 +01:00
d9fc9bdfc6
[ #103 ] Include fanstatic styles directory
2015-08-19 14:58:16 +02:00
Mark Winterbottom
amercader
David Read
Ross Jones
Raphael Stolt