ede50aa3fb
Merge branch 'immediate-harvest'
2015-11-20 12:28:35 +00:00
8a7bc9e1d8
Merge remote-tracking branch 'origin/master' into immediate-harvest
...
Conflicts:
README.rst
ckanext/harvest/commands/harvester.py
ckanext/harvest/logic/action/create.py
ckanext/harvest/logic/action/update.py
ckanext/harvest/logic/auth/update.py
2015-11-03 00:40:25 +00:00
c7fac36c1c
[ #107 ] "unchanged" response tested and related fixes
...
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
2015-11-03 00:22:53 +00:00
14f372aec6
Merge branch 'master' of github.com:ckan/ckanext-harvest into 157-version-three-apify
...
Conflicts:
README.rst
2015-11-02 17:01:22 +00:00
3f856c3b8c
Merge branch 'activate-pyenv'
2015-10-30 14:12:56 +00:00
f1d2d5fdc4
[ #111 ] Run jobs straight away.
2015-10-28 21:58:36 +00:00
a617979338
[ #141 ] Add install instruction to activate the pyenv. Fixes #141 .
2015-10-28 20:05:43 +00:00
421e6da660
Add run_test, job_abort, source commands
...
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
2015-10-28 17:51:58 +00:00
e2ab9e58e7
Merge remote-tracking branch 'origin/master' into 157-version-three-apify
...
Conflicts:
ckanext/harvest/harvesters/ckanharvester.py
2015-10-28 14:34:27 +00:00
55245b5091
[ #158 ] PEP8/formatting.
2015-10-27 17:43:11 +00:00
b56fae8aed
Fixes and tests
...
* Fix extras as a list of dicts
* Fix SOLR dates syntax - needed a Z
* Basic tests for this updated ckan harvester
* Now require CKAN 2.0 to be able to be able to save these packages in package_show form. Take advantage of this now we are such various imports from are definitely available, such as munge_tag.
* Add back compatibility for other harvesters supplying restful-like package_dicts to _create_or_update_package
TODO add back in the ability to harvest pre 2.0 CKANs with the RESTful calls (fallback or maybe configurable)
2015-10-23 17:30:28 +00:00
bc49149d5e
Merge branch 'master' into include-exclude-org
2015-10-23 14:36:53 +01:00
0f12c71aef
Merge branch 'docs'
2015-10-23 10:56:23 +01:00
6fb5728d02
Fixed Typos.
2015-10-21 17:48:07 +01:00
eb9aa17862
Include/exclude orgs funcationality based on work by memaldi and ross.
2015-10-21 16:33:16 +00:00
1f81fefcbb
Correct doc error for configuring amqp.
2015-10-21 16:11:11 +00:00
84126bb821
Update README
...
Simplify install instructions (no need to choose a branch on
ckanext-harvest, just use master). Updated examples and paths.
Added contributing and license info.
2015-06-10 15:57:28 +01:00
2a2d85f60c
Wording changes for clearsource and rmsource
2015-06-10 16:19:23 +02:00
8ebb843052
Add documentation for clearsource command
2015-06-10 11:29:24 +02:00
fd93a69da9
Document extra connection parameters
2015-06-01 17:55:37 +01:00
3e21ea4f82
Fix tests, set up Travis
...
TODO: sort out the tests properly, avoiding imports from the legacy ones
2015-04-07 13:31:45 +01:00
07e47acea3
Add a missing command to the install docs
2015-02-13 10:43:43 +00:00
c9aff29613
Recommend Redis as the preferred backend
2014-09-19 11:38:38 +01:00
2ba9908653
Config option to munge tags changed to be consistent with other config options in this extension, and noted in README.
2014-01-29 10:55:51 +01:00
d50eb6fca8
Harvesting of remote organisations similar to remote groups
2013-10-04 16:37:52 +02:00
ffea49ca62
[ #56 ] Update parameters on source create command
...
Add missing title and owner_org fields, remove deprecated user_id and
publisher_id
2013-08-14 11:54:51 +01:00
ad0bbea3a0
Added new 'name' parameter to README
2013-08-06 00:03:36 +02:00
87cae31c75
Fix api_version check in the group importer code
...
I have forgotten to update one check for the api_version 1 in the code
responsible for the remote group import feature. This commit fixes that.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 13:36:56 +02:00
05094090af
Change type of the API version to integer
...
The CKAN logic uses integers when dealing with the API version, e.g.
making checks which API version is in use. Currently, the harvester
uses strings to identify the API version. Instead of dealing with
type conversion the harvester could use integers directly.
This commit fixes okfn/ckanext-harvest#36 . When the API version is
parsed from the configuration it is passed through the int() function.
This way the harvesting will still work even if a harvest source was
configured with a string API version which makes this commit backward
compatible.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 12:51:48 +02:00
f1d11c1307
[ #17 ] Import remote groups in CKAN harvester
...
This is a cleaner commit of the great work done by @platzhirsch
implementing remote groups import on the CKAN harvester.
2013-05-24 16:55:05 +01:00
2dc1b6a178
Update install instructions
2013-05-14 17:14:03 +01:00
b316cc26a2
Update README for 2.0
...
General clean-up, mention redis, new auth, run command
2013-05-14 17:00:20 +01:00
2ee27164c3
[ #13 ] Remove or deprecate unused code
...
Mostly in controllers, dictization and plugin, either related to the old
templates pre-dataset type or old authorization.
2013-03-06 16:54:33 +00:00
8431182f01
Document method and cleanup the interface file
2013-01-24 18:39:19 +00:00
0609262731
[ #2 ] Update README link
2012-12-13 13:59:45 +00:00
4976ebe535
Only need one command to create sysadmin user 'harvest', if user does not exist it will be created
2012-06-15 13:07:17 +03:00
4474ee3e97
[doc][xs] Fix typo on harvester config example
2012-05-09 15:58:55 +01:00
60274333c0
README formatting.
2012-04-10 20:58:51 +01:00
b56859a8cd
README formatting.
2012-04-10 20:58:10 +01:00
1f5c4298e0
README formatting.
2012-04-10 20:55:59 +01:00
ddc136e0e1
Fix typos.
2012-04-10 20:53:12 +01:00
330c9b5391
[merge]
2012-04-10 16:35:17 +01:00
71646dd7c4
[merge]
2012-04-10 16:24:42 +01:00
48d7f4b7f8
Improved info on running paster commands.
2012-04-10 16:20:43 +01:00
f210455aef
[ckan harvester] Replace title on default extras
2012-03-13 12:38:14 +00:00
60e31094a3
[auth,docs] Add auth profiles docs
2012-03-08 17:29:05 +00:00
f482eb607c
[master][doc] Revert inclusion of external file as github doesn't support it.
...
This partially reverts commit 0aa7b7d199
2012-02-16 21:08:00 +00:00
0aa7b7d199
[master][doc] Provided an example supervidor config file
...
And reference to it from the README.
This allows the example conf file to be used as a base in an automatated
deployment.
2012-02-16 20:27:51 +00:00
1027d777e5
[master][docs] Updated docs a little
2012-02-16 16:03:15 +00:00
33aa6f9356
[doc] Added documentation regarding production setup
2012-02-16 12:52:43 +00:00
50537a6738
Merge branch 'master' into enh-1726-harvesting-model-update
2012-02-15 12:01:15 +00:00
78b61a09e9
[doc] Quick note about the current field
2012-02-06 10:17:27 +00:00
9ed152cbea
[ckan harvester] Add support for forcing gathering of all remote packages
2012-02-03 17:54:34 +00:00
38c7742b08
[doc] Fix typos
2012-01-10 17:55:45 +00:00
eb591521ff
Merge branch 'feature-new-ckan-harvester-features'
2012-01-10 17:27:33 +00:00
eb646b3385
[ckan harvester] Add support for defining default extras
2012-01-10 17:07:19 +00:00
a971f7be79
[doc]: Convert to github.
2011-12-12 11:10:11 +00:00
ac9a9100f8
[doc] Document api_key config option
2011-11-23 11:09:16 +00:00
994590531e
[ckan harvester] Support for creating read-only packages
2011-11-18 14:30:10 +00:00
c939d90dbb
[ckan harvester] Support for defining a custom user to do the harvesting
2011-11-18 14:12:30 +00:00
2018d9e513
[ckan harvester] Support for default tags and groups
2011-11-18 13:20:41 +00:00
c36d9bdd8e
Add new command to create new jobs for all active sources
2011-09-06 18:25:17 +01:00
fda4534994
Remove old tests
2011-07-19 14:18:25 +01:00
98384021fc
Update README. Thanks to Rolf Kleef for the patch
2011-07-18 17:34:24 +01:00
6e75d362e3
Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method.
2011-06-07 12:07:53 +01:00
565eaf3d0a
Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form
2011-05-13 18:39:36 +01:00
ce71379d25
Remove references to ckanext-dgu
2011-05-13 17:08:21 +01:00
36b93359fa
[merge] from 987-common-harvesting-framework.
2011-04-28 11:14:13 +02:00
f492ed9c7a
Update README with harvesting interface description and instructions for running the jobs
2011-04-15 15:36:53 +01:00
a11480aa8f
[refactoring] Use ckan.harvest for options in the ini file
2011-04-13 12:46:52 +01:00
040e4d6956
[refactoring] Add a command to create the necessary tables in the database
2011-04-13 12:39:53 +01:00
842bea8433
[refactoring] Remove all geo related functions as they now have it own extension ckanext-spatial
2011-04-11 19:19:09 +01:00
3ce7cf2dc5
documentation tweak
2011-04-05 22:31:49 +01:00
d6e35bc5cb
[controllers,tests]: Added requirement to be sysadmin for all controller operations. Provided test for this and some useful test infrastrure.
2011-03-25 17:01:26 +00:00
0ebe3c7a07
Update README and license notices
2011-03-23 10:20:40 +00:00
16641741ff
Update README with srid config option
2011-03-18 17:46:47 +00:00
4036858ac9
Add a CLI command to create or update the geometries for package extents
2011-03-18 15:44:40 +00:00
38150cca47
Update README with the new API call
2011-03-18 13:53:25 +00:00
7a325d4be3
Update README
2011-03-16 11:06:25 +00:00
abfd394f0b
Update README
2011-03-11 09:49:28 +00:00
66c65f53f4
First draft of the Harvesting extension
...
This extension will hold all the harvesting code that is now located in
different parts of the ckan core and ckanext.dgu. This revision still
uses the ckanext-dgu forms API.
Controllers and templates have been defined for listing, creating and
showing the details of harvesting sources.
2011-03-09 18:56:55 +00:00
amercader
David Read
Mark Winterbottom
Stefan Oderbolz
Sean Hammond
Rachel Knowler
Konrad Reiche
Sean Hammond
David Read
Ian Murray
ww