Update README
Simplify install instructions (no need to choose a branch on ckanext-harvest, just use master). Updated examples and paths. Added contributing and license info.
This commit is contained in:
parent
5b4bff329f
commit
84126bb821
64
README.rst
64
README.rst
|
@ -32,25 +32,9 @@ Installation
|
||||||
ckan.harvest.mq.type = rabbitmq
|
ckan.harvest.mq.type = rabbitmq
|
||||||
|
|
||||||
|
|
||||||
2. Install the extension into your python environment.
|
2. Install the extension into your python environment::
|
||||||
|
|
||||||
*Note:* Depending on the CKAN core version you are targeting you will need to
|
(pyenv) $ pip install -e git+https://github.com/ckan/ckanext-harvest.git#egg=ckanext-harvest
|
||||||
use a different branch from the extension.
|
|
||||||
|
|
||||||
For a production site, use the `stable` branch, unless there is a specific
|
|
||||||
branch that targets the CKAN core version that you are using.
|
|
||||||
|
|
||||||
To target the latest CKAN core release::
|
|
||||||
|
|
||||||
(pyenv) $ pip install -e git+https://github.com/okfn/ckanext-harvest.git@stable#egg=ckanext-harvest
|
|
||||||
|
|
||||||
To target an old release (if a release branch exists, otherwise use `stable`)::
|
|
||||||
|
|
||||||
(pyenv) $ pip install -e git+https://github.com/okfn/ckanext-harvest.git@release-v1.8#egg=ckanext-harvest
|
|
||||||
|
|
||||||
To target CKAN `master`, use the extension `master` branch (ie no branch defined)::
|
|
||||||
|
|
||||||
(pyenv) $ pip install -e git+https://github.com/okfn/ckanext-harvest.git#egg=ckanext-harvest
|
|
||||||
|
|
||||||
3. Install the rest of python modules required by the extension::
|
3. Install the rest of python modules required by the extension::
|
||||||
|
|
||||||
|
@ -418,8 +402,8 @@ interface:
|
||||||
|
|
||||||
Here you can also find other examples of custom harvesters:
|
Here you can also find other examples of custom harvesters:
|
||||||
|
|
||||||
* https://github.com/okfn/ckanext-pdeu/tree/master/ckanext/pdeu/harvesters
|
* https://github.com/ckan/ckanext-dcat/tree/master/ckanext/dcat/harvesters
|
||||||
* https://github.com/okfn/ckanext-inspire/ckanext/inspire/harvesters.py
|
* https://github.com/ckan/ckanext-spatial/tree/master/ckanext/spatial/harvesters
|
||||||
|
|
||||||
|
|
||||||
Running the harvest jobs
|
Running the harvest jobs
|
||||||
|
@ -466,7 +450,7 @@ have already installed and configured the harvesting extension (See
|
||||||
`Installation` if not).
|
`Installation` if not).
|
||||||
|
|
||||||
Note: It is recommended to run the harvest process from a non-root user
|
Note: It is recommended to run the harvest process from a non-root user
|
||||||
(generally the one you are running CKAN with). Replace the user `okfn` in the
|
(generally the one you are running CKAN with). Replace the user `ckan` in the
|
||||||
following steps with the one you are using.
|
following steps with the one you are using.
|
||||||
|
|
||||||
1. Install Supervisor::
|
1. Install Supervisor::
|
||||||
|
@ -494,10 +478,10 @@ following steps with the one you are using.
|
||||||
|
|
||||||
[program:ckan_gather_consumer]
|
[program:ckan_gather_consumer]
|
||||||
|
|
||||||
command=/var/lib/ckan/std/pyenv/bin/paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/std/std.ini
|
command=/usr/lib//ckan/default/bin/paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/std/std.ini
|
||||||
|
|
||||||
; user that owns virtual environment.
|
; user that owns virtual environment.
|
||||||
user=okfn
|
user=ckan
|
||||||
|
|
||||||
numprocs=1
|
numprocs=1
|
||||||
stdout_logfile=/var/log/ckan/std/gather_consumer.log
|
stdout_logfile=/var/log/ckan/std/gather_consumer.log
|
||||||
|
@ -508,10 +492,10 @@ following steps with the one you are using.
|
||||||
|
|
||||||
[program:ckan_fetch_consumer]
|
[program:ckan_fetch_consumer]
|
||||||
|
|
||||||
command=/var/lib/ckan/std/pyenv/bin/paster --plugin=ckanext-harvest harvester fetch_consumer --config=/etc/ckan/std/std.ini
|
command=/usr/lib//ckan/default/bin/paster --plugin=ckanext-harvest harvester fetch_consumer --config=/etc/ckan/std/std.ini
|
||||||
|
|
||||||
; user that owns virtual environment.
|
; user that owns virtual environment.
|
||||||
user=okfn
|
user=ckan
|
||||||
|
|
||||||
numprocs=1
|
numprocs=1
|
||||||
stdout_logfile=/var/log/ckan/std/fetch_consumer.log
|
stdout_logfile=/var/log/ckan/std/fetch_consumer.log
|
||||||
|
@ -574,20 +558,44 @@ following steps with the one you are using.
|
||||||
that will run the `run` harvester command periodically. To do so, edit the cron table with
|
that will run the `run` harvester command periodically. To do so, edit the cron table with
|
||||||
the following command (it may ask you to choose an editor)::
|
the following command (it may ask you to choose an editor)::
|
||||||
|
|
||||||
sudo crontab -e -u okfn
|
sudo crontab -e -u ckan
|
||||||
|
|
||||||
Note that we are running this command as the same user we configured the processes to be run with
|
Note that we are running this command as the same user we configured the processes to be run with
|
||||||
(`okfn` in our example).
|
(`ckan` in our example).
|
||||||
|
|
||||||
Paste this line into your crontab, again replacing the paths to paster and the ini file with yours::
|
Paste this line into your crontab, again replacing the paths to paster and the ini file with yours::
|
||||||
|
|
||||||
# m h dom mon dow command
|
# m h dom mon dow command
|
||||||
*/15 * * * * /var/lib/ckan/std/pyenv/bin/paster --plugin=ckanext-harvest harvester run --config=/etc/ckan/std/std.ini
|
*/15 * * * * /usr/lib/ckan/default/bin/paster --plugin=ckanext-harvest harvester run --config=/etc/ckan/std/std.ini
|
||||||
|
|
||||||
This particular example will check for pending jobs every fifteen minutes.
|
This particular example will check for pending jobs every fifteen minutes.
|
||||||
You can of course modify this periodicity, this `Wikipedia page <http://en.wikipedia.org/wiki/Cron#CRON_expression>`_
|
You can of course modify this periodicity, this `Wikipedia page <http://en.wikipedia.org/wiki/Cron#CRON_expression>`_
|
||||||
has a good overview of the crontab syntax.
|
has a good overview of the crontab syntax.
|
||||||
|
|
||||||
|
Community
|
||||||
|
=========
|
||||||
|
|
||||||
|
* Developer mailing list: `ckan-dev@lists.okfn.org <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_
|
||||||
|
* Developer IRC channel: `#ckan on irc.freenode.net <http://webchat.freenode.net/?channels=ckan>`_
|
||||||
|
* `Issue tracker <https://github.com/ckan/ckanext-harvest/issues>`_
|
||||||
|
|
||||||
|
|
||||||
|
Contributing
|
||||||
|
============
|
||||||
|
|
||||||
|
For contributing to ckanext-spatial or its documentation, follow the same
|
||||||
|
guidelines that apply to CKAN core, described in
|
||||||
|
`CONTRIBUTING <https://github.com/ckan/ckan/blob/master/CONTRIBUTING.rst>`_.
|
||||||
|
|
||||||
|
|
||||||
|
License
|
||||||
|
=======
|
||||||
|
|
||||||
|
This extension is open and licensed under the GNU Affero General Public License (AGPL) v3.0.
|
||||||
|
Its full text may be found at:
|
||||||
|
|
||||||
|
http://www.fsf.org/licensing/licenses/agpl-3.0.html
|
||||||
|
|
||||||
|
|
||||||
.. _Supervisor: http://supervisord.org
|
.. _Supervisor: http://supervisord.org
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue