harvester-d4science/README.rst

================================================
ckanext-harvest - Remote harvesting extension
================================================

This extension will contain all harvesting related code, now present
in ckan core, ckanext-dgu and ckanext-csw.

Configuration
=============

Run the following command (in the ckanext-harvest directory) to create 
the necessary tables in the database::

    paster harvester initdb --config=../ckan/development.ini


The extension needs a user with sysadmin privileges to perform the 
harvesting jobs. You can create such a user running these two commands in
the ckan directory::

    paster user add harvest

    paster sysadmin add harvest

The user's API key must be defined in the CKAN
configuration file (.ini) in the [app:main] section::

    ckan.harvest.api_key = 4e1dac58-f642-4e54-bbc4-3ea262271fe2


The API URL used can be also defined in the ini file (it defaults to 
http://localhost:5000/)::

    ckan.api_url = <api_url>

Tests
=====

To run the tests, this is the basic command::

    $ nosetests --ckan tests/

Or with postgres::

    $ nosetests --ckan --with-pylons=../ckan/test-core.ini tests/

(See the Ckan README for more information.)


Command line interface
======================

The following operations can be run from the command line using the 
``paster harvester`` command::

      harvester initdb
        - Creates the necessary tables in the database

      harvester source {url} {type} [{active}] [{user-id}] [{publisher-id}] 
        - create new harvest source

      harvester rmsource {id}
        - remove (inactivate) a harvester source

      harvester sources [all]        
        - lists harvest sources
          If 'all' is defined, it also shows the Inactive sources

      harvester job {source-id}
        - create new harvest job
  
      harvester jobs
        - lists harvest jobs

      harvester run
        - runs harvest jobs

      harvester gather_consumer
        - starts the consumer for the gathering queue

      harvester fetch_consumer
        - starts the consumer for the fetching queue
       
The commands should be run from the ckanext-harvest directory and expect
a development.ini file to be present. Most of the time you will specify 
the config explicitly though::

        paster harvester sources --config=../ckan/development.ini