CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
137 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

137 Commits

Author SHA1 Message Date
amercader eb646b3385 [ckan harvester] Add support for defining default extras 2012-01-10 17:07:19 +00:00
amercader ae51093213 [ckan harvester] Ignore __junk field, was causing imports to fail 2012-01-10 14:46:12 +00:00
Adrià Mercader ac9a9100f8 [doc] Document api_key config option 2011-11-23 11:09:16 +00:00
Adrià Mercader da469ab08e [base harvester] Custom tag munge function. TODO: check with flexible tags 2011-11-23 11:05:52 +00:00
Adrià Mercader cfaba6e1e8 [ckan harvester] Add support for sending an API key 2011-11-21 17:29:10 +00:00
Adrià Mercader 0ab5c53b47 [ckan harvester] Fix typo 2011-11-18 17:53:01 +00:00
Adrià Mercader f02ee45aae [ui] Show config options in harvest source details page 2011-11-18 14:35:46 +00:00
Adrià Mercader 994590531e [ckan harvester] Support for creating read-only packages 2011-11-18 14:30:10 +00:00
Adrià Mercader c939d90dbb [ckan harvester] Support for defining a custom user to do the harvesting 2011-11-18 14:12:30 +00:00
Adrià Mercader 2018d9e513 [ckan harvester] Support for default tags and groups 2011-11-18 13:20:41 +00:00
Adrià Mercader 8ec05bc3e3 Modify import command to avoid problems with the Session 2011-11-15 11:26:24 +00:00
Adrià Mercader c04d80e27e Use get_action function instead of directly calling the action functions 2011-10-26 17:26:18 +01:00
David Raznick 31dac7029a fix to make import stage work on its own 2011-09-28 14:27:28 +01:00
Adrià Mercader 63afd199a9 Add link to source documents from object errors 2011-09-08 10:27:36 +01:00
Adrià Mercader 31c1ea1c21 Separate gather and object errors in reports. Add info about guid and object id in the object ones 2011-09-08 09:58:21 +01:00
Adrià Mercader c36d9bdd8e Add new command to create new jobs for all active sources 2011-09-06 18:25:17 +01:00
David Read dd00e98d9d [model]: More careful about creating tables, since paster db upgrade loads the envrionment and therefore runs setup() before it does the migrations, and therefore in this instance we do not want to create the db tables. 2011-08-10 16:25:57 +01:00
Adrià Mercader 7927329536 Make harvesters work with latest ckan release 2011-07-29 11:31:03 +01:00
Adrià Mercader bbf623213e Added tag ckanext-harvest-0.1 for changeset 9bc6d9d11f17 2011-07-29 10:47:55 +01:00
Adrià Mercader fda4534994 Remove old tests 2011-07-19 14:18:25 +01:00
Adrià Mercader cabbb4922d Use API version defined in config if present 2011-07-18 17:35:32 +01:00
Adrià Mercader c867660e7d Add docs to base harvester functions 2011-07-18 17:35:03 +01:00
Adrià Mercader 98384021fc Update README. Thanks to Rolf Kleef for the patch 2011-07-18 17:34:24 +01:00
Adrià Mercader 54de6759fe Fix bug with empty config 2011-06-28 15:04:40 +01:00
Adrià Mercader c80e68a12f Ensure the correct configuration is used on each stage 2011-06-14 15:59:13 +01:00
Adrià Mercader 3125bb1514 Add a check to ensure sources with no packages are reharvested 2011-06-14 12:59:48 +01:00
Adrià Mercader 2ac9885150 Page packages in the harvest source details page 2011-06-14 10:27:48 +01:00
Friedrich Lindenberg 0d9d1f8096 reduce number of queries for harvest index to a less insane number. still heavy. 2011-06-13 17:36:35 +02:00
Adrià Mercader ef04ce1774 Add support for config options in CLI 2011-06-13 15:56:19 +01:00
Friedrich Lindenberg 13f2fb3b96 use hasattr for config validation 2011-06-09 11:35:58 +02:00
Adrià Mercader 7b61fb62bf Ignore missing config values 2011-06-07 15:32:46 +01:00
Adrià Mercader 98bfd50f47 Load config in the CKAN harvester 2011-06-07 13:35:11 +01:00
Adrià Mercader 6e75d362e3 Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method. 2011-06-07 12:07:53 +01:00
Adrià Mercader ca6af0249a Reverting previous changeset, as it conflicts with dgu_form_api 2011-06-07 11:58:35 +01:00
Friedrich Lindenberg f9c0ee37aa spacing in template paths 2011-06-06 10:16:34 +02:00
Friedrich Lindenberg 89934b8538 [harvesters] factor out a base harvester for use in generic harvesting apps 2011-06-02 12:07:07 +02:00
David Raznick 79fd966573 add database setup at configure time 2011-05-31 18:06:26 +01:00
David Raznick 264b606c48 take tables out of global scope at import time 2011-05-31 18:02:07 +01:00
Adrià Mercader 2b98080266 [merge] from new-forms, as forms refactoring has been merged in core 2011-05-20 13:50:15 +01:00
Adrià Mercader 235d822458 [ckan harvester] Request only packages modified since last harvest job. Also support older versions which do not include 'metadata_modified' 2011-05-17 17:26:42 +01:00
Adrià Mercader 565eaf3d0a Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form 2011-05-13 18:39:36 +01:00
Adrià Mercader ce71379d25 Remove references to ckanext-dgu 2011-05-13 17:08:21 +01:00
Adrià Mercader fecee82b1a Minor enhancements in the WUI 2011-05-13 17:02:18 +01:00
Adrià Mercader b3a88070e3 [forms] Adapt CLI commands to changes in lib 2011-05-13 16:00:36 +01:00
Adrià Mercader bbe459527f [forms] Major refactoring of the harvest forms. Forms no longer use the DGU form 
API, and are handled similarly to the new ones on CKAN core (logic, schema,
validators...). The UI is also more consistent with the CKAN one.
 2011-05-13 14:17:58 +01:00
Adrià Mercader 26cdc1089d Change date definitions in Harvest Objects. reference_date -> metadata_modified_date, created -> gathered 2011-05-11 17:07:05 +01:00
Adrià Mercader e1080e349e Set a flag to force harvesters to import objects 2011-05-10 17:11:12 +01:00
Adrià Mercader e320d0588f Add command to reimport existing harvest objects 2011-05-10 16:06:57 +01:00
Adrià Mercader f7c6854a1d Save reference date in Harvest Objects when harvesting CKAN instances 2011-05-10 12:57:57 +01:00
Adrià Mercader 329ca2dd29 Add a reference date to the Harvest Objects. This must be set during the harvest 
process.
 2011-05-10 11:05:44 +01:00