CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
321 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

321 Commits

Author SHA1 Message Date
David Read dd00e98d9d [model]: More careful about creating tables, since paster db upgrade loads the envrionment and therefore runs setup() before it does the migrations, and therefore in this instance we do not want to create the db tables. 2011-08-10 16:25:57 +01:00
Adrià Mercader 7927329536 Make harvesters work with latest ckan release 2011-07-29 11:31:03 +01:00
Adrià Mercader bbf623213e Added tag ckanext-harvest-0.1 for changeset 9bc6d9d11f17 2011-07-29 10:47:55 +01:00
Adrià Mercader fda4534994 Remove old tests 2011-07-19 14:18:25 +01:00
Adrià Mercader cabbb4922d Use API version defined in config if present 2011-07-18 17:35:32 +01:00
Adrià Mercader c867660e7d Add docs to base harvester functions 2011-07-18 17:35:03 +01:00
Adrià Mercader 98384021fc Update README. Thanks to Rolf Kleef for the patch 2011-07-18 17:34:24 +01:00
Adrià Mercader 54de6759fe Fix bug with empty config 2011-06-28 15:04:40 +01:00
Adrià Mercader c80e68a12f Ensure the correct configuration is used on each stage 2011-06-14 15:59:13 +01:00
Adrià Mercader 3125bb1514 Add a check to ensure sources with no packages are reharvested 2011-06-14 12:59:48 +01:00
Adrià Mercader 2ac9885150 Page packages in the harvest source details page 2011-06-14 10:27:48 +01:00
Friedrich Lindenberg 0d9d1f8096 reduce number of queries for harvest index to a less insane number. still heavy. 2011-06-13 17:36:35 +02:00
Adrià Mercader ef04ce1774 Add support for config options in CLI 2011-06-13 15:56:19 +01:00
Friedrich Lindenberg 13f2fb3b96 use hasattr for config validation 2011-06-09 11:35:58 +02:00
Adrià Mercader 7b61fb62bf Ignore missing config values 2011-06-07 15:32:46 +01:00
Adrià Mercader 98bfd50f47 Load config in the CKAN harvester 2011-06-07 13:35:11 +01:00
Adrià Mercader 6e75d362e3 Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method. 2011-06-07 12:07:53 +01:00
Adrià Mercader ca6af0249a Reverting previous changeset, as it conflicts with dgu_form_api 2011-06-07 11:58:35 +01:00
Friedrich Lindenberg f9c0ee37aa spacing in template paths 2011-06-06 10:16:34 +02:00
Friedrich Lindenberg 89934b8538 [harvesters] factor out a base harvester for use in generic harvesting apps 2011-06-02 12:07:07 +02:00
David Raznick 79fd966573 add database setup at configure time 2011-05-31 18:06:26 +01:00
David Raznick 264b606c48 take tables out of global scope at import time 2011-05-31 18:02:07 +01:00
Adrià Mercader 2b98080266 [merge] from new-forms, as forms refactoring has been merged in core 2011-05-20 13:50:15 +01:00
Adrià Mercader 235d822458 [ckan harvester] Request only packages modified since last harvest job. Also support older versions which do not include 'metadata_modified' 2011-05-17 17:26:42 +01:00
Adrià Mercader 565eaf3d0a Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form 2011-05-13 18:39:36 +01:00
Adrià Mercader ce71379d25 Remove references to ckanext-dgu 2011-05-13 17:08:21 +01:00
Adrià Mercader fecee82b1a Minor enhancements in the WUI 2011-05-13 17:02:18 +01:00
Adrià Mercader b3a88070e3 [forms] Adapt CLI commands to changes in lib 2011-05-13 16:00:36 +01:00
Adrià Mercader bbe459527f [forms] Major refactoring of the harvest forms. Forms no longer use the DGU form 
API, and are handled similarly to the new ones on CKAN core (logic, schema,
validators...). The UI is also more consistent with the CKAN one.
 2011-05-13 14:17:58 +01:00
Adrià Mercader 26cdc1089d Change date definitions in Harvest Objects. reference_date -> metadata_modified_date, created -> gathered 2011-05-11 17:07:05 +01:00
Adrià Mercader e1080e349e Set a flag to force harvesters to import objects 2011-05-10 17:11:12 +01:00
Adrià Mercader e320d0588f Add command to reimport existing harvest objects 2011-05-10 16:06:57 +01:00
Adrià Mercader f7c6854a1d Save reference date in Harvest Objects when harvesting CKAN instances 2011-05-10 12:57:57 +01:00
Adrià Mercader 329ca2dd29 Add a reference date to the Harvest Objects. This must be set during the harvest 
process.
 2011-05-10 11:05:44 +01:00
Adrià Mercader c697bc3350 Normalize https ports too (#736) 2011-05-09 18:47:30 +01:00
Adrià Mercader 5594f22be7 More robust URL checking (#736) 2011-05-09 14:03:46 +01:00
Adrià Mercader 0e56c0ab4f Abort pending jobs when removing sources 2011-05-05 17:13:07 +01:00
Adrià Mercader e0a1e5752d Fix redirects in view controller 2011-05-05 16:47:34 +01:00
David Read 35fdfaf58c [pip-requirements] Add carrot dependency. 2011-04-28 11:45:57 +02:00
David Read 36b93359fa [merge] from 987-common-harvesting-framework. 2011-04-28 11:14:13 +02:00
Adrià Mercader 43453b6938 Show GUID on object errors 2011-04-19 17:16:25 +01:00
David Raznick c9d43b2e4d overide default create schema 2011-04-19 15:34:56 +01:00
Adrià Mercader e3bca3ceee Add first version of the CKAN harvester [#985] 2011-04-19 14:54:59 +01:00
Adrià Mercader b5aea2d863 Allow sources with existing URLs if the existing ones are inactive 2011-04-18 17:19:04 +01:00
james bc8caebc30 [packaging] Moved templates and publics into source 2011-04-18 08:49:25 +00:00
Adrià Mercader 7e75673cff Add function to edit harvest sources 2011-04-15 18:28:58 +01:00
Adrià Mercader e3a83ea14d Source URL not unique 2011-04-15 18:28:38 +01:00
Adrià Mercader 1edf984f1b Log an error if no harvester could be found for the source 2011-04-15 16:12:47 +01:00
Adrià Mercader d92b1df8c6 Only send harvesting jobs from active sources to queue 2011-04-15 15:50:32 +01:00
Adrià Mercader f492ed9c7a Update README with harvesting interface description and instructions for running the jobs 2011-04-15 15:36:53 +01:00