CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
404 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

404 Commits

Author SHA1 Message Date
Adrià Mercader ca6af0249a Reverting previous changeset, as it conflicts with dgu_form_api 2011-06-07 11:58:35 +01:00
Friedrich Lindenberg f9c0ee37aa spacing in template paths 2011-06-06 10:16:34 +02:00
Friedrich Lindenberg 89934b8538 [harvesters] factor out a base harvester for use in generic harvesting apps 2011-06-02 12:07:07 +02:00
David Raznick 79fd966573 add database setup at configure time 2011-05-31 18:06:26 +01:00
David Raznick 264b606c48 take tables out of global scope at import time 2011-05-31 18:02:07 +01:00
Adrià Mercader 2b98080266 [merge] from new-forms, as forms refactoring has been merged in core 2011-05-20 13:50:15 +01:00
Adrià Mercader 235d822458 [ckan harvester] Request only packages modified since last harvest job. Also support older versions which do not include 'metadata_modified' 2011-05-17 17:26:42 +01:00
Adrià Mercader 565eaf3d0a Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form 2011-05-13 18:39:36 +01:00
Adrià Mercader ce71379d25 Remove references to ckanext-dgu 2011-05-13 17:08:21 +01:00
Adrià Mercader fecee82b1a Minor enhancements in the WUI 2011-05-13 17:02:18 +01:00
Adrià Mercader b3a88070e3 [forms] Adapt CLI commands to changes in lib 2011-05-13 16:00:36 +01:00
Adrià Mercader bbe459527f [forms] Major refactoring of the harvest forms. Forms no longer use the DGU form 
API, and are handled similarly to the new ones on CKAN core (logic, schema,
validators...). The UI is also more consistent with the CKAN one.
 2011-05-13 14:17:58 +01:00
Adrià Mercader 26cdc1089d Change date definitions in Harvest Objects. reference_date -> metadata_modified_date, created -> gathered 2011-05-11 17:07:05 +01:00
Adrià Mercader e1080e349e Set a flag to force harvesters to import objects 2011-05-10 17:11:12 +01:00
Adrià Mercader e320d0588f Add command to reimport existing harvest objects 2011-05-10 16:06:57 +01:00
Adrià Mercader f7c6854a1d Save reference date in Harvest Objects when harvesting CKAN instances 2011-05-10 12:57:57 +01:00
Adrià Mercader 329ca2dd29 Add a reference date to the Harvest Objects. This must be set during the harvest 
process.
 2011-05-10 11:05:44 +01:00
Adrià Mercader c697bc3350 Normalize https ports too (#736) 2011-05-09 18:47:30 +01:00
Adrià Mercader 5594f22be7 More robust URL checking (#736) 2011-05-09 14:03:46 +01:00
Adrià Mercader 0e56c0ab4f Abort pending jobs when removing sources 2011-05-05 17:13:07 +01:00
Adrià Mercader e0a1e5752d Fix redirects in view controller 2011-05-05 16:47:34 +01:00
David Read 35fdfaf58c [pip-requirements] Add carrot dependency. 2011-04-28 11:45:57 +02:00
David Read 36b93359fa [merge] from 987-common-harvesting-framework. 2011-04-28 11:14:13 +02:00
Adrià Mercader 43453b6938 Show GUID on object errors 2011-04-19 17:16:25 +01:00
David Raznick c9d43b2e4d overide default create schema 2011-04-19 15:34:56 +01:00
Adrià Mercader e3bca3ceee Add first version of the CKAN harvester [#985] 2011-04-19 14:54:59 +01:00
Adrià Mercader b5aea2d863 Allow sources with existing URLs if the existing ones are inactive 2011-04-18 17:19:04 +01:00
james bc8caebc30 [packaging] Moved templates and publics into source 2011-04-18 08:49:25 +00:00
Adrià Mercader 7e75673cff Add function to edit harvest sources 2011-04-15 18:28:58 +01:00
Adrià Mercader e3a83ea14d Source URL not unique 2011-04-15 18:28:38 +01:00
Adrià Mercader 1edf984f1b Log an error if no harvester could be found for the source 2011-04-15 16:12:47 +01:00
Adrià Mercader d92b1df8c6 Only send harvesting jobs from active sources to queue 2011-04-15 15:50:32 +01:00
Adrià Mercader f492ed9c7a Update README with harvesting interface description and instructions for running the jobs 2011-04-15 15:36:53 +01:00
Adrià Mercader 5fb6512d82 Do not output messages when running the 'run' command 2011-04-15 15:35:19 +01:00
David Raznick 9018beeedb use create_all instead of create 2011-04-15 12:19:24 +01:00
Adrià Mercader a11480aa8f [refactoring] Use ckan.harvest for options in the ini file 2011-04-13 12:46:52 +01:00
Adrià Mercader 040e4d6956 [refactoring] Add a command to create the necessary tables in the database 2011-04-13 12:39:53 +01:00
Adrià Mercader 312e9b8209 [refactoring] Update view controller and templates 2011-04-12 16:15:14 +01:00
Adrià Mercader aaf7885f43 [refactoring] Return default values if provided in getters. Also return dates as strings 2011-04-12 16:13:38 +01:00
Adrià Mercader a234d6ff6a [refactoring] Remove unneeded interface 2011-04-12 10:11:02 +01:00
Adrià Mercader 842bea8433 [refactoring] Remove all geo related functions as they now have it own extension ckanext-spatial 2011-04-11 19:19:09 +01:00
Adrià Mercader 280dce1626 [refactoring] Add status info to harvest sources. Updated dict functions and UI. 2011-04-11 16:30:56 +01:00
Adrià Mercader e2faa14b8d [refactoring] Save gathering starting and finishing time 2011-04-11 16:29:47 +01:00
Adrià Mercader 3c505693ff [refactoring] Do not delete sources, just inactivate them. Also don't delete jobs. 2011-04-08 17:07:19 +01:00
Adrià Mercader 2588352bc5 [refactoring] Simplify model relations 2011-04-08 16:48:29 +01:00
Adrià Mercader 9465d59d3a [refactoring] Remove old harvesting controller. This functionality now lives on ckanext-inspire 2011-04-08 15:54:33 +01:00
Adrià Mercader b38fc57ec5 [refactoring] Changes in the queue code and the IHarvester interface" 2011-04-07 16:59:11 +01:00
Adrià Mercader 90ae9d27db [refactoring] Tweaks in the model and a setup function to create the tables 2011-04-07 16:57:36 +01:00
Adrià Mercader 56d5acc867 [refactoring] Add code to handle queuing and the new IHarvester interface. Add 
new commands in the CLI to start the queue consumers and fire the harvesting
process.
 2011-04-06 12:45:00 +01:00
ww f3c84a6dba mention requirements explicitly in setup.py 2011-04-05 22:39:33 +01:00