CatalogueRevamping
/
harvester-d4science
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
828 Commits 3 Branches 0 Tags 2.5 MiB
Commit Graph

11 Commits

Author SHA1 Message Date
David Read f67029b993 Hint about rerunning the import_stage. 2015-12-01 17:59:11 +00:00
David Read b8c7d39e1a Document existing functionality for aborting during the gather stage. 2015-12-01 17:56:24 +00:00
David Read 25301e2152 Update README from interfaces.py. PEP8. Mention HarvestObject - Package relation in interface. 2015-12-01 17:51:35 +00:00
David Read b0780b2062 Fetch stage can also return "unchanged", same as the import stage. Used by DGU. It is useful to skip an object like this, to avoid saving the fetched content in a HarvestObject (saves disk usage). 2015-12-01 17:38:57 +00:00
David Read c7fac36c1c [#107] "unchanged" response tested and related fixes 
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
 2015-11-03 00:22:53 +00:00
amercader 8431182f01 Document method and cleanup the interface file 2013-01-24 18:39:19 +00:00
Adrià Mercader 565eaf3d0a Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form 2011-05-13 18:39:36 +01:00
Adrià Mercader 329ca2dd29 Add a reference date to the Harvest Objects. This must be set during the harvest 
process.
 2011-05-10 11:05:44 +01:00
Adrià Mercader 280dce1626 [refactoring] Add status info to harvest sources. Updated dict functions and UI. 2011-04-11 16:30:56 +01:00
Adrià Mercader b38fc57ec5 [refactoring] Changes in the queue code and the IHarvester interface" 2011-04-07 16:59:11 +01:00
Adrià Mercader 56d5acc867 [refactoring] Add code to handle queuing and the new IHarvester interface. Add 
new commands in the CLI to start the queue consumers and fire the harvesting
process.
 2011-04-06 12:45:00 +01:00