harvester-d4science

Author	SHA1	Message	Date
David Read	b0780b2062	Fetch stage can also return "unchanged", same as the import stage. Used by DGU. It is useful to skip an object like this, to avoid saving the fetched content in a HarvestObject (saves disk usage).	2015-12-01 17:38:57 +00:00
David Read	414c33ac6b	Easy fix to stop test_queue.py using MockHarvester in test_queue2.py by mistake.	2015-12-01 16:55:42 +00:00
David Read	c7fac36c1c	[#107 ] "unchanged" response tested and related fixes * fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object. * fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now. * "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that. * "unchanged" response is now documented * better report_status tests in test_queue2.	2015-11-03 00:22:53 +00:00

Author

SHA1

Message

Date

David Read

b0780b2062

Fetch stage can also return "unchanged", same as the import stage. Used by DGU. It is useful to skip an object like this, to avoid saving the fetched content in a HarvestObject (saves disk usage).

2015-12-01 17:38:57 +00:00

David Read

414c33ac6b

Easy fix to stop test_queue.py using MockHarvester in test_queue2.py by mistake.

2015-12-01 16:55:42 +00:00

David Read

c7fac36c1c

[#107 ] "unchanged" response tested and related fixes

* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.

2015-11-03 00:22:53 +00:00

3 Commits