[#42] Remove non-string extras from packages

Remove extras whose values are not strings (e.g. dicts, lists..) from
packages before attempting to create or update the packages on the
target site.

In CKAN 1 it was possible for the values of extras to be other types,
but in CKAN 2 they must be strings, so when harvesting from a CKAN 1 site
into a CKAN 2 site SQLAlchemy would crash when trying to create packages
with non-string extras.

The fix in this commit is to simply remove any non-string extras from
the harvested package. (Alternatively, we could try to convert them to a
string using JSON.)

Fixes #42.
This commit is contained in:
Sean Hammond 2013-05-31 14:56:53 +02:00
parent 0609262731
commit 85a013f2c9
1 changed files with 6 additions and 0 deletions

View File

@ -254,6 +254,12 @@ class CKANHarvester(HarvesterBase):
package_dict['groups'] = []
package_dict['groups'].extend([g for g in default_groups if g not in package_dict['groups']])
# Delete any extras whose values are not strings, as these cause
# errors from CKAN when trying to create/update the package.
for key in package_dict['extras'].keys():
if not isinstance(package_dict['extras'][key], basestring):
del package_dict['extras'][key]
# Set default extras if needed
default_extras = self.config.get('default_extras',{})
if default_extras: