9b3199b41b
[ #65 ] remove unused code
2013-09-17 17:02:38 +01:00
5da153c6b6
[ #65 ] harvest_object_create action
...
update to use schema and validators. Also accept more parameters to
data_dict.
2013-09-17 16:49:19 +01:00
71aedf3fd4
Makes ede45bd work in both CKAN 2.0 and 2.1
2013-09-05 15:36:00 +01:00
ede45bd1be
Fixes #66 by correcting the block name
2013-09-05 15:31:44 +01:00
1b663bbff4
add harvest_object_create action
2013-09-04 14:17:01 +01:00
52956feab9
Merge branch '62-default-package-name'
2013-08-19 18:23:37 +01:00
f51b8e905a
[ #58 ] Check properly for version numbers (patch numbers)
2013-08-19 18:13:01 +01:00
f028375ad3
[ #62 ] Use current name when updating package, if the user haven't sent a new one
...
It's hard for someone outside CKAN to make sure they're sending it in the format
we expect. And they'll also have to keep track of our name format, to keep in
sync whenever we change.
To fix this, we simply do what we already do when creating packages: use a
default name. In this case, the current one.
2013-08-18 12:08:30 -03:00
86dcd933ea
Merged master
2013-08-15 18:47:16 +01:00
712e150b52
[ #58 ] Fix to make merge nice
2013-08-15 18:43:46 +01:00
575df637b4
[ #58 ] Fixes to make harvest templates to work with both CKAN 2.0 and 2.1
2013-08-15 16:45:02 +01:00
05e6362c38
Merge branch 'fix-jinja-status-exception' of git://github.com/metaodi/ckanext-harvest into metaodi-fix-jinja-status-exception
2013-08-15 14:39:20 +01:00
01ca5c0dfd
[ #61 ] Ignore harvest sources on the CKAN harvester
2013-08-15 14:38:33 +01:00
b25fffda93
[ #36 ] Fix bug on API version checking
2013-08-15 14:37:55 +01:00
39ad78d90a
[ #59 ] Ignore auth in the CKAN harvester
2013-08-15 14:37:12 +01:00
f26baf6c09
Hide both the label and the number of datasets when 'status' is not available
2013-08-15 13:25:16 +02:00
1c36b33aaf
[ #59 ] Ignore auth when using site_user
2013-08-14 12:28:27 +01:00
ffea49ca62
[ #56 ] Update parameters on source create command
...
Add missing title and owner_org fields, remove deprecated user_id and
publisher_id
2013-08-14 11:54:51 +01:00
3494727d3f
[ #56 ] Increase max params number
2013-08-14 11:43:32 +01:00
8e33262026
[ #56 ] Fix syntax error and wrong type
2013-08-14 11:31:23 +01:00
4dfd091aec
Make the /harvest page more robust if source.status is not set
...
This prevents exceptions from appearing in the log from Jinja:
[error] [client 1.2.3.4] Error - <class 'jinja2.exceptions.UndefinedError'>: 'dict object' has no attribute 'status'
2013-08-14 11:52:11 +02:00
7ae9d6e208
Made print method more robust against KeyErrors
...
This is especially needed if you create a new harvest source which does not have all the optional arguments. Before this lead to a KeyError after the creation of the source. Now this simply output 'None'.
2013-08-05 23:50:30 +02:00
1249564be5
Add additional name argument when creating new harvest source
2013-08-05 23:46:21 +02:00
ade5f83e38
Change key of data_dict from 'type' to 'source_type'
2013-08-05 23:07:25 +02:00
cb745c3c3e
Avoid importing unnecessary functions from the harvest logic
2013-08-05 18:39:44 +01:00
70e53a7833
Fix bug where source was being treated as an object, when it's a dict
2013-07-29 07:06:58 -03:00
cc3f3d3426
[ #50 ] Fix objects deletion on gather exceptions
2013-07-05 13:29:11 +01:00
e2696b98bb
[ #50 ] Save all dates as UTC in the database
...
At some point we may want to transform these to local time at the
dictization level. We will need a library like dateutil to handle it
properly though.
2013-07-04 14:59:27 +01:00
c2283e3fdb
only migrate harvest sources which are active
2013-06-28 02:32:45 +01:00
a42991b8c9
fix so that non sysadmins can edit harvest sources of organizations they
...
are admins or editors of.
2013-06-27 12:16:11 +01:00
6540726c47
use correct limit for paging harvest listing
2013-06-26 11:14:38 +01:00
584c340583
Merge branch '42-remove-non-string-extras'
2013-06-03 10:33:59 +01:00
01df3a1db4
[ #42 ] Dump non-string extras with json
...
Convert any non-string extra values to strings using json.dumps(),
instead of just deleting them.
2013-05-31 20:35:06 +02:00
3a31db59b6
[ #36 ] Move validation code to validate_config
...
This ensures it is checked whenever the source is edited or created.
2013-05-31 17:23:40 +01:00
a6a0196a4e
Merge branch 'api-version-fix' of git://github.com/fraunhoferfokus/ckanext-harvest into fraunhoferfokus-api-version-fix
2013-05-31 17:15:43 +01:00
85a013f2c9
[ #42 ] Remove non-string extras from packages
...
Remove extras whose values are not strings (e.g. dicts, lists..) from
packages before attempting to create or update the packages on the
target site.
In CKAN 1 it was possible for the values of extras to be other types,
but in CKAN 2 they must be strings, so when harvesting from a CKAN 1 site
into a CKAN 2 site SQLAlchemy would crash when trying to create packages
with non-string extras.
The fix in this commit is to simply remove any non-string extras from
the harvested package. (Alternatively, we could try to convert them to a
string using JSON.)
Fixes #42 .
2013-05-31 15:43:42 +02:00
361abcfc07
[ #17 ] Fix bug with remote groups handling
...
If neither 'only_local' or 'create' are used the remote groups property
needs to be removed, otherwise it causes an exception when the group is
not found.
2013-05-30 18:06:15 +01:00
87cae31c75
Fix api_version check in the group importer code
...
I have forgotten to update one check for the api_version 1 in the code
responsible for the remote group import feature. This commit fixes that.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 13:36:56 +02:00
c858b9fe9f
Add exception handling for the API version parsing
...
I have added try-except clauses in order to prevent the process from
crashing if a non-parsable integer is used for the api_version option.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 13:12:05 +02:00
05094090af
Change type of the API version to integer
...
The CKAN logic uses integers when dealing with the API version, e.g.
making checks which API version is in use. Currently, the harvester
uses strings to identify the API version. Instead of dealing with
type conversion the harvester could use integers directly.
This commit fixes okfn/ckanext-harvest#36 . When the API version is
parsed from the configuration it is passed through the int() function.
This way the harvesting will still work even if a harvest source was
configured with a string API version which makes this commit backward
compatible.
Signed-off-by: Konrad Reiche <konrad.reiche@fokus.fraunhofer.de>
2013-05-27 12:51:48 +02:00
ff7287d4b4
[ #30 ] Remove lxml dependency
2013-05-24 18:12:02 +01:00
3d2867ca04
[ #17 ] Remove ckanclient dependency as it is not used
2013-05-24 17:55:37 +01:00
f1d11c1307
[ #17 ] Import remote groups in CKAN harvester
...
This is a cleaner commit of the great work done by @platzhirsch
implementing remote groups import on the CKAN harvester.
2013-05-24 16:55:05 +01:00
1792180e4f
Better harvest source dataset migration
...
Current implementation only checked for the first source to exist and
didn't allow to rerun the migration for other sources if there was an
error. With the new one, all non existing sources are migrated each
time.
2013-05-24 14:49:55 +01:00
1d54edfdaa
Fix bug in source datasets migration
...
Wrong dataset type was causing the default package schema to be used,
thus failing when providing an id.
2013-05-24 14:25:05 +01:00
751409ab7d
[ #34 ] Integrate clear command with delete source
...
When deleting a source, if clear_source equals true in the context,
harvest_source_clear will be called. Default is false. The UI shows a
select with the two options.
2013-05-20 14:30:22 +01:00
6d5d0fbaae
Add hover helper text to refresh and clear buttons
2013-05-20 12:09:14 +01:00
edfc49719b
Use page_heading helper consistently with the main CKAN templates.
2013-05-17 16:12:57 +01:00
d0bc52f2d8
[ #34 ] Fix typo in warning message
2013-05-16 18:07:32 +01:00
b9e2613458
[ #34 ] Allow all authorized users for a source to clear it
2013-05-16 17:57:59 +01:00
71349e658b
[ #34 ] Expose harvest source clear button
2013-05-16 17:51:48 +01:00
7b652542e7
[ #34 ] Fix harvest_source_clear action
...
Some typos in the SQL statements, and also the source needs to be
reindexed to update the status with the counts.
2013-05-16 17:33:39 +01:00
1efd7ab4cd
Ignore remote orgs in CKAN harvester
...
If #17 progresses we can do somethign similar for them, although it amy
be more complicated because of authorization issues.
2013-05-16 17:30:54 +01:00
1714e55110
simplify harvest_clear queries so they do not lock on big db
2013-04-30 13:59:23 +01:00
a2b8ab1994
make harvest source clear not create table
2013-04-30 12:40:46 +01:00
9041f3f3ad
Changes in Redis conusmer to make tests work
2013-04-22 18:08:19 +01:00
70dfee1a36
Update queue tests
2013-04-22 17:56:11 +01:00
dcfd201cdd
[ #32 ] redis queue support
2013-04-21 17:04:57 +01:00
0ce59a29b6
delete insead of update harvest objects when error
2013-04-12 12:32:33 +01:00
7d7657f94a
make gather phase as finished if there is an error
2013-04-12 10:35:08 +01:00
bd761498f0
make sure config dict is not jsonified if it contains an error
2013-04-08 18:52:36 +01:00
eaebeb4e6e
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-04-08 13:25:33 +01:00
5414b6c08d
Merge branch '29-new-idataset-form' into release-v2.0
2013-04-08 13:23:41 +01:00
66ff773f99
remove previous commit import
2013-03-29 12:47:14 +00:00
3ac065f0f0
fix package_schema import
2013-03-29 01:17:24 +00:00
cb8b808274
sanity check that harvest source id matches harvest dataset id
...
remove author_email, license_id, maintainer_email, maintainer and
author from pacakge_dict, these were not actually necessary
2013-03-29 00:59:20 +00:00
99bd17401c
Handle wrong JSON in harvest_source_extra_validator
2013-03-28 16:19:16 +00:00
a9b8be8f01
harvest source index clear
2013-03-28 15:36:44 +00:00
95ebb5bbf3
[ #29 ] Remove check_data_dict ✨
2013-03-28 15:01:21 +00:00
fbc8ecde97
[ #29 ] Fix some imports on actions and plugin
2013-03-28 15:00:44 +00:00
c754479014
#29 make new idatasets form work with harvest source form
2013-03-25 17:38:07 +00:00
548d3c1c2a
fix validation issue on db upgrade
2013-03-25 12:02:07 +00:00
b5a697ec87
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-25 11:58:31 +00:00
0b5c3c608a
catch and raise gather exception, acking the message
2013-03-25 11:57:57 +00:00
438ba672e2
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-03-25 11:44:37 +00:00
845c9927a8
add harvest source clear
2013-03-25 11:39:00 +00:00
d518b6709a
[ #27 ] fix package_list_for_source for unowned data sources
2013-03-21 15:59:22 +00:00
7bff041568
[ #25 ] Further tweaks on helpers texts
2013-03-21 13:47:23 +00:00
4d0dd9a4d3
[ #25 ] Small copy tweak to confirmation dialog
2013-03-21 12:14:33 +00:00
78bde974b9
[ #25 ] Adds confirmation dialog to reharvest button
2013-03-21 10:56:39 +00:00
3197162fe6
[ #25 ] Changed 'Refresh' to 'Reharvest' on button
2013-03-21 10:36:12 +00:00
a7583a7b8b
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-21 02:32:11 +00:00
b676fb02e1
only get out harvest items in interface and when indexing
2013-03-21 02:31:34 +00:00
15c44d9aa8
Merge branch '23-harvest-form-cleanup' into release-v2.0
2013-03-20 17:03:41 +00:00
4ba298fe58
[ #23 ] Make labels a little wider on harvest new form
2013-03-20 14:07:03 +00:00
02e90767f4
Fix source listing in organization page
...
It needed update after #515 in ckan core
2013-03-20 13:01:23 +00:00
86355fb9db
[ #23 ] Form cleanup after core bootstrap upgrade
2013-03-20 10:44:24 +00:00
634a0bbd30
return instead of continue
2013-03-19 01:21:20 +00:00
3adf38105e
readd code from old branch seperating the fetch and import logic
2013-03-19 01:16:43 +00:00
c2a6bd14eb
Add auth function for harvest_source_show_status
2013-03-18 16:48:27 +00:00
c76b7d95f3
Only count public datasets on the source status
...
This is more in line with what is done on the orgs/groups pages
2013-03-18 16:41:01 +00:00
cb80ac784e
Add logic to show private datasets to authorizaded users
2013-03-18 16:29:29 +00:00
341331ac53
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2013-03-14 17:33:21 +00:00
d77f16aba9
[ #21 ] Improve gather stage error handling
...
See issue for full details. Basically we don't want to catch any
exception at the queue.py level, as they prevent debugging. Harvesters
should deal with them and return a list of ids or an empty list if no
objects need to be fetched.
Also improved the debug messages.
2013-03-14 17:31:07 +00:00
b30cc54427
Fix for add harvest source button within org
2013-03-14 14:45:54 +00:00
91f18bffab
Fix pagination on org sources listing
2013-03-14 11:44:38 +00:00
8cac0977aa
Fix import on org sources listing
2013-03-14 11:44:22 +00:00
cd6c1b56a8
[ #18 ] Get package dict on after_delete to check type
...
No need for #615 in core then
2013-03-13 17:31:39 +00:00
1b11b00946
[ #18 ] Fix wrong logic for setting the source active field
2013-03-13 13:19:43 +00:00
cb5e06119e
Merge branch 'release-v2.0' of https://github.com/okfn/ckanext-harvest into release-v2.0
2013-03-12 23:31:58 +00:00
joetsoi
John Martin
amercader
Vitor Baptista
Stefan Oderbolz
kindly
Sean Hammond
Konrad Reiche
Tom Rees