f210455aef
[ckan harvester] Replace title on default extras
2012-03-13 12:38:14 +00:00
e0bef2ef9c
[base] Minor fix for harvesters without config
2012-03-12 14:46:28 +00:00
4fe38ec49d
[tests,auth] Add tests for the auth profiles
...
Note that only the tests related to the currently loaded auth
profile will be run.
2012-03-08 16:14:44 +00:00
4a7007460b
[logic] Fix broken imports
2012-03-07 17:08:17 +00:00
763f07fcad
[logic,cli] Add session to the context in cli commands
2012-03-07 15:20:49 +00:00
bf6df2dcd6
Fix merge
2012-03-07 15:04:50 +00:00
9fcaefe8ff
[ui] Fix source datasets paging
2012-03-07 15:03:33 +00:00
6cccbb61c9
Bug fix, new job count property had not been updated
2012-03-07 12:10:32 +00:00
124f3191c8
[ui] Add class to config fields so they can be hidden via CSS
2012-03-07 11:56:18 +00:00
d9cfc52643
[ui,auth] Aggregate sources by publisher on the sources list
2012-03-07 11:49:12 +00:00
97b390f3c1
[auth,logic,ui] Handle publishers on the UI
...
Add fields for publishers in the form when using the publihser auth
profile. Some changes related to the source schema.
2012-03-06 16:01:43 +00:00
aea785701f
[logic,auth] Check that users actually exist
2012-03-06 10:37:31 +00:00
f0e2521d9b
[logic,auth] Modify checks to ensure users are admins of their publishers
2012-03-06 10:16:27 +00:00
d98206858d
[plugin,auth] Check on startup if ckan is also using the publisher profile
2012-03-05 17:10:02 +00:00
2a2397c0ed
[logic,auth] Implement publisher auth profile
...
The publisher profile allows general users to handle harvest sources
based on membership to a certain group (publisher), as opposed to the
default auth profile where only sysadmins can perform any harvesting
task.
To enable it, put this directive in your ini file:
ckan.harvest.auth.profile = publisher
TODO:
* Save publisher id / user id when creating sources
* Show publisher in form and index page
2012-03-02 16:49:39 +00:00
3b68298bba
[logic,auth] Use the site user for CLI commands auth checks
2012-03-01 12:46:42 +00:00
a35eb75440
[logic,auth] Add auth logic layer
...
The first version of the auth layer is based on the current policy, i.e.
you need to be sysadmin to perform any action.
TODO: the CLI is still not working.
2012-03-01 12:02:16 +00:00
c798013752
[logic] Refactor the rest of the logic functions (create,update,delete)
2012-02-29 15:20:35 +00:00
651474e9f1
[logic] Refactor logic layer to follow CKAN core conventions
...
To make maintenance easier and better support the upcoming auth checks,
the logic layer has been refactored to mimic the structure of the one on
CKAN core: separate actions and dictize functions and logic functions
receive a context.
Only get functions are included in this commit.
2012-02-29 10:59:02 +00:00
50537a6738
Merge branch 'master' into enh-1726-harvesting-model-update
2012-02-15 12:01:15 +00:00
e03c2545ca
[ui,logic] Expose source title in the source form
2012-02-15 11:49:59 +00:00
3489a004ad
[ui] Minor tweak to support older themes
2012-02-14 17:23:17 +00:00
2990353533
[ui,logic] Expose source state (active/inactive) in the source form
2012-02-14 14:24:32 +00:00
4d7b8143b9
[lib] Renable unique constraint in url for inactive sources
2012-02-14 11:28:11 +00:00
9ed152cbea
[ckan harvester] Add support for forcing gathering of all remote packages
2012-02-03 17:54:34 +00:00
a5cf445fa6
[ #1727 ][lib] Use 'current' field in queries returning harvest objects
2012-02-02 13:20:03 +00:00
479750da09
[ #1726 ][base harvester] Set current field when importing
2012-02-02 13:18:43 +00:00
4c81c7c3a7
[ #1726 ][model] Harvest source reference compatibility
...
The 'source' property of harvest objects now comes from the actual
foreign key. For compatibility with old harvesters, an before insert
event listener has beeen added to check if the source id has been set,
and set it automatically from the job if not.
Note that this requires SQLAlchemy 0.7 (ie CKAN 1.5.1)
2012-02-01 12:52:52 +00:00
004210935a
[model] Avoid unicode warning
2012-02-01 11:10:44 +00:00
b64d97118c
[ #1726 ][model] Add scripts for populating source_id and current fields
2012-02-01 11:08:41 +00:00
d1783f5415
[model] Changes in harvest model
...
Added three changes to the harvest model:
* 'title' column in harvest_source table
* 'current' column in harvest_job table
* foreign key from harvest_object to harvest_source
Tables are checked on startup to see if they need to be updated.
TODO: populate current and harvest_source_id fields
2012-01-30 18:38:35 +00:00
f086e908bc
[model] Clearer table initialization
2012-01-30 17:09:28 +00:00
a997e45470
[lib] Ignore deleted packages in source stats
2012-01-25 17:47:35 +00:00
3a489bbb82
[ui] Cleanup sources list and details page
2012-01-24 16:55:47 +00:00
0f8c607187
[tests]: Another test moved in wholesale from dgu repo.
2012-01-11 10:35:37 +00:00
81ed69c4da
[tests]: Moved to this repo test code from dgu repo that might be useful or might not. Completely broken, but maybe be worth something.
2012-01-11 10:29:05 +00:00
a53b79c181
[ui] Show edit and refresh links in source page
2012-01-10 17:24:05 +00:00
2ad29df5c5
[lib] Fix bug: couldn't delete source conf
2012-01-10 17:15:56 +00:00
eb646b3385
[ckan harvester] Add support for defining default extras
2012-01-10 17:07:19 +00:00
ae51093213
[ckan harvester] Ignore __junk field, was causing imports to fail
2012-01-10 14:46:12 +00:00
da469ab08e
[base harvester] Custom tag munge function. TODO: check with flexible tags
2011-11-23 11:05:52 +00:00
cfaba6e1e8
[ckan harvester] Add support for sending an API key
2011-11-21 17:29:10 +00:00
0ab5c53b47
[ckan harvester] Fix typo
2011-11-18 17:53:01 +00:00
f02ee45aae
[ui] Show config options in harvest source details page
2011-11-18 14:35:46 +00:00
994590531e
[ckan harvester] Support for creating read-only packages
2011-11-18 14:30:10 +00:00
c939d90dbb
[ckan harvester] Support for defining a custom user to do the harvesting
2011-11-18 14:12:30 +00:00
2018d9e513
[ckan harvester] Support for default tags and groups
2011-11-18 13:20:41 +00:00
8ec05bc3e3
Modify import command to avoid problems with the Session
2011-11-15 11:26:24 +00:00
c04d80e27e
Use get_action function instead of directly calling the action functions
2011-10-26 17:26:18 +01:00
31dac7029a
fix to make import stage work on its own
2011-09-28 14:27:28 +01:00
63afd199a9
Add link to source documents from object errors
2011-09-08 10:27:36 +01:00
31c1ea1c21
Separate gather and object errors in reports. Add info about guid and object id in the object ones
2011-09-08 09:58:21 +01:00
c36d9bdd8e
Add new command to create new jobs for all active sources
2011-09-06 18:25:17 +01:00
dd00e98d9d
[model]: More careful about creating tables, since paster db upgrade loads the envrionment and therefore runs setup() before it does the migrations, and therefore in this instance we do not want to create the db tables.
2011-08-10 16:25:57 +01:00
7927329536
Make harvesters work with latest ckan release
2011-07-29 11:31:03 +01:00
cabbb4922d
Use API version defined in config if present
2011-07-18 17:35:32 +01:00
c867660e7d
Add docs to base harvester functions
2011-07-18 17:35:03 +01:00
54de6759fe
Fix bug with empty config
2011-06-28 15:04:40 +01:00
c80e68a12f
Ensure the correct configuration is used on each stage
2011-06-14 15:59:13 +01:00
3125bb1514
Add a check to ensure sources with no packages are reharvested
2011-06-14 12:59:48 +01:00
2ac9885150
Page packages in the harvest source details page
2011-06-14 10:27:48 +01:00
0d9d1f8096
reduce number of queries for harvest index to a less insane number. still heavy.
2011-06-13 17:36:35 +02:00
ef04ce1774
Add support for config options in CLI
2011-06-13 15:56:19 +01:00
13f2fb3b96
use hasattr for config validation
2011-06-09 11:35:58 +02:00
7b61fb62bf
Ignore missing config values
2011-06-07 15:32:46 +01:00
98bfd50f47
Load config in the CKAN harvester
2011-06-07 13:35:11 +01:00
6e75d362e3
Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method.
2011-06-07 12:07:53 +01:00
ca6af0249a
Reverting previous changeset, as it conflicts with dgu_form_api
2011-06-07 11:58:35 +01:00
f9c0ee37aa
spacing in template paths
2011-06-06 10:16:34 +02:00
89934b8538
[harvesters] factor out a base harvester for use in generic harvesting apps
2011-06-02 12:07:07 +02:00
79fd966573
add database setup at configure time
2011-05-31 18:06:26 +01:00
264b606c48
take tables out of global scope at import time
2011-05-31 18:02:07 +01:00
2b98080266
[merge] from new-forms, as forms refactoring has been merged in core
2011-05-20 13:50:15 +01:00
235d822458
[ckan harvester] Request only packages modified since last harvest job. Also support older versions which do not include 'metadata_modified'
2011-05-17 17:26:42 +01:00
565eaf3d0a
Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form
2011-05-13 18:39:36 +01:00
fecee82b1a
Minor enhancements in the WUI
2011-05-13 17:02:18 +01:00
b3a88070e3
[forms] Adapt CLI commands to changes in lib
2011-05-13 16:00:36 +01:00
bbe459527f
[forms] Major refactoring of the harvest forms. Forms no longer use the DGU form
...
API, and are handled similarly to the new ones on CKAN core (logic, schema,
validators...). The UI is also more consistent with the CKAN one.
2011-05-13 14:17:58 +01:00
26cdc1089d
Change date definitions in Harvest Objects. reference_date -> metadata_modified_date, created -> gathered
2011-05-11 17:07:05 +01:00
e1080e349e
Set a flag to force harvesters to import objects
2011-05-10 17:11:12 +01:00
e320d0588f
Add command to reimport existing harvest objects
2011-05-10 16:06:57 +01:00
f7c6854a1d
Save reference date in Harvest Objects when harvesting CKAN instances
2011-05-10 12:57:57 +01:00
329ca2dd29
Add a reference date to the Harvest Objects. This must be set during the harvest
...
process.
2011-05-10 11:05:44 +01:00
c697bc3350
Normalize https ports too ( #736 )
2011-05-09 18:47:30 +01:00
5594f22be7
More robust URL checking ( #736 )
2011-05-09 14:03:46 +01:00
0e56c0ab4f
Abort pending jobs when removing sources
2011-05-05 17:13:07 +01:00
e0a1e5752d
Fix redirects in view controller
2011-05-05 16:47:34 +01:00
43453b6938
Show GUID on object errors
2011-04-19 17:16:25 +01:00
c9d43b2e4d
overide default create schema
2011-04-19 15:34:56 +01:00
e3bca3ceee
Add first version of the CKAN harvester [ #985 ]
2011-04-19 14:54:59 +01:00
b5aea2d863
Allow sources with existing URLs if the existing ones are inactive
2011-04-18 17:19:04 +01:00
bc8caebc30
[packaging] Moved templates and publics into source
2011-04-18 08:49:25 +00:00
7e75673cff
Add function to edit harvest sources
2011-04-15 18:28:58 +01:00
e3a83ea14d
Source URL not unique
2011-04-15 18:28:38 +01:00
1edf984f1b
Log an error if no harvester could be found for the source
2011-04-15 16:12:47 +01:00
d92b1df8c6
Only send harvesting jobs from active sources to queue
2011-04-15 15:50:32 +01:00
5fb6512d82
Do not output messages when running the 'run' command
2011-04-15 15:35:19 +01:00
9018beeedb
use create_all instead of create
2011-04-15 12:19:24 +01:00
a11480aa8f
[refactoring] Use ckan.harvest for options in the ini file
2011-04-13 12:46:52 +01:00
040e4d6956
[refactoring] Add a command to create the necessary tables in the database
2011-04-13 12:39:53 +01:00
312e9b8209
[refactoring] Update view controller and templates
2011-04-12 16:15:14 +01:00
aaf7885f43
[refactoring] Return default values if provided in getters. Also return dates as strings
2011-04-12 16:13:38 +01:00
a234d6ff6a
[refactoring] Remove unneeded interface
2011-04-12 10:11:02 +01:00
842bea8433
[refactoring] Remove all geo related functions as they now have it own extension ckanext-spatial
2011-04-11 19:19:09 +01:00
280dce1626
[refactoring] Add status info to harvest sources. Updated dict functions and UI.
2011-04-11 16:30:56 +01:00
e2faa14b8d
[refactoring] Save gathering starting and finishing time
2011-04-11 16:29:47 +01:00
3c505693ff
[refactoring] Do not delete sources, just inactivate them. Also don't delete jobs.
2011-04-08 17:07:19 +01:00
2588352bc5
[refactoring] Simplify model relations
2011-04-08 16:48:29 +01:00
9465d59d3a
[refactoring] Remove old harvesting controller. This functionality now lives on ckanext-inspire
2011-04-08 15:54:33 +01:00
b38fc57ec5
[refactoring] Changes in the queue code and the IHarvester interface"
2011-04-07 16:59:11 +01:00
90ae9d27db
[refactoring] Tweaks in the model and a setup function to create the tables
2011-04-07 16:57:36 +01:00
56d5acc867
[refactoring] Add code to handle queuing and the new IHarvester interface. Add
...
new commands in the CLI to start the queue consumers and fire the harvesting
process.
2011-04-06 12:45:00 +01:00
4023bb7222
[refactoring] Use the common functions in the web interface.
...
Not yet implemented in create and edit harvest source as they use the DGU forms API.
Also TODO, think of what report info is needed in the listing and details page.
2011-04-05 13:39:23 +01:00
3d32a18802
[refactoring] Return dictionaries from the common functions, and use them in the CLI"
2011-04-05 12:55:58 +01:00
dca4ff328f
[refactoring] Add missing column in error table
2011-04-05 12:53:36 +01:00
a4d93e4ad3
[refactoring] Add a reference to objects from jobs
2011-04-05 12:36:27 +01:00
e819a68f21
[refactoring] Move common functions to lib. Adapt the CLI to use these common functions.
2011-04-05 11:53:39 +01:00
ce86cfde1b
[refactoring] Clean up the model definitions
2011-04-05 11:51:59 +01:00
d3f06b7b4f
[refactoring] Changes in model
2011-03-30 16:55:42 +01:00
0396838e36
[refactoring] First version of the new model
2011-03-30 14:42:59 +01:00
d35f86f559
Stop harvesting job if the package extent could not be saved. Still shows an ugly SA exception
2011-03-29 17:23:49 +01:00
95623b0f7a
Set format as WMS if the harvested document refers to a service
2011-03-29 16:24:14 +01:00
e855d2ad9a
Moving the extension points code to package as a .deb file
2011-03-28 15:52:43 +01:00
6a877666bd
Use the URL of a WMS type resource instead of the package URL
2011-03-28 09:40:17 +01:00
d6e35bc5cb
[controllers,tests]: Added requirement to be sysadmin for all controller operations. Provided test for this and some useful test infrastrure.
2011-03-25 17:01:26 +00:00
61e23196bf
Better handling of unauthorized access
2011-03-23 17:02:02 +00:00
0cf2c7e7ac
Nicer design for the WMS preview
2011-03-23 10:00:39 +00:00
32a180f39c
First version of the WMS viewer. The viewer is based on OpenLayers, which is used to parse the GetCapabilities response and build a map with the layers found.
2011-03-22 17:33:58 +00:00
57631a8931
Use the projection from harvested documents. Add a configure option to define the database projection
2011-03-18 17:42:43 +00:00
35537bbbcb
Update help in CLI
2011-03-18 15:48:08 +00:00
4036858ac9
Add a CLI command to create or update the geometries for package extents
2011-03-18 15:44:40 +00:00
b83443af4a
Create geometry when harvesting documents with geographic extent
2011-03-18 13:40:49 +00:00
51102c9330
Add spatial search based on PostGIS for packages with geographic extent
2011-03-18 12:05:16 +00:00
ee186376bd
Display an error if the API call fails
2011-03-15 16:19:13 +00:00
94ec121029
[merge]
2011-03-14 17:17:15 +00:00
2c8eb94344
#1030 Move harvesting controller to ckanext-harvest
2011-03-14 17:04:41 +00:00
898149bb8f
[merge]
2011-03-14 16:06:30 +00:00
52b17ffceb
[security] change public file path code to avoid a trailing , and thus adding / as a static file app in the Pylons middleware cascade
2011-03-14 16:05:34 +00:00
c2bfdd16fc
#1030 Move harvesting model to ckanext-harvest
2011-03-14 15:19:58 +00:00
544a88d18d
#1030 Move paster harvester CLI command to ckanext-harvest
2011-03-14 13:34:48 +00:00
dfd23d720c
Add option to edit harvesting sources
2011-03-11 13:42:21 +00:00
696a8dc058
Add option to delete harvesting sources
2011-03-11 12:41:13 +00:00
58164e915d
Fix typo
2011-03-11 12:38:09 +00:00
81569c0f2c
Show error notices for duplicate sources and existing jobs.
2011-03-11 12:35:27 +00:00
70bff06b48
Add refresh link to the UI.
2011-03-10 17:24:23 +00:00
4f2a233cf1
Harvesting jobs creation controller
2011-03-10 16:48:50 +00:00
6cc73aa1f7
Send the authz headers on all requests
2011-03-10 15:32:51 +00:00
ad239b93e8
Add controller to create a harvesting source
...
Before sending a create request to the forms API, an authz header is
added, with the API key defined in the config file
(ckan.harvesting.api_key). Also the data is formatted as the JSON object
expected by the forms API.
2011-03-10 14:02:21 +00:00
572650d26a
Use urllib2 and better error handling
2011-03-10 09:45:16 +00:00
66c65f53f4
First draft of the Harvesting extension
...
This extension will hold all the harvesting code that is now located in
different parts of the ckan core and ckanext.dgu. This revision still
uses the ckanext-dgu forms API.
Controllers and templates have been defined for listing, creating and
showing the details of harvesting sources.
2011-03-09 18:56:55 +00:00
amercader
David Read
David Raznick
Friedrich Lindenberg
james
james gardner