2ac9885150
Page packages in the harvest source details page
2011-06-14 10:27:48 +01:00
0d9d1f8096
reduce number of queries for harvest index to a less insane number. still heavy.
2011-06-13 17:36:35 +02:00
ef04ce1774
Add support for config options in CLI
2011-06-13 15:56:19 +01:00
13f2fb3b96
use hasattr for config validation
2011-06-09 11:35:58 +02:00
7b61fb62bf
Ignore missing config values
2011-06-07 15:32:46 +01:00
98bfd50f47
Load config in the CKAN harvester
2011-06-07 13:35:11 +01:00
6e75d362e3
Add a simple way for harvesters to store configuration options. If form_config_interface is Text on the info dictionary, the configuration field will be enabled in the form. Harvesters can also provide a validate_config method.
2011-06-07 12:07:53 +01:00
ca6af0249a
Reverting previous changeset, as it conflicts with dgu_form_api
2011-06-07 11:58:35 +01:00
f9c0ee37aa
spacing in template paths
2011-06-06 10:16:34 +02:00
89934b8538
[harvesters] factor out a base harvester for use in generic harvesting apps
2011-06-02 12:07:07 +02:00
79fd966573
add database setup at configure time
2011-05-31 18:06:26 +01:00
264b606c48
take tables out of global scope at import time
2011-05-31 18:02:07 +01:00
2b98080266
[merge] from new-forms, as forms refactoring has been merged in core
2011-05-20 13:50:15 +01:00
235d822458
[ckan harvester] Request only packages modified since last harvest job. Also support older versions which do not include 'metadata_modified'
2011-05-17 17:26:42 +01:00
565eaf3d0a
Add a new info method to the harvester interface so implementations can provide details. Use this to build the WUI form
2011-05-13 18:39:36 +01:00
fecee82b1a
Minor enhancements in the WUI
2011-05-13 17:02:18 +01:00
b3a88070e3
[forms] Adapt CLI commands to changes in lib
2011-05-13 16:00:36 +01:00
bbe459527f
[forms] Major refactoring of the harvest forms. Forms no longer use the DGU form
...
API, and are handled similarly to the new ones on CKAN core (logic, schema,
validators...). The UI is also more consistent with the CKAN one.
2011-05-13 14:17:58 +01:00
26cdc1089d
Change date definitions in Harvest Objects. reference_date -> metadata_modified_date, created -> gathered
2011-05-11 17:07:05 +01:00
e1080e349e
Set a flag to force harvesters to import objects
2011-05-10 17:11:12 +01:00
e320d0588f
Add command to reimport existing harvest objects
2011-05-10 16:06:57 +01:00
f7c6854a1d
Save reference date in Harvest Objects when harvesting CKAN instances
2011-05-10 12:57:57 +01:00
329ca2dd29
Add a reference date to the Harvest Objects. This must be set during the harvest
...
process.
2011-05-10 11:05:44 +01:00
c697bc3350
Normalize https ports too ( #736 )
2011-05-09 18:47:30 +01:00
5594f22be7
More robust URL checking ( #736 )
2011-05-09 14:03:46 +01:00
0e56c0ab4f
Abort pending jobs when removing sources
2011-05-05 17:13:07 +01:00
e0a1e5752d
Fix redirects in view controller
2011-05-05 16:47:34 +01:00
43453b6938
Show GUID on object errors
2011-04-19 17:16:25 +01:00
c9d43b2e4d
overide default create schema
2011-04-19 15:34:56 +01:00
e3bca3ceee
Add first version of the CKAN harvester [ #985 ]
2011-04-19 14:54:59 +01:00
b5aea2d863
Allow sources with existing URLs if the existing ones are inactive
2011-04-18 17:19:04 +01:00
bc8caebc30
[packaging] Moved templates and publics into source
2011-04-18 08:49:25 +00:00
7e75673cff
Add function to edit harvest sources
2011-04-15 18:28:58 +01:00
e3a83ea14d
Source URL not unique
2011-04-15 18:28:38 +01:00
1edf984f1b
Log an error if no harvester could be found for the source
2011-04-15 16:12:47 +01:00
d92b1df8c6
Only send harvesting jobs from active sources to queue
2011-04-15 15:50:32 +01:00
5fb6512d82
Do not output messages when running the 'run' command
2011-04-15 15:35:19 +01:00
9018beeedb
use create_all instead of create
2011-04-15 12:19:24 +01:00
a11480aa8f
[refactoring] Use ckan.harvest for options in the ini file
2011-04-13 12:46:52 +01:00
040e4d6956
[refactoring] Add a command to create the necessary tables in the database
2011-04-13 12:39:53 +01:00
312e9b8209
[refactoring] Update view controller and templates
2011-04-12 16:15:14 +01:00
aaf7885f43
[refactoring] Return default values if provided in getters. Also return dates as strings
2011-04-12 16:13:38 +01:00
a234d6ff6a
[refactoring] Remove unneeded interface
2011-04-12 10:11:02 +01:00
842bea8433
[refactoring] Remove all geo related functions as they now have it own extension ckanext-spatial
2011-04-11 19:19:09 +01:00
280dce1626
[refactoring] Add status info to harvest sources. Updated dict functions and UI.
2011-04-11 16:30:56 +01:00
e2faa14b8d
[refactoring] Save gathering starting and finishing time
2011-04-11 16:29:47 +01:00
3c505693ff
[refactoring] Do not delete sources, just inactivate them. Also don't delete jobs.
2011-04-08 17:07:19 +01:00
2588352bc5
[refactoring] Simplify model relations
2011-04-08 16:48:29 +01:00
9465d59d3a
[refactoring] Remove old harvesting controller. This functionality now lives on ckanext-inspire
2011-04-08 15:54:33 +01:00
b38fc57ec5
[refactoring] Changes in the queue code and the IHarvester interface"
2011-04-07 16:59:11 +01:00
90ae9d27db
[refactoring] Tweaks in the model and a setup function to create the tables
2011-04-07 16:57:36 +01:00
56d5acc867
[refactoring] Add code to handle queuing and the new IHarvester interface. Add
...
new commands in the CLI to start the queue consumers and fire the harvesting
process.
2011-04-06 12:45:00 +01:00
4023bb7222
[refactoring] Use the common functions in the web interface.
...
Not yet implemented in create and edit harvest source as they use the DGU forms API.
Also TODO, think of what report info is needed in the listing and details page.
2011-04-05 13:39:23 +01:00
3d32a18802
[refactoring] Return dictionaries from the common functions, and use them in the CLI"
2011-04-05 12:55:58 +01:00
dca4ff328f
[refactoring] Add missing column in error table
2011-04-05 12:53:36 +01:00
a4d93e4ad3
[refactoring] Add a reference to objects from jobs
2011-04-05 12:36:27 +01:00
e819a68f21
[refactoring] Move common functions to lib. Adapt the CLI to use these common functions.
2011-04-05 11:53:39 +01:00
ce86cfde1b
[refactoring] Clean up the model definitions
2011-04-05 11:51:59 +01:00
d3f06b7b4f
[refactoring] Changes in model
2011-03-30 16:55:42 +01:00
0396838e36
[refactoring] First version of the new model
2011-03-30 14:42:59 +01:00
d35f86f559
Stop harvesting job if the package extent could not be saved. Still shows an ugly SA exception
2011-03-29 17:23:49 +01:00
95623b0f7a
Set format as WMS if the harvested document refers to a service
2011-03-29 16:24:14 +01:00
e855d2ad9a
Moving the extension points code to package as a .deb file
2011-03-28 15:52:43 +01:00
6a877666bd
Use the URL of a WMS type resource instead of the package URL
2011-03-28 09:40:17 +01:00
d6e35bc5cb
[controllers,tests]: Added requirement to be sysadmin for all controller operations. Provided test for this and some useful test infrastrure.
2011-03-25 17:01:26 +00:00
61e23196bf
Better handling of unauthorized access
2011-03-23 17:02:02 +00:00
0cf2c7e7ac
Nicer design for the WMS preview
2011-03-23 10:00:39 +00:00
32a180f39c
First version of the WMS viewer. The viewer is based on OpenLayers, which is used to parse the GetCapabilities response and build a map with the layers found.
2011-03-22 17:33:58 +00:00
57631a8931
Use the projection from harvested documents. Add a configure option to define the database projection
2011-03-18 17:42:43 +00:00
35537bbbcb
Update help in CLI
2011-03-18 15:48:08 +00:00
4036858ac9
Add a CLI command to create or update the geometries for package extents
2011-03-18 15:44:40 +00:00
b83443af4a
Create geometry when harvesting documents with geographic extent
2011-03-18 13:40:49 +00:00
51102c9330
Add spatial search based on PostGIS for packages with geographic extent
2011-03-18 12:05:16 +00:00
ee186376bd
Display an error if the API call fails
2011-03-15 16:19:13 +00:00
94ec121029
[merge]
2011-03-14 17:17:15 +00:00
2c8eb94344
#1030 Move harvesting controller to ckanext-harvest
2011-03-14 17:04:41 +00:00
898149bb8f
[merge]
2011-03-14 16:06:30 +00:00
52b17ffceb
[security] change public file path code to avoid a trailing , and thus adding / as a static file app in the Pylons middleware cascade
2011-03-14 16:05:34 +00:00
c2bfdd16fc
#1030 Move harvesting model to ckanext-harvest
2011-03-14 15:19:58 +00:00
544a88d18d
#1030 Move paster harvester CLI command to ckanext-harvest
2011-03-14 13:34:48 +00:00
dfd23d720c
Add option to edit harvesting sources
2011-03-11 13:42:21 +00:00
696a8dc058
Add option to delete harvesting sources
2011-03-11 12:41:13 +00:00
58164e915d
Fix typo
2011-03-11 12:38:09 +00:00
81569c0f2c
Show error notices for duplicate sources and existing jobs.
2011-03-11 12:35:27 +00:00
70bff06b48
Add refresh link to the UI.
2011-03-10 17:24:23 +00:00
4f2a233cf1
Harvesting jobs creation controller
2011-03-10 16:48:50 +00:00
6cc73aa1f7
Send the authz headers on all requests
2011-03-10 15:32:51 +00:00
ad239b93e8
Add controller to create a harvesting source
...
Before sending a create request to the forms API, an authz header is
added, with the API key defined in the config file
(ckan.harvesting.api_key). Also the data is formatted as the JSON object
expected by the forms API.
2011-03-10 14:02:21 +00:00
572650d26a
Use urllib2 and better error handling
2011-03-10 09:45:16 +00:00
66c65f53f4
First draft of the Harvesting extension
...
This extension will hold all the harvesting code that is now located in
different parts of the ckan core and ckanext.dgu. This revision still
uses the ckanext-dgu forms API.
Controllers and templates have been defined for listing, creating and
showing the details of harvesting sources.
2011-03-09 18:56:55 +00:00
Adrià Mercader
Friedrich Lindenberg
David Raznick
james
james gardner
David Read