d57e73458a
Make harvest object - package FK deferrable
...
Allows eg to add the harvest object id to the package dict before
indexing.
2012-12-13 18:21:40 +00:00
b424ba1cea
Add flag to avoid returning all objects when getting a job
2012-12-13 18:20:49 +00:00
0dde483992
Set job status to Finished when actually finishing it
...
Until now, harvest jobs were set to Finished just after sending all
objects to the fetch stage. Now every time the run command is run, jobs
are set to Running, and all previous Running jobs are checked to see if
all harvest objects have a state of Complete or Error. Only then the job
is flagged as Finished.
2012-12-13 18:19:22 +00:00
81c3881a1a
Add active field to source dict
2012-12-13 18:00:07 +00:00
37efb3b978
Set harvest object state depending on the output of import_stage
...
Either to COMPLETE or ERROR, depending on whether it returns True or
False.
2012-12-13 14:30:13 +00:00
4da64a84ae
Add more elements to the harvest sources page (still provisional)
2012-12-12 18:49:38 +00:00
e0f3d47cb9
Add extra information to the harvest source page
...
The status object gives extra information about the source and there is
a helper function to build the dataset list for this particular source.
TODO: Pager still needs fixing.
2012-12-12 11:54:50 +00:00
b567e562f4
Add after_show extension point
...
We hook into the package_show extension point in order to:
1. For harvest_source type datasets, add extra information about the
source, jobs, etc (calling harvest_source_show_status)
2. For normal datasets, check if they were harvested, and if so, add a
reference to the harvest object and harvest source.
2012-12-12 11:49:55 +00:00
2557636994
Update endpoints to receive the context object
2012-12-12 11:47:57 +00:00
8e1621731b
Move harvest source status function as a logic function
...
The status dict is added automatically to harvest source packages.
Note that the actual queries still need to be updated as they proabably
won't scale.
2012-12-12 11:45:13 +00:00
b0407bb2ac
Update harvest_source_show logic function
2012-12-11 12:49:05 +00:00
fcbe6aa6de
Script for creating harvest source datasets on old versions
...
The way we check whether datasets need to be created might need to be
improved.
2012-12-05 18:54:28 +00:00
22ec9cb5af
Fix old controller import
2012-12-05 18:53:35 +00:00
697933f8d0
Add custom harvest source read page (provisional)
2012-12-05 15:47:02 +00:00
2dba7fbf78
Add custom harvest sources search page
2012-12-05 14:51:20 +00:00
a605564a41
Fix links to harvest sources page
2012-12-05 13:01:56 +00:00
d77bf255b4
Finish up create and edit forms, including breadcrumbs, links, etc
2012-11-30 18:53:13 +00:00
9d83322591
Fix config validator and add tests
2012-11-30 17:02:06 +00:00
803b228d1c
Update harvest source create and update logic functions
...
`harvest_source_create` and `harvest_source_update` now call
`package_create` and `package_update` respectively, making sure to
define a 'harvest_source' type. The returned dict uses the db_to_form
schema.
2012-11-30 14:11:24 +00:00
0e0aed0503
Clean up schemas
...
Better naming, remove old ones, ignore __extras field
2012-11-30 13:20:37 +00:00
875a773f1c
Check if type property is actually there
2012-11-30 11:10:21 +00:00
7db09fceb0
Various fixes for the harvest source dataset type forms
...
Add a db to form schema to show the fields stored in extras. Validate
the source url on the Package object.
2012-11-29 16:57:20 +00:00
ab7a379058
Behind the scenes creation and updating of HarvestSource objects
...
Taking advantage of the new after_create/after_update extensions points,
the extension checks if the dataset type is harvest source and creates
or updates the corresponding HarvestSource object. When creating a new
one, it will use the same id as the dataset.
2012-11-29 16:48:44 +00:00
9d36fd6841
First stub of the new dataset type forms
...
Adds a 'harvest_source' dataset type that mimics the original harvest
source form.
It works against the 3022 branch on CKAN core.
2012-11-29 12:31:48 +00:00
866fd69730
Do not remove XML declaration and add utf-8 charset to headers
2012-11-20 15:43:39 +00:00
c52ed3b163
Add line field to object error table
2012-11-20 11:29:58 +00:00
03fd1884f4
Implement retry times for harvest objects
2012-11-15 18:11:35 +00:00
202c9d9fcc
use correct queue for gather stage
2012-11-15 14:21:09 +00:00
c9c1eb4848
use generator to consume
2012-11-15 14:14:55 +00:00
33d5e09722
Change fetch_callback to proper acknowledge objects
2012-11-15 11:36:06 +00:00
13357893ad
Fix typo
2012-11-13 14:41:38 +00:00
54ff0526bb
Return original document if present when requesting an object
2012-11-13 12:06:36 +00:00
820443d58f
Add cascade option to harvest object extras and errors
2012-11-09 14:52:34 +00:00
5063626554
make sure state is changed to error on fetch error
2012-11-07 09:53:16 +00:00
28e5e9137a
add perge queues command
2012-11-07 09:51:25 +00:00
6db65b5826
made manual default not null
2012-11-05 13:17:32 +00:00
fdf01c09f2
Fix wrong check for harvest sources
2012-11-01 14:12:45 +00:00
d598c0707b
Ignore frequency field on the frontend for the time being
2012-11-01 14:12:01 +00:00
d7f8c9165c
Merge branch 'model_upgrade' into release-v2.0
2012-10-30 18:07:24 +00:00
d502b925a6
Remove old deprecated tests and some whitespace
2012-10-30 18:07:05 +00:00
a136cbf202
Fix typos in migration script
2012-10-30 17:52:10 +00:00
61b99e8eff
Merge branch 'pika' into release-v2.0
2012-10-30 17:31:30 +00:00
82a498d9fc
Rename function to be implementation independent
2012-10-30 17:13:39 +00:00
2529a17304
add jobs at certain frequencies
2012-10-29 17:15:02 +00:00
9fc0ae9937
add next run field
2012-10-26 10:50:35 +01:00
bc079c6644
model upgrade with tests and migration
2012-10-25 19:01:54 +01:00
1153c1c5c9
add full queue test and new test harvester
2012-10-24 11:58:00 +01:00
da125cdcc2
pika now used as queue library
2012-10-24 00:34:32 +01:00
8233b2ec23
Strip spaces from url when creating or updating a source
2012-08-17 12:25:06 +01:00
c1f83e0d3e
Strip spaces from url when creating or updating a source
2012-08-17 12:24:41 +01:00
a17e8208de
Very small text fix
2012-08-16 09:30:04 +01:00
7e940b497d
Text message minor fix
2012-08-16 09:27:36 +01:00
b6a32fd23b
Add descriptions for sources
2012-08-16 09:16:34 +01:00
c984727de5
Minor template tidy
2012-08-16 08:56:25 +01:00
5b7a9c0855
Flash messages to notices plus translatable
2012-08-16 08:49:35 +01:00
19ea538097
[#2852,#2853] Reword errors
2012-08-15 18:28:08 +01:00
7609a93422
Minor css tweaks on the forms
2012-08-15 18:26:36 +01:00
c6c4f6d098
Remove about text placeholder
2012-08-15 10:40:52 +01:00
3b8075b670
Only specify autoform items once
2012-08-14 18:01:29 +01:00
e1c74bdbe6
Fixes for autoform extra_text
2012-08-14 17:56:49 +01:00
8f6bab104e
Dirty form changes pending cleanup
2012-08-14 17:33:32 +01:00
1979517706
Widen url field
2012-08-14 12:02:02 +01:00
a76140650d
Merge branch 'release-v2.0' of github.com:okfn/ckanext-harvest into release-v2.0
2012-08-14 11:40:35 +01:00
4b68e4c31b
Fix details page template and style
2012-08-14 11:23:56 +01:00
eb12152089
Fix index page template and style
2012-08-14 11:04:17 +01:00
4b4e5dba62
fix broken show form
2012-08-14 00:44:00 +02:00
7efca28c22
Template updates
2012-08-10 13:05:54 +01:00
5557da653f
First draft of new source page
2012-08-10 10:06:37 +01:00
3feca92d55
First draft of index page
2012-08-10 10:00:02 +01:00
d8a98fd64a
Move to new plugins model
2012-08-10 09:59:18 +01:00
a8aebac965
Fix the harvest object show call
2012-08-09 13:38:17 +01:00
bb5ba43ebb
Allow showing harvest objects by default (on the default auth profile)
2012-08-09 13:37:28 +01:00
4c562e5f5f
Do not store the object when importing
2012-08-09 11:17:41 +01:00
e4b3cb440c
Do not use repo.are_tables_created
...
When checking whether the core tables have been alredy created it is
best to use package_table.exists(), as are_tables_created reflects the
tables, causing conflicts with other extensions.
This allows ckanext-harvest and ckanext-spatial to be used together on
ckan 1.8 onwards.
2012-08-09 11:06:05 +01:00
4d2fdeac57
Allow defining segments of harvest objects to import
...
Useful when importing large number of objects, as it allows
parallelization
2012-08-02 18:41:59 +01:00
7011efe5dc
Allow not linking to datasets when importing records
...
With the -j flag, harvest objects are not linked to datasets when
importing. This is useful sometimes when importing records for the first
time.
2012-07-30 12:11:55 +01:00
203bcb053b
Status can have links in it now.
2012-07-23 16:15:11 +01:00
a61ea06faf
Merge branch 'master' of github.com:okfn/ckanext-harvest
2012-07-19 15:27:04 +01:00
1a4e43a2a9
Status message added - change config to set the text.
2012-07-19 15:17:50 +01:00
4d00e665f1
[cli] Speed up run command
2012-06-29 11:32:18 +01:00
5df2b64dda
Merge branch 'master' of github.com:okfn/ckanext-harvest
2012-06-15 18:38:33 +01:00
c0a9965b52
Reword warning.
2012-06-15 18:38:22 +01:00
528e98120c
[ #2533 ] Fix some imports broken by ckan cleanups
2012-06-15 12:08:35 +02:00
ccf0cd3da2
Add copious logging to record what happens in harvesting.
2012-06-08 17:09:22 +01:00
a5fac2ac86
Added logic for getting a harvest source when viewing a dataset.
2012-06-01 17:03:40 +01:00
5151f4ee23
In publisher auth mode, any member of the group can make the changes. This brings things in line with the general idea that Admins have the power to do this plus authorize other editors/admins.
2012-05-29 15:21:34 +01:00
017222afd2
Withdrawn language introduced for uklp.
2012-05-28 18:41:03 +01:00
103eea9d50
Improve layout of sources to span page.
2012-05-28 18:13:34 +01:00
b1a4cc7721
[ui][xs] Nicer button
2012-05-10 11:53:19 +01:00
3941b691f4
[ui] Fix style for CKAN 1.7
2012-05-09 16:03:03 +01:00
f0a09e8299
[logic] Fix small bug in dictization
2012-05-09 15:58:23 +01:00
718202d886
Logging is now assured.
2012-04-10 20:53:29 +01:00
00e911a70c
Fix name of the queue logger. Moved imports of ckanext.harvest until after _load_config so that the loggers do not start disabled.
2012-04-10 20:10:17 +01:00
e797f50a05
[cli] Fix create job command
2012-03-19 17:28:53 +00:00
38a9a03355
[logic] Fix variables naming
2012-03-19 17:01:20 +00:00
1145e6ea72
[master][auth/publisher] Check for 'ignore_auth' in harvest_object_show
...
Use case: In ckanext-dgu we want to index the harvest_object.content field. As indexing is done synchronously we need
to provide a way for that harvest_object to be accessed when the current http request is made by a non-sysadmin user.
2012-03-15 18:14:57 +00:00
7f10418f44
[master][auth] get_obj_object() function was missing
2012-03-15 18:09:44 +00:00
871eae94b6
[ckan harvester] Fix bug on force all check
2012-03-15 11:31:12 +00:00
f210455aef
[ckan harvester] Replace title on default extras
2012-03-13 12:38:14 +00:00
e0bef2ef9c
[base] Minor fix for harvesters without config
2012-03-12 14:46:28 +00:00
amercader
kindly
tobes
David Read
Sean Hammond
David Read
Ian Murray