98364723d7
Apply flake8
2019-03-06 13:19:05 +02:00
68bf4893c4
Add an 'owner_org' to the v3 package migration
...
The auto-migration fails because the new package does not have an
owner organisation ID assigned, so fails validation
2018-12-03 14:13:04 +00:00
bfc9b8e0d9
[ #249 ] Test and fix docs for default_tags. Needed to improve error handling when saving ValidationError in a HOE.
2016-06-09 22:11:03 +00:00
1ec2af0590
Merge pull request #230 from keitaroinc/logging-module
...
Logging module
2016-05-17 14:12:03 +01:00
0be2c868cb
README updates
...
DBLogHandler updates
Added harvest_log table migration for existing users
Implemented database log scoping
2016-05-11 13:29:53 +02:00
009cc57e09
Added clean-up mechanism for the harvest log
2016-05-06 18:44:02 +02:00
633a32075e
create index to harvest_object
2016-04-27 09:27:12 +03:00
a79ad2e325
Added basic DBLogHandler
2016-04-05 20:21:04 +02:00
c7021933a0
Move creation of errors to the model as thats a more natural home. Provide backwards compatibility.
2015-12-02 08:15:13 +00:00
c7fac36c1c
[ #107 ] "unchanged" response tested and related fixes
...
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object.
* fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now.
* "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that.
* "unchanged" response is now documented
* better report_status tests in test_queue2.
2015-11-03 00:22:53 +00:00
421e6da660
Add run_test, job_abort, source commands
...
* run_test - for running a whole harvest on the command-line
* job_abort - for aborting a limbo job
* source - for showing a single harvest source
* allowing a source to be specified by name in several commands
2015-10-28 17:51:58 +00:00
2f4adfb338
Merge branch 'tests'
2015-10-23 13:18:15 +01:00
dc7af5d150
Remove prints.
2015-10-21 16:38:03 +00:00
f70c16bce7
Add framework for testing harvesters. Modernize existing tests.
2015-10-21 16:26:57 +00:00
2da918c2e4
Fix migration for old harvests so that ones that errored are correctly marked. Added helpful comments in model.
2015-07-22 10:13:02 +01:00
c2283e3fdb
only migrate harvest sources which are active
2013-06-28 02:32:45 +01:00
1792180e4f
Better harvest source dataset migration
...
Current implementation only checked for the first source to exist and
didn't allow to rerun the migration for other sources if there was an
error. With the new one, all non existing sources are migrated each
time.
2013-05-24 14:49:55 +01:00
1d54edfdaa
Fix bug in source datasets migration
...
Wrong dataset type was causing the default package schema to be used,
thus failing when providing an id.
2013-05-24 14:25:05 +01:00
cb8b808274
sanity check that harvest source id matches harvest dataset id
...
remove author_email, license_id, maintainer_email, maintainer and
author from pacakge_dict, these were not actually necessary
2013-03-29 00:59:20 +00:00
548d3c1c2a
fix validation issue on db upgrade
2013-03-25 12:02:07 +00:00
5956e5a9d5
Merge branch '4-new-auth-for-2.0' into 5-improve-job-errors-reporting
2013-02-05 12:36:26 +00:00
ca7819b885
Merge branch 'release-v2.0' into 2.0-dataset-sources
2013-02-05 12:35:14 +00:00
cca554c5ec
Fix typo and add missing column on v3 migration script
2013-02-05 12:33:56 +00:00
42bace3628
[ #5 ] Add new finished field for harvest job
...
When the run command flags a job as finished, it will query the most
recent harvest object for this job and use its import_finished value as
the job finishing time.
2013-01-28 17:19:28 +00:00
fdac761fba
Merge branch 'release-v2.0' into 2.0-dataset-sources
...
Conflicts:
ckanext/harvest/logic/action/get.py
ckanext/harvest/tests/test_queue.py
2012-12-20 16:16:30 +00:00
6b42d96fe0
add report_status field
2012-12-17 23:50:26 +00:00
c1b0415cb6
Merge branch 'release-v2.0' into 2.0-dataset-sources
...
Conflicts:
ckanext/harvest/model/__init__.py
2012-12-13 18:33:59 +00:00
d57e73458a
Make harvest object - package FK deferrable
...
Allows eg to add the harvest object id to the package dict before
indexing.
2012-12-13 18:21:40 +00:00
fcbe6aa6de
Script for creating harvest source datasets on old versions
...
The way we check whether datasets need to be created might need to be
improved.
2012-12-05 18:54:28 +00:00
9d36fd6841
First stub of the new dataset type forms
...
Adds a 'harvest_source' dataset type that mimics the original harvest
source form.
It works against the 3022 branch on CKAN core.
2012-11-29 12:31:48 +00:00
c52ed3b163
Add line field to object error table
2012-11-20 11:29:58 +00:00
03fd1884f4
Implement retry times for harvest objects
2012-11-15 18:11:35 +00:00
820443d58f
Add cascade option to harvest object extras and errors
2012-11-09 14:52:34 +00:00
6db65b5826
made manual default not null
2012-11-05 13:17:32 +00:00
a136cbf202
Fix typos in migration script
2012-10-30 17:52:10 +00:00
9fc0ae9937
add next run field
2012-10-26 10:50:35 +01:00
bc079c6644
model upgrade with tests and migration
2012-10-25 19:01:54 +01:00
e4b3cb440c
Do not use repo.are_tables_created
...
When checking whether the core tables have been alredy created it is
best to use package_table.exists(), as are_tables_created reflects the
tables, causing conflicts with other extensions.
This allows ckanext-harvest and ckanext-spatial to be used together on
ckan 1.8 onwards.
2012-08-09 11:06:05 +01:00
528e98120c
[ #2533 ] Fix some imports broken by ckan cleanups
2012-06-15 12:08:35 +02:00
ccf0cd3da2
Add copious logging to record what happens in harvesting.
2012-06-08 17:09:22 +01:00
4c81c7c3a7
[ #1726 ][model] Harvest source reference compatibility
...
The 'source' property of harvest objects now comes from the actual
foreign key. For compatibility with old harvesters, an before insert
event listener has beeen added to check if the source id has been set,
and set it automatically from the job if not.
Note that this requires SQLAlchemy 0.7 (ie CKAN 1.5.1)
2012-02-01 12:52:52 +00:00
004210935a
[model] Avoid unicode warning
2012-02-01 11:10:44 +00:00
b64d97118c
[ #1726 ][model] Add scripts for populating source_id and current fields
2012-02-01 11:08:41 +00:00
d1783f5415
[model] Changes in harvest model
...
Added three changes to the harvest model:
* 'title' column in harvest_source table
* 'current' column in harvest_job table
* foreign key from harvest_object to harvest_source
Tables are checked on startup to see if they need to be updated.
TODO: populate current and harvest_source_id fields
2012-01-30 18:38:35 +00:00
f086e908bc
[model] Clearer table initialization
2012-01-30 17:09:28 +00:00
dd00e98d9d
[model]: More careful about creating tables, since paster db upgrade loads the envrionment and therefore runs setup() before it does the migrations, and therefore in this instance we do not want to create the db tables.
2011-08-10 16:25:57 +01:00
0d9d1f8096
reduce number of queries for harvest index to a less insane number. still heavy.
2011-06-13 17:36:35 +02:00
264b606c48
take tables out of global scope at import time
2011-05-31 18:02:07 +01:00
bbe459527f
[forms] Major refactoring of the harvest forms. Forms no longer use the DGU form
...
API, and are handled similarly to the new ones on CKAN core (logic, schema,
validators...). The UI is also more consistent with the CKAN one.
2011-05-13 14:17:58 +01:00
26cdc1089d
Change date definitions in Harvest Objects. reference_date -> metadata_modified_date, created -> gathered
2011-05-11 17:07:05 +01:00
Jari Voutilainen
Bruce Bolt
David Read
Adrià Mercader
Petar Efnushev
kindly
joetsoi
Sean Hammond
David Read
Friedrich Lindenberg