D-Net
/
AriadnePlus
4
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
613 Commits 4 Branches 1 Tag 2.6 MiB
Commit Graph

452 Commits

Author SHA1 Message Date
Enrico Ottonello 8a341add4c sparql insert for 1972 2020-12-01 11:29:37 +01:00
Enrico Ottonello 8baf2d7941 added BST date format handling parsing, before creating es record 2020-11-30 13:27:28 +01:00
Enrico Ottonello fbc4fc8717 added has_creator field mandatory for records 2020-11-27 12:41:11 +01:00
Enrico Ottonello 4a9a5301b3 sparql insert for 304, fix for 1093 2020-11-27 11:00:03 +01:00
Enrico Ottonello f1b1f55ae5 sparql insert for 1093 2020-11-25 13:34:18 +01:00
Enrico Ottonello 1fb0a95ada added check on record mandatory field place_name 2020-11-02 14:57:54 +01:00
Enrico Ottonello d9ca483855 added a default value for record mandatory field has_spatial_coverage 2020-11-02 14:55:18 +01:00
Enrico Ottonello eb775fc787 sparql insert for ads 4 2020-10-30 18:22:17 +01:00
Enrico Ottonello 075490f4da removed duplicated log 2020-10-30 17:29:18 +01:00
Enrico Ottonello 529151828f when IRI created from periodo url is wrong, wf has to fail 2020-10-30 17:08:46 +01:00
Enrico Ottonello 7044bcdca4 sparql insert for ads 328 2020-10-28 17:04:25 +01:00
Enrico Ottonello 991df8d4d1 sparql insert for ads 324 2020-10-28 16:41:31 +01:00
Enrico Ottonello 665a717ca0 added check on identifier, before indexing 2020-10-23 12:18:58 +02:00
Enrico Ottonello 899f3eb4d9 added has_temporal_coverage check 2020-10-22 11:26:33 +02:00
Enrico Ottonello 292f3000c4 added temporal_coverage default, if not found 2020-10-22 10:45:39 +02:00
Enrico Ottonello 240be91c5e added rest method to index single record by its identifier 2020-10-19 12:20:56 +02:00
Enrico Ottonello 9174f4df14 add graphdb connection setup 2020-10-16 19:56:14 +02:00
Enrico Ottonello f88231ba18 added new methods to retrieve resource identifiers and to start indexing by resource identifiers 2020-10-16 19:37:21 +02:00
Enrico Ottonello 0e96774895 changed http request method for indexing 2020-10-15 17:09:35 +02:00
Enrico Ottonello f5cd3cdce5 sparql insert for ads 1054 2020-10-15 13:51:02 +02:00
Enrico Ottonello ba5632f783 added has_temporal_coverage default as not provided for all ads collection, because of is a mandatory field for the query 2020-10-15 12:19:29 +02:00
Enrico Ottonello 72d37b4238 sparql insert for ads 1787, 269, fix for date on 1786 2020-10-14 19:10:24 +02:00
Enrico Ottonello 57e0470aee sparql insert for ads 1788, 388 2020-10-14 17:43:55 +02:00
Enrico Ottonello cb595f2c1c sparql insert for ads 420_monument 2020-10-14 16:30:04 +02:00
Enrico Ottonello 78818738b9 sparql insert for ads 801, 1970, 276 2020-10-14 13:11:14 +02:00
Alessia Bardi 6d8750b00d Add default for time spans that are not provided by CENIEH 2020-10-13 09:37:16 +02:00
Enrico Ottonello 263c1beb7e fix date parsing on es record creation 2020-10-13 00:31:17 +02:00
Enrico Ottonello 8f3722da13 has_native_period section set optional because of notprovided case 2020-10-12 23:00:09 +02:00
Enrico Ottonello 8777d14506 replaced test record identifier with the original placeholder %record ... 2020-10-12 22:27:54 +02:00
Alessia Bardi a452133ac3 use the new values statically inserted by the construct query 2020-10-12 18:32:56 +02:00
Alessia Bardi 5329110d93 fixed properties 2020-10-12 18:32:21 +02:00
Alessia Bardi 1b6c989f13 fixed INSERTs for CENIEH 2020-10-12 18:18:20 +02:00
Alessia Bardi 8e2b8922f6 SND native subject in upper case to enable AAT enrichment 2020-10-12 18:17:45 +02:00
Alessia Bardi 83ecd94c6f uncommented logs and set them in debug mode 2020-10-12 18:15:15 +02:00
Alessia Bardi 031f4d92f5 set explicit type to distinguish records and collections: property aocat:has_type can have multiple values 2020-10-12 18:12:41 +02:00
Miriam Baglioni d8133be6d4 insert for CENIEH 2020-10-12 15:25:16 +02:00
Enrico Ottonello ab0d99f503 added periodO uri info to retrieved collection data 2020-10-12 11:04:19 +02:00
Enrico Ottonello 0867cf3ab2 fix from e until for collection 2020-10-11 19:37:56 +02:00
Enrico Ottonello 2e732a3b5b sparql insert for ads 1;replaced AsynchJobNode with SimpleJobNode to 404 problem 2020-10-10 18:05:02 +02:00
Enrico Ottonello 2aafbc8506 fix loop condition 2020-10-10 01:03:41 +02:00
Enrico Ottonello fe614ce2ed added retry loop on HTTPQueryEvaluationException (Heap space almost full) 2020-10-10 00:55:26 +02:00
Enrico Ottonello f3c93400a3 sparql insert for ads 1785, was_issued and was_modified check has to be restricted using 1785, because of 1785 and 1786 share the same collection 2020-10-09 20:14:36 +02:00
Enrico Ottonello c83c8bf931 sparql insert for ads 1786 2020-10-09 17:47:53 +02:00
Alessia Bardi a9749ffbf2 INSERT for SND zip API - preliminary 2020-10-09 15:33:57 +02:00
Alessia Bardi a0b3e27645 updated construct query for temporals 2020-10-09 11:36:05 +02:00
Enrico Ottonello 47f46009e1 sparql insert for ads 272_event 2020-10-06 23:29:42 +02:00
Enrico Ottonello c2d13f1f4d sparql insert for ads 292 2020-10-06 22:57:18 +02:00
Enrico Ottonello 101b6261f1 sparql insert for ads 367 2020-10-06 15:51:11 +02:00
Enrico Ottonello ab3086c70c sparql insert for ads 3 2020-10-06 13:14:29 +02:00
Enrico Ottonello 8612438e86 added native subject and temporal coverage generation from about section 2020-10-05 16:45:17 +02:00
Enrico Ottonello e0729986f8 added temporal coverage generation from about section 2020-10-05 12:55:53 +02:00
Enrico Ottonello d65252873e added native_subject generation from about section 2020-10-05 11:08:00 +02:00
Enrico Ottonello 95270ef3f7 sparql insert for ads 321 2020-10-02 12:08:06 +02:00
Enrico Ottonello 05c3682e0d fix HNM periodo graph creation with all needed information, fix HNM periodoplus graph creation retrieving from/until date in year format; 858 sparql insert queries 2020-10-01 15:01:52 +02:00
Enrico Ottonello 0d44e18674 added optional fields for collection 2020-09-30 12:30:57 +02:00
Enrico Ottonello e5d0fa5254 added has_native_subject insert with uppercase; added aatplus graph creation 2020-09-29 23:13:07 +02:00
Enrico Ottonello e36e1a0080 enriched query for 420_event collection 2020-09-25 15:14:14 +02:00
Enrico Ottonello 142a296166 changed exception type, if collection indexing failes 2020-09-24 12:53:08 +02:00
Enrico Ottonello 814c908663 collection is now indexed before records, if there is an error on collection indexing, wf failed immediately; enriched query for 397 collection 2020-09-24 12:35:59 +02:00
Enrico Ottonello 12bbcdd8be Merge branch 'master' of https://code-repo.d4science.org/D-Net/AriadnePlus 2020-09-24 11:16:55 +02:00
Enrico Ottonello e5e7d85996 added ns declaration in template; enriched query for 1091 collection 2020-09-24 11:16:51 +02:00
Alessia Bardi f0509be5aa Merge branch 'master' of https://code-repo.d4science.org/D-Net/AriadnePlus 2020-09-22 16:28:50 +02:00
Enrico Ottonello 6976a35611 has_spatial_coverage data set to optional for collection 2020-09-18 17:24:36 +02:00
Enrico Ottonello b4476f6ee7 sparql insert set for 272_monument; test for retrieving collection records for 272_monument, that does not retrieve any records, because of spatial_region data are not found 2020-09-17 16:32:43 +02:00
Alessia Bardi ea4ed22b4d Collection 272 does not exist, we have 272_monument and 272_event 2020-09-15 15:21:07 +02:00
Alessia Bardi be31a772a1 SPARQL INSERT for HNM 2020-09-07 15:46:07 +02:00
Alessia Bardi 12c80713d7 indexing result per record logged in debug mode instead of info 2020-09-07 15:45:26 +02:00
Alessia Bardi 832b95f064 parse must succeed if the date is already in the expected format 2020-09-07 15:37:40 +02:00
Alessia Bardi 19ebb63d88 fixed log 2020-09-07 15:00:09 +02:00
Alessia Bardi f1e3de1b07 avoid null pointer exception 2020-09-07 14:59:54 +02:00
Alessia Bardi 4fb5bb2941 it is legal to have no individual resources or no collections: let's index what we have 2020-09-04 16:57:38 +02:00
Alessia Bardi f55699a324 sparql insert for HNM (AAT missing) 2020-09-04 16:56:52 +02:00
Alessia Bardi 420e31d866 preparing sparql inserts for ADS 271 and 272 2020-09-04 16:56:17 +02:00
Alessia Bardi e63f25217c Insert defaults for collections of HNM 2020-09-04 11:29:01 +02:00
Alessia Bardi ca2f6dbb21 updated labels 2020-09-04 11:26:06 +02:00
Alessia Bardi 1b11d0ee1c default for collection has_type (needed by HNM) 2020-09-04 11:11:22 +02:00
Alessia Bardi 7338a759c8 removed duplicate check on access rights 2020-09-04 10:57:11 +02:00
Enrico Ottonello db2ad3f97e multithreads http client not needed on indexjobnode, maybe avoid 404 response after a few minutes, added log on each indexing operation 2020-08-12 15:47:55 +02:00
Alessia Bardi 56088d7779 need to add info from the collSubjects graph -- ARIADNE subject and collection id 2020-08-07 18:53:15 +02:00
Alessia Bardi ca42e0d01e fixed SPARQL INSERT for PeriodO enrichment 2020-08-07 17:45:45 +02:00
Alessia Bardi 23f1feb44a updated SPARQL inserts for ADS 2020-08-07 16:41:07 +02:00
Alessia Bardi 8329f6e9dc useless if removed 2020-08-07 16:40:21 +02:00
Alessia Bardi c49d6ce109 added query for spatial region of collections 2020-08-07 16:24:15 +02:00
Alessia Bardi 02c5014fd5 removed redundand isPart 2020-08-07 16:23:26 +02:00
Alessia Bardi e8d65287cc check additional mandatory fields 2020-08-07 16:22:19 +02:00
Alessia Bardi 3e2e9ff372 Insert default creator also for collections 2020-08-07 16:21:47 +02:00
Alessia Bardi ca4ad7ea3b Need to split the multiple inserts and perform separate transactions in GraphDB Free edition. See https://stackoverflow.com/questions/54428161/graphdb-read-check-and-update-in-a-transaction 2020-08-07 14:54:15 +02:00
Alessia Bardi 587887abd6 sparql inserts for ADS 836 2020-08-07 14:53:21 +02:00
Alessia Bardi 2c282d31b2 sparql inserts for ADS 836 2020-08-07 14:51:56 +02:00
Alessia Bardi 223c304702 sparql inserts for ADS 273 2020-08-07 12:57:21 +02:00
Alessia Bardi 69ca7eaade removed redundand lines of code 2020-08-07 12:44:36 +02:00
Alessia Bardi b3f5f38c02 sparql query to get the records without mandatory fields 2020-08-07 12:31:41 +02:00
Alessia Bardi f8a3bc5a0e sparql query to get the collections without mandatory fields 2020-08-07 12:28:17 +02:00
Alessia Bardi e82043d9eb added comments 2020-08-07 11:45:09 +02:00
Alessia Bardi 7040ad741b moved SPARQL query for ADS 270 in dedicated folder 2020-08-06 20:29:06 +02:00
Alessia Bardi 6365fa583d sparql templates for the different fix and enrichment phases 2020-08-06 20:28:16 +02:00
Alessia Bardi cf2f2e6f8b organised templates and prepared folders for ADS SPARQL inserts 2020-08-06 20:26:01 +02:00
Enrico Ottonello 54458be7f5 order in sparql insert execution is important, this query contains all sparql insert in the correct order 2020-07-29 16:37:22 +02:00
Enrico Ottonello 220c6be6db report returned to wf node 2020-07-24 13:15:27 +02:00
Enrico Ottonello 214502cc11 removed test publisher 2020-07-24 12:59:59 +02:00
Enrico Ottonello f61f794c25 removed a few logs 2020-07-24 12:34:06 +02:00
Enrico Ottonello 11e00c806d reset json parser status on each record 2020-07-24 01:48:17 +02:00
Enrico Ottonello b29b4e1fd7 added log on json parser 2020-07-24 01:21:25 +02:00
Enrico Ottonello eb61aebcd5 added other check on parsing json from rdf statements 2020-07-24 00:32:49 +02:00
Enrico Ottonello 5eaa655f01 added other check on es return code 2020-07-23 23:12:07 +02:00
Enrico Ottonello 173b0c289e to avoid overload on graphdb heap it is better to slowing down query execution 2020-07-22 23:47:08 +02:00
Enrico Ottonello 5bb65a8854 added response code from es; added sparql insert for copying ariadne_subject value into collection for ads; fix sparql insert for copying collection data into records, collection id condition is needed 2020-07-22 23:13:47 +02:00
Enrico Ottonello 766f8199f2 removed empty values/structures from record/collection to es 2020-07-17 17:42:08 +02:00
Enrico Ottonello d6aafdf97d es _id only for collection type record must be numeric format, for now we prepend 10000 to originalId value 2020-07-13 11:03:21 +02:00
Enrico Ottonello 6a804f176b removed extent and audience fields, empty values cause error on portal 2020-07-09 16:35:04 +02:00
Enrico Ottonello 9358f1eaf2 collection data are now indexed with all required fields by es mapping 2020-07-08 22:00:28 +02:00
Enrico Ottonello 364d25f2b7 archeologicalResourceType.id mapped 2020-07-08 11:54:32 +02:00
Enrico Ottonello e689623994 all ES fields are now mapped 2020-07-08 10:50:14 +02:00
Enrico Ottonello 9737184b1e start aligning es json fields to es mapping 2020-07-07 13:39:22 +02:00
Enrico Ottonello db9b70feb0 new node and workflow to index on ES 2020-06-16 02:36:16 +02:00
Enrico Ottonello 42c3f5e885 added indexing of collection data with dedicated query 2020-06-15 12:10:59 +02:00
Enrico Ottonello 06995feefa added rest method indexOnES(datasource,collectionId) to retrieve records data from graphDB and index to ES 2020-06-12 18:14:41 +02:00
Enrico Ottonello 064f298329 added query to retrieve record ids collection; datasource, collection id and record id are now paramters on a query template from file 2020-06-11 18:20:42 +02:00
Enrico Ottonello b80274fff3 added graphDB reader and ES indexer 2020-06-10 19:39:53 +02:00
Enrico Ottonello fa1d268b36 added query to link data from collection level to records level 2020-06-10 11:53:37 +02:00
Enrico Ottonello af2eb7bb91 added sparql query folder 2020-06-09 11:44:48 +02:00
Enrico Ottonello aa1c6c2fda added query to generate aat and periodo enriched graphs 2020-06-09 11:39:56 +02:00
Enrico Ottonello 9814069fb2 new rest method to import data into graphDB from url 2020-05-30 17:21:03 +02:00
Enrico Ottonello 30b3fa2140 New JobNode and workflow to enrich content on GraphDB 2020-05-29 16:19:55 +02:00
Enrico Ottonello 739dcc6b82 added rest method to execute sparql insert query on graphdb 2020-05-27 23:07:08 +02:00
Enrico Ottonello d23f08f750 removed BIND not necessary 2020-04-03 15:53:52 +02:00
Enrico Ottonello a005328dfe fix resource type variable name 2020-03-31 11:10:41 +02:00
Enrico Ottonello d52ad32f68 added narrow and broad for aat match 2020-03-20 17:46:55 +01:00
Enrico Ottonello 7ae6d400b6 temporal coverage joined with periodO data that must be directly imported in graphDb in turtle format together with periodO mapping 2020-03-20 13:01:54 +01:00
Enrico Ottonello ef42eb551a fix spatial subselect: region, region point and region bounding box data must be referenced by 3 different variables inside the query 2020-03-17 16:27:14 +01:00
Enrico Ottonello 32b91c277d all not madatory sets of conditions must be set as optional 2020-03-16 19:16:42 +01:00
Enrico Ottonello 5c3d50f309 originalId set to mandatory 2020-03-06 13:11:35 +01:00
Enrico Ottonello 2b85674659 modified prefix mine: <http://www.myprefix/> into aoprop: <https://www.ariadne-infrastructure.eu/property/> 2020-03-06 12:22:51 +01:00
Enrico Ottonello 3e230e98be added bounding box info, modified output predicates 2020-03-05 16:24:22 +01:00
Enrico Ottonello a8aa5ec6ed added legalResponsible, owner 2020-03-04 13:27:31 +01:00
Enrico Ottonello 7bd85b6946 retrieved contributor on collection sub-select 2020-03-03 16:57:07 +01:00
Enrico Ottonello 7d65ba2a0c matching with aat and fix spatial coverage 2020-02-27 17:19:12 +01:00
Enrico Ottonello 7f93ce6358 from named condition modified to from because of wrong triples in results 2020-02-25 16:16:06 +01:00
Enrico Ottonello bf862f20f2 fix wrong matches on native subjects 2020-02-19 16:13:07 +01:00
Enrico Ottonello 226cdde77d all records related to an api are published into the same graph 2020-02-19 14:33:54 +01:00
Enrico Ottonello f12eb7eb52 added aat collection clause 2020-02-18 16:05:51 +01:00
Enrico Ottonello ae91617d1b added optional surrounding union conditions, in case all those conditions do not match any statements 2020-02-11 12:04:45 +01:00
Enrico Ottonello 911ea12ac3 query with union 2020-02-07 15:31:47 +01:00
Enrico Ottonello 5d8cbece81 graphdb query 2020-02-07 14:44:26 +01:00
Enrico Ottonello ff357e728b modified context properties name because of update to springboot 2.1.3 2020-01-29 16:43:25 +01:00
Enrico Ottonello 019bbd2f6a new classes to graphdb querying via sparql exewcution, data retrieving, feeding to elastic search 2020-01-28 16:53:59 +01:00
Enrico Ottonello 17552bafa0 foreach published api, the datasource name is stored into graph datasourceApis 2020-01-24 10:52:04 +01:00
Enrico Ottonello 15cc87456c before any api publishing all named graph related to previous aggregation for that api are dropped 2020-01-15 12:30:18 +01:00
Enrico Ottonello 8530e488d0 all records related to a datasource api are now published on graphdb into n named graphs because of out of memory issue 2020-01-14 16:55:45 +01:00
Enrico Ottonello b3d8a9193b moved graphdb repository name and other properties in application.properties 2019-12-16 14:46:42 +01:00
Enrico Ottonello 3f8a4e9a46 new rest method to save datasourceApi information after records publishing on graphDB 2019-12-13 14:53:15 +01:00
Enrico Ottonello 2f5fb6fcb5 new module for graphdb publishing 2019-12-12 12:58:30 +01:00