D-Net
/
AriadnePlus
4
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
536 Commits 4 Branches 1 Tag 2.5 MiB
Commit Graph

46 Commits

Author SHA1 Message Date
Alessia Bardi 236f936b05 new index field to host has_identifier (https://support.d4science.org/issues/24160) 2022-11-22 13:47:04 +01:00
Alessia Bardi 194c1620ea more logs and bmped version 2022-11-21 18:59:58 +01:00
Alessia Bardi 23c857906d new library for parsing dates 2022-11-17 14:38:03 +01:00
Alessia Bardi fa5bce3c39 added support for dates like '27-Oct-2022' 2022-11-16 16:31:47 +01:00
Enrico Ottonello 0d0ca09983 open search record id generated with sha-256 function 2022-07-11 18:43:25 +02:00
Enrico Ottonello 4b94935f17 added rdfs:label to native period for collection query; fix iavp enrichment; added date formatter for es publishing 2022-02-01 14:19:13 +01:00
Enrico Ottonello 0010458a23 modified log level for a few message 2021-10-22 12:32:04 +02:00
Enrico Ottonello 3510d7d1be Merge branch 'new_es_mapping' 
# Conflicts:
#	dnet-ariadneplus-graphdb-publisher/src/main/resources/eu/dnetlib/ariadneplus/sparql/read_collection_data_template.sparql
#	dnet-ariadneplus-graphdb-publisher/src/main/resources/eu/dnetlib/ariadneplus/sparql/read_record_data_template.sparql
#	dnet-ariadneplus-graphdb-publisher/test/java/eu/dnetlib/ariadneplus/GraphDbReaderAndESIndexTest.java
 2021-09-08 15:56:12 +02:00
Alessia Bardi d169e1fad8 Merge branch 'master' of https://code-repo.d4science.org/D-Net/AriadnePlus 2021-08-26 17:59:35 +02:00
Alessia Bardi ea3b749792 test with read-only mode connection 2021-08-26 17:59:32 +02:00
Enrico Ottonello 42eb9604a5 new class to handle publish on graphdb operation, transaction throws OOM so currently one write operation at time is executed without transaction 2021-07-27 14:58:00 +02:00
Enrico Ottonello e91d82f32c model classes according to new es mapping; creation and indexing of a record with geopoint data 2021-06-15 23:34:59 +02:00
Enrico Ottonello 5682cc4bd2 added multiple descriptions for arup 2021-04-02 12:46:07 +02:00
Enrico Ottonello a7a1a466b8 fix year date parsing 2021-01-26 16:39:47 +01:00
Enrico Ottonello 8baf2d7941 added BST date format handling parsing, before creating es record 2020-11-30 13:27:28 +01:00
Enrico Ottonello 665a717ca0 added check on identifier, before indexing 2020-10-23 12:18:58 +02:00
Enrico Ottonello 263c1beb7e fix date parsing on es record creation 2020-10-13 00:31:17 +02:00
Alessia Bardi a452133ac3 use the new values statically inserted by the construct query 2020-10-12 18:32:56 +02:00
Alessia Bardi 83ecd94c6f uncommented logs and set them in debug mode 2020-10-12 18:15:15 +02:00
Enrico Ottonello 2aafbc8506 fix loop condition 2020-10-10 01:03:41 +02:00
Enrico Ottonello fe614ce2ed added retry loop on HTTPQueryEvaluationException (Heap space almost full) 2020-10-10 00:55:26 +02:00
Enrico Ottonello 142a296166 changed exception type, if collection indexing failes 2020-09-24 12:53:08 +02:00
Enrico Ottonello 814c908663 collection is now indexed before records, if there is an error on collection indexing, wf failed immediately; enriched query for 397 collection 2020-09-24 12:35:59 +02:00
Alessia Bardi 12c80713d7 indexing result per record logged in debug mode instead of info 2020-09-07 15:45:26 +02:00
Alessia Bardi 832b95f064 parse must succeed if the date is already in the expected format 2020-09-07 15:37:40 +02:00
Alessia Bardi f1e3de1b07 avoid null pointer exception 2020-09-07 14:59:54 +02:00
Alessia Bardi 4fb5bb2941 it is legal to have no individual resources or no collections: let's index what we have 2020-09-04 16:57:38 +02:00
Enrico Ottonello db2ad3f97e multithreads http client not needed on indexjobnode, maybe avoid 404 response after a few minutes, added log on each indexing operation 2020-08-12 15:47:55 +02:00
Alessia Bardi 8329f6e9dc useless if removed 2020-08-07 16:40:21 +02:00
Enrico Ottonello f61f794c25 removed a few logs 2020-07-24 12:34:06 +02:00
Enrico Ottonello 11e00c806d reset json parser status on each record 2020-07-24 01:48:17 +02:00
Enrico Ottonello b29b4e1fd7 added log on json parser 2020-07-24 01:21:25 +02:00
Enrico Ottonello eb61aebcd5 added other check on parsing json from rdf statements 2020-07-24 00:32:49 +02:00
Enrico Ottonello 5eaa655f01 added other check on es return code 2020-07-23 23:12:07 +02:00
Enrico Ottonello 173b0c289e to avoid overload on graphdb heap it is better to slowing down query execution 2020-07-22 23:47:08 +02:00
Enrico Ottonello 5bb65a8854 added response code from es; added sparql insert for copying ariadne_subject value into collection for ads; fix sparql insert for copying collection data into records, collection id condition is needed 2020-07-22 23:13:47 +02:00
Enrico Ottonello d6aafdf97d es _id only for collection type record must be numeric format, for now we prepend 10000 to originalId value 2020-07-13 11:03:21 +02:00
Enrico Ottonello 9358f1eaf2 collection data are now indexed with all required fields by es mapping 2020-07-08 22:00:28 +02:00
Enrico Ottonello e689623994 all ES fields are now mapped 2020-07-08 10:50:14 +02:00
Enrico Ottonello 9737184b1e start aligning es json fields to es mapping 2020-07-07 13:39:22 +02:00
Enrico Ottonello db9b70feb0 new node and workflow to index on ES 2020-06-16 02:36:16 +02:00
Enrico Ottonello 42c3f5e885 added indexing of collection data with dedicated query 2020-06-15 12:10:59 +02:00
Enrico Ottonello 06995feefa added rest method indexOnES(datasource,collectionId) to retrieve records data from graphDB and index to ES 2020-06-12 18:14:41 +02:00
Enrico Ottonello 064f298329 added query to retrieve record ids collection; datasource, collection id and record id are now paramters on a query template from file 2020-06-11 18:20:42 +02:00
Enrico Ottonello b80274fff3 added graphDB reader and ES indexer 2020-06-10 19:39:53 +02:00
Enrico Ottonello 019bbd2f6a new classes to graphdb querying via sparql exewcution, data retrieving, feeding to elastic search 2020-01-28 16:53:59 +01:00