Commit Graph

113 Commits

Author SHA1 Message Date
Alessia Bardi d9ea174948 better default values for wf template of periodo 2022-10-10 17:06:53 +02:00
Alessia Bardi fa71f9a7e1 Incremental support for thanados plugin 2022-05-05 16:13:32 +02:00
Alessia Bardi ae2b4b6388 protovol and plugin bean for thanados 2022-05-05 15:44:04 +02:00
Alessia Bardi 17a382d35e cleaned test 2022-05-05 15:26:39 +02:00
Alessia Bardi fcdec764f1 Fixed end iteration criteria 2022-04-19 17:03:54 +02:00
Alessia Bardi 48b135190e Added some printlns while testing 2022-04-15 14:42:59 +02:00
Alessia Bardi 305bc617ff Collector plugin for Thanados 2022-04-11 18:55:54 +02:00
Enrico Ottonello 62af1fc637 update to ARIADNEplus Generator Policy version 1.10 2022-03-30 10:53:00 +02:00
Enrico Ottonello f8e6e04e82 added charset on payload creation for nara publishing 2022-01-31 15:25:38 +01:00
Enrico Ottonello 98cccc4442 removed GZIPCompression on request, currently it is not needed 2021-12-17 11:44:16 +01:00
Enrico Ottonello a90f02487a record to publish now are sent to spring module inside body request, because of htmlsimple harvesting can produce large record (>10MB) 2021-12-17 11:39:37 +01:00
Alessia Bardi 0d0ebc4d77 testing DIME 2021-10-25 15:24:23 +02:00
Enrico Ottonello 788fc3b734 added lowercase function for derivedSubject.prefLabe, nativeSubject.prefLabel 2021-10-19 16:03:38 +02:00
Enrico Ottonello e9430ab460 changed title structure from field to object for multiple language handling 2021-10-18 18:06:40 +02:00
Enrico Ottonello 3510d7d1be Merge branch 'new_es_mapping'
# Conflicts:
#	dnet-ariadneplus-graphdb-publisher/src/main/resources/eu/dnetlib/ariadneplus/sparql/read_collection_data_template.sparql
#	dnet-ariadneplus-graphdb-publisher/src/main/resources/eu/dnetlib/ariadneplus/sparql/read_record_data_template.sparql
#	dnet-ariadneplus-graphdb-publisher/test/java/eu/dnetlib/ariadneplus/GraphDbReaderAndESIndexTest.java
2021-09-08 15:56:12 +02:00
Enrico Ottonello 03da34a897 added dynamic property set to false to avoid es mapping modification indexinga record; previous index has been deleted 2021-06-15 23:38:02 +02:00
Alessia Bardi fc71bc24ed new generator policy 2021-06-04 10:20:46 +02:00
Enrico Ottonello 06f20f7a8d shell command to create the new elastic search index with mapping on staging 2021-05-31 16:59:56 +02:00
Enrico Ottonello 5a68ab07b9 mapping loaded only once now 2021-01-29 17:16:48 +01:00
Enrico Ottonello 290259fbc2 new collector plugin (xmlsFolder) based on ads one, adding a new parameter which represents the list of namespaces that has to be added to the generated xml record during collection step 2021-01-15 13:33:23 +01:00
Enrico Ottonello 35b81aeee4 Merge branch 'master' of https://code-repo.d4science.org/D-Net/AriadnePlus 2021-01-09 00:53:43 +01:00
Enrico Ottonello a14c66f36e added default values for has_temporal_coverage and has_spatial_coverage 2021-01-09 00:53:33 +01:00
Miriam Baglioni 472ee2754c Aggiornare 'dnet-ariadneplus/src/main/resources/es/port.py' 2021-01-08 16:52:26 +01:00
Miriam Baglioni 060f7f8673 Aggiornare 'dnet-ariadneplus/src/main/resources/es/port.py' 2021-01-08 16:51:44 +01:00
Miriam Baglioni 538be9e035 Aggiungi 'dnet-ariadneplus/src/main/resources/es/port.py' 2021-01-08 16:51:07 +01:00
Enrico Ottonello b8d0e3f741 enrichment queries set is now splitted on workflow node side, because of timeout error on very large collection 2021-01-07 23:40:47 +01:00
Enrico Ottonello 4673c143ce curl command to create catalog index with mapping for elastic search on production 2021-01-07 11:48:00 +01:00
Enrico Ottonello 1ca29a2ebf alignment with current elastic-test instance mapping 2021-01-07 11:35:33 +01:00
Enrico Ottonello a05d8e97da replicated with a new id for having different execution dates with aggregation_to_graphdb_wf.xml 2020-12-18 17:39:15 +01:00
Enrico Ottonello 3e68138b81 new wf to publish a collection on staging graphdb and then on public one 2020-12-18 16:20:38 +01:00
Enrico Ottonello dc6bd40d1e let isStart true only for collection node, avoiding parallelism on multiple nodes 2020-12-18 16:19:43 +01:00
Alessia Bardi 5f695ee538 fixed worng copy&paste, probably from an st file...again 2020-12-17 20:15:51 +01:00
Alessia Bardi 0e30eccbae fixed worng copy&paste, probably from an st file 2020-12-17 17:56:13 +01:00
Alessia Bardi 6c150be5b2 added missing param 2020-12-17 13:53:26 +01:00
Alessia Bardi e5eefef411 fixed param name 2020-12-16 16:09:40 +01:00
Alessia Bardi fa3592b1b3 added missing wf node 2020-12-15 17:06:52 +01:00
Alessia Bardi 3e28f8fd57 increasing mid version number 2020-12-14 18:57:36 +01:00
Alessia Bardi 4edfc4af05 new_agg_wf.xml annotated with comments to guide the (manual) migration of old workflows to the new one 2020-12-14 18:56:49 +01:00
Alessia Bardi 687d5ab9ec only one start node 2020-12-14 18:27:23 +01:00
Alessia Bardi be64ce5dcf changed workflow name ad description 2020-12-14 18:22:13 +01:00
Alessia Bardi 8a9782cb1e re-organising workflows and templates for staging/public instances of graphdb and elasticsearch 2020-12-14 17:49:53 +01:00
Enrico Ottonello 5d29077a43 added counter on wf progress information 2020-10-20 19:34:40 +02:00
Enrico Ottonello 8073ac7131 the node retrieves resources identifier list for collection and record, then loops on the 2 identifiers lists and request indexing to rest module 2020-10-20 17:43:27 +02:00
Enrico Ottonello 1398c2495e test for new indexing methods on rest module 2020-10-20 16:42:44 +02:00
Enrico Ottonello 0e96774895 changed http request method for indexing 2020-10-15 17:09:35 +02:00
Enrico Ottonello 25de647504 set KeepAlive true on used socket; replaced SimpleJobNode with AsyncJobNode 2020-10-13 15:34:43 +02:00
Enrico Ottonello 2e732a3b5b sparql insert for ads 1;replaced AsynchJobNode with SimpleJobNode to 404 problem 2020-10-10 18:05:02 +02:00
Enrico Ottonello 8b3f8896a5 removed progress bar handler attached to resultset client (not used here) 2020-10-10 16:53:37 +02:00
Enrico Ottonello db2ad3f97e multithreads http client not needed on indexjobnode, maybe avoid 404 response after a few minutes, added log on each indexing operation 2020-08-12 15:47:55 +02:00
Enrico Ottonello 5c3ef5f25b retrieved body from server response 2020-07-24 18:32:28 +02:00