Commit Graph

36 Commits

Author SHA1 Message Date
Enrico Ottonello 290259fbc2 new collector plugin (xmlsFolder) based on ads one, adding a new parameter which represents the list of namespaces that has to be added to the generated xml record during collection step 2021-01-15 13:33:23 +01:00
Enrico Ottonello b8d0e3f741 enrichment queries set is now splitted on workflow node side, because of timeout error on very large collection 2021-01-07 23:40:47 +01:00
Enrico Ottonello 5d29077a43 added counter on wf progress information 2020-10-20 19:34:40 +02:00
Enrico Ottonello 8073ac7131 the node retrieves resources identifier list for collection and record, then loops on the 2 identifiers lists and request indexing to rest module 2020-10-20 17:43:27 +02:00
Enrico Ottonello 0e96774895 changed http request method for indexing 2020-10-15 17:09:35 +02:00
Enrico Ottonello 25de647504 set KeepAlive true on used socket; replaced SimpleJobNode with AsyncJobNode 2020-10-13 15:34:43 +02:00
Enrico Ottonello 2e732a3b5b sparql insert for ads 1;replaced AsynchJobNode with SimpleJobNode to 404 problem 2020-10-10 18:05:02 +02:00
Enrico Ottonello 8b3f8896a5 removed progress bar handler attached to resultset client (not used here) 2020-10-10 16:53:37 +02:00
Enrico Ottonello db2ad3f97e multithreads http client not needed on indexjobnode, maybe avoid 404 response after a few minutes, added log on each indexing operation 2020-08-12 15:47:55 +02:00
Enrico Ottonello 5c3ef5f25b retrieved body from server response 2020-07-24 18:32:28 +02:00
Enrico Ottonello e189283059 retrieved report on index results 2020-07-24 17:59:23 +02:00
Enrico Ottonello db9b70feb0 new node and workflow to index on ES 2020-06-16 02:36:16 +02:00
Alessia Bardi 4d68ecad58 the job fails if the endpoint returns client errors 4xx 2020-06-05 16:19:54 +02:00
Alessia Bardi e5cfbd01af ADS plugin now able to process one single remote file 2020-06-05 15:20:21 +02:00
Enrico Ottonello 819ef88520 new workflow to import periodo from url 2020-06-01 12:29:48 +02:00
Enrico Ottonello 30b3fa2140 New JobNode and workflow to enrich content on GraphDB 2020-05-29 16:19:55 +02:00
Enrico Ottonello 996150a315 modified access protocol parameter name (mandatory for aat and periodo), it is the name of the root node, inside the metadata record, that is generated converting original json record 2020-04-03 11:58:04 +02:00
Enrico Ottonello 7d6aa5de5e aat json collector that works on a folder 2020-02-19 16:22:22 +01:00
Enrico Ottonello 226cdde77d all records related to an api are published into the same graph 2020-02-19 14:33:54 +01:00
Enrico Ottonello a6b5a80984 modified generated xml according to new mapping 2020-02-14 16:56:34 +01:00
Enrico Ottonello c5945c386b plugin to collect one json file 2020-02-14 13:00:09 +01:00
Enrico Ottonello 17552bafa0 foreach published api, the datasource name is stored into graph datasourceApis 2020-01-24 10:52:04 +01:00
Enrico Ottonello 8530e488d0 all records related to a datasource api are now published on graphdb into n named graphs because of out of memory issue 2020-01-14 16:55:45 +01:00
Enrico Ottonello e748d3d802 added progress status data on publishToGraphDB workflow node ui 2019-12-18 15:55:53 +01:00
Enrico Ottonello 7980449ac0 CloseableHttpClient instance has to be closed after all post has been executed, same handling for PoolingHttpClientConnectionManager instance shutdown 2019-12-17 15:35:36 +01:00
Alessia Bardi 11e3992ab4 slow down the task producer to avoid OOM errors. See here for more: https://stackoverflow.com/questions/42108351/executorservice-giving-out-of-memory-error 2019-12-16 18:52:08 +01:00
Alessia Bardi c770fc40e0 cleaning up: let's remove the classes we do not need anymore, like Virtuoso-related classes and specific plugins that we have created for Parthenos 2019-12-16 18:34:08 +01:00
Enrico Ottonello 5087750f96 rdf handling move away from this node 2019-12-16 15:32:20 +01:00
Enrico Ottonello 98b452c59a new node to elastic search index 2019-12-16 15:25:55 +01:00
Enrico Ottonello 0f6f2e7b75 after records publishing the datasourceApi informations are saved on GraphDB 2019-12-13 14:55:51 +01:00
Enrico Ottonello 2f5fb6fcb5 new module for graphdb publishing 2019-12-12 12:58:30 +01:00
Enrico Ottonello b3a8f05c26 x3m mapping is now only taken from url; ads policy aligned to aggregator instance; removed wf parameter unused 2019-10-31 14:08:37 +01:00
Enrico Ottonello e968c5faa5 mapping url downloaded from remote url set in workflow parameters 2019-10-22 13:45:17 +02:00
Enrico Ottonello 4047d32587 all files under Access Protocol > Base URL folder are collected together 2019-10-21 15:43:29 +02:00
Enrico Ottonello 05209e8393 added custom plugin to ads datasource 2019-09-19 15:09:44 +02:00
Enrico Ottonello a9f6a0f6eb added modules related to ariadneplus 2019-06-18 16:15:23 +02:00