Claudio Atzori
bb12d0b4df
removed legacy actionmanager dependencies
2024-07-03 16:26:39 +02:00
Miriam Baglioni
a2b708bb71
[AffiliationIngestion]refactoring
2024-06-29 18:36:47 +02:00
Miriam Baglioni
9cbe966b4a
[AffiliationIngestion]refactoring
2024-06-29 18:35:49 +02:00
Miriam Baglioni
236b64d830
[AffiliationIngestion]Extended the ingestion of affiliation from open aire to include also links derived from Web Crawl. Extended the test. Inserted in Constatns the id and name of the webcrawl datasource to be used here and also in the ingestion of links from web crawl
2024-06-29 18:29:20 +02:00
Miriam Baglioni
67ff783e65
[Person]First implementation to include Person entity in the graph
2024-06-29 17:13:01 +02:00
Michele De Bonis
a10e8d9f05
implementation of countryMatch and addition of workflow parameters
2024-06-28 16:46:52 +02:00
Claudio Atzori
14539f9c8b
[graph provision] publicFormat worfklow parameter defined as optional
2024-06-28 14:55:18 +02:00
Claudio Atzori
1bc8c5d173
[graph provision] fixed serialization of the instancetypes
2024-06-28 14:54:28 +02:00
Claudio Atzori
1ccf01cdb8
Using the updated Solr JSON payload model classes
2024-06-28 12:38:07 +02:00
Claudio Atzori
b79cb155ba
Merge pull request 'Fix permissions-issue in Stats-workflow, step22a-createPDFsAggregated.' ( #450 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: #450
2024-06-26 10:11:34 +02:00
Claudio Atzori
33a02c5b9e
Merge pull request 'Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to converge to records having DOI pid' ( #446 ) from pivotselectionbypid into beta
...
Reviewed-on: #446
2024-06-26 10:10:13 +02:00
Claudio Atzori
1c30eacac2
updated index feeding procedure to exploit the collection aliases
2024-06-25 15:27:38 +02:00
Claudio Atzori
6055212f77
merged from the json_payload branch
2024-06-25 12:39:02 +02:00
Claudio Atzori
0031cf849e
Merge branch 'beta' into 9872-create-solr-collection-aliases
2024-06-25 09:58:01 +02:00
Serafeim Chatzopoulos
9f6e16a03c
Add support to cretate/update solr collection aliases
2024-06-20 16:03:15 +03:00
Lampros Smyrnaios
66cd28f70a
- Fix not using the "export HADOOP_USER_NAME" statement in "createPDFsAggregated.sh", which caused permission-issues when creating tables with Impala.
...
- Remove unused "--user" parameter in "impala-shell" calls.
- Code polishing.
2024-06-20 14:33:46 +03:00
Miriam Baglioni
d35edac212
[IrishFunderList]make changed according to 9635 comment 20, 21, 22 and 23
2024-06-20 12:28:28 +02:00
Miriam Baglioni
6421f8fece
Merge remote-tracking branch 'origin/beta' into beta
2024-06-19 11:12:15 +02:00
Miriam Baglioni
ac270f795b
[IrishFunderList]make changed according to 9635 comment 14, 15 and 16
2024-06-19 11:11:52 +02:00
Lampros Smyrnaios
285416c74e
Merge branch 'beta' into beta
2024-06-18 13:50:38 +02:00
Lampros Smyrnaios
3095047e5e
Miscellaneous updates to the copying operation to Impala Cluster:
...
- Fix not breaking out of the VIEWS-infinite-loop when the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" is set to "false".
- Exit the script when no HDFS-active-node was found, independently of the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR".
- Fix view_name-recognition in a log-message, by using the more advanced "Perl-Compatible Regular Expressions" in "grep".
- Add error-handling for "compute stats" errors.
2024-06-18 14:40:41 +03:00
Antonis Lempesis
0456f1b788
Merge remote-tracking branch 'origin/beta' into beta
2024-06-14 15:11:30 +03:00
Antonis Lempesis
38636942c7
filtering out deletedbyinference and invinsible results from accessroute
2024-06-14 15:11:19 +03:00
Lampros Smyrnaios
d942a1101b
Miscellaneous updates to the copying operation to Impala Cluster:
...
- Show some counts and the elapsed time for various sub-tasks.
- Code polishing.
2024-06-14 12:14:38 +03:00
Giambattista Bloisi
9bf2bda1c6
Fix: next returned a null value at end of stream
2024-06-12 13:28:51 +02:00
Giambattista Bloisi
d90cb099b8
Fix for paginationStart parameter management
2024-06-11 20:23:44 +02:00
Giambattista Bloisi
4f2a61e10f
Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to slowly converge to records having DOI pid
2024-06-11 15:33:56 +02:00
Claudio Atzori
11fe3a4fe0
[graph resolution] use sparkExecutorMemory to define also the memoryOverhead
2024-06-11 14:21:17 +02:00
Miriam Baglioni
8fe934810f
Merge remote-tracking branch 'origin/beta' into beta
2024-06-11 10:28:51 +02:00
Miriam Baglioni
9da006e98c
[SDGFoSActionSet]remove datainfo for the result. It is not needed (qualifier.classid = UPDATE) useless since subject do not go at the level of the instance
2024-06-11 10:28:32 +02:00
Giambattista Bloisi
85c1eae7e0
Fixes for pagination strategy looping at end of download
2024-06-10 19:03:58 +02:00
Claudio Atzori
b0eba210c0
[actionset promotion] use sparkExecutorMemory to define also the memoryOverhead
2024-06-10 16:15:24 +02:00
Claudio Atzori
3776327a8c
hostedby patching to work with the updated Crossref contents, resolved conflict
2024-06-10 15:24:12 +02:00
Claudio Atzori
0139f23d66
Merge pull request 'organization type from OpenOrgs' ( #445 ) from import_openorg_type into beta
...
Reviewed-on: #445
2024-06-07 12:17:31 +02:00
Michele Artini
c726572418
changed some parameters in OSF test
2024-06-07 12:03:26 +02:00
Claudio Atzori
ec79405cc9
[graph raw] set organization type from openorgs
2024-06-07 11:30:31 +02:00
Miriam Baglioni
1477406ecc
[bulkTag] fixed issue that made project disappear in graph_10_enriched
2024-06-06 10:45:41 +02:00
Claudio Atzori
92c3abd5a4
[graph cleaning] use sparkExecutorMemory to define also the memoryOverhead
2024-06-06 10:44:33 +02:00
Claudio Atzori
ce2364743a
applying changes from PR#442: Fix for missing collectedfrom after dedup
2024-06-06 10:43:43 +02:00
Claudio Atzori
f70dc76b61
minor
2024-06-06 10:43:10 +02:00
Claudio Atzori
73bd1938a5
[graph2hive] use sparkExecutorMemory to define also the memoryOverhead
2024-06-05 12:17:35 +02:00
Claudio Atzori
da5c1e73a4
Merge pull request 'Irish oaipmh exporter' ( #443 ) from irish-oaipmh-exporter into beta
...
Reviewed-on: #443
2024-06-05 10:55:09 +02:00
Claudio Atzori
81090ad593
[IE OAIPHM] added oozie workflow, minor changes, code formatting
2024-06-05 10:03:33 +02:00
Claudio Atzori
a02f3f0d2b
code formatting
2024-05-30 10:21:18 +02:00
Alessia Bardi
05ee783c07
Merge branch 'beta' into dblp_collection_plugin
2024-05-29 16:04:39 +02:00
Claudio Atzori
c272c4ad68
code formatting
2024-05-29 15:50:07 +02:00
Alessia Bardi
c5f4da16a4
Merge branch 'beta' into rest-collector-request-header-map
2024-05-29 15:46:23 +02:00
Alessia
1b165a14a0
Rest collector plugin on hadoop supports a new param to pass request headers
2024-05-29 15:41:36 +02:00
Michele Artini
e996787be2
OSF test
2024-05-29 15:05:17 +02:00
Claudio Atzori
62716141c5
Merge pull request 'Miscellaneous updates to the copying operation to Impala Cluster' ( #440 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: #440
2024-05-29 14:34:51 +02:00
Miriam Baglioni
5d85b70e1f
[NOAMI] removed Ireland funder id 501100011103. ticket 9635
2024-05-29 11:55:00 +02:00
Lampros Smyrnaios
e3f28338c1
Miscellaneous updates to the copying operation to Impala Cluster:
...
- Assign the WRITE and EXECUTE permissions to the DBs' HDFS-directories, in order to be able to create tables on top of them, in the Impala Cluster.
- Make sure the "copydb" function returns early, when it encounters a fatal error, while respecting the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" config.
2024-05-28 17:51:45 +03:00
Miriam Baglioni
75d5ddb999
Update to include a blackList that filters out the results we know are wrongly associated to IE - update workflow definition - the blacklist parameter
2024-05-27 12:01:28 +02:00
Miriam Baglioni
87c9c61b41
Update to include a blackList that filters out the results we know are wrongly associated to IE - refactoring
2024-05-27 12:01:16 +02:00
Miriam Baglioni
b55fed09f8
Update to include a blackList that filters out the results we know are wrongly associated to IE
2024-05-27 12:01:01 +02:00
Claudio Atzori
107d958b89
[org dedup] avoid NPEs in SparkPrepareNewOrgs
2024-05-27 11:59:54 +02:00
Claudio Atzori
3a7a6ecc32
[org dedup] avoid NPEs in SparkPrepareOrgRels
2024-05-27 11:59:45 +02:00
Claudio Atzori
1af4224d3d
[org dedup] avoid NPEs in SparkPrepareOrgRels
2024-05-27 11:59:33 +02:00
Claudio Atzori
0d5bdb2db0
Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta
2024-05-27 11:59:02 +02:00
Claudio Atzori
66548e6a83
Merge pull request 'changes in copy script' ( #438 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: #438
2024-05-27 11:54:03 +02:00
Antonis Lempesis
15b54a345a
added fos lvl4
2024-05-24 13:21:28 +03:00
Lampros Smyrnaios
b48ed6e617
Change configuration in the copy-operation to Impala Cluster:
...
Set the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" parameter to "false".
2024-05-23 16:58:12 +03:00
Lampros Smyrnaios
68322843e2
Small updates to the copy-operation to Impala Cluster:
...
- Add a configuration-"switch" to control whether the script exits upon an error or not.
- Allow the script to exit when a table could not be created.
- Show the elapsed time for processing each database.
2024-05-23 15:07:49 +03:00
Lampros Smyrnaios
c7b32bbacc
Update CopyDataToImpalaCluster:
...
Update the code of acquiring the entities from Ocean cluster, through hive, in order to optimize the process and account for additional reserved keywords in Impala.
Co-authored-by: Antonis Lempesis <antleb@di.uoa.gr>
2024-05-23 13:00:19 +03:00
Sandro La Bruzzo
f1fe363b19
merged again from beta (I hope for the last time)
2024-05-22 11:08:52 +02:00
Sandro La Bruzzo
66c1ffc866
merged again from beta (I hope for the last time)
2024-05-22 11:02:46 +02:00
Claudio Atzori
1ea67eba82
Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta
2024-05-21 13:48:48 +02:00
Claudio Atzori
834461ba26
[graph provision]fixed wf definition, revised serialization of the usage counts measures
2024-05-21 13:48:06 +02:00
Sandro La Bruzzo
e8a61d5dd5
removed plugin, use only FileGZip plugin
2024-05-21 13:45:29 +02:00
Sandro La Bruzzo
ca9414b737
Implement multiple node name splitter on GZipCollectorPlugin and all nodes that use XMLIterator. If the splitter name contains is a comma separated values it splits for all the values
2024-05-21 09:11:13 +02:00
Sandro La Bruzzo
032bcc8279
since last beta workflow we decide to introduce in the graph only MAG item with DOI and set them invisible ( this should be the same behaviour of the previous DOIBoost mapping).
...
This commit apply this type of mapping
2024-05-20 09:24:15 +02:00
Sandro La Bruzzo
103e2652b3
merged beta
2024-05-17 14:43:07 +02:00
Sandro La Bruzzo
a87f9ea643
fixed scholexplorer bug
2024-05-17 14:16:43 +02:00
Sandro La Bruzzo
6efab4d88e
fixed scholexplorer bug
2024-05-16 16:19:18 +02:00
Claudio Atzori
92f018d196
[graph provision] fixed path pointing to an intermediate data store in the working directory
2024-05-15 15:39:18 +02:00
Claudio Atzori
0611c81a2f
[graph provision] using Qualifier.classNames to populate the correponsing fields in the JSON payload
2024-05-15 15:33:10 +02:00
Michele Artini
2b3b5fe9a1
oai finalization and test
2024-05-15 14:13:16 +02:00
Claudio Atzori
1efe7f7e39
[graph provision] upgrade to dhp-schema:6.1.2, included project.oamandatepublications in the JSON payload mapping, fixed serialisation of the usageCounts measures
2024-05-14 12:39:31 +02:00
Claudio Atzori
f7d56e2ef2
Merge branch 'beta' into rest-collector-plugin-with-retry
2024-05-10 09:02:21 +02:00
Claudio Atzori
dc3a5858f7
Merge branch 'beta' into beta_provision_relation
2024-05-09 14:14:43 +02:00
Claudio Atzori
55f39f7850
[graph provision] adds the possibility to validate the XML records before storing them via the validateXML parameter
2024-05-09 14:06:04 +02:00
Claudio Atzori
39a2afe8b5
[graph provision] fixed XML serialization of the usage counts measures, renamed workflow actions to better reflect their role
2024-05-09 13:54:42 +02:00
Claudio Atzori
908ed9da7a
Merge pull request 'Various fixes in the stats wf' ( #430 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: #430
2024-05-08 13:41:02 +02:00
Antonis Lempesis
0cada3cc8f
every step is run in the analytics queue. Hardcoded for now, will make a parameter later
2024-05-08 13:42:53 +03:00
Antonis Lempesis
90a4fb3547
fixed typos
2024-05-08 13:17:58 +03:00
Claudio Atzori
18aa323ee9
cleanup unused classes, adjustments in the oozie wf definition
2024-05-08 11:36:46 +02:00
Michele Artini
c9a327bc50
refactoring of gzip method
2024-05-08 11:34:08 +02:00
Michele Artini
e234848af8
oaf record: xpath for root
2024-05-08 10:00:53 +02:00
Claudio Atzori
b4e3389432
fixed property mapping creating the RelatedEntity transient objects. spark cores & memory adjustments. Code formatting
2024-05-07 16:25:17 +02:00
Giambattista Bloisi
711048ceed
PrepareRelationsJob rewritten to use Spark Dataframe API and Windowing functions
2024-05-07 15:44:33 +02:00
Michele Artini
70bf6ac415
oai exporter tests
2024-05-07 09:36:26 +02:00
Michele Artini
aa40e53c19
oai exporter parameters
2024-05-07 08:01:19 +02:00
Michele Artini
ed052a3476
job for the population of the oai database
2024-05-06 16:08:33 +02:00
Claudio Atzori
26363060ed
fixed id prefix creation for the fosnodoi records, again
2024-05-03 15:53:52 +02:00
Claudio Atzori
0486227185
[cleaning] deactivating the cleaning of FOS subjects found in the metadata provided by repositories
2024-05-03 14:31:12 +02:00
Claudio Atzori
e1a0fb8933
fixed id prefix creation for the fosnodoi records
2024-05-03 14:14:18 +02:00
Sandro La Bruzzo
26bf8e763a
merged from beta
2024-05-02 15:20:23 +02:00
Sandro La Bruzzo
0646d0d064
Updated main sparkApplication to avoid to require master variable
2024-05-02 15:15:03 +02:00
Claudio Atzori
4355f64810
reverted to version 1.2.5-SNAPSHOT
2024-05-02 11:23:53 +02:00
Claudio Atzori
66680b8b9a
refactoring of common utilities
2024-05-02 11:16:58 +02:00