dnet-hadoop

sabeel

dnet-hadoop

forked from D-Net/dnet-hadoop

Fork 0

Commit Graph

Select branches

Hide Pull Requests

9559_DBLP_data

9647_datacite_affiliations

9872-create-solr-collection-aliases

CrossrefFundersMap

SWH_issue_377

UsageStatsRecordDS

WebCrowl

WebCrowlBeta

affRoFromRawString

affRoFromRawStringmain

base_stats_job_deprecated

beta

beta-release-1.2.5

beta_dedup_configuration

beta_indexing_May2024

broker_orcid

checkContextOnDatasourceAndProject

cleanup_actionmanager_deps

crossref_mapping_vocabularies

dedup-with-dataframe-spark34

dependency-revision

doiboost_authormerger

doidoost_dismiss

enrichment

fc4e-rsac

fix_mergedcliquesort

import_openorg_type

index_records

irish-oaipmh-exporter

json_payload

mag_only_doi

main

master

mergeutils

metadata_collection_java_upgrade

misc_fixes_merge_entities

oaf_country_beta

oldPropagationOrganizationCommunity

openorgs_fixes

openorgs_optimization

orcid_import

person

pivotselectionbypid

research_fi_collector_plugin

rest-collector-plugin-with-retry

scholix_data_type_openaire

scholix_flat_indexing

scholix_small_index

spark34-integration

spark_join_param_tuning

subjectPropagation

taggingProjects

ticket_8369

tweaking_spark_params

validation

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

archive/master_pre_stable_ids

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

september-2023

ef6c90cc64 implemented methods to extract fulltext link from an API call main sab 2024-09-11 14:57:38 +0200
df82f8beb9 code adapted as per Michele's recommendations sab 2024-09-04 15:29:13 +0200
53787dbf67 code refactored sab 2024-08-01 09:52:19 +0200
bbb79273a3 conversion to Dublin Core has been implemented sab 2024-08-01 01:23:04 +0200
7f39375ba8 data fetcher has been implemented sab 2024-07-31 18:05:11 +0200
d20a5e020a [graph provision] log the Solr admin application operations for alias deletion and creation Claudio Atzori 2024-07-15 16:30:43 +0200
38f8ed27fd [graph provision] log the Solr admin application operations for alias deletion and creation beta Claudio Atzori 2024-07-15 16:30:43 +0200
1fb44198fb renamed workflow to better reflect its purpose Claudio Atzori 2024-07-15 15:24:18 +0200
3d1d8e6036 renamed workflow to better reflect its purpose Claudio Atzori 2024-07-15 15:24:18 +0200
0b1c58358b Merge pull request '[broker] fixing the mapping of ORCID for the identification of the enrichments' (#458) from broker_orcid into main Claudio Atzori 2024-07-15 11:34:01 +0200
6f6e85ddf4 code formatting Claudio Atzori 2024-07-15 09:32:04 +0200
7fa3d51200 renamed class, updated criteria to consider the ORCIDs used in the matchers Claudio Atzori 2024-07-12 17:09:01 +0200
f99fb21040 tests Michele Artini 2024-07-12 15:29:45 +0200
b70a440aca renamed class, updated criteria to consider the ORCIDs used in the matchers broker_orcid Claudio Atzori 2024-07-12 17:09:01 +0200
36c3df1652 tests Michele Artini 2024-07-12 15:29:45 +0200
e17edb2581 [broker] fine tuned the workflow memory settings Claudio Atzori 2024-07-12 10:27:24 +0200
2f13683285 [broker] fine tuned the workflow memory settings Claudio Atzori 2024-07-12 10:27:24 +0200
61d1fa9b9f [metadata collection] added -Dcom.sun.security.enableAIAcaIssuers=true as a default for metadata collection Claudio Atzori 2024-07-12 10:25:51 +0200
5ab409dcab [metadata collection] added -Dcom.sun.security.enableAIAcaIssuers=true as a default for metadata collection Claudio Atzori 2024-07-12 10:25:51 +0200
d7b67469a7 [metadata collection] added -Dcom.sun.security.enableAIAcaIssuers=true as a default for metadata collection metadata_collection_java_upgrade Claudio Atzori 2024-07-12 10:25:51 +0200
f9ed2ae33c [metadata collection] added the possibility to specify the JAVA_HOME and the JAVA_OPTS parameters Claudio Atzori 2024-07-11 15:24:29 +0200
b756cfeb85 Merge pull request 'set JAVA_HOME and JAVA_OPTS in metadata collection' (#457) from metadata_collection_java_upgrade into main Claudio Atzori 2024-07-11 15:32:11 +0200
51d6a541bd [metadata collection] added the possibility to specify the JAVA_HOME and the JAVA_OPTS parameters Claudio Atzori 2024-07-11 15:24:29 +0200
bbe52584f7 log message research_fi_collector_plugin Michele Artini 2024-07-11 15:14:34 +0200
07ce92cef2 [OAI-PMH] fixed node name Claudio Atzori 2024-07-11 11:00:23 +0200
5cdba9172b implementeation of the new collector plugin: research_fi Michele Artini 2024-07-10 14:53:13 +0200
2a36ccb997 optimization of normalization stage in openorgs workflow, implementation of new comparators replacing older versions, openorgs configuration update, addition of inference flag in model definition, new test classes openorgs_optimization Michele De Bonis 2024-07-09 16:58:10 +0200
c465835061 [Person]new implementation for the extraction of the coAuthorship relations person Miriam Baglioni 2024-07-09 12:29:55 +0200
814e650e12 [Irish Tender]changed the irish.json file according to comments #26, #29, and #34 for 9635 Miriam Baglioni 2024-07-04 12:24:28 +0200
f043b7b096 [Irish Tender]changed the irish.json file according to comments #26, #29, and #34 for 9635 Miriam Baglioni 2024-07-04 12:22:56 +0200
ddd20e7f8e [Person]first implementation of the action set to include Person entity in the graph starting from the orcid data Miriam Baglioni 2024-07-04 12:08:46 +0200
153b56eeff make entity level pids unique by pidType:pidValue Claudio Atzori 2024-07-04 09:41:12 +0200
1180d78b71 make entity level pids unique by pidType:pidValue Claudio Atzori 2024-07-04 09:41:12 +0200
bb12d0b4df removed legacy actionmanager dependencies cleanup_actionmanager_deps Claudio Atzori 2024-07-03 16:26:39 +0200
ed97ba4565 Merge pull request '[prod] Openaire Affiliation Inference' (#453) from affRoFromRawStringmain into main Claudio Atzori 2024-07-03 12:32:26 +0200
7b398a6d0b updated import of organization types from OpenOrgs Claudio Atzori 2024-07-03 11:11:35 +0200
13f6506ce5 Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to slowly converge to records having DOI Claudio Atzori 2024-07-03 10:44:01 +0200
3d9ddaa23a importing organization types from OpenOrgs Claudio Atzori 2024-07-03 10:15:37 +0200
7d3292551b ignore dates containing 'null's Claudio Atzori 2024-07-02 15:43:11 +0200
c06dfdfd86 ignore dates containing 'null's Claudio Atzori 2024-07-02 15:43:11 +0200
8157f94ff2 [Provision]Branch to verify why the context tag for datasource and Projects are not visible in the index checkContextOnDatasourceAndProject Miriam Baglioni 2024-07-01 16:53:29 +0200
c7634c55c7 Merge pull request '[beta] implementation of countryMatch and addition of workflow parameters' (#451) from openorgs_fixes into beta Claudio Atzori 2024-07-01 09:22:56 +0200
b822b34abe code formatting Claudio Atzori 2024-07-01 09:22:35 +0200
ea1841fbd2 implementation of countryMatch and addition of workflow parameters Michele De Bonis 2024-06-28 16:46:52 +0200
4dbce39237 [AffiliationInference]Extended the affiliation ingestion from OpenAIRE to include also the links derived from web crawl. Changed the provenance from BIP! to OpenAIRE affRoFromRawStringmain Miriam Baglioni 2024-06-29 18:51:06 +0200
3ee8a7d18a [WebCrawl]moved to Constants web crawl name and id Miriam Baglioni 2024-06-29 18:47:23 +0200
a2b708bb71 [AffiliationIngestion]refactoring affRoFromRawString Miriam Baglioni 2024-06-29 18:36:47 +0200
9cbe966b4a [AffiliationIngestion]refactoring Miriam Baglioni 2024-06-29 18:35:49 +0200
236b64d830 [AffiliationIngestion]Extended the ingestion of affiliation from open aire to include also links derived from Web Crawl. Extended the test. Inserted in Constatns the id and name of the webcrawl datasource to be used here and also in the ingestion of links from web crawl Miriam Baglioni 2024-06-29 18:29:20 +0200
67ff783e65 [Person]First implementation to include Person entity in the graph Miriam Baglioni 2024-06-29 17:13:01 +0200
a10e8d9f05 implementation of countryMatch and addition of workflow parameters openorgs_fixes Michele De Bonis 2024-06-28 16:46:52 +0200
14539f9c8b [graph provision] publicFormat worfklow parameter defined as optional Claudio Atzori 2024-06-28 14:52:43 +0200
1bc8c5d173 [graph provision] fixed serialization of the instancetypes Claudio Atzori 2024-06-28 14:21:12 +0200
ee7deb3f60 [graph provision] publicFormat worfklow parameter defined as optional Claudio Atzori 2024-06-28 14:52:43 +0200
157cc8be87 [graph provision] fixed serialization of the instancetypes Claudio Atzori 2024-06-28 14:21:12 +0200
1ccf01cdb8 Using the updated Solr JSON payload model classes Claudio Atzori 2024-06-28 12:38:07 +0200
70b6e58914 minor change: wrong import updated master Michele De Bonis 2024-06-28 10:52:57 +0200
023099a921 imported from beta Claudio Atzori 2024-06-26 11:40:16 +0200
786c217085 Using the updated Solr JSON payload model classes Claudio Atzori 2024-06-26 11:11:33 +0200
b79cb155ba Merge pull request 'Fix permissions-issue in Stats-workflow, step22a-createPDFsAggregated.' (#450) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-06-26 10:11:34 +0200
c858c02111 - Fix not using the "export HADOOP_USER_NAME" statement in "createPDFsAggregated.sh", which caused permission-issues when creating tables with Impala. - Remove unused "--user" parameter in "impala-shell" calls. - Code polishing. Lampros Smyrnaios 2024-06-20 14:33:46 +0300
33a02c5b9e Merge pull request 'Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to converge to records having DOI pid' (#446) from pivotselectionbypid into beta Claudio Atzori 2024-06-26 10:10:13 +0200
1182bca9eb Merge pull request 'Add support to cretate/update solr collection aliases' (#449) from 9872-create-solr-collection-aliases into beta Claudio Atzori 2024-06-26 10:09:51 +0200
1c30eacac2 updated index feeding procedure to exploit the collection aliases 9872-create-solr-collection-aliases Claudio Atzori 2024-06-25 15:27:38 +0200
6055212f77 merged from the json_payload branch Claudio Atzori 2024-06-25 12:39:02 +0200
0031cf849e Merge branch 'beta' into 9872-create-solr-collection-aliases Claudio Atzori 2024-06-25 09:58:01 +0200
8220e27110 Merge pull request 'Align Solr JSON records to the explore portal requirements' (#448) from json_payload into beta_to_master_may2024 Claudio Atzori 2024-06-25 09:57:40 +0200
bc993d49c1 Update pom.xml json_payload Claudio Atzori 2024-06-25 09:57:06 +0200
1dc7458de2 added JSON payload to the SolrInputDocument, updated unit tests Claudio Atzori 2024-06-24 14:48:09 +0200
a7a54aab47 WIP: align Solr JSON records to the explore portal requirements Claudio Atzori 2024-06-20 15:48:45 +0200
9f6e16a03c Add support to cretate/update solr collection aliases Serafeim Chatzopoulos 2024-06-20 16:03:15 +0300
66cd28f70a - Fix not using the "export HADOOP_USER_NAME" statement in "createPDFsAggregated.sh", which caused permission-issues when creating tables with Impala. - Remove unused "--user" parameter in "impala-shell" calls. - Code polishing. Lampros Smyrnaios 2024-06-20 14:33:46 +0300
c6b1ab2a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Lampros Smyrnaios 2024-06-20 14:33:05 +0300
eaa00a4199 [IrishFunderList]make changed according to 9635 comment 20, 21, 22 and 23 Miriam Baglioni 2024-06-20 12:32:57 +0200
d35edac212 [IrishFunderList]make changed according to 9635 comment 20, 21, 22 and 23 Miriam Baglioni 2024-06-20 12:28:28 +0200
fb731b6d46 WIP: align Solr JSON records to the explore portal requirements Claudio Atzori 2024-06-19 15:38:43 +0200
6421f8fece Merge remote-tracking branch 'origin/beta' into beta Miriam Baglioni 2024-06-19 11:12:15 +0200
ac270f795b [IrishFunderList]make changed according to 9635 comment 14, 15 and 16 Miriam Baglioni 2024-06-19 11:11:52 +0200
b6da35e736 [IrishFunderList]make changed according to 9635 comment 14, 15 and 16 Miriam Baglioni 2024-06-19 11:06:58 +0200
2a7e5de094 added module for indexing scholexplorer scholix_small_index Sandro La Bruzzo 2024-06-18 17:06:01 +0200
236aed8954 Merge remote-tracking branch 'origin/beta' into beta Lampros Smyrnaios 2024-06-18 17:12:35 +0300
3c9b8de892 Miscellaneous updates to the copying operation to Impala Cluster: - Fix not breaking out of the VIEWS-infinite-loop when the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" is set to "false". - Exit the script when no HDFS-active-node was found, independently of the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR". - Fix view_name-recognition in a log-message, by using the more advanced "Perl-Compatible Regular Expressions" in "grep". - Add error-handling for "compute stats" errors. Lampros Smyrnaios 2024-06-18 14:40:41 +0300
c67ef157d3 filtering out deletedbyinference and invinsible results from accessroute Antonis Lempesis 2024-06-14 15:11:19 +0300
c23f3031ed Miscellaneous updates to the copying operation to Impala Cluster: - Show some counts and the elapsed time for various sub-tasks. - Code polishing. Lampros Smyrnaios 2024-06-14 12:14:38 +0300
8ec151aa3d [graph indexing] comment out setting the JSON payload from the SolrInputDocuments Claudio Atzori 2024-06-18 15:53:24 +0200
dd541f8cf5 Merge pull request 'Miscellaneous updates to the copying operation to Impala Cluster.' (#447) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-06-18 15:52:30 +0200
ff335578ea Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Lampros Smyrnaios 2024-06-18 14:52:31 +0300
285416c74e Merge branch 'beta' into beta Lampros Smyrnaios 2024-06-18 13:50:38 +0200
3095047e5e Miscellaneous updates to the copying operation to Impala Cluster: - Fix not breaking out of the VIEWS-infinite-loop when the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" is set to "false". - Exit the script when no HDFS-active-node was found, independently of the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR". - Fix view_name-recognition in a log-message, by using the more advanced "Perl-Compatible Regular Expressions" in "grep". - Add error-handling for "compute stats" errors. Lampros Smyrnaios 2024-06-18 14:40:41 +0300
0456f1b788 Merge remote-tracking branch 'origin/beta' into beta Antonis Lempesis 2024-06-14 15:11:30 +0300
38636942c7 filtering out deletedbyinference and invinsible results from accessroute Antonis Lempesis 2024-06-14 15:11:19 +0300
2636936162 [IE OAI-PMH] fixed oozie wf definition Claudio Atzori 2024-06-14 11:47:37 +0200
d942a1101b Miscellaneous updates to the copying operation to Impala Cluster: - Show some counts and the elapsed time for various sub-tasks. - Code polishing. Lampros Smyrnaios 2024-06-14 12:14:38 +0300
ef437a8cdf [Provision]temporarily removed Json paylod from indexed records (Shadow cannot support it) Miriam Baglioni 2024-06-13 16:48:03 +0200
9bf2bda1c6 Fix: next returned a null value at end of stream Giambattista Bloisi 2024-06-12 13:28:51 +0200
844a31f7a6 implemented new version of scholix using only the require index field and move the resource and target metadata on another index to reduce space Sandro La Bruzzo 2024-06-12 12:05:18 +0200
d90cb099b8 Fix for paginationStart parameter management Giambattista Bloisi 2024-06-11 20:23:44 +0200
86088ef26e Merge remote-tracking branch 'origin/beta_to_master_may2024' into beta_to_master_may2024 Miriam Baglioni 2024-06-11 17:04:07 +0200
143c525343 [WebCrawl]remove relations for pid not doi Miriam Baglioni 2024-06-11 17:03:59 +0200
4f2a61e10f Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to slowly converge to records having DOI pid pivotselectionbypid Giambattista Bloisi 2024-05-29 23:05:20 +0200