dnet-hadoop

lsmyrnaios

dnet-hadoop

forked from D-Net/dnet-hadoop

Fork 0

Commit Graph

Select branches

Hide Pull Requests

7096-fileGZip-collector-plugin

8172_impact_indicators_workflow

8232-mdstore-synch-improve

8876

9078_xml_records_irish_tender

SWH_integration

UsageStatsRecordDS

advConstraintsInBeta

apc_affiliation

beta

beta_dedup_configuration

bulkTag

bulkTagRefactor

citations_monodirectional

cleanCountryOnMaster

clean_context_master

clean_license_publisher

compatibility_order

continuous-validation

continuous_validation2

dedup-with-dataframe-spark34

dedup_increasenumofblocks

deduptesting

discard-non-wellformed

distinct_pids_from_openorgs_beta

doiboostFunderExtention

doiboostMappingExtention

doiboost_authormerger

doiboost_refactor

dump

dump_delta_projects

dump_new_funded_products

enrichment

eosc_context_tagging

fc4e-rsac

fulltext_url_validation

graph_cleaning

graph_cleaning_refactoring

h2020classification

horizontalConstraints

hostedByMap_oastartdate

hostedByMap_update

importpoci

instance_group_by_url

mag_citation_relation

master

master_july23

monitoring

native_records_migration

oaf_relation_mapping

orcid_import

organizationToRepresentative

propagationProjectThroughParentChils

pubmed_to_production

pubmed_update

removeDump

removeTaggingCondition

scholix_data_type_openaire

scholix_flat_indexing

scholix_to_solr

spark34-integration

subjectPropagation

sygma_indexing

ticket_8369

transformation_wf

tweaking_spark_params

updateCopyScripts

validation

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

archive/master_pre_stable_ids

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

september-2023

a81c555fe6 [graph provision] include only FoS L1..L2 in the record serialization Claudio Atzori 2024-07-25 15:26:47 +0200
359b8ebda8 [graph provision] include only FoS L1..L2 in the record serialization Claudio Atzori 2024-07-25 15:22:29 +0200
c7f6669f1a [webcrawl] the blacklist is now in json and no more in csv after the normalization process Miriam Baglioni 2024-07-25 15:20:18 +0200
7cff281d3e [webcrawl] the blacklist is now in json and no more in csv after the normalization process Miriam Baglioni 2024-07-25 15:16:42 +0200
d4bf449e8c minor Claudio Atzori 2024-07-25 14:53:06 +0200
fc60661ac5 [webcrawl] added code and test (code/resource) to verify the deletion of the relations related to results put in blacklist Miriam Baglioni 2024-07-25 12:25:14 +0200
d771a883f9 [dedup] updated sql query used to read organizations from the OpenOrgs DB to include their typology Claudio Atzori 2024-07-25 09:53:48 +0200
01958a3e07 [graph provision] addded filter to exclude records marked with datainfo.deletedbyinference = true Claudio Atzori 2024-07-24 10:00:10 +0200
ceb210993c Merge pull request 'SDG no DOI' (#464) from sdgnodoi into beta Claudio Atzori 2024-07-24 09:59:13 +0200
6f1801d7d1 [webcrawl]- Miriam Baglioni 2024-07-23 17:34:48 +0200
19806c2ae3 [SDG]fixed switch of methods Miriam Baglioni 2024-07-23 17:12:55 +0200
d0590e0e49 added latest institutions Antonis Lempesis 2024-07-23 15:17:15 +0300
7d2c0a3723 added new institutions Antonis Lempesis 2024-07-23 15:10:17 +0300
62649dc5c4 merging with branch beta Miriam Baglioni 2024-07-23 12:50:12 +0200
9573bf576d [SDG]added code to ingest also the SDG without DOI Miriam Baglioni 2024-07-23 12:47:57 +0200
d27e9ea50f added ODF invisible stores in raw_all workflow Michele Artini 2024-07-23 09:56:27 +0200
4f4c73d65b minor change: addition of missing parameter in sql query Michele De Bonis 2024-07-22 15:19:02 +0200
79985ad197 [Crossref]added mapping for DFG versus the unidentified project [https://support.openaire.eu/issues/9926?next_issue_id=9924&prev_issue_id=9927#note-4] Miriam Baglioni 2024-07-17 18:30:24 +0200
c25b048e12 Merge pull request 'PersonEntity' (#459) from person into beta Claudio Atzori 2024-07-17 12:02:24 +0200
06e3985b77 merged from beta Claudio Atzori 2024-07-17 12:01:40 +0200
83327239de fixed pom definitions, bumped dependency version for the dhp-schema module, removed unnecessary dependencies Claudio Atzori 2024-07-17 11:58:48 +0200
db9c54c944 Revert "removed legacy actionmanager dependencies" Claudio Atzori 2024-07-17 11:27:43 +0200
e39e8bbd47 Merge pull request '[WebCrawlAffiliation]remove from the creation of the action set the relations for pmc and pmid. Only doi are allowed' (#462) from affiliationFromWebCrawlOnlyDOI into beta Claudio Atzori 2024-07-17 11:12:32 +0200
e94ae771ff Merge pull request '[BulkTag]added tagging for the organization relevant for the community.' (#461) from tagOrganization into beta Claudio Atzori 2024-07-17 11:11:52 +0200
6c98d69215 reverted changed contens under dhp-pace-core Claudio Atzori 2024-07-17 11:09:37 +0200
78b5e4bb6f reverted changed contens under dhp-graph-provision Claudio Atzori 2024-07-17 10:48:20 +0200
40c5d87645 Merge pull request '[graph provision] entity level contexts' (#460) from entity_contexts into beta Claudio Atzori 2024-07-17 10:43:21 +0200
a65241fcaf Merge pull request 'implementation of the new collector plugin: research_fi' (#456) from research_fi_collector_plugin into beta Claudio Atzori 2024-07-17 10:25:38 +0200
6665976604 Merge pull request 'Optimizations for the Openorgs Dedup: normalization and inference of strings and implementation of new general-purpose comparators' (#455) from openorgs_optimization into beta Claudio Atzori 2024-07-17 10:25:20 +0200
c99f92efaa Merge pull request '[beta] OpenAIRE Affiliation Inference' (#452) from affRoFromRawString into beta Claudio Atzori 2024-07-17 10:24:39 +0200
f17e1243ba reverted changed contens under dhp-graph-provision Claudio Atzori 2024-07-17 10:23:50 +0200
6a19337dab Merge pull request 'removed legacy actionmanager dependencies' (#454) from cleanup_actionmanager_deps into beta Claudio Atzori 2024-07-17 10:20:44 +0200
d96215cb9b [UnpayWall]added othe : in the identifier construction Miriam Baglioni 2024-07-16 18:17:32 +0200
9246bdec1c [WebCrawlAffiliation]remove from the creation of the action set the relations for pmc and pmid. Only doi are allowed Miriam Baglioni 2024-07-16 14:07:37 +0200
9d27910144 [BulkTag]added tagging for the organization relevant for the community. Added test. Changed the tagging variables. Miriam Baglioni 2024-07-16 13:48:48 +0200
beb93cdfe9 [graph provision] expand the context info for each entity type Claudio Atzori 2024-07-16 11:43:48 +0200
5aa7847ea6 consider the transformative agreement text when merging results Claudio Atzori 2024-07-16 10:38:50 +0200
38f8ed27fd [graph provision] log the Solr admin application operations for alias deletion and creation Claudio Atzori 2024-07-15 16:30:43 +0200
1fb44198fb renamed workflow to better reflect its purpose Claudio Atzori 2024-07-15 15:24:18 +0200
6f6e85ddf4 code formatting Claudio Atzori 2024-07-15 09:32:04 +0200
7fa3d51200 renamed class, updated criteria to consider the ORCIDs used in the matchers Claudio Atzori 2024-07-12 17:09:01 +0200
f99fb21040 tests Michele Artini 2024-07-12 15:29:45 +0200
e17edb2581 [broker] fine tuned the workflow memory settings Claudio Atzori 2024-07-12 10:27:24 +0200
61d1fa9b9f [metadata collection] added -Dcom.sun.security.enableAIAcaIssuers=true as a default for metadata collection Claudio Atzori 2024-07-12 10:25:51 +0200
f9ed2ae33c [metadata collection] added the possibility to specify the JAVA_HOME and the JAVA_OPTS parameters Claudio Atzori 2024-07-11 15:24:29 +0200
bbe52584f7 log message Michele Artini 2024-07-11 15:14:34 +0200
5cdba9172b implementeation of the new collector plugin: research_fi Michele Artini 2024-07-10 14:53:13 +0200
2a36ccb997 optimization of normalization stage in openorgs workflow, implementation of new comparators replacing older versions, openorgs configuration update, addition of inference flag in model definition, new test classes Michele De Bonis 2024-07-09 16:58:10 +0200
c465835061 [Person]new implementation for the extraction of the coAuthorship relations Miriam Baglioni 2024-07-09 12:29:55 +0200
814e650e12 [Irish Tender]changed the irish.json file according to comments #26, #29, and #34 for 9635 Miriam Baglioni 2024-07-04 12:24:28 +0200
ddd20e7f8e [Person]first implementation of the action set to include Person entity in the graph starting from the orcid data Miriam Baglioni 2024-07-04 12:08:46 +0200
1180d78b71 make entity level pids unique by pidType:pidValue Claudio Atzori 2024-07-04 09:41:12 +0200
e9686365a2 Improve performance of creating the "result_fos" table, by using a temp-table to cache data, which is requested multiple times. Lampros Smyrnaios 2024-07-03 20:24:36 +0300
ce0aee21cc Improve performance of transferring the stats-DBs to another cluster and querying the DBs' tables, by ordering Spark to create up to 100 files per table, instead of thousands. Lampros Smyrnaios 2024-07-03 20:15:33 +0300
7b7dd32ad5 - Fix placement of some "set mapred.job.queue.name=analytics" statements and remove their unused "/*EOS*/" indicator. - Add stacktrace-info to failed actions. Lampros Smyrnaios 2024-07-03 19:53:24 +0300
7ce051d766 - Update the remaining hive-actions to spark-actions. - Update the version of shell-actions. - Fix missing "/*EOS*/" indicators. Lampros Smyrnaios 2024-07-03 19:49:19 +0300
aa4d7d5e20 Prioritize the rest of the stats-queries over other tasks on the cluster, by putting them in the "analytics" queue. Lampros Smyrnaios 2024-07-03 19:14:25 +0300
bb12d0b4df removed legacy actionmanager dependencies Claudio Atzori 2024-07-03 16:26:39 +0200
54e11b6a43 Improve performance and efficiency by rewriting the creation process of "publication", "project", "dataset", "datasource", "software", "otherresearchproduct" and "result" tables, to be performed in a single query, for each one. Lampros Smyrnaios 2024-07-03 13:03:15 +0300
7d3292551b ignore dates containing 'null's Claudio Atzori 2024-07-02 15:43:11 +0200
c7634c55c7 Merge pull request '[beta] implementation of countryMatch and addition of workflow parameters' (#451) from openorgs_fixes into beta Claudio Atzori 2024-07-01 09:22:56 +0200
a2b708bb71 [AffiliationIngestion]refactoring Miriam Baglioni 2024-06-29 18:36:47 +0200
9cbe966b4a [AffiliationIngestion]refactoring Miriam Baglioni 2024-06-29 18:35:49 +0200
236b64d830 [AffiliationIngestion]Extended the ingestion of affiliation from open aire to include also links derived from Web Crawl. Extended the test. Inserted in Constatns the id and name of the webcrawl datasource to be used here and also in the ingestion of links from web crawl Miriam Baglioni 2024-06-29 18:29:20 +0200
67ff783e65 [Person]First implementation to include Person entity in the graph Miriam Baglioni 2024-06-29 17:13:01 +0200
a10e8d9f05 implementation of countryMatch and addition of workflow parameters Michele De Bonis 2024-06-28 16:46:52 +0200
14539f9c8b [graph provision] publicFormat worfklow parameter defined as optional Claudio Atzori 2024-06-28 14:52:43 +0200
1bc8c5d173 [graph provision] fixed serialization of the instancetypes Claudio Atzori 2024-06-28 14:21:12 +0200
1ccf01cdb8 Using the updated Solr JSON payload model classes Claudio Atzori 2024-06-28 12:38:07 +0200
b79cb155ba Merge pull request 'Fix permissions-issue in Stats-workflow, step22a-createPDFsAggregated.' (#450) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-06-26 10:11:34 +0200
33a02c5b9e Merge pull request 'Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to converge to records having DOI pid' (#446) from pivotselectionbypid into beta Claudio Atzori 2024-06-26 10:10:13 +0200
1182bca9eb Merge pull request 'Add support to cretate/update solr collection aliases' (#449) from 9872-create-solr-collection-aliases into beta Claudio Atzori 2024-06-26 10:09:51 +0200
fe2275a9b0 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into convert_hive_to_spark_actions Lampros Smyrnaios 2024-06-25 20:17:47 +0300
1c30eacac2 updated index feeding procedure to exploit the collection aliases Claudio Atzori 2024-06-25 15:27:38 +0200
6055212f77 merged from the json_payload branch Claudio Atzori 2024-06-25 12:39:02 +0200
0031cf849e Merge branch 'beta' into 9872-create-solr-collection-aliases Claudio Atzori 2024-06-25 09:58:01 +0200
9f6e16a03c Add support to cretate/update solr collection aliases Serafeim Chatzopoulos 2024-06-20 16:03:15 +0300
66cd28f70a - Fix not using the "export HADOOP_USER_NAME" statement in "createPDFsAggregated.sh", which caused permission-issues when creating tables with Impala. - Remove unused "--user" parameter in "impala-shell" calls. - Code polishing. Lampros Smyrnaios 2024-06-20 14:33:46 +0300
c6b1ab2a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Lampros Smyrnaios 2024-06-20 14:33:05 +0300
d35edac212 [IrishFunderList]make changed according to 9635 comment 20, 21, 22 and 23 Miriam Baglioni 2024-06-20 12:28:28 +0200
6421f8fece Merge remote-tracking branch 'origin/beta' into beta Miriam Baglioni 2024-06-19 11:12:15 +0200
ac270f795b [IrishFunderList]make changed according to 9635 comment 14, 15 and 16 Miriam Baglioni 2024-06-19 11:11:52 +0200
236aed8954 Merge remote-tracking branch 'origin/beta' into beta Lampros Smyrnaios 2024-06-18 17:12:35 +0300
dd541f8cf5 Merge pull request 'Miscellaneous updates to the copying operation to Impala Cluster.' (#447) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-06-18 15:52:30 +0200
ff335578ea Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Lampros Smyrnaios 2024-06-18 14:52:31 +0300
285416c74e Merge branch 'beta' into beta Lampros Smyrnaios 2024-06-18 13:50:38 +0200
3095047e5e Miscellaneous updates to the copying operation to Impala Cluster: - Fix not breaking out of the VIEWS-infinite-loop when the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" is set to "false". - Exit the script when no HDFS-active-node was found, independently of the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR". - Fix view_name-recognition in a log-message, by using the more advanced "Perl-Compatible Regular Expressions" in "grep". - Add error-handling for "compute stats" errors. Lampros Smyrnaios 2024-06-18 14:40:41 +0300
0456f1b788 Merge remote-tracking branch 'origin/beta' into beta Antonis Lempesis 2024-06-14 15:11:30 +0300
38636942c7 filtering out deletedbyinference and invinsible results from accessroute Antonis Lempesis 2024-06-14 15:11:19 +0300
d942a1101b Miscellaneous updates to the copying operation to Impala Cluster: - Show some counts and the elapsed time for various sub-tasks. - Code polishing. Lampros Smyrnaios 2024-06-14 12:14:38 +0300
9bf2bda1c6 Fix: next returned a null value at end of stream Giambattista Bloisi 2024-06-12 13:28:51 +0200
d90cb099b8 Fix for paginationStart parameter management Giambattista Bloisi 2024-06-11 20:23:44 +0200
4f2a61e10f Change the selection criteria for the pivot record of a group so that by best pid type becomes the first criteria. This will have the effect to slowly converge to records having DOI pid Giambattista Bloisi 2024-05-29 23:05:20 +0200
11fe3a4fe0 [graph resolution] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-06-11 14:21:01 +0200
a8d68c9d29 avoid NPEs Claudio Atzori 2024-06-11 12:40:50 +0200
8fe934810f Merge remote-tracking branch 'origin/beta' into beta Miriam Baglioni 2024-06-11 10:28:51 +0200
9da006e98c [SDGFoSActionSet]remove datainfo for the result. It is not needed (qualifier.classid = UPDATE) useless since subject do not go at the level of the instance Miriam Baglioni 2024-06-11 10:28:32 +0200
85c1eae7e0 Fixes for pagination strategy looping at end of download Giambattista Bloisi 2024-06-10 19:03:30 +0200
b0eba210c0 [actionset promotion] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-06-10 16:15:07 +0200
3776327a8c hostedby patching to work with the updated Crossref contents, resolved conflict Claudio Atzori 2024-06-10 15:22:33 +0200