Master branch updates from beta December 2023 #369

claudio.atzori · 2023-12-15T10:15:25+01:00

claudio.atzori commented

2023-12-15 10:15:25 +01:00

This PR brings to the master branch the changes available from the beta branch at December 2023.

#350 COAR based resource types & Irish tender fields
#352 URL Validator to accept double slashes
#353 Add Pubmed affiliations (inferred by BIP) as actionsets
#355 StatsDB workflow to export actionsets about OA routes, diamond, and publicly-funded
#356 graph cleaning, suggestions from ticket 8898 - round 2
#359 [ENRICHMENT][BETA] Use of community API in enrichment process AND addition to tagging result for communities through projects
#363 Changes for tables and creation of the new indicator indi_is_result_accessible
#364 ORCID Enrichment and Download
#365, #368 [graph provision] added serialization for the new fields imported from the stats DB
#366 [graph cleaning] added cleaning for result.publisher and result.instance.license

This PR brings to the master branch the changes available from the beta branch at December 2023. * #350 COAR based resource types & Irish tender fields * #352 URL Validator to accept double slashes * #353 Add Pubmed affiliations (inferred by BIP) as actionsets * #355 StatsDB workflow to export actionsets about OA routes, diamond, and publicly-funded * #356 graph cleaning, suggestions from ticket 8898 - round 2 * #359 [ENRICHMENT][BETA] Use of community API in enrichment process AND addition to tagging result for communities through projects * #363 Changes for tables and creation of the new indicator `indi_is_result_accessible` * #364 ORCID Enrichment and Download * #365, #368 [graph provision] added serialization for the new fields imported from the stats DB * #366 [graph cleaning] added cleaning for `result.publisher` and `result.instance.license`

claudio.atzori added 101 commits 2023-12-15 10:15:26 +01:00

3d6be20989 changes to use the API instead of the IS the get the information for the communities to be used during bulktagging and context propagation

8448b9ebfb mergin with branch beta

a3d01ccb24 refactoring

554551682d [raw graph] adopting the new COAR based vocabularies for the resource typing

89184d5b4f used the API instead of the IS for bulktagging and propagation for community through organization. Added a new propagation step for communities through projects. Still using the API and not the IS

8e9493fad9 mergin with branch beta

54fbf09ac6 [raw graph] WIP: mapping original resource types

159388f9c2 testing and fix some issues

a5a89a702f new spark parrameter updated

39d24d5469 Merge branch 'beta' into resource_types

6dfcd0c9a2 [raw graph] mapping original resource types

d28b7085f6 more NPE checks

a24178cb93 Merge branch 'beta' into resource_types

f1b898c6b4 mergin with branch beta

2b9d0416ec [graph raw] URL Validator to accept double slashes

6b19dcee80 Add actionset creation for pubmed affiliations

a4214ced1e fixing issue on propagation organization. added --config to workflow definition. added oozie_app to communtiy project

aad5982bf1 Change the description of the workflow

89c4dfbaf4 StatsDB workflow to export actionsets about OA routes, diamond, and publicly-funded

A new oozie workflow capable to read from the stats db to produce a new actionSet for updating results with:
- green_oa ={true, false}
- openAccesColor = {gold, hybrid, bronze}
- in_diamond_journal={true, false}
- publicly_funded={true, false}

Inputs:

- outputPath
- statsDB

b3a61ea955 Merge branch 'beta' into url_validation

a82aaf57b2 Renaming input param for crossref input path

2090003ea9 Adjust tests to new WF input params

262d7c581b [graph cleaning] implemented further suggestions from https://support.openaire.eu/issues/8898

dde2fec035 [graph cleaning] cleanup

e9a20fc8f6 mergin with branch beta

c80b768af0 test for project propagation

a737dd47b6 removed not needed test class

937ff6a7c7 -

edcb17ca98 refactoring and test

638ad9e74f changing test for new implementation

7b1e34f159 refactoring

5bc97615d5 -

d524e30866 Changes to actionsets

Resolve comments from
#355

6ce36b3e41 Implemented ORCID Workflow on DHP-Aggregation for retrieving ORCID DUMP and generating tables

eaf0a702de -

a94a54a2d0 Changes for tables and creation of the new indicator indi_is_result_accessible

- Drop table statements for all tables to avoid duplicates in case of wf rerun
- Add pdfsaggregated step to create the indi_is_result_accessible table. This step is executed on the new impala cluster only, since the pdfaggregation_i is updated on this cluster.

11a1207f9c [graph cleaning] applying coar based vocabularies in bulk

008fdf9d8a Merge pull request 'URL Validator to accept double slashes' (#352 ) from url_validation into beta

Reviewed-on: #352

745039ad5b Merge branch 'beta' into 9117_pubmed_affiliations

836d7ec724 Merge pull request 'Add Pubmed affiliations (inferred by BIP) as actionsets' (#353 ) from 9117_pubmed_affiliations into beta

Reviewed-on: #353

2c77638bf5 Merge branch 'beta' into cleaning_8898

a21617732a Merge pull request 'graph cleaning, suggestions from ticket 8898 - round 2' (#356 ) from cleaning_8898 into beta

Reviewed-on: #356

359e81b7a6 Update StatsAtomicActionsJob.java

Bug fix for duplicate bronze checks

1ba582de3c [graph cleaning] added cleaning for result.publisher and result.instance.license

34a4b3cbdf Implemented ORCID Enrichment

48e0427a23 changed the parameter from production to baseURL. Fixed issue in tagging configuration

e3cce9a5a0 mergin with branch beta

8eb70e6657 refactoring

6f4d0c05ea Implemented Author MErger for ORCID that takes in account the case when name and surname are swapped

59111713fa added comment

aa239ec673 Changed implementation of check similarity to verify exact match of name instead of the first char

279100fa52 added test

af1c2634b3 added instanceTypeMapping original field in the mapping of

- DOIBoost
- Datacite
- PubMed
- Scholexplorer Datasource

c96ff54b45 Merge remote-tracking branch 'origin/resource_types' into resource_types

86b5775e08 added vocabulary in instanceTypeMapping for

- DOIBoost
- Datacite
- PubMed
- Scholexplorer Datasource

4e1aac2e2f resolved conflict in pom.xml before applying the changes from [COAR based resource types & Irish tender] #350

6f10791e77 Merge branch 'beta' into propagationapi

4cbabc9fbc Merge pull request '[ENRICHMENT][BETA] Use of community API in enrichment process AND addition to tagging result for communities through projects' (#359 ) from propagationapi into beta

Reviewed-on: #359

7b5e04f37e removed Orcid intersection on DOIBoost

f718caaac9 Added copy of the untouched entities of the graph

5e22b67b8a Merge remote-tracking branch 'origin/beta' into orcid_import

cdfb7588dd code formatting

48430a32a6 Update StatsAtomicActionsJob.java

Added indi_funded_result_with_fundref indicator

bf0fd27c36 Removed unused function

Applied PR Comment of Giambattista in the PR

622fafbd2e Merge branch 'beta' into orcid_import

76594ded23 Changes to indicators

Fixes on open access colours indicators
- indi_pub_green_oa
- indi_pub_gold_oa
- indi_pub_hybrid
- indi_pub_bronze_oa
- indi_pub_diamond

a397112cb8 Add new indicator

Add indi_pub_publicly_funded

c9d995dde0 New institutions added

33cb483c75 using objectSubType as originalType in Crossref2Oaf, code formatting

0c3c9ea43d Merge pull request 'StatsDB workflow to export actionsets about OA routes, diamond, and publicly-funded' (#355 ) from dimitris.pierrakos/dnet-hadoop:beta into beta

Reviewed-on: #355

93a700742a Merge pull request 'Changes for tables and creation of the new indicator indi_is_result_accessible' (#363 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #363

09d061e90b Merge branch 'beta' into orcid_import

c5ac593c07 Merge pull request 'ORCID Enrichment and Download' (#364 ) from orcid_import into beta

Reviewed-on: #364

d33f578e54 code formatting

511a98dd80 fixed doiboost process workflow, removed references to the ProcessORCID step

3caf6ff27e Extracted the correct original type to pass to instanceTypeMapping in Crossref Mapping

178a14c491 code formatting

2a233a89aa [graph grouping] added isLookupUrl to the workflow definition, passed to the grouping spark aciton

e6086efc53 avoid NPEs in Vocabulary.getTermBySynonym

74b185d07b avoid NPEs

7c3041b276 avoid NPEs

3c3bdb8318 [bulktagging] fixed workflow parameters

c5b7253130 [community_organization propagation] fixed workflow parameters

321922772b added serialization for the new fields imported for the Irish tender

982c0c110b Merge pull request '[graph provision] added serialization for the new fields imported from the stats DB' (#365 ) from 9078_xml_records_irish_tender into beta

Reviewed-on: #365

431c6bb08a [dedup] added isLookupUrl to the graph consistency workflow definition, required now by the entity grouping phase

259c69e446 [orcid enrichment] fixed workflow definition

c0cde53bf6 [bulktagging] setting first step of bulktaggin as the copy of the entities and relations not involved in the tagging'

336fb31d87 [community_result_propagation] adjusting starting poit of workflow

c381bacee0 [enrichment] passing the community API base URL

70eb1796b2 logging typo

cb71a7936b [graph cleaning] avoid stack overflow error when navigating Oaf objects declaring an Enum

34abd0fc43 Merge branch 'beta' into clean_license_publisher

2877839df0 Merge pull request '[graph cleaning] added cleaning for result.publisher and result.instance.license' (#366 ) from clean_license_publisher into beta

Reviewed-on: #366

aba95ed1d1 code formatting

7e8eff40c1 [graph provision] added tests for the new model fields

84d54643cf [cleaning] allow enriched orcids to pass the cleaning, rule out non-orcid author pids

98cce5bfb2 code formatting

a6d635e695 Merge branch 'beta' into 9078_xml_records_irish_tender

ff924215b8 [graph provision] added tests for new peerreviewed field

a59be5779e Merge pull request '9078_xml_records_irish_tender' (#368 ) from 9078_xml_records_irish_tender into beta

Reviewed-on: #368

claudio.atzori added 1 commit 2023-12-15 11:17:27 +01:00

1726f49790 code formatting

claudio.atzori merged commit c4ec35b6cd into master

2023-12-15 11:18:31 +01:00

claudio.atzori referenced this issue from a commit

2023-12-15 11:18:31 +01:00

Merge pull request 'Master branch updates from beta December 2023' (#369) from beta_to_master_dicember2023 into master

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#369