Master branch updates from beta October 2023 #358

Manually merged
claudio.atzori merged 0 commits from beta into master 2023-11-03 12:09:45 +01:00

This PR brings to the master branch the changes available from the beta branch at October 2023.

  • #340 import of POCI from Opencitations
  • #343 Integration with Software Heritage (result level PIDs)
  • #338 Run CC and RAM sequentieally in dhp-impact-indicators WF
  • #344 [DOIBoost] implemented relation to irish funder from a Json list from Crossref
  • #346 [UnresolvedEntities] changing in the creation of the unresolved entities to integrate FOS deeper levels
  • #349 [dedup] use common saveParquet and save methods to ensure outputs are compressed
  • #351 [dedup] FIX: GroupEntitiesSparkJob deletes whole graph outputPath instead of its temporary folder
  • #339 [dedup] Fix bug in conversion from dedup json model to Spark Dataset of Rows (instanceTypeMatch no longer working)
  • #333 [dedup] SparkPropagateRelation relations do not propagate deletedByInference and invisible
  • #334 [dedup] GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob
  • #332 [stats] several changes in the indicators and monitor DBs
  • #342 [cleaning] Extending the coverage of the peer non-unknown refereed instances
  • #345 [cleaning] Fix cleaning of Pmid where parsing of numbers stopped at first not leading 0 (zero) character
This PR brings to the master branch the changes available from the beta branch at October 2023. * #340 import of POCI from Opencitations * #343 Integration with Software Heritage (result level PIDs) * #338 Run CC and RAM sequentieally in dhp-impact-indicators WF * #344 [DOIBoost] implemented relation to irish funder from a Json list from Crossref * #346 [UnresolvedEntities] changing in the creation of the unresolved entities to integrate FOS deeper levels * #349 [dedup] use common saveParquet and save methods to ensure outputs are compressed * #351 [dedup] FIX: GroupEntitiesSparkJob deletes whole graph outputPath instead of its temporary folder * #339 [dedup] Fix bug in conversion from dedup json model to Spark Dataset of Rows (instanceTypeMatch no longer working) * #333 [dedup] SparkPropagateRelation relations do not propagate deletedByInference and invisible * #334 [dedup] GroupEntities and DispatchEntites are now merged in GroupEntitiesSparkJob * #332 [stats] several changes in the indicators and monitor DBs * #342 [cleaning] Extending the coverage of the peer non-unknown refereed instances * #345 [cleaning] Fix cleaning of Pmid where parsing of numbers stopped at first not leading 0 (zero) character
claudio.atzori added 79 commits 2023-11-03 12:04:59 +01:00
163b2ee2a8 Changes
1. Monitor updates
2. Bug fixes during copy to impala cluster
964c2f553e Changes in indicators step, monitor step
- graduatedoctorates for observatory
- result_apc_affiliations table
- new indicators
	indi_is_funder_plan_s
	indi_funder_fairness
	indi_ris_fairness
	indi_funder_openess
	indi_ris_openess
	indi_funder_findable
	indi_ris_findable
	indi_is_project_result_after
- cast year to int in composite indicators
- new institutions
     -- Universidade Católica Portuguesa
     -- Iscte - Instituto Universitário de Lisboa
     -- Munster Technological University
     -- Cardiff University
     -- Leibniz Institute of Ecological Urban and Regional Development
2caaaec42d Include SparkCleanRelation logic in SparkPropagateRelation
SparkPropagateRelation includes merge relations
Revised tests for SparkPropagateRelation
9ef971a146 Update step16-createIndicatorsTables.sql
Fix int year for:
indi_org_openess_year
indi_org_fairness_year
indi_org_findable_year
489a082f04 Update step16-createIndicatorsTables.sql
Change scripts for gold, hybrid, bronze indicators
17586f0ff8 Update step20-createMonitorDB.sql
Add result_orcid table to monitor dbs
claudio.atzori manually merged commit 8c03c41d5d into master 2023-11-03 12:09:45 +01:00
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#358
No description provided.