promoting beta branch to master - November 2021 #165

claudio.atzori · 2021-11-26T15:55:18+01:00

claudio.atzori commented

2021-11-26 15:55:18 +01:00

Graph model version [dhp-schemas:2.8.22]

[Graph model]

minor: added serializable to the Measures model class
added the following relation terms Describes/IsDescribedBy, IsMetadataFor/IsMetadataOf, HasAssociationWith/HasAssociationWith, IsRequiredBy/Requires. All these are used in combination with the relation subRelType "relationship"
added constants declaring the values used for hierarchical relationships among the - organizations IsParentOf / IsChildOf

[Dump model]

added dedicated BestAccessRight class, used at the result level instead of AccessRight beta

[Graph processing]
#150 integration of "IsParentOf" and "IsChildOf" organization relations from OpenOrgs
#164 preserve parent/child relations from OpenOrgs
#151 Graph dump workflow model update and refactoring
#157 Entity resolution
#158 action set for the relations from OpenCitations

[Stats wf]
#152 affiliations in stats and monitor dbs
#154 fixed a hardcoded reference to a deleted db
#156 RIs, affiliations, parquet

[DOIBoost]
#155 Instance creation policy - removed instance from MAG

[Documentation]
#161 dnet-hadoop documentation site created using the mvn site plugin

Graph model version [dhp-schemas:2.8.22] [Graph model] - minor: added serializable to the Measures model class - added the following relation terms Describes/IsDescribedBy, IsMetadataFor/IsMetadataOf, HasAssociationWith/HasAssociationWith, IsRequiredBy/Requires. All these are used in combination with the relation subRelType "relationship" - added constants declaring the values used for hierarchical relationships among the - organizations IsParentOf / IsChildOf [Dump model] - added dedicated BestAccessRight class, used at the result level instead of AccessRight beta [Graph processing] #150 integration of "IsParentOf" and "IsChildOf" organization relations from OpenOrgs #164 preserve parent/child relations from OpenOrgs #151 Graph dump workflow model update and refactoring #157 Entity resolution #158 action set for the relations from OpenCitations [Stats wf] #152 affiliations in stats and monitor dbs #154 fixed a hardcoded reference to a deleted db #156 RIs, affiliations, parquet [DOIBoost] #155 Instance creation policy - removed instance from MAG [Documentation] #161 dnet-hadoop documentation site created using the `mvn site` plugin

claudio.atzori added 257 commits 2021-11-26 15:55:19 +01:00

e13926cdd0 merged with master

2e70aa43f0 Merge pull request 'H2020Classification fix and possibility to add datasources in blacklist for propagation of result to organization' (#108 ) from miriam.baglioni/dnet-hadoop:master into master

Reviewed-on: #108

The changes look ok, but please drop a comment to describe how the parameters should be changed from the workflow caller for both workflows
* H2020Classification
* propagation of result to organization

ea9b00ce56 adjusted test

4d6c473bf1 removed redundant classes contained now in dhp-schema

ae7bd24d79 removed old workflows

0821d8e97d Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi

1265dadc90 workflow aligned with stable_ids

d0945c3c78 added temporary output folder, because of folder access rights are different on beta and prod

abdd0ade1f added temporary output folder as workflow parameter

783988af06 depending on dhp-schemas:2.6.14 (release)

28a66af425 updated URL in the issueManagement tag

e0061232e9 [aggregation] datacite wf: conditional creation of links, optional resume from intermediate phases

f5486ffb14 Fixed issues to tests

2f66fedfec changed the logic to verify if a community is contained in the list of context of a result

da88c850c6 changed the logic to verify if a community is contained in the list of context of a result

86e50f7311 modified code to split the Croatian funder

69fd40fd30 modified code to split the Croatian funder

87a6e2b967 extended test class

4b432fbee8 extended test class

65a242646d added resource for APC dump

6410ab71d8 added APC in the dump and test method

13c66e16be changed logic to split for communities

9a58f1b93d added logic to select only the valid relations: those not deletedbyinference and having both part of the relation as entities in the graph

39b1a6edf6 added test class for the selection of valid relations and description

8429aed6c6 Added resource for testing selection of valid relations

e9a17ec899 added check to verify not to add void APC

f13e11e3f7 [aggregation] datacite wf: defined parameter declaring the path used to store the OAF objects produced by the transformation phase

9629569e22 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

5295d10691 added check not to dump deletedByInference entities

72397ea1ba Added fix for community of arbitrary name length

8f322a73cb change because of the renaming of originalId in acronym

fa720c1da4 [doiboost] added workflow for the ActionSet update dedicated to production

084b4ef999 added the creation of the openaireId from funder and grant number if the element is not present in the context profile

59615da65e Add test to verify the creation of relation between context and projects

618d2de2da minor changes and refactoring

d418c309f5 removed the part after part-x- in the file name generated by spark. It was too long and created problems while creating the tar entries

734de62474 [doiboost] added workflow for the ActionSet update dedicated to production

5e38c7f42d dumping only communities with status all

d70f8c96fd funding contains and not starts with h2020

c028feef4f workflow for the dump as sub workflows

eae10c5894 modification to allow the dump for a single community

970b387b8d modification to allow dump of a single community

52ce35d57b -

320cf02d96 Changed the way to find results linked to projects. We verify to actually have the project on the graph before selecting the result

886617afd0 One result linked to more than on project is saved just once

774cdb190e changes to mirror the last dump of the graph with the ols data model.

4cb65bc64a fixed process doiboost workflow:

- splitted OrcidToOAF into two phase preprocess and process
- updated workflow used in production

1cdd09cd8e Tentative fix for testing of Jenkins

10068c00ea Code refactor:

- removed old workflows in doiboost
 - splitted workflow of doiboost in preprocess and process

a65667d217 added publication to dataset even if no contributors

7840cc6526 merged with master

66604bb2b4 added absolute path to process folder

2dc50c0999 added default value to process path

9913b6073c Merge pull request 'orcid-no-doi' (#123 ) from enrico.ottonello/dnet-hadoop:orcid-no-doi into master

Reviewed-on: #123

5e5f65a3c3 contents mapped from the stores with 'claim' interpretation will not change their identifier along their way towards the graph

13cf444f85 Merge pull request 'force orginalId for claimed records' (#124 ) from forceOrginalId_claims into master

Reviewed-on: #124

5947cddafc adding record identifier among the originalIds regardless of what IdentifierFactory produces

77e8c6c7f7 filtering 'old' OpenAIRE ids from the entity.originalId[] array in the OAF -> XML searialization procedure

9069958479 tests for enermaps

ffdb2a3ea3 [cleaning] fixed filtering function for missing titles

5b6844b969 mapping funding relations from Datacite should be done according to the actual result identifier

35e395eae8 merge with master

998b66855a updated assertions in eu.dnetlib.dhp.oa.graph.raw.MappersTest

d267dce520 [raw_all] added extra workflow step for patching the identifiers in the relations, given an id mapping dataset

9594343725 code formatting after mvn compile

a9961a1835 [cleaning] title cleaning based on the me.xuender:unidecode library

1923c1ce21 replaced full join + filtering with a left join

5f7330d407 Merge branch 'master' into fct_project_id_replacement

f83dd70e1c Merge pull request 'Patch the identifiers (source/target) in the relations' (#125 ) from fct_project_id_replacement into master

Reviewed-on: #125

e725c88ebb [raw_all] patching relation identifier phase to be run at the end, i.e. includes also claimed relations

6e3554a45e [provision] lowercase relation filter

576693d782 added unit test for PatchRelationsApplication

081fe92a21 Merge branch 'fct_project_id_replacement' of https://code-repo.d4science.org/D-Net/dnet-hadoop into fct_project_id_replacement

2fc89fc9b5 Merge branch 'fct_project_id_replacement' of https://code-repo.d4science.org/D-Net/dnet-hadoop into fct_project_id_replacement

9bc4fd3b69 Patch FCT relations - fixed issue with join

a6a38cca9e fixed implementation of PatchRelationsApplication, refined the relative unit test

4f78565c04 fixed implementation of PatchRelationsApplication, refined the relative unit test

5219d56be5 Merge pull request 'Patch the identifiers (source/target) in the relations, refinements' (#131 ) from fct_project_id_replacement into master

Reviewed-on: #131

11e26c020a Update 'README.md'

74afe43c3a fixed wrong test file

ee13da9258 merge branch with master

73dc082927 added new dumped field (openaccessroute, pid and alternate identifier at the level of the instance) and the bipFinder measure at the level of the result

6bd1eca7e0 merge branch with beta

c3931557e3 extended the logic of the dump to consider the validation date in the relation (also in the dumped result for communities and funders at the level of the project), the extention on the instance for the APC, the pid, the alternate identifiers, and the extention of the AccessRight to store the OpenAccessRoute. Added new resourec for testing and extended the old class to verify the new dump. Fixed also issue on relation dump: only relation whose source and target are entities in the graph are dumped. The same hold for references to projects

5d70f842eb mergin with branch beta

eff499af9f added new tests and changed the test example

bc9e3a06ba Graph Dump - extended the test class

8daaa32e90 Graph Dump - added resources for testing

bd0d7bfba7 Graph Dump - added resources for testing addition of validation date in project for communityresult

a90bac3bc9 Graph Dump - added method to test class to verify addition of validation date in projects for community result

f2118d771a first steps in the implementation of the integration of opencitations

eedf7c3310 mergin with branch beta

5ec69889db OpenCitations: creation of AS from OC

476a4708d6 mergin with branch beta

c8321ad31a merge with branch beta

c4ccd7b32c -

9814c3e700 mergin with branch beta

e653756e3d applied some suggestiond from Sonar Lint

8f99d2af86 Make the node of doiBoost to point to the correct OpenAire Organization in relations

de81007302 Add exploreTestConfig, a new Solr configuration folder

e468a7b96b Add tests to query Solr with different configurations

201ce71cc1 Add resultsubject, relprojectname and resultacceptanceyear to __all field

5606014b17 code refactor see ticket #7065

83f51f1812 refactoring

fec40bdd95 merging with branch beta - resolved conflicts

511da98d0c - fixed bug on download pmc Article

- removed unused line of code in SparkCreateActionset

914b3e92cb updating graph schema module dependency to version 2.8.20 to include organization parent/child relation constants

7387416e90 added params skip update to direct transform in OAF, this should be set to true in production

f2c8356ccf Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

63933808d4 added fix for mixing result types, added configuration default to funder subworkflow

16b28494a9 added new parameter in the doiboost process workflow to specify a folder for the process of MAG dataset

326bf63775 integration of parent child orgs relations

5d9cc2452d changed the working path parameter value as dependant from the dnet-workflow working dir parameter

995c1eddaf minor change

b292e4a700 [stats wf] added extra logging in the context data retrieval phase

14fbf92ad6 Merge branch 'beta' into beta_solr_config

dd568ec88b Merge pull request 'Refactoring Solr Configuration' (#148 ) from beta_solr_config into beta

Reviewed-on: #148

51a03c0a50 refactor code for EBI from dhp-graph-mapper into dhp-aggregation

4b7c8dff2d fetching affiliated results for 4 orgs in monitor. fixed affiliated orgs in stats db

41ecb1eb61 invalidating medatadata before context thingies

7b15b88d4c renamed wrong package, implemented last aggregation workflow for scholexplorer

46f82c7c8f removed not needed folder deletion

76d41602be Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

1cc09adfaa Opencitations: chenaged the test class to mirror the creation or not of duplicate dois for .refs oc original plus added optional parameter to duplicate the relation

1315952702 merge with branch beta

5f780a6ba1 bug fix in migrate entities: parameter name was wrong

a894d7adf3 updated version of dhp-schemas

c7f6cd2591 added again the setting for saXReader

7a73010acd WIP: worflow nodes for including Scholexplorer records in the RAW graph

f8329bc110 since dhp-schemas changed, introducing new Relation inverse model, this class has been updated

c9870c5122 code formatted

c8850456e9 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

98f37c8d81 WIP: worflow nodes for including Scholexplorer records in the RAW graph

e9157c67aa Merge branch 'beta' into dump

d517c71458 Merge branch 'dump' into beta

512e7b0170 code formatting

515e068a78 Merge branch 'beta' into hierarchical_orgs_relations

241dcf6df1 Merge branch 'beta' into beta

bc3372093e Merge pull request '[stats] affiliations in stats and monitor dbs' (#152 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #152

59f76b50d4 Merge branch 'beta' into hierarchical_orgs_relations

d0cf2963f0 Merge pull request 'hierarchical_orgs_relations' (#150 ) from hierarchical_orgs_relations into beta

Reviewed-on: #150

c01dd0c925 registered oaf model classes for the KryoSerializer

00b78b9c58 cleanup: mapping contents in the graph already defined in the OAF graph model doesn't require to be aware of the vocabularies

4f8970f8ed [stats] reducing the step22 wait time

ae4e99a471 Adapted workflow of resolution of PID to work into OpenAIRE data workflow

- Added relations in both verse on all Scholexplorer datasources

ab3a99d3e9 removed old datacite oozie workflow

aeeebd573b code refactor renamed datacite package

ac36aa7d1c fixed wrong Encoding during a map phase

3702fe478d cleanup

d147295c2f avoiding java.io.NotSerializableException: java.util.HashMap

6b34ba737e minor

034304b33a conflict resolved on merge

aafdffa6b3 resolved conflict

4acfa8fa2e Scholexplorer Datasource Aggregation:

- Added collectedfrom in the inverse relation generated
Relation resolution:
- increased number of partitions in workflow.xml
- using classid instead of classname to build the pid-dnetId mapping

1be9aa0a5f Removed filter of datacite items from the raw graph merging phase, Datacite is not an actionset anymore in beta

d9cbca83f7 moved filter on next phase

1225ba0b92 [resolution] increasing number of partitions to avoid OOM

f78afb5ef9 removed hardcoded reference

7fa49f6956 Merge pull request 'removed hardcoded reference' (#154 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #154

7bd224f051 implement first version of scholexplorer integration for the generation of final graph

2480e590d1 [DOIBoost - Mapping] changed the type on which to map dissertation from Crossref: from 006 Doctoral thesis to 0044 Thesis since dissertation could be either Doctoral or master thesis

779318961c [DOIBoost - Mapping] removed the url from crossref containing the api.elsevier.com... string in the url

b2bb8d9d79 [DOIBoost - Mapping] selecting the url from Crossref containing the doi

683fe093cf [DOIBoost - Mapping] Remove the addition of the instance to the MAG publication record

96769b4481 [DOIBoost - Mapping] Changed the logic which brought in in the instance urls that should not be there: The urld of the doi in the json is reachable from the root (json/"URL") other urls where added from the links element. Now the mapping from the link element has been removed

d97ea82a29 [DOIBoost Mapping] Added test to verify the instance created for Crossref will have just the url related to the doi

edf55395e9 added test resourse

d50057b2d9 [DOIBoost Mapping] changed the way to create the url for the instance: we use the crooref guidelines https://doi.org/doi

de63d29b6f [DOIBoost Mapping] Fix to avoid to produce results with null as identifier (probably due to the filtering function in the factory for the creation of the id)

df7ee77c7a [DOIBoost Mapping] removed not needed comments

91354c6068 - fetching all context related results

- storing tables as parquet

a7d50c499b [BypassAS] prepare FOS subject, test and model for FOS and BipFinder scores

6f7ca539c6 [BypassAS] update of results for bipFinder and FOS

6477a40670 implement filter of openCitation

9e214ce0eb [BypassAS] addition of OC relations

c371b23077 -

6d3c4c4abe mergin with branch beta

9cb195314f implemented and tested resolution of entities

2ca0a436ad added SparkResolveEntities node to the oozie wf

148289150f Merge branch 'beta' into doiboost_url

8bdca3413f Merge pull request 'DOIBoost Mapping: change the creation of the instance in the DOIBoost result' (#155 ) from doiboost_url into beta

Reviewed-on: #155

26f086dd64 removed the too restrctive clause. will discuss again

935062edec [Bypass Action Set] creation of unresolved entities

a7763d2492 removed alternate identifier in resolutionMap

3469cc2b1d Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

1f2a3d1af0 depending on dhp-schemas:2.8.22 (release)

716021546e [Bypass Action Set] minor fix

ffb0ce1d59 merge with beta - resolved conflict in pom

47ccb53c4f [Bypass Action Set] modification for comment #157 (comment)

881113743f [Bypass Action Set] refactoring

92d0e18b55 [Bypass Action Set] used constant DOI instead of "doi"

7b81607035 Merge pull request 'PR: Bypass Action Set' (#157 ) from bypass_acstionset into beta

Reviewed-on: #157

157d33ebf9 [Bypass Action Set] Refactoring

6f1a434e90 [Bypass Action Set] Fixed test to consider the new identifier utils

4ec88c718c merge with beta - resolved conflict in pom

1ecceea788 Merge pull request 'Open Citations' (#158 ) from openCitations into beta

Reviewed-on: #158

975b10b711 [actionmanager] increased spark.sql.shuffle.partitions to 5000

d2c787d416 [graph resolution] fixed sequence of the workflow steps

48923e46a1 added documentation to Pubmed Class and also added mvn site for dhp-aggregations

efa09057db Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

7c804acda8 [graph resolution] minor

941a50a2fc Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

7d0a03f607 [graph resolution] minor

668ac25224 [graph resolution] using existing argument parser file name

bafa2990f3 code formatting

0a727d325d [dedup] increased number of partitions in the consistency phase

a1cafaf2e3 added mvn site for dnet-hadoop project

18c1d70ef4 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into mvn_site_documentation

49f897ef29 [cleaning wf] fixed regex used to spot garbage in result titles; adjusted threshold for filtering titles

2d67020c59 added dhp-enrichment maven site template

ec8b0219ff [Documentation] Added first page for Integration via unresolved entities generation

4094f2bb9a added integration md file

cded363b55 code refactor, created and moved scala code on the correct maven folder under src/main/scala and src/test/scala

2506d7a679 Merge branch 'mvn_site_documentation' of code-repo.d4science.org:D-Net/dnet-hadoop into mvn_site_documentation

60ae874dcb Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into mvn_site_documentation

2fd9ceac13 code refactor, created and moved scala code on the correct maven folder under src/main/scala and src/test/scala

1f5ee116ed code refactor, created and moved scala code on the correct maven folder under src/main/scala and src/test/scala

fixed test

9c82d670b8 make class public in order to create javadoc

6d4a1c57ee [Resolve Entities] Change test dataset to mirror the modification in the creation of the map between the pids and the unresolved

82a4e4efae [cleaning wf] fixed methodology to rule out invalid result titles, based on https://support.openaire.eu/issues/7206

e0395719d7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

c283406829 added Universidad Polytecnica de Madrid

cb3adb90f4 Merge branch 'beta' into beta

10a32f287f Merge pull request '[stats wf] RIs, affiliations, parquet' (#156 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #156

793b5a8e5f Aggiornare 'dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/dump/ResultMapper.java'

Removing the dump of Measure at the level of the result. We decided not to map it

bb5dca7979 cleanup

c0750fb17c avoid non necessary count operations over large spark datasets

a24b9f8268 [dedup] trivial refactoring

bd9a43cefd Revert to 4094f2bb9a

62fa61f3cf merge from beta

0c0d561bc4 added public class into tests to create correct javadoc

8a7c7d36db Merge pull request 'mvn_site_documentation' (#161 ) from mvn_site_documentation into beta

Reviewed-on: #161

fc03c99805 fixed javadocs url after deploying site

2b46b87f56 fixed filtering criteria applied in SparkCopyRelationsNoOpenorgs to keep the parent/child relations from OpenOrgs

f4538f3c4c cleanup

e5a2c596b2 Merge branch 'beta' into preserve_openorg_parent_child_relations

90c2a4987e Merge pull request '[fix] preserve parent/child relations from OpenOrgs' (#164 ) from preserve_openorg_parent_child_relations into beta

Reviewed-on: #164

4542a2338b updated site configuration to deploy on website

75298ec442 added site.xml to code style

3426451d3f Merge remote-tracking branch 'origin/beta' into beta

155d8bf83f updated maven site plugin on dhp-code-style

65ebe1019b updated wagon-ssh version

6110a2b984 reverted version

fdb75b180e [Cleaning] added couple of tests for DOIBOOST publications

35e20b0647 updated resolution wf:

- generate a new version of the graph
 - changed merge from union to join

93fe8ce8b2 entity resolution: fix test

483d3039d1 entity resolution: added distcpt of missing entities in graph materialization

a7cf277d98 Datacite: Removed HostedBy Patch as described on ticket #7219 , Now all the records will have hosted by Unknown Repository

2164a2a889 Datacite: Code Refactor generated a general SparkApplication Scala where all the spark scala have to inherit

Commented a little the Datacite transformation code

028a8acad8 add test resources

feea154e89 remove working dir after test

5fd0e610bf [DOIBOOST Process] fix filtering to filter results with non null id

ac07ed8251 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

1e1f5e4fe0 minor fix

bb7f556eff Merge remote-tracking branch 'origin/beta' into beta

5c6d328537 code formatting

014e872ae1 [resolution wf] added optional parameter to skip the entity resolution

claudio.atzori referenced this issue from a commit

2021-11-26 16:15:27 +01:00

resolved conflicts for #165

claudio.atzori manually merged commit 1de881b796 into master

2021-11-26 16:17:09 +01:00

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#165