promoting beta branch to master - March 2022 #197

claudio.atzori · 2022-02-24T12:47:37+01:00

claudio.atzori commented

2022-02-24 12:47:37 +01:00

[Graph model `dhp-schemas:2.10.29`]

merge logics changed to consider invisble in dataInfo
Added APC information at the level of the result
change name and id of OpenAPC datasource
OpenAIRE ids of the delegated authorities and enrichment providers in the identifier creation strategy
implemented enrichment on Result specializing in the method mergefrom
added utility method and constants for checking weather is an OafEntity represents an enrichment
Added Instance.measures field, allowing to maintain the association between them and the individual result instance

[Graph dump]

#176 - fixed issue on extraction of relation between entities and contexts: the relationship name and type were swapped
change the names of the classes to be able to automatically create the json schema with specific descriptions
added json schemas

[Graph processing]

#168 - Cleaning of invisible records
#170 - Graph cleaning consider terms as synonyms in the vocabulary lookup
#172 - Graph cleaning improved instance type defaults
#173 - Citation scores from Bip!Finder, action set for the result level scores
#174 - Citation scores from Bip!Finder, unresolved entities for the integration at the instance level
#185 - Updated result merging procedure to integration instance level Bip!Finder scores
#175 - FOS as unresolved entity
#178 - SDG as resolved entity
#181, #182 - fixed issues for the SDG & FOS integration
#180 - tolerant parsing of OAF-store-graph mdstores
#183 - Affiliation propagation through semantic relation, further additions to the workflow
#187 - Priority to records from delegated authorities
#188 - Raw graph mapping: grab the title types from XML records in the Datacite format (ODF)
#192 - APC at the level of the result
#194 - Various fixes on the OpenCitations data
#169 - Refactoring of scala classes
#184 - ScalaFmt, code style for scala classes
3aa2020b24 - added script to regenerate hostedBy Map following instruction defined on ticket #7539

[Graph provision]

#167 - XML serialisation of instances with the same URLs
#171 - Serialization of fields in XML records for Sygma
#177 - Normalising DOI urls in the XML record serialization
#196 - Serialize measures defined on the result level in the XML index records

#### [Graph model `dhp-schemas:2.10.29`] * merge logics changed to consider invisble in dataInfo * Added APC information at the level of the result * change name and id of OpenAPC datasource * OpenAIRE ids of the delegated authorities and enrichment providers in the identifier creation strategy * implemented enrichment on Result specializing in the method mergefrom * added utility method and constants for checking weather is an OafEntity represents an enrichment * Added Instance.measures field, allowing to maintain the association between them and the individual result instance #### [Graph dump] * #176 - fixed issue on extraction of relation between entities and contexts: the relationship name and type were swapped * change the names of the classes to be able to automatically create the json schema with specific descriptions * added json schemas #### [Graph processing] * #168 - Cleaning of invisible records * #170 - Graph cleaning consider terms as synonyms in the vocabulary lookup * #172 - Graph cleaning improved instance type defaults * #173 - Citation scores from Bip!Finder, action set for the result level scores * #174 - Citation scores from Bip!Finder, unresolved entities for the integration at the instance level * #185 - Updated result merging procedure to integration instance level Bip!Finder scores * #175 - FOS as unresolved entity * #178 - SDG as resolved entity * #181, #182 - fixed issues for the SDG & FOS integration * #180 - tolerant parsing of OAF-store-graph mdstores * #183 - Affiliation propagation through semantic relation, further additions to the workflow * #187 - Priority to records from delegated authorities * #188 - Raw graph mapping: grab the title types from XML records in the Datacite format (ODF) * #192 - APC at the level of the result * #194 - Various fixes on the OpenCitations data * #169 - Refactoring of scala classes * #184 - ScalaFmt, code style for scala classes * 3aa2020b24 - added script to regenerate hostedBy Map following instruction defined on ticket [#7539](https://support.openaire.eu/issues/7539) #### [Graph provision] * #167 - XML serialisation of instances with the same URLs * #171 - Serialization of fields in XML records for Sygma * #177 - Normalising DOI urls in the XML record serialization * #196 - Serialize measures defined on the result level in the XML index records

claudio.atzori added 223 commits 2022-02-24 12:47:37 +01:00

652114c641 [affiliationPropagation] first try. preparetion

d66e20e7ac added hierarchy rel in ROR actionset

d0ef7d91c5 adding test resource

d02caef185 Merge branch 'beta' into hierarchical_orgs_relations

09f36cffb8 [Enrichment: Propagation through parent-child relationships] First implementation, testing, and wf for propagation of result to organization through semantic relation

2aca6bfa0a mergin with branch beta

b9d124bb7c [Enrichment: Propagation through parent-child relationships] Added counters, and changed constraint to verify if filtering out the relation (from classname = harvested to classid != propagation)

0857849a86 [Graph DUMP] Remove dump of measure until it will be clear where to put it (at the level of result or at the level of the instance)

c10ff6928c [Graph DUMP] add schema of the dump related to the model as in dhp-schemas.2.8.31. Note the measere element at the level of the result has been removed because of issues on where to display it: at the level of the result or at the level of the entity

a22c29fba1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

5681e89544 Update 'dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/dump/schemas/result_schema.json'

8442efd8d1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

4c70201412 mergin with branch beta

9cb8e4ad21 Merge branch 'beta' into hierarchical_orgs_relations

94918a673c [Graph DUMP] Fix issue for empty origilaId list

88b73f4f49 mergin with branch beta

8cc50ecee0 [Graph Dump] changed AccessRight with BestAccessRight in the dump and modified the dependency to the schema to the SNAPSHOT

e5498052e8 [Graph DUmp] removed OpenAccessRoute from test in best access right

b8bdabfae9 [Graph DUmp] removed OpenAccessRoute from test in best access right

b3f9370125 merge with beta - resolved conflict in pom

43cae4ad88 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

6595135a1a [Dump Schemas] changed the schema of the dumped result according to the modifications in the bestAccessRight type

99d86134f5 [Graph Dump] changed the dump since the measures have been moded at the level of the instance

c6a9f0a1a8 mergin with branch beta

c7c0c3187b [AFFILIATION PROPAGATION] Applied some SonarLint suggestions

7c96e3fd46 removed not useful dir

28ea532ece [Affilaition Propagation] moved the selection of graph relation as a preparation step

2bbece2ca5 mergin with branch beta

1b79c0ee79 mergin with branch beta

0136a8c266 [Graph Dump] Change test to mirror that measure is at the level of the isntance

5dc5792722 [Graph Dump] Change test resource to mirror the movement of the measure element

3974fa7dc1 Merge branch 'beta' into affiliationPropagation

3a4d925386 Merge branch 'beta' into hierarchical_orgs_relations

9fae872181 [Graph Dump] changed to mirror the changes in the model

0506fa2654 [Graph Dump] changed to mirror the changes in the model

01e5e0142a added test to verify the relation inverse lookup operation

4fe7888817 code formatting

d85af6fc25 [cleaning wf] fixed OAF record navigation, a mapping defined on a container object would have prevented the natvigation to continue on its properties

cfa4560769 minor: fixed hive action name

3b19821f3c added stats computation on the graph hive DB tables

87eedad898 -

8905a39bf3 mergin with branch beta

58bc3f223a [GRAPH DUMP] Add filtering for relation we do not want to dump. It is based on the relclass

d9f80488cc [GRAPH DUMP] Add one more test to check the filtering of the relations

9cac283bec implemented Instance serialization features requested in https://support.openaire.eu/issues/7156

863a2f9db3 avoid to filter OAF records defined as invisible = true

c4c705aa46 Merge pull request 'Cleaning of invisible records' (#168 ) from clean_invisible_records into beta

Reviewed-on: #168

dd0b2e5244 Merge branch 'beta' into instance_group_by_url

372633880f Merge pull request 'XML serialisation of instances with the same URLs' (#167 ) from instance_group_by_url into beta

Reviewed-on: #167

f7011b90d8 format code

0fa0ce33d6 removed duplicated on gitignore

4bb1d43afc -

f430688ff7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

08795cbd30 using helper method from ModelSupport to find the inverse relation descriptor

9132727793 fixed date cleaning test

7af0bbd0b1 [scala-refactor] Module dhp-aggregation:

Moved all scala source into src/main/scala and src/test/scala

81bf604059 [scala-refactor] Module dhp-common:

Moved all scala source into src/main/scala and src/test/scala

bf880e2508 [scala-refactor] Module dhp-graph-mapper:

Moved all scala source into src/main/scala and src/test/scala

e9f285ec4d [scala-refactor] Module dhp-doiboost:

Moved all scala source into src/main/scala and src/test/scala

96a7d46278 [Graph Dump] fixed tests

ed0c352799 [test-fixing] fixed wrong test

d1df01ff1e [Graph Dump] fixed resource for test

d9836f0cf3 [OpenCitations] fixed test when executed one after the other

5d51b3dd4a Merge pull request 'scala_refactor' (#169 ) from scala_refactor into beta

Reviewed-on: #169

b113586207 resolved conflicts

6b5d7688a4 #7275 serialize license information in XML records

e6e177dda0 vocabulary based cleaning considers also the term label when looking up for a synonym

adf17452b0 Merge pull request '[graph cleaning] consider terms as synonyms in the vocabulary lookup' (#170 ) from graph_cleaning into beta

Reviewed-on: #170

e53228401b style

cba63e9f82 Merge branch 'beta' into sygma_indexing

8f1e018ceb Merge pull request 'Serialization of fields in XML records for Sygma (and not only)' (#171 ) from sygma_indexing into beta

Reviewed-on: #171

41c70c607d cleaning workflow assigns the proper default instance type when a value could not be cleaned using the vocabularies

eb43eda42a Merge branch 'beta' into graph_cleaning

25dc7929a9 Merge pull request '[graph cleaning] improved instance type defaults' (#172 ) from graph_cleaning into beta

Reviewed-on: #172

b70ecccea0 avoid NPEs merging XMLInstance(s)

5e17247bb6 avoid NPEs merging XMLInstance(s)

98eb292c59 avoid NPEs merging XMLInstance(s)

8d755cca80 -

936578aaf1 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

4eb8276493 -

573bd17cbb Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

aff3ddc8d2 added cleaning for the format field, removing carrige return and tab characters

6fb6236cd4 changed the way to produce the AS for bipFinder.

22d4b5619b [BipFinder Result] last changes to test and resources files

a3592b463a Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

56409d1281 [Dump] resolved conflicts with beta and merging

1790fa2d44 Merge branch 'beta' into affiliationPropagation

e30e5ac8a8 Merge pull request '[Affiliation Propagation]' (#162 ) from affiliationPropagation into beta

Reviewed-on: #162

e5bff64f2e [scholexplorer]

- Minor fix on SparkConvertRDDtoDataset
-first implementation of retrieve datacite dump

63952018c0 [scholexplorer]

-moved SparkRetrieveDataciteDelta in scala folder

b881ee5ef8 [scholexplorer]

- implemented generation of scholix of delta update of datacite

63b648b0dd Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

5e5dfd619c Merge branch 'beta' into dump

3cc1b7b153 mergin with branch beta

c1cdec09a9 Sprint 5 and other changes

58996972d9 added first indicator of sprint 5

3920d68992 Fixed workflow generation of delta in datacite

460e6b95d6 [Graph Dump] -

d1ae219cb4 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

e24a7f3496 mergin with branch beta

a4c0cbb98c fixed typos in indicators. Added extra views in monitor

3edd661608 fixed column names

16539d7360 added usage stats

89ea9fa0e1 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

be0acccf42 Merge branch 'beta' into dump

9d19b057b8 Merge pull request '[GRAPH DUMP]Moving Measures' (#159 ) from dump into beta

Reviewed-on: #159

8d18500069 using dhp-schema:2.9.24

4849270c55 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

da5f6260aa mergin with branch beta

c5739c4266 [BipFinder] create action set for the measures at the level of the result

b5e11a3a0a [BipFinder] put in common package BipFinder model

0807fdb65a [BipFinder] remove not needed resources

bf52a1847b Merge pull request 'bipFinder at the level of the result' (#173 ) from bipFinder into beta

Reviewed-on: #173

2c126ed014 [BipFinder] create unresolved entities with measures at the level of the instance

813f856d3f [BipFinder] removing left over parameter in wf

20ef1d657f refactoring

34ac56565d refactoring

b352fbe453 Merge pull request 'bipFinder: unresolved entities' (#174 ) from bipFinder into beta

Reviewed-on: #174

de6c4c8968 [FOS]creation of the unresolved entities: remove the split for the doi: no more needed since each row is related to one doi

73175ba086 mergin with branch beta

b81efb6a9d [FOS]changed the mapping between the csv and the model. Changed Test classes and resources

6116fc5d40 [FOS]added logic to include only different subjects. Test refactoring and extention

10579c0dd0 [FOS]fixed doi value in test

bf3a9505e0 Merge pull request 'FOS' (#175 ) from FOS into beta

Reviewed-on: #175

31b26d48ac [Graph Dump] fixed issue on extraction of relation between entities and contexts: the relationship name and type were swapped

69e9ea9eeb [Graph Dump] Test for extraction of rels from entities extended

5c4fee3533 Merge pull request '[Graph Dump] fixed issue on extraction of relation between entities and contexts: the relationship name and type were swapped' (#176 ) from dump into beta

Reviewed-on: #176

2a67ee13ec [SDG] added model class

cccb16900c https://support.openaire.eu/issues/7330 normalising DOI urls

278cf08421 Merge pull request 'Normalising DOI urls' (#177 ) from instance_group_by_url into beta

Reviewed-on: #177

7a1b440413 [SDG] logic to create unresolved entities out of SDG input. This changes also some classes related to FOS to reuse the same code. The code under createunresolvedentities create results with the merged update of the the inputs provided (bip at the level of the isntance, fos and sdg for subjects)

7551e52e95 fixed a typo

81ee654271 added result_result relations

a706ba0c08 Merge pull request 'SDG Integration' (#178 ) from SDG into beta

Reviewed-on: #178

bcea4e3a9b added dnet workflow profile for the orchestration of the simplified and complete graph construction and processing pipeline, where the IIS works on the non-deduplicated graph

fe7e5f4748 Merge pull request '[stats wf] result_result relations, usage stats, monitor views, indicator for sprint 5' (#179 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #179

92fd69e25d [SDG-FOS] alternative way to get input data to avoid OOM error while getting csv

4c60ee1718 mergin with branch beta

a6977197b3 serialise records in the OAF-store-graph mdstores in json format. Read them again in the graph construction phase using a tolerant parser to support backward compatible changes in the evolution of the schema

bd59b58efb test for the tolerant deserialisation utility method

cb30770a0b Merge pull request 'tolerant parsing of OAF-store-graph mdstores' (#180 ) from graph_interpretation_mdstores into beta

Reviewed-on: #180

045d767013 OAF-store-graph mdstores: save them in text format

174c3037e1 OAF-store-graph mdstores: save them in text format

58f8998e3d OAF-store-graph mdstores: save them in text format

908294d86e OAF-store-graph mdstores: firther fix for PR#180

adccc2346a [SDG-FOS] to lower case for the doi

639190370a mergin with branch beta

af8a33638d Merge pull request 'SDG - FOS' (#181 ) from SDG into beta

Reviewed-on: #181

b7e450070b [SDG-FOS] to import SDG file not considering the header

93f26fb742 Merge pull request '[SDG-FOS] to import SDG file not considering the header' (#182 ) from SDG into beta

Reviewed-on: #182

064f9bbd87 [AFFPropSR] added new paprameter for the number of iterations and new code for just one iteration

904e1c2667 Merge pull request 'Affiliation Propagation through semantic relation' (#183 ) from enrichment into beta

Reviewed-on: #183

0163dadb7f [doiboost]

- update MAG schema, new filed added on version dec-2021

4f212652ca scalafmt: code formatting

dcd282977c pulled from beta

0f2144b5e0 scalafmt: code formatting

b78d2b71f0 updated scala format configuration

57e2c4b749 formatted code

1b9e8378b3 Merge pull request 'scalafmt: code style for scala' (#184 ) from scalafmt into beta

Reviewed-on: #184

4993666d73 [BipFinderInstanceLevel] changed creation of the instance to allow to enrich existing instances with same pid

dbd6fa1d65 scalafmt: remote referencing the common definition files makes it work compiling the entire project as well as the individual submodules

e7d5a39c03 [BipFinderInstanceLevel] added tests in test class

4d517ed9ec mergin with branch beta

a75fb8c47a [BipFinderInstanceLevel] change pom to align to the dhp-schema release 2.10.24 and refactoring

7bf12ad24a Merge pull request 'BipInstance' (#185 ) from BipInstance into beta

Reviewed-on: #185

a7c4d0d16d [DoiBoost Organizations] added parameter to specify the action in the wf raw_organizations to be able to load the openorgs organization as in the loading step for the construction of the graph

42e8f76778 [GraphCleaning] change the return value in the filtering function to avoid to lose the APC entities

44a937f4ed factored out entity grouping implementation, extended to consider results from delegated authorities rather than identical records from other sources

62f135262e code formatting

391aa1373b added unit test

abfa9c6045 code formatting

3b9020c1b7 added unit test for the DispatchEntitiesJob

b37bc277c4 reintroduced the hostedby patching to the datacite records

f0ea2410e5 improved mapping titles from datacite records to consider title types

7a3819144d Merge pull request 'title types from datacite records' (#188 ) from datacite_title_mapping into beta

Reviewed-on: #188

4983d6536d Merge branch 'beta' into delegated_authorities

dd52bf1bb8 copy relations to the graphOutputPath

2f385b3ac6 updated dnet workflow profile definitions

c42623f006 added NPE checks

97ad94d7d9 [graph resolution] drop output path at the beginning

1322379741 Merge branch 'beta' into delegated_authorities

a70b0990c9 Merge pull request 'Priority to records from delegated authorities' (#187 ) from delegated_authorities into beta

Reviewed-on: #187

4fc44edb71 depending on dhp-schemas:2.10.26

af61e44acc ported changes to the GraphCleaningFunctionsTest from 8de9788308

db299dd8ab fixed typo

8eb75ca169 adapted GenerateEntitiesApplicationTest behaviour

dce7f5fea8 [BULK TAGGING] changed to fix issue that should have been fixed already

73eba34d42 [UnresolvedEntities] Changed the way to merge the unresolved because the new merge removed the dataInfo from the merged result. Added also data info for subjects

37784209c9 [dhp-schemas-] updated the version of dhp-schema to 2.10.27 for APC name and id modification

2e215abfa8 test for instances with URLs for OpenAPC

dbd2646d59 fixed the result_result creation for monitor

c2b44530a3 typo...

89922156c9 Merge pull request '[stats-wf]fixed the result_result creation for monitor' (#190 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #190

aae667e6b6 [APC at the result level] added the APC at the level of the result and modified test class

ae633c566b fixed the result_result table

0547fd6ee7 Merge pull request '[stats-wf]fixed the result_result table' (#191 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #191

9fd2ef468e [APC at the result level] changed dependecy in external pom

ac8b8f224f Merge branch 'beta' into extendResult

b04ecfcf13 Merge pull request 'extendResult' (#192 ) from extendResult into beta

Reviewed-on: #192

5f762cbd09 fixed yet another typo

1f8302dc37 Merge pull request '[stats-wf]fixed yet another typo' (#193 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #193

fbc28ee8c3 [OpenCitation] change the integration logic to consider dois with commas inside

b071f8e415 [OpenCitation] change to extract in json format each folder just onece

759ed519f2 [OpenCitation] added logic to avoid the genration of self citations relations

5c4043dba8 [OpenCitation] refactoring

600ede1798 serialisation of APCs int he XML records

6158170334 testing delegated authority and bumped dep to schemas

c191080965 mergin with branch beta

1490867cc7 [OpenCitation] cleaning of the COCI model

a1013e62d4 Merge pull request 'openCitations' (#194 ) from openCitations into beta

Reviewed-on: #194

be64055cfe [OpenCitation] changed the name of destination folders

90e197a563 Merge pull request '[OpenCitation] changed the name of destination folders' (#195 ) from openCitations into beta

Reviewed-on: #195

3aa2020b24 added script to regenerate hostedBy Map following instruction defined on ticket #7539

updated hosted By Map

89c7313fc5 Merge branch 'beta' into hierarchical_orgs_relations

56e048ef21 Merge pull request 'added hierarchy rel in ROR actionset' (#153 ) from hierarchical_orgs_relations into beta

Reviewed-on: #153

d3f03abd51 fixed wrong json path

891781ee3f Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

cf8443780e added processingchargeamount to the result view

401dd38074 code formatting

99f5b14469 [graph raw] invisible records stored among the raw graph rather than the claimed subgraph

5226d0a100 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

86cdb7a38f [provision] serialize measures defined on the result level

fbf192d6ba Merge pull request '[provision wf] serialize measures defined on the result level' (#196 ) from xml_measures into beta

Reviewed-on: #196

claudio.atzori added 1 commit 2022-02-24 12:51:59 +01:00

e7016c3981 Merge branch 'master_202203' into beta

claudio.atzori manually merged commit a87c070447 into master_202203

2022-02-24 12:52:29 +01:00

claudio.atzori commented

2022-02-25 09:40:41 +01:00

Integrate in the master branch with fe1c966cbf.

Integrate in the master branch with fe1c966cbf535418d7307090d74bb4bf442a7290.

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#197

promoting beta branch to master - March 2022 #197

[Graph model dhp-schemas:2.10.29]

[Graph dump]

[Graph processing]

[Graph provision]

[Graph model `dhp-schemas:2.10.29`]