WIP: Graph footprint optimisation #287

claudio.atzori · 2023-04-06T11:54:18+02:00

claudio.atzori commented

2023-04-06 11:54:18 +02:00

This PR introduces several enhancements on the disk/memory footprint of the graph processing pipeline.

lighter definition of the model
customised serialization that avoids to serialise empty fields
using built-in json instead of serde using ObjectMapper
using file parquet as encoding file of the graph

This PR introduces several enhancements on the disk/memory footprint of the graph processing pipeline. * lighter definition of the model * customised serialization that avoids to serialise empty fields * using built-in json instead of serde using ObjectMapper * using file parquet as encoding file of the graph

claudio.atzori added 21 commits 2023-04-06 11:54:19 +02:00

1845dcfedf WIP: refactoring the internal graph data model and its utilities

67735f7e9d WIP: refactoring model utilities

d9c9482a5b WIP: refactoring the internal graph data model and its utilities

606cada7a4 trying to adapt to the new data model

934c1846f8 wip: large refactoring

04b12a35cd Moved cressref Utility to dhp-aggregation

6bd5a792dd Added vocabolary mocks

d04610480a wip: large refactoring

508648e1d8 Merge branch 'ticket_8369' of https://code-repo.d4science.org/D-Net/dnet-hadoop into ticket_8369

125657ed4c -

63c5c5848d wip: large refactoring

8f777af827 increased version number of dhp-code-style

990e3e2f60 code refactor

6b0a08e29c Merge branch 'ticket_8369' of https://code-repo.d4science.org/D-Net/dnet-hadoop into ticket_8369

8af8b2ea27 Now Crossref mapping and dhp-aggregation compile

56495cc727 Merge branch 'ticket_8369' of https://code-repo.d4science.org/D-Net/dnet-hadoop into ticket_8369

8ddcf10075 resolved conflicts

624c62f62d [enrichment] changed to make it compile against the new model

eca7ebab6d [enrichment] adding relations in one side only

74c40fd4e2 Merge branch 'ticket_8369' of https://code-repo.d4science.org/D-Net/dnet-hadoop into ticket_8369

2a914e4463 merged from beta. It compiles

claudio.atzori added 3 commits 2023-04-21 08:47:24 +02:00

41e00bcd07 [graph provision] avoid to parse again the XML records, apparently the escaped XML characters get unescaped invalidating the record

518618f1a9 [graph cleaning] avoid to overwrite the subject class to 'keyword' for those with provenance 'subject:fos'

c3dff7e199 Merge branch 'beta' into ticket_8369

claudio.atzori added 1 commit 2023-04-21 08:47:53 +02:00

cdb5aad82b merge from beta

claudio.atzori added 10 commits 2023-04-26 16:02:11 +02:00

6d3d18d8b5 [graph cleaning] WIP: refactoring of the cleaning stages

4f5ba0ed52 [graph cleaning] WIP: refactoring of the cleaning stages, unit tests

488d9a5eaa [graph cleaning] WIP: refactoring of the cleaning stages, unit tests

90e61a8aba [graph cleaning] WIP: refactoring of the cleaning stages, unit tests

c07857fa37 [graph cleaning] unit tests & cleanup

b502f86523 fixed input path supplemented to GetDatasourceFromCountry; adjusted the various spark.sql.shuffle.partitions

2a6ba29b64 [graph cleaning] unit tests & cleanup

dead87917f [graph cleaning] cleanup

864f4051d3 [graph cleaning] added missing case

8c9a77d7eb WIP merged from graph_cleaning_refactoring, applying model simplification

claudio.atzori changed title from ~~Graph footprint optimisation~~ to WIP: Graph footprint optimisation

2023-05-02 09:57:59 +02:00

sandro.labruzzo added 3 commits 2023-05-09 12:23:54 +02:00

f8f4b9a018 -Renamed RelationInverse into RelationLabel.

-Removed findRelation from ModelSupport
-code formatted

88fffa6dbd - Removed ModelConstants and update Relation enum

ba380f5826 Makes the code compilable

sandro.labruzzo added 1 commit 2023-05-09 13:55:02 +02:00

706631586b Now it compiles

sandro.labruzzo added 1 commit 2023-05-10 09:05:29 +02:00

883dcd910e Make the tests compilable

alessia.bardi added 1 commit 2023-06-07 10:29:53 +02:00

af1f8af788 Updated officialnmae of pangaea in hostedbymap for Datacite to avoid duplicate entries in the source filter of the portal

claudio.atzori added this to the OpenAIRE - DNet project 2023-10-26 09:59:22 +02:00

This pull request has changes conflicting with the target branch.

dhp-common/src/main/java/eu/dnetlib/dhp/oa/merge/AuthorMerger.java
dhp-common/src/main/java/eu/dnetlib/dhp/oa/merge/DispatchEntitiesSparkJob.java
dhp-common/src/main/java/eu/dnetlib/dhp/oa/merge/GroupEntitiesSparkJob.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/CleaningFunctions.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/IdentifierFactory.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/MergeUtils.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/ModelHardLimits.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/OafMapperUtils.java
dhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/OrganizationPidComparator.java