[GRAPH DUMP]Moving Measures #159

miriam.baglioni · 2021-11-16T12:09:34+01:00

miriam.baglioni commented

2021-11-16 12:09:34 +01:00

This PR moves the Measure element at the level of the instance. Removes also the originalId of OpenAIRE (the ones starting with 50|) from the set of dumped originalIds.

It also adds filtering criteria to remove some relation from the dump. The filtering is based on the relclass value: all the relations having the relclass in the set specified by the new parameter removeSet will be filtered out from the dump. The relclass have to be split by ';'

This PR moves the Measure element at the level of the instance. Removes also the originalId of OpenAIRE (the ones starting with 50|) from the set of dumped originalIds. It also adds filtering criteria to remove some relation from the dump. The filtering is based on the relclass value: all the relations having the relclass in the set specified by the new parameter *removeSet* will be filtered out from the dump. The relclass have to be split by ';'

claudio.atzori was assigned by miriam.baglioni

2021-11-16 12:09:34 +01:00

miriam.baglioni added 15 commits 2021-11-16 12:09:35 +01:00

0857849a86 [Graph DUMP] Remove dump of measure until it will be clear where to put it (at the level of result or at the level of the instance)

c10ff6928c [Graph DUMP] add schema of the dump related to the model as in dhp-schemas.2.8.31. Note the measere element at the level of the result has been removed because of issues on where to display it: at the level of the result or at the level of the entity

a22c29fba1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

5681e89544 Update 'dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/dump/schemas/result_schema.json'

8442efd8d1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

94918a673c [Graph DUMP] Fix issue for empty origilaId list

88b73f4f49 mergin with branch beta

8cc50ecee0 [Graph Dump] changed AccessRight with BestAccessRight in the dump and modified the dependency to the schema to the SNAPSHOT

e5498052e8 [Graph DUmp] removed OpenAccessRoute from test in best access right

b8bdabfae9 [Graph DUmp] removed OpenAccessRoute from test in best access right

b3f9370125 merge with beta - resolved conflict in pom

43cae4ad88 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

6595135a1a [Dump Schemas] changed the schema of the dumped result according to the modifications in the bestAccessRight type

99d86134f5 [Graph Dump] changed the dump since the measures have been moded at the level of the instance

c6a9f0a1a8 mergin with branch beta

miriam.baglioni changed title from ~~Moving Measures~~ to [GRAPH DUMP]Moving Measures

2021-11-16 12:11:57 +01:00

miriam.baglioni added 14 commits 2021-11-19 15:56:45 +01:00

91354c6068 - fetching all context related results

- storing tables as parquet

26f086dd64 removed the too restrctive clause. will discuss again

49f897ef29 [cleaning wf] fixed regex used to spot garbage in result titles; adjusted threshold for filtering titles

6d4a1c57ee [Resolve Entities] Change test dataset to mirror the modification in the creation of the map between the pids and the unresolved

82a4e4efae [cleaning wf] fixed methodology to rule out invalid result titles, based on https://support.openaire.eu/issues/7206

e0395719d7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

c283406829 added Universidad Polytecnica de Madrid

cb3adb90f4 Merge branch 'beta' into beta

10a32f287f Merge pull request '[stats wf] RIs, affiliations, parquet' (#156 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #156

1b79c0ee79 mergin with branch beta

0136a8c266 [Graph Dump] Change test to mirror that measure is at the level of the isntance

5dc5792722 [Graph Dump] Change test resource to mirror the movement of the measure element

9fae872181 [Graph Dump] changed to mirror the changes in the model

0506fa2654 [Graph Dump] changed to mirror the changes in the model

miriam.baglioni added 3 commits 2021-12-02 14:11:27 +01:00

87eedad898 -

8905a39bf3 mergin with branch beta

58bc3f223a [GRAPH DUMP] Add filtering for relation we do not want to dump. It is based on the relclass

miriam.baglioni added 1 commit 2021-12-02 14:15:25 +01:00

d9f80488cc [GRAPH DUMP] Add one more test to check the filtering of the relations

miriam.baglioni added 1 commit 2021-12-03 12:36:01 +01:00

4bb1d43afc -

miriam.baglioni added 25 commits 2021-12-14 15:03:55 +01:00

9cac283bec implemented Instance serialization features requested in https://support.openaire.eu/issues/7156

863a2f9db3 avoid to filter OAF records defined as invisible = true

c4c705aa46 Merge pull request 'Cleaning of invisible records' (#168 ) from clean_invisible_records into beta

Reviewed-on: #168

dd0b2e5244 Merge branch 'beta' into instance_group_by_url

372633880f Merge pull request 'XML serialisation of instances with the same URLs' (#167 ) from instance_group_by_url into beta

Reviewed-on: #167

f7011b90d8 format code

0fa0ce33d6 removed duplicated on gitignore

f430688ff7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

08795cbd30 using helper method from ModelSupport to find the inverse relation descriptor

9132727793 fixed date cleaning test

7af0bbd0b1 [scala-refactor] Module dhp-aggregation:

Moved all scala source into src/main/scala and src/test/scala

81bf604059 [scala-refactor] Module dhp-common:

Moved all scala source into src/main/scala and src/test/scala

bf880e2508 [scala-refactor] Module dhp-graph-mapper:

Moved all scala source into src/main/scala and src/test/scala

e9f285ec4d [scala-refactor] Module dhp-doiboost:

Moved all scala source into src/main/scala and src/test/scala

96a7d46278 [Graph Dump] fixed tests

ed0c352799 [test-fixing] fixed wrong test

d1df01ff1e [Graph Dump] fixed resource for test

d9836f0cf3 [OpenCitations] fixed test when executed one after the other

5d51b3dd4a Merge pull request 'scala_refactor' (#169 ) from scala_refactor into beta

Reviewed-on: #169

b113586207 resolved conflicts

8d755cca80 -

936578aaf1 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

573bd17cbb Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

a3592b463a Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

56409d1281 [Dump] resolved conflicts with beta and merging

miriam.baglioni added 1 commit 2021-12-15 17:21:56 +01:00

5e5dfd619c Merge branch 'beta' into dump

miriam.baglioni added 4 commits 2021-12-22 12:37:09 +01:00

63b648b0dd Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

3cc1b7b153 mergin with branch beta

460e6b95d6 [Graph Dump] -

89ea9fa0e1 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

miriam.baglioni added 1 commit 2021-12-22 12:39:58 +01:00

be0acccf42 Merge branch 'beta' into dump

miriam.baglioni merged commit 9d19b057b8 into beta

2021-12-22 12:40:35 +01:00

miriam.baglioni referenced this issue from a commit

2021-12-22 12:40:36 +01:00

Merge pull request '[GRAPH DUMP]Moving Measures' (#159) from dump into beta

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#159