[GRAPH DUMP]Moving Measures #159

Merged

miriam.baglioni merged 39 commits from dump into beta

miriam.baglioni commented

Collaborator

This PR moves the Measure element at the level of the instance. Removes also the originalId of OpenAIRE (the ones starting with 50|) from the set of dumped originalIds.

It also adds filtering criteria to remove some relation from the dump. The filtering is based on the relclass value: all the relations having the relclass in the set specified by the new parameter removeSet will be filtered out from the dump. The relclass have to be split by ';'

This PR moves the Measure element at the level of the instance. Removes also the originalId of OpenAIRE (the ones starting with 50|) from the set of dumped originalIds. It also adds filtering criteria to remove some relation from the dump. The filtering is based on the relclass value: all the relations having the relclass in the set specified by the new parameter *removeSet* will be filtered out from the dump. The relclass have to be split by ';'

claudio.atzori was assigned by miriam.baglioni

miriam.baglioni added 15 commits

0857849a86 [Graph DUMP] Remove dump of measure until it will be clear where to put it (at the level of result or at the level of the instance)

c10ff6928c [Graph DUMP] add schema of the dump related to the model as in dhp-schemas.2.8.31. Note the measere element at the level of the result has been removed because of issues on where to display it: at the level of the result or at the level of the entity

a22c29fba1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

5681e89544 Update 'dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/dump/schemas/result_schema.json'

8442efd8d1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

94918a673c [Graph DUMP] Fix issue for empty origilaId list

88b73f4f49 mergin with branch beta

8cc50ecee0 [Graph Dump] changed AccessRight with BestAccessRight in the dump and modified the dependency to the schema to the SNAPSHOT

e5498052e8 [Graph DUmp] removed OpenAccessRoute from test in best access right

b8bdabfae9 [Graph DUmp] removed OpenAccessRoute from test in best access right

b3f9370125 merge with beta - resolved conflict in pom

43cae4ad88 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

6595135a1a [Dump Schemas] changed the schema of the dumped result according to the modifications in the bestAccessRight type

99d86134f5 [Graph Dump] changed the dump since the measures have been moded at the level of the instance

c6a9f0a1a8 mergin with branch beta

miriam.baglioni changed title from ~~Moving Measures~~ to [GRAPH DUMP]Moving Measures

miriam.baglioni added 14 commits

91354c6068 - fetching all context related results

- storing tables as parquet

26f086dd64 removed the too restrctive clause. will discuss again

49f897ef29 [cleaning wf] fixed regex used to spot garbage in result titles; adjusted threshold for filtering titles

6d4a1c57ee [Resolve Entities] Change test dataset to mirror the modification in the creation of the map between the pids and the unresolved

82a4e4efae [cleaning wf] fixed methodology to rule out invalid result titles, based on https://support.openaire.eu/issues/7206

e0395719d7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

c283406829 added Universidad Polytecnica de Madrid

cb3adb90f4 Merge branch 'beta' into beta

10a32f287f Merge pull request '[stats wf] RIs, affiliations, parquet' (#156 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #156

1b79c0ee79 mergin with branch beta

0136a8c266 [Graph Dump] Change test to mirror that measure is at the level of the isntance

5dc5792722 [Graph Dump] Change test resource to mirror the movement of the measure element

9fae872181 [Graph Dump] changed to mirror the changes in the model

0506fa2654 [Graph Dump] changed to mirror the changes in the model

miriam.baglioni added 3 commits

87eedad898 -

8905a39bf3 mergin with branch beta

58bc3f223a [GRAPH DUMP] Add filtering for relation we do not want to dump. It is based on the relclass

miriam.baglioni added 1 commit

d9f80488cc [GRAPH DUMP] Add one more test to check the filtering of the relations

miriam.baglioni added 1 commit

4bb1d43afc -

miriam.baglioni added 25 commits

9cac283bec implemented Instance serialization features requested in https://support.openaire.eu/issues/7156

863a2f9db3 avoid to filter OAF records defined as invisible = true

c4c705aa46 Merge pull request 'Cleaning of invisible records' (#168 ) from clean_invisible_records into beta

Reviewed-on: #168

dd0b2e5244 Merge branch 'beta' into instance_group_by_url

372633880f Merge pull request 'XML serialisation of instances with the same URLs' (#167 ) from instance_group_by_url into beta

Reviewed-on: #167

f7011b90d8 format code

0fa0ce33d6 removed duplicated on gitignore

f430688ff7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

08795cbd30 using helper method from ModelSupport to find the inverse relation descriptor

9132727793 fixed date cleaning test

7af0bbd0b1 [scala-refactor] Module dhp-aggregation:

Moved all scala source into src/main/scala and src/test/scala

81bf604059 [scala-refactor] Module dhp-common:

Moved all scala source into src/main/scala and src/test/scala

bf880e2508 [scala-refactor] Module dhp-graph-mapper:

Moved all scala source into src/main/scala and src/test/scala

e9f285ec4d [scala-refactor] Module dhp-doiboost:

Moved all scala source into src/main/scala and src/test/scala

96a7d46278 [Graph Dump] fixed tests

ed0c352799 [test-fixing] fixed wrong test

d1df01ff1e [Graph Dump] fixed resource for test

d9836f0cf3 [OpenCitations] fixed test when executed one after the other

5d51b3dd4a Merge pull request 'scala_refactor' (#169 ) from scala_refactor into beta

Reviewed-on: #169

b113586207 resolved conflicts

8d755cca80 -

936578aaf1 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

573bd17cbb Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

a3592b463a Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

56409d1281 [Dump] resolved conflicts with beta and merging

miriam.baglioni added 1 commit

5e5dfd619c Merge branch 'beta' into dump

miriam.baglioni added 4 commits

63b648b0dd Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

3cc1b7b153 mergin with branch beta

460e6b95d6 [Graph Dump] -

89ea9fa0e1 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

miriam.baglioni added 1 commit

be0acccf42 Merge branch 'beta' into dump

miriam.baglioni merged commit 9d19b057b8 into beta 2 years ago

miriam.baglioni referenced this issue from a commit

Merge pull request '[GRAPH DUMP]Moving Measures' (#159) from dump into beta

The pull request has been merged as 9d19b057b8.

You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.

git checkout -b dump beta

git pull origin dump

Step 2:

Merge the changes and update on Gitea.

git checkout beta

git merge --no-ff dump

git push origin beta

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Loading…

There is no content yet.

Labels Milestones

[GRAPH DUMP]Moving Measures #159

Step 1:

Step 2: