promoting beta branch to master - July 2022 #227

claudio.atzori · 2022-07-13T17:05:13+02:00

claudio.atzori commented

2022-07-13 17:05:13 +02:00

[Graph model `dhp-schemas:2.12.0`]

introduced support for EOSC Services

[Content aggregation]

#211 fileGZip-collector-plugin
#214 UsageCounts actionset as result.measure
88d531dc91 exclude FAIRsharing records from Datacite
634869ce95 deleted hierarchical rels from ror action set

[Graph processing]

#226 mapping oaf:fulltext element in the result.fulltext field
#225 score class in the XML serialization
#220 author name parsing
#219 mapping relationship from trasformed records based on oaf:relation
#217 updated dhp-schema to include the datasource/eosc model, follow up updates
#216 [cleaning wf] Oozie workflow for cleancontext
#215 [cleaning wf] eosc subjects, rough implementation
#208 [cleaning wf] Remove Context from result
d098ad0d93 [hb patch] updated map using deduped datasource ids

[Graph stats]

#218 fixed *_organization tables, removed double result_apc table creation from monitor
#213 added open citations & more orgs in monitor, removed collab indicator
#210 computing stats in each step
#207 added APCs in monitor db
#206 fix: views cannot be stored as parquet...

#### [Graph model `dhp-schemas:2.12.0`] * introduced support for EOSC Services ### [Content aggregation] * #211 fileGZip-collector-plugin * #214 UsageCounts actionset as result.measure * 88d531dc91 exclude FAIRsharing records from Datacite * 634869ce95 deleted hierarchical rels from ror action set ### [Graph processing] * #226 mapping `oaf:fulltext` element in the `result.fulltext` field * #225 score class in the XML serialization * #220 author name parsing * #219 mapping relationship from trasformed records based on `oaf:relation` * #217 updated dhp-schema to include the datasource/eosc model, follow up updates * #216 [cleaning wf] Oozie workflow for cleancontext * #215 [cleaning wf] eosc subjects, rough implementation * #208 [cleaning wf] Remove Context from result * d098ad0d93 [hb patch] updated map using deduped datasource ids ### [Graph stats] * #218 fixed *_organization tables, removed double result_apc table creation from monitor * #213 added open citations & more orgs in monitor, removed collab indicator * #210 computing stats in each step * #207 added APCs in monitor db * #206 fix: views cannot be stored as parquet...

alessia.bardi was assigned by claudio.atzori

2022-07-13 17:05:14 +02:00

miriam.baglioni was assigned by claudio.atzori

2022-07-13 17:05:14 +02:00

sandro.labruzzo was assigned by claudio.atzori

2022-07-13 17:05:14 +02:00

claudio.atzori added 123 commits 2022-07-13 17:05:16 +02:00

79336d46c5 [Clean Context] first naive implementation of a functionality to clean not wanted contextes from one result. This implementation simply verifies the main title of the results start with a given string

c442c91f89 computing stats in each step

4190c9f6bc [graph raw] avoid NPEs importing datasource consent fields

e612489670 Add fileGZip collector plugin and respective test

bc1bf55507 Add AbstractSplittedRecordPlugin

91e32f12ed Merge branch 'master' into beta

d0b84d3297 Add FileCollectorPlugin and respective test

4eff7856f5 Merge pull request '[stats-wf] computing stats in each step' (#210 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #210

48b580b45c [graph enrichment] fixed country_propagation oozie workflow definition, parameter saveGraph is not needed anymore by the SparkCountryPropagationJob

73c172926a [Doiboost] fixed fundingReference extraction from the Crossref records

4314db55c8 migration to services: update sql queries

c96a8613f8 update SQL queries

b7cd2c6ca1 added open citations

869407c6e2 [Measures] added new measure (usagecounts) as action set. Measure added at the level of the result. Ref #7587

5feae77937 [Measures] last changes to accomodate tests

dbfbe8841a [Clean Context] changed the description in input parameters

a38f0f5ea7 mergin with branch beta

61c0266a44 Merge pull request 'Remove Context from result' (#208 ) from cleancontext into beta

Reviewed-on: #208

5295effc96 [Measures] fixed issue

c304657d91 [Measures] put the logic in common, no need to change the schema

eabb40fccc Merge branch 'beta' into 7096-fileGZip-collector-plugin

88acad76f9 Merge branch 'beta' into eosc_dimitris

d012d125d7 [EOSCTag] -

b61efd613b [Measures] addressed comments in the PR

bebb2a0560 Merge branch 'eosc_dimitris' of https://code-repo.d4science.org/D-Net/dnet-hadoop into eosc_dimitris

20de75ca64 [Measures] removed typo

a289c9eae2 Merge pull request '[Measures] added new measure (UsageCounts)' (#214 ) from eosc_dimitris into beta

Reviewed-on: #214

ccba1a3db1 [Clean Context] added logic to cleaning workflow to accomodate also context cleaning

5b7d9e741c [Clean Context] added logic to cleaning workflow to accomodate also context cleaning

29150a5d0c code formatting

9a961a0092 [Clean Context] fixed issue in param name

6dc68c48e0 [EOSCTag] -

e0915061c2 [Clean Context] fixed issue in param name

7cb7066472 [EoscTag] first "rough" implementation

aa12429f50 Modified last intersection since we lost many titles.

a82ec3aaaf code formatter

30105f0722 Merge branch 'beta' into 7096-fileGZip-collector-plugin

bbb77052d3 [EOSCTag] first test

54162f5c4f Merge branch 'beta' into cleancontext

19d90658fc [Clean Context] added description to parameters

911ce0780a Merge branch 'cleancontext' of https://code-repo.d4science.org/D-Net/dnet-hadoop into cleancontext

81242538e6 Merge pull request 'Oozie workflow for cleancontext' (#216 ) from cleancontext into beta

Reviewed-on: #216

Looks good. We need to extend the cleaning workflow parameters to enable the extra step only when it is needed.

87bff36d9e mergin with branch beta

27c85e901a [EOSCTag] added resources and finalized test for Jupyter Notebook tagging

dfbd2bcbea [EOSC TAG] added logic in case subject is null

88562c0930 [EOSC TAG] added test for galaxy for title and description criterias

e342ec93f0 [EOSCTag] prepared resources for test

81c4496d32 Merge branch 'beta' into 7096-fileGZip-collector-plugin

8c22e5c30a added fix to include date array with only year or year and month

78015a5733 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

5ffc24d1ba EOSC Services - ongoing update

623f7be26d Fix reading files from HDFS in FileCollector & FileGZipCollector plugins

f5f532d134 EOSC Services - ongoing update

05c1ea92e9 EOSC Services - added Service-specific fields in the XML record serialization

a8c51f6f16 EOSC Services - fixed query and testing preparation

e37177e1ce mergin with branch beta

b6a7ff3a99 EOSC Services - removed fields from mapping, testing preparation

2ade69dea6 EOSC Services - minor

a21fe310e5 [EOSCTag] last test and change in the implementation to search in title and descriptio

9e12cb3c92 EOSC Services - removed field knowledgegraph; depending on the released schema module

da611cfbbd [eosc_services] resolved merge conflicts

3aeedd931a [EOSCTag] fixed issue in case description is null. Modified test resources and classes

bd1108f98b mergin with branch beta

8a72de4011 [EOSCTag] modified workflow to execute all the steps and not only the last one

5fe25cc51c Merge pull request '[eosc tag] set the eosc subjects, rough implementation' (#215 ) from eosc_tag into beta

Reviewed-on: #215

846975c886 [eosc_services] using the correct 'keyword' subject type, as declared in the dnet:subject_classification_typologies vocabulary

658450d9a3 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

cfbbcaf7c4 commented out indi_result_org_country_collab

61b4c19e65 restored indi_result_org_country_collab, removed indi_result_org_collab

a056f59c6e [UsageCount] make it as an action set as it should be, plus changed the test to make them work as well now

89657a0b78 [UsageCount] refactoring

378020e30a [eosc_services] unit test adaptation

77bc9863e9 [openorgs] mapping parent/child relations without massaging the semantic labels

23334479bb removed yet another collab, added more orgs in monitor

5d3b4a9c25 [graph merge beta] merge datasource originalid, collectedfrom, and pid lists

ca8d26bcb4 added better filter for openCitations

22f65680b9 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

c25134f28d fixed typo

e4eac1d20b [EOSC TAG] added code to remove EOSC Jupyter Notebook from subjects and put EOSC as classid in the qualifier

3fc9efeab6 fixed typo, addded open citations and apcs in monitor

8160763330 fixed conflict

0dc33ea391 [openorgs] fixed parent/child query, using the correct semantic labels

4c50f35c8b update publication Date format

c1971d52c4 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

997c50078e [graph grouping] drop relation target path before copying from source

6442763f97 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

d098ad0d93 [hb patch] updated map

f5207885e3 [EOSCTag] changed code to remove EOSC Jupyter Notebook and modified test to exclude galaxy + software from the tagging for Galaxy

eaf9385ae5 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

c298c148cb [CountryPropagation] fix NPE issue

5e0b8f9b5f [CountryPropagation] refactoring

5c2949a864 Merge pull request '[stats wf] added open citations & more orgs in monitor, removed collab indicator' (#213 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #213

db088cc69c fixed *_organization tables

4b6913787b [DOI-BOOST] added one method in test of crossref mapping to aof and one resource. Related to ticket 7807

574492c659 removed double result_apc table creation from monitor

ab18c9daa9 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta

b229c6e7af Merge pull request 'beta' (#218 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #218

68bd58d6a4 tests for ROHub

922c6d66ef Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

634869ce95 deleted hierarchical rels from ror action set

b8cda65487 code formatting

116902c028 mapping relationship from trasformed records based on oaf:relation

88d531dc91 exclude FAIRsharing records from Datacite

4c8e820ff0 mapping relationship from trasformed records based on oaf:relation

06b5533d4c Merge branch 'beta' into 7096-fileGZip-collector-plugin

e03c0c7794 Merge branch 'beta' into oaf_relation_mapping

875ae29961 Merge pull request 'mapping relationship from trasformed records based on `oaf:relation`' (#219 ) from oaf_relation_mapping into beta

Reviewed-on: #219

c7b09c6225 Merge branch 'beta' into 7096-fileGZip-collector-plugin

b295a40d9c restored use of name_particles when parsing author names

c76ff6c613 Merge pull request '7096-fileGZip-collector-plugin' (#211 ) from 7096-fileGZip-collector-plugin into beta

Reviewed-on: #211

929b145130 code formatting

316b0fd73c added 'von' to the name particles file

4829b96bb5 Merge branch 'beta' into author_name_particles

cba9c2b7cc Merge pull request 'author name parsing' (#220 ) from author_name_particles into beta

Reviewed-on: #220

ee1f1eeca2 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

7da24c1dec added more logging

a88103bcf9 [action manager] added more testing

072f192853 include the class information in the measure XML serialization

138d1dfbf8 Merge pull request 'score class in the XML serialization' (#225 ) from measure_serialization into beta

Reviewed-on: #225

c09fcdb40b Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

fae681fea1 [Country Propagation] add check to avoid NPE on datasource.getDatasourceType().getClassis()

0c1cfee396 mapping oaf:fulltext elements in the result.fulltext field

28a32facf6 Merge pull request 'mapping `oaf:fulltext` element in the `result.fulltext` field' (#226 ) from oaf_fulltext_mapping into beta

Reviewed-on: #226

claudio.atzori added 1 commit 2022-07-19 14:18:04 +02:00

18b505d6a3 Merge branch 'master' into beta

claudio.atzori manually merged commit 1138b2ac8e into master

2022-07-19 14:18:17 +02:00

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#227

promoting beta branch to master - July 2022 #227

[Graph model dhp-schemas:2.12.0]

[Content aggregation]

[Graph processing]

[Graph stats]

[Graph model `dhp-schemas:2.12.0`]