promoting beta branch to master - April 2022 #209

Manually merged

claudio.atzori merged 0 commits from beta into master

2022-04-06 10:28:15 +02:00

claudio.atzori commented

2022-04-06 09:55:32 +02:00

Owner

[Graph model `dhp-schemas:2.10.32`]

consent fields for datasources
added comparator for refereed field instances

[Graph processing]

#201 Updated construction of the HostedByMap. It uses the DOAJ json file instead of the csv one to get also information about the open access start date. Extends also the peerReview information that was already collected.
#202 Added mapping for integrating the datasource consent fields as part of the graph.
#203 Fix on the country propagation. Revises the implementation, getting rid of hive and compute the preparation step through spark actions.
#204 Instances grouped by URL expose only one refereed field.

[Graph stats]

#207 added APCs in monitor db
#206 fix: views cannot be stored as parquet...
#205 various fixes, organization ids for inst. dashboard

Note: the stats related PRs are being integrated in the master branch, but as their testing is still ongoing on beta, the production workflow will still use the previous version of the stats update/promote workflows, deployed at

/lib/dnet/PROD/oa/graph/stats_promote_202203
/lib/dnet/PROD/oa/graph/stats_update_202203

#### [Graph model `dhp-schemas:2.10.32`] * consent fields for datasources * added comparator for refereed field instances ### [Graph processing] * #201 Updated construction of the HostedByMap. It uses the DOAJ json file instead of the csv one to get also information about the open access start date. Extends also the peerReview information that was already collected. * #202 Added mapping for integrating the datasource consent fields as part of the graph. * #203 Fix on the country propagation. Revises the implementation, getting rid of hive and compute the preparation step through spark actions. * #204 Instances grouped by URL expose only one refereed field. ### [Graph stats] * #207 added APCs in monitor db * #206 fix: views cannot be stored as parquet... * #205 various fixes, organization ids for inst. dashboard Note: the stats related PRs are being integrated in the master branch, but as their testing is still ongoing on beta, the production workflow will still use the previous version of the stats update/promote workflows, deployed at ``` /lib/dnet/PROD/oa/graph/stats_promote_202203 /lib/dnet/PROD/oa/graph/stats_update_202203 ```

miriam.baglioni was assigned by claudio.atzori

2022-04-06 09:55:33 +02:00

claudio.atzori self-assigned this 2022-04-06 09:55:33 +02:00

claudio.atzori added 43 commits 2022-04-06 09:55:33 +02:00

efeeebfee1 fixed query after the change in the indicator table

3970651ee1 Merge pull request 'fixed query after the change in the indicator table' (#200 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #200

ad78e505da yet another fix

44b0c03080 [HostedByMap] update to download the json instead of the csv

8a41f63348 [HostedByMap] update to download the json instead of the csv

b7c2340952 [HostedByMap - DOIBoost] changed to use code moved to common since used also from hostedbymap now

5d608d6291 [HostedByMap] changed the model to include also oaStart date and review process that could be possibly used in the future

2c5087d55a [HostedByMap] download of doaj from json, modification of test resources, deletion of class no more needed for the CSV download

9f3036c847 Merge pull request 'HostedByMap' (#201 ) from hostedByMap_update into beta

Reviewed-on: #201

f25407bbe2 added mapping for datasource consent fields to integrate them in the graph

2b643059fa [Country Propagation] changed the logic to get the collectedfrom at the result level. To fix issue when no instance is created for a result that should have the country associated. Change the code to use spark instead of hive to prepare the data needed for the propagation step. Added new tests for the intermediate steps and new verification for the propagation itself

4437f9345d [Country Propagation] left out from previous commit

2fbb35ade5 mergin with branch beta

12de9acb0d [Country Propagation] left out from previous commit

d48ccfd65e Merge pull request 'enrichment_country' (#203 ) from enrichment_country into beta

Looks good to me

Reviewed-on: #203

f430029596 cleanup

c959639bd5 dependency updated to the new pace-core version

89fd275480 [HostedByMap] added left over from PR and fixed issue on workflow

dcfbeb8142 yet more typos

2e8394ecf8 creating aaall tables as parquet

62f91b0869 cleanup

43733c1a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

f10066547b increased spark.sql.shuffle.partitions in affiliation_from_semrel_propagation

48d32466e4 instances grouped by URL expose only one refereed

7b8f85692e [Enrichment country] fixed issues with parameters and workflow args

d8503cd191 added moooar organizations

61319b2e83 updated dhp-schema version; set entity-level dataInfo before & after merging the fields from the group of duplicates

3610f1749a merged pom.xml from beta branch

741bc99c47 Merge branch 'beta' into datasource_pdf_consent

fa3cb84f77 Merge pull request 'Datasource consent fields' (#202 ) from datasource_pdf_consent into beta

Reviewed-on: #202

395ac6ecec merged pom.xml from beta branch

96aa2a5d0d Merge branch 'beta' into instance_group_by_url

5d53ac95aa Merge pull request 'XML serialisation of instances with the same URLs - 2nd round' (#204 ) from instance_group_by_url into beta

Reviewed-on: #204

9fa3dd78fe Merge pull request '[stats wf] various fixes, organization ids for inst. dashboard' (#205 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #205

eca82e30c9 updated dhp-schema version

0a0ae84c22 [graph raw] DOI based instance URLs on https

1b11010169 minor fix

ee24f3eb2c views cannot be stored as parquet...

de85367695 Merge pull request '[stats wf] fix: views cannot be stored as parquet...' (#206 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #206

fff0b3cc19 added apcs in monitor db

7112806a73 views cannot be stored as parquet...

873369af1c Merge pull request '[stats wf] added apcs in monitor db' (#207 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #207

e77d104951 [OC] added / to workflow path

claudio.atzori manually merged commit 8c457f1b2c into master

2022-04-06 10:28:15 +02:00

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#209

No description provided.