promoting beta branch to master - October 2021 #149

Manually merged

claudio.atzori merged 2 commits from beta into master 2021-10-12 10:42:01 +02:00

Conversation 0 Commits 2 Files Changed 33 +157 -162

claudio.atzori commented 2021-10-11 17:29:24 +02:00

Owner

Copy Link

Graph model version [dhp-schemas:2.7.18]

• added constants declaring vocabulary names for relation fields
• aligned the graph dump schema to mirror the changes in the model
• Added openaccessroute at the level of the instance inside the AccessRight element
• Added pid and the alternate identifiers at the level of the instance;
• Added the bipFinder measures beta
• Updated the casing of the following terms (relation.relClass):
  -- isRelatedTo -> IsRelatedTo
• Added the following relClass terms:
  -- IsAmongTopNSimilarDocuments
  -- HasAmongTopNSimilarDocuments

Deduplication

• #144 implementation of the whitelist for similarity relations

Content aggregation

• #138 introduced workflow for downloading and unpackick the last official Crossref dumps
• #139 adaptations of the doiboost construction workflow related to the introduciton of the automatic download and unpack of the crossref official dump
• #130 towards EOSC datasource profiles, first version
• #141 UnknowHostException handling for ORCID collector api

Graph cleaning

• #136 #5644 introduced bulk hostedby patching
• #142 introduced support for relation label cleaning
• #143 Propagation relations not Cleaned

Stats workflow

• #145 updates in the mapping, indicators, wf
• #146 latest indicators and fix for RIs

Misc

• #137 refactoring of the libs used to download the CSV files. It replaces the old ones and moves it to dhp-common

Graph model version [dhp-schemas:2.7.18] - added constants declaring vocabulary names for relation fields - aligned the graph dump schema to mirror the changes in the model - Added openaccessroute at the level of the instance inside the AccessRight element - Added pid and the alternate identifiers at the level of the instance; - Added the bipFinder measures beta - Updated the casing of the following terms (relation.relClass): -- isRelatedTo -> IsRelatedTo - Added the following relClass terms: -- IsAmongTopNSimilarDocuments -- HasAmongTopNSimilarDocuments Deduplication - #144 implementation of the whitelist for similarity relations Content aggregation - #138 introduced workflow for downloading and unpackick the last official Crossref dumps - #139 adaptations of the doiboost construction workflow related to the introduciton of the automatic download and unpack of the crossref official dump - #130 towards EOSC datasource profiles, first version - #141 UnknowHostException handling for ORCID collector api Graph cleaning - #136 [#5644](https://code-repo.d4science.org/D-Net/dnet-hadoop/issues/5644) introduced bulk hostedby patching - #142 introduced support for relation label cleaning - #143 Propagation relations not Cleaned Stats workflow - #145 updates in the mapping, indicators, wf - #146 latest indicators and fix for RIs Misc - #137 refactoring of the libs used to download the CSV files. It replaces the old ones and moves it to dhp-common

claudio.atzori added the

enhancement

label 2021-10-11 17:29:24 +02:00

claudio.atzori self-assigned this 2021-10-11 17:29:24 +02:00

claudio.atzori added 204 commits 2021-10-11 17:29:25 +02:00

acd6056330 added shell action to automatically download the new dump and put it in a specified hdfs location

c4b18e6ccb changed the download.sh, added skip step to allow to not execute one phase and changed the workflow sequence of steps

63553a76b3 added code to download gold issn list from unibi

eb07f7f40f Hosted By Map

74f801b689 mergin with branch beta

52e2315ba2 removed trick for datasourcetypeui

0424f47494 HostedByMap fixing issues

708d0ade34 Merge branch 'beta' into hostedbymap

1fb572a33a added eosc fields

c72c960ffb added eosc fields

e6f1773d63 mapping of new eosc fields

cc0d3d8a7b mergin with branch beta

3d2bba3d5d removing not needed classes

9f1c7b8e17 tests

3e2a2d6e71 added new fields in xml

baad01cadc hostedbymap

1d6ac3715b merge branch with beta

d1807781c0 mergin with branch beta

613bd3bde0 Hosted By Map - refactor of the first attemp to prepare a new hosted by map dependent on the datasource in the graph and on two external sources: the gold list from unibi ad the doaj list of open access journal. Both the lists are downloaded from provided url parameter

d8b9b0553b Hosted By Map - model classes to store the intermediate information to be used to apply the hosted by map

7c6ea2f4c7 Hosted By Map - first attempt for the creation of intermedia information to be used to applu the hosted by map on the graph entities

1695d45bd4 Hosted By Map - Test class to verify the preparation of the intermediate information

ff1ce75e33 Hosted By Map - modification in the code to prepare the info needed to apply the HostedByMap. There is no need to join datasources with the hbm: all the information needed is in the hosted by map already

72df8f9232 Hosted By Map - removed the aggregator for the datasource (it is no more needed) and added a new aggregator for the results. Changed also the hostedBYMap aggregator

1e859706a3 Hosted By Map - Classes to apply the HBM to results and datasources

90e91486e2 Hosted By Map - test class to verify each step in the preparation process

ee7ccb98dc Hosted By Map - test class to verify the application of the hbm to results and datasource

17292c6641 Hosted By Map - resources for testing purposes

327cddde33 Hosted By Map - refactoring

461b8a29a0 removed not needed class

100e54e6c8 mergin with branch beta

9831725073 Hosted By Map - remove from workflow a step not needed. The hbm will be take care also of the integration of the unibi list of gold openaccess journals

a7bf314fd2 Hosted By Map - added new aggregator to get just one result per datasource id

8f7623e77a Hosted By Map - refactoring and application of the new aggregator

8ba8c77f92 Hosted By Map - refactoring

1e952cccf6 Hosted By Map - refactoring and deletion of not needed methods

eccf3851b0 Hosted By Map - refactoring

67ba4c40e0 Hosted By Map - added parameter resources

e94ae0b1de Hosted By Map - extention of the workflow to consider also the application of the map to publications and datasources

eb8c3f8594 Hosted By Map - test modified because of the application of the new aggregator on datasources

c7b71647c6 Hosted By Map - modification of the resource for testing the presence of only one entry per datasource id

b4eb026c8b mergin with branch beta

1965e4eece new workflow for downloading the dump of crossref and unpack it

5faeefbda8 added script to download the dump,changed the workflow input paramenters

bd096f5170 removed not needed param file

b6dcf89d22 mergin with branch beta

a5f82f442b Merge branch 'beta' into doiboost_wf

b7079804cb CrossrefDump - put token among the parameters

54a6cbb244 CrossrefDump - put token among the parameters

964f97ed4d cleanup

da20fceaf7 removed all the part related to the crossref dump download since it is done in a separate workflow

577f3b1ac8 added dnet workflows responsible for the graph construction, enrichment, provision

2efa5abda5 refactoring

9731a6144a hostedbymap - in case the journal is open access the access may be changed also for the best access right in the result

b688567db5 hostedbymap - modified part of test to check the bestaccessright changed

b954fe9ba8 mergin with branch beta

2ee21da43b suggestions from SonarLint

61d811ba53 suggestions from intellij

9f4db73f30 updated/fixed unit tests

8da3a25cf6 merging with branch beta

52c18c2697 removed not needed test class. Teh functionality has been moved

b1c6140ebf removed all comments in Italian

8229632839 adding assertions to the mapping of the unibi part of gold list

f9b6b45d85 reverting

55fc500d8d reverting

7aa3260729 reverting

524c06e028 reverting

d688749ad9 reverting

804589eb30 reverting

b6b58bba28 reverting

c6a2a780a9 reverting

0e1a6bec20 reverting

8ad7c71417 reverting

b966329833 reverting

cc3d72df0e removing not needed dependency

95e5482bbb removing not needed dependency

785db1d5b2 refactoring

9650eea497 reverting

e33daaeee8 reverting

ac417ca798 removed not needed test resource

08dd2b2102 moving the dependency version to the external pom file

8cdce59e0e [graph raw] let the mapping exceptions propagate

86d940044c added test to verify bad records from FWF-E-Book-Library

9587d4aee8 Merge branch 'beta' into hostedbymap

e91ffcd2f3 Merge pull request 'hostedbymap' (#136) from hostedbymap into beta ...

Reviewed-on: #136

6e84b3951f GetCSV refactoring - moving classes to dhp-common that have dependency with GetCSV class (that was located in graph-mapper)

bfe8f5335c GetCSV refactoring - copied model classes in test path

733bcaecf6 GetCSV refactoring - added test class (all the tests are disabled since they refer to remote resource)

7402daf51a GetCSV refactoring - added dependency to open-csv lib

d36e925277 GetCSV refactoring - moved under model package

b62cd656a7 GetCSV refactoring - changed the model to store only the information needed

4317211a2b GetCSV refactoring - refactoring due to movement

e9fc3ef3bc GetCSV refactoring - changed to use the new class to get and write the csv file

7a789423aa GetCSV refactoring - refactoring due to movement of classes

f0845e9865 GetCSV refactoring - refactoring due to movement of classes

335a824e34 GetCSV refactoring - fixed issue

ab8abd61bb GetCSV refactoring - refactoring due to movement of classes

9da74b544a GetCSV refactoring - refactoring due to movement of classes

d57b2bb927 GetCSV refactoring - removing not needed dependency

6b9e1bf2e3 GetCSV refactoring - removing not needed dependency

8769dd8eef GetCSV refactoring - refactoring due to movement of classes

ed183d878e GetCSV refactoring - modified test classes due to change in the model of projects and programme

a5f6edfa6c GetCSV refactoring - changed to mirror the original model class

f3d575f749 GetCSV refactoring - changed due to changes in input resource

58f241f4a2 GetCSV refactoring - changed due to change of input resource

5cd5714530 GetCSV refactoring - added ignore annotation for fields not in input csv

5f674efb0c moved dependency version in external pom

eaf077fc34 GetCSV refactoring - removed not needed dependency

964a46ca21 GetCSV refactoring - modified due to movement of classes

01db1f8bc4 GetCSV refactoring - removed not needed import

dfd1e53c69 added external dependency for version

32fd75691f refactoring

dc8b05b39e Hosted By Map - changed the association with the datasource id for the hostedby element: there is no more the need to compute it. With the new HBM it is already the id in the graph

f4ec81c92c mergin with branch beta

4e6575a428 Merge pull request 'Moving Download CSV' (#137) from refactoring_download_csv into beta ...

Reviewed-on: #137

3359f73fcf cleanup & best practices

baed5e3337 test classes moved in specific components

c3ad4ab701 minor fixes

7ee2757fcd fixed DownloadCSV parameters spec; workflow patching the hostedby replaces the graph content (publication, datasource) rather than creating a copy

17cefe6a97 [HBM] removed stale replace option

5f0903d50d fixed CSV downloader & tests

f74adc4752 added DownloadCSV2 as alternative implementation of the same download procedure

6eb7508995 mergin with branch beta

7743d0f919 consolidated dnet wf profiles into the same submodule

ed7e28490a change in sh

6fec71e8d2 removed the specific of the infra we are running the wf from the wf name

5856ca8a7b merging with branch beta - resolved conflicts

d43667d857 Merge pull request 'Automatic download of Crossref' (#138) from crossref_dw_wf into beta ...

Reviewed-on: #138

2c0a05f11a manually merged PR#139

bc7068106c added crossref download oozie workflow

82086f3422 fixed directory name

7c0c67bdd6 added mock pom

e5cf11d088 change open access route to result matching hbm to gold

a65d3caaea Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

812bd54c57 different funders in blacklist from BETA and PROD aggregator

a053e1513c different funders in blacklist from BETA and PROD aggregator

65822400ce CrossrefDump - added new parameter file that was missing

f3b6c392c1 CrossrefDump - moving parameter file under folder crossref_dump_reader

35880c0e7b CrossrefDump - changed the wf to be able to resume from one of the steps

45c62609af CrossrefDump - modified because parameter file was moved

882abb40e4 CrossrefDump -

5f8ccbc365 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

4c1474e693 Dealing with #6859#note-2: we have to decode URLs to avoid & and other chars encoded becasue of the original XML representation of data

931f430129 Merge branch 'beta' into datasource_model_eosc_beta

3bcac7e88c Merge pull request 'towards EOSC datasource profiles' (#130) from datasource_model_eosc_beta into beta ...

Reviewed-on: #130

412d2cb16a added dependencies to classgraph and opencsv. Bumped version of dhp-schemas

f19b04d41b code formatting after mvn compile

00a28c0080 originalId was renamed to acronym

45898c71ac fixed wrong doi in pubmed

ccf4103a25 keep the original url if the decoder fails for any reason

e8b3cb9147 Implemented method to download delta updates in EBI Links

3762b17f7b added VERSIOn and PART relationship and re-ordered according to my personal and obviously possibly biased ...

ordering

9f8a80deb7 fixed wrong import of unresolved relation in openaire

d4dadf6d77 reduced max number of PID in Relatedentity

3c6fc2096c fix bug on oai iterator that skip record cleaned

aed29156c7 changed behavior in transformation job, that doesn't fail at first error

ccee451dde added indicators of sprint 2 in monitor db

1250ae197f using new indicators for the definition of peerreviewed, gold, and green

c6ada217a1 fixed typo

f13cca7e83 moved dependencies of indicators before them...

43852bac0e creating other::other concept for all contexts

461bf90ca6 fixed the gold_oa definition

8fc89ae822 moved context table creation before indicators

9b1936701c fixed yet another typo

de9bf3a161 added cc_licences and abstracts in observatory db

aefa36c54b other task executions go ahead if UnknownHostException happens on a single task

8b804e7fe1 removed unused imports

ebf53a1616 added cleaning for relation fields: subRelType & relClass according to dedicated vocabs

663b1556d7 manually integrating PR#140 #140

12766bf5f2 Merge branch 'beta' into clean_relations

e9ccdf853f related to #132

954a16c213 Merge pull request 'Propagation relations not Cleaned' (#143) from enrichment into beta ...

Reviewed-on: #143

09c2eb7f62 Merge branch 'beta' into clean_relations

dd2329849f fixed the definition of cc_licence

2943287d10 fixed the definition of cc_licence, part II

71cfa386bc Merge pull request 'cleaning for relation fields' (#142) from clean_relations into beta ...

Reviewed-on: #142

8b681dcf1b attempt to make the observatory wf run in hive

853333bdde implementation of the whitelist for similarity relations

0c74f5667e Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

92a63f78fe multiple download attempts handling if a connection to orcid server fails

421d55265d created hive action for observatory queries

7fa60e166e Merge branch 'beta' into dedup_whitelist

df15a4dc9f Merge pull request 'UnknowHostException handling for orcid collector api' (#141) from enrico.ottonello/dnet-hadoop:beta into beta ...

Reviewed-on: #141

f358cabb2b fixed typo

a1e1cf32d7 fixed an impala error

b924276e18 tests to generate records for the EOSC-Future demo with the EOSC Jupyter Notebbok subject

a53acfbc06 Merge pull request '[stats] updates in the mapping, indicators, wf' (#145) from antonis.lempesis/dnet-hadoop:beta into beta ...

Reviewed-on: #145

474117c2e8 Merge branch 'beta' into dedup_whitelist

35619b93ee Merge pull request 'implementation of the whitelist for similarity relations' (#144) from dedup_whitelist into beta ...

Reviewed-on: #144

be79d74e3d Fixed DoiBoost generation to point to correct organization in affiliation relation

477cb10715 Merge remote-tracking branch 'origin/beta' into beta

66702b1973 Added node to update datacite

60a6a9a583 [graph2hive] added field 'measures' to the result view

13687fd887 Sprint 3 indicators update

3f25d2efb2 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta

cb7efe12ac Merge pull request 'beta' (#146) from antonis.lempesis/dnet-hadoop:beta into beta ...

Reviewed-on: #146

991b06bd0b removed generation of EBI links from old dump, now EBI link dump is created by another wf

f258bbb927 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

b84e0cabeb Implemented new method for update baseline

2557bb41f5 Implemented new method for update baseline inside scala node

9646b9fd98 implementation of the http call for the update of openorgs suggestions

611ca511db set configuration property in openorgs duplicates wf

8d3b60f446 test for patching records for EOSC Future

c48c43fa9e Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

claudio.atzori referenced this issue from a commit 2021-10-11 18:29:50 +02:00

manually merging PR#149 https://code-repo.d4science.org/D-Net/dnet-hadoop/pulls/149

sandro.labruzzo added 2 commits 2021-10-12 08:12:08 +02:00

8f99d2af86 Make the node of doiBoost to point to the correct OpenAire Organization in relations

5606014b17 code refactor see ticket #7065

claudio.atzori manually merged commit 83c90c7180 into master 2021-10-12 10:42:01 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature / refactoring

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

RDGraph

EOSC Research Discovery Graph

  

RSAC

EOSC Research Software APIs and Connectors

  

wontfix

This won't be fixed

No Label

bug

duplicate

enhancement

help wanted

invalid

question

RDGraph

RSAC

wontfix

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

claudio.atzori

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#149

No description provided.