WebCrowl #427

miriam.baglioni · 2024-04-22T10:35:26+02:00

miriam.baglioni commented

2024-04-22 10:35:26 +02:00

This PR is to include the relationsh of affiliations given from a webcrawl source.
This inclused all the relations found for IE, while for other countries only relations before 2021

This PR is to include the relationsh of affiliations given from a webcrawl source. This inclused all the relations found for IE, while for other countries only relations before 2021

claudio.atzori was assigned by miriam.baglioni

2024-04-22 10:35:26 +02:00

miriam.baglioni added 234 commits 2024-04-22 10:35:27 +02:00

cbd48bc645 Merge commit 'efd96e7e664e4139321e35e8d172b884ba4b61a1' into beta2master_sept_2022

a07c876922 Merge commit '27a91841e7fa2a1b615b4d1e161d606db5bead96' into beta2master_sept_2022

2abe2bc137 Merge commit '08ce2cadc2d84aa982726e429c280a905536a715' into beta2master_sept_2022

a387807d43 Merge commit 'b78889a0ce27a79c7ab2d8da05b118ee4f1bcb36' into beta2master_sept_2022

0ec2eaba35 Merge commit 'c1f2ffc53dc41f1fac3855b2d2df7d6a5ea15e3e' into beta2master_sept_2022

300ae6221c Merge commit '32cee1f619eb30d2e2ac6083435b76b1aba7db09' into beta2master_sept_2022

725c3c68d0 Merge commit '844f6eb46533cdd4be3210401b10401322079640' into beta2master_sept_2022

ab0efecab4 Merge commit '84598c75356cf580de6c81653a9351e9b8173639' into beta2master_sept_2022

d872d1cdd9 Merge commit 'a4815f6bec87f05be8cd740d236707949a0f746e' into beta2master_sept_2022

01d5ad6361 Merge commit 'd85ba3c1a9d7f0e80565742161ff6c9ecffd52b7' into beta2master_sept_2022

1c05aaaa2e Merge commit '3418ce50ac9b28fed4fa949919e6c8208738cdcf' into beta2master_sept_2022

45fc5e12be Merge commit 'cb7c07c54e59675e8dffe42b7f2a13f16c956068' into beta2master_sept_2022

281239249e Merge commit 'b7c387c21f946adbc9da90ded95166205195edb0' into beta2master_sept_2022

0849ebfd80 merged a11eb38065

c7e8ad853e Merge commit '2b5f8c9c9a3611c57ee5febfe262a455a39ad801' into beta2master_sept_2022

dbb567251a merged 853c996fa2

61aacb3271 Merge commit '1203378441dc6d8e8435cacd42e76e11746f6d1b' into beta2master_sept_2022

3e8499ce47 Merge commit '71b069ca90a2f7ec09d64241c60917d3636fc81e' into beta2master_sept_2022

d72a64ded3 Merge commit '690be4482fc84327dc7617acbc8d976d559df512' into beta2master_sept_2022

65209359bc Merge commit 'b5f7bd30be7f7adaaa28170740da0484b50a77ed' into beta2master_sept_2022

c527112e33 Merge commit 'ff6f789b6d9be0567b6ad72f8a0e75fe3f52726a' into beta2master_sept_2022

fd87571506 code formatting

35bb7c423f updated dhp-schemas version to 2.12.1

96062164f9 Merge pull request '[Aggregator graph|master] Discard invalid records' (#245 ) from discard-non-wellformed into master

Reviewed-on: #245

a0ee1a8640 [EOSC BulkTag] remove addition of eosc context for result with eosc if guidelines set

1329aa8479 [EOSC BulkTag] modified test to remove association of result to eosc when eoscifguidelines are set

b42e2c9df6 [Patch Hosted By] added fix to avoi NPE error when datasource official name is not provided

ec87149cb3 [Patch Hosted By] added fix to avoi NPE error when datasource official name is not provided. Removing datasources if no officialname has been provided

bdc8f993d0 [Patch Hosted By] check also the presence of datasource.officialname.value

840465958b [EOSC BulkTag] filtering aout the datasources registered in the eosc with compatibility different from 3.0, 4.0 for literature, data and CRIS to add the context eosc to the results

869e129288 [EOSC BulkTag] refactoring

7225fe9cbe integrated changes from discard-non-wellformed

10ec074f79 Merge remote-tracking branch 'antonis.lempesis/beta' into beta2master_sept_2022

b2c3071e72 Merge branch 'master' into beta2master_sept_2022

930f118673 fixed semantic (subreltype) for ServiceOrganization relations

e6d788d27a [stats wf] adding missing changes lost in PR#248

c01d528ab2 suppressing hyper verbose spark logs during unit test execution

80c5e0f637 code formatting

55fa3b2a17 Hive memory parameters

b18bbca8af [EOSC TAG] adding search in orp for jupyter notebook criteria

5f9383b2d9 [EOSC TAG] remove reduntant check for jupyter notebook

a929dc5fee integrated changes for mapping ROHub contents in the Graph

eb45ba7af0 extended mapping from ODF relations (PR#251)

af15b1e48d [eosc tag] extending criteria for Jupyter Notebook (adding to ORP the same constraint)

13cc592f39 code formatting

8742934843 added spark.sql.shuffle.partitions in the last join phase of the result to community through semantic relation propagation

ed64618235 increased spark.sql.shuffle.partitions in the last join phase of the result (publication) to community through semantic relation propagation

cee7a45b1d [Bulk Tag Datasource] fixed issue with verb name and add new test for neanias selection for orcid

122e75aa17 fixed conflicts

a34c8b6f81 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

b6da9b67ff [BulkTag] fixed typo in annotation for verb name

43aedbdfe5 [BulkTag] changed verb name in configuration

935aa367d8 [BulkTag] removed commented code

c6df8327b3 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

33a2b1b5dc [Bulk Tag] fixed typo in test configuration

29d3da85f1 [EOSC DUMP] added resources needed for the review as test

08c4588d47 Merge pull request 'Changes from beta stats wf to prod' (#264 ) from antonis.lempesis/dnet-hadoop:beta into master

Reviewed-on: #264

6901ac91b1 [Clean Country] moving source and resources to master

fc95a550c3 [Clean Country] moving other resources for testing purposes

059e100ec7 [Clean Country] moving other resources for testing purposes

da85ca697d Merge pull request 'cleanCountryOnMaster' (#265 ) from cleanCountryOnMaster into master

Reviewed-on: #265

46972f8393 [orcid propagation] skip empty directory

5816ded93f code formatting

33bdad104e [cleaning] align parameter names

f910b7379d [cleaning] recovering missing resources from #265

91c70b15a5 updated lines function to it's implementation linesWithSeparators.map(l => l.stripLineEnd) in this way we force scala plugin compiler to consider this pipeline scala code and not java.string.lines() pipeline

11f2b470d3 [Cleaning] adding missing classes

b713132db7 [Cleaning] adding missing classes

7becdaf31d Merge pull request 'Workaround to use new version of intellij on Master' (#266 ) from master_intellij into master

Reviewed-on: #266

c60d3a2b46 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

4953ae5649 fixed an invalid char

d40e20f437 Considering instance pids and alteternative identifiers

f86e19b282 code formatting

699736addc NPE prevention

3800361033 [country propagation] fixes error 'cannot resolve countrySet given input columns: []' when there is no prepared information driving the propagation process for a given result type

c1e2460293 [cleaning] the datasource master-duplicate fixup should not be brought to production yet

1b37516578 [bulk tagging] better node naming

5f0906be60 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

7473093c84 [FoS] changed the default separator from comma to tab to solve the issue in subject value split

7184cc0804 [FoS] added check for null on level1 subject

32870339f5 refactoring after compile

9c1df15071 null values in date range conditions

200098b683 updated the order of the compatibilities

9a9cc6a1dd changed the way the tar archive is build to support renaming in case we need to change .tt.gz into .json.gz

42442ccd39 Merge pull request 'updated the order of the compatibilities' (#275 ) from compatibility_order into master

Reviewed-on: #275

ecc05fe0f3 Added the code for the advancedConstraint implementation during the bulkTagging

c5a9f39141 Extended the association project - result in the mapping from CrossRef

e093f04874 Merge pull request 'AdvancedConstraint' (#285 ) from advConstraintsInBeta into master

Reviewed-on: #285

4f67225fbc Merge pull request 'doiboostMappingExtention' (#286 ) from doiboostMappingExtention into master

Reviewed-on: #286

cf3d0f4f83 fixed issue on bulktagging for the advanced constraints

831055a1fc change of the property for test purposes, addition of two new verbs, and fix of issue for advanced constraints

c6a7602b3e refactoring after compilation

4a4ca634f0 Merge pull request 'advConstraintsInBeta' (#288 ) from advConstraintsInBeta into master

Reviewed-on: #288

2e465915b4 [graph to Solr] using dedicated sparkExecutorCores, sparkExecutorMemory, sparkDriverMemory in convert_to_xml

688e3b7936 added eoscifguidelines in the result view; removed compute statistics statements

087b5a7973 [ZenodiAPIClient] new version of the API to connect to Zenodo (change the http client

24c41806ac [ZenodoApiClienttest] change test to mirror change in the omplementation

9fc8ebe98b refactoring

382f46a8e4 tests to generate the XML records for the index for the EDITH demo on digital twins, integrating output from the FoS classifier

1c173642f0 removed level5 from test records

5ffe82ffd8 aligned to current DMF index layout on production

b88f009d9f combined level 4 and 6 for the demo

04141fe259 tests for records from D4Science catalogues

db625e548d [UsageCount] addition of usagecount for Projects and datasources

654ffcba60 Merge pull request '[UsageCount] addition of usagecount for Projects and datasources' (#296 ) from master_datasource_project_usagecounts into master

Reviewed-on: #296

b64a5eb4a5 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

cae92cf811 update sql query to return distinct pids

5befd93d7d test records for Solr indexing

118e72d7db Updated officialnmae of pangaea in hostedbymap for Datacite to avoid duplicate entries in the source filter of the portal

d9506035e4 [ZenodoApi] gone back to okhttp3 to send the payload.

a92206dab5 re-added the name of a column (pid)

485f9d18cb REmove duplicated code and ensure that load and initialization is done through "DedupConfig.load" method

758e662ab8 Revert "REmove duplicated code and ensure that load and initialization is done through "DedupConfig.load" method"

This reverts commit 485f9d18cb.

e4b27182d0 [master] refactoring

009d7f312f fixed a datasource Id

d029bf0b94 Merge branch 'master' into distinct_pids_from_openorgs

b93e1541aa Merge pull request 'update sql query to return distinct pids' (#301 ) from distinct_pids_from_openorgs into master

Reviewed-on: #301

e10ce92fe5 [stats wf] merged workflows from branch beta

5f32edd9bf adopting dhp-schema:3.17.1

288ec0b7d6 [doiboost] merged workflow from branch beta

4ef0f2ec26 added dependency commons-validator:commons-validator:1.7

f3a85e224b merged from branch beta the bulk tagging (single step, negative constraints), the cleanig worflow (single step, pid type based cleaning), instance level fulltext

ef2dd7a980 resolved conflicts

8621377917 [UsageCount] fixed typo in attribute name for datasource table

8dcd028eed [UsageCount] fixed typo in attribute name for datasource table

aedd279f78 Updates Promotion DBs

- Add a step for promoting the splitted monitor DBs

706092bc19 Update updateProductionViews.sh

8af129b0c7 merged stats promotion step from antonis/promotion-prod-only

373a5f2c83 Merge pull request 'Master branch updates from beta July 2023' (#317 ) from master_july23 into master

Reviewed-on: #317

9e8e39f78a -

59764145bb cherry picked & fixed commit 270df939c4

d512df8612 code formatting

ccac6a7f75 rule out records with NULL dataInfo

da1727f93f rule out records with NULL dataInfo, except for Relations

7180911ded [graph cleaning] fixed regex behaviour for cleaning ROR and GRID identifiers, added tests

0bc74e2000 code formatting

599828ce35 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

da0e9828f7 resolved conflicts for PR#337

265180bfd2 added Archive ouverte UNIGE (ETHZ.UNIGENF, opendoar____::1400) to the Datacite hostedBy_map

77a2199837 updated test for EOSC comunity

dc80ab14d3 [graph dedup] consistency wf should not remove the relations while dispatching the entities

4dc4862011 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

2aed5a74be Run CC and RAM sequentieally in dhp-impact-indicators WF

c2f179800c Merge pull request 'Run CC and RAM sequentieally in dhp-impact-indicators WF' (#338 ) from run_cc_and_ram_sequentially into master

Reviewed-on: #338

19abba8fa7 tests for d4science catalog

d94b9bebf7 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

6186cdc2cc Use v5 of the UNIBI Gold ISSN list in test

1f226d1dce Fix defect #8997 : GenerateEventsJob is generating huge amounts of logs because broker entity similarity calculation consistently failed

4853c19b5e code formatting

7152d47f84 Use asScala to convert java List to Scala Sequence

423ef30676 minor fix on the aggregation of uniprot and pdb

9c3ab11d5b Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop

775c3f704a Fix bug in conversion from dedup json model to Spark Dataset of Rows: list of strings contained the json escaped representation of the value instead of the plain value, this caused instanceTypeMatch failures because of the leading and trailing double quotes

d13bb534f0 Merge branch 'master' into fix_8997

ef02648399 Merge pull request 'fixed dedup configuration management in the Broker workflow' (#341 ) from fix_8997 into master

Reviewed-on: #341

fa692b3629 Merge branch 'master' into fix_dedupfailsonmatchinginstances

4ac06c9e37 Merge pull request 'Fix bug in conversion from dedup json model to Spark Dataset of Rows (instanceTypeMatch no longer working)' (#339 ) from fix_dedupfailsonmatchinginstances into master

Reviewed-on: #339

2c235e82ad Fix cleaning of Pmid where parsing of numbers stopped at first not leading 0' character

ba5475ed4c Merge pull request 'Fix cleaning of Pmid where parsing of numbers stopped at first not leading 0 (zero) character' (#345 ) from fix_truncated_pmid into master

Reviewed-on: #345

af3ffad6c4 [AMF] docs

242d647146 cleanup & docs

a9ede1e989 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

69dac91659 adding the new code to use the API instead of the Information Service

18bfff8af3 adding test classes and modifying test for bulktag

34358afe75 modified resource file, workflow anf default-config. Add 3g of memory Overhead and specified the shuffle partition in the wf confiduration. Removed the multiple instantiation in the wf because of different implementation of the spark job

f206ff42d6 modified code to use the the API. Removing not needed parameters. Rewritten the code to exploit the parallel stream on the entity types

70b78a40c7 removed file from different propagation

5c5a195e97 refactoring and fixing issue on property name

0097f4e64b Removed Query community testing. Removed package from common related to the interaction with Zenodo since it was moved to the dump-project

94089878fd Adjust tests to new WF input params

6ce9b600c1 Add actionset creation for pubmed affiliations

24c3f92d87 Change the description of the workflow

7e34dde774 Renaming input param for crossref input path

97454e9594 Merge pull request '9117_pubmed_affiliations_prod' (#357 ) from 9117_pubmed_affiliations_prod into master

Reviewed-on: #357

8c03c41d5d applying changes from beta

5f1ed61c1f merging from bulkTag branch

671ba8a5a7 Clear working dir in bipranker workflow

b177cd5a0a Project propagation via communityAPI instead of using IS via IIS

2b626815ff Merge pull request 'Project propagation via communityAPI instead of using IS via IIS' (#362 ) from projectPropagation into master

Reviewed-on: #362

8fb05888fd Merge branch 'master' into 9120_bipranker_clean_working_dir

a0311e8a90 Merge pull request 'Clear working dir in bipranker workflow' (#360 ) from 9120_bipranker_clean_working_dir into master

Reviewed-on: #360

1763d377ad code formatting

1726f49790 code formatting

c4ec35b6cd Merge pull request 'Master branch updates from beta December 2023' (#369 ) from beta_to_master_dicember2023 into master

Reviewed-on: #369

1f1a6a5f5f updated the transformation Baseline workflow to include mdstore rollback/commit action

1fbd4325f5 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop

9d39845d1f uploaded input parameters on CreateBaseline WF

37e36baf76 updated workflow for generation of Scholix Datasource's to use mdstore transactions

a8a4db96f0 added metaresourcetype to the result hive DB view

106968adaa Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

3c8c88bdd3 Fixed problem on missing author in crossref Mapping

4d0c59669b merged changes from beta

4c8706efee [orcid-enrichment] change the value of parameters.

1a8b609ed2 code formatting

f28c63d5ef [orcid enrichment] fixed directory cleanup before distcp

e8630a6d03 [graph cleaning] rule out datasources without an officialname

08162902ab [actiosets] introduced support for the PromoteAction strategy

d86b909db2 [actiosets] fixed join type

f21133229a Merge pull request 'Support for the PromoteAction strategy [master]' (#391 ) from promote_actions_join_type_master into master

Reviewed-on: #391

8dd666aedd Dedup aliases, created when a dedup in a previous build has been merged in a new dedup, need to be marked as "deletedbyinference", since they are "merged" in the new dedup

079085286c Merge branch 'master' into fix_dedupaliases_deletedbyinference

ba1a0e7b4f Merge pull request 'Set deletedbyinference =true to dedup aliases, created when a dedup in a previous build has been merged in a new dedup' (#392 ) from fix_dedupaliases_deletedbyinference into master

Reviewed-on: #392

1416f16b35 [graph raw] fixed mapping of the original resource type from the Datacite format

b3ddbaed58 fixed import of ORPs stored on HDFS in the internal graph format (e.g. Datacite)

071d044971 Merge branch 'master' into crossref_missing_author_fix_master

07d009007b Merge pull request 'Fixed problem on missing author in crossref Mapping' (#384 ) from crossref_missing_author_fix_master into master

Reviewed-on: #384

4374d7449e mapping of project PIDs

c94d94035c [BulkTagging] added check to verify if field is present in the pathMap

3067ea390d Use SparkSQL in place of Hive for executing step16-createIndicatorsTables.sql of stats update wf

3bcfc40293 new plugin to collect from a dump of BASE

841ca92246 Merge pull request 'new plugin to collect from a dump of BASE' (#400 ) from base-collector-plugin into master

Reviewed-on: #400

d9b23a76c5 comments

7f7083f53e updated sql query for filtering BASE records

a99942f7cf filter by base types

cd4c3c934d openorgs wf updated

f6601ea7d1 default parameters for openorgs updated

d70793847d resolving conflicts on step16-createIndicatorsTables.sql

09a6d17059 Merge pull request '[Stats wf] #372 , #405 to production' (#406 ) from antonis.lempesis/dnet-hadoop:beta into master

Reviewed-on: #406

d16c15da8d adjusted pom files

5592ccc37a Merge pull request 'added missing EOS, Generate tables with parquet-files, instead of csv in the contexts.sh script' (#408 ) from antonis.lempesis/dnet-hadoop:beta into master

Reviewed-on: #408

f01390702e Merge pull request 'fixed typo in indicator query' (#410 ) from antonis.lempesis/dnet-hadoop:beta into master

Reviewed-on: #410

5add51f38c Merge pull request 'fixed the result_country definition and updated the stats DB copy procedure' (#412 ) from antonis.lempesis/dnet-hadoop:beta into master

Reviewed-on: #412

519db1ddef Extended mapping of funder from crossref (#9169 , #9277 ) and change the correspondece files for the irish fundrs (#9635 ). Extended the datacite map to include the association between metadata and the EBRAINS datasource (SciLake)

6132bd028e Merge pull request 'Extend Crossref-funders mapping and datacite hostedbymap' (#417 ) from CrossrefFundersMap into master

Reviewed-on: #417

013935c593 Merge pull request 'Improvements to copying data from ocean to impala' (#420 ) from antonis.lempesis/dnet-hadoop:beta into master

Reviewed-on: #420

0486cea4c4 removed the funder id : 100011062 Asian Spinal Cord Network, wrongly associated to Ireland

96bddcc921 revised query implementation for indi_pub_gold_oa

62a07b7add added missing end of statement /*EOS*/

43e123c624 added column alias

18fdaaf548 integrating suggestion from #9699 to improve the result_country table construction

8fdd0244ad Merge pull request 'Various fixes for the stats DB update workflow, step16-createIndicatorsTables.sql' (#425 ) from stats_step16_fix into master

Reviewed-on: #425

aecf3b4f2e [WebCrawl] first implementation

48ed49055e [WebCrawl] filtering out all the results published before 2020 not associated to IE

c987beb723 [WebCrawl] resolving conflicts

miriam.baglioni closed this pull request

2024-04-22 10:44:01 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#427