slight optimization in indi_pub_gold_oa definition #423

antonis.lempesis · 2024-04-17T22:55:18+02:00

antonis.lempesis commented

2024-04-17 22:55:18 +02:00

No description provided.

antonis.lempesis added 1 commit 2024-04-17 22:55:18 +02:00

1f5aba12fa slight optimization in indi_pub_gold_oa definition

antonis.lempesis added 1 commit 2024-04-17 22:59:59 +02:00

27d22bd8f9 slight optimization in indi_pub_gold_oa definition

antonis.lempesis added 2 commits 2024-04-18 10:08:11 +02:00

308ae580a9 slight optimization in indi_pub_gold_oa definition

e728a0897c fixed the definition of indi_pub_bronze_oa

antonis.lempesis added 1 commit 2024-04-18 10:53:58 +02:00

43d05dbebb fixed the definition of result_country

antonis.lempesis added 1 commit 2024-04-18 11:01:34 +02:00

0c71c58df6 fixed the definition of gold_oa

antonis.lempesis added 1 commit 2024-04-19 11:45:52 +02:00

c3fe9662b2 all indicator tables are now stored as parquet

antonis.lempesis added 172 commits 2024-04-19 14:29:51 +02:00

55ea485783 [UsageCount] split the count for result at the level of the datasource. for each indicator one unit is specified for each datasource contrinuting to that indicator value. The datasource key is the value of the key element in the unit for the measure, while the count for that datasource is in the value

4c9bc4c3a5 refactoring

b00771c7cc [Transformative Agreement] added code to extract relations from the transformative agreement file for the IE products got from OpenAPC

354e02e6a9 [Transformative Agreement] removed not needed class. Read directly the json and no need to pass from the csv

624f5f3f21 [Transformative Agreement] added check to verify the APC were paid byu the IReL funder

1a42a5c10d Implemented Download update of ORCID

8f61063201 Added workflow

9d5a7c3b22 code refactor

39ebb60b38 Merge remote-tracking branch 'origin/beta' into orcid_update

859babf722 added some useful comment

e328bc0ade fixed missing parameter on download update

e0753f19da Fixed error of connection timeout

e0ec800d7e [BulkTagging] extend the definition of the pathMap to include also actions that should be performed of the value extracted from the result befor applying the constraint

6e58d79623 mergin with branch beta

f7d06dc661 compilation after merging

43e0bba7ed logg added during download

0386f36385 Added workflow to update ORCID and replaced some parsing, because the update works and employments xml differs from the dump one.

9aebca77a0 Added exception throwing in Hadoop transformation when TR is not syntactically valid

e9131f4e4a mergin with branch beta

a418dacb47 [UsageCount] code extention to include also the name of the datasource

a5995ab557 [orcid-enrichment] change the value of parameters.

ead08b0dd4 mergin with branch beta

07a373a0bd [bulkTagging] removing checks while performing the substring action so that it will fire an Exception if the paramneters are wrongly set

f2a08d8cc2 test for Italian records from IRS repositories

42f5506306 [orcid enrichment] fixed directory cleanup before distcp

bb82052c40 [graph cleaning] rule out datasources without an officialname

009dcf6aea [actiosets] introduced support for the PromoteAction strategy

fd17c1f17c [actiosets] fixed join type

38c9001147 fixed import of ORPs stored on HDFS in the internal graph format (e.g. Datacite)

e6bdee86d1 Merge pull request 'Support for the PromoteAction strategy' (#389 ) from promote_actions_join_type into beta

Reviewed-on: #389

b19643f6eb Dedup aliases, created when a dedup in a previous build has been merged in a new dedup, need to be marked as "deletedbyinference", since they are "merged" in the new dedup

d85d2df6ad [graph raw] fixed mapping of the original resource type from the Datacite format

bdb6bbb365 mergin with branch beta

eca021f4d6 [Transformative Agreement] add results with information abount the agreement and the country of the organization paid for it

3f7d262a4e mergin with branch beta

6e1f383e4a [Tagging Projects and Datasource] first extention of bulktagging to add the context to projects and datasource

83bb97be83 [Tagging Projects and Datasource] added test to check datasource tagging. Fixed issue

8dae10b442 -

29194472a7 Promote "Research" to a jolly instanceType in dedup comparisons

Compare Part of book or chapter of book with Article

d65285da7f Promote "Research" to a jolly instanceType in dedup comparisons

Compare "Journal" and "Part of book or chapter of book" with "Article"

85aeff72f1 Merge pull request 'Revised instance type comparisons in dedup phase' (#393 ) from revisedInstanceType into beta

Reviewed-on: #393

a63b091bae Merge branch 'beta' into import_orps_fix

753c2a72bd Merge pull request 'fix import of ORPs' (#390 ) from import_orps_fix into beta

Reviewed-on: #390

f0dc12634b Add Action Set creation for affiliations inferred from the OpenAPC data

43da7e1191 [Tagging Projects and Datasource] changed the way the pathMap parameter is passed. It was too long and was truncated

72bae7af76 [Transformative Agreement] removed the relations from the ActionSet waiting to have the gree light from Ioanna

3268570b2c mapping of project PIDs

b32a9d1994 Implemented workflow for updating table , added step to check if the new generated table is valid

a712df1e1d Merge remote-tracking branch 'origin/beta' into orcid_update

773e856550 Revised procedure when converting json data into xml:

- json object keys are renamed to be conformant to xml tag elements, special characters are substituted or removed
- json string values are no longer post-processed as they are already escaped by the org.json.XML.toString method

915a76a796 following the comment on the pull requests:

- Added #NUM_OF_THREADS complete job in the queue at the end of  the main loop to avoid deadlock

b63994dcc4 Merge remote-tracking branch 'origin/beta' into orcid_update

e468e99100 Merge pull request 'Orcid Update Procedure' (#394 ) from orcid_update into beta

Reviewed-on: #394

7d806a434c formatted code

3f07390a58 WIP

6fcf872daa Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into index_records

56dd05f85c Merge pull request 'Revised procedure when converting json data into xml' (#395 ) from restiterator_xmlcleanup into beta

Reviewed-on: #395

3cd5590f3b When converting json to XML, remove characters that are not allowed in the XML 1.0 specs, as they will cause xpath failures even if escaped

9d94648f3b code formatted

ad0e9aa80c added first part of refactoring of the code generating MAG,

make it more readable using spark sql queries

f417515e43 Implemented class that generates a normalized table of MAG, which is the starting point for the creation of the mag source

3b837d38ce added oozie workflow

d34cef3f8d Merge remote-tracking branch 'origin/beta' into doidoost_dismiss

fd34372c40 [OCNEW] first implementation

7827a2d66b [OCNEW] added creation of the actionset for the results classified with FoS based ont he OpenAIRE identifier

5180b6ec8a [FOSNEW] removed test class

d4871b31e8 WIP: extended provision workflow to create the JSON based payload

cbd4e5e4bb update mag mapping

9092075760 Enrich authors with ORCID info using new matching algorithm

db66555ebb WIP: updated provision workflow to create a JSON based representation of the payload

48c052215c [OC New] last fix

c532831718 Moved Crossref Mapping on dhp-aggregations,

refactored code, avoid to use utility for create part of the oaf defined in DOIBoostMappingUtils, used instead utility in OafMappingUtils

5a32bb9578 [OC New] last fix

ee1fcb672b code refactor

5281f010a5 applied cherry pick

eb5887cb9a including related organization url in the XML record serialization (ticket #9498 )

7863c92466 expanded paper abstract in the result/children XML element (ticket #9497 )

af154d4456 implemented changes from #9497 : sort abstracts by string length, included author fullnames in the related results, expanded instance details within each children/result XML element

078169b922 cleanup

88fef367b9 new plugin to collect from a dump of BASE

30167aa882 mapped oaf:country from results

455f2e1e07 apply commits from master

85b844d57e updated BASE filter param

cb29b9773c xslt rules

664a381d31 Unify merge logic of entities in MergeUtils.class

58dbe71d39 update crossref mapping to be runnable separately as a single datasource outside doiboost

7ae7e8aa06 Merge pull request 'Unify merge logic of entities in MergeUtils.class' (#370 ) from mergeutils into beta

Reviewed-on: #370

f9c74c98fa fixed an identifier xpath

7faa115ba0 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

0ff7faad72 Fix conditions that prevented ORCID Enrichment

aaa73f89d1 refactoring the Oaf records merge utilities into dhp-common

c8683eb13c Merge branch 'beta' into mergeutils

3f22c101d9 Merge pull request 'Enrich authors with ORCID info using new matching algorithm' (#398 ) from new_orcid_enhancement into beta

Reviewed-on: #398

6ffb1faf09 fixed a problem with multiple nodes

3f174ad90f Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

e1149eb5c4 xslt rules and tests

9fc70a9451 implemented default merge procedure applied to result.instance

54936b7f42 Merge branch 'beta' into transformativeagreement

e0c315b07b Merge pull request 'Extract Information from Transformative Agreement' (#371 ) from transformativeagreement into beta

Reviewed-on: #371

63067d4b24 align dhp-schema.version with the beta branch

91b61687fa Merge branch 'beta' into bulkTaggingPathMapExtention

c3c9bdb59c Merge pull request 'bulkTaggingPathMapExtention' (#381 ) from bulkTaggingPathMapExtention into beta

Reviewed-on: #381

71c1f81b54 Merge branch 'beta' into exception_on_invalid_transofmation_rule

069803f34a Merge pull request 'Added exception throwing in Hadoop transformation when TR is not syntactically valid' (#387 ) from exception_on_invalid_transofmation_rule into beta

Reviewed-on: #387

9a5b134ddf Merge branch 'beta' into FOSNew

e9eb590f87 Merge pull request 'FOS ActionSet for the classification of results without a doi' (#397 ) from FOSNew into beta

Reviewed-on: #397

6c3b692f60 integrated minor change from beta branch

74e5d05577 Merge branch 'beta' into ocnew

fa4b3e6d2b Merge pull request 'Open Citation integration' (#401 ) from ocnew into beta

Reviewed-on: #401

4b978ffa2d align dhp-schema.version with the beta branch

82fc609c4f Merge branch 'beta' into index_records

eae88c0fe3 Merge pull request 'Solr JSON payload' (#399 ) from index_records into beta

Reviewed-on: #399

538b180fe0 Merge branch 'beta' into oaf_country_beta

25c2025223 Merge pull request 'mapped oaf:country from results' (#403 ) from oaf_country_beta into beta

Reviewed-on: #403

ecff0b4825 merge from beta

414acd4ef4 Merge pull request 'refactoring the Oaf records merge utilities into dhp-common' (#404 ) from mergeutils into beta

Reviewed-on: #404

ece56f0178 update crossref mapping to be transformed together with UnpayWall

d72e7b7487 Merge pull request 'Changes to indicators and funders definition' (#372 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #372

bfba71a95c further follow up changes from integrating the mergeutils branch

ef52128c55 included new stats* workflows in parent pom list of modules, code formatting

730eaffc85 Merge pull request 'correctly selecting the active hdfs node for the impala cluster' (#405 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #405

3b209261f2 [BulkTagging - tag datasource and projects]merging with branch beta

94b931f7bd [BulkTagging - tag datasource and projects]merging with branch beta

75551ad4ec code formatting

9e700a8b0d Merge pull request 'adding context information to projects and datasources' (#407 ) from taggingProjects into beta

Reviewed-on: #407

73a67c0e4a Improved Crossref mapping to include also unpaywall tested

cff6040424 Merge pull request '[BETA] added missing EOS, Generate tables with parquet-files, instead of csv in the contexts.sh script' (#409 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #409

24227ab598 Merge pull request '[BETA] fixed typo in indicator query' (#411 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #411

0eb0701b26 Add action set creation for Datacite affiliations

64cbd8abe9 Merge pull request '[UsageCount] Usage count per result split by datasource' (#318 ) from UsageStatsRecordDS into beta

Reviewed-on: #318

b42bdd5fb3 [UsageCount] add check in case the datasource is not matched against those present in the graph

2c4440951f Merge pull request '[UsageCount] add check in case the datasource is not matched against those present in the graph' (#414 ) from UsageStatsRecordDS into beta

Reviewed-on: #414

9c9a9562ae [UsageCount] fixed error

cbe13a5c61 Fix datacite input path in properties file

4bb504e693 Merge pull request '[UsageCount] fixed error' (#415 ) from UsageStatsRecordDS into beta

Reviewed-on: #415

4f0a044245 Merge pull request 'Add action set creation for Datacite affiliations' (#413 ) from 9647_datacite_affiliations into beta

Reviewed-on: #413

42846d3b91 [OpenCitation] add compression option when writing the sequence file

02c9a311c8 base datainfo with trust=0.89

71d6e02886 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

50fbebf186 [NOAMI] removed entry for Health and Social Care Board from the list of funders. Modified IRC putting 1596 and 1597 as synonyms, as required in ticket 9635

26b97aa5ed Merge pull request '[BETA] fixed the result_country definition and updated the stats DB copy procedure' (#416 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #416

f0f6abf892 [MapToFunderLink]added references for HFRI and Erasmus+ for the creation of links for funders

6f3e925cae Implemented first part of the new MAG mapping

31e152d2bb Merge remote-tracking branch 'origin/doidoost_dismiss' into doidoost_dismiss

c8a88b2187 [DataciteHostedByMap] added entry for EBRAINS

4b1de076ac [DataciteHostedByMap] added entry for EBRAINS

0794e0667b Merge branch 'doidoost_dismiss' of https://code-repo.d4science.org/D-Net/dnet-hadoop into doidoost_dismiss

5142f462b5 completed mapping from paper to OAF, not tested

ef582948a7 Updated mapping

98dc042db5 mapping generated for MAG,

missing generation of Organization Action set

a0642bd190 added instanceTypeMapping field on MAG

3cad4a415d fixed duplicated property dhp-schemas.version

2581672c11 updated wf of MAG and crossref to use transaction

1e30454ee0 added vocabulary tu instanceTypeMApping of Mag

843dc95340 resolved conflict

41a42dde64 code formatted

43b454399f - Bug fix in matchOrderedTokenAndAbbreviations algorithms where tokens with same initial character were always considered equal

- AuthorsMatch exploits the new matching strategy used for ORCID enhancements in #PR398: split author names in tokens, order the tokens, then check for matches of ordered full tokens or abbreviations

78b9d84e4a test

d070db4a32 added a couple more invalid author names

43fd1de681 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

da333e9f4d Merge pull request 'Enhance Dedup authors matching with algorithms used for ORCID enhancements (task 9690)' (#419 ) from dedup_authorsmatch_bytoken into beta