8232-mdstore-synch-improve #272

sandro.labruzzo · 2023-02-13T09:32:37+01:00

sandro.labruzzo commented

2023-02-13 09:32:37 +01:00

This is the implementation described on ticket 8232
The Mongo MDStores are already synchronized on base path:

/data/mongo_mdstores

The only change in the workflow parameter which create the raw graph is to remove the two sequential file path of the two mdstores MDFormat and include:

/data/mongo_mdstores/PROD/*/*
in case of production

/data/mongo_mdstores/BETA/*/*
in case of BETA

This is the implementation described on ticket [8232](https://support.openaire.eu/issues/8232) The Mongo MDStores are already synchronized on base path: `/data/mongo_mdstores` The only change in the workflow parameter which create the raw graph is to remove the two sequential file path of the two mdstores MDFormat and include: `/data/mongo_mdstores/PROD/*/*` in case of production `/data/mongo_mdstores/BETA/*/*` in case of BETA

claudio.atzori was assigned by sandro.labruzzo

2023-02-13 09:32:37 +01:00

sandro.labruzzo added 747 commits 2023-02-13 09:32:38 +01:00

1699d41d39 relations for openorgs: not it choose only one master

3f2d3253e4 Merge branch 'stable_ids' into deduptesting

a45b95ccc1 resolving conflicts for PR#134

d64a942a76 fixed MappersTest

370dddb2fa fix bug on oai iterator that skip record cleaned

ec94cc9b93 IndexNotificationsJob test: persist contents on HDFS instead of passing them to ES

b01cd521b0 removed configuration specifying the limit to 8 for spark.dynamicAllocation.maxExecutors

31a6ad1d79 optimization of verifySubsriptions()

0a9ef34b56 test

8bbaa17335 reimplemented of conditions cache as a non static variable

69008e20c2 log and tests

210d6c0e6d generateNotificationsJob and indexNotificationsJob

d6e1f22408 max numbers of workers for indexing

83c90c7180 manually merging PR#149 #149

2f61054cd1 code formatting

8db39c86e2 added new parameter in the doiboost process workflow to specify a folder for the process of MAG dataset

4b1920f008 changed the working path parameter value as dependant from the dnet-workflow working dir parameter

e15a1969a5 applying fix on the DOIBoost construction process that somehow wasn't part of the merge done in 83c90c7180

e471f12d5e hotfix: recovered implementation removing the hardcoded working_dirs

bdffa86c2f undo last commit

172363e7f1 [broker] integrating PR#147, notification record creation phase separated from indexing on ES

c4fce785ab fixed a compilation problem of a unit test

652114c641 [affiliationPropagation] first try. preparetion

43f4eb492b fetching affiliated results for 4 orgs in monitor. fixed affiliated orgs in stats db

a7376907c2 invalidating medatadata before context thingies

cece432adc [stats] reducing the step22 wait time

d66e20e7ac added hierarchy rel in ROR actionset

d0ef7d91c5 adding test resource

d02caef185 Merge branch 'beta' into hierarchical_orgs_relations

09f36cffb8 [Enrichment: Propagation through parent-child relationships] First implementation, testing, and wf for propagation of result to organization through semantic relation

2aca6bfa0a mergin with branch beta

b97b78f874 removed hardcoded reference

b9d124bb7c [Enrichment: Propagation through parent-child relationships] Added counters, and changed constraint to verify if filtering out the relation (from classname = harvested to classid != propagation)

0857849a86 [Graph DUMP] Remove dump of measure until it will be clear where to put it (at the level of result or at the level of the instance)

c10ff6928c [Graph DUMP] add schema of the dump related to the model as in dhp-schemas.2.8.31. Note the measere element at the level of the result has been removed because of issues on where to display it: at the level of the result or at the level of the entity

a22c29fba1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

5681e89544 Update 'dhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/dump/schemas/result_schema.json'

8442efd8d1 [Graph DUMP] Filtering out from the originalIds the id of the result in OpenAIRE

4c70201412 mergin with branch beta

9cb8e4ad21 Merge branch 'beta' into hierarchical_orgs_relations

94918a673c [Graph DUMP] Fix issue for empty origilaId list

b9d4f115cc fixed Crossref mappign for SFI projects

6cd91004e3 fixed DOI for Wellcome Trust in mapping relationships from Crossref

fc8fceaac3 create direct link to WT projects as well

88b73f4f49 mergin with branch beta

8cc50ecee0 [Graph Dump] changed AccessRight with BestAccessRight in the dump and modified the dependency to the schema to the SNAPSHOT

e5498052e8 [Graph DUmp] removed OpenAccessRoute from test in best access right

b8bdabfae9 [Graph DUmp] removed OpenAccessRoute from test in best access right

b3f9370125 merge with beta - resolved conflict in pom

43cae4ad88 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

6595135a1a [Dump Schemas] changed the schema of the dumped result according to the modifications in the bestAccessRight type

99d86134f5 [Graph Dump] changed the dump since the measures have been moded at the level of the instance

c6a9f0a1a8 mergin with branch beta

c7c0c3187b [AFFILIATION PROPAGATION] Applied some SonarLint suggestions

7c96e3fd46 removed not useful dir

a1cafaf2e3 added mvn site for dnet-hadoop project

18c1d70ef4 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into mvn_site_documentation

28ea532ece [Affilaition Propagation] moved the selection of graph relation as a preparation step

2d67020c59 added dhp-enrichment maven site template

2bbece2ca5 mergin with branch beta

ec8b0219ff [Documentation] Added first page for Integration via unresolved entities generation

4094f2bb9a added integration md file

cded363b55 code refactor, created and moved scala code on the correct maven folder under src/main/scala and src/test/scala

2506d7a679 Merge branch 'mvn_site_documentation' of code-repo.d4science.org:D-Net/dnet-hadoop into mvn_site_documentation

60ae874dcb Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into mvn_site_documentation

2fd9ceac13 code refactor, created and moved scala code on the correct maven folder under src/main/scala and src/test/scala

1f5ee116ed code refactor, created and moved scala code on the correct maven folder under src/main/scala and src/test/scala

fixed test

9c82d670b8 make class public in order to create javadoc

10a32f287f Merge pull request '[stats wf] RIs, affiliations, parquet' (#156 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #156

1b79c0ee79 mergin with branch beta

0136a8c266 [Graph Dump] Change test to mirror that measure is at the level of the isntance

5dc5792722 [Graph Dump] Change test resource to mirror the movement of the measure element

793b5a8e5f Aggiornare 'dhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/dump/ResultMapper.java'

Removing the dump of Measure at the level of the result. We decided not to map it

bb5dca7979 cleanup

c0750fb17c avoid non necessary count operations over large spark datasets

a24b9f8268 [dedup] trivial refactoring

3974fa7dc1 Merge branch 'beta' into affiliationPropagation

3a4d925386 Merge branch 'beta' into hierarchical_orgs_relations

bd9a43cefd Revert to 4094f2bb9a

62fa61f3cf merge from beta

0c0d561bc4 added public class into tests to create correct javadoc

8a7c7d36db Merge pull request 'mvn_site_documentation' (#161 ) from mvn_site_documentation into beta

Reviewed-on: #161

fc03c99805 fixed javadocs url after deploying site

9fae872181 [Graph Dump] changed to mirror the changes in the model

2b46b87f56 fixed filtering criteria applied in SparkCopyRelationsNoOpenorgs to keep the parent/child relations from OpenOrgs

f4538f3c4c cleanup

e5a2c596b2 Merge branch 'beta' into preserve_openorg_parent_child_relations

90c2a4987e Merge pull request '[fix] preserve parent/child relations from OpenOrgs' (#164 ) from preserve_openorg_parent_child_relations into beta

Reviewed-on: #164

4542a2338b updated site configuration to deploy on website

75298ec442 added site.xml to code style

3426451d3f Merge remote-tracking branch 'origin/beta' into beta

155d8bf83f updated maven site plugin on dhp-code-style

65ebe1019b updated wagon-ssh version

6110a2b984 reverted version

0506fa2654 [Graph Dump] changed to mirror the changes in the model

fdb75b180e [Cleaning] added couple of tests for DOIBOOST publications

35e20b0647 updated resolution wf:

- generate a new version of the graph
 - changed merge from union to join

93fe8ce8b2 entity resolution: fix test

483d3039d1 entity resolution: added distcpt of missing entities in graph materialization

a7cf277d98 Datacite: Removed HostedBy Patch as described on ticket #7219 , Now all the records will have hosted by Unknown Repository

3f9b2ba8ce [Hosted By Map] fix issue in test

2164a2a889 Datacite: Code Refactor generated a general SparkApplication Scala where all the spark scala have to inherit

Commented a little the Datacite transformation code

028a8acad8 add test resources

feea154e89 remove working dir after test

5fd0e610bf [DOIBOOST Process] fix filtering to filter results with non null id

ac07ed8251 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

1e1f5e4fe0 minor fix

bb7f556eff Merge remote-tracking branch 'origin/beta' into beta

5c6d328537 code formatting

014e872ae1 [resolution wf] added optional parameter to skip the entity resolution

1de881b796 resolved conflicts for #165

0df9574a6f Merge pull request '[stats wf] Added sprint 3&4 of indicators' (#166 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #166

01e5e0142a added test to verify the relation inverse lookup operation

4fe7888817 code formatting

d85af6fc25 [cleaning wf] fixed OAF record navigation, a mapping defined on a container object would have prevented the natvigation to continue on its properties

cfa4560769 minor: fixed hive action name

3b19821f3c added stats computation on the graph hive DB tables

87eedad898 -

8905a39bf3 mergin with branch beta

58bc3f223a [GRAPH DUMP] Add filtering for relation we do not want to dump. It is based on the relclass

d9f80488cc [GRAPH DUMP] Add one more test to check the filtering of the relations

9cac283bec implemented Instance serialization features requested in https://support.openaire.eu/issues/7156

863a2f9db3 avoid to filter OAF records defined as invisible = true

c4c705aa46 Merge pull request 'Cleaning of invisible records' (#168 ) from clean_invisible_records into beta

Reviewed-on: #168

dd0b2e5244 Merge branch 'beta' into instance_group_by_url

372633880f Merge pull request 'XML serialisation of instances with the same URLs' (#167 ) from instance_group_by_url into beta

Reviewed-on: #167

f7011b90d8 format code

0fa0ce33d6 removed duplicated on gitignore

4bb1d43afc -

f430688ff7 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

08795cbd30 using helper method from ModelSupport to find the inverse relation descriptor

9132727793 fixed date cleaning test

7af0bbd0b1 [scala-refactor] Module dhp-aggregation:

Moved all scala source into src/main/scala and src/test/scala

81bf604059 [scala-refactor] Module dhp-common:

Moved all scala source into src/main/scala and src/test/scala

bf880e2508 [scala-refactor] Module dhp-graph-mapper:

Moved all scala source into src/main/scala and src/test/scala

e9f285ec4d [scala-refactor] Module dhp-doiboost:

Moved all scala source into src/main/scala and src/test/scala

96a7d46278 [Graph Dump] fixed tests

ed0c352799 [test-fixing] fixed wrong test

d1df01ff1e [Graph Dump] fixed resource for test

d9836f0cf3 [OpenCitations] fixed test when executed one after the other

5d51b3dd4a Merge pull request 'scala_refactor' (#169 ) from scala_refactor into beta

Reviewed-on: #169

b113586207 resolved conflicts

6b5d7688a4 #7275 serialize license information in XML records

e6e177dda0 vocabulary based cleaning considers also the term label when looking up for a synonym

adf17452b0 Merge pull request '[graph cleaning] consider terms as synonyms in the vocabulary lookup' (#170 ) from graph_cleaning into beta

Reviewed-on: #170

cd9c51fd7a vocabulary based cleaning considers also the term label when looking up for a synonym

e53228401b style

cba63e9f82 Merge branch 'beta' into sygma_indexing

8f1e018ceb Merge pull request 'Serialization of fields in XML records for Sygma (and not only)' (#171 ) from sygma_indexing into beta

Reviewed-on: #171

41c70c607d cleaning workflow assigns the proper default instance type when a value could not be cleaned using the vocabularies

eb43eda42a Merge branch 'beta' into graph_cleaning

25dc7929a9 Merge pull request '[graph cleaning] improved instance type defaults' (#172 ) from graph_cleaning into beta

Reviewed-on: #172

c1b6ae47cd cleaning workflow assigns the proper default instance type when a value could not be cleaned using the vocabularies

b70ecccea0 avoid NPEs merging XMLInstance(s)

5e17247bb6 avoid NPEs merging XMLInstance(s)

98eb292c59 avoid NPEs merging XMLInstance(s)

8d755cca80 -

936578aaf1 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

4eb8276493 -

573bd17cbb Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

aff3ddc8d2 added cleaning for the format field, removing carrige return and tab characters

6fb6236cd4 changed the way to produce the AS for bipFinder.

22d4b5619b [BipFinder Result] last changes to test and resources files

a3592b463a Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

56409d1281 [Dump] resolved conflicts with beta and merging

1790fa2d44 Merge branch 'beta' into affiliationPropagation

e30e5ac8a8 Merge pull request '[Affiliation Propagation]' (#162 ) from affiliationPropagation into beta

Reviewed-on: #162

e5bff64f2e [scholexplorer]

- Minor fix on SparkConvertRDDtoDataset
-first implementation of retrieve datacite dump

63952018c0 [scholexplorer]

-moved SparkRetrieveDataciteDelta in scala folder

b881ee5ef8 [scholexplorer]

- implemented generation of scholix of delta update of datacite

f0b523cfa7 removed the too restrctive clause. will discuss again

63b648b0dd Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

5e5dfd619c Merge branch 'beta' into dump

3cc1b7b153 mergin with branch beta

3920d68992 Fixed workflow generation of delta in datacite

460e6b95d6 [Graph Dump] -

d1ae219cb4 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

e24a7f3496 mergin with branch beta

89ea9fa0e1 Merge branch 'dump' of https://code-repo.d4science.org/D-Net/dnet-hadoop into dump

be0acccf42 Merge branch 'beta' into dump

9d19b057b8 Merge pull request '[GRAPH DUMP]Moving Measures' (#159 ) from dump into beta

Reviewed-on: #159

8d18500069 using dhp-schema:2.9.24

4849270c55 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

da5f6260aa mergin with branch beta

c5739c4266 [BipFinder] create action set for the measures at the level of the result

b5e11a3a0a [BipFinder] put in common package BipFinder model

0807fdb65a [BipFinder] remove not needed resources

bf52a1847b Merge pull request 'bipFinder at the level of the result' (#173 ) from bipFinder into beta

Reviewed-on: #173

2c126ed014 [BipFinder] create unresolved entities with measures at the level of the instance

813f856d3f [BipFinder] removing left over parameter in wf

20ef1d657f refactoring

34ac56565d refactoring

b352fbe453 Merge pull request 'bipFinder: unresolved entities' (#174 ) from bipFinder into beta

Reviewed-on: #174

de6c4c8968 [FOS]creation of the unresolved entities: remove the split for the doi: no more needed since each row is related to one doi

73175ba086 mergin with branch beta

b81efb6a9d [FOS]changed the mapping between the csv and the model. Changed Test classes and resources

6116fc5d40 [FOS]added logic to include only different subjects. Test refactoring and extention

10579c0dd0 [FOS]fixed doi value in test

bf3a9505e0 Merge pull request 'FOS' (#175 ) from FOS into beta

Reviewed-on: #175

31b26d48ac [Graph Dump] fixed issue on extraction of relation between entities and contexts: the relationship name and type were swapped

69e9ea9eeb [Graph Dump] Test for extraction of rels from entities extended

5c4fee3533 Merge pull request '[Graph Dump] fixed issue on extraction of relation between entities and contexts: the relationship name and type were swapped' (#176 ) from dump into beta

Reviewed-on: #176

2a67ee13ec [SDG] added model class

cccb16900c https://support.openaire.eu/issues/7330 normalising DOI urls

278cf08421 Merge pull request 'Normalising DOI urls' (#177 ) from instance_group_by_url into beta

Reviewed-on: #177

7a1b440413 [SDG] logic to create unresolved entities out of SDG input. This changes also some classes related to FOS to reuse the same code. The code under createunresolvedentities create results with the merged update of the the inputs provided (bip at the level of the isntance, fos and sdg for subjects)

a706ba0c08 Merge pull request 'SDG Integration' (#178 ) from SDG into beta

Reviewed-on: #178

bcea4e3a9b added dnet workflow profile for the orchestration of the simplified and complete graph construction and processing pipeline, where the IIS works on the non-deduplicated graph

fe7e5f4748 Merge pull request '[stats wf] result_result relations, usage stats, monitor views, indicator for sprint 5' (#179 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #179

92fd69e25d [SDG-FOS] alternative way to get input data to avoid OOM error while getting csv

4c60ee1718 mergin with branch beta

a6977197b3 serialise records in the OAF-store-graph mdstores in json format. Read them again in the graph construction phase using a tolerant parser to support backward compatible changes in the evolution of the schema

bd59b58efb test for the tolerant deserialisation utility method

cb30770a0b Merge pull request 'tolerant parsing of OAF-store-graph mdstores' (#180 ) from graph_interpretation_mdstores into beta

Reviewed-on: #180

045d767013 OAF-store-graph mdstores: save them in text format

174c3037e1 OAF-store-graph mdstores: save them in text format

58f8998e3d OAF-store-graph mdstores: save them in text format

9458ee7938 serialise records in the OAF-store-graph mdstores in json format. Read them again in the graph construction phase using a tolerant parser to support backward compatible changes in the evolution of the schema

8d13effa31 test for the tolerant deserialisation utility method

f82db765db OAF-store-graph mdstores: save them in text format

3dc48c7ab5 OAF-store-graph mdstores: save them in text format

3bd3653be9 OAF-store-graph mdstores: save them in text format

908294d86e OAF-store-graph mdstores: firther fix for PR#180

8ae46ca789 OAF-store-graph mdstores: firther fix for PR#180

adccc2346a [SDG-FOS] to lower case for the doi

639190370a mergin with branch beta

af8a33638d Merge pull request 'SDG - FOS' (#181 ) from SDG into beta

Reviewed-on: #181

b7e450070b [SDG-FOS] to import SDG file not considering the header

93f26fb742 Merge pull request '[SDG-FOS] to import SDG file not considering the header' (#182 ) from SDG into beta

Reviewed-on: #182

064f9bbd87 [AFFPropSR] added new paprameter for the number of iterations and new code for just one iteration

904e1c2667 Merge pull request 'Affiliation Propagation through semantic relation' (#183 ) from enrichment into beta

Reviewed-on: #183

0163dadb7f [doiboost]

- update MAG schema, new filed added on version dec-2021

4f212652ca scalafmt: code formatting

dcd282977c pulled from beta

0f2144b5e0 scalafmt: code formatting

b78d2b71f0 updated scala format configuration

57e2c4b749 formatted code

1b9e8378b3 Merge pull request 'scalafmt: code style for scala' (#184 ) from scalafmt into beta

Reviewed-on: #184

a30a98a716 Applying PR#166 in the master branch (Added sprint 3&4 of indicators). Merge commit '0df9574a6f5d9d75bc840decb023561ae941f9d6'

bb17e070d8 added result_result relations

a4316bafed fixed a typo

4878d7485c added usage stats

3ce1976627 fixed column names

e91f06f39b fixed typos in indicators. Added extra views in monitor

b6b4bc0df9 added first indicator of sprint 5

b053b0178e Sprint 5 and other changes

9acc32faa6 [stats wf] final touches for the integration of PRs #166 , #179 in the master branch

4993666d73 [BipFinderInstanceLevel] changed creation of the instance to allow to enrich existing instances with same pid

dbd6fa1d65 scalafmt: remote referencing the common definition files makes it work compiling the entire project as well as the individual submodules

e7d5a39c03 [BipFinderInstanceLevel] added tests in test class

4d517ed9ec mergin with branch beta

a75fb8c47a [BipFinderInstanceLevel] change pom to align to the dhp-schema release 2.10.24 and refactoring

7bf12ad24a Merge pull request 'BipInstance' (#185 ) from BipInstance into beta

Reviewed-on: #185

a7c4d0d16d [DoiBoost Organizations] added parameter to specify the action in the wf raw_organizations to be able to load the openorgs organization as in the loading step for the construction of the graph

42e8f76778 [GraphCleaning] change the return value in the filtering function to avoid to lose the APC entities

44a937f4ed factored out entity grouping implementation, extended to consider results from delegated authorities rather than identical records from other sources

62f135262e code formatting

391aa1373b added unit test

abfa9c6045 code formatting

3b9020c1b7 added unit test for the DispatchEntitiesJob

f2fde5566b using helper method from ModelSupport to find the inverse relation descriptor

b37bc277c4 reintroduced the hostedby patching to the datacite records

f0ea2410e5 improved mapping titles from datacite records to consider title types

7a3819144d Merge pull request 'title types from datacite records' (#188 ) from datacite_title_mapping into beta

Reviewed-on: #188

4983d6536d Merge branch 'beta' into delegated_authorities

dd52bf1bb8 copy relations to the graphOutputPath

2f385b3ac6 updated dnet workflow profile definitions

c42623f006 added NPE checks

8de9788308 applied fix for avoiding ruling out the invisible (APC) records during the graph cleaning

97ad94d7d9 [graph resolution] drop output path at the beginning

59a250337c [graph resolution] drop output path at the beginning

1322379741 Merge branch 'beta' into delegated_authorities

a70b0990c9 Merge pull request 'Priority to records from delegated authorities' (#187 ) from delegated_authorities into beta

Reviewed-on: #187

4fc44edb71 depending on dhp-schemas:2.10.26

af61e44acc ported changes to the GraphCleaningFunctionsTest from 8de9788308

db299dd8ab fixed typo

8eb75ca169 adapted GenerateEntitiesApplicationTest behaviour

dce7f5fea8 [BULK TAGGING] changed to fix issue that should have been fixed already

73eba34d42 [UnresolvedEntities] Changed the way to merge the unresolved because the new merge removed the dataInfo from the merged result. Added also data info for subjects

37784209c9 [dhp-schemas-] updated the version of dhp-schema to 2.10.27 for APC name and id modification

2e215abfa8 test for instances with URLs for OpenAPC

89922156c9 Merge pull request '[stats-wf]fixed the result_result creation for monitor' (#190 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #190

3c60e53a96 [stats-wf]fixed the result_result creation for monitor PR#190 on beta

bcfdf9a0d7 iis repository with https

aae667e6b6 [APC at the result level] added the APC at the level of the result and modified test class

0547fd6ee7 Merge pull request '[stats-wf]fixed the result_result table' (#191 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #191

493caef358 [stats-wf]fixed the result_result table related to PR#191

9fd2ef468e [APC at the result level] changed dependecy in external pom

ac8b8f224f Merge branch 'beta' into extendResult

b04ecfcf13 Merge pull request 'extendResult' (#192 ) from extendResult into beta

Reviewed-on: #192

1f8302dc37 Merge pull request '[stats-wf]fixed yet another typo' (#193 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #193

78be2975f0 [stats-wf]fixed another typo related to PR#193

fbc28ee8c3 [OpenCitation] change the integration logic to consider dois with commas inside

b071f8e415 [OpenCitation] change to extract in json format each folder just onece

759ed519f2 [OpenCitation] added logic to avoid the genration of self citations relations

5c4043dba8 [OpenCitation] refactoring

600ede1798 serialisation of APCs int he XML records

6158170334 testing delegated authority and bumped dep to schemas

c191080965 mergin with branch beta

1490867cc7 [OpenCitation] cleaning of the COCI model

a1013e62d4 Merge pull request 'openCitations' (#194 ) from openCitations into beta

Reviewed-on: #194

be64055cfe [OpenCitation] changed the name of destination folders

90e197a563 Merge pull request '[OpenCitation] changed the name of destination folders' (#195 ) from openCitations into beta

Reviewed-on: #195

3aa2020b24 added script to regenerate hostedBy Map following instruction defined on ticket #7539

updated hosted By Map

89c7313fc5 Merge branch 'beta' into hierarchical_orgs_relations

56e048ef21 Merge pull request 'added hierarchy rel in ROR actionset' (#153 ) from hierarchical_orgs_relations into beta

Reviewed-on: #153

d3f03abd51 fixed wrong json path

891781ee3f Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

cf8443780e added processingchargeamount to the result view

401dd38074 code formatting

99f5b14469 [graph raw] invisible records stored among the raw graph rather than the claimed subgraph

5226d0a100 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

9d6203f79b test mapping datasource

86cdb7a38f [provision] serialize measures defined on the result level

fbf192d6ba Merge pull request '[provision wf] serialize measures defined on the result level' (#196 ) from xml_measures into beta

Reviewed-on: #196

ab36154e3e added more ignores

55caa389d5 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

a87c070447 conflicts resolved, merged from beta

e7016c3981 Merge branch 'master_202203' into beta

66c09b1bc7 code formatting

a40b59b7d5 [ResultToOrgFromInstRepoTest] fixed issue in model of the input resources

859cb7ac9d [DoiBoost AR] changed test resource to be sure the result will always have EMBARGO as value for AccessRight

b0f369dc78 bug fix in the DedupRecordFactory, DataInfo set before merge

fe1c966cbf Merge branch 'master_202203' of code-repo.d4science.org:D-Net/dnet-hadoop into master_202203

8991d097b4 bug fix in the DedupRecordFactory, DataInfo set before merge

f5b0a6f89c [master to beta] fixed issues in test files

1932a65d1c Merge pull request '[Stats wf] sprint 6 indicators' (#198 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #198

580d904aae manually merging PR#199 #199

3970651ee1 Merge pull request 'fixed query after the change in the indicator table' (#200 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #200

3be8737c32 [graph-stats] fixed query after the change in the indicator table related to PR#200

44b0c03080 [HostedByMap] update to download the json instead of the csv

8a41f63348 [HostedByMap] update to download the json instead of the csv

b7c2340952 [HostedByMap - DOIBoost] changed to use code moved to common since used also from hostedbymap now

5d608d6291 [HostedByMap] changed the model to include also oaStart date and review process that could be possibly used in the future

2c5087d55a [HostedByMap] download of doaj from json, modification of test resources, deletion of class no more needed for the CSV download

9f3036c847 Merge pull request 'HostedByMap' (#201 ) from hostedByMap_update into beta

Reviewed-on: #201

f25407bbe2 added mapping for datasource consent fields to integrate them in the graph

2b643059fa [Country Propagation] changed the logic to get the collectedfrom at the result level. To fix issue when no instance is created for a result that should have the country associated. Change the code to use spark instead of hive to prepare the data needed for the propagation step. Added new tests for the intermediate steps and new verification for the propagation itself

4437f9345d [Country Propagation] left out from previous commit

2fbb35ade5 mergin with branch beta

12de9acb0d [Country Propagation] left out from previous commit

d48ccfd65e Merge pull request 'enrichment_country' (#203 ) from enrichment_country into beta

Looks good to me

Reviewed-on: #203

f430029596 cleanup

0f7d8ca2e0 [HostedByMap] change on master to align to PR 201 on beta merged as 9f3036c847

c959639bd5 dependency updated to the new pace-core version

c763aded70 dependency updated to the new pace-core version

89fd275480 [HostedByMap] added left over from PR and fixed issue on workflow

43733c1a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

f10066547b increased spark.sql.shuffle.partitions in affiliation_from_semrel_propagation

48d32466e4 instances grouped by URL expose only one refereed

7b8f85692e [Enrichment country] fixed issues with parameters and workflow args

61319b2e83 updated dhp-schema version; set entity-level dataInfo before & after merging the fields from the group of duplicates

3610f1749a merged pom.xml from beta branch

741bc99c47 Merge branch 'beta' into datasource_pdf_consent

fa3cb84f77 Merge pull request 'Datasource consent fields' (#202 ) from datasource_pdf_consent into beta

Reviewed-on: #202

395ac6ecec merged pom.xml from beta branch

96aa2a5d0d Merge branch 'beta' into instance_group_by_url

5d53ac95aa Merge pull request 'XML serialisation of instances with the same URLs - 2nd round' (#204 ) from instance_group_by_url into beta

Reviewed-on: #204

9fa3dd78fe Merge pull request '[stats wf] various fixes, organization ids for inst. dashboard' (#205 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #205

eca82e30c9 updated dhp-schema version

0a0ae84c22 [graph raw] DOI based instance URLs on https

1b11010169 minor fix

de85367695 Merge pull request '[stats wf] fix: views cannot be stored as parquet...' (#206 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #206

873369af1c Merge pull request '[stats wf] added apcs in monitor db' (#207 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #207

79336d46c5 [Clean Context] first naive implementation of a functionality to clean not wanted contextes from one result. This implementation simply verifies the main title of the results start with a given string

e77d104951 [OC] added / to workflow path

8c457f1b2c conflicts resolved, merged from beta

05fafa1408 [graph raw] avoid NPEs importing datasource consent fields

4190c9f6bc [graph raw] avoid NPEs importing datasource consent fields

e612489670 Add fileGZip collector plugin and respective test

66e718981e [maven-release-plugin] prepare release dhp-1.2.4

505420fd61 [maven-release-plugin] prepare for next development iteration

bc05b6168a [maven-release-plugin] rollback the release of dhp-1.2.4

85bc722ff4 [maven-release-plugin] prepare release dhp-1.2.4

2ac2d928bd [maven-release-plugin] prepare for next development iteration

3bba6d6e38 [maven-release-plugin] rollback the release of dhp-1.2.4

f03dea4f49 allow to skip maven site

db3d9877a5 [maven-release-plugin] prepare release dhp-1.2.4

598e11dfd7 [maven-release-plugin] prepare for next development iteration

ad85d88eaf [maven-release-plugin] rollback the release of dhp-1.2.4

86585a6b27 [maven-release-plugin] prepare release dhp-1.2.4

c26222623f [maven-release-plugin] prepare for next development iteration

bc1bf55507 Add AbstractSplittedRecordPlugin

91e32f12ed Merge branch 'master' into beta

d0b84d3297 Add FileCollectorPlugin and respective test

4eff7856f5 Merge pull request '[stats-wf] computing stats in each step' (#210 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #210

21f32b83c6 [graph enrichment] fixed country_propagation oozie workflow definition, parameter saveGraph is not needed anymore by the SparkCountryPropagationJob

48b580b45c [graph enrichment] fixed country_propagation oozie workflow definition, parameter saveGraph is not needed anymore by the SparkCountryPropagationJob

73c172926a [Doiboost] fixed fundingReference extraction from the Crossref records

b93a141d6c [Doiboost] fixed fundingReference extraction from the Crossref records

8e8933d41a [BulkTagging] added fix if result.dataInfo is null

d5b29d96a7 fix merging in crossrefAggregator which creates dataInfo null

c5a863132c [BulkTagging] revert it

0012e57bf9 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

4314db55c8 migration to services: update sql queries

c96a8613f8 update SQL queries

869407c6e2 [Measures] added new measure (usagecounts) as action set. Measure added at the level of the result. Ref #7587

5feae77937 [Measures] last changes to accomodate tests

dbfbe8841a [Clean Context] changed the description in input parameters

a38f0f5ea7 mergin with branch beta

61c0266a44 Merge pull request 'Remove Context from result' (#208 ) from cleancontext into beta

Reviewed-on: #208

5295effc96 [Measures] fixed issue

d580e15442 Modified last intersection since we lost many titles.

this is my last resource, after that, I've to  change my job

c304657d91 [Measures] put the logic in common, no need to change the schema

eabb40fccc Merge branch 'beta' into 7096-fileGZip-collector-plugin

88acad76f9 Merge branch 'beta' into eosc_dimitris

d012d125d7 [EOSCTag] -

b61efd613b [Measures] addressed comments in the PR

bebb2a0560 Merge branch 'eosc_dimitris' of https://code-repo.d4science.org/D-Net/dnet-hadoop into eosc_dimitris

20de75ca64 [Measures] removed typo

a289c9eae2 Merge pull request '[Measures] added new measure (UsageCounts)' (#214 ) from eosc_dimitris into beta

Reviewed-on: #214

ccba1a3db1 [Clean Context] added logic to cleaning workflow to accomodate also context cleaning

5b7d9e741c [Clean Context] added logic to cleaning workflow to accomodate also context cleaning

29150a5d0c code formatting

9a961a0092 [Clean Context] fixed issue in param name

6dc68c48e0 [EOSCTag] -

e0915061c2 [Clean Context] fixed issue in param name

d660895b30 fixed wrong mapping type of dataset

7cb7066472 [EoscTag] first "rough" implementation

aa12429f50 Modified last intersection since we lost many titles.

a82ec3aaaf code formatter

30105f0722 Merge branch 'beta' into 7096-fileGZip-collector-plugin

bbb77052d3 [EOSCTag] first test

54162f5c4f Merge branch 'beta' into cleancontext

19d90658fc [Clean Context] added description to parameters

911ce0780a Merge branch 'cleancontext' of https://code-repo.d4science.org/D-Net/dnet-hadoop into cleancontext

81242538e6 Merge pull request 'Oozie workflow for cleancontext' (#216 ) from cleancontext into beta

Reviewed-on: #216

Looks good. We need to extend the cleaning workflow parameters to enable the extra step only when it is needed.

87bff36d9e mergin with branch beta

27c85e901a [EOSCTag] added resources and finalized test for Jupyter Notebook tagging

dfbd2bcbea [EOSC TAG] added logic in case subject is null

88562c0930 [EOSC TAG] added test for galaxy for title and description criterias

e342ec93f0 [EOSCTag] prepared resources for test

81c4496d32 Merge branch 'beta' into 7096-fileGZip-collector-plugin

8c22e5c30a added fix to include date array with only year or year and month

78015a5733 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

5ffc24d1ba EOSC Services - ongoing update

623f7be26d Fix reading files from HDFS in FileCollector & FileGZipCollector plugins

f5f532d134 EOSC Services - ongoing update

05c1ea92e9 EOSC Services - added Service-specific fields in the XML record serialization

a8c51f6f16 EOSC Services - fixed query and testing preparation

e37177e1ce mergin with branch beta

b6a7ff3a99 EOSC Services - removed fields from mapping, testing preparation

2ade69dea6 EOSC Services - minor

a21fe310e5 [EOSCTag] last test and change in the implementation to search in title and descriptio

9e12cb3c92 EOSC Services - removed field knowledgegraph; depending on the released schema module

da611cfbbd [eosc_services] resolved merge conflicts

3aeedd931a [EOSCTag] fixed issue in case description is null. Modified test resources and classes

bd1108f98b mergin with branch beta

8a72de4011 [EOSCTag] modified workflow to execute all the steps and not only the last one

5fe25cc51c Merge pull request '[eosc tag] set the eosc subjects, rough implementation' (#215 ) from eosc_tag into beta

Reviewed-on: #215

846975c886 [eosc_services] using the correct 'keyword' subject type, as declared in the dnet:subject_classification_typologies vocabulary

658450d9a3 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

a056f59c6e [UsageCount] make it as an action set as it should be, plus changed the test to make them work as well now

89657a0b78 [UsageCount] refactoring

378020e30a [eosc_services] unit test adaptation

77bc9863e9 [openorgs] mapping parent/child relations without massaging the semantic labels

2a8e0fb72f [openorgs] mapping parent/child relations without massaging the semantic labels

5d3b4a9c25 [graph merge beta] merge datasource originalid, collectedfrom, and pid lists

ca8d26bcb4 added better filter for openCitations

22f65680b9 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

e4eac1d20b [EOSC TAG] added code to remove EOSC Jupyter Notebook from subjects and put EOSC as classid in the qualifier

0dc33ea391 [openorgs] fixed parent/child query, using the correct semantic labels

6031acb2e3 [openorgs] fixed parent/child query, using the correct semantic labels

46c07e0724 deleted hierarchical rels from ror action set

4c50f35c8b update publication Date format

c1971d52c4 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta

997c50078e [graph grouping] drop relation target path before copying from source

6442763f97 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

52cb086506 [graph grouping] drop relation target path before copying from source

8dd5517548 code formatting

1dda11e031 [hb patch] updated map

d098ad0d93 [hb patch] updated map

f5207885e3 [EOSCTag] changed code to remove EOSC Jupyter Notebook and modified test to exclude galaxy + software from the tagging for Galaxy

eaf9385ae5 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

c298c148cb [CountryPropagation] fix NPE issue

5e0b8f9b5f [CountryPropagation] refactoring

5c2949a864 Merge pull request '[stats wf] added open citations & more orgs in monitor, removed collab indicator' (#213 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #213

31d4557e8d Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop

4b6913787b [DOI-BOOST] added one method in test of crossref mapping to aof and one resource. Related to ticket 7807

ab8868bd3a [ZENODO-API] changed to iterate in all the deposited products and not just the last ten

b94a791bc5 unit tests to transform cnr explora

b229c6e7af Merge pull request 'beta' (#218 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #218

68bd58d6a4 tests for ROHub

922c6d66ef Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

634869ce95 deleted hierarchical rels from ror action set

b8cda65487 code formatting

116902c028 mapping relationship from trasformed records based on oaf:relation

88d531dc91 exclude FAIRsharing records from Datacite

4c8e820ff0 mapping relationship from trasformed records based on oaf:relation

06b5533d4c Merge branch 'beta' into 7096-fileGZip-collector-plugin

e03c0c7794 Merge branch 'beta' into oaf_relation_mapping

875ae29961 Merge pull request 'mapping relationship from trasformed records based on `oaf:relation`' (#219 ) from oaf_relation_mapping into beta

Reviewed-on: #219

c7b09c6225 Merge branch 'beta' into 7096-fileGZip-collector-plugin

b295a40d9c restored use of name_particles when parsing author names

c76ff6c613 Merge pull request '7096-fileGZip-collector-plugin' (#211 ) from 7096-fileGZip-collector-plugin into beta

Reviewed-on: #211

7423577a08 [Graph DUMP] add code to produce the delta of new projects with respect to the previous delta/dump

b98f904d48 [Funder Products Dump] new way to avoid using hive

ff74e73369 [DUMP NEW FUNDED PRODUCTS] change in resources

f561f13dd9 [Funder Products Dump] fixed names of parameters in workflow

edddfc6c63 [DUMP DELTA PROJECTS] adding test and resource

929b145130 code formatting

5130eac247 mapping by participant project contribution

316b0fd73c added 'von' to the name particles file

4829b96bb5 Merge branch 'beta' into author_name_particles

cba9c2b7cc Merge pull request 'author name parsing' (#220 ) from author_name_particles into beta

Reviewed-on: #220

a8773af0cb Merge branch 'beta' into project_organization_contribution

1d1fe3b151 [DUMP DELTA PROJECTS] refactoring

71744a1f52 [DUMP DELTA PROJECTS] refactoring

ee1f1eeca2 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

7da24c1dec added more logging

a88103bcf9 [action manager] added more testing

072f192853 include the class information in the measure XML serialization

2f998b2429 Merge pull request '[Graph DUMP] add code to produce the delta of new projects with respect to the previous delta/dump' (#221 ) from dump_delta_projects into master

IMO looks good, I think it can be integrated in the master branch.

Reviewed-on: #221

4ec13e2b66 Merge branch 'master' into dump_new_funded_products

0cb1c70788 code formatting

446699c59d Merge pull request '[Graph Dump] New funded products dump' (#222 ) from dump_new_funded_products into master

Reviewed-on: #222

138d1dfbf8 Merge pull request 'score class in the XML serialization' (#225 ) from measure_serialization into beta

Reviewed-on: #225

c09fcdb40b Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

fae681fea1 [Country Propagation] add check to avoid NPE on datasource.getDatasourceType().getClassis()

0c1cfee396 mapping oaf:fulltext elements in the result.fulltext field

28a32facf6 Merge pull request 'mapping `oaf:fulltext` element in the `result.fulltext` field' (#226 ) from oaf_fulltext_mapping into beta

Reviewed-on: #226

0a4f4d98fa added PMCId to PmArticle

00168303db Added unit test to verify the generation in the OriginalID the old openaire Identifier generated by OAI

1138b2ac8e code formatting

18b505d6a3 Merge branch 'master' into beta

5b76321d9c implemented oozie workflow to generate scholix dump filtering relclass semantic

65cc736e2f [Clean Country] first implementation to remove country NL from results collected from NARCIS when doi starts with mendely prefix

438abdf96f [EOSC TAG] adding eosc interoperability guidelines in the specific element in the result. Removed from subjects. Removed also the deletion of EOSC Jupyter Notebook from subject since now the criteria are searchd for in a different place

5f651f2316 changed filter relation on SubRelType

d900a02b74 Merge pull request 'implemented oozie workflow to generate scholix dump filtering relclass semantic' (#229 ) from opencitation_enrichments into beta

Reviewed-on: #229

5143a80232 [EOSC TAG] modification of test class to align with new element

56d09e6348 [EOSC TAG] before adding the tag added a step to verify the same tag is not already present

e61b8e6b03 mergin with branch beta

3be036f290 [EOSC TAG] refactoring after compilation

317a4a56ef [EOSC context TAG] first implementation of the logic to tag results imported from datasources registered in the EOSC

ddc414b258 fixed wrong json param

7a1c1b6f53 [EOSC context TAG] Add test class and resourcesK

627332526b [EOSC context TAG] workflow start from reset_outputpath action

ffb0ce3fb9 mergin with branch beta

06a95daf60 [EOSC context TAG] refactoring after compilation

2c933f1158 mergin with branch beta

a12d28c644 [Clean Country] added logic not to remove country from result if it exist a hosting datasource with that country. Moreover the country will be removed only if added with propagation

5968ec018d [Clean Country] modified workflow and added param file

d091866e48 [EOSC Context Tagging] refactoring

657b0208a2 multiple works download (<=100) for single request

144c103b67 [EOSC Context Tagging] add check to avoid the insertion of the context if already present

c3ede1b379 Merge branch 'beta' into pubmed_update

3f883c4ecc Merge pull request 'pubmed_update' (#228 ) from pubmed_update into beta

Reviewed-on: #228

612b7a5530 Merge branch 'beta' into tagEosc

3c23d634eb Merge pull request 'EOSC IF' (#230 ) from tagEosc into beta

Reviewed-on: #230

0172bab251 [EOSC Context Tagging] refactoring

68cb637832 merge with branch beta

1c82acb168 [EOSC Context Tagging] refactoring: moved EOSC IF tagging in package eosc under bulkTag

35bcd9422d [EOSC Context Tagging] removed not needed specification in path

d43663d30f adapted RorActionSet test, it should not create parent/child rels

c03e20be39 Merge pull request 'EOSC COntext Tagging' (#231 ) from eosc_context_tagging into beta

Reviewed-on: #231

26104826c4 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

67525076ec fixed test, now it compiles after commit a6977197b3

09ccc7b472 Merge branch 'beta' into project_organization_contribution

ed98a6d9d0 [Datacite mapping] include the older datacite prefixed OpenAIRE id among the originalId[]

60e4fbd78b Merge branch 'beta' into project_organization_contribution

1dd1e4fe3a extended test for mapping project_organization relations

37cfda0fc5 Merge pull request 'participant project contribution' (#223 ) from project_organization_contribution into beta

Reviewed-on: #223

3329b6ce6b [EOSC TAG] added fix for NPE on subjects

0727f0ef48 [EOSC tag] avoid NPEs

f62c4e05cd code formatting

92e48f12f7 [metadata collection] updated collector plugin name

64311b8be4 removed unuseful accumulator

209c7e9dab [datacite] avoid UnsupportedOperationException

27681cf6bf Merge pull request '[stats wf] latest version of indicators + added FOS classification' (#232 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #232

eb53b52f7c code formatting

efd96e7e66 Merge pull request 'fixed the datasourceOrganization relations' (#233 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #233

e62018e95d [aggregator graph] added more assertions in test

b09d7ddc74 fixed the datasourceOrganization relations

27a91841e7 WIP: cleaning of subjects

499826ead1 serialising field eoscifguidelines field in the Solr XML records

a7a18d7630 [Graph Dump] removed code for the dump from the project. Fixed issues in tests when possible

08ce2cadc2 Merge pull request '[Graph Dump] Remove code from dnet-hadoop' (#235 ) from removeDump into beta

Reviewed-on: #235

b78889a0ce WIP: cleaning of subjects

6c0fd9284b merge from beta

c1f2ffc53d Merge pull request 'commenting out the collab indicators because they still fail' (#237 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #237

32cee1f619 WIP: cleaning of subjects

844f6eb465 Merge branch 'beta' into clean_subjects

84598c7535 Merge pull request 'restored some collab indicators' (#240 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #240

4eaa063b1f cleaning of subjects

29c4cde42e Merge branch 'clean_subjects' of https://code-repo.d4science.org/D-Net/dnet-hadoop into clean_subjects

a4815f6bec Merge branch 'beta' into clean_subjects

3937ff04de Merge branch 'beta' into tagEosc

d85ba3c1a9 Merge pull request 'serialising field eoscifguidelines field in the Solr XML records' (#234 ) from tagEosc into beta

Reviewed-on: #234

390013a4b2 mergin with branch beta

a78028dabc Merge branch 'beta' into clean_subjects

3418ce50ac cleaning of subjects: perform the cleaning when the given value is equivalent to one of the terms in the vocabulary

62d2138806 [Clean Context] changed a bit the logic. Added the check not to have result hosted by a datasource of type institutional repository from NL. Added also the check that the country should have been included in the result via propagation for it to be removed

51ad93e545 [scholix] fixed OpenCitation dump procedure

7dbdd4a0fe [Clean Country]changes related to #241 (comment)

2aa16d0432 [scholix] fixed OpenCitation dump procedure

cb7c07c54e [scholix] added step to create tar archive

adb526b0e1 Merge branch 'beta' into clean_subjects

b7c387c21f cleaning of subjects: avoid duplicated subjects, prioritise collected vs inferred or other sources

27445ccdaa cleaned log

da1cf561e6 alignment with beta

bfdf2dc390 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid_multipleworks_download

a11eb38065 testing for RO-Hub

5c45d52af3 testing for RiuNet

9ef063d502 #7861#note-8 instance url from handle

a431e01383 Merge pull request 'orcid_multipleworks_download' (#242 ) from enrico.ottonello/dnet-hadoop:orcid_multipleworks_download into beta

Reviewed-on: #242

853c996fa2 Merge branch 'beta' into handle_as_instance_urls

14dc909a14 Merge branch 'beta' into clean_country

1203378441 Merge branch 'beta' into clean_subjects

a539c6ccaf https for handle URLs

f14107ad77 Merge branch 'handle_as_instance_urls' of https://code-repo.d4science.org/D-Net/dnet-hadoop into handle_as_instance_urls

690be4482f Merge pull request '#7861#note-8 instance url from handle' (#243 ) from handle_as_instance_urls into beta

Reviewed-on: #243

b5f7bd30be Merge branch 'beta' into clean_subjects

b5d6966c01 Merge branch 'beta' into clean_country

ff6f789b6d code formatting

5066db3386 Merge pull request 'subjects cleaning' (#239 ) from clean_subjects into beta

Reviewed-on: #239

27af5122d2 logs for non well formed XML files

b99a011345 return empty Oaf list if record cannot be parsed

a0919ed495 [aggregator graph] save invalid records aside for further inspection

c48f6e9c57 [aggregator graph] save invalid records aside for further inspection

9e7ec4198f fixed test

1e42d984e1 [aggregator graph] save invalid records aside for further inspection

465e941214 Merge pull request '[stats wf] Changes to indicators tables' (#244 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #244

e370e940d8 [aggregator graph] save invalid records aside for further inspection

192215a18e merged from branch discard-non-wellformed

5240ac3d7b [EOSC Tag] remove addition of eosc context for result with eosc if guidelines set

26e1badded added instance.url syntactical validation, avoid creating multiple duplicated URLs

e45ec15221 Merge branch 'beta' into clean_country

33bb79459e Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

c42850328e fixed semantic (subreltype) for ServiceOrganization relations

960cb861a0 refactoring

982bcc1e35 test wrid pid and record identifier

ba33ff71fd refactoring for the generation of relationships from related identifier of type 'OPENAIRE'

c86cc53520 suppressing hyper verbose spark logs during unit test execution

c5eb722170 relationships from relatedIdentifier whose target id type is one of the pid type with an authority

55da4d8715 [BulkTagging] modifying code to represent constraints horizontally on all the results. Added subject to the set of field used to express the constraint. Modified resorces to test the new approach. Modified test calss

f0e303abf9 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

4a3e119b73 mergin with branch beta

3e6b0f58bb [BulkTagging] changed the query to the IS to get also the information for the advancedConstraint from the profile

ca216a92ad [BulkTagging] changed the query to the IS to insert values for FOS and SDG as subject in the configuration used for the tagging

fd63e9bfac Mapping all relationships supported in ModelConstants and ModelSupport

25e9d92aad Merge branch 'beta' into clean_country

6ad38ade74 Merge pull request 'Clean Country' (#241 ) from clean_country into beta

Reviewed-on: #241

3b60642ef9 Merge pull request 'Synchronize indicators in stats-db with monitor-db' (#249 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #249

b60985cf68 Merge branch 'beta' into horizontalConstraints

57dbeb08d2 code formatting

b4b6a4457c Merge pull request 'BulkTagging extension' (#250 ) from horizontalConstraints into beta

Reviewed-on: #250

0b3e44e521 Merge branch 'beta' into relation-from-odf

3f90d159e3 code formatting

de7bc9350e Merge pull request 'relation-from-odf' (#251 ) from relation-from-odf into beta

Reviewed-on: #251

f3f7604e6c trying to fix a test that fails only on Jenkins

1cb79719a7 [BulkTag] fixed issues

3ec044600d [BulkTag] fixed conflicts

f1d7d45cf7 [BulkTag] fixed issue

b5b5a4c192 [CleanCountry] fixed issue

49360770d7 map w3id as instance url

89f7007080 Merge pull request '[stats wf] misc changes' (#254 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #254

188f25eefa [BipFinder] Fixed issue for wrong escaped char in doi

28dc317350 [BipFinder] refactoring

7324853a17 Revert "[BipFinder] refactoring"

This reverts commit 28dc317350.

4d8339614b Revert "[BipFinder] Fixed issue for wrong escaped char in doi"

This reverts commit 188f25eefa.

8d97949316 [cleaning] fixed loop in wf nodes

d51275a965 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

ece40adc09 [cleaning] fixing NPE in the country cleaning phase

b301e9fdff [cleaning] renamed action name/description

6163ecbf63 [cleaning] renamed parameters in wf action

b47aaf4dd1 [cleaning] subjects declared as belonging to specific vocabularies whose values are not found in the vocab are set to type keyword

a1f94530a3 added documentation

135cf81151 Merge remote-tracking branch 'origin/beta' into beta

ae7cd0735a [graph2hive] more partitions

9b449110c6 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

72f0d88d6c formatted code

bca4a61710 suppressing hyper verbose spark logs during unit test execution

ec39b84898 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

31a10f000b Map the field oaf:eoscifguidelines from mdstores. Currently we can find it in ROHub metadata

ee759ac92d file format after mvn compile

208ed32315 fixed xpath for semantic relation

2b9a20a4a3 Changed the way Scholexplorer filter the relationships, I found that filter all relation coming from openCitation is wrong, because we loose a lot of relation than intersect OpenCitation, but they don't come only from there

22873c9172 Merge pull request 'Added fields: totalcost, fundedamount, currency, in project table' (#257 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #257

7c3390ac10 Merge branch 'beta' into eoscifguidelines-from-mdstores

0419953470 merge from beta

5af5a8ae42 added IdentifierComparator

ddff0e8999 merging duplicates using IdentifierComparator

24f99d7310 Merge pull request 'Map oaf:eoscifguidelines from mdstore to the graph' (#256 ) from eoscifguidelines-from-mdstores into beta

Reviewed-on: #256

3dbc637d3e code formatting

0aa725083f extended dedup testing

1d5143b0b6 Merge branch 'beta' into deduptesting

2687fc9f73 tests for EOSC Future review - ROhub

3c08269a4d Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

2832117f23 added eoscifguidelines in test

a79c47522d updated ORCID datasource identifier

0e3edc5018 [Bulk Tag] fixed issue in verb name

90c8f9cb61 tests for EOSC Future

24ef301cc1 [graph cleaning] patch the result's collectedfrom and hostedby identifiers according to the datasource master-duplicate mapping

6082d235d3 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into graph_cleaning

11695ba649 [graph cleaning] patch also the result's collectedfrom and hostedby datasource name according to the datasource master-duplicate mapping

0628df7a3a resolving conflicts

9c70c5dbd6 [Bulk Tag horizontal] added new path in definition of constraint (to recognize fos subjects) - changed test and resource class to test this new aspect

7d264a1d69 Merge pull request 'horizontalConstraints' (#259 ) from horizontalConstraints into beta

Reviewed-on: #259

58c05731f9 [graph cleaning] WIP: testing the collectedfron and hostedby patch procedure

8e3edba318 [graph cleaning] testing the collectedfron and hostedby patch procedure

bb0ddc1c44 [BulkTag] adding verb starts_with

ce020f2c83 [EOSC FUTURE] added resources and test for review

a38116546d Merge branch 'beta' into deduptesting

5a48a2fb18 implemented synch for single mdstore

41f7f1bbc5 Merge pull request '[graph dedup] records stability and testing' (#44 ) from deduptesting into beta

Reviewed-on: #44

ddf065756f Merge pull request 'Two organizations are added for monitor' (#258 ) from antonis.lempesis/dnet-hadoop:beta into beta

Reviewed-on: #258

8248da40d9 Merge branch 'beta' into graph_cleaning

71b121e9f8 Merge pull request '[graph cleaning] update collectedfrom & hostedby references as consequence of the datasource deduplication' (#260 ) from graph_cleaning into beta

Reviewed-on: #260

062abfd669 fixed NPE, removed unused stuff

cd1b58483e [bulk tag] fixed Community configuration parsing to void NPE

1b8488976b code formatting

3d99b78d94 [Cleaning] fixed error in parameter (workingPath to workingDir)

a485a94956 [Cleaning] fixed parameter name in property file

2094fa6db0 [cleaning] align wf parameter names

730228d73d [cleaning] align wf parameter names in test

389dd25430 [cleaning] avoid NPE

8b44afe5e5 [cleaning] avoid NPE

c18b8048c3 [cleaning] avoid NPE

5e4866d033 implemented synch for single mdstore

b8bafab8a0 [cleaning] improved vocabulary based mapping, specialization for the strict vocab cleaning

7b80b24f82 [cleaning] country cleaning must use both PID and AlternateIdentifier fields

d791840b82 [Clean Country] added test to verify remove of country:

dc0ec88a58 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

8685eaa706 [Clean Country] added test to verify remove of country

f37113a941 [BulkTag] moving xquery to get community configuration in dedicated file

6674cccb94 [BulkTag] description of parameters more comprehensive for those who do not implement it

9cf0a98699 [cleaning] set the common subject classid/name

6aa91204a5 [orcid propagation] skip empty directories

3c9826f186 updated lines function to it's implementation linesWithSeparators.map(l => l.stripLineEnd) in this way we force scala plugin compiler to consider this pipeline scala code and not java.string.lines() pipeline

8893389895 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

18a7aa2d78 Merge pull request 'Workaround to use new version of intellij on Beta' (#267 ) from beta_intellij into beta

Reviewed-on: #267

d6895f0387 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

7b7520850b fixed an invalid char

1f367122e4 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

91b845f611 Considering instance pids and alteternative identifiers

04cb96001c [broker] d40e20f437 adapted to the beta graph model

0a58bc7ba7 [broker] prevent NPEs

0a5c6010b0 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

ecd398fe51 refactoring

505867bce9 [bulk tagging] better node naming

b254a0375f [Affiliation from institutionalrepo] changed the field to check to verify the datasource type. Now it is in the field jurisdiction

e82e009b46 added missing close tag for XML produced by the xquery to get information for the community from the IS

d05ca53a14 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta

6c81a161d2 Merge remote-tracking branch 'origin/beta' into 8231-mdstore-synch-improve

0b9819f1ab Code formatted

8920932dd8 Code formatted

sandro.labruzzo changed target branch from master to beta

2023-02-21 15:11:15 +01:00

claudio.atzori added 1 commit 2023-02-22 09:57:09 +01:00

cd3a51a15f Merge branch 'beta' into 8232-mdstore-synch-improve

claudio.atzori merged commit a590c371a9 into beta

2023-02-22 10:02:27 +01:00

claudio.atzori referenced this issue from a commit

2023-02-22 10:02:27 +01:00

Merge pull request '8232-mdstore-synch-improve' (#272) from 8232-mdstore-synch-improve into beta

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#272