Miriam Baglioni
|
eccf3851b0
|
Hosted By Map - refactoring
|
2021-08-04 10:16:30 +02:00 |
Sandro La Bruzzo
|
74afe43c3a
|
fixed wrong test file
|
2021-08-04 10:16:17 +02:00 |
Miriam Baglioni
|
1e952cccf6
|
Hosted By Map - refactoring and deletion of not needed methods
|
2021-08-04 10:15:43 +02:00 |
Miriam Baglioni
|
8ba8c77f92
|
Hosted By Map - refactoring
|
2021-08-04 10:14:57 +02:00 |
Miriam Baglioni
|
8f7623e77a
|
Hosted By Map - refactoring and application of the new aggregator
|
2021-08-04 10:14:20 +02:00 |
Sandro La Bruzzo
|
3fc820203b
|
fixed wrong test file
|
2021-08-04 10:13:59 +02:00 |
Miriam Baglioni
|
a7bf314fd2
|
Hosted By Map - added new aggregator to get just one result per datasource id
|
2021-08-04 10:13:30 +02:00 |
Miriam Baglioni
|
100e54e6c8
|
mergin with branch beta
|
2021-08-03 10:47:11 +02:00 |
Miriam Baglioni
|
461b8a29a0
|
removed not needed class
|
2021-08-03 10:46:51 +02:00 |
Miriam Baglioni
|
327cddde33
|
Hosted By Map - refactoring
|
2021-08-03 10:44:13 +02:00 |
Miriam Baglioni
|
17292c6641
|
Hosted By Map - resources for testing purposes
|
2021-08-02 19:37:08 +02:00 |
Miriam Baglioni
|
ee7ccb98dc
|
Hosted By Map - test class to verify the application of the hbm to results and datasource
|
2021-08-02 19:36:18 +02:00 |
Miriam Baglioni
|
90e91486e2
|
Hosted By Map - test class to verify each step in the preparation process
|
2021-08-02 19:35:52 +02:00 |
Miriam Baglioni
|
1e859706a3
|
Hosted By Map - Classes to apply the HBM to results and datasources
|
2021-08-02 19:35:23 +02:00 |
Miriam Baglioni
|
72df8f9232
|
Hosted By Map - removed the aggregator for the datasource (it is no more needed) and added a new aggregator for the results. Changed also the hostedBYMap aggregator
|
2021-08-02 19:34:44 +02:00 |
Miriam Baglioni
|
ff1ce75e33
|
Hosted By Map - modification in the code to prepare the info needed to apply the HostedByMap. There is no need to join datasources with the hbm: all the information needed is in the hosted by map already
|
2021-08-02 19:32:59 +02:00 |
Claudio Atzori
|
e826aae848
|
using constants from ModelConstants
|
2021-08-02 14:28:59 +02:00 |
Miriam Baglioni
|
1695d45bd4
|
Hosted By Map - Test class to verify the preparation of the intermediate information
|
2021-07-30 17:57:01 +02:00 |
Miriam Baglioni
|
7c6ea2f4c7
|
Hosted By Map - first attempt for the creation of intermedia information to be used to applu the hosted by map on the graph entities
|
2021-07-30 17:56:27 +02:00 |
Miriam Baglioni
|
d8b9b0553b
|
Hosted By Map - model classes to store the intermediate information to be used to apply the hosted by map
|
2021-07-30 17:55:39 +02:00 |
Miriam Baglioni
|
613bd3bde0
|
Hosted By Map - refactor of the first attemp to prepare a new hosted by map dependent on the datasource in the graph and on two external sources: the gold list from unibi ad the doaj list of open access journal. Both the lists are downloaded from provided url parameter
|
2021-07-30 17:54:45 +02:00 |
Miriam Baglioni
|
d1807781c0
|
mergin with branch beta
|
2021-07-30 14:34:07 +02:00 |
Miriam Baglioni
|
1d6ac3715b
|
merge branch with beta
|
2021-07-30 11:58:29 +02:00 |
Claudio Atzori
|
19620eed46
|
applying PR#131, Patch the identifiers (source/target) in the relations, refinements
|
2021-07-30 11:09:32 +02:00 |
Claudio Atzori
|
4f78565c04
|
fixed implementation of PatchRelationsApplication, refined the relative unit test
|
2021-07-30 11:07:09 +02:00 |
Claudio Atzori
|
a6a38cca9e
|
fixed implementation of PatchRelationsApplication, refined the relative unit test
|
2021-07-30 11:06:11 +02:00 |
Miriam Baglioni
|
9bc4fd3b69
|
Patch FCT relations - fixed issue with join
|
2021-07-30 10:34:05 +02:00 |
Miriam Baglioni
|
2fc89fc9b5
|
Merge branch 'fct_project_id_replacement' of https://code-repo.d4science.org/D-Net/dnet-hadoop into fct_project_id_replacement
|
2021-07-30 10:20:43 +02:00 |
Claudio Atzori
|
081fe92a21
|
Merge branch 'fct_project_id_replacement' of https://code-repo.d4science.org/D-Net/dnet-hadoop into fct_project_id_replacement
|
2021-07-30 10:13:56 +02:00 |
Claudio Atzori
|
576693d782
|
added unit test for PatchRelationsApplication
|
2021-07-30 10:13:33 +02:00 |
Miriam Baglioni
|
baad01cadc
|
hostedbymap
|
2021-07-29 13:04:39 +02:00 |
Claudio Atzori
|
e725c88ebb
|
[raw_all] patching relation identifier phase to be run at the end, i.e. includes also claimed relations
|
2021-07-29 13:03:43 +02:00 |
Claudio Atzori
|
5d08ad86ae
|
[raw_all] patching relation identifier phase to be run at the end, i.e. includes also claimed relations
|
2021-07-29 13:03:16 +02:00 |
Claudio Atzori
|
e87e1805c4
|
[raw_all] added extra workflow step for patching the identifiers in the relations, given an id mapping dataset
|
2021-07-29 12:13:06 +02:00 |
Claudio Atzori
|
5f7330d407
|
Merge branch 'master' into fct_project_id_replacement
|
2021-07-29 11:38:22 +02:00 |
Claudio Atzori
|
1923c1ce21
|
replaced full join + filtering with a left join
|
2021-07-29 11:36:20 +02:00 |
Claudio Atzori
|
a9961a1835
|
[cleaning] title cleaning based on the me.xuender:unidecode library
|
2021-07-28 16:36:33 +02:00 |
Claudio Atzori
|
e1797c0a42
|
Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta
|
2021-07-28 16:21:36 +02:00 |
Claudio Atzori
|
6dddad86ee
|
[cleaning] title cleaning based on the me.xuender:unidecode library
|
2021-07-28 16:21:29 +02:00 |
Alessia Bardi
|
c806387d4b
|
tests for enermaps
|
2021-07-28 11:54:36 +02:00 |
Claudio Atzori
|
2fff24df55
|
code formatting
|
2021-07-28 11:34:19 +02:00 |
Michele Artini
|
9f1c7b8e17
|
tests
|
2021-07-28 11:32:34 +02:00 |
Michele Artini
|
e6f1773d63
|
mapping of new eosc fields
|
2021-07-28 11:17:11 +02:00 |
Michele Artini
|
c72c960ffb
|
added eosc fields
|
2021-07-28 11:03:15 +02:00 |
Michele Artini
|
1fb572a33a
|
added eosc fields
|
2021-07-28 10:52:24 +02:00 |
Miriam Baglioni
|
708d0ade34
|
Merge branch 'beta' into hostedbymap
|
2021-07-28 10:37:22 +02:00 |
Miriam Baglioni
|
0424f47494
|
HostedByMap fixing issues
|
2021-07-28 10:24:13 +02:00 |
Claudio Atzori
|
d267dce520
|
[raw_all] added extra workflow step for patching the identifiers in the relations, given an id mapping dataset
|
2021-07-27 17:18:29 +02:00 |
Claudio Atzori
|
5aa7d16d1b
|
updated assertions in eu.dnetlib.dhp.oa.graph.raw.MappersTest
|
2021-07-27 15:11:58 +02:00 |
Claudio Atzori
|
998b66855a
|
updated assertions in eu.dnetlib.dhp.oa.graph.raw.MappersTest
|
2021-07-27 15:11:37 +02:00 |
Miriam Baglioni
|
74f801b689
|
mergin with branch beta
|
2021-07-27 13:18:31 +02:00 |
Miriam Baglioni
|
35e395eae8
|
merge with master
|
2021-07-27 12:34:59 +02:00 |
Miriam Baglioni
|
eb07f7f40f
|
Hosted By Map
|
2021-07-27 12:27:26 +02:00 |
Sandro La Bruzzo
|
848aabbb6c
|
minor fix
|
2021-07-25 12:06:41 +02:00 |
Sandro La Bruzzo
|
8fac10c91e
|
fixed defintion wf of creation final infospace of scholexplorer
|
2021-07-25 11:15:37 +02:00 |
Sandro La Bruzzo
|
3920c69bc8
|
change implementation of resolve Relation to generate jsonRdd in output
|
2021-07-25 09:51:36 +02:00 |
Sandro La Bruzzo
|
d9e3b89937
|
implemented last part of workflows to generate scholixGraph
|
2021-07-23 16:38:32 +02:00 |
Sandro La Bruzzo
|
cfde63a7c3
|
fixed resolve relation join
|
2021-07-23 14:17:29 +02:00 |
Sandro La Bruzzo
|
4a439c3863
|
NPE fixed
|
2021-07-23 14:17:29 +02:00 |
Sandro La Bruzzo
|
ca74e8dd02
|
create a separate wf for resolving relation
|
2021-07-23 11:40:06 +02:00 |
Sandro La Bruzzo
|
43e9380cd3
|
update resolve relation to use the same format of openaire graph
|
2021-07-23 11:25:18 +02:00 |
Sandro La Bruzzo
|
62ae36a3d2
|
fixed NPE
|
2021-07-22 15:41:38 +02:00 |
Sandro La Bruzzo
|
31d2d6d41e
|
Scholexplorer: introduction of dedup openaire
|
2021-07-21 18:09:32 +02:00 |
Alessia Bardi
|
9069958479
|
tests for enermaps
|
2021-07-20 19:31:43 +02:00 |
Claudio Atzori
|
65934888a1
|
adding record identifier among the originalIds regardless of what IdentifierFactory produces
|
2021-07-19 17:52:52 +02:00 |
Claudio Atzori
|
5947cddafc
|
adding record identifier among the originalIds regardless of what IdentifierFactory produces
|
2021-07-19 17:52:24 +02:00 |
Claudio Atzori
|
0977baf41d
|
contents mapped from the stores with 'claim' interpretation will not change their identifier along their way towards the graph
|
2021-07-19 17:43:52 +02:00 |
Claudio Atzori
|
5e5f65a3c3
|
contents mapped from the stores with 'claim' interpretation will not change their identifier along their way towards the graph
|
2021-07-19 15:56:55 +02:00 |
Sandro La Bruzzo
|
7e2caafe84
|
Scholexplorer: fixed mapping typologies
|
2021-07-15 09:53:12 +02:00 |
Miriam Baglioni
|
774cdb190e
|
changes to mirror the last dump of the graph with the ols data model.
|
2021-07-13 18:57:24 +02:00 |
Miriam Baglioni
|
886617afd0
|
One result linked to more than on project is saved just once
|
2021-07-13 18:15:35 +02:00 |
Miriam Baglioni
|
320cf02d96
|
Changed the way to find results linked to projects. We verify to actually have the project on the graph before selecting the result
|
2021-07-13 18:13:32 +02:00 |
Miriam Baglioni
|
52ce35d57b
|
-
|
2021-07-13 18:08:46 +02:00 |
Miriam Baglioni
|
970b387b8d
|
modification to allow dump of a single community
|
2021-07-13 18:08:10 +02:00 |
Miriam Baglioni
|
eae10c5894
|
modification to allow the dump for a single community
|
2021-07-13 18:07:25 +02:00 |
Miriam Baglioni
|
c028feef4f
|
workflow for the dump as sub workflows
|
2021-07-13 18:06:44 +02:00 |
Miriam Baglioni
|
d70f8c96fd
|
funding contains and not starts with h2020
|
2021-07-13 17:34:53 +02:00 |
Miriam Baglioni
|
5e38c7f42d
|
dumping only communities with status all
|
2021-07-13 17:32:38 +02:00 |
Miriam Baglioni
|
618d2de2da
|
minor changes and refactoring
|
2021-07-13 17:10:02 +02:00 |
Miriam Baglioni
|
59615da65e
|
Add test to verify the creation of relation between context and projects
|
2021-07-13 17:09:15 +02:00 |
Miriam Baglioni
|
084b4ef999
|
added the creation of the openaireId from funder and grant number if the element is not present in the context profile
|
2021-07-13 17:07:46 +02:00 |
Miriam Baglioni
|
8f322a73cb
|
change because of the renaming of originalId in acronym
|
2021-07-13 16:22:58 +02:00 |
Miriam Baglioni
|
72397ea1ba
|
Added fix for community of arbitrary name length
|
2021-07-13 16:18:35 +02:00 |
Miriam Baglioni
|
5295d10691
|
added check not to dump deletedByInference entities
|
2021-07-13 16:11:46 +02:00 |
Miriam Baglioni
|
e9a17ec899
|
added check to verify not to add void APC
|
2021-07-13 15:53:35 +02:00 |
Miriam Baglioni
|
8429aed6c6
|
Added resource for testing selection of valid relations
|
2021-07-13 15:49:38 +02:00 |
Miriam Baglioni
|
39b1a6edf6
|
added test class for the selection of valid relations and description
|
2021-07-13 15:23:09 +02:00 |
Miriam Baglioni
|
9a58f1b93d
|
added logic to select only the valid relations: those not deletedbyinference and having both part of the relation as entities in the graph
|
2021-07-13 15:20:39 +02:00 |
Miriam Baglioni
|
13c66e16be
|
changed logic to split for communities
|
2021-07-13 15:15:27 +02:00 |
Miriam Baglioni
|
6410ab71d8
|
added APC in the dump and test method
|
2021-07-13 15:13:58 +02:00 |
Miriam Baglioni
|
65a242646d
|
added resource for APC dump
|
2021-07-13 14:45:25 +02:00 |
Miriam Baglioni
|
4b432fbee8
|
extended test class
|
2021-07-13 14:40:39 +02:00 |
Miriam Baglioni
|
87a6e2b967
|
extended test class
|
2021-07-13 14:38:28 +02:00 |
Miriam Baglioni
|
69fd40fd30
|
modified code to split the Croatian funder
|
2021-07-13 14:35:26 +02:00 |
Miriam Baglioni
|
86e50f7311
|
modified code to split the Croatian funder
|
2021-07-13 14:31:45 +02:00 |
Miriam Baglioni
|
da88c850c6
|
changed the logic to verify if a community is contained in the list of context of a result
|
2021-07-13 14:22:44 +02:00 |
Miriam Baglioni
|
2f66fedfec
|
changed the logic to verify if a community is contained in the list of context of a result
|
2021-07-13 14:22:23 +02:00 |
Sandro La Bruzzo
|
bbe8193930
|
merged stable ids
|
2021-07-12 17:00:43 +02:00 |
Sandro La Bruzzo
|
09fccf8000
|
added workflow to serialize scholix and summary in json
|
2021-07-09 11:01:42 +02:00 |
Sandro La Bruzzo
|
0ea576745f
|
updated CreateInputGraph because ggenerics don't work on Spark Dataset
|
2021-07-09 10:29:24 +02:00 |
Sandro La Bruzzo
|
cd17e19044
|
implemented branch workflow to import datacite and crossref in scholexplorer
|
2021-07-08 21:20:19 +02:00 |
Sandro La Bruzzo
|
8a034e46e1
|
updated baseline workflow
|
2021-07-08 11:11:41 +02:00 |
Claudio Atzori
|
b7b8e0986e
|
[raw_all] The claim merge procedure includes the claimed contexts in the merged result
|
2021-07-08 10:42:31 +02:00 |
Sandro La Bruzzo
|
0799ac9fb6
|
fixed wrong path
|
2021-07-08 10:36:37 +02:00 |
Sandro La Bruzzo
|
4d53402712
|
extended ebiLinks to create a dataset before generation of OAF
|
2021-07-08 10:26:21 +02:00 |
Sandro La Bruzzo
|
a4a54a3786
|
code refactor
|
2021-07-08 09:08:25 +02:00 |
Sandro La Bruzzo
|
a01dbe0ab0
|
completed workflow of generation of scholix and summaries
|
2021-07-07 23:10:34 +02:00 |
Claudio Atzori
|
fdcff42e46
|
[raw_all] Aggregator graph creation merges claims (updates) with the corresponding entity
|
2021-07-07 19:01:59 +02:00 |
Claudio Atzori
|
bc014023c8
|
Merge pull request 'to solve the scala SI-3623' (#122) from andreas.czerniak/BrStableId_dnet-hadoop:stable_ids into stable_ids
Reviewed-on: #122
|
2021-07-07 11:13:51 +02:00 |
Claudio Atzori
|
32bdfdccbc
|
[raw_all] Aggregator graph creation merges claims (updates) with the corresponding entity
|
2021-07-07 11:08:27 +02:00 |
Claudio Atzori
|
f580cb77e1
|
added mapping for claim relation 'resultResult_publicationDataset_isRelatedTo' (present on BETA)
|
2021-07-06 21:11:11 +02:00 |
Sandro La Bruzzo
|
8535506c22
|
added scholix generation
|
2021-07-06 17:18:06 +02:00 |
Sandro La Bruzzo
|
4c54bd8742
|
add test to verify merge scholix on source
|
2021-07-06 11:32:14 +02:00 |
Andreas Czerniak
|
3531802710
|
to solve the scala SI-3623
|
2021-07-06 11:30:56 +02:00 |
Sandro La Bruzzo
|
7d8db2eb8a
|
betterRenamingMethod
|
2021-07-06 09:56:32 +02:00 |
Sandro La Bruzzo
|
c952c8d236
|
generate first side of scholix mapping
|
2021-07-06 09:53:14 +02:00 |
Sandro La Bruzzo
|
e4b84ef5d6
|
fixed mapping OAF to Scholix summary
|
2021-07-02 16:48:48 +02:00 |
Sandro La Bruzzo
|
c6fa8598e1
|
massive code refactor:
removed modules dhp-*-scholexplorer
|
2021-07-01 22:13:45 +02:00 |
Sandro La Bruzzo
|
84b834c893
|
added test dataset test for pangaea
|
2021-06-30 17:31:09 +02:00 |
Sandro La Bruzzo
|
1a6b398968
|
implemented Creation of Raw Graph and Resolution
|
2021-06-30 17:27:55 +02:00 |
Sandro La Bruzzo
|
623a0c4edb
|
code Refactor, renaming packages
|
2021-06-30 11:09:30 +02:00 |
Sandro La Bruzzo
|
7e08655e5f
|
added relation dates in all scholexplorer Datasources
|
2021-06-29 12:02:03 +02:00 |
Sandro La Bruzzo
|
075055eaca
|
added relation dates in bio mapping
|
2021-06-29 10:33:09 +02:00 |
Sandro La Bruzzo
|
f36f92287d
|
implemented mapping from Crossref Event Data to Oaf
|
2021-06-29 10:21:23 +02:00 |
Sandro La Bruzzo
|
511ec14c63
|
implemented mapping from EBI and Scholix Resolved to OAF
|
2021-06-28 22:04:22 +02:00 |
Sandro La Bruzzo
|
ad50415167
|
Merge remote-tracking branch 'origin/stable_ids' into stable_id_scholexplorer
|
2021-06-24 17:20:50 +02:00 |
Sandro La Bruzzo
|
80e15cc455
|
implemented mapping from uniprot, pdb and ebi links
|
2021-06-24 17:20:00 +02:00 |
Claudio Atzori
|
2e8fd2c531
|
cleanup
|
2021-06-23 14:38:24 +02:00 |
Claudio Atzori
|
4dc9ebf217
|
[raw_all] fixed unit test
|
2021-06-23 14:38:07 +02:00 |
Claudio Atzori
|
50fc5a64a0
|
[raw_all] Aggregator graph creation merges claims (updates) with the corresponding entity
|
2021-06-23 11:49:42 +02:00 |
Sandro La Bruzzo
|
080a280bea
|
added pdb to Oaf Transformation
|
2021-06-21 16:23:59 +02:00 |
Sandro La Bruzzo
|
507e42102a
|
added pdb to oaf class
|
2021-06-21 09:36:40 +02:00 |
Sandro La Bruzzo
|
4fe7b75644
|
renamed packages
|
2021-06-18 16:41:24 +02:00 |
Sandro La Bruzzo
|
3100166d29
|
Merge remote-tracking branch 'origin/stable_ids' into stable_id_scholexplorer
|
2021-06-16 16:22:16 +02:00 |
Claudio Atzori
|
7243a40c88
|
code formatting
|
2021-06-16 15:03:03 +02:00 |
Sandro La Bruzzo
|
dfcf78cf24
|
removed wrong code
|
2021-06-16 14:57:42 +02:00 |
Sandro La Bruzzo
|
cc0f2b11fb
|
Implemented mapping from pubmed baseline to OAF
|
2021-06-16 14:56:24 +02:00 |
Michele Artini
|
ada063ce70
|
fixed a problem with empty mdstore list (2)
|
2021-06-14 12:04:47 +02:00 |
Michele Artini
|
83132ee99a
|
fixed a problem with empty mdstore list
|
2021-06-14 11:57:00 +02:00 |
Sandro La Bruzzo
|
aeb8132627
|
Merged branch stable_ids
|
2021-06-14 10:07:29 +02:00 |
Claudio Atzori
|
2039bb9f5f
|
orcid / orcid_pending cleaning backported from master branch
|
2021-06-14 09:40:50 +02:00 |
Claudio Atzori
|
dd19c4ac5a
|
Merge pull request 'import_new_mdstores' (#112) from import_new_mdstores into stable_ids
Reviewed-on: #112
|
2021-06-14 09:23:55 +02:00 |
Claudio Atzori
|
a900bfb874
|
delegating the date parsing to https://github.com/sisyphsu/dateparser
|
2021-06-11 16:53:01 +02:00 |
Sandro La Bruzzo
|
5b724d9972
|
added relations to datacite mapping
|
2021-06-04 10:14:22 +02:00 |
Sandro La Bruzzo
|
e57294ac99
|
implemented changes on PUBMed dataflow
|
2021-06-03 10:52:09 +02:00 |
Michele Artini
|
ede2749822
|
orcid pid type
|
2021-06-01 12:42:43 +02:00 |
Michele Artini
|
f0fbfdcfae
|
Merge branch 'stable_ids' into import_new_mdstores
|
2021-06-01 12:03:00 +02:00 |
Michele Artini
|
e950750262
|
add nodes to import hdfs mdstores
|
2021-06-01 10:48:50 +02:00 |
Michele Artini
|
03a510859a
|
removed coalesce(1)
|
2021-05-31 14:10:51 +02:00 |
Michele Artini
|
e9f2b6037c
|
patch of mdstore records
|
2021-05-31 11:36:26 +02:00 |