 Enrico Ottonello
|
e13926cdd0
|
merged with master
|
2021-05-14 18:10:31 +02:00 |
 Sandro La Bruzzo
|
d9a0bbda7b
|
implemented new phase in doiboost to make the dataset Distinct by ID
|
2021-05-13 12:25:14 +02:00 |
|
Sandro La Bruzzo
|
7dc824fc23
|
imported changes in stable_id into master
|
2021-05-07 12:53:50 +02:00 |
|
Sandro La Bruzzo
|
1adfc41d23
|
merged manually changes on stable_id for doiboost into master
|
2021-05-05 10:23:32 +02:00 |
|
Enrico Ottonello
|
59ec5137e1
|
improvement related to https://issue.openaire.research-infrastructures.eu/issues/6501
|
2021-03-31 16:25:41 +02:00 |
|
Enrico Ottonello
|
ebd67b8c8f
|
removed duplicates orcid data on authors set
|
2021-03-25 11:20:52 +01:00 |
|
Enrico Ottonello
|
bd3b16402b
|
added result typologies
|
2021-03-01 10:16:02 +01:00 |
|
Enrico Ottonello
|
53d7023460
|
dateOfCollection taken from orcid last_update.txt on hdfs; cleaned wf parameters
|
2021-02-25 18:43:29 +01:00 |
|
Enrico Ottonello
|
d43ea88caf
|
aligned orcid result typologies with openaire vocabulary
|
2021-02-25 15:02:10 +01:00 |
|
Enrico Ottonello
|
975823b968
|
data from last updated orcid
|
2021-02-23 15:35:04 +01:00 |
|
Enrico Ottonello
|
ee4ba7298b
|
fix last update read/write from file on hdfs
|
2021-02-09 23:24:57 +01:00 |
|
Enrico Ottonello
|
c238561001
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi
|
2021-02-04 10:44:21 +01:00 |
|
Enrico Ottonello
|
465ce39f75
|
job execution now based on file last_update.txt on hdfs
|
2021-02-04 10:44:04 +01:00 |
 Claudio Atzori
|
8879704ba0
|
[DOIBoost] configurable ES server url and index name in crossref importer
|
2021-01-05 10:00:13 +01:00 |
|
Sandro La Bruzzo
|
7834a35768
|
avoid to save intermediate dataset before generation of Sequence file
|
2021-01-04 17:54:57 +01:00 |
|
Sandro La Bruzzo
|
8765020b85
|
minor fix
|
2021-01-04 17:37:08 +01:00 |
|
Sandro La Bruzzo
|
b0dc92786f
|
defined a single oozie workflow for the generation of doiboost
|
2021-01-04 17:01:35 +01:00 |
|
Claudio Atzori
|
28460c2cd1
|
using com.fasterxml.jackson.databind.ObjectMapper instead of org.codehaus.jackson.map.ObjectMapper
|
2020-12-23 16:59:52 +01:00 |
|
Sandro La Bruzzo
|
1f6c8a9e83
|
added orcid_pending type to records coming from Crossref
|
2020-12-15 11:47:15 +01:00 |
|
Enrico Ottonello
|
b2de598c1a
|
all actions from download lambda file to merge updated data into one wf
|
2020-12-15 10:42:55 +01:00 |
|
Enrico Ottonello
|
efe4c2a9c5
|
authors and works are now updated in two separate spark actions of the wf
|
2020-12-12 02:06:21 +01:00 |
|
Enrico Ottonello
|
858efbfad1
|
fix dataset creation for downloaded works
|
2020-12-11 16:49:54 +01:00 |
|
Enrico Ottonello
|
2233750a37
|
original orcid xml data are stored in a field of the class that models orcid data
|
2020-12-09 09:45:19 +01:00 |
|
Sandro La Bruzzo
|
302baab67b
|
fixed doiboost mapping and workflows
|
2020-12-07 19:59:33 +01:00 |
|
Enrico Ottonello
|
5c65e602d3
|
wf doi_authors generates one json data foreach row
|
2020-12-07 15:28:10 +01:00 |
|
Enrico Ottonello
|
fa1855a4b8
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi
|
2020-12-07 11:02:59 +01:00 |
|
Enrico Ottonello
|
b1b589ada1
|
wf to generate orcid dataset
|
2020-12-07 11:02:32 +01:00 |
|
Sandro La Bruzzo
|
b31dd126fb
|
fixed crossref workflow added common ORCID Class
|
2020-12-07 10:42:38 +01:00 |
|
Enrico Ottonello
|
8812ab65e1
|
completed download function to wf; added accumulators
|
2020-12-04 21:13:49 +01:00 |
|
Enrico Ottonello
|
1b1e9ea67c
|
wf to generate doi_author_list for doiboost; wf to download updated works
|
2020-12-02 23:20:16 +01:00 |
|
Sandro La Bruzzo
|
7da679542f
|
fixed wrong projectId
|
2020-12-02 14:28:09 +01:00 |
|
Claudio Atzori
|
cfb55effd9
|
code formatting
|
2020-12-02 11:23:49 +01:00 |
|
Enrico Ottonello
|
f2df3ead74
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi
|
2020-11-30 14:22:46 +01:00 |
|
Enrico Ottonello
|
40c4559e92
|
added datainfo on authors pid with "sysimport:crosswalk:entityregistry",
|
2020-11-30 14:19:22 +01:00 |
|
Claudio Atzori
|
db0181b8af
|
Merge pull request 'added bidirectionality to relations from project and result coming from crossref' (#60) from miriam.baglioni/dnet-hadoop:sxBidirectionality into master
|
2020-11-25 17:17:40 +01:00 |
|
Sandro La Bruzzo
|
ec3e238de6
|
Fixed problem on duplicated identifier
|
2020-11-25 17:15:54 +01:00 |
|
Enrico Ottonello
|
99a086f0c6
|
max concurrent executors set to 10, according to ORCID Director of Technology mail request
|
2020-11-24 17:49:32 +01:00 |
 Miriam Baglioni
|
00874a8ce6
|
added bidirectionality to relations from project and result
|
2020-11-24 15:17:23 +01:00 |
|
Enrico Ottonello
|
5c17e768b2
|
set wf configuration with spark.dynamicAllocation.maxExecutors 20 over 20 input partitions
|
2020-11-23 16:01:23 +01:00 |
|
Enrico Ottonello
|
97c8111847
|
action to convert lambda file in seq file; spark action to download updated authors
|
2020-11-23 09:49:22 +01:00 |
|
Enrico Ottonello
|
c0c2e05eae
|
added wf to extracting authors and works xml data from orcid dump to hdfs; added wf to download the lamda file (containing last orcid update informations) from orcid to hdfs
|
2020-11-17 18:23:12 +01:00 |
|
Enrico Ottonello
|
005f849674
|
added compression to output dataset
|
2020-11-13 12:45:31 +01:00 |
|
Enrico Ottonello
|
13f28fa225
|
moved AuthorData to dhp-schemas; added other names to author data
|
2020-11-12 17:43:32 +01:00 |
|
Enrico Ottonello
|
1f861f2b0d
|
now wf output is a sequence file with the format seq("eu.dnetlib.dhp.schema.oaf.Publication",eu.dnetlib.dhp.schema.action.AtomicActions)
|
2020-11-11 17:38:50 +01:00 |
|
Enrico Ottonello
|
1513174d7e
|
added further test case
|
2020-11-10 11:44:55 +01:00 |
|
Enrico Ottonello
|
6bc7dbeca7
|
first version of dataset successful generated from orcid dump 2020
|
2020-11-06 13:47:50 +01:00 |
|
Enrico Ottonello
|
9818e74a70
|
added dependency version in main pom.xml for orcid no doi
|
2020-10-22 16:38:00 +02:00 |
|
Enrico Ottonello
|
210a50e4f4
|
replaced null value
|
2020-10-22 16:24:42 +02:00 |
|
Enrico Ottonello
|
1139d6568d
|
replaced null value with a more safe empty string as return value
|
2020-10-22 15:32:26 +02:00 |
|
Enrico Ottonello
|
c58db1c8ea
|
added filter on null value after map function
|
2020-10-22 15:11:02 +02:00 |