 Enrico Ottonello
|
a65667d217
|
added publication to dataset even if no contributors
|
2021-07-14 15:07:07 +02:00 |
|
Enrico Ottonello
|
abdd0ade1f
|
added temporary output folder as workflow parameter
|
2021-05-21 12:08:16 +02:00 |
|
Enrico Ottonello
|
d0945c3c78
|
added temporary output folder, because of folder access rights are different on beta and prod
|
2021-05-20 19:14:31 +02:00 |
|
Enrico Ottonello
|
1265dadc90
|
workflow aligned with stable_ids
|
2021-05-20 19:01:28 +02:00 |
|
Enrico Ottonello
|
ae7bd24d79
|
removed old workflows
|
2021-05-20 18:32:22 +02:00 |
|
Enrico Ottonello
|
e13926cdd0
|
merged with master
|
2021-05-14 18:10:31 +02:00 |
 Sandro La Bruzzo
|
d9a0bbda7b
|
implemented new phase in doiboost to make the dataset Distinct by ID
|
2021-05-13 12:25:14 +02:00 |
|
Sandro La Bruzzo
|
54217d73ff
|
removed old parameters from oozie workflow
|
2021-05-11 09:59:02 +02:00 |
|
Sandro La Bruzzo
|
7dc824fc23
|
imported changes in stable_id into master
|
2021-05-07 12:53:50 +02:00 |
|
Sandro La Bruzzo
|
1adfc41d23
|
merged manually changes on stable_id for doiboost into master
|
2021-05-05 10:23:32 +02:00 |
|
Enrico Ottonello
|
27068aacd1
|
wf to move orcid-no-doi dataset on the folder ready the import
|
2021-04-16 17:17:47 +02:00 |
|
Enrico Ottonello
|
59ec5137e1
|
improvement related to https://issue.openaire.research-infrastructures.eu/issues/6501
|
2021-03-31 16:25:41 +02:00 |
|
Enrico Ottonello
|
ebd67b8c8f
|
removed duplicates orcid data on authors set
|
2021-03-25 11:20:52 +01:00 |
|
Enrico Ottonello
|
70cb100647
|
added updating last orcid dataset folders after completion
|
2021-03-01 10:17:04 +01:00 |
|
Enrico Ottonello
|
bd3b16402b
|
added result typologies
|
2021-03-01 10:16:02 +01:00 |
|
Enrico Ottonello
|
53d7023460
|
dateOfCollection taken from orcid last_update.txt on hdfs; cleaned wf parameters
|
2021-02-25 18:43:29 +01:00 |
|
Enrico Ottonello
|
d43ea88caf
|
aligned orcid result typologies with openaire vocabulary
|
2021-02-25 15:02:10 +01:00 |
|
Enrico Ottonello
|
975823b968
|
data from last updated orcid
|
2021-02-23 15:35:04 +01:00 |
|
Enrico Ottonello
|
ee4ba7298b
|
fix last update read/write from file on hdfs
|
2021-02-09 23:24:57 +01:00 |
|
Enrico Ottonello
|
c238561001
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi
|
2021-02-04 10:44:21 +01:00 |
|
Enrico Ottonello
|
465ce39f75
|
job execution now based on file last_update.txt on hdfs
|
2021-02-04 10:44:04 +01:00 |
 Claudio Atzori
|
ab2fe9266a
|
[DOIBoost] minor fixes in workflow definition
|
2021-01-05 10:26:39 +01:00 |
|
Claudio Atzori
|
7c722f3fdc
|
[DOIBoost] fixed typo
|
2021-01-05 10:25:54 +01:00 |
|
Claudio Atzori
|
8879704ba0
|
[DOIBoost] configurable ES server url and index name in crossref importer
|
2021-01-05 10:00:13 +01:00 |
|
Sandro La Bruzzo
|
7834a35768
|
avoid to save intermediate dataset before generation of Sequence file
|
2021-01-04 17:54:57 +01:00 |
|
Sandro La Bruzzo
|
e79445a8b4
|
minor fix for claudio polemica
|
2021-01-04 17:39:25 +01:00 |
|
Sandro La Bruzzo
|
8765020b85
|
minor fix
|
2021-01-04 17:37:08 +01:00 |
|
Sandro La Bruzzo
|
b0dc92786f
|
defined a single oozie workflow for the generation of doiboost
|
2021-01-04 17:01:35 +01:00 |
|
Claudio Atzori
|
28460c2cd1
|
using com.fasterxml.jackson.databind.ObjectMapper instead of org.codehaus.jackson.map.ObjectMapper
|
2020-12-23 16:59:52 +01:00 |
|
Sandro La Bruzzo
|
1f6c8a9e83
|
added orcid_pending type to records coming from Crossref
|
2020-12-15 11:47:15 +01:00 |
|
Enrico Ottonello
|
b2de598c1a
|
all actions from download lambda file to merge updated data into one wf
|
2020-12-15 10:42:55 +01:00 |
|
Enrico Ottonello
|
efe4c2a9c5
|
authors and works are now updated in two separate spark actions of the wf
|
2020-12-12 02:06:21 +01:00 |
|
Enrico Ottonello
|
858efbfad1
|
fix dataset creation for downloaded works
|
2020-12-11 16:49:54 +01:00 |
|
Enrico Ottonello
|
2233750a37
|
original orcid xml data are stored in a field of the class that models orcid data
|
2020-12-09 09:45:19 +01:00 |
|
Sandro La Bruzzo
|
302baab67b
|
fixed doiboost mapping and workflows
|
2020-12-07 19:59:33 +01:00 |
|
Enrico Ottonello
|
5c65e602d3
|
wf doi_authors generates one json data foreach row
|
2020-12-07 15:28:10 +01:00 |
|
Enrico Ottonello
|
fa1855a4b8
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi
|
2020-12-07 11:02:59 +01:00 |
|
Enrico Ottonello
|
b1b589ada1
|
wf to generate orcid dataset
|
2020-12-07 11:02:32 +01:00 |
|
Sandro La Bruzzo
|
b31dd126fb
|
fixed crossref workflow added common ORCID Class
|
2020-12-07 10:42:38 +01:00 |
|
Enrico Ottonello
|
8812ab65e1
|
completed download function to wf; added accumulators
|
2020-12-04 21:13:49 +01:00 |
|
Enrico Ottonello
|
1b1e9ea67c
|
wf to generate doi_author_list for doiboost; wf to download updated works
|
2020-12-02 23:20:16 +01:00 |
|
Sandro La Bruzzo
|
7da679542f
|
fixed wrong projectId
|
2020-12-02 14:28:09 +01:00 |
|
Claudio Atzori
|
cfb55effd9
|
code formatting
|
2020-12-02 11:23:49 +01:00 |
|
Enrico Ottonello
|
f2df3ead74
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop into orcid-no-doi
|
2020-11-30 14:22:46 +01:00 |
|
Enrico Ottonello
|
40c4559e92
|
added datainfo on authors pid with "sysimport:crosswalk:entityregistry",
|
2020-11-30 14:19:22 +01:00 |
|
Claudio Atzori
|
db0181b8af
|
Merge pull request 'added bidirectionality to relations from project and result coming from crossref' (#60) from miriam.baglioni/dnet-hadoop:sxBidirectionality into master
|
2020-11-25 17:17:40 +01:00 |
|
Sandro La Bruzzo
|
ec3e238de6
|
Fixed problem on duplicated identifier
|
2020-11-25 17:15:54 +01:00 |
|
Enrico Ottonello
|
99a086f0c6
|
max concurrent executors set to 10, according to ORCID Director of Technology mail request
|
2020-11-24 17:49:32 +01:00 |
 Miriam Baglioni
|
00874a8ce6
|
added bidirectionality to relations from project and result
|
2020-11-24 15:17:23 +01:00 |
|
Enrico Ottonello
|
5c17e768b2
|
set wf configuration with spark.dynamicAllocation.maxExecutors 20 over 20 input partitions
|
2020-11-23 16:01:23 +01:00 |