235 Commits

Author	SHA1	Message	Date
Enrico Ottonello	2dc50c0999	added default value to process path	2021-07-14 17:02:22 +02:00
Enrico Ottonello	66604bb2b4	added absolute path to process folder	2021-07-14 16:44:51 +02:00
Enrico Ottonello	7840cc6526	merged with master	2021-07-14 15:33:59 +02:00
Enrico Ottonello	a65667d217	added publication to dataset even if no contributors	2021-07-14 15:07:07 +02:00
Sandro La Bruzzo	10068c00ea	Code refactor: ... - removed old workflows in doiboost - splitted workflow of doiboost in preprocess and process	2021-07-14 14:45:50 +02:00
Sandro La Bruzzo	4cb65bc64a	fixed process doiboost workflow: ... - splitted OrcidToOAF into two phase preprocess and process - updated workflow used in production	2021-07-14 09:44:32 +02:00
Claudio Atzori	734de62474	[doiboost] added workflow for the ActionSet update dedicated to production	2021-07-13 17:26:04 +02:00
Claudio Atzori	fa720c1da4	[doiboost] added workflow for the ActionSet update dedicated to production	2021-07-13 16:59:30 +02:00
Sandro La Bruzzo	bbe8193930	merged stable ids	2021-07-12 17:00:43 +02:00
Miriam Baglioni	7177c25261	added check for null value during doi normalization	2021-07-05 16:22:38 +02:00
Miriam Baglioni	0892cad4e8	the normalization of the content of value was not visible outside the block. Moved doi normalization operation while returning value	2021-07-05 16:21:42 +02:00
Sandro La Bruzzo	c6fa8598e1	massive code refactor: ... removed modules dhp-*-scholexplorer	2021-07-01 22:13:45 +02:00
Miriam Baglioni	cf758f4f91	added normalization step for the doi	2021-06-30 10:03:15 +02:00
Miriam Baglioni	801763a0fa	there is no more the need to lower case the doi since it is done in the first step. Also changed the creation of the id by using the factory	2021-06-29 19:07:23 +02:00
Miriam Baglioni	a74de1cda2	added normalization step to the doi	2021-06-29 18:51:11 +02:00
Miriam Baglioni	06074ea7d3	added normalization step to the doi	2021-06-29 18:46:08 +02:00
Miriam Baglioni	8b8ffe82dc	added step of normalization for the doi	2021-06-29 18:41:39 +02:00
Miriam Baglioni	50cc21d92e	Added method to normalize doi values (lower case, remove all preceeding 10., filtering out doi not starting with 10.)	2021-06-29 18:35:28 +02:00
Sandro La Bruzzo	80e15cc455	implemented mapping from uniprot, pdb and ebi links	2021-06-24 17:20:00 +02:00
Sandro La Bruzzo	a167543637	Merge branch 'stable_ids' of code-repo.d4science.org:D-Net/dnet-hadoop into stable_id_scholexplorer	2021-06-21 09:14:11 +02:00
Miriam Baglioni	13c96622c9	-	2021-06-18 09:45:16 +02:00
Miriam Baglioni	464c2ddde3	changed to split in two steps the generation of the crossref dataset	2021-06-18 09:42:31 +02:00
Miriam Baglioni	6aca0d8ebb	added kryo encoding for input files	2021-06-18 09:42:07 +02:00
Miriam Baglioni	3585e53da3	changed to split in two steps the generation of the crossref dataset	2021-06-18 09:41:23 +02:00
Sandro La Bruzzo	3100166d29	Merge remote-tracking branch 'origin/stable_ids' into stable_id_scholexplorer	2021-06-16 16:22:16 +02:00
Miriam Baglioni	95885bcf12	forces executor Executor memory and driver executor memory to be 7G (trying to avoid OOM)	2021-06-16 10:17:52 +02:00
Miriam Baglioni	2550a73981	-	2021-06-16 10:04:41 +02:00
Miriam Baglioni	1c47c0d786	modified the number of executors trying to avoid OOM exception	2021-06-15 21:05:39 +02:00
Miriam Baglioni	7deac55138	added one option for resume from in the wf	2021-06-15 18:38:20 +02:00
Miriam Baglioni	66e7ef892f	changed the parameter name	2021-06-15 11:08:54 +02:00
Miriam Baglioni	4f47ad0891	no need to rename the folders, just write in overwrite mode, so I changed the name of the output folder	2021-06-15 09:28:31 +02:00
Miriam Baglioni	9f9dd00b94	refactoring	2021-06-15 09:24:46 +02:00
Miriam Baglioni	63d74ee379	refactoring	2021-06-15 09:24:11 +02:00
Miriam Baglioni	6ebc236657	added needed property: outputPath	2021-06-15 09:23:24 +02:00
Miriam Baglioni	f7379255b6	changed the workflow to extract info from the dump	2021-06-15 09:22:54 +02:00
Miriam Baglioni	d6e21bb6ea	creates the crossref dataset used for doiboost together with unpacking part from tar	2021-06-14 17:27:19 +02:00
Miriam Baglioni	ce0cfd79e0	creates the crossref dataset used for doiboost	2021-06-14 13:40:19 +02:00
Miriam Baglioni	93efe4de82	split the construction of crossref dataset in two parts. This one just unpacks the tar entries	2021-06-14 13:39:40 +02:00
Miriam Baglioni	8873e6b6d1	workflow and parameter	2021-06-14 10:15:57 +02:00
Miriam Baglioni	0f1acdf6b6	workflow and parameter	2021-06-14 10:08:55 +02:00
Sandro La Bruzzo	efbea1e01a	minor fix	2021-06-14 09:45:14 +02:00
Miriam Baglioni	75780fc636	extraction of the tar for the dump of crossref, and creation of the dataset	2021-06-14 09:45:07 +02:00
Miriam Baglioni	8d2e086e48	changes to avoid reassignment to val	2021-06-07 17:50:37 +02:00
Miriam Baglioni	f33521d338	Aggiornare 'dhp-workflows/dhp-doiboost/src/main/java/eu/dnetlib/doiboost/orcid/SparkConvertORCIDToOAF.scala' ... to be able to replace the aboject assigned to author val has been replaced by var	2021-06-07 17:27:07 +02:00
Miriam Baglioni	bc12e9819e	Aggiornare 'dhp-workflows/dhp-doiboost/src/main/java/eu/dnetlib/doiboost/orcid/SparkConvertORCIDToOAF.scala' ... The change is to fix the issue that arises when the same work appears more than once on the same ORCID profile. The change avoid to replicate the association doi -> author when the orcid id is already associated to the doi.	2021-06-07 16:37:01 +02:00
Claudio Atzori	5e4b91d9ef	more pervasive use of constants from ModelConstants, especially for ORCID	2021-05-26 18:20:23 +02:00
Enrico Ottonello	abdd0ade1f	added temporary output folder as workflow parameter	2021-05-21 12:08:16 +02:00
Enrico Ottonello	d0945c3c78	added temporary output folder, because of folder access rights are different on beta and prod	2021-05-20 19:14:31 +02:00
Enrico Ottonello	1265dadc90	workflow aligned with stable_ids	2021-05-20 19:01:28 +02:00
Enrico Ottonello	ae7bd24d79	removed old workflows	2021-05-20 18:32:22 +02:00