14 Commits

Author	SHA1	Message	Date
miconis	c39c82dfe9	modification of the jobs for the integration of openorgs in the provision, dedup records are no more created by merging but simply taking results of openorgs portal	2021-04-06 14:31:00 +02:00
miconis	2355cc4e9b	minor changes and bug fix	2021-03-29 10:07:12 +02:00
miconis	98854b0124	minor changes	2021-03-19 16:57:40 +01:00
miconis	4b2124a18e	implementation of the openorgs wfs, implementation of the raw_all wf to migrate openorgs db entities	2021-02-10 11:51:50 +01:00
miconis	8fea29177c	refactoring, minor changes and implementation of the wf for openorgs with integration of organization phases into the scan wf	2021-01-18 16:48:08 +01:00
Claudio Atzori	8c67938ad0	configurable number of partitions used in the SparkCreateSimRels phase	2020-07-13 16:07:07 +02:00
Claudio Atzori	c79e2f5977	drop workingPath before starting the dedup workflow	2020-05-06 11:27:44 +02:00
Claudio Atzori	71813795f6	various refactorings on the dnet-dedup-openaire workflow	2020-04-18 12:06:23 +02:00
Claudio Atzori	038ac7afd7	relation consistency workflow separated from dedup scan and creation of CCs	2020-04-17 13:12:44 +02:00
Claudio Atzori	ec5dfc068d	added spark.sql.shuffle.partitions=3840 to dedup scan wf	2020-04-16 14:41:28 +02:00
miconis	9b36458b6a	Merge branch 'deduptesting' of code-repo.d4science.org:D-Net/dnet-hadoop into deduptesting	2020-04-16 12:13:58 +02:00
miconis	cd4d9a148f	creating temporary directories in dedup test	2020-04-16 12:13:26 +02:00
Claudio Atzori	b39ff36c16	improving the wf definitions	2020-04-16 12:11:37 +02:00
Claudio Atzori	673e744649	moved openaire specific implementations under dedicated package eu.dnetlib.dhp.oa	2020-03-27 10:42:17 +01:00