1199 Commits

Author	SHA1	Message	Date
Michele Artini	1bc9adc10d	default trust for validated rels	2020-12-09 16:18:37 +01:00
Michele Artini	5f21a356fd	reindent	2020-12-09 11:24:30 +01:00
Michele Artini	370a5e650b	validation attributes in resultProject relations	2020-12-09 11:18:26 +01:00
Claudio Atzori	a104a632df	cleanup	2020-12-04 16:32:47 +01:00
Miriam Baglioni	5fb65ffc4a	merge branch with master	2020-12-03 11:24:35 +01:00
Miriam Baglioni	ea88dc3401	fixed issue in property name	2020-12-03 11:24:23 +01:00
Claudio Atzori	cfb55effd9	code formatting	2020-12-02 11:23:49 +01:00
Claudio Atzori	57f448b7a4	graph cleaning workflow separate orcid_pending from orcid, depending on the author pid provenance	2020-12-02 10:44:05 +01:00
Alessia Bardi	a417624670	tests for raw graph mapping	2020-12-02 10:15:26 +01:00
Claudio Atzori	893ac4a77b	GenerateEntitiesApplication can be configured to hash the id value or not	2020-12-02 09:30:06 +01:00
Claudio Atzori	2c407e775e	GenerateEntitiesApplication can be configured to hash the id value or not	2020-11-30 12:00:38 +01:00
Claudio Atzori	e731a7658d	cleaning texts to remove tab characters too	2020-11-27 09:00:04 +01:00
Claudio Atzori	c1b9a4045a	grouping of records will be performed by the dedup workflow	2020-11-26 10:59:10 +01:00
Miriam Baglioni	124591a7f3	refactoring	2020-11-25 18:23:28 +01:00
Miriam Baglioni	1a89f8211c	#61 (comment)	2020-11-25 18:12:40 +01:00
Miriam Baglioni	5fbe54ef54	#61 (comment)	2020-11-25 18:10:28 +01:00
Miriam Baglioni	ed01e5a5e1	#61 (comment)	2020-11-25 18:09:34 +01:00
Miriam Baglioni	d4ddde2ef2	changed because of #61 (comment)	2020-11-25 18:01:01 +01:00
Miriam Baglioni	f5e5e92a10	changed because of #61 (comment)	2020-11-25 17:58:53 +01:00
Miriam Baglioni	1df94b85b4	changed because of #61 (comment)	2020-11-25 17:57:43 +01:00
Claudio Atzori	dfd6205b95	Consistency graph workflow merges all the entities by ID	2020-11-25 14:55:32 +01:00
Miriam Baglioni	90d4369fd2	added test to verify the compression in writing community info on hdfs	2020-11-25 14:34:58 +01:00
Miriam Baglioni	6750e33d69	merge branch with master	2020-11-25 14:09:01 +01:00
Miriam Baglioni	b2c455f883	added java doc	2020-11-25 14:08:09 +01:00
Miriam Baglioni	1f130cdf92	changed the relation (produces -> isProducedBy) due to the change in the code	2020-11-25 14:04:26 +01:00
Miriam Baglioni	e758d5d9b4	refactoring	2020-11-25 13:46:39 +01:00
Miriam Baglioni	87a9f616ae	refactoring and addition of the funder nsp first part as nome for the dump insteasd of the whole nsp	2020-11-25 13:45:41 +01:00
Miriam Baglioni	e7e418e444	added decision node to verify if to upload in Zenodo	2020-11-25 13:44:10 +01:00
Miriam Baglioni	305e3d0c9c	added resource file for relation with relClass = isProducedBy	2020-11-25 13:43:41 +01:00
Miriam Baglioni	21ce175d17	added FilterFunction specification if filter operation	2020-11-25 13:42:31 +01:00
Miriam Baglioni	bde6d337dd	test classes for dump of results related to funders	2020-11-25 13:42:01 +01:00
Miriam Baglioni	b37b9352d7	added constant value for semantic relationship between projects and results	2020-11-25 13:41:08 +01:00
Claudio Atzori	36173c13a5	reverted filters in the clening process	2020-11-25 10:24:42 +01:00
Claudio Atzori	eeebd5a920	Cleanig workflow: remove newlines from titles, descriptions, subjects	2020-11-24 18:40:25 +01:00
Claudio Atzori	e1a1bb3ee4	moved class CleaningFunctions in the correct package. Remove newlines from titles, descriptions, subjects	2020-11-24 18:34:03 +01:00
Miriam Baglioni	72bb0fe360	changed directory name	2020-11-24 16:47:07 +01:00
Miriam Baglioni	39f4a20873	chenged the path and the name for saving the communities_infrastructures dump file	2020-11-24 14:47:32 +01:00
Miriam Baglioni	7e14452a87	final versione of the wf to get the dump of results associated to at least one funder per funder	2020-11-24 14:46:34 +01:00
Miriam Baglioni	c167a18057	added new parameter for the dumpType	2020-11-24 14:45:50 +01:00
Miriam Baglioni	54a309bb6b	refactoring	2020-11-24 14:45:30 +01:00
Miriam Baglioni	35ecea8842	changed to consider the modification for the specification of the type of dump	2020-11-24 14:45:15 +01:00
Miriam Baglioni	b9b6bdb2e6	fixing issue on previous implementation	2020-11-24 14:44:53 +01:00
Miriam Baglioni	7e940f1991	changed to consider the modification for the specification of the type of dump	2020-11-24 14:43:34 +01:00
Miriam Baglioni	62928ef7a5	changed to save the communities_infrastructures information as the other entity dumps: in a json.gz file	2020-11-24 14:42:41 +01:00
Claudio Atzori	33bae02451	reverted behaviour of the cleaning workflow: grouping entities by ID will be managed differently	2020-11-24 14:42:33 +01:00
Miriam Baglioni	3319440c53	changed the direction of the relation between projects and result considered to select the results linked to projects	2020-11-24 14:41:09 +01:00
Miriam Baglioni	00c377dac2	added specification of MapFunction types in map	2020-11-24 14:40:22 +01:00
Miriam Baglioni	44db258dc4	added enumerated for the dump type	2020-11-24 14:38:06 +01:00
Miriam Baglioni	1832708c42	modified boolean variable with string one whcih specify the type of dump we are performing: complete, community or funder	2020-11-24 14:37:36 +01:00
Miriam Baglioni	259c67ce36	fixed issue in path name	2020-11-20 12:32:23 +01:00
Miriam Baglioni	0a9db67eec	-	2020-11-20 12:21:33 +01:00
Miriam Baglioni	d362f2637d	merge branch with master	2020-11-19 19:17:20 +01:00
Miriam Baglioni	cf3f47563f	new parameter files	2020-11-19 19:16:05 +01:00
Miriam Baglioni	24c56fa7a3	new logic and workflow for dump of results with link to projects. In this implementation the result match the model of the communityresult.	2020-11-19 19:15:39 +01:00
Claudio Atzori	fcbb05eb21	cleanup	2020-11-19 15:14:33 +01:00
Claudio Atzori	3f34757c63	merged from master	2020-11-19 14:34:54 +01:00
Miriam Baglioni	fafb688887	-	2020-11-18 18:56:48 +01:00
Miriam Baglioni	906db690d2	-	2020-11-18 17:43:08 +01:00
Claudio Atzori	ede7fae6c8	Merge pull request 'XML record indexing test' (#58) from provision_indexing into master	2020-11-18 17:04:34 +01:00
Miriam Baglioni	5402062ff5	changed parameter file with the ono associated to the job	2020-11-18 16:58:20 +01:00
Miriam Baglioni	a172a37ad1	fixed typo	2020-11-18 16:55:07 +01:00
Miriam Baglioni	46ba3793f6	code, workflow and parameters for the dump of the results associated to funders	2020-11-18 16:47:31 +01:00
Miriam Baglioni	57cac36898	changed the workflow name	2020-11-18 13:38:03 +01:00
Claudio Atzori	8177ce7939	test for XmlIndexingJob based on a local miniSolrCluster	2020-11-18 10:58:05 +01:00
Alessia Bardi	10e673660f	Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop	2020-11-18 10:01:23 +01:00
Alessia Bardi	be7b310cef	rel semantcis ignore case	2020-11-18 10:01:20 +01:00
Michele Artini	33da2e3d6c	xpaths for dateOfCollection and dateOfTransformation	2020-11-18 09:26:20 +01:00
Alessia Bardi	8f87020a50	#56: map relevantDates from aggregated ODF records	2020-11-17 18:42:09 +01:00
Alessia Bardi	7e0a76a8ac	test fr TextGrid	2020-11-17 18:39:25 +01:00
Claudio Atzori	cfc01f136e	PID filtering based on a blacklist	2020-11-17 12:27:06 +01:00
Claudio Atzori	6ab1ce53c9	fixed condition in result pid cleaning; cleanup	2020-11-16 10:09:17 +01:00
Claudio Atzori	4de8c8b237	fixed workflow variable name	2020-11-16 10:03:11 +01:00
Claudio Atzori	331d621800	added test resource	2020-11-14 12:16:15 +01:00
Claudio Atzori	5d4e34e26a	fixed typo in variable name	2020-11-14 10:32:26 +01:00
Claudio Atzori	768bc5304c	Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop	2020-11-13 15:40:34 +01:00
Claudio Atzori	93f7b7974f	Merge pull request 'trust truncated to 3 decimals' (#24) from trunc_trust into master ... LGTM	2020-11-13 15:40:02 +01:00
Claudio Atzori	528231a287	grouping graph entities by id turned out to be an easy extension for the already existing cleaning workflow	2020-11-13 15:37:48 +01:00
Claudio Atzori	2bed29eb09	WIP: added oozie workflow for grouping graph entities by id	2020-11-13 10:05:12 +01:00
Claudio Atzori	13e36a4da0	WIP: added oozie workflow for grouping graph entities by id	2020-11-13 10:05:02 +01:00
Claudio Atzori	9b0fb9e958	merged from master	2020-11-12 09:27:12 +01:00
Michele Artini	40160d171f	organizations pids	2020-11-09 12:58:36 +01:00
Sandro La Bruzzo	027ef2326c	Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop	2020-11-06 17:12:42 +01:00
Sandro La Bruzzo	cd27df91a1	fixed bug on missing relation in ANDS	2020-11-06 17:12:31 +01:00
Claudio Atzori	d10447e747	re-packaged graph dump workflow sources	2020-11-05 17:38:18 +01:00
Claudio Atzori	2d76497488	cleanup	2020-11-05 17:10:24 +01:00
Miriam Baglioni	f8e9bda24c	merge branch with master	2020-11-05 16:31:18 +01:00
Miriam Baglioni	be5ed8f554	added check to avoid sending empty metadata.	2020-11-05 16:10:17 +01:00
Claudio Atzori	2148a51fae	minor changes	2020-11-05 11:24:12 +01:00
Claudio Atzori	4625b7486e	code formatting	2020-11-04 18:12:43 +01:00
Miriam Baglioni	e9ac471ae9	removed dependency from classes for the pid graph dump	2020-11-04 18:04:42 +01:00
Miriam Baglioni	b90a945c49	removed property files for pid graph dump	2020-11-04 17:28:33 +01:00
Miriam Baglioni	bac307155a	removed properties specific for pid graph dump	2020-11-04 17:28:04 +01:00
Miriam Baglioni	9c9d50f486	removed code specific for pid graph dump	2020-11-04 17:26:22 +01:00
Miriam Baglioni	5669890934	removed commented lines	2020-11-04 17:15:21 +01:00
Miriam Baglioni	6a89f59be9	removed commented lines	2020-11-04 17:13:59 +01:00
Miriam Baglioni	56150d7e5e	removed all code related to the dump of pids graph	2020-11-04 17:13:12 +01:00
Miriam Baglioni	16c54a96f8	removed pid dump	2020-11-04 17:11:32 +01:00
Miriam Baglioni	0cac5436ff	Merge branch 'dump' of code-repo.d4science.org:miriam.baglioni/dnet-hadoop into dump	2020-11-04 13:21:11 +01:00
Alessia Bardi	51808b5afd	Updated descriptions	2020-11-04 12:29:48 +01:00
Alessia Bardi	e6becf8659	Updated descriptions	2020-11-04 12:17:57 +01:00
Alessia Bardi	0abe0eee33	Updated descriptions	2020-11-04 12:15:30 +01:00
Alessia Bardi	f6ab238f5d	Updated descriptions	2020-11-04 11:50:47 +01:00
Miriam Baglioni	c010a8442f	fixed issue on test code	2020-11-03 17:26:51 +01:00
Miriam Baglioni	8ec7a61188	merge branch with master	2020-11-03 16:59:08 +01:00
Miriam Baglioni	c209284ca7	new schemas for the entities in the dump with added descriptions	2020-11-03 16:58:08 +01:00
Miriam Baglioni	08806deddf	added the splitSize non mandatory parameter. Default size 10G	2020-11-03 16:57:34 +01:00
Miriam Baglioni	7d2eda43ca	added new non mandatory property publish to determine if to publish the upload or leave it pending. Default value flase	2020-11-03 16:57:01 +01:00
Miriam Baglioni	cbbb1bdc54	moved business logic to new class in common for handling the zip of hte archives	2020-11-03 16:55:50 +01:00
Miriam Baglioni	d4382b54df	moved the tar archive with maz size on common module	2020-11-03 16:54:50 +01:00
Claudio Atzori	86d6fbe95b	refactoring: CleaningFunctions and OafMapperUtils moved in dhp-commong	2020-11-03 12:19:46 +01:00
Claudio Atzori	8471888ad3	Merge branch 'graph_cleaning' into stable_ids	2020-11-03 11:52:47 +01:00
Claudio Atzori	5310e56dba	remove empy PIDs	2020-11-03 11:52:10 +01:00
Claudio Atzori	3fcd669e99	result merge operation leverage on custom ResultTypeComparator in the aggregator graph construction	2020-11-03 10:53:23 +01:00
Claudio Atzori	09e44dabff	Merge branch 'master' into stable_ids	2020-11-02 12:16:01 +01:00
Sandro La Bruzzo	754c86f33e	fixed test to work on jenkins	2020-11-02 09:35:01 +01:00
Miriam Baglioni	dabb33e018	changed the discriminant for which split the file	2020-10-30 17:52:22 +01:00
Miriam Baglioni	0fba08eae4	max allowed size per file 10 Gb	2020-10-30 16:05:55 +01:00
Claudio Atzori	4ca75d6951	Merge pull request 'Dedup ID creation policy' (#48) from deduptesting into stable_ids	2020-10-30 15:15:32 +01:00
Miriam Baglioni	b828587252	prevent the code to cicle indefinetly	2020-10-30 15:01:25 +01:00
Miriam Baglioni	f747e303ac	classes for dumping of the graph as ttl file	2020-10-30 14:13:45 +01:00
Miriam Baglioni	16baf5b69e	formatting	2020-10-30 14:13:14 +01:00
Miriam Baglioni	a9eef9c852	added check for possible Optional value in relation dataInfo	2020-10-30 14:12:28 +01:00
Miriam Baglioni	5f4de9a962	formatting	2020-10-30 14:11:40 +01:00
Miriam Baglioni	14bf2e7238	added option to split dumps bigger that 40Gb on different files	2020-10-30 14:09:04 +01:00
Claudio Atzori	58f28296ea	ProvisionConstants moved as ModelHardLimits in dhp-common and applied to truncate long abstracts (len > 150000). Further filtering for empty PID values	2020-10-30 10:56:42 +01:00
Miriam Baglioni	78fdb11c3f	merge branch with master	2020-10-29 12:55:22 +01:00
Sandro La Bruzzo	1d9fdb7367	fixed spark memory issue in SparkSplitOafTODLIEntities	2020-10-28 12:30:32 +01:00
Miriam Baglioni	d2374e3b9e	added code to handle cases where the funding tree is not existing	2020-10-27 16:15:21 +01:00
Miriam Baglioni	5d3012eeb4	changed code to dump only the programme list and not the classification list	2020-10-27 16:14:18 +01:00
Miriam Baglioni	3241ec1777	added connection timeout and socket timeout 600 sec	2020-10-27 16:12:11 +01:00
Alessia Bardi	1425d810a8	testing mapping	2020-10-19 17:46:14 +02:00
Claudio Atzori	266bf1a221	common IdentifierFactory in use on the mapping from the aggregator data; merge the entities sharing the same id; code formatting	2020-10-16 17:02:10 +02:00
Claudio Atzori	34f1d0904b	common IdentifierFactory in use on the mapping from the aggregator data	2020-10-16 16:00:19 +02:00
Sandro La Bruzzo	fed711da80	Merge remote-tracking branch 'origin/master' into merge_record_to_common	2020-10-13 15:32:45 +02:00
Alessia Bardi	8775a64bc1	Merge pull request 'Merging different compatibility levels (pinocchio operator)' (#47) from merge_graph into master	2020-10-09 14:44:52 +02:00
Claudio Atzori	e751c1402f	Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop	2020-10-09 13:53:21 +02:00
Claudio Atzori	b961dc7d1e	added originalid to the fields in the result graph view	2020-10-09 13:53:15 +02:00
Sandro La Bruzzo	eec418cd26	moved AuthoreMerger into dhp-common	2020-10-08 10:33:55 +02:00
Sandro La Bruzzo	fe0a7870e6	Added test to check if merge authors works	2020-10-08 10:33:12 +02:00
Sandro La Bruzzo	cd9c377d18	adpted scholexplorer Dump generation to the new Dataset definition	2020-10-08 10:10:13 +02:00
Claudio Atzori	a3f37a9414	javadoc	2020-10-07 16:44:22 +02:00
Claudio Atzori	8d85a2fced	[BETA wf only] datasources involved in the merge operation doesn't obey to the infra precedence policy, but relies on a custom behaviour that, given two datasources from beta and prod returns the one from prod with the highest compatibility among the two	2020-10-07 16:28:52 +02:00
Miriam Baglioni	ae08b3c0dd	merge branch with master	2020-10-05 11:35:55 +02:00
Miriam Baglioni	11b7eaae09	changed the name of the folder where to store the context entity from context to communities_infrastructures	2020-10-05 11:24:54 +02:00
Miriam Baglioni	32bffb0134	changed the name from communities_infrastructures to communities_infrastuctures.json	2020-10-05 11:24:17 +02:00
Miriam Baglioni	25cbcf6114	changed to solve issues about names. context renamed communities_infrastructure.json and removed the double json.gz extention to the name of the part in the tar	2020-10-02 12:17:46 +02:00
Claudio Atzori	49ae3450a9	code formatting	2020-10-02 09:43:24 +02:00
Claudio Atzori	c2a6e2a9bf	fixed mapping for datasource journal info (ISSNs)	2020-10-02 09:37:08 +02:00
Miriam Baglioni	01117a46e1	whole workflow activated	2020-10-01 17:19:21 +02:00
Miriam Baglioni	cfb5766c6b	removed double json.gz from names of files in the tar	2020-10-01 17:18:34 +02:00
Miriam Baglioni	fcaedac980	merge branch with master	2020-10-01 16:46:59 +02:00
Miriam Baglioni	c6e6ed1bd8	merge branch with master	2020-10-01 16:24:41 +02:00
Claudio Atzori	2e9e13444d	author pids made unique by value	2020-10-01 12:50:40 +02:00
Claudio Atzori	e265c3e125	cleaning functions factored out in a dedicated class	2020-10-01 10:50:15 +02:00
Claudio Atzori	4287164aba	include relevantdate field in the result view	2020-10-01 10:28:55 +02:00
Miriam Baglioni	7b6a7333e6	merge branch with master	2020-09-25 16:42:07 +02:00
Miriam Baglioni	983a12ed15	temporary modification to allow the upload of files in the sandbox without the neew to recreate the mapping from scratch	2020-09-25 16:41:51 +02:00
Miriam Baglioni	8b36d19182	added property depositionId and chenage property newVersion that became string from boolean to handle the three possible distinct values	2020-09-25 16:41:15 +02:00
Miriam Baglioni	ed5239f9ec	added new code to handle the new possibility to upload files to an already open deposition	2020-09-25 16:34:32 +02:00
Miriam Baglioni	3a8c524fce	refactor	2020-09-25 16:34:02 +02:00
Miriam Baglioni	54800fb9b0	enabled only the step to upload in zenodo	2020-09-25 14:40:22 +02:00
Miriam Baglioni	de6c4d46d8	fixed conflicts	2020-09-24 15:35:01 +02:00
Claudio Atzori	044d3a0214	fixed query used to load datasources in the Graph	2020-09-24 13:48:58 +02:00
Claudio Atzori	27df1cea6d	code formatting	2020-09-24 12:16:00 +02:00
Claudio Atzori	fb22f4d70b	included values for projects fundedamount and totalcost fields in the mapping tests. Swapped expected and actual values in junit test assertions	2020-09-24 12:10:59 +02:00
Claudio Atzori	42f55395c8	fixed order of the ISSNs returned by the SQL query	2020-09-24 12:09:58 +02:00
Claudio Atzori	9a7e72d528	using concat_ws to join textual columns from PSQL. When using || to perform the concatenation, Null columns makes the operation result to be Null	2020-09-24 10:42:47 +02:00
Claudio Atzori	9e3e93c6b6	setting the correct issn type in the datasource.journal element	2020-09-24 10:39:16 +02:00
Miriam Baglioni	39eb8ab25b	changed the dump to move from h2020programme to h2020classification	2020-09-23 17:33:00 +02:00
Miriam Baglioni	c2b5c780ff	-	2020-09-14 14:34:03 +02:00
Miriam Baglioni	e2ceefe9be	-	2020-09-14 14:33:28 +02:00
Miriam Baglioni	1f893e63dc	-	2020-09-14 14:33:10 +02:00
Claudio Atzori	8a523474b7	code formatting	2020-09-07 11:40:16 +02:00
Miriam Baglioni	b72a7dad46	resuorce for pid graph dump	2020-08-24 17:09:01 +02:00
Miriam Baglioni	8694bb9b31	refactoring due to compilation	2020-08-24 17:07:34 +02:00
Miriam Baglioni	8a069a4fea	-	2020-08-24 17:01:30 +02:00
Miriam Baglioni	34fa96f3b1	-	2020-08-24 17:00:20 +02:00
Miriam Baglioni	5fb2949cb8	added utils methods	2020-08-24 17:00:09 +02:00
Miriam Baglioni	2a540b6c01	added constants for the pid graph dump	2020-08-24 16:55:35 +02:00
Miriam Baglioni	da103c399a	resources for the pid graph dump test	2020-08-24 16:52:07 +02:00
Miriam Baglioni	630a6a1fe7	first tests for the pid graph dump	2020-08-24 16:51:26 +02:00
Miriam Baglioni	40c8d2de7b	test resources for the dump of the pids graph	2020-08-24 16:50:39 +02:00
Miriam Baglioni	bef79d3bdf	first attempt to the dump of pids graph	2020-08-24 16:49:38 +02:00
Miriam Baglioni	85203c16e3	merge branch with master	2020-08-19 11:49:03 +02:00
Miriam Baglioni	2c783793ba	removed the affiliation from the author to mirror the changes in the model	2020-08-19 11:48:12 +02:00
Miriam Baglioni	f6bf888016	removed affiliation from author to mirror the changes in the model	2020-08-19 11:41:41 +02:00
Miriam Baglioni	66d0e0d3f2	-	2020-08-19 11:31:50 +02:00
Miriam Baglioni	1c593a9cfe	-	2020-08-19 11:29:51 +02:00
Miriam Baglioni	e42b2f5ae2	-	2020-08-19 11:29:09 +02:00
Miriam Baglioni	f81ee22418	changed to mirror the changes in the model (Instance, CommunityInstance, GraphResult)	2020-08-19 11:28:26 +02:00
Miriam Baglioni	387be43fd4	changed to discriminate if dumping all the results type together or each one in its own archive	2020-08-19 11:25:27 +02:00
Miriam Baglioni	c5858afb88	added parameter to guide the dump for the result (resultAggregation). true if all the result types should be dump together, false otherwise.	2020-08-19 11:24:14 +02:00
Miriam Baglioni	d407852ac2	changed to reflect the changed in the model	2020-08-19 11:15:05 +02:00
Miriam Baglioni	47c21a8961	refactoring due to compilation	2020-08-19 11:11:57 +02:00
Miriam Baglioni	5570678c65	changed parameter name from hfdsNameNode to nameNode	2020-08-19 10:59:26 +02:00
Miriam Baglioni	dc5096a327	refactoring due to compilation	2020-08-19 10:57:36 +02:00
Miriam Baglioni	96600ed04a	modified test resource for mirroring the deletion of affiliation from author parameters	2020-08-14 20:41:49 +02:00
Miriam Baglioni	09f5b92763	added specific reference to class	2020-08-14 20:00:09 +02:00
Miriam Baglioni	37e7c43652	changed parameter name from hdfsNaemNode to nameNode	2020-08-14 18:18:25 +02:00
Miriam Baglioni	d2a8a4961a	refactoring	2020-08-13 18:50:33 +02:00
Miriam Baglioni	a5043de5da	added method to get the mapped instance	2020-08-13 18:45:50 +02:00
Miriam Baglioni	fcd10f452c	changed because of #40 (comment)	2020-08-13 12:55:32 +02:00
Miriam Baglioni	fd48ae3b85	changed because of #40 (comment)	2020-08-13 12:19:15 +02:00
Miriam Baglioni	04a3e1ab38	disabled tests	2020-08-13 12:18:13 +02:00
Miriam Baglioni	2ede397933	Apply change because of #40 (comment)	2020-08-13 12:16:39 +02:00
Miriam Baglioni	bfd1fcde6d	removed not useful method and changed because of #40 (comment) and #40 (comment)	2020-08-13 12:14:37 +02:00
Miriam Baglioni	7fd8397123	apply changes in #40 (comment)	2020-08-13 12:13:15 +02:00
Miriam Baglioni	753d448cc9	apply changes in #40 (comment)	2020-08-13 12:12:58 +02:00
Miriam Baglioni	c0e071fa26	apply changes in #40 (comment)	2020-08-13 12:12:40 +02:00
Miriam Baglioni	526db915bc	apply changes in #40 (comment)	2020-08-13 12:12:16 +02:00
Miriam Baglioni	b0fab0d138	apply changes in #40 (comment)	2020-08-13 12:11:57 +02:00
Miriam Baglioni	1b6320b251	apply changes in #40 (comment)	2020-08-13 12:11:41 +02:00
Miriam Baglioni	743d31be22	apply changes in #40 (comment)	2020-08-13 12:11:22 +02:00
Miriam Baglioni	65b48df652	apply changes in #40 (comment)	2020-08-13 12:11:06 +02:00
Miriam Baglioni	90b54d3efb	apply changes in #40 (comment)	2020-08-13 12:08:24 +02:00
Miriam Baglioni	69bbb9592a	apply changes in #40 (comment)	2020-08-13 12:07:39 +02:00
Miriam Baglioni	945323299a	apply changes in #40 (comment)	2020-08-13 12:07:24 +02:00
Miriam Baglioni	e04c993247	apply changes in #40 (comment)	2020-08-13 12:07:07 +02:00
Miriam Baglioni	ed0812d0ce	apply changes in #40 (comment)	2020-08-13 12:06:49 +02:00
Miriam Baglioni	d55cfe0ea5	apply changes in #40 (comment)	2020-08-13 12:06:20 +02:00
Miriam Baglioni	80866bec7d	apply changes in #40 (comment)	2020-08-13 12:06:05 +02:00
Miriam Baglioni	1400978c0a	apply changes in #40 (comment)	2020-08-13 12:05:44 +02:00
Miriam Baglioni	7b941a2e0a	apply changes in #40 (comment)	2020-08-13 12:05:17 +02:00
Miriam Baglioni	f7474f50fe	apply changes in #40 (comment)	2020-08-13 12:04:52 +02:00
Miriam Baglioni	367203f412	apply changes in #40 (comment)	2020-08-13 12:04:33 +02:00
Miriam Baglioni	3ab4809d31	apply changes in #40 (comment)	2020-08-13 12:04:10 +02:00
Miriam Baglioni	02a4986e7b	Applying changed from code reviews #40 (comment) and #40 (comment) and #40 (comment)	2020-08-13 11:53:01 +02:00
Miriam Baglioni	235d4e4d6e	moved Context as relevant for Communities dump	2020-08-12 18:16:45 +02:00
Miriam Baglioni	adf9f96a67	test for extraction of relation between organizations and context	2020-08-12 10:04:47 +02:00
Miriam Baglioni	7400cd019d	removed not needed variable	2020-08-12 10:03:33 +02:00
Miriam Baglioni	98d28bab5c	fixed missing _ in context nsprefix	2020-08-12 10:00:18 +02:00
Miriam Baglioni	25f4fbceea	draft of test and resources	2020-08-11 17:37:22 +02:00
Miriam Baglioni	30a2b19b65	changed metadata for deposition od covid-19 dump in Zenodo	2020-08-11 17:36:56 +02:00
Miriam Baglioni	49788b532a	changed to mirror changes in the schema	2020-08-11 16:05:03 +02:00
Miriam Baglioni	b08511287b	-	2020-08-11 16:01:36 +02:00
Miriam Baglioni	7e81a17068	changed the XQUERY to mirror the change in the code	2020-08-11 16:00:33 +02:00
Miriam Baglioni	37ad2f28e9	removed added | in prefix for datasource	2020-08-11 15:55:06 +02:00
Miriam Baglioni	f31c2e9461	enabled test	2020-08-11 15:49:25 +02:00
Miriam Baglioni	2d67476417	merge branch with master	2020-08-11 15:46:04 +02:00
Miriam Baglioni	6d3804e24c	-	2020-08-11 15:45:12 +02:00
Miriam Baglioni	0603ec4757	changed test to upload the dump for covid-19 community	2020-08-11 15:43:25 +02:00
Miriam Baglioni	7dfd56df9d	-	2020-08-11 15:42:35 +02:00
Miriam Baglioni	a169d7e7c1	added test file for the MakeTar class	2020-08-11 15:40:41 +02:00
Miriam Baglioni	acb0926b2e	json schemas for the dumped entities and relation	2020-08-11 15:39:48 +02:00
Miriam Baglioni	ff52c51f92	added the communityMapPath parameter and removed the isLookUpUrl parameter	2020-08-11 15:39:22 +02:00
Miriam Baglioni	6f43acda5e	added the maketar and send to zenodo step. Adjusted wf parameters	2020-08-11 15:38:20 +02:00
Miriam Baglioni	ddc19de2e9	removed the isLookUpUrl among the parameters	2020-08-11 15:37:47 +02:00
Miriam Baglioni	592a8ea573	added parameter file for maketar class	2020-08-11 15:37:14 +02:00
Miriam Baglioni	77a0951b32	added the make archive step in the workflow	2020-08-11 15:32:32 +02:00
Miriam Baglioni	cf4d918787	added description, changed parameter name and added method	2020-08-11 15:27:31 +02:00
Miriam Baglioni	dc5fc5366d	Creation of an archive for each related dump part	2020-08-11 15:26:06 +02:00
Miriam Baglioni	0ce49049d6	added description	2020-08-11 15:25:11 +02:00
Miriam Baglioni	9bae991167	added description of the class	2020-08-11 11:20:43 +02:00
Miriam Baglioni	341dc59ead	removed the repartition(1). Added code for the creation of an archive containing all the parts dumped for each community	2020-08-11 11:18:58 +02:00
Miriam Baglioni	1991a49f70	removed reference to isLookUp to get the communityMap	2020-08-10 18:02:56 +02:00
Miriam Baglioni	c378c38546	disabled test. The testing functionalities for hte upload in Zenode are moved to common	2020-08-10 12:41:11 +02:00
Miriam Baglioni	63ad0ed209	changed to use communityMapPath instead of IsLookUp	2020-08-10 12:40:19 +02:00
Miriam Baglioni	cec795f2ea	changed resources to mirror changes in the model	2020-08-10 12:39:35 +02:00
Miriam Baglioni	f50e3e7333	changed the class for which to generate the schema	2020-08-10 12:03:49 +02:00
Miriam Baglioni	b8c26f656c	test using communityMapPath instead of isLookUp	2020-08-10 12:02:55 +02:00
Miriam Baglioni	fe88904df0	changed the wf definition	2020-08-10 12:01:14 +02:00
Miriam Baglioni	87856467e2	removed isLookUpUrl and added code to read from HDSF the communitymap	2020-08-10 11:38:41 +02:00
Miriam Baglioni	1cf7043e26	removed isLookUoUrl from the parameters	2020-08-10 11:38:03 +02:00
Sandro La Bruzzo	0ade33ad15	updated mergeFrom function for DLI Unknown	2020-08-10 10:18:35 +02:00
Miriam Baglioni	46986aae2d	added the new parameter for newdeposion/newversion and concept_record_id	2020-08-07 18:00:06 +02:00
Miriam Baglioni	3aedfdf0d6	added option to do a new deposition or new version of an old deposition	2020-08-07 17:49:14 +02:00
Miriam Baglioni	1b3ad1bce6	filter out authors pid (only orcid). Added check to get unique provenance for context id. filtr out countries with code UNKNOWN	2020-08-07 17:48:18 +02:00
Miriam Baglioni	5ceb8c5f0a	moved constants from graph.Constants	2020-08-07 17:46:47 +02:00
Miriam Baglioni	6c65c93c0e	refactoring	2020-08-07 17:45:35 +02:00
Miriam Baglioni	68adf86fe4	refactoring	2020-08-07 17:43:20 +02:00
Miriam Baglioni	26d2ad6ebb	refactoring	2020-08-07 17:41:56 +02:00
Miriam Baglioni	9675af7965	refactoring	2020-08-07 17:41:07 +02:00
Miriam Baglioni	346a91f4d9	Added constants	2020-08-07 17:35:39 +02:00
Miriam Baglioni	d52b0e1797	no use of IsLookUp. The query is done once and its result stored on HDFS. The path to the result is given instead of the isLookUpUrl	2020-08-07 17:34:40 +02:00
Miriam Baglioni	ae1b7fbfdb	changed method signature from set of mapkey entries to String representing path on file system where to find the map	2020-08-07 17:32:27 +02:00
Miriam Baglioni	931fa2ff00	removed dependencies	2020-08-07 16:46:37 +02:00
Miriam Baglioni	545ea9f77e	moved in common. Zenodo response model and APIClient to deposit in Zenodo	2020-08-07 16:44:51 +02:00
Miriam Baglioni	da9b012c15	fixed dewcription	2020-08-06 11:55:44 +02:00
Miriam Baglioni	6dbadcf181	the new schema for the dumped result	2020-08-06 11:05:56 +02:00
Sandro La Bruzzo	4fb1821fab	Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop	2020-08-06 10:28:31 +02:00
Sandro La Bruzzo	9d9e9edbd2	improved extractEntity Relation workflows using dataset	2020-08-06 10:28:24 +02:00
Miriam Baglioni	adf0ca5aa7	test to send is from hdfs	2020-08-05 14:24:43 +02:00
Miriam Baglioni	14eda4f46e	added method to try to put inputstream to zenodo	2020-08-05 14:18:25 +02:00
Miriam Baglioni	e737a47270	added classes to try to send input stream to zenodo for the upload	2020-08-05 14:17:40 +02:00
Miriam Baglioni	873e9cd50c	changed hadoop setting to connect to s3	2020-08-04 15:37:25 +02:00
Alessia Bardi	a29565ff57	code formatting	2020-08-04 12:55:27 +02:00
Alessia Bardi	01db29e208	fixes redmine issue #5846: datacite and its different namespace declarations	2020-08-04 12:53:48 +02:00
Alessia Bardi	b4e4e5f858	do not duplicate result PIDs	2020-08-04 12:52:14 +02:00
Alessia Bardi	09a323d18d	testing a dataset from Nakala	2020-08-04 12:50:52 +02:00
Alessia Bardi	c35bf486cc	added handle among the possible PIDs	2020-08-04 12:50:12 +02:00
Miriam Baglioni	5b651abf82	merge branch with master	2020-08-04 10:14:07 +02:00
Miriam Baglioni	901ae37f7b	added step to workflow	2020-08-03 18:12:54 +02:00
Miriam Baglioni	fa38cdb10b	added resource	2020-08-03 18:11:12 +02:00
Miriam Baglioni	e9fcc0b2f1	commented test unit - to decide change for mirroring the changed logics	2020-08-03 18:10:53 +02:00
Miriam Baglioni	e43aeb139a	added new property file and changed some parameter to old files	2020-08-03 18:07:28 +02:00
Miriam Baglioni	aa9f3d9698	changed logic for save in s3 directly	2020-08-03 18:06:18 +02:00
Miriam Baglioni	d465f0eec9	added fulltext to result	2020-08-03 18:03:27 +02:00
Miriam Baglioni	ec4b392d12	added new dependencies for writing on s3	2020-08-03 17:57:04 +02:00
Miriam Baglioni	c892c7dfa7	changed to query for community map just once and save the result for remaining executions	2020-08-03 17:56:31 +02:00
Alessia Bardi	8cc067fe76	specific test for claims	2020-08-03 11:17:50 +02:00
Michele Artini	652b13abb6	Merge branch 'master' into nsprefix_blacklist	2020-07-31 07:58:37 +02:00
Claudio Atzori	cd631bb5bc	defaults fixed in the cleaning workflow forces result.publisher to NULL when result.publisher.value in empty	2020-07-30 17:03:53 +02:00
Miriam Baglioni	872d7783fc	-	2020-07-30 16:50:36 +02:00
Miriam Baglioni	57c87b7653	re-implemented to fix issue on not serializable Set<String> variable	2020-07-30 16:43:43 +02:00
Miriam Baglioni	ef8e5957b5	added specific directory where to save results	2020-07-30 16:42:46 +02:00
Miriam Baglioni	75f3361c85	-	2020-07-30 16:41:31 +02:00
Miriam Baglioni	3f695b25fa	refactoring	2020-07-30 16:40:15 +02:00
Miriam Baglioni	e623f12bef	refactoring	2020-07-30 16:32:59 +02:00
Miriam Baglioni	ff7d05abb4	added support class to store the couple organizationId representativeId gaot from sql query on hive	2020-07-30 16:32:04 +02:00
Miriam Baglioni	cf6d80b2ab	added command to close the writer	2020-07-30 16:31:22 +02:00
Miriam Baglioni	f985bca37b	added USER_CLAIM constant value	2020-07-30 16:25:26 +02:00
Claudio Atzori	4bbfcf1ac6	Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop	2020-07-30 16:25:06 +02:00
Claudio Atzori	4ff8007518	added function to set the missing vocabulary names, used in the cleaning workflow as a pre-cleaning step	2020-07-30 16:24:39 +02:00
Miriam Baglioni	6f1c40a933	-	2020-07-30 16:24:28 +02:00
Miriam Baglioni	2b66a93f9e	added property file that was missing	2020-07-30 16:24:17 +02:00
Michele Artini	bdece15ca0	blacklist of nsprefix	2020-07-30 16:13:38 +02:00
Sandro La Bruzzo	c97c8f0c44	implemented new oozie job to extract entities in a separate dataset	2020-07-30 12:13:58 +02:00
Sandro La Bruzzo	3010a362bc	updated changing in the workflow of provision in the phase of aggregation. Removed serialization in JSON RDD and used spark Dataset	2020-07-30 09:25:56 +02:00
Sandro La Bruzzo	487226f669	Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop	2020-07-30 09:25:39 +02:00
Sandro La Bruzzo	16ae3c9ccf	updated changing in the workflow of provision in the phase of aggregation. Removed serialization in JSON RDD and used spark Dataset	2020-07-30 09:25:32 +02:00
Miriam Baglioni	ee8420c6b3	added resource for datasource test	2020-07-29 18:28:43 +02:00
Miriam Baglioni	76bcab98ce	added code to filter out null originalId from the dump	2020-07-29 18:28:21 +02:00
Miriam Baglioni	ef1d8aef17	added one test to verify the dump for the datasources	2020-07-29 18:27:46 +02:00
Miriam Baglioni	86bab79512	-	2020-07-29 18:20:22 +02:00
Miriam Baglioni	31791dcf3d	fixed wrong property file path name	2020-07-29 18:20:08 +02:00
Miriam Baglioni	9e722aa1ef	-	2020-07-29 18:00:08 +02:00
Miriam Baglioni	d22f106f27	added constant to identify datasource associated to funders	2020-07-29 17:56:55 +02:00
Miriam Baglioni	40e194fe2f	added check to not dump datasources related to funders	2020-07-29 17:56:18 +02:00
Miriam Baglioni	b48934f6df	changed the workflow name	2020-07-29 17:43:43 +02:00
Miriam Baglioni	1433db825d	refactorign	2020-07-29 17:43:24 +02:00
Miriam Baglioni	074e9ab75e	refactoring	2020-07-29 17:42:50 +02:00
Miriam Baglioni	8ad8dac7d4	merge branch with fork master	2020-07-29 17:38:28 +02:00
Miriam Baglioni	9fa82dc93b	fixed issue	2020-07-29 17:36:16 +02:00
Miriam Baglioni	8907648d6a	-	2020-07-29 17:35:47 +02:00
Miriam Baglioni	536e7f6352	added and changed resources for testing of the whole graph dump and of community related products dumps	2020-07-29 17:33:34 +02:00
Miriam Baglioni	4d7f590493	testings for the whole graph dump	2020-07-29 17:32:37 +02:00
Miriam Baglioni	a2f73ec2c7	changed due to changes in the model	2020-07-29 17:32:02 +02:00
Miriam Baglioni	481585e9d3	-	2020-07-29 17:31:41 +02:00
Miriam Baglioni	40a8dafbdc	-	2020-07-29 17:30:44 +02:00
Miriam Baglioni	de2ebb467e	changed due to changes in the model	2020-07-29 17:08:02 +02:00
Miriam Baglioni	d0ff2a56fb	-	2020-07-29 17:06:53 +02:00
Miriam Baglioni	b96dedb56b	changed due to changes in the model	2020-07-29 17:05:31 +02:00
Miriam Baglioni	6d0f08277b	classes to implement the dump of the whole graph.	2020-07-29 17:03:19 +02:00
Miriam Baglioni	8d4327b292	input parameters and workflow definition for the dump of the whole graph	2020-07-29 17:00:34 +02:00
Miriam Baglioni	b5f995ab12	refactoring	2020-07-29 16:59:48 +02:00
Miriam Baglioni	f7a87cc447	added new constants value	2020-07-29 16:58:40 +02:00
Miriam Baglioni	b71d12cf26	refactoring	2020-07-29 16:52:44 +02:00
Miriam Baglioni	a8d65b68cb	changed to delete the part to check if it was a test or a real execution	2020-07-29 16:47:57 +02:00
Miriam Baglioni	3ec2392904	Added new class to move the place the split is effectively run	2020-07-29 16:46:50 +02:00
Miriam Baglioni	178c2729a7	changed the path to reach the java class to be executed	2020-07-29 12:29:51 +02:00
Miriam Baglioni	437ac12139	removed unused parameter	2020-07-29 12:28:16 +02:00
Michele Artini	35e6e9c064	tests	2020-07-28 12:02:15 +02:00
Miriam Baglioni	6c2223d1fc	added code to get the openaire id for contexts	2020-07-24 17:30:15 +02:00
Miriam Baglioni	afd54c1684	removed not needed upload and refactoring	2020-07-24 17:28:56 +02:00
Miriam Baglioni	7b0569d989	changed to map also the result associated to the whole graph	2020-07-24 17:28:11 +02:00
Miriam Baglioni	082225ad61	-	2020-07-24 17:27:26 +02:00
Miriam Baglioni	968c59d97a	added teh logic to dump also the products for the whole graph. They will miss collected from and context information that will be materialized as new relations	2020-07-24 17:25:19 +02:00
Miriam Baglioni	332258d199	split the classes related to the communities dump and to the whole graph dump	2020-07-24 17:21:48 +02:00
Claudio Atzori	56bbfdc65d	introduced parameter 'numParitions', driving the hive DB table data partitioning. Currently specified only for table 'project'	2020-07-23 08:54:10 +02:00
Sandro La Bruzzo	9ab594ccf6	fixed test	2020-07-21 10:36:21 +02:00
Claudio Atzori	ebf60020ac	map results as OPRs in case of missing //CobjCategory/@type and the vocabulary dnet:result_typologies doesn't resolve the super type	2020-07-20 19:01:10 +02:00
Miriam Baglioni	355d7e426e	added dumo for project - not finished	2020-07-20 18:54:43 +02:00
Miriam Baglioni	a2f01e5259	added getter and setter	2020-07-20 18:54:17 +02:00
Miriam Baglioni	40bbe94f7c	merge with master fork	2020-07-20 18:10:03 +02:00
Miriam Baglioni	23160b4d29	realignment of the workflow classes with the changes in the structure of the module	2020-07-20 18:04:30 +02:00
Miriam Baglioni	3aab7680f6	changed the test results	2020-07-20 18:00:43 +02:00
Miriam Baglioni	5076e4f320	changed test to comply with the modifications	2020-07-20 17:55:18 +02:00
Miriam Baglioni	08dbd99455	changed to dump the whole results graph by usign classes already implemented for communities. Added class to dump also organization	2020-07-20 17:54:28 +02:00
Miriam Baglioni	e47ea9349c	extended some types by adding provenance as the couple (provenance, trust) and moved some classes to be used by the complete graph dump also	2020-07-20 17:46:27 +02:00
Claudio Atzori	32f5e466e3	imports cleanup	2020-07-20 17:42:58 +02:00
Claudio Atzori	54ac583923	code formatting	2020-07-20 17:37:08 +02:00
Claudio Atzori	124e7ce19c	in case of missing attribute //dr:CobjCategory/@type the resulttype is derived by looking up the vocabulary dnet:result_typologies with the 1st instance type available	2020-07-20 17:33:37 +02:00
Claudio Atzori	050dda223d	Merge pull request 'removed duplicated fields' (#25) from unique_field_in_lists into master ... Looks good as a temporary workaround. I agree the model could seamlessly make the distinct operation by using HashSets instead of Linked (or Array) Lists. The task to update the model in such a way is added on #9#issuecomment-1583 Thanks!	2020-07-20 12:12:50 +02:00
Claudio Atzori	e0c4cf6f7b	added parameter to drive the graph merge strategy: priority (BETA|PROD)	2020-07-20 10:48:01 +02:00
Claudio Atzori	94ccdb4852	Merge branch 'master' into merge_graph	2020-07-20 10:14:55 +02:00
Michele Artini	331a3cbdd0	fixed originalId	2020-07-20 09:50:29 +02:00
Sandro La Bruzzo	9116d75b3e	Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop	2020-07-17 18:01:30 +02:00
Miriam Baglioni	47c7122773	changed priority from beta to production	2020-07-17 12:56:35 +02:00
Michele Artini	442f30930c	removed duplicated fields	2020-07-17 12:25:36 +02:00
Michele Artini	3adedd0a68	trust truncated to 3 decimals	2020-07-17 11:58:11 +02:00
Claudio Atzori	1781609508	code formatting	2020-07-16 19:06:56 +02:00
Claudio Atzori	878f2b931c	Merge branch 'master' into merge_graph	2020-07-16 16:34:24 +02:00
Miriam Baglioni	f9ad6f3255	Merge branch 'dump' of code-repo.d4science.org:miriam.baglioni/dnet-hadoop into dump	2020-07-10 19:42:53 +02:00
Miriam Baglioni	c27f12d6e8	avoid to consider _SUCCESS file	2020-07-10 19:42:23 +02:00
Claudio Atzori	31071e363f	Merge branch 'provision_indexing'	2020-07-10 19:03:57 +02:00
Claudio Atzori	cc77446dc4	added dbSchema parameter to the raw_db workflow	2020-07-10 19:01:50 +02:00
Michele Artini	e1ae964bc4	stats	2020-07-10 16:12:08 +02:00
Sandro La Bruzzo	c01efed79b	Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop	2020-07-10 14:44:57 +02:00
Sandro La Bruzzo	a7d3977481	added generation of EBI Dataset	2020-07-10 14:44:50 +02:00
Claudio Atzori	67e1d222b6	bulk cleaning when found null or empty, sets bestaccessrights evaluating the result instances	2020-07-08 17:53:35 +02:00
Claudio Atzori	610d377d57	first implementation of the BETA & PROD graphs merge procedure	2020-07-08 16:54:26 +02:00
Alessia Bardi	9a898c0e4c	Json schema generator	2020-07-08 12:52:00 +02:00
Alessia Bardi	636f9ce7d6	json schema generator lib	2020-07-08 12:50:57 +02:00
Alessia Bardi	8f83b726fa	Dump json schema compliant to json schema Draft 7	2020-07-08 12:48:46 +02:00
Miriam Baglioni	1b0b968548	fixed issue on substring	2020-07-08 12:11:51 +02:00
Miriam Baglioni	7fe00cb4fb	-	2020-07-08 10:29:37 +02:00
Miriam Baglioni	375ef07d7b	changed the description for the upload	2020-07-07 18:41:27 +02:00
Miriam Baglioni	35c8265793	added the json extention to filename	2020-07-07 18:29:49 +02:00
Miriam Baglioni	81434f8e5e	added method newInstance	2020-07-07 18:26:10 +02:00
Miriam Baglioni	817cddfc52	-	2020-07-07 18:25:12 +02:00
Miriam Baglioni	a66aa9bd83	removed unuseful tests	2020-07-07 18:25:00 +02:00
Miriam Baglioni	9b20a21b24	removed unuseful tests	2020-07-07 18:23:37 +02:00
Miriam Baglioni	8a1b42ff21	added check to verify that dump contains at least one product	2020-07-07 18:21:35 +02:00
Miriam Baglioni	d86adb82a7	-	2020-07-07 18:20:51 +02:00
Miriam Baglioni	b2782025f6	enabled the whole workflow to run. Added property to give priority to depenedency in the classpath - to solve conflicts	2020-07-07 18:10:47 +02:00
Miriam Baglioni	83d2c84b77	added constraints to xquery so that to get only profiles with status manager or all	2020-07-07 18:09:48 +02:00
Miriam Baglioni	4c8d86493c	-	2020-07-07 18:09:06 +02:00
Miriam Baglioni	0208bc18f3	added new resource for testing	2020-07-07 17:47:24 +02:00
Miriam Baglioni	f5bb65c9ef	the json schema for the dump of the results	2020-07-07 17:34:40 +02:00
Miriam Baglioni	c19818a3f8	merge branch with fork master	2020-07-06 13:58:23 +02:00
Miriam Baglioni	f8bf4acd76	-	2020-07-02 16:03:11 +02:00
Miriam Baglioni	e6c79d44e6	-	2020-07-02 16:02:02 +02:00
Miriam Baglioni	d7f6f0c216	changed code to use other lib	2020-07-02 16:01:34 +02:00
Miriam Baglioni	8fdc9e070c	added dependency to OkHttp	2020-07-02 16:01:08 +02:00
Miriam Baglioni	94500a581b	merge branch with fork master	2020-07-02 14:25:39 +02:00
Claudio Atzori	ed1c7e5d75	fixed workflow for the import of the claims alone	2020-07-02 12:40:21 +02:00
Sandro La Bruzzo	1d420eedb4	added generation of EBI Dataset	2020-07-02 12:37:43 +02:00
Claudio Atzori	e4a29a4513	fixed workflow for the import of the claims alone	2020-07-02 12:36:33 +02:00
Claudio Atzori	6f5771c1c9	sets author.rank when null	2020-06-25 14:06:21 +02:00
Claudio Atzori	2d77d3a388	Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop	2020-06-25 12:54:30 +02:00
Miriam Baglioni	05a99cfb61	change the position of value and description elements in the workflow definition	2020-06-25 12:36:08 +02:00
Claudio Atzori	7df2712824	Merge branch 'provision_indexing'	2020-06-25 12:22:41 +02:00
Michele Artini	abcbebcbb4	fixed generation of ids	2020-06-25 09:50:46 +02:00
Michele Artini	77d2a1b1c4	params to choose sql queries for beta or production	2020-06-25 09:28:13 +02:00
Claudio Atzori	0e723d378b	added default from vocab for missing instance.refereed; remove spurious prefixes from orcid values; WIP: prepare relation job	2020-06-24 18:34:42 +02:00
Miriam Baglioni	3e5570de7a	-	2020-06-23 15:44:54 +02:00
Michele Artini	38bb45d0b6	test osf:refereed	2020-06-23 10:14:39 +02:00
Miriam Baglioni	e4b21be004	-	2020-06-22 17:31:50 +02:00
Miriam Baglioni	afa19b0c84	changed the way to PUT the files to the rest API	2020-06-22 17:20:07 +02:00
Miriam Baglioni	250fd1c854	merge branch with fork master	2020-06-22 16:25:48 +02:00
Claudio Atzori	9cd27183b6	[maven-release-plugin] prepare for next development iteration	2020-06-22 11:27:44 +02:00
Claudio Atzori	1e3dab0631	[maven-release-plugin] prepare release dhp-1.2.3	2020-06-22 11:27:39 +02:00
Miriam Baglioni	df80ae5c1b	merge branch with fork master	2020-06-22 10:51:23 +02:00
Miriam Baglioni	e8f914f8b3	-	2020-06-22 10:50:41 +02:00
Miriam Baglioni	edeb862476	excluded dependency in module that generates conflict	2020-06-22 10:49:56 +02:00
Miriam Baglioni	185facb8e5	change the deprecated DefaultHttpClient with the CLoseableHttpClient	2020-06-22 10:49:10 +02:00
Claudio Atzori	7d416f08d8	graph cleaning workflow: set hostedby to unknown repository when defined as NULL	2020-06-22 09:50:43 +02:00
Miriam Baglioni	669a509430	-	2020-06-19 17:39:46 +02:00
Claudio Atzori	d0ac7514b2	cleaning workflow to include cleaning of default values	2020-06-18 19:37:25 +02:00
Miriam Baglioni	44a12d244f	-	2020-06-18 18:38:54 +02:00
Miriam Baglioni	fb80353018	-	2020-06-18 14:21:36 +02:00
Miriam Baglioni	65bf312360	merge branch with fork master	2020-06-18 11:35:27 +02:00
Miriam Baglioni	3953f56bd3	added dependency to pom	2020-06-18 11:34:47 +02:00
Miriam Baglioni	a118b66858	-	2020-06-18 11:34:30 +02:00
Miriam Baglioni	f9578312b5	-	2020-06-18 11:34:15 +02:00
Miriam Baglioni	8b145e6aba	-	2020-06-18 11:25:28 +02:00
Miriam Baglioni	e8b3e972f2	changed the input params and the workflow definition to tackle the Result as all result product produced	2020-06-18 11:25:05 +02:00
Miriam Baglioni	3233b01089	changes due to adding all the result type under Result	2020-06-18 11:22:58 +02:00
Miriam Baglioni	5c8533d1a1	changed in the testing classes	2020-06-18 11:20:08 +02:00
Miriam Baglioni	bc8611a95a	added new resources for testing	2020-06-18 11:19:20 +02:00
Sandro La Bruzzo	9bf67f5de1	resolved conflicts	2020-06-17 09:15:43 +02:00
Sandro La Bruzzo	1d4275acc4	implemented first version of exportation of Scholexplorer into ActionSet	2020-06-17 09:10:38 +02:00
Claudio Atzori	1bc1d15eaf	stubbing for mock datasource.identities must be typed as array	2020-06-16 16:54:28 +02:00
Claudio Atzori	5441f01586	Merge pull request 'missing landingPage urls in instances' (#22) from instances-with-landing-page into master ... Looks good, thanks!	2020-06-16 15:32:44 +02:00
Claudio Atzori	89859111ee	Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop	2020-06-16 15:28:29 +02:00
Claudio Atzori	4ec262db53	included externalreference(s) in the result view on the Hive graph DB	2020-06-16 15:28:20 +02:00
Michele Artini	8a4f84f8c0	refactoring	2020-06-16 12:34:13 +02:00
Claudio Atzori	2a4f65795f	WIP: graph cleaner implementation	2020-06-15 18:32:24 +02:00
Claudio Atzori	c15c8c0ad0	map datasource identities (including piwik ids) as original IDs	2020-06-15 16:07:30 +02:00
Miriam Baglioni	9dd3ef22c5	merge branch with fork master	2020-06-15 11:23:26 +02:00
Miriam Baglioni	68cf0fd03f	test input	2020-06-15 11:14:42 +02:00
Miriam Baglioni	0467145ae3	test for graph dump	2020-06-15 11:13:51 +02:00
Miriam Baglioni	e43eedb5b0	added resources and workflow for dump of community products	2020-06-15 11:13:21 +02:00
Miriam Baglioni	f96ca900e1	fixed issues while running on cluster	2020-06-15 11:12:14 +02:00
Miriam Baglioni	20b9e67728	added new class funder	2020-06-15 11:06:18 +02:00
Claudio Atzori	0d52816244	WIP: graph cleaner implementation	2020-06-13 13:06:04 +02:00
Claudio Atzori	bed65a1be6	WIP: graph cleaner implementation	2020-06-12 18:25:47 +02:00
Claudio Atzori	c4d9f1837f	[maven-release-plugin] prepare for next development iteration	2020-06-12 12:21:08 +02:00
Claudio Atzori	f0746a7605	[maven-release-plugin] prepare release dhp-1.2.2	2020-06-12 12:21:03 +02:00
Claudio Atzori	463489f59f	code formatting	2020-06-12 12:03:25 +02:00
Claudio Atzori	4bcad1c9c3	Merge branch 'graph_cleaning'	2020-06-12 11:40:25 +02:00
Claudio Atzori	cdb1956fe9	WIP: graph cleaner implementation	2020-06-12 11:36:59 +02:00
Alessia Bardi	b347499745	do not use deprecated subreltype	2020-06-12 10:58:02 +02:00
Claudio Atzori	97b1c4057c	WIP: graph cleaner implementation	2020-06-12 10:45:18 +02:00
Claudio Atzori	ba8a024af9	avoid NPEs merging titles	2020-06-12 10:45:11 +02:00
Miriam Baglioni	e145972962	-	2020-06-11 13:08:39 +02:00
Miriam Baglioni	a01800224c	-	2020-06-11 13:02:04 +02:00
Miriam Baglioni	356dd582a3	map construction moved in class	2020-06-11 12:59:22 +02:00
Michele Artini	a41e0cb648	missing landingPage urls in instances	2020-06-11 12:28:34 +02:00
Michele Artini	99f88e1cb8	fixed generation entities from claims	2020-06-11 10:51:57 +02:00
Miriam Baglioni	db27663750	-	2020-06-11 10:49:01 +02:00
Miriam Baglioni	bb9f21d0e7	job test for class producing first step of results dump	2020-06-11 10:20:05 +02:00
Claudio Atzori	d1d92c4d8c	fixed integration of claims in the graph	2020-06-11 10:12:00 +02:00
Claudio Atzori	953da4a427	Merge branch 'master' into graph_cleaning	2020-06-10 21:36:56 +02:00
Claudio Atzori	f1bce64391	WIP: graph cleaner implementation	2020-06-10 21:36:31 +02:00
Michele Artini	c08e66e01e	fixed a workflow parameter	2020-06-10 10:11:56 +02:00
Michele Artini	7177a32d75	import of invisible stores	2020-06-10 10:04:00 +02:00
Claudio Atzori	a2fdf85ba1	WIP: graph cleaner implementation	2020-06-09 19:52:53 +02:00
Claudio Atzori	d9f33582c5	WIP: graph cleaner implementation	2020-06-09 17:20:40 +02:00
Miriam Baglioni	a089db18f1	workflow and parameters to exucute the dump	2020-06-09 15:39:38 +02:00
Miriam Baglioni	6bbe27587f	new classes to execute the dump for products associated to community, enrich each result with project information and assign the result to each community it belongs to	2020-06-09 15:39:03 +02:00
Miriam Baglioni	5121cbaf6a	new classes for external dump. Only classes functional to dump products	2020-06-09 15:37:46 +02:00
Claudio Atzori	b2349659cf	WIP: graph property fixing implementation	2020-06-05 18:37:38 +02:00
Claudio Atzori	5e23fb3a74	code formatting	2020-05-30 10:52:56 +02:00
Claudio Atzori	54ca8ed6c3	uniformed param name (isLookupUrl), Vocab model classes defined as Serializable	2020-05-29 18:17:30 +02:00
Claudio Atzori	1577bd5b8b	added IsLookupUrl to the raw_db workflow parameters	2020-05-29 16:18:16 +02:00
Michele Artini	adb798faa5	import from db using is vocabularies	2020-05-29 12:03:51 +02:00
Michele Artini	f5ce7d76e1	resolve conflicts	2020-05-27 12:49:17 +02:00
Michele Artini	b81f2741d2	xquery	2020-05-27 12:10:20 +02:00
Michele Artini	a25598140a	result pids (new xpaths + IS vocabularies)	2020-05-27 12:10:20 +02:00
Michele Artini	7a7272d9ec	result pids (new xpaths + IS vocabularies)	2020-05-27 12:10:20 +02:00
Michele Artini	3ceb2d2853	match terms with vocabularies	2020-05-27 11:34:13 +02:00
Michele Artini	c15d997925	xquery	2020-05-26 13:13:17 +02:00
Michele Artini	c6af36496a	result pids (new xpaths + IS vocabularies)	2020-05-26 13:11:09 +02:00
Michele Artini	093f1aff03	result pids (new xpaths + IS vocabularies)	2020-05-26 13:06:55 +02:00
Miriam Baglioni	54d869e618	merge upstream	2020-05-26 09:22:04 +02:00
Claudio Atzori	7582532e73	[maven-release-plugin] prepare for next development iteration	2020-05-25 19:48:18 +02:00
Claudio Atzori	01c2e93395	[maven-release-plugin] prepare release dhp-1.2.1	2020-05-25 19:48:14 +02:00
Miriam Baglioni	d3d36647d2	merge upstream	2020-05-25 10:38:22 +02:00
Miriam Baglioni	dbde2d243a	changed due to move of PacePerson from dhp-graph-mapper to dhp-common	2020-05-25 10:35:39 +02:00
Miriam Baglioni	8f6ce970f9	moved PacePerson to dhp-common to avoid conflict in dependency with graph-mapper	2020-05-25 10:25:55 +02:00
Claudio Atzori	de108f54d6	code formatting	2020-05-23 10:21:19 +02:00
Claudio Atzori	6b56cae57d	added mapping for bestaccessrights	2020-05-23 09:57:39 +02:00
Claudio Atzori	3cf2796ac6	code formatting	2020-05-22 12:34:00 +02:00
Michele Artini	dc4621b3cb	filter ORCID e MAG identifiers	2020-05-22 12:25:01 +02:00
Michele Artini	9f2d0f1b08	filter ORCID e MAG identifiers	2020-05-22 11:00:27 +02:00
Michele Artini	9de71e54a8	filter ORCID e MAG identifiers	2020-05-22 10:47:39 +02:00
Michele Artini	c5f7e17348	author fullnames	2020-05-22 10:08:02 +02:00
Michele Artini	e43d4d7778	added a coalesce in sql query	2020-05-21 11:08:07 +02:00
Michele Artini	b3bcbb3129	resolve name of organization countries	2020-05-21 08:41:32 +02:00
Claudio Atzori	7838f2c63f	init the empty list for author pids mapped from OAF	2020-05-15 17:06:01 +02:00
Claudio Atzori	7a89507ab1	code formatting	2020-05-15 15:16:54 +02:00
Claudio Atzori	cfc8948717	fixed mapping OdfToGraph: pick the correct element to map author pids and author affiliations; extended mapping Oaf2Graph: added support for author pids	2020-05-15 12:26:16 +02:00
Claudio Atzori	a832658296	code formatting	2020-05-15 10:21:09 +02:00
Claudio Atzori	18f46e47b9	added relations to the graph2hive import workflow	2020-05-15 09:34:48 +02:00
Claudio Atzori	9d028ffe1c	cleanup	2020-05-15 09:28:55 +02:00
Claudio Atzori	fd62359538	cleanup	2020-05-15 09:28:15 +02:00
Claudio Atzori	eb64335a54	parallel implementation for graph Hive importer	2020-05-15 09:05:26 +02:00
Claudio Atzori	f044d09315	revised mapping: more accurate mapping for name/surname from datacite format; improved mapping of null values	2020-05-14 15:07:24 +02:00
Claudio Atzori	ab37953332	added global properties in wf definitions to avoid repeating name-node and job-tracker in the (many) distcp actions; reintroduced output directory removal at the beginning of each spark action	2020-05-14 10:25:41 +02:00
Claudio Atzori	5ecacad70a	fixed default resource typing in Oaf/Odf mapping	2020-05-13 17:01:11 +02:00
Miriam Baglioni	f5d785e096	used the DbClient moved in dhp-common	2020-05-11 13:59:42 +02:00
Miriam Baglioni	2abb84877d	Merge branch 'master' into blacklist	2020-05-11 10:37:49 +02:00
Miriam Baglioni	bb59bdd60f	merge upstream	2020-05-11 10:33:17 +02:00
Miriam Baglioni	5e3548add6	-	2020-05-11 10:33:08 +02:00
Miriam Baglioni	871e079b45	merged with master	2020-05-11 10:20:00 +02:00
Claudio Atzori	60c40618d3	[maven-release-plugin] prepare for next development iteration	2020-05-11 10:17:14 +02:00
Claudio Atzori	c267d958d5	[maven-release-plugin] prepare release dhp-1.2.0	2020-05-11 10:17:10 +02:00
Miriam Baglioni	391b2399cc	merge upstream	2020-05-11 10:08:51 +02:00
Claudio Atzori	42f1a2bf94	bumped project version to 1.2.0-SNAPSHOT	2020-05-11 10:05:57 +02:00
Miriam Baglioni	32301451ec	merge upstream	2020-05-11 09:42:23 +02:00
Claudio Atzori	0ccc864ad9	[maven-release-plugin] prepare for next development iteration	2020-05-08 17:01:31 +02:00
Claudio Atzori	6e47c724c6	[maven-release-plugin] prepare release dhp-1.1.7	2020-05-08 17:01:27 +02:00
Miriam Baglioni	4c94231cad	merge with master fork	2020-05-08 12:25:57 +02:00
Claudio Atzori	62ea19f1d3	introduced mapping for ExternalReferences, made urls defined within an instance unique	2020-05-08 09:43:26 +02:00
Miriam Baglioni	207b899d6d	merged with upstream	2020-05-07 11:43:53 +02:00
Miriam Baglioni	5efae3acb9	new workflow for job3	2020-05-07 11:38:10 +02:00
Claudio Atzori	17860d3ab6	general changes in the RAW graph mapping: missing collectedfrom/hostedby causes records to be skipped; factored out most of the constants in ModelConstants class (dhp-schemas)	2020-05-06 13:20:02 +02:00
Michele Artini	8f30a09d84	Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop	2020-05-05 17:12:22 +02:00
Michele Artini	ccc609f909	new module for the production of broker events	2020-05-05 17:09:00 +02:00