42 Commits

Author	SHA1	Message	Date
Claudio Atzori	ee832f358e	Merge pull request 'stats_wf_extensions_and_corrections' (#28) from spyros/dnet-hadoop:stats_wf_extensions_and_corrections into master ... Thank you Guys! The update workflow will be made available to the beta & production orchestration systems under the HDFS path ```/lib/dnet/oa/graph/stats/oozie_app```	2020-07-27 16:02:03 +02:00
Antonis Lempesis	4ac8ebe427	correctly calculating the project duration	2020-07-24 19:50:40 +03:00
Antonis Lempesis	18d9464b52	creating shadow db only if it not exists...	2020-07-24 19:50:40 +03:00
Antonis Lempesis	e217d496ab	added the dest db...	2020-07-24 19:50:40 +03:00
Antonis Lempesis	b16bb68b9f	added the target db name...	2020-07-24 19:50:40 +03:00
Antonis Lempesis	1ee7eeedf3	added the source db name...	2020-07-24 19:50:40 +03:00
Antonis Lempesis	cecbbfa0fc	added missing tables and views: contexts, creation_date, funder	2020-07-24 19:50:40 +03:00
Antonis Lempesis	25b7a615f5	moved datasource_sources table creating in the datasource section	2020-07-24 19:50:40 +03:00
Antonis Lempesis	a8da4ab9c0	years in projects are now integers	2020-07-24 19:50:40 +03:00
Antonis Lempesis	c9cfc165d9	not using impala since the resulting tables are not visible	2020-07-24 19:50:40 +03:00
Antonis Lempesis	dd3d6a6e15	compute stats for the used and new impala tables	2020-07-24 19:50:40 +03:00
Antonis Lempesis	e6f50de6ef	Separated impala from hive steps	2020-07-24 19:50:40 +03:00
Antonis Lempesis	de49173420	fixed a typo in queries	2020-07-24 19:50:40 +03:00
antleb	391cf80fb8	Added peer-reviewed, green, gold tables and fields in result. Added shortcuts from result-country	2020-07-24 19:50:40 +03:00
antleb	68389d0125	Corrected the script used by the last step of the wf	2020-07-24 19:50:40 +03:00
antleb	ec52141f1a	changed refereed type from value to clssname	2020-07-24 19:50:40 +03:00
Spyros Zoupanos	63cd797aba	Comment out step 15 to make it work with the new schema of Claudio	2020-07-24 19:50:40 +03:00
Spyros Zoupanos	138c6ddffa	Insert statement to datasource table that takes into account the piwik_id of the openAIRE graph	2020-07-24 19:50:40 +03:00
Spyros Zoupanos	3630794cef	Fix to consider the relationships that have been 'virtually deleted' for project_results - defect #5607	2020-07-24 19:50:40 +03:00
Spyros Zoupanos	5546f29e63	Corrections on the shadow schema and the impala table stats calculation	2020-07-24 19:50:40 +03:00
Spyros Zoupanos	adf8a025d2	Adding more relations (Sources, Licences, Additional) and shadow schema as provided and discussed with Antonis Lempesis	2020-07-24 19:50:40 +03:00
Spyros Zoupanos	657a40536b	Corrections by Spyros: Scipt cleanup, corrections and re-arrangement	2020-07-24 19:50:40 +03:00
Giorgos Alexiou	477fa6234d	Script re-organisation and adding table invalidations needed for impala	2020-07-24 19:50:40 +03:00
Claudio Atzori	9cd27183b6	[maven-release-plugin] prepare for next development iteration	2020-06-22 11:27:44 +02:00
Claudio Atzori	1e3dab0631	[maven-release-plugin] prepare release dhp-1.2.3	2020-06-22 11:27:39 +02:00
Claudio Atzori	c4d9f1837f	[maven-release-plugin] prepare for next development iteration	2020-06-12 12:21:08 +02:00
Claudio Atzori	f0746a7605	[maven-release-plugin] prepare release dhp-1.2.2	2020-06-12 12:21:03 +02:00
Spyros Zoupanos	3576dd186b	Adding hive timeout as workflow parameter	2020-06-05 22:29:54 +03:00
Claudio Atzori	7582532e73	[maven-release-plugin] prepare for next development iteration	2020-05-25 19:48:18 +02:00
Claudio Atzori	01c2e93395	[maven-release-plugin] prepare release dhp-1.2.1	2020-05-25 19:48:14 +02:00
Claudio Atzori	60c40618d3	[maven-release-plugin] prepare for next development iteration	2020-05-11 10:17:14 +02:00
Claudio Atzori	c267d958d5	[maven-release-plugin] prepare release dhp-1.2.0	2020-05-11 10:17:10 +02:00
Claudio Atzori	42f1a2bf94	bumped project version to 1.2.0-SNAPSHOT	2020-05-11 10:05:57 +02:00
Spyros Zoupanos	ae0f535c73	Fixing hardcoded reference to main openAIRE graph db	2020-05-09 22:34:48 +03:00
Claudio Atzori	0ccc864ad9	[maven-release-plugin] prepare for next development iteration	2020-05-08 17:01:31 +02:00
Claudio Atzori	6e47c724c6	[maven-release-plugin] prepare release dhp-1.1.7	2020-05-08 17:01:27 +02:00
Claudio Atzori	077ccd8743	stats wf properties cleanup	2020-05-04 11:41:46 +02:00
Claudio Atzori	77ac995770	cleaned up poms, added descriptions	2020-04-29 18:44:17 +02:00
Claudio Atzori	8fd81e863d	added default value for the external_stats_db_name	2020-04-29 15:36:24 +02:00
Claudio Atzori	c6f3ff4462	stats workflow content relocated into common package; added <global> property definitions in stats workflow.xml	2020-04-29 14:29:27 +02:00
Michele Artini	c43b4c8962	formatting	2020-04-29 12:56:58 +02:00
Spyros Zoupanos	1ab97bbe00	Adding the full stats workflow to the dnet-hadoop hierarchy	2020-04-01 22:22:05 +03:00