Claudio Atzori
35e42a86ed
Merge pull request 'Update step15_5.sql' ( #313 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#313
2023-06-21 10:26:16 +02:00
dimitrispie
74cb060bfe
Update step15_5.sql
...
Add "if not exists" clause
2023-06-21 11:24:06 +03:00
Claudio Atzori
85e016df17
Merge pull request 'Update step16-createIndicatorsTables.sql' ( #312 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#312
2023-06-21 09:52:33 +02:00
dimitrispie
a475cfcb7b
Update step16-createIndicatorsTables.sql
...
Rename a field in indi_pub_interdisciplinarity
2023-06-21 10:42:02 +03:00
Claudio Atzori
979cf9cd87
Merge pull request 'Update step15.sql' ( #311 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#311
2023-06-21 09:20:01 +02:00
dimitrispie
4648cd88d4
Update step15.sql
...
Cast score to double
2023-06-21 10:02:19 +03:00
dimitrispie
94d2573c77
Update step15.sql
...
Bug Fix
2023-06-21 09:22:39 +03:00
Claudio Atzori
0561362de2
Merge pull request 'Update step20-createMonitorDB_institutions.sql' ( #309 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#309
2023-06-20 15:07:09 +02:00
dimitrispie
be2caedb04
Update step20-createMonitorDB_institutions.sql
...
Add openorgs____::1624ff7c01bb641b91f4518539a0c28a Vrije Universiteit Amsterdam
2023-06-19 12:12:17 +03:00
dimitrispie
4c770a5e29
Update finalizeImpalaCluster.sh
...
Drop views in shadow dbs before dropping the db
2023-06-15 13:25:37 +03:00
dimitrispie
e06d962a6a
Update step15.sql
2023-06-15 12:20:35 +03:00
dimitrispie
afcad08396
Update step20-createMonitorDB_institutions.sql
...
Added openorgs____::c0b262bd6eab819e4c994914f9c010e2 -- National Institute of Geophysics and Volcanology
2023-06-15 10:28:49 +03:00
Claudio Atzori
b9748763e2
Merge pull request '[stats wf] Bug fixes' ( #308 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#308
2023-06-14 21:57:03 +02:00
dimitrispie
42b8ce2ba4
Update copyDataToImpalaCluster.sh
2023-06-14 19:23:42 +03:00
dimitrispie
2032b0df40
Bug fixes
...
1. Remove tables/views from old databases in the new cluster, before dropping the dbs
2. Fix id in result_accessroute, indi_impact_measures, indi_pub_bronze_oa
2023-06-14 19:09:09 +03:00
Claudio Atzori
da7b66c542
Merge pull request '[stats wf] Added memory to hive' ( #305 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#305
2023-06-08 08:58:48 +02:00
dimitrispie
c5f42c7f5b
Added memory to hive
2023-06-07 18:18:23 +03:00
Claudio Atzori
afb76ebf0f
Merge pull request '[stats wf] Bug fix on indicators step' ( #304 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#304
2023-06-07 16:49:09 +02:00
dimitrispie
fa24e2e18f
Bug fix on indicators step
...
indi_pub_gold_oa table was missing during the creation of other indicators
2023-06-07 17:43:37 +03:00
Claudio Atzori
01c67e697d
Merge pull request '[ stats wf] Bug fix' ( #303 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#303
2023-06-07 14:41:44 +02:00
dimitrispie
28272c1b0e
Bug fix
2023-06-07 15:34:01 +03:00
Claudio Atzori
8f651f1225
Merge pull request 'Changes to beta stats wf' ( #300 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#300
2023-06-06 11:41:36 +02:00
dimitrispie
ad07fbf053
Add names to organizations for collaboration indicators
2023-06-02 14:13:10 +03:00
dimitrispie
2324670714
Split Monitor DBs-Interdisciplinarity indicators
...
- Split DBs Monitor for faster rendering of visualizations
- Add interdisciplinarity indicators from result_fos
2023-06-02 13:34:16 +03:00
dimitrispie
ebe586b1d1
Impact indicators/Unpaywall
...
- Added Impact indicators
- Added unpaywall open access colours
2023-05-26 10:25:28 +03:00
dimitrispie
d6102dd576
Update step16-createIndicatorsTables.sql
...
- Add org names to indi_project_collab_org
- Add indi_pub_bronze_oa
- Changes to indi_pub_hybrid_oa_with_cc
2023-05-25 14:52:34 +03:00
Claudio Atzori
a235d2a24a
Merge pull request 'Updates to steps related to transfer data to impala cluster' ( #295 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#295
2023-05-18 08:46:15 +02:00
dimitrispie
86f4f63daf
Updates to steps related to transfer data to impala cluster
...
1. Remove external table definitions in stats_ext
2. Fix the issue where some views are not created.
3. Added two workflow parameters for copying also the usage stats dbs
2023-05-18 09:33:05 +03:00
Claudio Atzori
0c314d5e09
Merge pull request 'Update copyDataToImpalaCluster.sh' ( #293 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#293
2023-05-15 12:05:54 +02:00
dimitrispie
b3f9633205
Update copyDataToImpalaCluster.sh
...
Added option --user to impala-shell command
2023-05-15 12:51:44 +03:00
Miriam Baglioni
12cd179d2d
Merge pull request 'Update copyDataToImpalaCluster.sh' ( #291 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#291
2023-05-12 11:36:34 +02:00
dimitrispie
00d0d162b6
Update copyDataToImpalaCluster.sh
...
Added a temporary folder to copy the files to avoid permission issues
2023-05-12 12:31:13 +03:00
Claudio Atzori
0704e186f6
Merge pull request 'Stats wf executed on hive only' ( #283 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#283
2023-05-02 14:05:12 +02:00
dimitrispie
c3d58e58e1
Bug fixes
2023-05-02 11:54:07 +03:00
dimitrispie
e57ecdaf98
Update step20-createMonitorDB.sql
...
Add University of Manitoba
2023-04-30 17:52:23 +03:00
dimitrispie
5247cb7115
Bug fix
2023-04-19 11:11:19 +03:00
dimitrispie
25dafccc24
Merge branch 'hive' into beta
2023-04-12 11:36:59 +03:00
dimitrispie
c85de8fa1f
-Added Technological University Dublin
...
-Added project_organization_contribution table
-Add Delft University of Technology
2023-04-07 09:22:59 +03:00
dimitrispie
9b41dff33c
Update step20-createMonitorDB.sql
...
Added Delft University of Technology
2023-04-07 09:21:38 +03:00
dimitrispie
91e18ac7f4
Added project_organization_contribution table
2023-04-06 10:53:11 +03:00
dimitrispie
9e1335df4c
-Added Technological University Dublin
...
-Added project_organization_contribution table
2023-04-04 13:22:40 +03:00
dimitrispie
fad7fa4af8
Added Technological University Dublin
2023-03-22 09:44:00 +02:00
dimitrispie
43b23a9bf3
Update step20-createMonitorDB.sql
...
Added Technological University Dublin
2023-03-15 09:57:12 +02:00
dimitrispie
1547611246
Merge branch 'beta' into hive
2023-02-22 16:57:12 +02:00
Claudio Atzori
c20c1c9159
Merge pull request 'Added 4 institutions:' ( #261 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#261
2023-02-22 09:53:45 +01:00
dimitrispie
032a401cbf
Bug fixes
2023-02-20 09:29:20 +02:00
dimitrispie
595192d510
Bug fix
2023-02-14 16:24:08 +02:00
dimitrispie
f3aaff3688
Remove duplicate orgs
2023-02-14 09:48:36 +02:00
dimitrispie
3400133c2f
Bug fix
2023-02-13 09:44:00 +02:00
dimitrispie
935db0ab25
Added organizations for Monitor
2023-02-13 09:29:09 +02:00
dimitrispie
7b78b15c81
Changes for copying to Impala Cluster
2023-02-13 09:27:00 +02:00
dimitrispie
d71f5672d3
Add monitor post step
2023-02-09 13:44:14 +02:00
dimitrispie
35ba8bb328
Bug fixes
2023-02-09 12:57:57 +02:00
dimitrispie
3ba11d64a1
Changes 07022023
2023-02-07 12:53:51 +02:00
dimitrispie
98c34263ed
Update step20-createMonitorDB.sql
...
Add University of Cape Town organization
2023-02-07 08:14:48 +02:00
dimitrispie
2dc6d47270
Changes 06022023
2023-02-06 13:18:53 +02:00
dimitrispie
973d78a4d6
Update step15_5.sql
...
Added unpaywalls open access colors
2023-02-02 08:03:54 +02:00
dimitrispie
cf58e4a5e4
Added Arts et Métiers ParisTech
2023-01-25 16:03:16 +02:00
dimitrispie
db7d625ba9
Addedd Arts et Métiers ParisTech organization
2023-01-25 12:22:21 +02:00
dimitrispie
686580a220
- New Monitor DB workflow
...
- New Organization added
2023-01-12 11:18:03 +02:00
dimitrispie
dcb958e146
Changes to execute the stats wf only in hive
2023-01-04 11:39:01 +02:00
dimitrispie
592013d5dd
Added more steps in decision node
2022-12-23 09:43:16 +02:00
dimitrispie
2a4bf32d4c
Merge branch 'hive' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into hive
...
# Conflicts:
# dhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step10.sql
# dhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step13.sql
# dhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step14.sql
# dhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step16_1-definitions.sql
# dhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step7.sql
2022-12-22 10:22:46 +02:00
dimitrispie
6449ff4207
1. Added a decision node to enables the workflow to make a selection on the execution path to follow
...
2. Added new organization
3. Added 5 new tables from Eurostast
2022-12-22 10:18:21 +02:00
Antonis Lempesis
c8309fe18e
addded command line params to allow hive actions to run
2022-12-21 12:41:33 +02:00
Antonis Lempesis
028873cc51
added new hive opts
2022-12-21 12:41:33 +02:00
Antonis Lempesis
1ddea4f442
removed 'stored as parquet' from views..
2022-12-21 12:41:33 +02:00
Antonis Lempesis
2754c3dd62
moving data to impala cluster and creating shadow databases there
2022-12-21 12:41:29 +02:00
Antonis Lempesis
778a1a724f
finished migration to hive only
2022-12-21 12:41:25 +02:00
Antonis Lempesis
e84dd5fe26
first
2022-12-21 12:41:23 +02:00
dimitrispie
2a52a42169
Added 4 institutions:
...
-University of Modena and Reggio Emilia
-Bilkent University
-Saints Cyril and Methodius University of Skopje
-University of Milan
2022-12-06 10:10:21 +02:00
Claudio Atzori
ddf065756f
Merge pull request 'Two organizations are added for monitor' ( #258 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#258
2022-12-02 14:45:27 +01:00
dimitrispie
992fc5b628
Added McMaster University Institution
2022-11-03 11:02:18 +02:00
dimitrispie
7fda05e380
Added Autonomous University of Barcelona
2022-11-01 13:59:40 +02:00
Claudio Atzori
22873c9172
Merge pull request 'Added fields: totalcost, fundedamount, currency, in project table' ( #257 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#257
2022-10-31 13:49:27 +01:00
dimitrispie
7861c472e0
Hive memory parameters
2022-10-28 19:00:32 +03:00
dimitrispie
5df9c63963
Added fields: totalcost, fundedamount, currency, in project table
2022-10-27 16:44:26 +03:00
Claudio Atzori
89f7007080
Merge pull request '[stats wf] misc changes' ( #254 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#254
2022-10-03 10:32:05 +02:00
dimitrispie
2c0c3f1806
Cast amount to float for table result_apcs
2022-09-28 19:33:24 +03:00
dimitrispie
bdc46e3eaa
Remove denormalization of results to fix downloads numbers in monitor
2022-09-28 14:59:08 +03:00
dimitrispie
2ebb1459a9
Fixed type in no_downloads
2022-09-28 14:36:57 +03:00
Claudio Atzori
3b60642ef9
Merge pull request 'Synchronize indicators in stats-db with monitor-db' ( #249 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#249
2022-09-27 14:37:33 +02:00
dimitrispie
dcd85f8cd7
- Synchronize indicators in stats-db with monitor-db
...
- added new openorg id for Nanyang Technological University
- changed openorg id for University of Helsinki #8088 ticket
2022-09-22 13:33:07 +03:00
Claudio Atzori
465e941214
Merge pull request '[stats wf] Changes to indicators tables' ( #244 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#244
2022-09-16 10:13:58 +02:00
dimitrispie
3bf3127251
Changes to monitor and indicator scripts
2022-09-14 16:36:19 +03:00
dimitrispie
71b069ca90
Changes to indicator and monitor scripts
2022-09-09 13:15:58 +03:00
dimitrispie
2b5f8c9c9a
comment out duplicate table creation
2022-09-06 12:27:53 +03:00
Claudio Atzori
84598c7535
Merge pull request 'restored some collab indicators' ( #240 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#240
2022-08-05 15:50:39 +02:00
Antonis Lempesis
fcef5294e2
restored some collab indicators
2022-08-05 13:45:01 +03:00
Claudio Atzori
c1f2ffc53d
Merge pull request 'commenting out the collab indicators because they still fail' ( #237 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#237
2022-08-05 11:57:36 +02:00
Antonis Lempesis
227e10f4b3
commenting out the collab indicators because they still fail
2022-08-05 12:54:36 +03:00
Claudio Atzori
efd96e7e66
Merge pull request 'fixed the datasourceOrganization relations' ( #233 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#233
2022-08-03 12:25:05 +02:00
Antonis Lempesis
8b0407d8ec
fixed the datasourceOrganization relations
2022-08-03 12:26:59 +03:00
Claudio Atzori
27681cf6bf
Merge pull request '[stats wf] latest version of indicators + added FOS classification' ( #232 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#232
2022-08-02 12:57:15 +02:00
Antonis Lempesis
1778d40c40
latest version of indicators
2022-08-02 13:39:34 +03:00
Antonis Lempesis
6fc9ef53f6
addded command line params to allow hive actions to run
2022-07-29 16:36:20 +03:00
Antonis Lempesis
9886fe87ec
- Added FOS classification
...
- Added extra orgs in monitor
- Fixed result-project and organization-project tables
2022-07-29 16:34:50 +03:00
Miriam Baglioni
b229c6e7af
Merge pull request 'beta' ( #218 ) from antonis.lempesis/dnet-hadoop:beta into beta
...
Reviewed-on: D-Net/dnet-hadoop#218
2022-06-10 11:03:48 +02:00
Antonis Lempesis
ab18c9daa9
Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta
2022-06-09 15:48:21 +03:00
Antonis Lempesis
574492c659
removed double result_apc table creation from monitor
2022-06-09 15:48:13 +03:00