 Lampros Smyrnaios
|
bc8c97182d
|
Automatically select the ACTIVE HDFS NODE for Impala cluster, in all "copyDataToImpalaCluster.sh" scripts.
|
2024-03-26 13:01:12 +02:00 |
|
Lampros Smyrnaios
|
92cc27e7eb
|
Use the ACTIVE HDFS NODE for Impala cluster, in "copyDataToImpalaCluster.sh" script.
|
2024-03-26 12:34:11 +02:00 |
 Antonis Lempesis
|
4c40c96e30
|
code cleanup
|
2024-03-22 10:16:49 +02:00 |
|
Antonis Lempesis
|
459167ac2f
|
Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta
|
2024-03-21 12:44:58 +02:00 |
|
Antonis Lempesis
|
07f634a46d
|
code cleanup
|
2024-03-21 12:44:30 +02:00 |
|
Antonis Lempesis
|
9521625a07
|
code cleanup
|
2024-03-21 11:45:08 +02:00 |
|
Antonis Lempesis
|
67a5aa0a38
|
Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta
|
2024-03-19 11:24:54 +02:00 |
dimitrispie
|
a3a570e9a0
|
Commit monitor-updates-wf
|
2024-03-19 09:42:21 +02:00 |
|
Antonis Lempesis
|
f74c7e8689
|
selecting distinct peer_reviewed
|
2024-03-12 02:13:04 +02:00 |
|
Antonis Lempesis
|
3c79720342
|
fixed the irish result subset
|
2024-03-07 14:08:57 +02:00 |
|
Antonis Lempesis
|
5ae4b4286c
|
Merge branch 'beta' of https://code-repo.d3science.org/antonis.lempesis/dnet-hadoop into beta
|
2024-03-07 12:15:19 +02:00 |
|
Antonis Lempesis
|
316d585c8a
|
using distinct apcs per publication to avoid huge sums
|
2024-03-07 02:07:59 +02:00 |
|
Antonis Lempesis
|
dd4c27f4f3
|
added 2 new institutions in monitor
|
2024-02-08 12:57:57 +02:00 |
|
Antonis Lempesis
|
a512ead447
|
changed orcid ids to all capital
|
2024-01-30 16:54:47 +02:00 |
|
Antonis Lempesis
|
bb10a22290
|
merged changes from dnet-hadoop
|
2024-01-29 21:51:47 +02:00 |
 Claudio Atzori
|
f804c58bc7
|
Merge pull request 'Use SparkSQL in place of Hive for executing step16-createIndicatorsTables.sql of stats update wf' (#386) from stats_with_spark_sql into beta
Reviewed-on: #386
|
2024-01-29 09:11:59 +01:00 |
|
Claudio Atzori
|
926903b06b
|
Merge branch 'beta' into stats_with_spark_sql
|
2024-01-29 09:11:45 +01:00 |
 Giambattista Bloisi
|
078df0b4d1
|
Use SparkSQL in place of Hive for executing step16-createIndicatorsTables.sql of stats update wf
|
2024-01-26 21:56:55 +01:00 |
|
Antonis Lempesis
|
c548796463
|
Changed step16-createIndicatorsTables to use a spark oozie action instead of hive
|
2024-01-26 02:04:48 +02:00 |
|
Antonis Lempesis
|
fd43b0e84a
|
max mem of joins (hive.mapjoin.followby.gby.localtask.max.memory.usage) now 80%, up from 55%.
|
2024-01-25 15:06:34 +01:00 |
|
Antonis Lempesis
|
e024718f73
|
creating result_instances even when no pids exist for the instance
|
2024-01-10 22:25:50 +01:00 |
|
dimitrispie
|
b920307bdd
|
Changes to indicators
|
2024-01-09 00:47:09 +02:00 |
|
dimitrispie
|
8b2cbb611e
|
Changes to beta db names
|
2024-01-09 00:40:56 +02:00 |
|
Antonis Lempesis
|
2e4cab026c
|
fixed the result_country definition
|
2024-01-08 16:01:26 +02:00 |
|
dimitrispie
|
6b823100ae
|
Update buildIrishMonitorDB.sql
New indicators added
|
2024-01-07 22:54:39 +02:00 |
|
dimitrispie
|
75bfde043c
|
Historical Snapshots Workflow
Create historical snapshots db with parameters:
hist_db_name=openaire_beta_historical_snapshots_xxx
hist_db_name_prev=openaire_beta_historical_snapshots_xxx (previous run of wf)
stats_db_name=openaire_beta_stats_xxx
stats_irish_db_name=openaire_beta_stats_monitor_ie_xxx
monitor_db_name=openaire_beta_stats_monitor_xxx
monitor_db_prod_name=openaire_beta_stats_monitor
monitor_irish_db_name=openaire_beta_stats_monitor_ie_xxx
monitor_irish_db_prod_name=openaire_beta_stats_monitor_ie
hist_db_prod_name=openaire_beta_historical_snapshots
hist_db_shadow_name=openaire_beta_historical_snapshots_shadow
hist_date=122023
hive_timeout=150000
hadoop_user_name=xxx
resumeFrom=CreateDB
|
2024-01-04 15:11:04 +02:00 |
|
dimitrispie
|
ffdd03d2f4
|
Monitor Irish Stats WF
Parameters (with examples):
stats_db_name=openaire_beta_stats_20231208
monitor_irish_db_name=openaire_beta_stats_monitor_ie_20231208b
monitor_irish_db_prod_name=openaire_beta_stats_monitor_ie
graph_db_name=openaire_beta_20231208
monitor_irish_db_shadow_name=openaire_beta_stats_monitor_ie_shadow
hive_timeout=150000
hadoop_user_name=dnet.beta
resumeFrom=Step1-buildIrishMonitorDB
|
2023-12-22 11:05:24 +02:00 |
|
dimitrispie
|
40b98d8182
|
Changes to indicators and funders definition
- Changes result_refereed definition
- Added result_country indicator
- Added indi_pub_green_with_license indicator
- Added country from jurisdiction to funders
|
2023-12-22 10:29:20 +02:00 |