D-Net
/
dnet-hadoop
14
5
Fork 11
Code Issues 14 Pull Requests 13 Projects 2 Releases 2 Wiki Activity

updates on stats-update workflow #100

Merged
claudio.atzori merged 8 commits from :master into master 2021-04-02 16:13:36 +02:00
Conversation 1 Commits 8 Files Changed 12 +117 -266
antonis.lempesis commented 2021-03-17 13:48:27 +01:00
Member
Copy Link

Changes

stats-update

contexts? check

result_concepts? check

monitor db creation? check

stats-promote:

revamp of the openaire_prod_stats{DATE} -> openaire_prod_stats? check

same treatment for monitor db? check

tested? as much as possible!

Properties

stats-update

 stats_db_name (openaire_prod_stats_YYYYMMDD)
 openaire_db_name (openaire_prod_YYYYMMDD)
 external_stats_db_name (stats_ext)
 stats_db_shadow_name (openaire_prod_stats_shadow)
 monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD)
 monitor_db_shadow_name (openaire_prod_stats_monitor_shadow)
 stats_tool_api_url (https://services.openaire.eu/stats-tool/)
 hive_timeout (15000)
 context_api_url (https://services.openaire.eu/openaire)

stats-promote

stats_db_name (openaire_prod_stats_YYYYMMDD)
stats_db_production_name (openaire_prod_stats)
monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD)
monitor_db_production_name (openaire_prod_stats_monitor)
stats_tool_api_url (https://services.openaire.eu/stats-tool/)
hive_timeout (15000)

Next steps:

make the sleep while updating the cache configurable. Now it's hardcoded to 20h

# Changes ## stats-update contexts? check result_concepts? check monitor db creation? check ## stats-promote: revamp of the openaire_prod_stats{DATE} -> openaire_prod_stats? check same treatment for monitor db? check tested? as much as possible! # Properties ## stats-update ``` stats_db_name (openaire_prod_stats_YYYYMMDD) openaire_db_name (openaire_prod_YYYYMMDD) external_stats_db_name (stats_ext) stats_db_shadow_name (openaire_prod_stats_shadow) monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD) monitor_db_shadow_name (openaire_prod_stats_monitor_shadow) stats_tool_api_url (https://services.openaire.eu/stats-tool/) hive_timeout (15000) context_api_url (https://services.openaire.eu/openaire) ``` ## stats-promote ``` stats_db_name (openaire_prod_stats_YYYYMMDD) stats_db_production_name (openaire_prod_stats) monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD) monitor_db_production_name (openaire_prod_stats_monitor) stats_tool_api_url (https://services.openaire.eu/stats-tool/) hive_timeout (15000) ``` # Next steps: make the sleep while updating the cache configurable. Now it's hardcoded to 20h
claudio.atzori changed title from master to updates on stats-update workflow 2021-03-17 13:51:57 +01:00
claudio.atzori changed target branch from master to stable_ids 2021-03-29 15:55:10 +02:00
claudio.atzori changed target branch from stable_ids to master 2021-03-29 15:56:20 +02:00
claudio.atzori commented 2021-03-29 16:05:48 +02:00
Owner
Copy Link

PR integrated for testing in the branch stable_ids with 3becaa5539

PR integrated for testing in the branch `stable_ids` with https://code-repo.d4science.org/D-Net/dnet-hadoop/commit/3becaa5539aee14592873f90ebfd7275206f4f61
claudio.atzori merged commit 37b65cc3ad into master 2021-04-02 16:13:36 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature / refactoring

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
RDGraph

EOSC Research Discovery Graph

  
RSAC

EOSC Research Software APIs and Connectors

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#100
No description provided.
Delete Branch ":master"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?