D-Net
/
dnet-hadoop
15
4
Fork 10
Code Issues 14 Pull Requests 8 Projects 2 Releases 1 Wiki Activity

Labels Milestones

updates on stats-update workflow #100

Merged
claudio.atzori merged 8 commits from :master into master 3 years ago
Conversation 1 Commits 8 Files Changed 12
antonis.lempesis commented 3 years ago
Collaborator

Changes

stats-update

contexts? check

result_concepts? check

monitor db creation? check

stats-promote:

revamp of the openaire_prod_stats{DATE} -> openaire_prod_stats? check

same treatment for monitor db? check

tested? as much as possible!

Properties

stats-update

 stats_db_name (openaire_prod_stats_YYYYMMDD)
 openaire_db_name (openaire_prod_YYYYMMDD)
 external_stats_db_name (stats_ext)
 stats_db_shadow_name (openaire_prod_stats_shadow)
 monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD)
 monitor_db_shadow_name (openaire_prod_stats_monitor_shadow)
 stats_tool_api_url (https://services.openaire.eu/stats-tool/)
 hive_timeout (15000)
 context_api_url (https://services.openaire.eu/openaire)

stats-promote

stats_db_name (openaire_prod_stats_YYYYMMDD)
stats_db_production_name (openaire_prod_stats)
monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD)
monitor_db_production_name (openaire_prod_stats_monitor)
stats_tool_api_url (https://services.openaire.eu/stats-tool/)
hive_timeout (15000)

Next steps:

make the sleep while updating the cache configurable. Now it's hardcoded to 20h

# Changes ## stats-update contexts? check result_concepts? check monitor db creation? check ## stats-promote: revamp of the openaire_prod_stats{DATE} -> openaire_prod_stats? check same treatment for monitor db? check tested? as much as possible! # Properties ## stats-update ``` stats_db_name (openaire_prod_stats_YYYYMMDD) openaire_db_name (openaire_prod_YYYYMMDD) external_stats_db_name (stats_ext) stats_db_shadow_name (openaire_prod_stats_shadow) monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD) monitor_db_shadow_name (openaire_prod_stats_monitor_shadow) stats_tool_api_url (https://services.openaire.eu/stats-tool/) hive_timeout (15000) context_api_url (https://services.openaire.eu/openaire) ``` ## stats-promote ``` stats_db_name (openaire_prod_stats_YYYYMMDD) stats_db_production_name (openaire_prod_stats) monitor_db_name (openaire_prod_stats_monitor_YYYYMMDD) monitor_db_production_name (openaire_prod_stats_monitor) stats_tool_api_url (https://services.openaire.eu/stats-tool/) hive_timeout (15000) ``` # Next steps: make the sleep while updating the cache configurable. Now it's hardcoded to 20h
claudio.atzori changed title from master to updates on stats-update workflow 3 years ago
claudio.atzori changed target branch from master to stable_ids 3 years ago
claudio.atzori changed target branch from stable_ids to master 3 years ago
claudio.atzori commented 3 years ago
Owner

PR integrated for testing in the branch stable_ids with 3becaa5539

PR integrated for testing in the branch `stable_ids` with https://code-repo.d4science.org/D-Net/dnet-hadoop/commit/3becaa5539aee14592873f90ebfd7275206f4f61
claudio.atzori merged commit 37b65cc3ad into master 3 years ago
The pull request has been merged as 37b65cc3ad.
Sign in to join this conversation.
Reviewers
Request review
No reviewers
Labels
Apply labels
Clear labels   
bug

Something is not working   
duplicate

This issue or pull request already exists   
enhancement

New feature / refactoring   
help wanted

Need some help   
invalid

Something is wrong   
question

More information is needed   
RDGraph

EOSC Research Discovery Graph   
RSAC

EOSC Research Software APIs and Connectors   
wontfix

This won't be fixed
No Label
bug
duplicate
enhancement
help wanted
invalid
question
RDGraph
RSAC
wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Set Project
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: D-Net/dnet-hadoop#100
Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch ':master'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes