Compare commits
No commits in common. "hadoop-cdh4-legacy" and "main" have entirely different histories.
hadoop-cdh
...
main
86
ChangeLog
86
ChangeLog
|
@ -1,86 +0,0 @@
|
|||
2013-11-06 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/hadoop-nrpe.cfg.j2: Add the correct entry for the tasktracker event handler.
|
||||
|
||||
2013-11-04 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/iptables-rules.v4.j2: Add access to the hbase master rpc port from the isti network.
|
||||
|
||||
2013-10-10 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/hbase-site.j2: Add the property hbase.master.loadbalance.bytable to try a better balance for our workload. Reference here: http://answers.mapr.com/questions/7049/table-only-on-single-region-server
|
||||
|
||||
2013-10-09 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/nagios-server/hadoop-cluster/services.cfg.j2: Handler to restart the tasktracker when it fails.
|
||||
* templates/iptables-rules.*.j2: iptables rules to block access to the services ports from outside CNR.
|
||||
|
||||
2013-10-07 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/nagios-server: added checks for the logstash host and services
|
||||
* logstash.yml: Add logstash with remote syslog to aggregate all the workers logs. Needed for solr.
|
||||
|
||||
2013-10-01 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/: management portal that redirects to the services web interfaces.
|
||||
|
||||
2013-09-23 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* tasks/jobtracker-ha.yml: HA configuration for the jobtracker. jobtracker.t.hadoop and quorum4.t.hadoop are the two masters.
|
||||
|
||||
2013-09-19 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* all.yml: HDFS is now HA. All the datanodes lists are generated from the hosts file and are not static anymore. Changed nagios to reflect the new configuration.
|
||||
|
||||
2013-09-17 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates: Changed the system-* scripts to manage the second namenode instance. Removed the secondary namenode start/stop script
|
||||
|
||||
2013-09-12 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* hadoop-test.yml: New quorum4.t.hadoop node. Zookeeper now has 5
|
||||
quorum nodes. HBASE master HA. quorum4 is the other instance.
|
||||
|
||||
2013-07-29 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* templates/datanode-hdfs-site.j2: Added "support_append" as "true" and max_xcievers as 1024
|
||||
|
||||
2013-06-20 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* hadoop-ganglia.yml: The ganglia configuration is now differentiated between datanodes, jobtracker, hbase master, hdfs namenode, hdfs secondary namenode
|
||||
|
||||
2013-02-27 Andrea Dell'Amico <adellam@sevenseas.org>
|
||||
|
||||
* vars/hadoop-global-vars.yml: mapred_tasktracker_reduce_tasks_maximum: 5, hbase_regionserver_heap_size: 3192
|
||||
|
||||
2013-02-22 Andrea Dell'Amico <adellam@isti.cnr.it>
|
||||
|
||||
* init.sh: Create hdfs directory /jobtracker to store the jobtracker history
|
||||
* templates/mapred-site-jobtracker.j2: Activate permanent jobtracker history
|
||||
* jobtracker.yml: Cleanup
|
||||
|
||||
2013-02-18 Andrea Dell'Amico <adellam@isti.cnr.it>
|
||||
|
||||
* vars/hadoop-global-vars.yml: mapred_child_java_opts: "-Xmx3092M", mapred_map_child_java_opts: "-Xmx2048M", mapred_reduce_child_java_opts: "-Xmx1512M", hbase_regionserver_heap_size: 4092
|
||||
|
||||
2013-02-18 Andrea Dell'Amico <adellam@isti.cnr.it>
|
||||
|
||||
* vars/hadoop-global-vars.yml: hbase_master_heap_size: 5120, hbase_regionserver_heap_size: 3192
|
||||
|
||||
2013-02-18 Andrea Dell'Amico <adellam@isti.cnr.it>
|
||||
|
||||
* vars/hadoop-global-vars.yml (hbase_thrift_heap_size): mapred_child_java_opts: "-Xmx1512M", mapred_map_child_java_opts: "-Xmx3092M", mapred_reduce_child_java_opts: "-Xmx2048M", hbase_master_heap_size: 3072
|
||||
|
||||
2013-02-16 Andrea Dell'Amico <adellam@isti.cnr.it>
|
||||
|
||||
* templates/hbase-thrift-env.sh.j2: Disabled the jmx console for hbase thrift
|
||||
|
||||
* templates/hbase-master-env.sh.j2: disabled the master jmx console
|
||||
|
||||
* vars/hadoop-global-vars.yml: zookeeper_max_timeout: 240000, fixed the zookeeper quorum host naming
|
||||
|
||||
2013-02-16 Andrea Dell'Amico <adellam@isti.cnr.it>
|
||||
|
||||
* vars/hadoop-global-vars.yml: mapred_child_java_opts: "-Xmx2G", mapred_reduce_child_java_opts: "-Xmx2512M", hbase_regionserver_heap_size: 5000
|
||||
|
||||
|
304
README.md
304
README.md
|
@ -1,303 +1,3 @@
|
|||
# hadoop-ansible
|
||||
|
||||
# Hadoop cluster based on the CDH 4 packages.
|
||||
|
||||
This is the playbook that I used to install and configure the Hadoop cluster @CNR, based on the deb packages found in the Cloudera repositories.
|
||||
No cloudera manager was used nor installed.
|
||||
|
||||
## The cluster.
|
||||
|
||||
The cluster structure is the following:
|
||||
|
||||
- jobtracker.t.hadoop.research-infrastructures.eu (3.5GB RAM, 4 CPUs):
|
||||
- mapreduce HA jobtracker
|
||||
- zookeeper quorum
|
||||
- HA HDFS journal
|
||||
- quorum4.t.hadoop.research-infrastructures.eu (3.5GB RAM, 4 CPUs):
|
||||
- mapreduce HA jobtracker
|
||||
- zookeeper quorum
|
||||
- HA HDFS journal
|
||||
- nn1.t.hadoop.research-infrastructures.eu (3.5GB RAM, 4 CPUs):
|
||||
- hdfs HA namenode
|
||||
- zookeeper quorum
|
||||
- HA HDFS journal
|
||||
- nn2.t.hadoop.research-infrastructures.eu (3.5GB RAM, 4 CPUs):
|
||||
- hdfs HA namenode
|
||||
- zookeeper quorum
|
||||
- HA HDFS journal
|
||||
- hbase-master.t.hadoop.research-infrastructures.eu (3.5GB RAM, 4 CPUs):
|
||||
- hbase primary master
|
||||
- hbase thrift
|
||||
- zookeeper quorum
|
||||
- HA HDFS journal
|
||||
- hbase-master2.t.hadoop.research-infrastructures.eu (2GB RAM, 2 CPUs):
|
||||
- HBASE secondary master
|
||||
- hbase thrift
|
||||
|
||||
- node{2..13}.t.hadoop.research-infrastructures.eu (9GB RAM, 8 CPUs, 1000GB external storage for HDFS each):
|
||||
- mapreduce tasktracker
|
||||
- hdfs datanode
|
||||
- hbase regionserver
|
||||
- solr (sharded)
|
||||
|
||||
- hive.t.hadoop.research-infrastructures.eu:
|
||||
- hue
|
||||
- hive
|
||||
- oozie
|
||||
- sqoop
|
||||
|
||||
- db.t.hadoop.research-infrastructures.eu:
|
||||
- postgresql instance for hue and hive
|
||||
|
||||
|
||||
Su jobtracker.t.hadoop.research-infrastructures.eu sono installati gli
|
||||
script che gestiscono tutti i servizi. È possibile fermare/attivare i
|
||||
singoli servizi oppure tutto il cluster, rispettando l'ordine
|
||||
corretto.
|
||||
|
||||
Hanno tutti prefisso "service-" e il nome dello script dà un'idea delle operazioni che verranno eseguite:
|
||||
service-global-hadoop-cluster
|
||||
service-global-hbase
|
||||
service-global-hdfs
|
||||
service-global-mapred
|
||||
service-global-zookeeper
|
||||
service-hbase-master
|
||||
service-hbase-regionserver
|
||||
service-hbase-rest
|
||||
service-hdfs-datanode
|
||||
service-hdfs-httpfs
|
||||
service-hdfs-journalnode
|
||||
service-hdfs-namenode
|
||||
service-hdfs-secondarynamenode
|
||||
service-mapreduce-jobtracker
|
||||
service-mapreduce-tasktracker
|
||||
service-zookeeper-server
|
||||
|
||||
Prendono come parametro "start,stop,status,restart"
|
||||
|
||||
|
||||
- jobtracker URL:
|
||||
http://jobtracker.t.hadoop.research-infrastructures.eu:50030/jobtracker.jsp
|
||||
- HDFS URL:
|
||||
http://namenode.t.hadoop.research-infrastructures.eu:50070/dfshealth.jsp
|
||||
- HBASE master URL:
|
||||
http://hbase-master.t.hadoop.research-infrastructures.eu:60010/master-status
|
||||
- HUE Web Interface:
|
||||
http://quorum2.t.hadoop.research-infrastructures.eu:8888
|
||||
|
||||
- URL ganglia, per le metriche del cluster:
|
||||
http://monitoring.research-infrastructures.eu/ganglia/?r=hour&cs=&ce=&s=by+name&c=Openaire%252B%2520Hadoop%2520TEST&tab=m&vn=
|
||||
- URL Nagios, per lo stato dei servizi (da attivare):
|
||||
http://monitoring.research-infrastructures.eu/nagios3/
|
||||
------------------------------------------------------------------------------------------------
|
||||
dom0/nodes/san map data
|
||||
|
||||
dlib18x: *node8* e90.6 (dlibsan9)
|
||||
dlib19x: *node9* e90.7 (dlibsan9)
|
||||
dlib20x: *node10* e90.8 (dlibsan9)
|
||||
dlib22x: *node11* e90.5 (dlibsan9)
|
||||
*node7* e63.4 (dlibsan6)
|
||||
dlib23x: *node12* e80.3 (dlibsan8)
|
||||
*node13* e80.4 (dlibsan8)
|
||||
dlib24x: *node2* e25.1 (dlibsan2)
|
||||
*node3* e74.1 (dlibsan7)
|
||||
dlib25x: *node4* e83.4 (dlibsan8)
|
||||
dlib26x: *node5* e72.1 (dlibsan7)
|
||||
*node6* e63.3 (dlibsan6)
|
||||
|
||||
------------------------------------------------------------------------------------------------
|
||||
Submitting a job (supporting multiple users)
|
||||
To support multiple users you create UNIX user accounts only in the master node.
|
||||
|
||||
Sul namenode:
|
||||
|
||||
#groupadd supergroup
|
||||
(da eseguire una sola volta)
|
||||
|
||||
#adduser claudio
|
||||
...
|
||||
|
||||
# su - hdfs
|
||||
$ hadoop dfs -mkdir /home/claudio
|
||||
$ hadoop dfs -chown -R claudio:supergroup /home/claudio
|
||||
|
||||
(aggiungere claudio al gruppo supergroup)
|
||||
|
||||
|
||||
Important:
|
||||
|
||||
If you do not create /tmp properly, with the right permissions as shown below, you may have problems with CDH components later. Specifically, if you don't create /tmp yourself, another process may create it automatically with restrictive permissions that will prevent your other applications from using it.
|
||||
|
||||
Create the /tmp directory after HDFS is up and running, and set its permissions to 1777 (drwxrwxrwt), as follows:
|
||||
|
||||
$ sudo -u hdfs hadoop fs -mkdir /tmp
|
||||
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
|
||||
|
||||
Note:
|
||||
|
||||
If Kerberos is enabled, do not use commands in the form sudo -u <user> <command>; they will fail with a security error. Instead, use the following commands: $ kinit <user> (if you are using a password) or $ kinit -kt <keytab> <principal> (if you are using a keytab) and then, for each command executed by this user, $ <command>
|
||||
Step 8: Create MapReduce /var directories
|
||||
|
||||
sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
|
||||
sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
|
||||
sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred
|
||||
|
||||
Step 9: Verify the HDFS File Structure
|
||||
|
||||
$ sudo -u hdfs hadoop fs -ls -R /
|
||||
|
||||
You should see:
|
||||
|
||||
drwxrwxrwt - hdfs supergroup 0 2012-04-19 15:14 /tmp
|
||||
drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var
|
||||
drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var/lib
|
||||
drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var/lib/hadoop-hdfs
|
||||
drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var/lib/hadoop-hdfs/cache
|
||||
drwxr-xr-x - mapred supergroup 0 2012-04-19 15:19 /var/lib/hadoop-hdfs/cache/mapred
|
||||
drwxr-xr-x - mapred supergroup 0 2012-04-19 15:29 /var/lib/hadoop-hdfs/cache/mapred/mapred
|
||||
drwxrwxrwt - mapred supergroup 0 2012-04-19 15:33 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
|
||||
|
||||
Step 10: Create and Configure the mapred.system.dir Directory in HDFS
|
||||
|
||||
After you start HDFS and create /tmp, but before you start the JobTracker (see the next step), you must also create the HDFS directory specified by the mapred.system.dir parameter (by default ${hadoop.tmp.dir}/mapred/system and configure it to be owned by the mapred user.
|
||||
|
||||
To create the directory in its default location:
|
||||
|
||||
$ sudo -u hdfs hadoop fs -mkdir /tmp/mapred/system
|
||||
$ sudo -u hdfs hadoop fs -chown mapred:hadoop /tmp/mapred/system
|
||||
|
||||
Important:
|
||||
|
||||
If you create the mapred.system.dir directory in a different location, specify that path in the conf/mapred-site.xml file.
|
||||
|
||||
When starting up, MapReduce sets the permissions for the mapred.system.dir directory to drwx------, assuming the user mapred owns that directory.
|
||||
Step 11: Start MapReduce
|
||||
|
||||
To start MapReduce, start the TaskTracker and JobTracker services
|
||||
|
||||
On each TaskTracker system:
|
||||
|
||||
$ sudo service hadoop-0.20-mapreduce-tasktracker start
|
||||
|
||||
On the JobTracker system:
|
||||
|
||||
$ sudo service hadoop-0.20-mapreduce-jobtracker start
|
||||
|
||||
Step 12: Create a Home Directory for each MapReduce User
|
||||
|
||||
Create a home directory for each MapReduce user. It is best to do this on the NameNode; for example:
|
||||
|
||||
$ sudo -u hdfs hadoop fs -mkdir /user/<user>
|
||||
$ sudo -u hdfs hadoop fs -chown <user> /user/<user>
|
||||
|
||||
where <user> is the Linux username of each user.
|
||||
|
||||
Alternatively, you can log in as each Linux user (or write a script to do so) and create the home directory as follows:
|
||||
|
||||
sudo -u hdfs hadoop fs -mkdir /user/$USER
|
||||
sudo -u hdfs hadoop fs -chown $USER /user/$USER
|
||||
|
||||
|
||||
|
||||
------------------------------------------------------------------------------------------------
|
||||
We use the jobtracker as provisioning server
|
||||
Correct start order (reverse to obtain the stop order):
|
||||
• HDFS (NB: substitute secondarynamenode with journalnode when we will have HA)
|
||||
• MapReduce
|
||||
• Zookeeper
|
||||
• HBase
|
||||
• Hive Metastore
|
||||
• Hue
|
||||
• Oozie
|
||||
• Ganglia
|
||||
• Nagios
|
||||
|
||||
|
||||
I comandi di init si trovano nel file "init.sh" nel repository ansible.
|
||||
|
||||
|
||||
Errore da indagare:
|
||||
http://stackoverflow.com/questions/6153560/hbase-client-connectionloss-for-hbase-error
|
||||
|
||||
# GC hints
|
||||
http://stackoverflow.com/questions/9792590/gc-tuning-preventing-a-full-gc?rq=1
|
||||
|
||||
HBASE troubleshooting
|
||||
|
||||
- Se alcune region rimangono in "transition" indefinitamente, è possibile provare a risolvere il problema da shell:
|
||||
|
||||
# su - hbase
|
||||
$ hbase hbck -fixAssignments
|
||||
|
||||
Potrebbe essere utile anche
|
||||
$ hbase hbck -repairHoles
|
||||
|
||||
-----------------------------------------------------
|
||||
Quando si verifica: "ROOT stuck in assigning forever"
|
||||
|
||||
bisogna:
|
||||
- verificare che non ci siano errori relativi a zookeeper. Se ci sono, far ripartire zookeeper e poi tutto il cluster hbase
|
||||
- Far ripartire il solo hbase master
|
||||
-----------------------------------------------------
|
||||
Quando ci sono tabelle disabilitate, ma che risultano impossibili da abilitare o eliminare:
|
||||
# su - hbase
|
||||
$ hbase hbck -fixAssignments
|
||||
|
||||
* Restart del master hbase
|
||||
|
||||
-----------------------------------------------------
|
||||
Vedi: http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/32838
|
||||
Ed in generale, per capire il funzionamento: http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
|
||||
|
||||
Tool per il monitoraggio di hbase quando è configurato per il manual splitting:
|
||||
https://github.com/sentric/hannibal
|
||||
|
||||
---------------------------------------------------------------------------------
|
||||
|
||||
2013-02-22 10:24:46,492 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@41a7fead
|
||||
2013-02-22 10:24:46,492 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_node2.t.hadoop.research-infrastructures.eu:localhost/127.0.0.1:47798
|
||||
2013-02-22 10:24:46,492 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1 and reserved physical memory is not configured. TaskMemoryManager is disabled.
|
||||
2013-02-22 10:24:46,571 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
|
||||
|
||||
---
|
||||
Post interessante che tratta la configurazione ed i vari parametri: http://gbif.blogspot.it/2011/01/setting-up-hadoop-cluster-part-1-manual.html
|
||||
|
||||
|
||||
Lista di nomi di parametri deprecati e il loro nuovo nome: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
|
||||
|
||||
---
|
||||
How to decommission a worker node
|
||||
|
||||
1. If they are many, reduce the hdfs redundancy factor
|
||||
2. Stop the regionserver on the node
|
||||
3. Add the node to the hdfs and jobtracker exclude list
|
||||
|
||||
./run.sh mapred.yml -i inventory/hosts.production -l jt_masters --tags=hadoop_workers
|
||||
./run.sh hadoop-hdfs.yml -i inventory/hosts.production -l hdfs_masters --tags=hadoop_workers
|
||||
|
||||
4. Refresh the hdfs and jobtracker configuration
|
||||
|
||||
hdfs dfsadmin -refreshNodes
|
||||
mapred mradmin -refreshNodes
|
||||
|
||||
5. Remove the node from the list of allowed ones
|
||||
|
||||
5a. Edit the inventory
|
||||
|
||||
5b. Run
|
||||
./run.sh hadoop-common.yml -i inventory/hosts.production --tags=hadoop_workers
|
||||
./run.sh mapred.yml -i inventory/hosts.production -l jt_masters --tags=hadoop_workers
|
||||
./run.sh hadoop-hdfs.yml -i inventory/hosts.production -l hdfs_masters --tags=hadoop_workers
|
||||
|
||||
---------------------------------------------------------------------------------
|
||||
Nagios monitoring
|
||||
|
||||
- The handlers to restart the services are managed via nrpe. To get them work, we need to:
|
||||
- Add an entry in nrpe.cfg. The command name needs to start with "global_restart_" and
|
||||
the remaining part of the name must coincide with the name of the service.
|
||||
For example:
|
||||
command[global_restart_hadoop-0.20-mapreduce-tasktracker]=/usr/bin/sudo /usr/sbin/service hadoop-0.20-mapreduce-tasktracker restart
|
||||
- Add a handler to the nagios service. The command needs the service name as parameter
|
||||
Example:
|
||||
event_handler restart-service!hadoop-0.20-mapreduce-tasktracker
|
||||
|
||||
---------------------------------------------------------------------------------
|
||||
Ansible playbook that installs and configures a Hadoop cluster.
|
32
README.new
32
README.new
|
@ -1,32 +0,0 @@
|
|||
|
||||
httpfs: We need to install it on only one machine (two for redundancy). Let's use the namenodes.
|
||||
|
||||
Move the second jobtracker on a dedicated machine.
|
||||
|
||||
hbase thrift: let's have two of them, on the nodes that run the hbase masters
|
||||
|
||||
Impala: needs to be installed on all the datanodes. After that, hue-impala can be installed on the hue server
|
||||
|
||||
NB: /etc/zookeeper/conf/zoo.cfg needs to be distributed on all
|
||||
datanodes.
|
||||
|
||||
Create the new disks: lvcreate -l 238465 -n node11.t.hadoop.research-infrastructures.eu-data-hdfs dlibsan6 /dev/md3
|
||||
# Move the data:
|
||||
rsync -qaxvH --delete --numeric-ids /mnt/disk/ dlibsan7:/mnt/disk/
|
||||
|
||||
----------
|
||||
dfs.socket.timeout, for read timeout
|
||||
dfs.datanode.socket.write.timeout, for write timeout
|
||||
|
||||
In fact, the read timeout value is used for various connections in
|
||||
DFSClient, if you only increase dfs.datanode.socket.write.timeout, the
|
||||
timeout can continue to happen.
|
||||
|
||||
I tried to generate 1TB data with teragen across more than 40 data
|
||||
nodes, increasing writing timeout has not fixed the problem. When I
|
||||
increased both values above 600000, it disappeared.
|
||||
----------
|
||||
|
||||
|
||||
To configure yarn:
|
||||
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html
|
|
@ -1,2 +0,0 @@
|
|||
monitorRole readonly
|
||||
controlRole readwrite
|
|
@ -1,374 +0,0 @@
|
|||
---
|
||||
# Generic machines data
|
||||
time_zone: 'Europe/Rome'
|
||||
cpu_cores: 8
|
||||
datanode_ram: 11000
|
||||
nagios_enabled: True
|
||||
ganglia_enabled: False
|
||||
ganglia_gmond_hdfs_datanodes_port: "8640:8660"
|
||||
ganglia_gmond_jobtracker_port: "8640:8660"
|
||||
ganglia_gmond_hbmaster_port: "8640:8660"
|
||||
ganglia_gmond_namenode_port: "8640:8660"
|
||||
configure_munin: True
|
||||
|
||||
# JDK (Oracle)
|
||||
jdk_version:
|
||||
- 7
|
||||
- 8
|
||||
jdk_default: 8
|
||||
java_home: '/usr/lib/jvm/java-{{ jdk_default }}-oracle'
|
||||
jdk_java_home: '{{ java_home }}'
|
||||
|
||||
# PKG state: latest or present. Set to 'latest' when you want to upgrade the installed packages version.
|
||||
hadoop_pkg_state: present
|
||||
#
|
||||
#
|
||||
# Global data
|
||||
#
|
||||
worker_nodes_num: 4
|
||||
worker_node_start: 2
|
||||
worker_node_end: 5
|
||||
worker_node_swappiness: 0
|
||||
|
||||
dns_domain: t.hadoop.research-infrastructures.eu
|
||||
namenode_hostname: 'nn1.{{ dns_domain }}'
|
||||
secondary_nm_hostname: 'nn2.{{ dns_domain }}'
|
||||
quorum_0_node_hostname: 'quorum0.{{ dns_domain }}'
|
||||
quorum_1_node_hostname: 'quorum1.{{ dns_domain }}'
|
||||
quorum_2_node_hostname: 'quorum2.{{ dns_domain }}'
|
||||
quorum_3_node_hostname: 'quorum3.{{ dns_domain }}'
|
||||
quorum_4_node_hostname: 'quorum4.{{ dns_domain }}'
|
||||
hbase_master_1_hostname: 'hbase-master1.{{ dns_domain }}'
|
||||
hbase_master_2_hostname: 'hbase-master2.{{ dns_domain }}'
|
||||
|
||||
ldap:
|
||||
server: ldap://ldap.sub.research-infrastructures.eu
|
||||
search_bind_auth: False
|
||||
username_pattern: "uid=<username>,ou=People,o=Users,ou=Organizations,dc=research-infrastructures,dc=eu"
|
||||
|
||||
hadoop_ldap_uri: ldap://ldap.sub.research-infrastructures.eu
|
||||
hadoop_ldap_base_dn: "dc=research-infrastructures,dc=eu"
|
||||
hadoop_ldap_search_bind_auth: False
|
||||
hadoop_ldap_username_pattern: "uid=<username>,ou=People,o=Users,ou=Organizations,dc=research-infrastructures,dc=eu"
|
||||
|
||||
#
|
||||
# LOGGING
|
||||
#
|
||||
# WARN,INFO,DEBUG,ERROR
|
||||
hadoop_log_level: INFO
|
||||
#
|
||||
# RFA is the rolling file appender
|
||||
hadoop_log_appender: RFA
|
||||
hadoop_log_appender_max_filesize: 256MB
|
||||
# max backup index is ignored if the appender is daily rolling file
|
||||
hadoop_log_appender_max_backupindex: 10
|
||||
#
|
||||
# We can use a logstash collector
|
||||
hadoop_send_to_logstash: False
|
||||
# Ditch the local appender if you want a logstash only solution
|
||||
hadoop_logstash_appender: RFA,LOGSTASH
|
||||
hadoop_logstash_collector_host: 'logstash.{{ dns_domain }}'
|
||||
hadoop_logstash_collector_socketappender_port: 4560
|
||||
hadoop_logstash_collector_socketappender_reconndelay: 10000
|
||||
#
|
||||
# rsyslog
|
||||
rsyslog_install_newer_package: True
|
||||
rsyslog_send_to_elasticsearch: False
|
||||
rsyslog_use_queues: False
|
||||
rsyslog_use_elasticsearch_module: False
|
||||
rsys_elasticsearch_collector_host: '{{ hadoop_logstash_collector_host }}'
|
||||
rsys_elasticsearch_collector_port: 9200
|
||||
|
||||
#
|
||||
# General hadoop
|
||||
#
|
||||
initialize_hadoop_cluster: False
|
||||
|
||||
hadoop_cluster_name: "nmis-hadoop-cluster"
|
||||
hadoop_data_dir: /data
|
||||
hadoop_conf_dir: '/etc/hadoop/conf.{{ hadoop_cluster_name|lower }}'
|
||||
hadoop_mapred_home: /usr/lib/hadoop-0.20-mapreduce
|
||||
|
||||
hadoop_hdfs_data_disk:
|
||||
- { mountpoint: '/data', device: 'xvda3', fstype: 'xfs' }
|
||||
#
|
||||
# Hadoop default heapsize
|
||||
# The default is 1000
|
||||
hadoop_default_heapsize: 1024
|
||||
hadoop_default_java_opts: "-server -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8"
|
||||
hadoop_jmx_enabled: False
|
||||
|
||||
#
|
||||
# HDFS
|
||||
#
|
||||
hdfs_cluster_id: '{{ hadoop_cluster_name }}'
|
||||
hdfs_cluster_nn_id_1: nn1
|
||||
hdfs_cluster_nn_id_2: nn2
|
||||
hdfs_cluster_ids: "{{ hdfs_cluster_nn_id_1 }},{{ hdfs_cluster_nn_id_2 }}"
|
||||
hdfs_namenode_1_hostname: '{{ namenode_hostname }}'
|
||||
hdfs_namenode_2_hostname: '{{ secondary_nm_hostname }}'
|
||||
hdfs_data_dir: '{{ hadoop_data_dir }}/dfs'
|
||||
hdfs_nn_data_dir: nn
|
||||
hdfs_dn_data_dir: dn
|
||||
hdfs_dn_balance_bandwidthPerSec: 2097152
|
||||
hdfs_support_append: "true"
|
||||
hdfs_nn_rpc_port: 8020
|
||||
hdfs_nn_http_port: 50070
|
||||
hdfs_nn_client_port: 57045
|
||||
# handler count. Recommended: ln(number of datanodes) * 20
|
||||
hdfs_nn_handler_count: 50
|
||||
# Recommended: up to 128MB, 134217728 bytes (this is the default, is a client parameter)
|
||||
hdfs_block_size: 16777216
|
||||
hdfs_repl_max: 256
|
||||
hdfs_replication: 1
|
||||
# Set to 0 to disable the trash use. Note that the client can enable it.
|
||||
hdfs_fs_trash_interval: 10060
|
||||
hdfs_datanode_max_xcievers: 1024
|
||||
hdfs_datanode_http_port: 50075
|
||||
hdfs_datanode_ipc_port: 50020
|
||||
hdfs_datanode_rpc_port: 50010
|
||||
hdfs_dfs_socket_timeout: 600000
|
||||
hdfs_dfs_socket_write_timeout: 600000
|
||||
# See http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_6.html
|
||||
hdfs_read_shortcircuit: True
|
||||
hdfs_read_shortcircuit_cache_size: 3000
|
||||
hdfs_read_shortcircuit_cache_expiry: 50000
|
||||
hdfs_read_shortcircuit_cache_dir: '/var/run/hadoop-hdfs'
|
||||
hdfs_journal_id: '{{ hdfs_cluster_id }}'
|
||||
hdfs_journal_port: 8485
|
||||
hdfs_journal_0: '{{ quorum_0_node_hostname }}'
|
||||
hdfs_journal_1: '{{ quorum_1_node_hostname }}'
|
||||
hdfs_journal_2: '{{ quorum_2_node_hostname }}'
|
||||
hdfs_journal_3: '{{ quorum_3_node_hostname }}'
|
||||
hdfs_journal_4: '{{ quorum_4_node_hostname }}'
|
||||
hdfs_journal_data_dir: jn
|
||||
hdfs_journal_http_port: 8480
|
||||
hdfs_zkfc_port: 8019
|
||||
hdfs_webhdfs_enabled: True
|
||||
hdfs_users_supergroup: supergroup
|
||||
# The following is used to retrieve the ssh key needed for the HA failover
|
||||
hdfs_user_home: /usr/lib/hadoop
|
||||
|
||||
httpfs_user: httpfs
|
||||
httpfs_host: 'hue.{{ dns_domain }}'
|
||||
httpfs_host_1: 'nn1.{{ dns_domain }}'
|
||||
httpfs_host_2: 'nn2.{{ dns_domain }}'
|
||||
httpfs_port: 14000
|
||||
httpfs_catalina_work_dir: /usr/lib/hadoop-httpfs/work
|
||||
|
||||
#
|
||||
# Zookeeper
|
||||
zookeeper_conf_dir: '/etc/zookeeper/conf.{{ hadoop_cluster_name|lower }}'
|
||||
zookeeper_log_dir: '/var/log/zookeeper'
|
||||
zookeeper_client_port: 2182
|
||||
zookeeper_quorum_port: 4182
|
||||
zookeeper_leader_port: 3182
|
||||
zookeeper_min_timeout: 30000
|
||||
zookeeper_max_timeout: 240000
|
||||
zookeeper_quorum_0: '{{ quorum_0_node_hostname }}'
|
||||
zookeeper_quorum_1: '{{ quorum_1_node_hostname }}'
|
||||
zookeeper_quorum_2: '{{ quorum_2_node_hostname }}'
|
||||
zookeeper_quorum_3: '{{ quorum_3_node_hostname }}'
|
||||
zookeeper_quorum_4: '{{ quorum_4_node_hostname }}'
|
||||
zookeeper_maxclient_connections: 240
|
||||
zookeeper_nodes: "{{ zookeeper_quorum_0 }},{{ zookeeper_quorum_1 }},{{ zookeeper_quorum_2 }},{{ zookeeper_quorum_3 }},{{ zookeeper_quorum_4 }}"
|
||||
zookeeper_cluster: "{{ zookeeper_quorum_0 }}:{{ zookeeper_client_port }},{{ zookeeper_quorum_1 }}:{{ zookeeper_client_port }},{{ zookeeper_quorum_2 }}:{{ zookeeper_client_port }},{{ zookeeper_quorum_3 }}:{{ zookeeper_client_port }},{{ zookeeper_quorum_4 }}:{{ zookeeper_client_port }}"
|
||||
|
||||
#
|
||||
# Jobtracker
|
||||
#
|
||||
jobtracker_cluster_id: nmis-hadoop-jt
|
||||
jobtracker_node_1_hostname: 'jobtracker.{{ dns_domain }}'
|
||||
jobtracker_node_2_hostname: 'jobtracker2.{{ dns_domain }}'
|
||||
jobtracker_cluster_id_1: jt1
|
||||
jobtracker_cluster_id_2: jt2
|
||||
jobtracker_cluster_id1_rpc_port: 8021
|
||||
jobtracker_cluster_id2_rpc_port: 8022
|
||||
jobtracker_cluster_id1_ha_rpc_port: 8023
|
||||
jobtracker_cluster_id2_ha_rpc_port: 8024
|
||||
jobtracker_cluster_id1_http_port: 50030
|
||||
jobtracker_cluster_id2_http_port: 50031
|
||||
jobtracker_http_port: 9290
|
||||
jobtracker_persistent_jobstatus: 'true'
|
||||
jobtracker_restart_recover: 'false'
|
||||
jobtracker_failover_connect_retries: 3
|
||||
jobtracker_auto_failover_enabled: 'true'
|
||||
jobtracker_zkfc_port: 8018
|
||||
# handler count. Recommended: ln(number of datanodes) * 20
|
||||
jobtracker_handler_count: 50
|
||||
|
||||
|
||||
# We have 12 nodes and 6 CPUs per node
|
||||
# reduce tasks forumla: 0.95 or 1.75 * (nodes * mapred.tasktracker.tasks.maximum)
|
||||
# Cloudera defaults: 2 mappers, 2 reducers max
|
||||
# ------
|
||||
# tested. too much stress on the hardware
|
||||
#mapred_tasktracker_map_tasks_maximum: 6
|
||||
#mapred_tasktracker_reduce_tasks_maximum: 68
|
||||
#mapred_reduce_child_java_opts: "-Xmx2G"
|
||||
# ------
|
||||
mapred_tasktracker_http_port: 50060
|
||||
mapred_tasktracker_map_tasks_maximum: 2
|
||||
mapred_tasktracker_reduce_tasks_maximum: 4
|
||||
mapred_use_fair_scheduler: True
|
||||
mapred_fair_scheduler_pools:
|
||||
- { name: 'solr', map: '12', reduce: '18' }
|
||||
mapred_fair_scheduler_use_poolnameproperty: True
|
||||
mapred_fair_scheduler_poolnameproperty: user.name
|
||||
mapred_fair_scheduler_undecl_pools: True
|
||||
mapred_fair_scheduler_preemption: False
|
||||
mapred_fair_scheduler_assignmultiple: True
|
||||
mapred_fair_scheduler_allocation_file: '{{ hadoop_conf_dir }}/fair-scheduler.xml'
|
||||
# reducer parallel copies. Recommended: ln(number of datanodes) * 4
|
||||
# with a minimum of 10
|
||||
mapred_reduce_parallel_copies: 10
|
||||
# Recommended: 80
|
||||
mapred_tasktracker_http_threads: 80
|
||||
# Default: 0.05. Recommended: 0.8. Used by the jobtracker
|
||||
mapred_reduce_slowstart_maps: 0.9
|
||||
# Default: 100. We could increase it
|
||||
mapred_tasktracker_io_sort_mb: 256
|
||||
mapred_io_sort_factor: 25
|
||||
mapreduce_job_counters_max: 5000
|
||||
mapred_userlog_retain_hours: 24
|
||||
mapred_jt_completeuserjobs_max: 150
|
||||
mapred_jt_persist_jobstatus_hours: 4320
|
||||
mapred_user_jobconf_limit: 5242880
|
||||
mapred_jt_retirejob_interval: 86400000
|
||||
mapreduce_jt_split_metainfo_maxsize: 10000000
|
||||
mapred_queue_names: default
|
||||
#
|
||||
mapred_staging_root_dir: /user
|
||||
mapred_old_staging_root_dir: /home
|
||||
mapred_local_dir: /data/mapred/local
|
||||
# Java parameters
|
||||
mapred_child_java_opts: "-Xmx3092M"
|
||||
mapred_map_child_java_opts: "-Xmx3092M"
|
||||
#mapred_reduce_child_java_opts: "-Xmx1512M"
|
||||
mapred_reduce_child_java_opts: "-Xmx2048M"
|
||||
|
||||
#
|
||||
# HBASE
|
||||
#
|
||||
# Raw formula to calculate the needed regionserver heap size:
|
||||
# regions.hbase.hregion.max.filesize /
|
||||
# hbase.hregion.memstore.flush.size *
|
||||
# dfs.replication *
|
||||
# hbase.regionserver.global.memstore.lowerLimit
|
||||
# See: http://hadoop-hbase.blogspot.it/2013/01/hbase-region-server-memory-sizing.html
|
||||
#
|
||||
hbase_user: hbase
|
||||
hbase_conf_dir: '/etc/hbase/conf.{{ hadoop_cluster_name|lower }}'
|
||||
# HBASE heap size
|
||||
hbase_master_heap_size: 5120
|
||||
hbase_thrift_heap_size: 1024
|
||||
hbase_regionserver_heap_size: 4500
|
||||
hbase_master_java_opts: '-Xmx{{ hbase_master_heap_size }}M'
|
||||
hbase_regionserver_maxdirectmemory_size: "-XX:MaxDirectMemorySize=2G"
|
||||
hbase_regionserver_java_opts: '-Xmx{{ hbase_regionserver_heap_size }}M'
|
||||
hbase_thrift_java_opts: '-Xmx{{ hbase_thrift_heap_size }}M'
|
||||
hbase_zookeeper_java_opts: -Xmx1G
|
||||
hbase_thrift_port: 9090
|
||||
hbase_thrift_jmx_port: 9591
|
||||
# hbase zookeeper timeout
|
||||
hbase_zookeeper_timeout: '{{ zookeeper_max_timeout }}'
|
||||
# rpc timeout needs to be greater than lease period
|
||||
# See http://hbase.apache.org/book/trouble.client.html
|
||||
hbase_rpc_timeout: 600000
|
||||
hbase_lease_period: 400000
|
||||
hbase_open_files: 65536
|
||||
hbase_master_rpc_port: 60000
|
||||
hbase_master_http_port: 60010
|
||||
hbase_regionserver_http_port: 60030
|
||||
hbase_regionserver_http_1_port: 60020
|
||||
# This is controversial. When set to 'true' hdfs balances
|
||||
# each table without paying attention to the global balancing
|
||||
hbase_loadbalance_bytable: True
|
||||
# Default is 0.2
|
||||
hbase_regions_slop: 0.15
|
||||
# Default is 10. The recommendation is to keep it low when the payload per request grows
|
||||
# We have mixed payloads.
|
||||
hbase_handler_count: 12
|
||||
# Default was 256M. It's 10737418240 (10GB) since 0.94
|
||||
# The recommendation is to have it big to decrease the total number of regions
|
||||
# 1288490188 is circa 1.2GB
|
||||
hbase_hregion_max_file_size: 1288490188
|
||||
hbase_hregion_memstore_mslab_enabled: True
|
||||
# The default 134217728, 128MB. We set it to 256M
|
||||
hbase_hregion_memstore_flush_size: 268435456
|
||||
# The default is 0.4
|
||||
hbase_regionserver_global_memstore_lowerLimit: 0.35
|
||||
#
|
||||
hbase_regionserver_global_memstore_upperLimit: 0.45
|
||||
hbase_hregion_memstore_block_multiplier: 3
|
||||
# HBASE thrift server
|
||||
hbase_thrift_server_1: '{{ hbase_master_1_hostname }}'
|
||||
hbase_thrift_server_2: '{{ hbase_master_2_hostname }}'
|
||||
|
||||
#
|
||||
# nginx uses as reverse proxy to all the web interfaces
|
||||
#
|
||||
nginx_use_ldap_pam_auth: True
|
||||
nginx_pam_svc_name: nginx
|
||||
nginx_ldap_uri: '{{ hadoop_ldap_uri }}'
|
||||
nginx_ldap_base_dn: '{{ hadoop_ldap_base_dn }}'
|
||||
|
||||
portal_nginx_conf: management-portal
|
||||
portal_pam_svc_name: '{{ nginx_pam_svc_name }}'
|
||||
portal_title: "NeMIS Hadoop Cluster"
|
||||
portal_web_root: /usr/share/nginx/www
|
||||
|
||||
#
|
||||
# OOZIE and HIVE DB data
|
||||
#
|
||||
oozie_db_type: postgresql
|
||||
oozie_db_name: oozie
|
||||
oozie_db_user: oozie
|
||||
oozie_db_host: db.t.hadoop.research-infrastructures.eu
|
||||
hive_db_type: '{{ oozie_db_type }}'
|
||||
hive_db_name: hive
|
||||
hive_db_user: hive
|
||||
hive_db_host: '{{ oozie_db_host }}'
|
||||
hive_metastore_db_type: '{{ oozie_db_type }}'
|
||||
hive_metastore_db_name: metastore
|
||||
hive_metastore_db_user: metastore
|
||||
hive_metastore_db_host: '{{ oozie_db_host }}'
|
||||
hue_db_type: '{{ oozie_db_type }}'
|
||||
hue_db_name: hue
|
||||
hue_db_user: hue
|
||||
hue_db_host: '{{ oozie_db_host }}'
|
||||
hue_http_port: 8888
|
||||
oozie_ip: 146.48.123.66
|
||||
hive_ip: '{{ oozie_ip }}'
|
||||
hue_ip: '{{ oozie_ip }}'
|
||||
|
||||
# Iptables
|
||||
|
||||
other_networks:
|
||||
# Marek
|
||||
icm_pl: 213.135.59.0/24
|
||||
# eri.katsari
|
||||
icm_pl_1: 195.134.66.216/32
|
||||
# Antonis addresses, need to reach hdfs and zookeeper (ARC). And Glykeria Katsari
|
||||
ilsp_gr: [ '194.177.192.226/32', '194.177.192.223/32', '195.134.66.96/32', '194.177.192.218/32', '194.177.192.231/32', '195.134.66.216/32', '195.134.66.145/32', '194.177.192.118/32', '195.134.66.244' ]
|
||||
# Needed by marek. It's the IIS cluster gateway.
|
||||
iis_pl_1: 213.135.60.74/32
|
||||
# Jochen
|
||||
icm_1: 129.70.43.118/32
|
||||
|
||||
monitoring_group_name: hadoop-cluster
|
||||
|
||||
nagios_local_plugins_dir: /usr/lib/nagios/plugins/hadoop
|
||||
nagios_common_lib: check_library.sh
|
||||
nagios_monitoring_dir: '/etc/nagios3/objects/{{ monitoring_group_name }}'
|
||||
nagios_root_disk: /
|
||||
nagios_check_disk_w: 10%
|
||||
nagios_check_disk_c: 7%
|
||||
nagios_service_contacts:
|
||||
- andrea.dellamico
|
||||
- claudio.atzori
|
||||
nagios_contactgroup: hadoop-managers
|
||||
nagios_monitoring_server_ip: 146.48.123.23
|
||||
|
||||
iptables_default_policy: REJECT
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
---
|
||||
# Ganglia
|
||||
ganglia_unicast_mode: False
|
||||
|
||||
ganglia_gmond_jobtracker_cluster: "Openaire+ Hadoop Cluster - Jobtrackers"
|
||||
ganglia_gmond_namenode_cluster: "Openaire+ Hadoop Cluster - HDFS namenodes"
|
||||
ganglia_gmond_hbmaster_cluster: "Openaire+ Hadoop Cluster - HBASE masters"
|
||||
ganglia_gmond_workers_cluster: "Openaire+ Hadoop Cluster - Worker nodes"
|
||||
|
||||
ganglia_gmond_cluster: '{{ ganglia_gmond_workers_cluster }}'
|
||||
|
||||
#
|
||||
# To play nice with iptables
|
||||
ganglia_gmond_mcast_addr: 239.2.11.0
|
||||
ganglia_gmond_cluster_port: "8640:8660"
|
||||
|
||||
# jmx ports
|
||||
hadoop_namenode_jmx_port: 10103
|
||||
hadoop_secondary_namenode_jmx_port: 10104
|
||||
hadoop_datanode_jmx_port: 10105
|
||||
hadoop_balancer_jmx_port: 10106
|
||||
hadoop_jobtracker_jmx_port: 10107
|
||||
hbase_master_jmx_port: 10101
|
||||
hbase_regionserver_jmx_port: 10102
|
||||
hbase_thrift_jmx_port: 10109
|
||||
hbase_zookeeper_jmx_port: 10110
|
||||
zookeeper_jmx_port: 10108
|
|
@ -1,32 +0,0 @@
|
|||
---
|
||||
# jmx ports
|
||||
hadoop_namenode_jmx_port: 10103
|
||||
hadoop_secondary_namenode_jmx_port: 10104
|
||||
hadoop_datanode_jmx_port: 10105
|
||||
hadoop_balancer_jmx_port: 10106
|
||||
hadoop_jobtracker_jmx_port: 10107
|
||||
hbase_master_jmx_port: 10101
|
||||
hbase_regionserver_jmx_port: 10102
|
||||
hbase_thrift_jmx_port: 10109
|
||||
hbase_zookeeper_jmx_port: 10110
|
||||
zookeeper_jmx_port: 10108
|
||||
#
|
||||
# Used by nagios
|
||||
hadoop_plugins_dir: /usr/lib/nagios/plugins/hadoop
|
||||
root_disk: /dev/xvda2
|
||||
data_disk: /dev/xvda3
|
||||
root_disk_warn: 20%
|
||||
disk_warn: '{{ root_disk_warn }}'
|
||||
root_disk_crit: 10%
|
||||
disk_crit: '{{ root_disk_crit }}'
|
||||
data_disk_warn: 7%
|
||||
data_disk_crit: 4%
|
||||
|
||||
hbase_check_user: hbasecheck
|
||||
hbase_check_timeout: 560
|
||||
hdfs_warn: 90
|
||||
hdfs_crit: 95
|
||||
|
||||
nagios_proclist_red: '{{ redprocs }}'
|
||||
nagios_proclist_yellow: '{{ yellowprocs }}'
|
||||
nagios_nrpe_port: 5666
|
|
@ -1,23 +0,0 @@
|
|||
---
|
||||
#
|
||||
# The OOZIE users are a subset of the hdfs users.
|
||||
#
|
||||
hadoop_users:
|
||||
- { login: 'marek.horst', name: "Marek Horst", ssh_key: '{{ marek_horst }}', shell: '/bin/bash' }
|
||||
- { login: 'claudio.atzori', name: "Claudio Atzori", ssh_key: '{{ claudio_atzori }}', shell: '/bin/bash' }
|
||||
- { login: 'sandro.labruzzo', name: "Sandro Labruzzo", ssh_key: '{{ sandro_labruzzo }}', shell: '/bin/bash' }
|
||||
- { login: 'michele.artini', name: "Michele Artini", ssh_key: '{{ michele_artini }}', shell: '/bin/bash' }
|
||||
- { login: 'alessia.bardi', name: "Alessia Bardi", ssh_key: '{{ alessia_bardi }}', shell: '/bin/bash' }
|
||||
- { login: 'andrea.mannocci', name: "Andrea Mannocci", ssh_key: '{{ andrea_mannocci }}', shell: '/bin/bash' }
|
||||
- { login: 'andrea.dellamico', name: "Andrea Dell'Amico", ssh_key: '{{ andrea_dellamico }}', shell: '/bin/bash' }
|
||||
- { login: 'giorgos.alexiou', name: "Giorgos Alexiou", ssh_key: '{{ giorgos_alexiou }}', shell: '/bin/bash' }
|
||||
- { login: 'antonis.lempesis', name: "Antonis Lempesis", ssh_key: '{{ antonis_lempesis }}', shell: '/bin/bash' }
|
||||
- { login: 'dnet' }
|
||||
- { login: 'claudio' }
|
||||
- { login: 'michele' }
|
||||
- { login: 'sandro' }
|
||||
- { login: 'alessia' }
|
||||
- { login: 'andrea' }
|
||||
- { login: 'adellam' }
|
||||
- { login: 'hbasecheck' }
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
$ANSIBLE_VAULT;1.1;AES256
|
||||
63613435386665626236306331353063626137386531346461646463623436376232303461653436
|
||||
3934313830326366373364396630356630623935633230360a646439346530363762363966643534
|
||||
30373331666537666266353666333632616465666331383231356661633838633432656536653233
|
||||
3738636134393763650a623637326339653932323563346336366433333732373733656532353137
|
||||
36306364343430303535373961646632656535666162363862613036356461343865
|
|
@ -1,10 +0,0 @@
|
|||
$ANSIBLE_VAULT;1.1;AES256
|
||||
39646636653439616665643935326563653435646462306639646266376232633436393834643933
|
||||
3364336430396530646637383438663037366362663135320a373065343862653035653838323739
|
||||
61646135626431643330363963666433303737663464396663353632646339653562666162393034
|
||||
3363383435346364310a356439323431343336366635306461613462663436326431383266366231
|
||||
39636262313038366135316331343939373064356336356239653631633435613736306131656363
|
||||
37613864353931396435353431633765623330663266646666643632626666643436623939303538
|
||||
34343461383338663466303131663336326230666532326335373862636437343739336136616435
|
||||
35653763353436383537633932316434303539373237336161303165353962356336666161323765
|
||||
6336
|
|
@ -1,18 +0,0 @@
|
|||
---
|
||||
psql_version: 9.1
|
||||
psql_db_host: localhost
|
||||
psql_db_data:
|
||||
- { name: '{{ oozie_db_name }}', encoding: 'UTF8', user: '{{ oozie_db_user }}', roles: 'CREATEDB,NOSUPERUSER', pwd: '{{ psql_db_pwd }}', allowed_hosts: [ '{{ oozie_ip }}/32' ] }
|
||||
- { name: '{{ hue_db_name }}', encoding: 'UTF8', user: '{{ hue_db_user }}', roles: 'CREATEDB,NOSUPERUSER', pwd: '{{ psql_db_pwd }}', allowed_hosts: [ '{{ hue_ip }}/32' ] }
|
||||
- { name: '{{ hive_metastore_db_name }}', encoding: 'UTF8', user: '{{ hive_metastore_db_user }}', roles: 'CREATEDB,NOSUPERUSER', pwd: '{{ psql_db_pwd }}', allowed_hosts: [ '{{ hive_ip }}/32' ] }
|
||||
psql_listen_on_ext_int: True
|
||||
|
||||
pg_backup_pgdump_bin: /usr/lib/postgresql/9.1/bin/pg_dump
|
||||
pg_backup_retain_copies: 10
|
||||
pg_backup_build_db_list: "no"
|
||||
pg_backup_db_list: "'{{ oozie_db_name }}' '{{ hue_db_name }}' '{{ hive_metastore_db_name }}'"
|
||||
pg_backup_destdir: /data/pgsql/backups
|
||||
pg_backup_logfile: '{{ pg_backup_logdir }}/postgresql-backup.log'
|
||||
pg_backup_use_nagios: "yes"
|
||||
|
||||
user_ssh_key: [ '{{ claudio_atzori }}' ]
|
|
@ -1,2 +0,0 @@
|
|||
---
|
||||
user_ssh_key: [ '{{ claudio_atzori }}', '{{ hadoop_test_cluster }}', '{{ sandro_labruzzo }}' ]
|
|
@ -1,18 +0,0 @@
|
|||
---
|
||||
#
|
||||
# The hadoop logs are now sent to logstash directly by log4j
|
||||
# - adellam 2015-02-04
|
||||
#
|
||||
# the log_state_file names must be unique when using the old rsyslog syntax. In the new one
|
||||
# they are not used
|
||||
# rsys_logfiles:
|
||||
# - { logfile: '/var/log/hadoop-0.20-mapreduce/hadoop-{{ hadoop_cluster_name }}-jobtrackerha-{{ ansible_hostname }}.log', log_tag: 'hadoop-jobtracker', log_state_file: 'hadoop-jobtracker'}
|
||||
# - { logfile: '/var/log/hadoop-0.20-mapreduce/hadoop-{{ hadoop_cluster_name }}-mrzkfc-{{ ansible_hostname }}.log', log_tag: 'hadoop-jt-mrzkfc', log_state_file: 'hadoop-jt-mrzkfc'}
|
||||
# - { logfile: '/var/log/hadoop-0.20-mapreduce/mapred-audit.log', log_tag: 'hadoop-mapred-audit', log_state_file: 'hadoop-mapred-audit'}
|
||||
# - { logfile: '/var/log/hadoop-hdfs/hadoop-{{ hadoop_cluster_name }}-namenode-{{ ansible_hostname }}.log', log_tag: 'hadoop-hdfs-namenode', log_state_file: 'hadoop-hdfs-namenode'}
|
||||
# - { logfile: '/var/log/hadoop-hdfs/hadoop-{{ hadoop_cluster_name }}-zkfc-{{ ansible_hostname }}.log', log_tag: 'hadoop-hdfs-zkfc', log_state_file: 'hadoop-hdfs-zkfc'}
|
||||
# - { logfile: '/var/log/hadoop-hdfs/hadoop-{{ hadoop_cluster_name }}-journalnode-{{ ansible_hostname }}.log', log_tag: 'hadoop-hdfs-journal', log_state_file: 'hadoop-hdfs-journal'}
|
||||
# - { logfile: '/var/log/hbase/hbase.log', log_tag: 'hbase-master-log', log_state_file: 'hbase-master-log'}
|
||||
# - { logfile: '/var/log/hbase/hbase-hbase-master-{{ ansible_hostname }}.log', log_tag: 'hbase-master-ha', log_state_file: 'hbase-master-ha'}
|
||||
# - { logfile: '/var/log/hbase/hbase-hbase-thrift-{{ ansible_hostname }}.log', log_tag: 'hbase-thrift', log_state_file: 'hbase-thrift'}
|
||||
# - { logfile: '{{ zookeeper_log_dir }}/zookeeper.log', log_tag: 'hadoop-zookeeper', log_state_file: 'hadoop-zookeeper'}
|
|
@ -1,6 +0,0 @@
|
|||
---
|
||||
# Ganglia gmond port
|
||||
ganglia_gmond_cluster: '{{ ganglia_gmond_workers_cluster }}'
|
||||
ganglia_gmond_cluster_port: '{{ ganglia_gmond_hdfs_datanodes_port }}'
|
||||
ganglia_gmond_mcast_address: '{{ ganglia_gmond_workers_mcast_addr }}'
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
---
|
||||
iptables:
|
||||
tcp_rules: True
|
||||
tcp:
|
||||
- { port: '{{ hdfs_datanode_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}' ] }
|
||||
- { port: '{{ hdfs_datanode_ipc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}' ] }
|
||||
- { port: '{{ hdfs_datanode_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ mapred_tasktracker_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hbase_regionserver_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hbase_regionserver_http_1_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
|
@ -1,10 +0,0 @@
|
|||
---
|
||||
#
|
||||
# The hadoop logs are now sent to logstash directly by log4j
|
||||
# - adellam 2015-02-04
|
||||
#
|
||||
# IMPORTANT: the log_state_file names must be unique
|
||||
# rsys_logfiles:
|
||||
# - { logfile: '/var/log/hadoop-0.20-mapreduce/hadoop-{{ hadoop_cluster_name }}-tasktracker-{{ ansible_hostname }}.log', log_tag: 'hadoop-tasktracker', log_state_file: 'hadoop-tasktracker'}
|
||||
# - { logfile: '/var/log/hadoop-hdfs/hadoop-{{ hadoop_cluster_name }}-datanode-{{ ansible_hostname }}.log', log_tag: 'hadoop-hdfs-datanode', log_state_file: 'hadoop-hdfs-datanode'}
|
||||
# - { logfile: '/var/log/hbase/hbase-hbase-regionserver-{{ ansible_hostname }}.log', log_tag: 'hbase-regionserver', log_state_file: 'hbase-regionserver'}
|
|
@ -1,6 +0,0 @@
|
|||
---
|
||||
# Ganglia gmond port
|
||||
ganglia_gmond_cluster: '{{ ganglia_gmond_hbmaster_cluster }}'
|
||||
ganglia_gmond_cluster_port: '{{ ganglia_gmond_hbmaster_port }}'
|
||||
ganglia_gmond_mcast_address: '{{ ganglia_gmond_hbmaster_mcast_addr }}'
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
---
|
||||
iptables:
|
||||
tcp_rules: True
|
||||
tcp:
|
||||
- { port: '{{ hbase_master_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ hbase_master_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ hbase_thrift_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_journal_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_journal_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_leader_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_quorum_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_client_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
|
@ -1,6 +0,0 @@
|
|||
---
|
||||
# Ganglia gmond port
|
||||
ganglia_gmond_cluster: '{{ ganglia_gmond_namenode_cluster }}'
|
||||
ganglia_gmond_cluster_port: '{{ ganglia_gmond_namenode_port }}'
|
||||
ganglia_gmond_mcast_address: '{{ ganglia_gmond_namenode_mcast_addr }}'
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
iptables:
|
||||
tcp_rules: True
|
||||
tcp:
|
||||
- { port: '{{ hdfs_nn_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ hdfs_nn_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ hdfs_nn_client_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_zkfc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_journal_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_journal_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_leader_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_quorum_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_client_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
|
@ -1,6 +0,0 @@
|
|||
---
|
||||
# Ganglia gmond port
|
||||
ganglia_gmond_cluster: '{{ ganglia_gmond_jobtracker_cluster }}'
|
||||
ganglia_gmond_cluster_port: '{{ ganglia_gmond_jobtracker_port }}'
|
||||
ganglia_gmond_mcast_address: '{{ ganglia_gmond_jobtracker_mcast_addr }}'
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
---
|
||||
iptables:
|
||||
tcp_rules: True
|
||||
tcp:
|
||||
- { port: '80:95' }
|
||||
- { port: '8100:8150' }
|
||||
- { port: '8200:8250' }
|
||||
- { port: '8300:8350' }
|
||||
- { port: '8400:8450' }
|
||||
- { port: '{{ jobtracker_cluster_id1_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ jobtracker_cluster_id2_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ jobtracker_cluster_id1_ha_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ jobtracker_cluster_id2_ha_rpc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ jobtracker_cluster_id1_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ jobtracker_cluster_id2_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.icm_1 }}' ] }
|
||||
- { port: '{{ jobtracker_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ jobtracker_zkfc_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_journal_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ hdfs_journal_http_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_leader_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_quorum_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '{{ zookeeper_client_port }}', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.ilsp_gr }}', '{{ other_networks.iis_pl_1 }}', '{{ other_networks.icm_1 }}' ] }
|
|
@ -1,52 +0,0 @@
|
|||
---
|
||||
user_ssh_key: [ '{{ claudio_atzori }}' ]
|
||||
|
||||
|
||||
# logstash
|
||||
logstash_collector_host: logstash.t.hadoop.research-infrastructures.eu
|
||||
logstash_collector_listen_port: 5544
|
||||
logstash_version: 1.3.3
|
||||
logstash_file: 'logstash-{{ logstash_version }}-flatjar.jar'
|
||||
logstash_url: 'download.elasticsearch.org/logstash/logstash/{{ logstash_file }}'
|
||||
logstash_install_dir: /opt/logstash
|
||||
logstash_conf_dir: '{{ logstash_install_dir }}/etc'
|
||||
logstash_lib_dir: '{{ logstash_install_dir }}/share'
|
||||
logstash_log_dir: /var/log/logstash
|
||||
logstash_user: logstash
|
||||
logstash_indexer_jvm_opts: "-Xms2048m -Xmx2048m"
|
||||
|
||||
kibana_nginx_conf: kibana
|
||||
kinaba_nginx_root: /var/www/kibana/src
|
||||
kibana_virtual_host: logs.t.hadoop.research-infrastructures.eu
|
||||
|
||||
elasticsearch_user: elasticsearch
|
||||
elasticsearch_group: elasticsearch
|
||||
elasticsearch_version: 1.0.0
|
||||
elasticsearch_http_port: 9200
|
||||
elasticsearch_transport_tcp_port: 9300
|
||||
elasticsearch_download_path: download.elasticsearch.org/elasticsearch/elasticsearch
|
||||
elasticsearch_cluster: hadoop-logstash
|
||||
elasticsearch_node_name: logstash
|
||||
elasticsearch_node_master: "true"
|
||||
elasticsearch_node_data: "true"
|
||||
elasticsearch_max_local_storage_nodes: 1
|
||||
elasticsearch_log_dir: /var/log/elasticsearch
|
||||
elasticsearch_heap_size: 5
|
||||
elasticsearch_host: localhost
|
||||
elasticsearch_curator_close_after: 10
|
||||
elasticsearch_curator_retain_days: 20
|
||||
elasticsearch_curator_optimize_days: 10
|
||||
elasticsearch_curator_bloom_days: 7
|
||||
elasticsearch_curator_timeout: 1200
|
||||
elasticsearch_curator_manage_marvel: True
|
||||
elasticsearch_disable_dynamic_scripts: True
|
||||
|
||||
# We use the nginx defaults here
|
||||
nginx_use_ldap_pam_auth: True
|
||||
|
||||
iptables:
|
||||
tcp_rules: True
|
||||
tcp:
|
||||
- { port: '{{ logstash.collector_listen_port }}', allowed_hosts: [ '{{ network.nmis }}' ] }
|
||||
- { port: '{{ elasticsearch.http_port }}', allowed_hosts: [ '{{ ansible_fqdn }}', '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
||||
- { port: '80', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}' ] }
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
user_ssh_key: [ '{{ claudio_atzori }}', '{{ michele_artini }}' ]
|
||||
|
||||
iptables:
|
||||
tcp_rules: True
|
||||
tcp:
|
||||
- { port: '11000', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.icm_pl }}', '{{ other_networks.icm_pl_1 }}' ] }
|
||||
- { port: '10000', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.icm_pl }}', '{{ other_networks.icm_pl_1 }}' ] }
|
||||
- { port: '9083', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.icm_pl }}', '{{ other_networks.icm_pl_1 }}' ] }
|
||||
- { port: '8888', allowed_hosts: [ '{{ network.isti }}', '{{ network.nmis }}', '{{ network.eduroam }}', '{{ other_networks.icm_pl }}', '{{ other_networks.icm_pl_1 }}', '0.0.0.0/0' ] }
|
||||
|
||||
oozie:
|
||||
host: 'oozie.{{ dns_domain }}'
|
||||
conf_dir: /etc/oozie/conf
|
||||
user: oozie
|
||||
catalina_work_dir: /usr/lib/oozie/oozie-server-0.20/work
|
||||
http_port: 11000
|
||||
|
||||
#
|
||||
# HIVE
|
||||
#
|
||||
hive:
|
||||
host: 'hive.{{ dns_domain }}'
|
||||
conf_dir: /etc/hive/conf
|
||||
user: hive
|
||||
metastore_port: 9083
|
||||
server2_http_port: 10000
|
||||
setugi: True
|
||||
|
||||
#
|
||||
# HUE
|
||||
#
|
||||
hue:
|
||||
user: hue
|
||||
group: hue
|
||||
host: 'hue.{{ dns_domain }}'
|
||||
http_port: 8888
|
||||
conf_dir: /etc/hue
|
||||
hive_interface: hiveserver2
|
||||
exec_path: /usr/share/hue/build/env/bin/hue
|
||||
encoding: 'utf-8'
|
||||
setuid_path: /usr/share/hue/apps/shell/src/shell/build/setuid
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
$ANSIBLE_VAULT;1.1;AES256
|
||||
35656164616131366466393935373064383333633237616435353030613234323463393363643961
|
||||
6366343466396563666662396332666661636462313861630a376235623035633530656238623464
|
||||
37636231343837363431396564363632343466306166343365356137646266656637313534353834
|
||||
3561323334346135300a643731653463353564356332376162613864336539376530333534363032
|
||||
36643532626433393939353030653762643636353331326565666164343761393533623461383165
|
||||
33313736346537373364646332653538343034376639626335393065346637623664303264343237
|
||||
326630336139303531346238383733633335
|
|
@ -1,17 +0,0 @@
|
|||
---
|
||||
- hosts: hadoop_worker_nodes:hadoop_masters
|
||||
remote_user: root
|
||||
max_fail_percentage: 10
|
||||
serial: "25%"
|
||||
|
||||
# vars_files:
|
||||
# - ../library/vars/isti-global.yml
|
||||
roles:
|
||||
- common
|
||||
- cdh_common
|
||||
- chkconfig
|
||||
- hadoop_common
|
||||
- hadoop_config
|
||||
- hadoop_zookeeper
|
||||
- hadoop_zookeeper_config
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
[zookeeper_cluster]
|
||||
quorum0.t.hadoop.research-infrastructures.eu zoo_id=0
|
||||
quorum1.t.hadoop.research-infrastructures.eu zoo_id=1
|
||||
quorum2.t.hadoop.research-infrastructures.eu zoo_id=2
|
||||
quorum3.t.hadoop.research-infrastructures.eu zoo_id=3
|
||||
quorum4.t.hadoop.research-infrastructures.eu zoo_id=4
|
||||
|
||||
[monitoring]
|
||||
monitoring.research-infrastructures.eu
|
||||
|
14
nagios.yml
14
nagios.yml
|
@ -1,14 +0,0 @@
|
|||
---
|
||||
- hosts: monitoring
|
||||
user: root
|
||||
vars_files:
|
||||
- ../library/vars/isti-global.yml
|
||||
roles:
|
||||
- nagios-server
|
||||
|
||||
- hosts: hadoop_cluster:other_services:db
|
||||
user: root
|
||||
vars_files:
|
||||
- ../library/vars/isti-global.yml
|
||||
roles:
|
||||
- nagios-monitoring
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
dependencies:
|
||||
- { role: ../library/roles/openjdk }
|
||||
- role: '../../library/roles/ssh-keys'
|
|
@ -1,14 +0,0 @@
|
|||
---
|
||||
- name: Install the common CDH hadoop packages
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop
|
||||
- hadoop-0.20-mapreduce
|
||||
- hadoop-client
|
||||
- hadoop-hdfs
|
||||
- hadoop-mapreduce
|
||||
tags:
|
||||
- hadoop
|
||||
- mapred
|
||||
- hdfs
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
---
|
||||
- name: Install the D-NET repository key
|
||||
action: apt_key url=http://ppa.research-infrastructures.eu/dnet/keys/dnet-archive.asc
|
||||
tags:
|
||||
- hadoop
|
||||
- cdh
|
||||
|
||||
- name: Install the CDH repository key
|
||||
action: apt_key url=http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/archive.key
|
||||
tags:
|
||||
- hadoop
|
||||
- cdh
|
||||
|
||||
- apt_repository: repo='{{ item }}' update_cache=yes
|
||||
with_items:
|
||||
- deb http://ppa.research-infrastructures.eu/dnet lucid main
|
||||
- deb http://ppa.research-infrastructures.eu/dnet unstable main
|
||||
- deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/ precise-cdh4 contrib
|
||||
- deb [arch=amd64] http://archive.cloudera.com/gplextras/ubuntu/precise/amd64/gplextras precise-gplextras4 contrib
|
||||
register: update_apt_cache
|
||||
tags:
|
||||
- hadoop
|
||||
- cdh
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
- import_tasks: cdh-setup.yml
|
||||
- import_tasks: cdh-pkgs.yml
|
||||
# See meta/main.yml for the involved library playbooks
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
dependencies:
|
||||
- role: '../../library/roles/ubuntu-deb-general'
|
||||
- { role: '../../library/roles/iptables', when: iptables is defined }
|
|
@ -1,2 +0,0 @@
|
|||
---
|
||||
# See meta/main.yml for the involved library playbooks
|
|
@ -1,3 +0,0 @@
|
|||
---
|
||||
dependencies:
|
||||
- role: '../../library/roles/ganglia'
|
|
@ -1,31 +0,0 @@
|
|||
---
|
||||
# See meta/main.yml for the basic installation and configuration steps
|
||||
# The hadoop conf directory always exists
|
||||
- name: Distribute the ganglia hadoop metrics properties
|
||||
template: src={{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=444
|
||||
with_items:
|
||||
- hadoop-metrics.properties
|
||||
- hadoop-metrics2.properties
|
||||
tags: [ 'monitoring', 'ganglia', 'ganglia_conf' ]
|
||||
|
||||
- name: Check if the hbase conf directory exists
|
||||
stat: path={{ hbase_conf_dir }}
|
||||
register: check_hbase_confdir
|
||||
tags: [ 'monitoring', 'ganglia', 'ganglia_conf' ]
|
||||
|
||||
- name: Distribute the ganglia hbase metrics properties
|
||||
template: src={{ item }}.properties.j2 dest={{ hbase_conf_dir }}/{{ item }}-hbase.properties owner=root group=root mode=444
|
||||
with_items:
|
||||
- hadoop-metrics
|
||||
- hadoop-metrics2
|
||||
when: check_hbase_confdir.stat.exists
|
||||
tags: [ 'monitoring', 'ganglia', 'ganglia_conf' ]
|
||||
|
||||
- name: Distribute the ganglia hbase metrics properties, maintain the old file name
|
||||
file: src={{ hbase_conf_dir }}/{{ item }}-hbase.properties dest={{ hbase_conf_dir }}/{{ item }}.properties state=link force=yes
|
||||
with_items:
|
||||
- hadoop-metrics
|
||||
- hadoop-metrics2
|
||||
when: check_hbase_confdir.stat.exists
|
||||
tags: [ 'monitoring', 'ganglia', 'ganglia_conf' ]
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
# Configuration of the "dfs" context for null
|
||||
dfs.class=org.apache.hadoop.metrics.spi.NullContext
|
||||
|
||||
# Configuration of the "dfs" context for file
|
||||
#dfs.class=org.apache.hadoop.metrics.file.FileContext
|
||||
#dfs.period=10
|
||||
#dfs.fileName=/tmp/dfsmetrics.log
|
||||
|
||||
# Configuration of the "dfs" context for ganglia
|
||||
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
|
||||
# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||
# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# dfs.period=10
|
||||
# dfs.servers=localhost:8649
|
||||
|
||||
|
||||
# Configuration of the "mapred" context for null
|
||||
mapred.class=org.apache.hadoop.metrics.spi.NullContext
|
||||
|
||||
# Configuration of the "mapred" context for file
|
||||
#mapred.class=org.apache.hadoop.metrics.file.FileContext
|
||||
#mapred.period=10
|
||||
#mapred.fileName=/tmp/mrmetrics.log
|
||||
|
||||
# Configuration of the "mapred" context for ganglia
|
||||
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
|
||||
# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||
# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# mapred.period=10
|
||||
# mapred.servers=localhost:8649
|
||||
|
||||
|
||||
# Configuration of the "jvm" context for null
|
||||
#jvm.class=org.apache.hadoop.metrics.spi.NullContext
|
||||
|
||||
# Configuration of the "jvm" context for file
|
||||
#jvm.class=org.apache.hadoop.metrics.file.FileContext
|
||||
#jvm.period=10
|
||||
#jvm.fileName=/tmp/jvmmetrics.log
|
||||
|
||||
# Configuration of the "jvm" context for ganglia
|
||||
# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||
# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# jvm.period=10
|
||||
# jvm.servers=localhost:8649
|
||||
|
||||
# Configuration of the "rpc" context for null
|
||||
rpc.class=org.apache.hadoop.metrics.spi.NullContext
|
||||
|
||||
# Configuration of the "rpc" context for file
|
||||
#rpc.class=org.apache.hadoop.metrics.file.FileContext
|
||||
#rpc.period=10
|
||||
#rpc.fileName=/tmp/rpcmetrics.log
|
||||
|
||||
# Configuration of the "rpc" context for ganglia
|
||||
# rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||
# rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# rpc.period=10
|
||||
# rpc.servers=localhost:8649
|
||||
|
||||
|
||||
# Configuration of the "ugi" context for null
|
||||
ugi.class=org.apache.hadoop.metrics.spi.NullContext
|
||||
|
||||
# Configuration of the "ugi" context for file
|
||||
#ugi.class=org.apache.hadoop.metrics.file.FileContext
|
||||
#ugi.period=10
|
||||
#ugi.fileName=/tmp/ugimetrics.log
|
||||
|
||||
# Configuration of the "ugi" context for ganglia
|
||||
# ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext
|
||||
# ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# ugi.period=10
|
||||
# ugi.servers=localhost:8649
|
||||
|
||||
# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# dfs.period=10
|
||||
# dfs.servers={{ hdfs_namenode_1_hostname }}:{{ ganglia_gmond_namenode_port }}
|
||||
# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# mapred.period=10
|
||||
# mapred.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
# hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# hbase.period=10
|
||||
# hbase.servers={{ hbase_master_1_hostname }}:{{ ganglia_gmond_cluster_port }}
|
||||
# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# jvm.period=10
|
||||
# jvm.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
# rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# rpc.period=10
|
||||
# rpc.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
# ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# ugi.period=10
|
||||
# ugi.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
# fairscheduler.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
|
||||
# fairscheduler.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
# Ganglia 3.1+ support
|
||||
*.period=60
|
||||
|
||||
*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
|
||||
*.sink.ganglia.period=10
|
||||
# default for supportsparse is false
|
||||
*.sink.ganglia.supportsparse=true
|
||||
*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both
|
||||
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40
|
||||
|
||||
|
||||
namenode.sink.ganglia.servers={{ hdfs_namenode_1_hostname }}:{{ ganglia_gmond_namenode_port }},{{ hdfs_namenode_2_hostname }}:{{ ganglia_gmond_namenode_port }}
|
||||
datanode.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
jobtracker.sink.ganglia.servers={{ jobtracker_node_1_hostname }}:{{ ganglia_gmond_jobtracker_port }},{{ jobtracker_node_2_hostname }}:{{ ganglia_gmond_jobtracker_port }}
|
||||
#tasktracker.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_tasktracker_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_tasktracker_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_tasktracker_port }}
|
||||
#maptask.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_maptask_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_maptask_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_maptask_port }}
|
||||
#reducetask.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_reducetask_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_reducetask_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_reducetask_port }}
|
||||
tasktracker.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
maptask.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
reducetask.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
hbase.extendedperiod = 3600
|
||||
hbase.sink.ganglia.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
hbase.servers=node2.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node5.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }},node11.{{ dns_domain }}:{{ ganglia_gmond_hdfs_datanodes_port }}
|
||||
#hbase.sink.ganglia.servers={{ hbase_master_1_hostname }}:{{ ganglia_gmond_hbmaster_port }},{{ hbase_master_2_hostname }}:{{ ganglia_gmond_hbmaster_port }}
|
||||
#hbase.servers={{ hbase_master_1_hostname }}:{{ ganglia_gmond_hbmaster_port }},{{ hbase_master_2_hostname }}:{{ ganglia_gmond_hbmaster_port }}
|
||||
|
||||
#resourcemanager.sink.ganglia.servers=
|
||||
#nodemanager.sink.ganglia.servers=
|
||||
#historyserver.sink.ganglia.servers=
|
||||
#journalnode.sink.ganglia.servers=
|
||||
#nimbus.sink.ganglia.servers=
|
||||
#supervisor.sink.ganglia.servers=
|
||||
#resourcemanager.sink.ganglia.tagsForPrefix.yarn=Queue
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
- name: Directory for hdfs root under /data
|
||||
file: dest={{ hdfs_data_dir }} state=directory
|
||||
tags:
|
||||
- hadoop
|
||||
- mapred
|
||||
- hdfs
|
||||
|
||||
# TODO: split and move to more specific roles.
|
||||
- name: Directories for the hdfs services
|
||||
file: dest={{ hdfs_data_dir}}/{{ item }} state=directory owner=hdfs group=hdfs mode=700
|
||||
with_items:
|
||||
- '{{ hdfs_dn_data_dir }}'
|
||||
- '{{ hdfs_journal_data_dir }}'
|
||||
tags:
|
||||
- hadoop
|
||||
- mapred
|
||||
- hdfs
|
||||
|
||||
- name: Directories for mapred under /data/mapred
|
||||
file: dest=/data/mapred state=directory
|
||||
tags:
|
||||
- hadoop
|
||||
- mapred
|
||||
- hdfs
|
||||
|
||||
- name: Directories for mapred under /data/mapred
|
||||
file: dest=/data/mapred/{{ item }} state=directory owner=mapred group=hadoop mode=700
|
||||
with_items:
|
||||
- jt
|
||||
- local
|
||||
tags:
|
||||
- hadoop
|
||||
- mapred
|
||||
- hdfs
|
||||
|
||||
- name: JMX secrets directory
|
||||
file: dest=/etc/hadoop-jmx/conf state=directory owner=hdfs group=root mode=0750
|
||||
when: hadoop_jmx_enabled
|
||||
tags:
|
||||
- hadoop
|
||||
- jmx
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
node13.t.hadoop.research-infrastructures.eu
|
||||
node12.t.hadoop.research-infrastructures.eu
|
||||
node11.t.hadoop.research-infrastructures.eu
|
||||
node10.t.hadoop.research-infrastructures.eu
|
||||
node9.t.hadoop.research-infrastructures.eu
|
||||
node8.t.hadoop.research-infrastructures.eu
|
||||
node7.t.hadoop.research-infrastructures.eu
|
||||
node6.t.hadoop.research-infrastructures.eu
|
|
@ -1,39 +0,0 @@
|
|||
---
|
||||
- name: Restart HDFS namenode
|
||||
service: name=hadoop-hdfs-namenode state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
||||
- name: Restart HDFS journalnode
|
||||
service: name=hadoop-hdfs-journalnode state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
||||
- name: Restart HDFS datanode
|
||||
service: name=hadoop-hdfs-datanode state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
||||
- name: Restart HDFS httpfs
|
||||
service: name=hadoop-httpfs state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
||||
- name: Disable HDFS httpfs
|
||||
service: name=hadoop-httpfs state=stopped enabled=no
|
||||
ignore_errors: true
|
||||
|
||||
- name: Enable HDFS httpfs
|
||||
service: name=hadoop-httpfs state=started enabled=yes
|
||||
ignore_errors: true
|
||||
|
||||
- name: Refresh HDFS datanodes
|
||||
become: True
|
||||
become_user: hdfs
|
||||
command: hdfs dfsadmin -refreshNodes
|
||||
ignore_errors: true
|
||||
|
||||
- name: Restart mapreduce jobtracker
|
||||
service: name=hadoop-0.20-mapreduce-jobtrackerha state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
||||
- name: Restart mapreduce tasktracker
|
||||
service: name=hadoop-0.20-mapreduce-tasktracker state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
---
|
||||
# Base environment for all the hadoop services
|
||||
- name: Base environment for all the hadoop services
|
||||
template: src=templates/bigtop-utils.default.j2 dest=/etc/default/bigtop-utils owner=root group=root mode=444
|
||||
tags: [ 'hadoop', 'configuration' ]
|
||||
|
||||
- name: copy /etc/hadoop/conf.empty to {{ hadoop_conf_dir }}
|
||||
command: creates={{ hadoop_conf_dir }} cp -R -p /etc/hadoop/conf.empty {{ hadoop_conf_dir }}
|
||||
tags: [ 'hadoop', 'configuration' ]
|
||||
|
||||
- name: run 'update-alternatives' to set our hadoop configuration directory
|
||||
alternatives: name=hadoop-conf link=/etc/hadoop/conf path={{ hadoop_conf_dir }}
|
||||
tags: [ 'hadoop', 'configuration' ]
|
||||
|
||||
- name: Install the common configuration files
|
||||
template: src={{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=444
|
||||
with_items:
|
||||
- slaves
|
||||
- masters
|
||||
- log4j.properties
|
||||
tags: [ 'hadoop', 'configuration', 'log4j', 'hadoop_workers' ]
|
||||
|
||||
- name: Install the shared configuration files
|
||||
template: src=templates/{{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=444
|
||||
with_items:
|
||||
- hadoop-env.sh
|
||||
tags: [ 'hadoop', 'configuration' ]
|
||||
|
||||
- name: Install the mapreduce scheduler configuration file
|
||||
template: src=templates/{{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=444
|
||||
with_items:
|
||||
- fair-scheduler.xml
|
||||
when: mapred_use_fair_scheduler
|
||||
tags: [ 'hadoop', 'configuration', 'scheduler' ]
|
|
@ -1,238 +0,0 @@
|
|||
# Copyright 2011 The Apache Software Foundation
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Define some default values that can be overridden by system properties
|
||||
#hadoop.root.logger=INFO,console
|
||||
{% if hadoop_send_to_logstash %}
|
||||
hadoop.root.logger={{ hadoop_log_level }},{{ hadoop_logstash_appender }}
|
||||
{% else %}
|
||||
hadoop.root.logger={{ hadoop_log_level }},{{ hadoop_log_appender }}
|
||||
{% endif %}
|
||||
hadoop.log.dir=.
|
||||
hadoop.log.file=hadoop.log
|
||||
|
||||
# Define the root logger to the system property "hadoop.root.logger".
|
||||
log4j.rootLogger=${hadoop.root.logger}, EventCounter
|
||||
|
||||
# Logging Threshold
|
||||
log4j.threshold=ALL
|
||||
|
||||
# Null Appender
|
||||
log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender
|
||||
|
||||
#
|
||||
# Rolling File Appender - cap space usage at 5gb.
|
||||
#
|
||||
hadoop.log.maxfilesize={{ hadoop_log_appender_max_filesize }}
|
||||
hadoop.log.maxbackupindex={{ hadoop_log_appender_max_backupindex }}
|
||||
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
|
||||
|
||||
log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}
|
||||
log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}
|
||||
|
||||
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
|
||||
|
||||
# Pattern format: Date LogLevel LoggerName LogMessage
|
||||
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||
# Debugging Pattern format
|
||||
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
|
||||
|
||||
|
||||
#
|
||||
# Daily Rolling File Appender
|
||||
#
|
||||
|
||||
log4j.appender.DRFA=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
|
||||
|
||||
# Rollver at midnight
|
||||
#log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
|
||||
|
||||
log4j.appender.DRFA.MaxFileSize=${hadoop.log.maxfilesize}
|
||||
log4j.appender.DRFA.MaxBackupIndex=${hadoop.log.maxbackupindex}
|
||||
|
||||
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
|
||||
|
||||
# Pattern format: Date LogLevel LoggerName LogMessage
|
||||
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||
# Debugging Pattern format
|
||||
#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
|
||||
|
||||
|
||||
#
|
||||
# console
|
||||
# Add "console" to rootlogger above if you want to use this
|
||||
#
|
||||
|
||||
log4j.appender.console=org.apache.log4j.ConsoleAppender
|
||||
log4j.appender.console.target=System.err
|
||||
log4j.appender.console.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
|
||||
|
||||
#
|
||||
# TaskLog Appender
|
||||
#
|
||||
|
||||
#Default values
|
||||
hadoop.tasklog.taskid=null
|
||||
hadoop.tasklog.iscleanup=false
|
||||
hadoop.tasklog.noKeepSplits=4
|
||||
hadoop.tasklog.totalLogFileSize=100
|
||||
hadoop.tasklog.purgeLogSplits=true
|
||||
hadoop.tasklog.logsRetainHours=12
|
||||
|
||||
log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender
|
||||
log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
|
||||
log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}
|
||||
log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}
|
||||
|
||||
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||
|
||||
#
|
||||
# HDFS block state change log from block manager
|
||||
#
|
||||
# Uncomment the following to suppress normal block state change
|
||||
# messages from BlockManager in NameNode.
|
||||
#log4j.logger.BlockStateChange=WARN
|
||||
|
||||
#
|
||||
#Security appender
|
||||
#
|
||||
#hadoop.security.logger={{ hadoop_log_level }},NullAppender
|
||||
hadoop.security.logger={{ hadoop_log_level }},RFAS
|
||||
hadoop.security.log.maxfilesize=64MB
|
||||
hadoop.security.log.maxbackupindex=4
|
||||
log4j.category.SecurityLogger=${hadoop.security.logger}
|
||||
hadoop.security.log.file=SecurityAuth-${user.name}.audit
|
||||
log4j.appender.RFAS=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}
|
||||
log4j.appender.RFAS.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||
log4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize}
|
||||
log4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}
|
||||
|
||||
#
|
||||
# (fake) Daily Rolling Security appender
|
||||
#
|
||||
log4j.appender.DRFAS=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}
|
||||
log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
|
||||
#log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd
|
||||
log4j.appender.DRFAS.MaxFileSize=${hadoop.security.log.maxfilesize}
|
||||
log4j.appender.DRFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}
|
||||
|
||||
#
|
||||
# hdfs audit logging
|
||||
#
|
||||
hdfs.audit.logger={{ hadoop_log_level }},RFAAUDIT
|
||||
hdfs.audit.log.maxfilesize=64MB
|
||||
hdfs.audit.log.maxbackupindex=4
|
||||
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
|
||||
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
|
||||
log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.RFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
|
||||
log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
|
||||
log4j.appender.RFAAUDIT.MaxFileSize=${hdfs.audit.log.maxfilesize}
|
||||
log4j.appender.RFAAUDIT.MaxBackupIndex=${hdfs.audit.log.maxbackupindex}
|
||||
|
||||
#
|
||||
# mapred audit logging
|
||||
#
|
||||
mapred.audit.logger={{ hadoop_log_level }},MRAUDIT
|
||||
mapred.audit.log.maxfilesize=64MB
|
||||
mapred.audit.log.maxbackupindex=4
|
||||
log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}
|
||||
log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false
|
||||
log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log
|
||||
log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
|
||||
log4j.appender.MRAUDIT.MaxFileSize=${mapred.audit.log.maxfilesize}
|
||||
log4j.appender.MRAUDIT.MaxBackupIndex=${mapred.audit.log.maxbackupindex}
|
||||
|
||||
# Custom Logging levels
|
||||
|
||||
log4j.logger.org.apache.hadoop.mapred.JobTracker={{ hadoop_log_level }}
|
||||
log4j.logger.org.apache.hadoop.mapred.TaskTracker={{ hadoop_log_level }}
|
||||
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit={{ hadoop_log_level }}
|
||||
|
||||
# Jets3t library
|
||||
log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR
|
||||
|
||||
#
|
||||
# Event Counter Appender
|
||||
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
|
||||
#
|
||||
log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
|
||||
|
||||
#
|
||||
# Job Summary Appender
|
||||
#
|
||||
# Use following logger to send summary to separate file defined by
|
||||
# hadoop.mapreduce.jobsummary.log.file :
|
||||
# hadoop.mapreduce.jobsummary.logger={{ hadoop_log_level }},JSA
|
||||
#
|
||||
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
|
||||
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log
|
||||
hadoop.mapreduce.jobsummary.log.maxfilesize=256MB
|
||||
hadoop.mapreduce.jobsummary.log.maxbackupindex=20
|
||||
log4j.appender.JSA=org.apache.log4j.RollingFileAppender
|
||||
log4j.appender.JSA.File=${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file}
|
||||
log4j.appender.JSA.MaxFileSize=${hadoop.mapreduce.jobsummary.log.maxfilesize}
|
||||
log4j.appender.JSA.MaxBackupIndex=${hadoop.mapreduce.jobsummary.log.maxbackupindex}
|
||||
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
|
||||
log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=${hadoop.mapreduce.jobsummary.logger}
|
||||
log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=false
|
||||
|
||||
#
|
||||
# Yarn ResourceManager Application Summary Log
|
||||
#
|
||||
# Set the ResourceManager summary log filename
|
||||
#yarn.server.resourcemanager.appsummary.log.file=rm-appsummary.log
|
||||
# Set the ResourceManager summary log level and appender
|
||||
#yarn.server.resourcemanager.appsummary.logger={{ hadoop_log_level }},RMSUMMARY
|
||||
|
||||
# Appender for ResourceManager Application Summary Log
|
||||
# Requires the following properties to be set
|
||||
# - hadoop.log.dir (Hadoop Log directory)
|
||||
# - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename)
|
||||
# - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender)
|
||||
|
||||
#log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger}
|
||||
#log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false
|
||||
#log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender
|
||||
#log4j.appender.RMSUMMARY.File=${hadoop.log.dir}/${yarn.server.resourcemanager.appsummary.log.file}
|
||||
#log4j.appender.RMSUMMARY.MaxFileSize=256MB
|
||||
#log4j.appender.RMSUMMARY.MaxBackupIndex=20
|
||||
#log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout
|
||||
#log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
|
||||
|
||||
{% if hadoop_send_to_logstash %}
|
||||
log4j.appender.LOGSTASH=org.apache.log4j.net.SocketAppender
|
||||
log4j.appender.LOGSTASH.remoteHost={{ hadoop_logstash_collector_host }}
|
||||
log4j.appender.LOGSTASH.port={{ hadoop_logstash_collector_socketappender_port }}
|
||||
log4j.appender.LOGSTASH.ReconnectionDelay={{ hadoop_logstash_collector_socketappender_reconndelay }}
|
||||
log4j.appender.LOGSTASH.LocationInfo=true
|
||||
log4j.appender.LOGSTASH.layout = org.apache.log4j.PatternLayout
|
||||
log4j.appender.LOGSTASH.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
|
||||
{% endif %}
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['hdfs_masters'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,9 +0,0 @@
|
|||
---
|
||||
- name: Install the HDFS datanode packages
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-hdfs-datanode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
---
|
||||
# TODO: find a better condition for the restart
|
||||
#
|
||||
- name: Restart HDFS datanode
|
||||
service: name=hadoop-hdfs-datanode state=restarted sleep=20
|
||||
when: ansible_fqdn | match ("node.*\.t\.hadoop\.research\-infrastructures\.eu")
|
||||
ignore_errors: true
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
---
|
||||
# HDFS datanode
|
||||
- name: Ensure that the hdfs data disks are mounted
|
||||
mount: name={{ item.mountpoint }} src=/dev/{{ item.device }} fstype={{ item.fstype }} state=mounted
|
||||
with_items: hadoop_hdfs_data_disk
|
||||
when: ansible_fqdn | match ("node.*\.t\.hadoop\.research\-infrastructures\.eu")
|
||||
tags:
|
||||
- hdfs
|
||||
- datanodes
|
||||
- disk
|
||||
|
||||
- name: Set the vm.swappiness parameter
|
||||
sysctl: name=vm.swappiness value={{ worker_node_swappiness }} sysctl_file=/etc/sysctl.d/90-swappiness.conf state=present reload=yes
|
||||
when: ansible_fqdn | match ("node.*\.t\.hadoop\.research\-infrastructures\.eu")
|
||||
tags:
|
||||
- hdfs
|
||||
- datanodes
|
||||
|
||||
- name: Install the HDFS datanode config files.
|
||||
template: src=templates/{{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=0444
|
||||
with_items:
|
||||
- core-site.xml
|
||||
notify:
|
||||
- Restart HDFS datanode
|
||||
tags:
|
||||
- datanode
|
||||
- hdfs
|
||||
|
||||
- name: Ensure that the hdfs shortcircuit cache directory exists
|
||||
file: dest={{ hdfs_read_shortcircuit_cache_dir }} state=directory owner=hdfs group=hdfs mode=0755
|
||||
when:
|
||||
- ansible_fqdn | match ("node.*\.t\.hadoop\.research\-infrastructures\.eu")
|
||||
- hdfs_read_shortcircuit
|
||||
tags:
|
||||
- datanode
|
||||
- hdfs
|
||||
|
||||
- name: Install the HDFS datanode config files.
|
||||
template: src=templates/datanode-hdfs-site.xml.j2 dest={{ hadoop_conf_dir }}/hdfs-site.xml owner=root group=root mode=0444
|
||||
notify:
|
||||
- Restart HDFS datanode
|
||||
tags:
|
||||
- datanode
|
||||
- hdfs
|
||||
- hdfs_site
|
||||
|
||||
- name: Check if the hadoop-jmx conf directory exists
|
||||
stat: path=/etc/hadoop-jmx/conf
|
||||
register: check_jmx_confdir
|
||||
|
||||
- name: Distribute the jmx authorization files for hadoop hdfs
|
||||
template: src=templates/jmxremote.passwd.j2 dest=/etc/hadoop-jmx/conf/jmxremote.passwd owner=hdfs mode=0600
|
||||
when: check_jmx_confdir.stat.exists and hadoop_jmx_enabled
|
||||
notify:
|
||||
- Restart HDFS datanode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
- jmx
|
||||
|
||||
- name: Distribute the jmx role files for hadoop hdfs
|
||||
copy: src=files/jmxremote.access dest=/etc/hadoop-jmx/conf/jmxremote.access owner=root mode=0644
|
||||
when: check_jmx_confdir.stat.exists and hadoop_jmx_enabled
|
||||
notify:
|
||||
- Restart HDFS datanode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
- jmx
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
- name: Start HDFS journal
|
||||
service: name=hadoop-hdfs-journalnode state=started enabled=yes
|
||||
ignore_errors: True
|
|
@ -1,19 +0,0 @@
|
|||
---
|
||||
- name: Directories for the hdfs journal under /data/dfs
|
||||
file: dest={{ hdfs_data_dir}}/{{ item }} state=directory owner=hdfs group=hdfs mode=700
|
||||
with_items:
|
||||
- '{{ hdfs_journal_data_dir }}'
|
||||
tags:
|
||||
- hdfs
|
||||
- journal
|
||||
|
||||
- name: Install the journalnode server package
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-hdfs-journalnode
|
||||
notify:
|
||||
- Start HDFS journal
|
||||
tags:
|
||||
- hdfs
|
||||
- journal
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
---
|
||||
- name: Install the common CDH hadoop packages
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-hdfs-namenode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
node13.t.hadoop.research-infrastructures.eu
|
||||
node12.t.hadoop.research-infrastructures.eu
|
||||
node11.t.hadoop.research-infrastructures.eu
|
||||
node10.t.hadoop.research-infrastructures.eu
|
||||
node9.t.hadoop.research-infrastructures.eu
|
||||
node8.t.hadoop.research-infrastructures.eu
|
||||
node7.t.hadoop.research-infrastructures.eu
|
||||
node6.t.hadoop.research-infrastructures.eu
|
|
@ -1,5 +0,0 @@
|
|||
---
|
||||
- name: Restart HDFS namenode
|
||||
service: name=hadoop-hdfs-namenode state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
---
|
||||
# HDFS namenode
|
||||
- name: Install the hdfs config files.
|
||||
template: src=templates/{{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=0444
|
||||
with_items:
|
||||
- core-site.xml
|
||||
notify:
|
||||
- Restart HDFS namenode
|
||||
tags:
|
||||
- namenode
|
||||
- hdfs
|
||||
|
||||
- name: Install the hdfs config files.
|
||||
template: src=templates/namenode-{{ item }}.j2 dest={{ hadoop_conf_dir }}/{{ item }} owner=root group=root mode=0444
|
||||
with_items:
|
||||
- hdfs-site.xml
|
||||
notify:
|
||||
- Restart HDFS namenode
|
||||
tags:
|
||||
- namenode
|
||||
- hdfs
|
||||
- hdfs_site
|
||||
|
||||
- name: Install the dfs hosts allow file
|
||||
template: src=dfs_hosts_allow.txt.j2 dest={{ hadoop_conf_dir }}/dfs_hosts_allow.txt owner=root group=root mode=0444
|
||||
notify:
|
||||
- Restart HDFS namenode
|
||||
tags:
|
||||
- namenode
|
||||
- hdfs
|
||||
- hadoop_workers
|
||||
|
||||
- name: Install the dfs hosts exclude file
|
||||
copy: src=dfs_hosts_exclude.txt dest={{ hadoop_conf_dir }}/dfs_hosts_exclude.txt owner=root group=root mode=0444
|
||||
tags:
|
||||
- namenode
|
||||
- hdfs
|
||||
- hadoop_workers
|
||||
- hadoop_exclude
|
||||
|
||||
- name: Check if the hadoop-jmx conf directory exists
|
||||
stat: path=/etc/hadoop-jmx/conf
|
||||
register: check_jmx_confdir
|
||||
|
||||
- name: Distribute the jmx authorization files for hadoop hdfs
|
||||
template: src=templates/jmxremote.passwd.j2 dest=/etc/hadoop-jmx/conf/jmxremote.passwd owner=hdfs mode=0600
|
||||
when: check_jmx_confdir.stat.exists and hadoop_jmx_enabled
|
||||
notify:
|
||||
- Restart HDFS namenode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
- jmx
|
||||
|
||||
- name: Distribute the jmx role files for hadoop hdfs
|
||||
copy: src=files/jmxremote.access dest=/etc/hadoop-jmx/conf/jmxremote.access owner=root mode=0644
|
||||
when: check_jmx_confdir.stat.exists and hadoop_jmx_enabled
|
||||
notify:
|
||||
- Restart HDFS namenode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
- jmx
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,20 +0,0 @@
|
|||
---
|
||||
# Manage the hdfs ssh keys used by the HDFS HA
|
||||
- name: Create a ssh key for the hdfs user. Needed by NN automatic failover
|
||||
user: name=hdfs generate_ssh_key=yes ssh_key_type=rsa ssh_key_bits=2048
|
||||
tags:
|
||||
- hdfs-ssh
|
||||
|
||||
- name: Fetch the ssh public key. Needed to populate authorized_keys
|
||||
fetch: src=/usr/lib/hadoop/.ssh/id_rsa.pub dest=/var/tmp/prefix-hdfs-{{ ansible_fqdn }}-id_rsa.pub fail_on_missing=yes flat=yes
|
||||
tags:
|
||||
- hdfs-ssh
|
||||
|
||||
- name: Authorize the hdfs user ssh key. Needed by NN automatic failover
|
||||
authorized_key: user=hdfs key="{{ lookup('file', '/var/tmp/prefix-hdfs-{{ item }}.t.hadoop.research-infrastructures.eu-id_rsa.pub') }}"
|
||||
with_items:
|
||||
- nn1
|
||||
- nn2
|
||||
tags:
|
||||
- hdfs-ssh
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
---
|
||||
- name: Activate the users on hdfs, with the right permissions.
|
||||
become: yes
|
||||
become_user: hdfs
|
||||
shell: . /etc/profile.d/jdk.sh ; hadoop fs -mkdir {{ mapred_staging_root_dir }}/{{ item.login }} ; hadoop fs -chown {{ item.login }}:{{ item.login }} {{ mapred_staging_root_dir }}/{{ item.login }} ; hadoop fs -chmod 755 {{ mapred_staging_root_dir }}/{{ item.login }} ; touch /var/lib/hadoop-hdfs/.{{ item.login }}
|
||||
with_items: '{{ hadoop_users }}'
|
||||
args:
|
||||
creates: '/var/lib/hadoop-hdfs/.{{ item.login }}'
|
||||
register: create_user
|
||||
tags:
|
||||
- hdfs
|
||||
- users
|
||||
- hadoop_users
|
|
@ -1,10 +0,0 @@
|
|||
---
|
||||
- name: Install the common CDH hadoop packages
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-hdfs-zkfc
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
- zkfc
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
---
|
||||
- name: Create a /tmp directory on HDFS
|
||||
become: True
|
||||
become_user: hdfs
|
||||
command: hadoop fs -mkdir /tmp
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Fix the HDFS /tmp directory permissions
|
||||
command: hadoop fs -chmod -R 1777 /tmp
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Create a /user/history directory on HDFS
|
||||
command: hadoop fs -mkdir /user/history
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Fix the HDFS /user/history permissions
|
||||
command: hadoop fs -chmod -R 1777 /user/history
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Fix the /user/history directory owner and group
|
||||
command: hadoop fs -chown mapred:supergroup /user/history
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Create the /hbase directory on HDFS
|
||||
command: hadoop fs -mkdir /hbase
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Fix the /hbase directory ownership
|
||||
command: hadoop fs -chown hbase:hbase /hbase
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
|
@ -1,19 +0,0 @@
|
|||
---
|
||||
- name: Create the 'supergroup' user group
|
||||
group: name={{ hdfs_users_supergroup }} state=present
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Create the /data/dfs directory on the NN filesystem
|
||||
file: path={{ hdfs_data_dir }} owner=root group=root state=directory
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Create the /data/dfs/nn directory on the NN filesystem
|
||||
file: path={{ hdfs_data_dir }}/{{ hdfs_nn_data_dir }} owner=hdfs group=hdfs state=directory
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
---
|
||||
- name: Format the namenode, if it\'s not already formatted. Runs on the first namenode only
|
||||
become: True
|
||||
become_user: hdfs
|
||||
command: creates={{ hdfs_data_dir }}/{{ hdfs_nn_data_dir }}/current/VERSION hdfs namenode -format -force
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
---
|
||||
- name: Wait for the first namenode before doing anything
|
||||
wait_for: host={{ hostvars[groups['hdfs_primary_master'][0]].ipv4_address|default(hostvars[groups['hdfs_primary_master'][0]].ansible_default_ipv4.address) }} port={{ hdfs_nn_http_port }}
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
||||
- name: Bootstrap the second namenode
|
||||
become: True
|
||||
become_user: hdfs
|
||||
command: hdfs namenode -bootstrapStandby
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
---
|
||||
- name: Restart hdfs namenode
|
||||
service: name=hadoop-hdfs-namenode state=restarted
|
||||
|
||||
- name: Restart hdfs zkfc
|
||||
service: name=hadoop-hdfs-zkfc state=restarted
|
|
@ -1,11 +0,0 @@
|
|||
---
|
||||
- name: Format zkfc
|
||||
become: True
|
||||
become_user: hdfs
|
||||
command: hdfs zkfc -formatZK -force
|
||||
notify:
|
||||
- Restart hdfs zkfc
|
||||
- Restart hdfs namenode
|
||||
tags:
|
||||
- hadoop
|
||||
- hdfs
|
|
@ -1,12 +0,0 @@
|
|||
<?xml version="1.0"?>
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>mapred.queue.default.acl-submit-job</name>
|
||||
<value>*</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.queue.default.acl-administer-jobs</name>
|
||||
<value> </value>
|
||||
</property>
|
||||
</configuration>
|
|
@ -1,8 +0,0 @@
|
|||
node13.t.hadoop.research-infrastructures.eu
|
||||
node12.t.hadoop.research-infrastructures.eu
|
||||
node11.t.hadoop.research-infrastructures.eu
|
||||
node10.t.hadoop.research-infrastructures.eu
|
||||
node9.t.hadoop.research-infrastructures.eu
|
||||
node8.t.hadoop.research-infrastructures.eu
|
||||
node7.t.hadoop.research-infrastructures.eu
|
||||
node6.t.hadoop.research-infrastructures.eu
|
|
@ -1,117 +0,0 @@
|
|||
<?xml version="1.0"?>
|
||||
|
||||
<!-- This is the configuration file for the resource manager in Hadoop. -->
|
||||
<!-- You can configure various scheduling parameters related to queues. -->
|
||||
<!-- The properties for a queue follow a naming convention,such as, -->
|
||||
<!-- mapred.capacity-scheduler.queue.<queue-name>.property-name. -->
|
||||
|
||||
<configuration>
|
||||
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.queue.default.capacity</name>
|
||||
<value>100</value>
|
||||
<description>Percentage of the number of slots in the cluster that are
|
||||
to be available for jobs in this queue.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.queue.default.maximum-capacity</name>
|
||||
<value>-1</value>
|
||||
<description>
|
||||
maximum-capacity defines a limit beyond which a queue cannot use the capacity of the cluster.
|
||||
This provides a means to limit how much excess capacity a queue can use. By default, there is no limit.
|
||||
The maximum-capacity of a queue can only be greater than or equal to its minimum capacity.
|
||||
Default value of -1 implies a queue can use complete capacity of the cluster.
|
||||
|
||||
This property could be to curtail certain jobs which are long running in nature from occupying more than a
|
||||
certain percentage of the cluster, which in the absence of pre-emption, could lead to capacity guarantees of
|
||||
other queues being affected.
|
||||
|
||||
One important thing to note is that maximum-capacity is a percentage , so based on the cluster's capacity
|
||||
the max capacity would change. So if large no of nodes or racks get added to the cluster , max Capacity in
|
||||
absolute terms would increase accordingly.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.queue.default.supports-priority</name>
|
||||
<value>false</value>
|
||||
<description>If true, priorities of jobs will be taken into
|
||||
account in scheduling decisions.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.queue.default.minimum-user-limit-percent</name>
|
||||
<value>100</value>
|
||||
<description> Each queue enforces a limit on the percentage of resources
|
||||
allocated to a user at any given time, if there is competition for them.
|
||||
This user limit can vary between a minimum and maximum value. The former
|
||||
depends on the number of users who have submitted jobs, and the latter is
|
||||
set to this property value. For example, suppose the value of this
|
||||
property is 25. If two users have submitted jobs to a queue, no single
|
||||
user can use more than 50% of the queue resources. If a third user submits
|
||||
a job, no single user can use more than 33% of the queue resources. With 4
|
||||
or more users, no user can use more than 25% of the queue's resources. A
|
||||
value of 100 implies no user limits are imposed.
|
||||
</description>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.queue.default.maximum-initialized-jobs-per-user</name>
|
||||
<value>2</value>
|
||||
<description>The maximum number of jobs to be pre-initialized for a user
|
||||
of the job queue.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
<!-- The default configuration settings for the capacity task scheduler -->
|
||||
<!-- The default values would be applied to all the queues which don't have -->
|
||||
<!-- the appropriate property for the particular queue -->
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.default-supports-priority</name>
|
||||
<value>false</value>
|
||||
<description>If true, priorities of jobs will be taken into
|
||||
account in scheduling decisions by default in a job queue.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.default-minimum-user-limit-percent</name>
|
||||
<value>100</value>
|
||||
<description>The percentage of the resources limited to a particular user
|
||||
for the job queue at any given point of time by default.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.default-maximum-initialized-jobs-per-user</name>
|
||||
<value>2</value>
|
||||
<description>The maximum number of jobs to be pre-initialized for a user
|
||||
of the job queue.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
|
||||
<!-- Capacity scheduler Job Initialization configuration parameters -->
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.init-poll-interval</name>
|
||||
<value>5000</value>
|
||||
<description>The amount of time in miliseconds which is used to poll
|
||||
the job queues for jobs to initialize.
|
||||
</description>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.capacity-scheduler.init-worker-threads</name>
|
||||
<value>5</value>
|
||||
<description>Number of worker threads which would be used by
|
||||
Initialization poller to initialize jobs in a set of queue.
|
||||
If number mentioned in property is equal to number of job queues
|
||||
then a single thread would initialize jobs in a queue. If lesser
|
||||
then a thread would get a set of queues assigned. If the number
|
||||
is greater then number of threads would be equal to number of
|
||||
job queues.
|
||||
</description>
|
||||
</property>
|
||||
|
||||
</configuration>
|
|
@ -1,3 +0,0 @@
|
|||
<?xml version="1.0"?>
|
||||
<allocations>
|
||||
</allocations>
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
- name: Restart mapreduce HA jobtracker
|
||||
service: name=hadoop-0.20-mapreduce-jobtrackerha state=restarted sleep=20
|
||||
ignore_errors: true
|
|
@ -1,48 +0,0 @@
|
|||
---
|
||||
- name: Install the mapred-site config file for the jobtracker HA
|
||||
template: src=mapred-site-jobtracker.j2 dest=/etc/hadoop/conf/mapred-site.xml owner=root group=root mode=0444
|
||||
notify:
|
||||
- Restart mapreduce HA jobtracker
|
||||
tags: [ 'hadoop', 'jobtracker', 'jt_conf' ]
|
||||
|
||||
- name: Install the mapred-queue-acls config
|
||||
copy: src=mapred-queue-acls.xml dest=/etc/hadoop/conf/mapred-queue-acls.xml owner=root group=root mode=0444
|
||||
notify:
|
||||
- Restart mapreduce HA jobtracker
|
||||
tags: [ 'hadoop', 'jobtracker' ]
|
||||
|
||||
- name: Install the mapreduce schedulers config
|
||||
copy: src=mapreduce-{{ item }} dest=/etc/hadoop/conf/{{ item }} owner=root group=root mode=0444
|
||||
with_items:
|
||||
- fair-scheduler.xml
|
||||
- capacity-scheduler.xml
|
||||
tags: [ 'hadoop', 'jobtracker', 'jt_scheduler' ]
|
||||
|
||||
- name: Install the mapred_hosts_allow config
|
||||
template: src=mapred_hosts_allow.txt.j2 dest=/etc/hadoop/conf/mapred_hosts_allow.txt owner=root group=root mode=0444
|
||||
notify:
|
||||
- Restart mapreduce HA jobtracker
|
||||
tags: [ 'hadoop', 'jobtracker', 'hadoop_workers' ]
|
||||
|
||||
- name: Install the mapred_hosts_exclude config
|
||||
copy: src=mapred_hosts_exclude.txt dest=/etc/hadoop/conf/mapred_hosts_exclude.txt owner=root group=root mode=0444
|
||||
tags: [ 'hadoop', 'jobtracker', 'hadoop_workers', 'hadoop_exclude' ]
|
||||
|
||||
- name: Check if the hadoop-jmx conf directory exists
|
||||
stat: path=/etc/hadoop-jmx/conf
|
||||
register: check_jmx_confdir
|
||||
tags: [ 'hadoop', 'jobtracker', 'jmx' ]
|
||||
|
||||
- name: Distribute the jmx authorization files for the jobtracker
|
||||
template: src=templates/jmxremote.passwd.j2 dest=/etc/hadoop-jmx/conf/jmxremote.passwd owner=mapred mode=0600
|
||||
when: check_jmx_confdir.stat.exists and hadoop_jmx_enabled
|
||||
notify:
|
||||
- Restart mapreduce HA jobtracker
|
||||
tags: [ 'hadoop', 'jobtracker', 'jmx' ]
|
||||
|
||||
- name: Distribute the jmx role files for hadoop jobtracker
|
||||
copy: src=files/jmxremote.access dest=/etc/hadoop-jmx/conf/jmxremote.access owner=root mode=0644
|
||||
when: check_jmx_confdir.stat.exists and hadoop_jmx_enabled
|
||||
notify:
|
||||
- Restart mapreduce HA jobtracker
|
||||
tags: [ 'hadoop', 'jobtracker', 'jmx' ]
|
|
@ -1,219 +0,0 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
|
||||
<configuration>
|
||||
<property>
|
||||
<name>mapred.job.tracker</name>
|
||||
<value>{{ jobtracker_cluster_id }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtrackers.{{ jobtracker_cluster_id }}</name>
|
||||
<value>{{ jobtracker_cluster_id_1 }},{{ jobtracker_cluster_id_2 }}</value>
|
||||
<description>Comma-separated list of JobTracker IDs.</description>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.rpc-address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_1 }}</name>
|
||||
<!-- RPC address for {{ jobtracker_cluster_id_1 }} -->
|
||||
<value>{{ jobtracker_node_1_hostname }}:{{ jobtracker_cluster_id1_rpc_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.rpc-address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_2 }}</name>
|
||||
<!-- RPC address for {{ jobtracker_cluster_id_2 }} -->
|
||||
<value>{{ jobtracker_node_2_hostname }}:{{ jobtracker_cluster_id2_rpc_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.job.tracker.http.address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_1 }}</name>
|
||||
<!-- HTTP bind address for {{ jobtracker_cluster_id_1 }} -->
|
||||
<value>0.0.0.0:{{ jobtracker_cluster_id1_http_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.job.tracker.http.address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_2 }}</name>
|
||||
<!-- HTTP bind address for {{ jobtracker_cluster_id_2 }} -->
|
||||
<value>0.0.0.0:{{ jobtracker_cluster_id2_http_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.jobtracker.rpc-address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_1 }}</name>
|
||||
<!-- RPC address for {{ jobtracker_cluster_id_1 }} HA daemon -->
|
||||
<value>{{ jobtracker_node_1_hostname }}:{{ jobtracker_cluster_id1_ha_rpc_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.jobtracker.rpc-address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_2 }}</name>
|
||||
<!-- RPC address for {{ jobtracker_cluster_id_2 }} HA daemon -->
|
||||
<value>{{ jobtracker_node_2_hostname }}:{{ jobtracker_cluster_id2_ha_rpc_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.jobtracker.http-redirect-address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_1 }}</name>
|
||||
<!-- HTTP redirect address for {{ jobtracker_cluster_id_1 }} -->
|
||||
<value>{{ jobtracker_node_1_hostname }}:{{ jobtracker_cluster_id1_http_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.jobtracker.http-redirect-address.{{ jobtracker_cluster_id }}.{{ jobtracker_cluster_id_2 }}</name>
|
||||
<!-- HTTP redirect address for {{ jobtracker_cluster_id_2 }} -->
|
||||
<value>{{ jobtracker_node_2_hostname }}:{{ jobtracker_cluster_id2_http_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.restart.recover</name>
|
||||
<value>{{ jobtracker_restart_recover }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.client.failover.proxy.provider.{{ jobtracker_cluster_id }}</name>
|
||||
<value>org.apache.hadoop.mapred.ConfiguredFailoverProxyProvider</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.client.failover.max.attempts</name>
|
||||
<value>15</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.client.failover.sleep.base.millis</name>
|
||||
<value>500</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.client.failover.sleep.max.millis</name>
|
||||
<value>1500</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.client.failover.connection.retries</name>
|
||||
<value>{{ jobtracker_failover_connect_retries }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.client.failover.connection.retries.on.timeouts</name>
|
||||
<value>{{ jobtracker_failover_connect_retries }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.fencing.methods</name>
|
||||
<!-- We don't need a real fencing command (?) -->
|
||||
<value>shell(/bin/true)</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.automatic-failover.enabled</name>
|
||||
<value>{{ jobtracker_auto_failover_enabled }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.ha.zkfc.port</name>
|
||||
<value>{{ jobtracker_zkfc_port }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.hosts</name>
|
||||
<value>/etc/hadoop/conf/mapred_hosts_allow.txt</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.hosts.exclude</name>
|
||||
<value>/etc/hadoop/conf/mapred_hosts_exclude.txt</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.system.dir</name>
|
||||
<value>/tmp/mapred/system</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapreduce.jobtracker.staging.root.dir</name>
|
||||
<value>{{ mapred_staging_root_dir }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.acls.enabled</name>
|
||||
<value>false</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.local.dir</name>
|
||||
<value>/data/mapred/jt</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>hadoop.job.history.location</name>
|
||||
<!-- <value>file:////var/log/hadoop-0.20-mapreduce/history</value> -->
|
||||
<value>/jobtracker/history</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.taskScheduler</name>
|
||||
<value>org.apache.hadoop.mapred.FairScheduler</value>
|
||||
</property>
|
||||
{% if mapred_use_fair_scheduler %}
|
||||
<property>
|
||||
<name>mapred.jobtracker.taskScheduler</name>
|
||||
<value>org.apache.hadoop.mapred.FairScheduler</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.fairscheduler.allocation.file</name>
|
||||
<value>{{ mapred_fair_scheduler_allocation_file }}</value>
|
||||
</property>
|
||||
{% if mapred_fair_scheduler_use_poolnameproperty %}
|
||||
<property>
|
||||
<name>mapred.fairscheduler.poolnameproperty</name>
|
||||
<value>{{ mapred_fair_scheduler_poolnameproperty }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.fairscheduler.allow.undeclared.pools</name>
|
||||
<value>{{ mapred_fair_scheduler_undecl_pools }}</value>
|
||||
</property>
|
||||
{% endif %}
|
||||
<property>
|
||||
<name>mapred.fairscheduler.weight.adjuster</name>
|
||||
<value></value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.fairscheduler.assignmultiple</name>
|
||||
<value>{{ mapred_fair_scheduler_assignmultiple }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.fairscheduler.preemption</name>
|
||||
<value>{{ mapred_fair_scheduler_preemption }}</value>
|
||||
</property>
|
||||
{% endif %}
|
||||
<property>
|
||||
<name>mapred.job.tracker.handler.count</name>
|
||||
<value>{{ jobtracker_handler_count }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.reduce.slowstart.completed.maps</name>
|
||||
<value>{{ mapred_reduce_slowstart_maps }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapreduce.jobtracker.split.metainfo.maxsize</name>
|
||||
<value>{{ mapreduce_jt_split_metainfo_maxsize }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.user.jobconf.limit</name>
|
||||
<value>{{ mapred_user_jobconf_limit }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapreduce.job.counters.max</name>
|
||||
<value>{{ mapreduce_job_counters_max }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.retirejob.interval</name>
|
||||
<value>{{ mapred_jt_retirejob_interval }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.job.tracker.persist.jobstatus.active</name>
|
||||
<value>{{ jobtracker_persistent_jobstatus }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.job.tracker.persist.jobstatus.hours</name>
|
||||
<value>{{ mapred_jt_persist_jobstatus_hours }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.job.tracker.persist.jobstatus.dir</name>
|
||||
<value>/jobtracker/jobsInfo</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.completeuserjobs.maximum</name>
|
||||
<value>{{ mapred_jt_completeuserjobs_max }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.job.restart.recover</name>
|
||||
<value>{{ jobtracker_restart_recover }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>hadoop.rpc.socket.factory.class.JobSubmissionProtocol</name>
|
||||
<value></value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.jobtracker.plugins</name>
|
||||
<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>mapred.queue.names</name>
|
||||
<value>{{ mapred_queue_names }}</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>jobtracker.thrift.address</name>
|
||||
<value>0.0.0.0:{{ jobtracker_http_port }}</value>
|
||||
</property>
|
||||
</configuration>
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['mapred_tasktrackers'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,33 +0,0 @@
|
|||
---
|
||||
- name: Remove the non HA jobtracker package
|
||||
apt: pkg={{ item }} state=absent
|
||||
with_items:
|
||||
- hadoop-0.20-mapreduce-jobtracker
|
||||
tags:
|
||||
- jobtracker
|
||||
- mapred
|
||||
|
||||
- name: Install the HA jobtracker package
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-0.20-mapreduce-jobtrackerha
|
||||
tags:
|
||||
- jobtracker
|
||||
- mapred
|
||||
|
||||
- name: Install the hue-plugins package, needed to access the jobtracker from hue
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hue-plugins
|
||||
tags:
|
||||
- jobtracker
|
||||
- mapred
|
||||
|
||||
- name: Install the dsh package
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- dsh
|
||||
tags:
|
||||
- jobtracker
|
||||
- mapred
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
---
|
||||
- name: Install the ZKFC jobtracker package
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-0.20-mapreduce-zkfc
|
||||
tags:
|
||||
- jobtracker
|
||||
- mapred
|
||||
- jt_zkfc
|
|
@ -1,17 +0,0 @@
|
|||
---
|
||||
- name: Install the mapred tasktracker
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- hadoop-0.20-mapreduce
|
||||
- hadoop-0.20-mapreduce-tasktracker
|
||||
tags:
|
||||
- tasktracker
|
||||
- mapred
|
||||
|
||||
- name: Install non hadoop packages needed by some hadoop jobs
|
||||
apt: pkg={{ item }} state={{ hadoop_pkg_state }}
|
||||
with_items:
|
||||
- python-apsw
|
||||
tags:
|
||||
- tasktracker
|
||||
- mapred
|
|
@ -1,5 +0,0 @@
|
|||
---
|
||||
- name: Restart mapreduce tasktracker
|
||||
service: name=hadoop-0.20-mapreduce-tasktracker state=restarted sleep=20
|
||||
ignore_errors: true
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
---
|
||||
- name: Install the mapred-site config file for the mapreduce tasktracker
|
||||
template: src=templates/mapred-site-tasktracker.xml.j2 dest=/etc/hadoop/conf/mapred-site.xml owner=root group=root mode=0444
|
||||
notify:
|
||||
- Restart mapreduce tasktracker
|
||||
tags:
|
||||
- hadoop
|
||||
- tasktracker
|
||||
- mapred_conf
|
||||
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
export PATH="/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH"
|
||||
|
||||
function restart_ntp() {
|
||||
restart-ntp
|
||||
}
|
|
@ -1,3 +0,0 @@
|
|||
---
|
||||
dependencies:
|
||||
- role: '../../library/roles/nginx'
|
|
@ -1,79 +0,0 @@
|
|||
---
|
||||
- name: Create the configuration directory for dsh
|
||||
file: path=/root/.dsh/group owner=root group=root state=directory
|
||||
|
||||
- name: Install the dsh host groups
|
||||
template: src=dsh-{{ item }}.j2 dest=/root/.dsh/group/{{ item }}
|
||||
with_items:
|
||||
- quorum
|
||||
- namenodes
|
||||
- datanodes
|
||||
- hbase-master
|
||||
- jobtrackers
|
||||
tags:
|
||||
- system
|
||||
|
||||
# Install the global start/stop/restart scripts. jobtracker drives all the other nodes
|
||||
- name: Install the Hadoop cluster start/stop scripts
|
||||
template: src={{item }}.j2 dest=/usr/local/bin/{{ item }} owner=root group=root mode=555
|
||||
with_items:
|
||||
- service-hdfs-journalnode
|
||||
- service-hdfs-zkfc
|
||||
- service-hdfs-namenode
|
||||
- service-hdfs-secondarynamenode
|
||||
- service-hdfs-datanode
|
||||
- service-hdfs-httpfs
|
||||
- service-zookeeper-server
|
||||
- service-hbase-master
|
||||
- service-hbase-regionserver
|
||||
- service-hbase-rest
|
||||
- service-hbase-thrift
|
||||
- service-mapreduce-jobtracker
|
||||
- service-mapreduce-jobtracker-zkfc
|
||||
- service-mapreduce-tasktracker
|
||||
- service-global-hadoop-cluster
|
||||
- service-global-hbase
|
||||
- service-global-mapred
|
||||
- service-global-hdfs
|
||||
tags:
|
||||
- system
|
||||
|
||||
- name: Install the shell functions library
|
||||
copy: src={{ item }}.sh dest=/usr/local/lib/{{ item }} owner=root group=root mode=444
|
||||
with_items:
|
||||
- service-hadoop-common-functions
|
||||
tags:
|
||||
- system
|
||||
|
||||
- name: Another name for the zookeeper script
|
||||
file: src=/usr/local/bin/service-zookeeper-server dest=/usr/local/bin/service-global-zookeeper state=link
|
||||
tags:
|
||||
- system
|
||||
|
||||
- name: update nginx config
|
||||
template: src={{ portal_nginx_conf }}-nginx.conf.j2 dest=/etc/nginx/sites-available/{{ portal_nginx_conf }}
|
||||
notify: Reload nginx
|
||||
tags:
|
||||
- portal
|
||||
|
||||
- name: symlink nginx config
|
||||
file: src=/etc/nginx/sites-available/{{ portal_nginx_conf }} dest=/etc/nginx/sites-enabled/{{ portal_nginx_conf }} state=link
|
||||
notify: Reload nginx
|
||||
tags:
|
||||
- portal
|
||||
|
||||
- name: Create the web root if it doesn''t exist
|
||||
file: dest={{ portal_web_root }} state=directory
|
||||
tags:
|
||||
- portal
|
||||
|
||||
- name: Create a fake favicon
|
||||
copy: content="" dest={{ portal_web_root }}/favicon.ico owner=root group=root mode=0444
|
||||
tags:
|
||||
- portal
|
||||
|
||||
- name: Install the index file
|
||||
template: src=management-portal-index.html.j2 dest={{ portal_web_root }}/index.html mode=444
|
||||
tags:
|
||||
- portal
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['hbase_masters'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['jt_masters'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['hdfs_masters'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,3 +0,0 @@
|
|||
{% for host in groups['zookeeper_cluster'] %}
|
||||
{{ host }}
|
||||
{% endfor %}
|
|
@ -1,49 +0,0 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>
|
||||
{{ portal_title }}
|
||||
</title>
|
||||
</head>
|
||||
<body>
|
||||
<ul> <h2>HUE interface</h2>
|
||||
{% for host in groups['hue'] %}
|
||||
<li><a href="http://{{ host }}:{{ hue_http_port }}/jobbrowser">HUE interface</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>Jobtracker</h2>
|
||||
{% for host in groups['jt_masters'] %}
|
||||
<li><a href="http://{{ jobtracker_node_1_hostname }}:{{ hostvars[host]['jt_http'] }}">{{ host }} Jobtracker master</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>Mapred tasktrackers</h2>
|
||||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
<li><a href="http://{{ jobtracker_node_1_hostname }}:{{ hostvars[host]['mapred_http'] }}">{{ host }} mapred tasktracker</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>HDFS namenode</h2>
|
||||
{% for host in groups['hdfs_masters'] %}
|
||||
<li><a href="http://{{ jobtracker_node_1_hostname }}:{{ hostvars[host]['hdfs_m_http'] }}">{{ host }} HDFS namenode</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>HDFS datanodes</h2>
|
||||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
<li><a href="http://{{ jobtracker_node_1_hostname }}:{{ hostvars[host]['hdfs_http'] }}">{{ host }} HDFS datanode</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>HBASE master</h2>
|
||||
{% for host in groups['hbase_masters'] %}
|
||||
<li><a href="http://{{ jobtracker_node_1_hostname }}:{{ hostvars[host]['hbase_m_http'] }}">{{ host }} HBASE master</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>HBASE regionservers</h2>
|
||||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
<li><a href="http://{{ jobtracker_node_1_hostname }}:{{ hostvars[host]['hbase_http'] }}">{{ host }} HBASE regionserver</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<ul> <h2>Logstash collector</h2>
|
||||
{% for host in groups['logstash'] %}
|
||||
<li><a href="http://{{ host }}/">{{ host }} Logstash collector</a></li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
</body>
|
||||
</html>
|
|
@ -1,279 +0,0 @@
|
|||
server {
|
||||
root {{ portal_web_root }};
|
||||
index index.html;
|
||||
|
||||
gzip on;
|
||||
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
|
||||
gzip_types text/javascript text/css application/x-javascript application/javascript application/json image/svg+xml;
|
||||
gzip_vary on;
|
||||
gzip_proxied any;
|
||||
|
||||
server_name {{ ansible_fqdn }};
|
||||
|
||||
location / {
|
||||
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
# HUE is a real mess
|
||||
{% if hue_servers is defined %}
|
||||
{% for host in groups['hue_servers'] %}
|
||||
location /jobbrowser {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/jobbrowser;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "nginx";
|
||||
}
|
||||
location /accounts {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/accounts;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /beeswax {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/beeswax;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /oozie {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/oozie;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /help {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/help;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /static {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/static;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /jobsub {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/jobsub;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /shell {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/shell;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /useradmin {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/useradmin;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
location /filebrowser {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }}/filebrowser;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Access";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
}
|
||||
|
||||
# HUE
|
||||
{% for host in groups['hue'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['hue_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ hue_http_port }};
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# Jobtracker HA masters
|
||||
{% for host in groups['jt_masters'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['jt_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ jobtracker_cluster_id1_http_port }}/;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# Map/Reduce tasktrackers
|
||||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['mapred_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ mapred_tasktracker_http_port }}/;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# HDFS masters
|
||||
{% for host in groups['hdfs_masters'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['hdfs_m_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ hdfs_nn_http_port }}/;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# HDFS datanodes
|
||||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['hdfs_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ hdfs_datanode_http_port }}/;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# HBASE masters
|
||||
{% for host in groups['hbase_masters'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['hbase_m_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ hbase_master_http_port }}/;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# HBASE regionservers
|
||||
{% for host in groups['hadoop_worker_nodes'] %}
|
||||
server {
|
||||
listen {{ hostvars[host]['hbase_http'] }};
|
||||
location / {
|
||||
proxy_pass http://{{ host }}:{{ hbase_regionserver_http_port }}/;
|
||||
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
proxy_redirect off;
|
||||
proxy_buffering off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
auth_pam "NeMIS Hadoop Cluster Access";
|
||||
auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
|
||||
# Logstash
|
||||
# -- NB: that doesn't work, kibana keeps searching the elasticsearch instance as jobtracker.t.hadoop.
|
||||
#{% for host in groups['logstash'] %}
|
||||
#server {
|
||||
# listen {{ hostvars[host]['log_http'] }};
|
||||
# location / {
|
||||
# proxy_pass http://{{ host }}/;
|
||||
# proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
# proxy_redirect off;
|
||||
# proxy_buffering off;
|
||||
# proxy_set_header Host $host;
|
||||
# proxy_set_header X-Real-IP $remote_addr;
|
||||
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
# auth_pam "NeMIS Hadoop Cluster Logstash data";
|
||||
# auth_pam_service_name "{{ portal_pam_svc_name }}";
|
||||
# }
|
||||
#}
|
||||
#{% endfor %}
|
|
@ -1,53 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
# Correct start order (reverse to obtain the stop order):
|
||||
#
|
||||
# • HDFS
|
||||
# • MapReduce
|
||||
# • Zookeeper
|
||||
# • HBase
|
||||
# (• Hive Metastore )
|
||||
# (• Hue )
|
||||
# (• Oozie)
|
||||
# • Ganglia
|
||||
# • Nagios
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
SERVICES_START_ORDER="service-global-zookeeper service-global-hdfs service-global-mapred service-global-hbase"
|
||||
SERVICES_STOP_ORDER="service-global-hbase service-global-mapred service-global-hdfs service-global-zookeeper"
|
||||
|
||||
SH_LIB_PATH=/usr/local/lib
|
||||
if [ -f $SH_LIB_PATH/service-hadoop-common-functions ] ; then
|
||||
. $SH_LIB_PATH/service-hadoop-common-functions
|
||||
else
|
||||
echo "Library file: $SH_LIB_PATH/service-hadoop-common-functions is missing"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SERVICES=$SERVICES_START_ORDER
|
||||
ARG=$1
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
if [ "$ACTION" == "stop" ] ; then
|
||||
SERVICES=$SERVICES_STOP_ORDER
|
||||
fi
|
||||
for SRV in $SERVICES ; do
|
||||
$SRV $ACTION
|
||||
done
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload|status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,38 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
# Correct start order (reverse to obtain the stop order):
|
||||
#
|
||||
# • HBase master
|
||||
# • HBase regionservers
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
SERVICES_START_ORDER="service-hbase-master service-hbase-regionserver service-hbase-thrift"
|
||||
SERVICES_STOP_ORDER="service-hbase-thrift service-hbase-regionserver service-hbase-master"
|
||||
|
||||
SERVICES=$SERVICES_START_ORDER
|
||||
ARG=$1
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
if [ "$ACTION" == "stop" ] ; then
|
||||
SERVICES=$SERVICES_STOP_ORDER
|
||||
fi
|
||||
for SRV in $SERVICES ; do
|
||||
$SRV $ACTION
|
||||
done
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload|status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,40 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
# Correct start order (reverse to obtain the stop order):
|
||||
#
|
||||
# • HDFS namenode 1
|
||||
# • HDFS namenode 2
|
||||
# • HDFS journalnodes
|
||||
# • HDFS datanodes
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
SERVICES_START_ORDER="service-hdfs-journalnode service-hdfs-zkfc service-hdfs-namenode service-hdfs-datanode"
|
||||
SERVICES_STOP_ORDER="service-hdfs-datanode service-hdfs-namenode service-hdfs-journalnode service-hdfs-zkfc"
|
||||
|
||||
SERVICES=$SERVICES_START_ORDER
|
||||
ARG=$1
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
if [ "$ACTION" == "stop" ] ; then
|
||||
SERVICES=$SERVICES_STOP_ORDER
|
||||
fi
|
||||
for SRV in $SERVICES ; do
|
||||
$SRV $ACTION
|
||||
done
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload|status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,38 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
# Correct start order (reverse to obtain the stop order):
|
||||
#
|
||||
# • MapReduce jobtracker
|
||||
# • MapReduce tasktrackers
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
SERVICES_START_ORDER="service-mapreduce-jobtracker-zkfc service-mapreduce-jobtracker service-mapreduce-tasktracker"
|
||||
SERVICES_STOP_ORDER="service-mapreduce-tasktracker service-mapreduce-jobtracker-zkfc service-mapreduce-jobtracker"
|
||||
|
||||
SERVICES=$SERVICES_START_ORDER
|
||||
ARG=$1
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
if [ "$ACTION" == "stop" ] ; then
|
||||
SERVICES=$SERVICES_STOP_ORDER
|
||||
fi
|
||||
for SRV in $SERVICES ; do
|
||||
$SRV $ACTION
|
||||
done
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload|status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,37 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
# Correct start order (reverse to obtain the stop order):
|
||||
#
|
||||
# • Zookeeper server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
SERVICES_START_ORDER="service-zookeeper-server"
|
||||
SERVICES_STOP_ORDER=$SERVICES_START_ORDER
|
||||
|
||||
SERVICES=$SERVICES_START_ORDER
|
||||
ARG=$1
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
if [ "$ACTION" == "stop" ] ; then
|
||||
SERVICES=$SERVICES_STOP_ORDER
|
||||
fi
|
||||
for SRV in $SERVICES ; do
|
||||
$SRV $ACTION
|
||||
done
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload|status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,32 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hbase-master
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=hbase-master
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hbase-regionserver
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=datanodes
|
||||
|
||||
function ntp_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- restart-ntp
|
||||
}
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
# ntp_loop
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,38 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hbase-rest
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=datanodes
|
||||
|
||||
function ntp_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- restart-ntp
|
||||
}
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
# ntp_loop
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,33 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
export PATH="/sbin:/usr/sbin:$PATH"
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hbase-thrift
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=hbase-master
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,38 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hadoop-hdfs-datanode
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=datanodes
|
||||
|
||||
function ntp_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- restart-ntp
|
||||
}
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
# ntp_loop
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,40 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
export PATH="/sbin:/usr/sbin:$PATH"
|
||||
|
||||
ARG=$1
|
||||
NAMENODE={{ secondary_nm_hostname }}
|
||||
SERVICE_SCRIPT=service
|
||||
STARTUP_SCRIPT=hadoop-httpfs
|
||||
REMOTE_CMD=ssh
|
||||
|
||||
case "$ARG" in
|
||||
start)
|
||||
# $REMOTE_CMD $NAMENODE restart-ntp
|
||||
echo "Running $STARTUP_SCRIPT $ARG on host $NAMENODE"
|
||||
$REMOTE_CMD $NAMENODE $SERVICE_SCRIPT $STARTUP_SCRIPT start
|
||||
;;
|
||||
restart|reload|force-reload)
|
||||
# $REMOTE_CMD $NAMENODE restart-ntp
|
||||
echo "Running $STARTUP_SCRIPT $ARG on host $NAMENODE"
|
||||
$REMOTE_CMD $NAMENODE $SERVICE_SCRIPT $STARTUP_SCRIPT restart
|
||||
;;
|
||||
status)
|
||||
echo "Running $STARTUP_SCRIPT $ARG on host $NAMENODE"
|
||||
$REMOTE_CMD $NAMENODE $SERVICE_SCRIPT $STARTUP_SCRIPT status
|
||||
;;
|
||||
stop)
|
||||
echo "Running $STARTUP_SCRIPT $ARG on host $NAMENODE"
|
||||
$REMOTE_CMD $NAMENODE $SERVICE_SCRIPT $STARTUP_SCRIPT stop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
|
@ -1,39 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
DOMAIN_N="t.hadoop.research-infrastructures.eu"
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hadoop-hdfs-journalnode
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=quorum
|
||||
|
||||
function ntp_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- restart-ntp
|
||||
}
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
# ntp_loop
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# We use the jobtracker as provisioning server
|
||||
#
|
||||
HOSTNAME=$( hostname -f )
|
||||
export PATH="/sbin:/usr/sbin:$PATH"
|
||||
|
||||
ARG=$1
|
||||
SERVICE_CMD=/usr/sbin/service
|
||||
STARTUP_SCRIPT=hadoop-hdfs-namenode
|
||||
REMOTE_CMD=dsh
|
||||
DSH_GROUPNAME=namenodes
|
||||
|
||||
function ntp_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- restart-ntp
|
||||
}
|
||||
|
||||
function action_loop(){
|
||||
ACTION=$ARG
|
||||
dsh -g ${DSH_GROUPNAME} -cM -- ${SERVICE_CMD} ${STARTUP_SCRIPT} ${ACTION}
|
||||
}
|
||||
|
||||
case "$ARG" in
|
||||
start|restart|reload|force-reload)
|
||||
# ntp_loop
|
||||
action_loop
|
||||
;;
|
||||
status|stop)
|
||||
action_loop
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 start|stop|restart|status" >&2
|
||||
exit 3
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue