1 搭建准备
HostName | IP | 内存 | CPU | 磁盘 |
---|---|---|---|---|
cdp01 | 172.29.0.110 | 16G | 4核 | 100G |
cdp02 | 172.29.0.111 | 16G | 4核 | 100G |
cdp03 | 172.29.0.112 | 16G | 4核 | 100G |
组件 | 版本 |
---|---|
Cloudera Manager | 7.1.4 |
CDH | 7.1.7 |
OpenJDK | 1.8.0_372 |
CentOS | 7.6 |
MySQL | 8.0.34 |
2 基础配置2.1 hosts配置(所有主机)
- 编辑hosts文件
vim /etc/hosts
删除原IP配置,填入以下内容172.29.0.110 cdp01172.29.0.111 cdp02172.29.0.112 cdp03
- 设置主机名称
cdp01节点hostnamectl set-hostname cdp01 && bash
cdp02节点hostnamectl set-hostname cdp02 && bash
cdp03节点hostnamectl set-hostname cdp03 && bash
2.2 关闭 Selinux(所有主机)
setenforce 0sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
2.3 关闭防火墙(所有主机)
systemctl stop firewalldsystemctl disable firewalld
2.4 设置swap(所有主机)
cat >> /etc/sysctl.conf <<EOFvm.swappiness=1EOF
2.5 设置 limits(所有主机)
cat >> /etc/security/limits.conf <<EOF* soft nofile 65535* hard nofile 65535hdfs - nofile 32768mapred - nofile 32768hbase - nofile 32768hdfs - noproc 32768mapred - noproc 32768hbase - noproc 32768EOF
2.6 关闭透明大页(所有主机)
vim /etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/enabledecho never > /sys/kernel/mm/transparent_hugepage/defragif test -f /sys/kernel/mm/transparent_hugepage/enabled; thenecho never > /sys/kernel/mm/transparent_hugepage/enabledfiif test -f /sys/kernel/mm/transparent_hugepage/defrag; thenecho never > /sys/kernel/mm/transparent_hugepage/defragfi
2.7 安装ntpd服务(所有主机)
yum install -y ntp ntpdate
- cdp01 节点
vim /etc/ntp.conf
// 注:192.168.191.2 和 255.255.255.0是集群所在网段的网关和子网掩码restrict 192.168.191.2 mask 255.255.255.0 nomodify notrapserver 127.127.1.0fudge 127.127.1.0 stratum 8
- cdp02 节点
vim /etc/ntp.conf
restrict 192.168.191.2 mask 255.255.255.0 nomodify notrapserver cdp02 preferserver 127.127.1.0fudge 127.127.1.0 stratum 9
- cdp03 节点
vim /etc/ntp.conf
server cdp01 preferserver cdp02
- 在manager( cdp01 )启动ntp之后,其余各节点启动ntp服务之前,master(cdp02 )和worker( cdp03 )节点执行命令,同步manager时间启动 cdp01 节点ntpd服务
systemctl start ntpdsystemctl enable ntpd
- 在cdp02、cdp03 节点进行时间同步
ntpdate cdp01
- 启动cdp02、cdp03节点ntpd服务
systemctl start ntpdsystemctl enable ntpd
- 查看ntp状态
ntpq -p’*’ 表示当前使用的时钟源,’+’ 表示这些源可作为 NTP 源
2.8 安装http服务(所有主机)
- 安装httpd服务
yum install httpd -y
- 开启httpd服务
systemctl start httpd.service
- 开机⾃动启动httpd服务
chkconfig httpd on
2.9 配置SSH免密(cdp01节点)
- 在要免密登录的主机上生成密钥
ssh-keygen -t rsa # 一路回车
在~/.ssh目录下创建两个文件id_rsa(私钥),id_rsa.pub(公钥) - 将生成的公钥发到要远程连接的服务器上
ssh-copy-id cdp01ssh-copy-id cdp02ssh-copy-id cdp03
注:这一次需要输入密码,后面就不用了;如果主机未设置账号密码登录需要先设置账号密码;配置后可以通过将/etc/ssh/sshd_config文件中的PasswordAuthentication yes修改为PasswordAuthentication no来关闭账号密码登录
vim /etc/ssh/sshd_config修改如下:PasswordAuthentication no修改后使其生效:systemctl restart sshd
- 验证免密登录效果
ssh root@cdp02退出输入:exit
3 配置本地yum源3.1 配置cm
- 安装createrepo
yum install createrepo
- 下载cm软件包
mkdir -p /var/www/html/cm7.1.4cd /var/www/html/cm7.1.4
下载所需软件包,如果下载慢可以自己下载然后上传到服务器
wget -nd -r -l1 --no-parent https://archive.cloudera.com/cm7/7.1.4/redhat7/yum/RPMS/x86_64/ wget https://archive.cloudera.com/cm7/7.1.4/redhat7/yum/RPM-GPG-KEY-clouderawget https://archive.cloudera.com/cm7/7.1.4/allkeys.asc
- 创建repo
createrepo .
- 下载cdh软件包
mkdir -p /var/www/html/cdh7.1.7cd /var/www/html/cdh7.1.7
下载所需软件包,如果下载慢可以自己下载然后上传到服务器
wget -nd -r -l1 --no-parent https://archive.cloudera.com/cdh7/7.1.7.0/parcels/
- 创建repo
createrepo .
- 配置本地yum源
cd /etc/yum.repos.dvim cm7.repo
配置以下内容:
[cm]name=cm,Version 7.1.4baseurl=http://172.29.0.110/cm7.1.4/enable=1gpgcheck=0[cdh]name=cdhbaseurl=http://172.29.0.110/cdh7.1.7/enable=1gpgcheck=0
退出保存后,执⾏以下命令:
yum clean allyum makecache
查看配置结果
yum list |grep cloudera
可以看到以下内容:
可以通过外网地址查看配置结果//xxx.xxx.xxx.xxx为cdp01外网IPhttp://xxx.xxx.xxx.xxx/cm7.1.4/http://xxx.xxx.xxx.xxx/cdh7.1.7/
- 分发到其他节点
在cdp01上配置好本地yum仓库之后,将cm7.repo⽂件分发到其他节点/etc/yum.repos.d目录
4 安装jdk(所有主机)
yum -y install java-1.8.0-openjdk
5 安装MySQL(cdp01)5.1 安装并启用MySQL源
使用wget从MySQL网站下载MySQL源rpm文件:
cd /root/downloads/wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
使用rpm命令来安装下载的rpm包:
rpm -ivh mysql80-community-release-el7-3.noarch.rpm
检查MySQL 8.0源已经成功添加到yum程序库中:
yum repolist enabled | grep "mysql.*-community.*"
5.2 使用yum命令安装MySQL 8.0
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022yum install -y mysql-community-server
5.3 启动并检查MySQL服务器
systemctl start mysqldsystemctl status mysqld
5.4 修改Root密码
查找密码并登陆Mysql
cat /var/log/mysqld.log | grep passwordmysql -u root -p #输入密码
alter user root@localhost identified by '密码';FLUSH PRIVILEGES;
use mysql;update user set host='%' where user = 'root';commit;exit;
5.5 my.cnf配置
vi /etc/my.cnf
#设置编码为utf-8character_set_server=utf8init_connect='SET NAMES utf8'
5.6 重启服务
systemctl stop mysqldsystemctl enable mysqldsystemctl start mysqld
5.7 创建CM元数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'scm'@'%' IDENTIFIED BY '密码';GRANT ALL ON scm.* TO 'scm'@'%';CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'amon'@'%' IDENTIFIED BY '密码';GRANT ALL ON amon.* TO 'amon'@'%';CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'rman'@'%' IDENTIFIED BY '密码';GRANT ALL ON rman.* TO 'rman'@'%';CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'hue'@'%' IDENTIFIED BY '密码';GRANT ALL ON hue.* TO 'hue'@'%';CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'hive'@'%' IDENTIFIED BY '密码';GRANT ALL ON hive.* TO 'hive'@'%';CREATE DATABASE ranger DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'rangeradmin'@'%' IDENTIFIED BY '密码';GRANT ALL ON ranger.* TO 'rangeradmin'@'%';CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'nav'@'%' IDENTIFIED BY '密码';GRANT ALL ON nav.* TO 'nav'@'%';CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'navms'@'%' IDENTIFIED BY '密码';GRANT ALL ON navms.* TO 'navms'@'%';CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'oozie'@'%' IDENTIFIED BY '密码';GRANT ALL ON oozie.* TO 'oozie'@'%';flush privileges;
6 安装jdbc驱动(所有主机)
cd /root/downloads/wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gzmkdir -p /usr/share/javatar zxvf mysql-connector-java-5.1.46.tar.gzscp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jarscp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp02:/usr/share/java/mysql-connector-java.jarscp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp03:/usr/share/java/mysql-connector-java.jar
7 安装cm server(cdp01)7.1 在cdp01上通过yum安装Cloudera Manager Server
yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
7.2 安装完后可以查看服务状态
systemctl status cloudera-scm-server
7.3 初始化mysql
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm '之前设置的密码'
7.4 启动Cloudera Manager Server
systemctl start cloudera-scm-server.servicesystemctl status cloudera-scm-server.service
7.5 检查服务日志,等待启动完成
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
7.6 检查端口是否监听
netstat -nltp | grep 7180
7.7 访问CM 创建集群
通过 http://cdp01:7180/ 访问CM,输⼊初始账号密码 admin/admin 即可登陆