# 命令行部署 --- ## 前提 首先登录到zookeeper1，然后切换到root用户 ``` sh ssh zookeeper1 su - root ``` 在开始前，请配置yum源，安装lava命令行管理工具 ```sh # 从yum源所在机器(假设为192.168.1.10)获取repo文件 scp root@192.168.1.10:/etc/yum.repos.d/oushu.repo /etc/yum.repos.d/oushu.repo # 追加yum源所在机器信息到/etc/hosts文件 # 安装lava命令行管理工具 yum clean all yum makecache yum install -y lava ``` 创建一个zkhostfile包含要安装ZooKeeper的机器： ``` sh touch ${HOME}/zkhostfile ``` 添加下面主机名到zkhostfile，该文件记录安装ZooKeeper的主机名称： ``` sh zookeeper1 zookeeper2 zookeeper3 ``` 交换key并分发repo，在首台机器上和集群内其他节点交换公钥，以便ssh免密码登陆和分发配置文件 ```sh # 和集群内其他机器交换公钥 lava ssh-exkeys -f ${HOME}/zkhostfile -p ******** # 将repo文件分发给集群内其他机器 lava scp -f ${HOME}/zkhostfile /etc/yum.repos.d/oushu.repo =:/etc/yum.repos.d ``` ### Kerberos认证依赖(可选) 若开启Kerberos认证，Kerberos安装部署请参考：[Kerberos 安装](../kerberos/installation-kerberos.rst)。 Kerberos 服务地址假定为`kdc1` ## 准备 检查机器hosts设置 ```shell cat /etc/hosts ``` ```shell 127.0.0.1 localhost.localdomain localhost 127.0.0.1 localhost4.localdomain4 localhost4 10.0.128.13 zookeeper1 10.0.128.5 zookeeper2 10.0.128.8 zookeeper3 ``` 如果hosts文件中包含行形如下面信息，删除此行： ```shell 127.0.0.1 hostname hostname ``` ### kerberos准备(可选) 如果开启Kerberos，则需要在所有HDFS节点安装Kerberos客户端。 ```shell lava ssh -f ${HOME}/hdfshost -e "yum install -y krb5-libs krb5-workstation" ``` 登录到kdc1机器上 ```shell ssh kdc1 mkdir -p /etc/security/keytabs kadmin.local ``` 创建ZooKeeper相关用户，生成的ZooKeeper.keytab。 注意：这里对应的hostname不管大小写，都需要用小写 登录kdc ```shell kadmin.local addprinc -randkey zookeeper/zookeeper1@OUSHU.COM addprinc -randkey zookeeper/zookeeper2@OUSHU.COM addprinc -randkey zookeeper/zookeeper3@OUSHU.COM addprinc -randkey HTTP/zookeeper1@OUSHU.COM addprinc -randkey HTTP/zookeeper2@OUSHU.COM addprinc -randkey HTTP/zookeeper3@OUSHU.COM ktadd -k /etc/security/keytabs/zookeeper.keytab zookeeper/zookeeper1@OUSHU.COM ktadd -k /etc/security/keytabs/zookeeper.keytab zookeeper/zookeeper2@OUSHU.COM ktadd -k /etc/security/keytabs/zookeeper.keytab zookeeper/zookeeper3@OUSHU.COM ktadd -norandkey -k /etc/security/keytabs/zookeeper.keytab HTTP/zookeeper1@OUSHU.COM ktadd -norandkey -k /etc/security/keytabs/zookeeper.keytab HTTP/zookeeper2@OUSHU.COM ktadd -norandkey -k /etc/security/keytabs/zookeeper.keytab HTTP/zookeeper3@OUSHU.COM ``` 生成的keytab进行分发： ```sh ssh zookeeper1 lava ssh -f ${HOME}/hdfshost -e 'mkdir -p /etc/security/keytabs/' scp root@kdc1:/etc/krb5.conf /etc/krb5.conf scp root@kdc1:/etc/security/keytabs/zookeeper.keytab /etc/security/keytabs/zookeeper.keytab lava scp -r -f ${HOME}/zkhostfile /etc/krb5.conf =:/etc/krb5.conf lava scp -r -f ${HOME}/zkhostfile /etc/security/keytabs/hdfs.keytab =:/etc/security/keytabs/zookeeper.keytab lava ssh -f ${HOME}/zkhostfile -e 'chown zookeeper:hadoop /etc/security/keytabs/zookeeper.keytab' ``` ## 安装 使用yum进行安装 ``` sh lava ssh -f ${HOME}/zkhostfile -e 'yum install -y zookeeper' ``` 创建ZooKeeper数据目录： ``` sh lava ssh -f ${HOME}/zkhostfile -e 'mkdir -p /data1/zookeeper/data' lava ssh -f ${HOME}/zkhostfile -e 'chown -R zookeeper:zookeeper /data1/zookeeper' ``` 分别在三台配置了ZooKeeper的节点上配置myid文件，分别节点zookeeper1、zookeeper2、zookeeper3创建文件myid，内容分别是1、2和3，然后为zookeeper用户加上权限： ``` echo 1 > myid lava ssh -h zookeeper1 ./myid =:/data1/zookeeper/data echo 2 > myid lava ssh -h zookeeper2 ./myid =:/data1/zookeeper/data echo 3 > myid lava ssh -h zookeeper3 ./myid =:/data1/zookeeper/data lava ssh -f ${HOME}/zkhostfile -e "chown -R zookeeper:zookeeper /data1/zookeeper/data" ``` ## 配置 修改在/usr/local/oushu/conf/zookeeper/下的配置文件zoo.cfg，添加集群信息： ``` sh clientPort=2181 server.1=zookeeper1:2888:3888 server.2=zookeeper2:2888:3888 server.3=zookeeper3:2888:3888 ``` ### kerberos配置(可选) 修改在/usr/local/oushu/conf/zookeeper/下的配置文件zoo.cfg，添加一下信息： ```shell authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider jaasLoginRenew=3600000 ``` 并在/usr/local/oushu/conf/zookeeper/目录下分别创建server-jaas.conf文件和client-jaas.conf文件并分别配置如下参数： vim /usr/local/oushu/conf/zookeeper/server-jaas.conf ```shell Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/zookeeper.keytab" storeKey=true useTicketCache=false principal="zookeeper/zookeeper1@OUSHU.COM"; }; ``` vim /usr/local/oushu/conf/zookeeper/client-jaas.conf ```shell Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/zookeeper.keytab" storeKey=true useTicketCache=false principal="zookeeper/zookeeper1@OUSHU.COM"; }; ``` 在/usr/local/oushu/conf/zookeeper/java.env文件，添加以下内容: ```shell export JVMFLAGS="-Djava.security.auth.login.config=/usr/local/oushu/conf/zookeeper/server-jaas.conf" ``` 当zkClient单独部署的时候可以如下使用，同样适用于使用了zk的客户端都其他组件。 如hive开启kerberos连接zk时，需要在hive-env.sh中添加，后重启hive。 ```shell export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/usr/local/oushu/conf/zookeeper/client-jaas.conf" ``` ## 启动 **同步配置文件后**启动ZooKeeper ``` shell lava scp -r -f ${HOME}/zkhostfile /usr/local/oushu/conf/zookeeper/* =:/usr/local/oushu/conf/zookeeper/ lava ssh -f ${HOME}/zkhostfile -e 'sudo -u zookeeper /usr/local/oushu/zookeeper/bin/zkServer.sh start' ``` ## 检查 查看所有ZooKeeper状态 ``` sh lava ssh -f ${HOME}/zkhostfile -e 'sudo -u zookeeper /usr/local/oushu/zookeeper/bin/zkServer.sh status' ZooKeeper JMX enabled by default Using config:/usr/local/oushu/zookeeper/../conf/zookeeper/zoo.cfg Client port found: 2181. Client address: zookeeper1. Mode:leader ZooKeeper JMX enabled by default Using config:/usr/local/oushu/zookeeper/../conf/zookeeper/zoo.cfg Client port found: 2181. Client address: zookeeper2. Mode:follower ZooKeeper JMX enabled by default Using config:/usr/local/oushu/zookeeper/../conf/zookeeper/zoo.cfg Client port found: 2181. Client address: zookeeper3. Mode:follower ``` ## 常用命令 Zookeeper停止 ```shell lava ssh -f ${HOME}/zkhostfile -e 'sudo -u zookeeper /usr/local/oushu/zookeeper/bin/zkServer.sh stop' ``` 进入客户端 ```shell sudo -u zookeeper /usr/local/oushu/zookeeper/bin/zkCli.sh [zk: 127.0.0.1:2181(CONNECTED) 0] ``` ## 注册到Skylab(可选) Kerberos将要安装的机器需要通过机器管理添加到skylab中，如果您尚未添加，请参考[注册机器](../start/install-lava.md)。 在zookeeper1上修改/usr/local/oushu/lava/conf配置`server.json`，替换localhost为skylab的服务器ip，具体skylab的基础服务lava安装步骤请参考：[lava安装](../start/start-installation.rst)。 然后创建`~/zookeeper.json`文件，文件内容参考如下: ```json { "data": { "name": "ZooKeeperCluster", "group_roles": [ { "role": "zk.worker", "cluster_name": "ZooKeeper-id", "group_name": "ZooKeeper", "machines": [ { "id": 1, "name": "zookeeper1", "subnet": "lava", "data_ip": "192.168.1.11", "manage_ip": "", "assist_port": 1622, "ssh_port": 22 }, { "id": 2, "name": "zookeeper2", "subnet": "lava", "data_ip": "192.168.1.12", "manage_ip": "", "assist_port": 1622, "ssh_port": 22 },{ "id": 3, "name": "zookeeper3", "subnet": "lava", "data_ip": "192.168.1.13", "manage_ip": "", "assist_port": 1622, "ssh_port": 22 } ] } ] } } ``` 上述配置文件中，需要根据实际情况修改machines数组中的机器信息，在平台基础组件lava所安装的机器执行： ``` psql lavaadmin -p 4432 -U oushu -c "select m.id,m.name,s.name as subnet,m.private_ip as data_ip,m.public_ip as manage_ip,m.assist_port,m.ssh_port from machine as m,subnet as s where m.subnet_id=s.id;" ``` 获取到所需的机器信息，根据服务角色对应的节点，将机器信息添加到machines数组中。 例如zookeeper1对应的ZooKeeper Worker角色，zookeeper1的机器信息需要备添加到zookeeper.worker角色对应的machines数组中。 调用lava命令注册集群： ``` lava login -u oushu -p ******** -T {租户id} lava onprem-register service -s ZooKeeper -f ~/zookeeper.json ``` 如果返回值为： ``` Add service by self success ``` 则表示注册成功，如果有错误信息，请根据错误信息处理。 从页面登录后，在自动部署模块对应服务中可以查看到新添加的集群，同时列表中会实时监控ZooKeeper进程在机器上的状态。 ![](./images/skylabui.png)