行业知识
【Ambari】HDP单机自动化安装(基础环境和MySQL脚本一键安装)
个人主页——开���拖拉机回家_Linux,大数据运维-CSDN博客 ✨
感谢点赞和关注 ,每天进步一点点!加油!
目录
一、版本信息
1.1大数据组件版本
1.2Apache Components
1.3Databases支持版本
二、安装包上传
三 、服务器基础环境配置
3.1配置修改
3.2服务器环境配置
3.3MySQL数据库安装
四、安装Ambari-server
4.1 安装ambari-server
4.2 检查REPO源
五、HDP 安装
Get Started
Select Version
Install Options
Confirm Hosts
Choose Services
Assign Masters
Assign Slaves and Clients
Customize Services
CREDENTIALS
DATABASES
DIRECTORIES
ALL CONFIGURATIONS
Review
六、开启Kerberos
6.1 kerberos服务检查
6.2 Ambari启动kerberos
Get Started
Configure Kerberos
Install and Test Kerberos Client
Configure Identities
Confirm Configuration
Stop Services
Kerberize Cluster
Start and Test Services
七、脚本附录说明
一、版本信息
1.1大数据组件版本
| 组件 | 版本 |
| 操作系统 | CentOS7.2-7.9 |
| ambari | 2.7.4 |
| HDP | 3.3.1.0 |
| HDP-GPL | 3.3.1.0 |
| HDP-UTILS | 1.1.0.22 |
| JDK | jdk-8u162-linux-x64.tar.gz |
| MySQL | 5.7 |
1.2Apache Components
| 组件名称 | Apache版本 |
| Apache Ambari | 2.7.4 |
| Apache Zookeeper | 3.4.6 |
| Apache Hadoop | 3.1.1 |
| Apache Hive | 3.1.0 |
| Apache HBase | 2.0.2 |
| Apache Ranger | 1.2.0.3.1 |
| Apache Spark 2 | 2.3.0 |
| Apache TEZ | 0.9.1 |
1.3Databases支持版本
| Name | Version |
| PostgreSQL | 10.7 10.5 10.2 9.6 |
| MySQL | 5.7 |
| MariaDB | 10.2 |
二、安装包上传
使用工具将安装包上传到Linux服务器,如上传到/opt 目录,如下命令进入/opt目录并解压安装包,包名日期可能有所变动。
cd /opttar -zxvf windp-aio-deploy-build20231114.tar.gzcd /opt/windp-aio-deploy
安装包解压后目录结构:
windp-aio-deploy-buildxxxxxxxx.tar.gz 包内容说明
| 包、脚本和目录 | 说明 |
| HDP-WinDP-AIO-Platform-final.tar.gz | Ambari安装包 |
| jdk-8u162-linux-x64.tar.gz | JDK安装包 |
| mysql-5.7.44-el7-x86_64.tar.gz | MySQL安装包 |
| install_base.sh | 基础环境配置和初始化 |
| install_mysql.sh | MySQL安装 |
| install_repo.sh | HDP源配置和Ambari安装 |
| config | 配置文件 |
| repo | HDP-repo源文件 |
三 、服务器基础环境配置
3.1配置修改
确认是否/opt为安装目录,如果不是则要修改路径,root 用户密码修改,需要修改部分已标红,修改完成后在Linux服务器命令行执行如下命令:
cat > /opt/windp-aio-deploy/config/global.sh /etc/hosts /etc/rc.local /sys/kernel/mm/transparent_hugepage/defragfiif test -f /sys/kernel/mm/transparent_hugepage/enabled; thenecho never > /sys/kernel/mm/transparent_hugepage/enabledfiEOF## chronysudo yum install chrony -ysudo systemctl restart chronydsudo systemctl enable chronydsudo systemctl status chronyd # httpd sudo yum -y install httpdsudo systemctl restart httpd.servicesudo systemctl enable httpd.service sudo systemctl status httpd.service ## kerberos clientsudo yum install -y krb5-workstation krb5-libs}######################### 生成本地ssh公钥########################create_ssh_pub(){echo "生成本地ssh公钥"/usr/bin/expect > /etc/profile echo 'export CLASSPATH=$JAVA_HOME/lib/' >> /etc/profile echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile fisource /etc/profilejava -version echo "********** JDK installation completed **********"}######################### 配置kerberos,安装启动########################config_krb5() { # kerberos server and client sudo yum install krb5-server krb5-libs krb5-workstation -y # config file cat $BASEDIR/config/krb5.conf > /etc/krb5.confcat $BASEDIR/config/kdc.conf > /var/kerberos/krb5kdc/kdc.confcat $BASEDIR/config/kadm5.acl > /var/kerberos/krb5kdc/kadm5.acl echo "******* 创建kdc数据库 *********" /usr/bin/expect $stack_path/3.0/services/DRUID/metainfo.xmlcat $BASEDIR/repo/services/LOGSEARCH/metainfo.xml > $stack_path/3.0/services/LOGSEARCH/metainfo.xmlcat $BASEDIR/repo/services/SUPERSET/metainfo.xml > $stack_path/3.0/services/SUPERSET/metainfo.xmlcat $BASEDIR/repo/services/ATLAS/metainfo.xml > $stack_path/3.1/services/ATLAS/metainfo.xmlcat $BASEDIR/repo/services/ZEPPELIN/metainfo.xml > $stack_path/3.0/services/ZEPPELIN/metainfo.xml cat $BASEDIR/repo/services/STORM/metainfo.xml > $stack_path/3.0/services/STORM/metainfo.xmlcat $BASEDIR/repo/services/RANGER_KMS/metainfo.xml > $stack_path/3.1/services/RANGER_KMS/metainfo.xmlcat $BASEDIR/repo/services/OOZIE/metainfo.xml > $stack_path/3.0/services/OOZIE/metainfo.xmlcat $BASEDIR/repo/services/KNOX/metainfo.xml > $stack_path/3.1/services/KNOX/metainfo.xmlcat $BASEDIR/repo/services/SQOOP/metainfo.xml > $stack_path/3.0/services/SQOOP/metainfo.xmlcat $BASEDIR/repo/services/SMARTSENSE/metainfo.xml > $stack_path/3.0/services/SMARTSENSE/metainfo.xml ambari-server restart}config_repoconfig_dbinstall_ambariconfig_metainfo_modify cat repo/services/ACCUMULO/metainfo.xml
我们把这些组件选择隐藏了,默认不安装,不会出现在组件安装的选择项里面。
2.0 ACCUMULO true
global.sh
########################部署相关全局参数定义######################## root 密码保存root_passwd=winner@001# mysql配置myurl=127.0.0.1myuser=rootmypwd=Winner001myport=3306mydb=ipvacloud# HDP包解压目录install_path=/hadoophdp_tar_name=HDP-WinDP-AIO-Platform-final.tar.gz# mysql 安装目录mysql_install_path=/usr/local
init_db.sql
use mysql;GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Winner001' WITH GRANT OPTION;create database ambari character set utf8;CREATE USER 'ambari'@'%' IDENTIFIED BY 'Winner001';GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';FLUSH PRIVILEGES;create database hive character set utf8;CREATE USER 'hive'@'%' IDENTIFIED BY 'Winner001';GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';FLUSH PRIVILEGES;create database ranger character set utf8;CREATE USER 'ranger'@'%' IDENTIFIED BY 'Winner001';GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'%';FLUSH PRIVILEGES;
ambari.properties,ambari配置文件需要修改数据库连接和JDK路径
## Copyright 2011 The Apache Software Foundation## Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements. See the NOTICE file# distributed with this work for additional information# regarding copyright ownership. The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.##Sat Nov 04 12:10:44 CST 2023agent.package.install.task.timeout=1800agent.stack.retry.on_repo_unavailability=falseagent.stack.retry.tries=5agent.task.timeout=900agent.threadpool.size.max=25ambari-server.user=rootambari.python.wrap=ambari-python-wrapbootstrap.dir=/var/run/ambari-server/bootstrapbootstrap.script=/usr/lib/ambari-server/lib/ambari_server/bootstrap.pybootstrap.setup_agent.script=/usr/lib/ambari-server/lib/ambari_server/setupAgent.pycheck_database_skipped=falseclient.threadpool.size.max=25common.services.path=/var/lib/ambari-server/resources/common-servicescustom.action.definitions=/var/lib/ambari-server/resources/custom_action_definitionscustom.mysql.jdbc.name=mysql-connector-java.jarextensions.path=/var/lib/ambari-server/resources/extensionsgpl.license.accepted=truehttp.cache-control=no-storehttp.charset=utf-8http.pragma=no-cachehttp.strict-transport-security=max-age=31536000http.x-content-type-options=nosniffhttp.x-frame-options=DENYhttp.x-xss-protection=1; mode=blockjava.home=/usr/java/jdk1.8.0_162java.releases=jdk1.8java.releases.ppc64le=jce.download.supported=truejdk.download.supported=truejdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8jdk1.8.dest-file=jdk-8u112-linux-x64.tar.gzjdk1.8.home=/usr/jdk64/jdk1.8.jcpol-file=jce_policy-8.zipjdk1.8.jcpol-url=http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zipjdk1.8.re=(jdk.*)/jrejdk1.8.url=http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gzkerberos.keytab.cache.dir=/var/lib/ambari-server/data/cachekerberos.operation.verify.kdc.trust=truemetadata.path=/var/lib/ambari-server/resources/stacksmpacks.staging.path=/var/lib/ambari-server/resources/mpackspid.dir=/var/run/ambari-serverprevious.custom.mysql.jdbc.name=mysql-connector-java.jarrecommendations.artifacts.lifetime=1wrecommendations.dir=/var/run/ambari-server/stack-recommendationsresources.dir=/var/lib/ambari-server/resourcesrolling.upgrade.skip.packages.prefixes=security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5security.server.keys_dir=/var/lib/ambari-server/keysserver.connection.max.idle.millis=900000server.execution.scheduler.isClustered=falseserver.execution.scheduler.maxDbConnections=5server.execution.scheduler.maxThreads=5server.execution.scheduler.misfire.toleration.minutes=480server.fqdn.service.url=http://169.254.169.254/latest/meta-data/public-hostnameserver.http.session.inactive_timeout=1800server.jdbc.connection-pool=c3p0server.jdbc.connection-pool.acquisition-size=5server.jdbc.connection-pool.idle-test-interval=7200server.jdbc.connection-pool.max-age=0server.jdbc.connection-pool.max-idle-time=14400server.jdbc.connection-pool.max-idle-time-excess=0server.jdbc.database=mysqlserver.jdbc.database_name=ambariserver.jdbc.driver=com.mysql.jdbc.Driverserver.jdbc.driver.path=/usr/share/java/mysql-connector-java.jarserver.jdbc.hostname=localhostserver.jdbc.port=3306server.jdbc.rca.driver=com.mysql.jdbc.Driverserver.jdbc.rca.url=jdbc:mysql://localhost:3306/ambariserver.jdbc.rca.user.name=ambariserver.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.datserver.jdbc.url=jdbc:mysql://localhost:3306/ambariserver.jdbc.user.name=ambariserver.jdbc.user.passwd=/etc/ambari-server/conf/password.datserver.os_family=redhat7server.os_type=centos7server.persistence.type=remoteserver.python.log.level=INFOserver.python.log.name=ambari-server-command.logserver.stages.parallel=trueserver.task.timeout=1200server.tmp.dir=/var/lib/ambari-server/data/tmpserver.version.file=/var/lib/ambari-server/resources/versionshared.resources.dir=/usr/lib/ambari-server/lib/ambari_commons/resourcesskip.service.checks=falsestack.java.home=/usr/java/jdk1.8.0_162stackadvisor.script=/var/lib/ambari-server/resources/scripts/stack_advisor.pyulimit.open.files=65536upgrade.parameter.convert.hive.tables.timeout=86400upgrade.parameter.move.hive.tables.timeout=86400user.inactivity.timeout.default=0user.inactivity.timeout.role.readonly.default=0views.ambari.request.connect.timeout.millis=30000views.ambari.request.read.timeout.millis=45000views.http.cache-control=no-storeviews.http.charset=utf-8views.http.pragma=no-cacheviews.http.strict-transport-security=max-age=31536000views.http.x-content-type-options=nosniffviews.http.x-frame-options=SAMEORIGINviews.http.x-xss-protection=1; mode=blockviews.request.connect.timeout.millis=5000views.request.read.timeout.millis=10000views.skip.home-directory-check.file-system.list=wasb,adls,adlwebapp.dir=/usr/lib/ambari-server/web
password.dat(数据库密码)
Winner001
my.cnf
[mysql]socket=/var/lib/mysql/mysql.sock# set mysql client default chararterdefault-character-set=utf8[mysqld]socket=/var/lib/mysql/mysql.sock# set mysql server port port = 3306# set mysql install base dirbasedir=/usr/local/mysql# set the data store dirdatadir=/usr/local/mysql/data# set the number of allow max connnectionmax_connections=200# set server charactre default encodingcharacter-set-server=utf8# the storage enginedefault-storage-engine=INNODBlower_case_table_names=1max_allowed_packet=16Mexplicit_defaults_for_timestamp=true[mysql.server]user=mysqlbasedir=/usr/local/mysql
krb5.conf (kerberos配置)
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = WINNER.COM ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5[logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log[realms] WINNER.COM = { admin_server = windp-aio kdc = windp-aio } repo源
ambari.repo
#VERSION_NUMBER=2.7.4.0-118[ambari-2.7.4.0]#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.jsonname=ambari Version - ambari-2.7.4.0baseurl=http://windp-aio/ambari/centos7/2.7.4.0-118/gpgcheck=1gpgkey=http://windp-aio/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1
hdp.repo
#VERSION_NUMBER=3.1.4.0-315[HDP-3.1.4.0]name=HDP Version - HDP-3.1.4.0baseurl=http://windp-aio/HDP/centos7/3.1.4.0-315gpgcheck=1gpgkey=http://windp-aio/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1[HDP-UTILS-1.1.0.22]name=HDP-UTILS Version - HDP-UTILS-1.1.0.22baseurl=http://windp-aio/HDP-UTILS/centos7/1.1.0.22gpgcheck=1gpgkey=http://windp-aio/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1
hdp.gpl.repo
#VERSION_NUMBER=3.1.4.0-315[HDP-GPL-3.1.4.0]name=HDP-GPL Version - HDP-GPL-3.1.4.0baseurl=http://windp-aio/HDP-GPL/centos7/3.1.4.0-315gpgcheck=1gpgkey=http://windp-aio/HDP-GPL/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1
总结:如上是单机测试环境,我们可以将集群扩展到多台 完成集群安装。
下一篇:黄墅湖服务器是哪里?