小工具知识网

当前位置: 首页 > 行业知识 > 【Ambari】HDP单机自动化安装(基础环境和MySQL脚本一键安装)

行业知识

【Ambari】HDP单机自动化安装(基础环境和MySQL脚本一键安装)

2026-11-18 06:22:00 来源:互联网转载

个人主页——开���拖拉机回家_Linux,大数据运维-CSDN博客 ✨

感谢点赞和关注 ,每天进步一点点!加油!

目录

一、版本信息

1.1大数据组件版本

1.2Apache Components

1.3Databases支持版本

二、安装包上传

三 、服务器基础环境配置

3.1配置修改

3.2服务器环境配置

3.3MySQL数据库安装

四、安装Ambari-server

4.1 安装ambari-server

4.2 检查REPO源

五、HDP 安装

Get Started

Select Version

Install Options

Confirm Hosts

Choose Services

Assign Masters

Assign Slaves and Clients

Customize Services

CREDENTIALS

DATABASES

DIRECTORIES

ALL CONFIGURATIONS

Review

六、开启Kerberos

6.1 kerberos服务检查

6.2 Ambari启动kerberos

Get Started

Configure Kerberos

Install and Test Kerberos Client

Configure Identities

Confirm Configuration

Stop Services

Kerberize Cluster

Start and Test Services

七、脚本附录说明


一、版本信息


1.1大数据组件版本


组件

版本

操作系统

CentOS7.2-7.9

ambari

2.7.4

HDP

3.3.1.0

HDP-GPL

3.3.1.0

HDP-UTILS

1.1.0.22

JDK

jdk-8u162-linux-x64.tar.gz

MySQL

5.7

1.2Apache Components


组件名称

Apache版本

Apache Ambari

2.7.4

Apache Zookeeper

3.4.6

Apache Hadoop

3.1.1

Apache Hive

3.1.0

Apache HBase

2.0.2

Apache Ranger

1.2.0.3.1

Apache Spark 2

2.3.0

Apache TEZ

0.9.1

1.3Databases支持版本


Name

Version

PostgreSQL

10.7 10.5 10.2 9.6

MySQL

5.7

MariaDB

10.2


二、安装包上传


使用工具将安装包上传到Linux服务器,如上传到/opt 目录,如下命令进入/opt目录并解压安装包,包名日期可能有所变动。

cd /opttar -zxvf windp-aio-deploy-build20231114.tar.gzcd  /opt/windp-aio-deploy

安装包解压后目录结构:

windp-aio-deploy-buildxxxxxxxx.tar.gz 包内容说明

包、脚本和目录

说明

HDP-WinDP-AIO-Platform-final.tar.gz

Ambari安装包

jdk-8u162-linux-x64.tar.gz

JDK安装包

mysql-5.7.44-el7-x86_64.tar.gz

MySQL安装包

install_base.sh

基础环境配置和初始化

install_mysql.sh

MySQL安装

install_repo.sh

HDP源配置和Ambari安装

config

配置文件

repo

HDP-repo源文件


三 、服务器基础环境配置


3.1配置修改


确认是否/opt为安装目录,如果不是则要修改路径,root 用户密码修改,需要修改部分已标红,修改完成后在Linux服务器命令行执行如下命令:

cat > /opt/windp-aio-deploy/config/global.sh  /etc/hosts  /etc/rc.local  /sys/kernel/mm/transparent_hugepage/defragfiif test -f /sys/kernel/mm/transparent_hugepage/enabled; thenecho never > /sys/kernel/mm/transparent_hugepage/enabledfiEOF## chronysudo yum install chrony -ysudo systemctl restart  chronydsudo systemctl enable chronydsudo systemctl status chronyd                # httpd  sudo yum -y install httpdsudo systemctl restart httpd.servicesudo systemctl enable httpd.service sudo systemctl status httpd.service ## kerberos clientsudo yum install -y krb5-workstation krb5-libs}######################### 生成本地ssh公钥########################create_ssh_pub(){echo "生成本地ssh公钥"/usr/bin/expect > /etc/profile     echo 'export CLASSPATH=$JAVA_HOME/lib/' >> /etc/profile    echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile        fisource /etc/profilejava -version        echo "********** JDK installation completed **********"}######################### 配置kerberos,安装启动########################config_krb5() {        # kerberos server and client        sudo yum install krb5-server krb5-libs krb5-workstation -y        # config file        cat $BASEDIR/config/krb5.conf > /etc/krb5.confcat $BASEDIR/config/kdc.conf  > /var/kerberos/krb5kdc/kdc.confcat $BASEDIR/config/kadm5.acl > /var/kerberos/krb5kdc/kadm5.acl                echo "******* 创建kdc数据库 *********"        /usr/bin/expect   $stack_path/3.0/services/DRUID/metainfo.xmlcat $BASEDIR/repo/services/LOGSEARCH/metainfo.xml   >  $stack_path/3.0/services/LOGSEARCH/metainfo.xmlcat $BASEDIR/repo/services/SUPERSET/metainfo.xml    >  $stack_path/3.0/services/SUPERSET/metainfo.xmlcat $BASEDIR/repo/services/ATLAS/metainfo.xml       >  $stack_path/3.1/services/ATLAS/metainfo.xmlcat $BASEDIR/repo/services/ZEPPELIN/metainfo.xml    >  $stack_path/3.0/services/ZEPPELIN/metainfo.xml      cat $BASEDIR/repo/services/STORM/metainfo.xml       >  $stack_path/3.0/services/STORM/metainfo.xmlcat $BASEDIR/repo/services/RANGER_KMS/metainfo.xml  >  $stack_path/3.1/services/RANGER_KMS/metainfo.xmlcat $BASEDIR/repo/services/OOZIE/metainfo.xml       >  $stack_path/3.0/services/OOZIE/metainfo.xmlcat $BASEDIR/repo/services/KNOX/metainfo.xml        >  $stack_path/3.1/services/KNOX/metainfo.xmlcat $BASEDIR/repo/services/SQOOP/metainfo.xml       >  $stack_path/3.0/services/SQOOP/metainfo.xmlcat $BASEDIR/repo/services/SMARTSENSE/metainfo.xml  >  $stack_path/3.0/services/SMARTSENSE/metainfo.xml        ambari-server restart}config_repoconfig_dbinstall_ambariconfig_metainfo_modify

cat repo/services/ACCUMULO/metainfo.xml

我们把这些组件选择隐藏了,默认不安装,不会出现在组件安装的选择项里面。

  2.0            ACCUMULO      true            

global.sh

########################部署相关全局参数定义######################## root 密码保存root_passwd=winner@001# mysql配置myurl=127.0.0.1myuser=rootmypwd=Winner001myport=3306mydb=ipvacloud# HDP包解压目录install_path=/hadoophdp_tar_name=HDP-WinDP-AIO-Platform-final.tar.gz# mysql 安装目录mysql_install_path=/usr/local

init_db.sql

use mysql;GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Winner001' WITH GRANT OPTION;create database ambari character set utf8;CREATE USER 'ambari'@'%' IDENTIFIED BY 'Winner001';GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';FLUSH PRIVILEGES;create database hive character set utf8;CREATE USER 'hive'@'%' IDENTIFIED BY 'Winner001';GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';FLUSH PRIVILEGES;create database ranger character set utf8;CREATE USER 'ranger'@'%' IDENTIFIED BY 'Winner001';GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'%';FLUSH PRIVILEGES;

ambari.properties,ambari配置文件需要修改数据库连接和JDK路径

## Copyright 2011 The Apache Software Foundation## Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.##Sat Nov 04 12:10:44 CST 2023agent.package.install.task.timeout=1800agent.stack.retry.on_repo_unavailability=falseagent.stack.retry.tries=5agent.task.timeout=900agent.threadpool.size.max=25ambari-server.user=rootambari.python.wrap=ambari-python-wrapbootstrap.dir=/var/run/ambari-server/bootstrapbootstrap.script=/usr/lib/ambari-server/lib/ambari_server/bootstrap.pybootstrap.setup_agent.script=/usr/lib/ambari-server/lib/ambari_server/setupAgent.pycheck_database_skipped=falseclient.threadpool.size.max=25common.services.path=/var/lib/ambari-server/resources/common-servicescustom.action.definitions=/var/lib/ambari-server/resources/custom_action_definitionscustom.mysql.jdbc.name=mysql-connector-java.jarextensions.path=/var/lib/ambari-server/resources/extensionsgpl.license.accepted=truehttp.cache-control=no-storehttp.charset=utf-8http.pragma=no-cachehttp.strict-transport-security=max-age=31536000http.x-content-type-options=nosniffhttp.x-frame-options=DENYhttp.x-xss-protection=1; mode=blockjava.home=/usr/java/jdk1.8.0_162java.releases=jdk1.8java.releases.ppc64le=jce.download.supported=truejdk.download.supported=truejdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8jdk1.8.dest-file=jdk-8u112-linux-x64.tar.gzjdk1.8.home=/usr/jdk64/jdk1.8.jcpol-file=jce_policy-8.zipjdk1.8.jcpol-url=http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zipjdk1.8.re=(jdk.*)/jrejdk1.8.url=http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gzkerberos.keytab.cache.dir=/var/lib/ambari-server/data/cachekerberos.operation.verify.kdc.trust=truemetadata.path=/var/lib/ambari-server/resources/stacksmpacks.staging.path=/var/lib/ambari-server/resources/mpackspid.dir=/var/run/ambari-serverprevious.custom.mysql.jdbc.name=mysql-connector-java.jarrecommendations.artifacts.lifetime=1wrecommendations.dir=/var/run/ambari-server/stack-recommendationsresources.dir=/var/lib/ambari-server/resourcesrolling.upgrade.skip.packages.prefixes=security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5security.server.keys_dir=/var/lib/ambari-server/keysserver.connection.max.idle.millis=900000server.execution.scheduler.isClustered=falseserver.execution.scheduler.maxDbConnections=5server.execution.scheduler.maxThreads=5server.execution.scheduler.misfire.toleration.minutes=480server.fqdn.service.url=http://169.254.169.254/latest/meta-data/public-hostnameserver.http.session.inactive_timeout=1800server.jdbc.connection-pool=c3p0server.jdbc.connection-pool.acquisition-size=5server.jdbc.connection-pool.idle-test-interval=7200server.jdbc.connection-pool.max-age=0server.jdbc.connection-pool.max-idle-time=14400server.jdbc.connection-pool.max-idle-time-excess=0server.jdbc.database=mysqlserver.jdbc.database_name=ambariserver.jdbc.driver=com.mysql.jdbc.Driverserver.jdbc.driver.path=/usr/share/java/mysql-connector-java.jarserver.jdbc.hostname=localhostserver.jdbc.port=3306server.jdbc.rca.driver=com.mysql.jdbc.Driverserver.jdbc.rca.url=jdbc:mysql://localhost:3306/ambariserver.jdbc.rca.user.name=ambariserver.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.datserver.jdbc.url=jdbc:mysql://localhost:3306/ambariserver.jdbc.user.name=ambariserver.jdbc.user.passwd=/etc/ambari-server/conf/password.datserver.os_family=redhat7server.os_type=centos7server.persistence.type=remoteserver.python.log.level=INFOserver.python.log.name=ambari-server-command.logserver.stages.parallel=trueserver.task.timeout=1200server.tmp.dir=/var/lib/ambari-server/data/tmpserver.version.file=/var/lib/ambari-server/resources/versionshared.resources.dir=/usr/lib/ambari-server/lib/ambari_commons/resourcesskip.service.checks=falsestack.java.home=/usr/java/jdk1.8.0_162stackadvisor.script=/var/lib/ambari-server/resources/scripts/stack_advisor.pyulimit.open.files=65536upgrade.parameter.convert.hive.tables.timeout=86400upgrade.parameter.move.hive.tables.timeout=86400user.inactivity.timeout.default=0user.inactivity.timeout.role.readonly.default=0views.ambari.request.connect.timeout.millis=30000views.ambari.request.read.timeout.millis=45000views.http.cache-control=no-storeviews.http.charset=utf-8views.http.pragma=no-cacheviews.http.strict-transport-security=max-age=31536000views.http.x-content-type-options=nosniffviews.http.x-frame-options=SAMEORIGINviews.http.x-xss-protection=1; mode=blockviews.request.connect.timeout.millis=5000views.request.read.timeout.millis=10000views.skip.home-directory-check.file-system.list=wasb,adls,adlwebapp.dir=/usr/lib/ambari-server/web

password.dat(数据库密码)

Winner001

my.cnf

[mysql]socket=/var/lib/mysql/mysql.sock# set mysql client default chararterdefault-character-set=utf8[mysqld]socket=/var/lib/mysql/mysql.sock# set mysql server port  port = 3306# set mysql install base dirbasedir=/usr/local/mysql# set the data store dirdatadir=/usr/local/mysql/data# set the number of allow max connnectionmax_connections=200# set server charactre default encodingcharacter-set-server=utf8# the storage enginedefault-storage-engine=INNODBlower_case_table_names=1max_allowed_packet=16Mexplicit_defaults_for_timestamp=true[mysql.server]user=mysqlbasedir=/usr/local/mysql

krb5.conf (kerberos配置)

[libdefaults]  renew_lifetime = 7d  forwardable = true  default_realm = WINNER.COM  ticket_lifetime = 24h  dns_lookup_realm = false  dns_lookup_kdc = false  default_ccache_name = /tmp/krb5cc_%{uid}  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5[logging]  default = FILE:/var/log/krb5kdc.log  admin_server = FILE:/var/log/kadmind.log  kdc = FILE:/var/log/krb5kdc.log[realms]  WINNER.COM = {    admin_server = windp-aio    kdc = windp-aio  }

repo源

ambari.repo

#VERSION_NUMBER=2.7.4.0-118[ambari-2.7.4.0]#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.jsonname=ambari Version - ambari-2.7.4.0baseurl=http://windp-aio/ambari/centos7/2.7.4.0-118/gpgcheck=1gpgkey=http://windp-aio/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1

hdp.repo

#VERSION_NUMBER=3.1.4.0-315[HDP-3.1.4.0]name=HDP Version - HDP-3.1.4.0baseurl=http://windp-aio/HDP/centos7/3.1.4.0-315gpgcheck=1gpgkey=http://windp-aio/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1[HDP-UTILS-1.1.0.22]name=HDP-UTILS Version - HDP-UTILS-1.1.0.22baseurl=http://windp-aio/HDP-UTILS/centos7/1.1.0.22gpgcheck=1gpgkey=http://windp-aio/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1

hdp.gpl.repo

#VERSION_NUMBER=3.1.4.0-315[HDP-GPL-3.1.4.0]name=HDP-GPL Version - HDP-GPL-3.1.4.0baseurl=http://windp-aio/HDP-GPL/centos7/3.1.4.0-315gpgcheck=1gpgkey=http://windp-aio/HDP-GPL/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1

总结:如上是单机测试环境,我们可以将集群扩展到多台 完成集群安装。

上一篇:罗大背后的故事揭秘,引人深思的真相!

下一篇:黄墅湖服务器是哪里?