目录

Hadoop部署

一、配置core-site.xml。代理主机

二、HDFS配置文件

三、YARN配置

四、MapReduce配置文件

五、配置workers

配置历史服务器

一、配置mapred-site.xml

配置日志的聚集

Hadoop部署

一、配置core-site.xml。代理主机

[doudou@hadoop101 module]$ cd hadoop-3.1.3/

[doudou@hadoop101 hadoop-3.1.3]$ cd etc/hadoop/

[doudou@hadoop101 hadoop]$ cd core-site.xml

[doudou@hadoop101 hadoop]$ vim /core-site.xml

         fs.defaultFS     hdfs://hadoop102:8020         hadoop.tmp.dir     /opt/module/hadoop-3.1.3/data         hadoop.http.staticuser.user     doudou           hadoop.proxyuser.doudou.hosts       *         hadoop.proxyuser.doudou.grups     *           hadoop.proxyuser.doudou.users         *  

二、HDFS配置文件

配置hdfs-site.xml

[doudou@hadoop101 hadoop]$ vim hdfs-site.xml 

     dfs.namenode.http-address   hadoop101:9870        dfs.namenode.secondary.http-address    hadoop103:9868         dfs.replication     1  ~                         

分发

[doudou@hadoop101 hadoop]$ xsync hdfs-site.xml 

三、YARN配置

配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml

               yarn.nodemanager.aux-services        mapreduce_shuffle                    yarn.resourcemanager.hostname        hadoop103                    yarn.nodemanager.env-whitelist        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME              yarn.scheduler.minimum-allocation-mb       512             yarn.scheduler.maximum-allocation-mb       4096              yarn.nodemanager.resource.memory-mb        4096            yarn.nodemanager.vmem-check-enabled      false   

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml

四、MapReduce配置文件

配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml 

            mapreduce.framework.name        yarn    

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml 

五、配置workers

[doudou@hadoop101 ~]$ vim /opt/module/hadoop-3.1.3/etc/hadoop/workers

hadoop101hadoop102hadoop103

[doudou@hadoop101 hadoop]$ xsync workers 

配置历史服务器

一、配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml 

    mapreduce.jobhistory.webapp.address    hadoop101:19888>mapreduce.framework.name        yarn    

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml 

配置日志的聚集

一、配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml 

    yarn.log-aggregation-enable    true      yarn.log.server.url      http://hadoop101:19888/jobhistory/logs    yarn.log-aggregation.retain-seconds    604800

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml