Windows下 IDEA編譯調(diào)試 hive2.3.9的過程解析

更新時間：2023年07月12日 10:03:26 作者：顧棟

這篇文章主要介紹了Windows下 IDEA編譯調(diào)試 hive2.3.9的過程,本文給大家介紹的非常詳細(xì)，對大家的學(xué)習(xí)或工作具有一定的參考借鑒價值，需要的朋友可以參考下

Windows下 IDEA編譯調(diào)試 hive2.3.9

環(huán)境

IDEA 2021.2
JDK1.8（試過用高版本的JDK17編譯，不兼容編譯不過）
一個Hadoop集群，涉及配置文件core-site.xml，hdfs-site.xml，yarn-site.xml，mapred-site.xml
Windows環(huán)境中配置hadoop客戶端。
git

源碼編譯

在Git Bash中執(zhí)行編譯命令，hive2版本的編譯命令 mvn clean package -DskipTests -Pdist。

編譯結(jié)果圖

在這里插入圖片描述

編譯問題

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (generate-version-annotation) on project hive-standalone-metastore-common: An Ant BuildException has occured: exec returned: 127
[ERROR] around Ant part ...<exec failonerror="true" executable="bash">... @ 4:46 in F:\subsys\mystudy\hive-2\hive\standalone-metastore\metastore-common\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hive-standalone-metastore-common

編譯命令需要在shell環(huán)境下執(zhí)行，，所以采用git bash環(huán)境編譯。

導(dǎo)入IDEA

用IDEA打開項(xiàng)目

問題

Could not find artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT

原因是：丟失了依賴解決方案：查看社區(qū)，發(fā)現(xiàn)存在一個issue。issue號是HIVE-21777。在hive service 模塊下的pom.xml文件排掉這個依賴項(xiàng)

        <dependency>
            <groupId>org.apache.directory.server</groupId>
            <artifactId>apacheds-server-integ</artifactId>
            <version>${apache-directory-server.version}</version>
            <scope>test</scope>
            <!-- 增加的部分-->
            <exclusions>
                <exclusion>
                    <groupId>org.apache.directory.client.ldap</groupId>
                    <artifactId>ldap-client-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

啟動Cli

1.分別把集群的配置文件copy到hive-cli模塊的resources(需要自己新建和配置)下core-site.xml，hdfs-site.xml，yarn-site.xml，mapred-site.xml，新建hive-log4j2.properties和hive-site.xml文件。

2.變更配置項(xiàng)

mapred-site.xml中新增，

    <!-- 支持跨平臺提交任務(wù) -->
    <property>
        <name>mapreduce.app-submission.cross-platform</name>
        <value>true</value>
    </property>

hive-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" ?>
<configuration>
    <property>
        <name>hive.cli.print.current.db</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.cli.print.header</name>
        <value>true</value>
    </property>
    <!-- hive 元數(shù)據(jù)鏈接地址 HA-->
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://192.168.1.1:9083,thrift://192.168.1.2:9083</value>
        <final>true</final>
    </property>
    <property>
        <name>hive.cluster.delegation.token.store.class</name>
        <value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value>
        <description>Hive defaults to MemoryTokenStore, or ZooKeeperTokenStore</description>
    </property>
    <!-- 測試環(huán)境的 hive客戶端本地模式-->
    <property>
        <name>hive.exec.mode.local.auto</name>
        <value>true</value>
    </property>
    <!--輸入最大數(shù)據(jù)量，默認(rèn)128MB-->
    <property>
        <name>hive.exec.mode.local.auto.inputbytes.max</name>
        <value>134217728</value>
    </property>
    <!--    最大任務(wù)數(shù)-->
    <property>
        <name>hive.exec.mode.local.auto.input.files.max</name>
        <value>10</value>
    </property>
</configuration>

hive-log4j2.properties變更

property.hive.log.dir = /opt/hive/logs
property.hive.log.file = hive-${sys:user.name}.log

3.修改hive-cli模塊中的pom文件，將commons-io，和disruptor范圍取消test限制。

4.配置CliDriver啟動，JVM參數(shù)-Djline.WindowsTerminal.directConsole=false,環(huán)境配置,HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native;HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib

在這里插入圖片描述

問題：import org.apache.hive.tmpl.QueryProfileTmpl 缺失

解決在git bash執(zhí)行mvn generate-sources ,然后拷貝hive service模塊目錄target/generated-jamon/org/apache/hive/tmpl生成的源文件至指定目錄（需要手動創(chuàng)建）,QueryProfileTmpl和QueryProfileTmplImpl。

本地模式下進(jìn)行基礎(chǔ)操作

建庫

create database testdb;

建表

create table test001(id int,dummy string) stored as orc;

插入數(shù)據(jù)

use testdb;
insert into table testdb.test001 values(1,"X");

查詢數(shù)據(jù)

select * from test001;

刪除數(shù)據(jù)，不建議進(jìn)行單行或者多行的條件式刪除，可以進(jìn)行分區(qū)或整表的刪除，重新寫入數(shù)據(jù)的方式。

執(zhí)行記錄：

hive (default)> create database testdb;
create database testdb;
OK
Time taken: 0.072 seconds
hive (default)> use testdb;
use testdb;
OK
Time taken: 0.045 seconds
hive (testdb)> create table test001(id int,dummy string) stored as orc;
create table test001(id int,dummy string) stored as orc;
OK
Time taken: 0.53 seconds
hive (testdb)> insert into table test001 values(1,"X");
insert into table test001 values(1,"X");
Automatically selecting local only mode for query
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = 19057578_20230222162806_f6ce7921-08d3-41db-8408-dfd18339e594
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Job running in-process (local Hadoop)
2023-02-22 16:28:13,266 Stage-1 map = 0%,  reduce = 0%
2023-02-22 16:28:14,295 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_local2039626018_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hadoopdemo/user/bigdata/hive/warehouse/testdb.db/test001/.hive-staging_hive_2023-02-22_16-28-06_738_8793114222801018317-1/-ext-10000
Loading data to table testdb.test001
MapReduce Jobs Launched: 
Stage-Stage-1:  HDFS Read: 4 HDFS Write: 348 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 8.293 seconds
_col0	_col1
hive (testdb)> select * from test001;
select * from test001;
OK
test001.id	test001.dummy
1	X
hive (testdb)> Time taken: 0.241 seconds, Fetched: 1 row(s)
hive (testdb)> drop table test001;
drop table test001;
OK
Time taken: 0.218 seconds
hive (testdb)> show tables;
show tables;
OK
tab_name
values__tmp__table__1
hive (testdb)> Time taken: 0.063 seconds, Fetched: 1 row(s)

values__tmp__table__1為客戶端內(nèi)存中的臨時數(shù)據(jù)，會在關(guān)閉客戶端后刪除，不會落盤。

遺留問題

提交任務(wù)到y(tǒng)arn集群失敗

取消本地模式，在將任務(wù)提交到y(tǒng)arn是會有錯誤。判斷是客戶端環(huán)境配置問題，但是沒有在客戶端處解決。

yarn上的日志

************************************************************/
2023-02-22 16:35:04,263 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
2023-02-22 16:35:04,263 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2023-02-22 16:35:04,263 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1 cluster_timestamp: 1677047289113 } attemptId: 2 } keyId: 176967782)
2023-02-22 16:35:04,365 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2023-02-22 16:35:04,367 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2427)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:545)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:522)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1764)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:522)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:308)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1722)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2395)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2419)
	... 12 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2299)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2393)
	... 13 more
2023-02-22 16:35:04,369 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2427)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:545)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:522)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1764)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:522)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:308)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1722)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2395)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2419)
	... 12 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2299)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2393)
	... 13 more
2023-02-22 16:35:04,372 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found

到此這篇關(guān)于Windows下 IDEA編譯調(diào)試 hive2.3.9的文章就介紹到這了,更多相關(guān)IDEA編譯調(diào)試 hive2.3.9內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！

您可能感興趣的文章: