Windows下 IDEA編譯調(diào)試 hive2.3.9的過(guò)程解析
Windows下 IDEA編譯調(diào)試 hive2.3.9
環(huán)境
- IDEA 2021.2
- JDK1.8(試過(guò)用高版本的JDK17編譯,不兼容編譯不過(guò))
- 一個(gè)Hadoop集群,涉及配置文件core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml
- Windows環(huán)境中配置hadoop客戶端。
- git
源碼編譯
在Git Bash中執(zhí)行編譯命令,hive2版本的編譯命令 mvn clean package -DskipTests -Pdist
。
編譯結(jié)果圖
編譯問(wèn)題
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (generate-version-annotation) on project hive-standalone-metastore-common: An Ant BuildException has occured: exec returned: 127
[ERROR] around Ant part ...<exec failonerror="true" executable="bash">... @ 4:46 in F:\subsys\mystudy\hive-2\hive\standalone-metastore\metastore-common\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hive-standalone-metastore-common
編譯命令需要在shell環(huán)境下執(zhí)行,,所以采用git bash環(huán)境編譯。
導(dǎo)入IDEA
用IDEA打開(kāi)項(xiàng)目
問(wèn)題
Could not find artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT
原因是:丟失了依賴 解決方案:查看社區(qū),發(fā)現(xiàn)存在一個(gè)issue。issue號(hào)是HIVE-21777。在hive service 模塊下的pom.xml文件排掉這個(gè)依賴項(xiàng)
<dependency> <groupId>org.apache.directory.server</groupId> <artifactId>apacheds-server-integ</artifactId> <version>${apache-directory-server.version}</version> <scope>test</scope> <!-- 增加的部分--> <exclusions> <exclusion> <groupId>org.apache.directory.client.ldap</groupId> <artifactId>ldap-client-api</artifactId> </exclusion> </exclusions> </dependency>
啟動(dòng)Cli
1.分別把集群的配置文件copy到hive-cli
模塊的resources
(需要自己新建和配置)下core-site.xml
,hdfs-site.xml
,yarn-site.xml
,mapred-site.xml
,新建hive-log4j2.properties
和hive-site.xml
文件。
2.變更配置項(xiàng)
mapred-site.xml中新增,
<!-- 支持跨平臺(tái)提交任務(wù) --> <property> <name>mapreduce.app-submission.cross-platform</name> <value>true</value> </property>
hive-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" ?> <configuration> <property> <name>hive.cli.print.current.db</name> <value>true</value> </property> <property> <name>hive.cli.print.header</name> <value>true</value> </property> <!-- hive 元數(shù)據(jù)鏈接地址 HA--> <property> <name>hive.metastore.uris</name> <value>thrift://192.168.1.1:9083,thrift://192.168.1.2:9083</value> <final>true</final> </property> <property> <name>hive.cluster.delegation.token.store.class</name> <value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value> <description>Hive defaults to MemoryTokenStore, or ZooKeeperTokenStore</description> </property> <!-- 測(cè)試環(huán)境的 hive客戶端本地模式--> <property> <name>hive.exec.mode.local.auto</name> <value>true</value> </property> <!--輸入最大數(shù)據(jù)量,默認(rèn)128MB--> <property> <name>hive.exec.mode.local.auto.inputbytes.max</name> <value>134217728</value> </property> <!-- 最大任務(wù)數(shù)--> <property> <name>hive.exec.mode.local.auto.input.files.max</name> <value>10</value> </property> </configuration>
hive-log4j2.properties變更
property.hive.log.dir = /opt/hive/logs property.hive.log.file = hive-${sys:user.name}.log
3.修改hive-cli
模塊中的pom文件,將commons-io
,和disruptor
范圍取消test限制。
4.配置CliDriver啟動(dòng),JVM參數(shù)-Djline.WindowsTerminal.directConsole=false
,環(huán)境配置,HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native;HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib
問(wèn)題:import org.apache.hive.tmpl.QueryProfileTmpl
缺失
解決 在git bash執(zhí)行mvn generate-sources
,然后拷貝hive service
模塊目錄target/generated-jamon/org/apache/hive/tmpl
生成的源文件至指定目錄(需要手動(dòng)創(chuàng)建),QueryProfileTmpl
和QueryProfileTmplImpl
。
本地模式下進(jìn)行基礎(chǔ)操作
建庫(kù)
create database testdb;
建表
create table test001(id int,dummy string) stored as orc;
插入數(shù)據(jù)
use testdb; insert into table testdb.test001 values(1,"X");
查詢數(shù)據(jù)
select * from test001;
刪除數(shù)據(jù),不建議進(jìn)行單行或者多行的條件式刪除,可以進(jìn)行分區(qū)或整表的刪除,重新寫入數(shù)據(jù)的方式。
執(zhí)行記錄:
hive (default)> create database testdb; create database testdb; OK Time taken: 0.072 seconds hive (default)> use testdb; use testdb; OK Time taken: 0.045 seconds hive (testdb)> create table test001(id int,dummy string) stored as orc; create table test001(id int,dummy string) stored as orc; OK Time taken: 0.53 seconds hive (testdb)> insert into table test001 values(1,"X"); insert into table test001 values(1,"X"); Automatically selecting local only mode for query WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Query ID = 19057578_20230222162806_f6ce7921-08d3-41db-8408-dfd18339e594 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Job running in-process (local Hadoop) 2023-02-22 16:28:13,266 Stage-1 map = 0%, reduce = 0% 2023-02-22 16:28:14,295 Stage-1 map = 100%, reduce = 0% Ended Job = job_local2039626018_0001 Stage-4 is selected by condition resolver. Stage-3 is filtered out by condition resolver. Stage-5 is filtered out by condition resolver. Moving data to directory hdfs://hadoopdemo/user/bigdata/hive/warehouse/testdb.db/test001/.hive-staging_hive_2023-02-22_16-28-06_738_8793114222801018317-1/-ext-10000 Loading data to table testdb.test001 MapReduce Jobs Launched: Stage-Stage-1: HDFS Read: 4 HDFS Write: 348 SUCCESS Total MapReduce CPU Time Spent: 0 msec OK Time taken: 8.293 seconds _col0 _col1 hive (testdb)> select * from test001; select * from test001; OK test001.id test001.dummy 1 X hive (testdb)> Time taken: 0.241 seconds, Fetched: 1 row(s) hive (testdb)> drop table test001; drop table test001; OK Time taken: 0.218 seconds hive (testdb)> show tables; show tables; OK tab_name values__tmp__table__1 hive (testdb)> Time taken: 0.063 seconds, Fetched: 1 row(s)
values__tmp__table__1
為客戶端內(nèi)存中的臨時(shí)數(shù)據(jù),會(huì)在關(guān)閉客戶端后刪除,不會(huì)落盤。
遺留問(wèn)題
提交任務(wù)到y(tǒng)arn集群失敗
取消本地模式,在將任務(wù)提交到y(tǒng)arn是會(huì)有錯(cuò)誤。判斷是客戶端環(huán)境配置問(wèn)題,但是沒(méi)有在客戶端處解決。
yarn上的日志
************************************************************/ 2023-02-22 16:35:04,263 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration 2023-02-22 16:35:04,263 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2023-02-22 16:35:04,263 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1 cluster_timestamp: 1677047289113 } attemptId: 2 } keyId: 176967782) 2023-02-22 16:35:04,365 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter 2023-02-22 16:35:04,367 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2427) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:545) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:522) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1764) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:522) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:308) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1722) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2395) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2419) ... 12 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2299) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2393) ... 13 more 2023-02-22 16:35:04,369 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2427) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:545) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:522) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1764) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:522) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:308) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1722) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2395) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2419) ... 12 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2299) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2393) ... 13 more 2023-02-22 16:35:04,372 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter not found
到此這篇關(guān)于Windows下 IDEA編譯調(diào)試 hive2.3.9的文章就介紹到這了,更多相關(guān)IDEA編譯調(diào)試 hive2.3.9內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!
相關(guān)文章
SpringBoot集成JWT實(shí)現(xiàn)token驗(yàn)證的流程
Json web token (JWT), 是為了在網(wǎng)絡(luò)應(yīng)用環(huán)境間傳遞聲明而執(zhí)行的一種基于JSON的開(kāi)放標(biāo)準(zhǔn)((RFC 7519).這篇文章主要介紹了SpringBoot集成JWT實(shí)現(xiàn)token驗(yàn)證,需要的朋友可以參考下2020-01-01@JsonSerialize(using = LongToStringUtil.class)注解的使
這篇文章主要介紹了@JsonSerialize(using = LongToStringUtil.class)注解的使用方式,具有很好的參考價(jià)值,希望對(duì)大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-08-08Maven pom.xml文件中build,plugin標(biāo)簽的使用小結(jié)
本文主要介紹了Maven pom.xml文件中build,plugin標(biāo)簽的使用小結(jié),文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2025-03-03詳解springboot和vue前后端分離開(kāi)發(fā)跨域登陸問(wèn)題
這篇文章主要介紹了詳解springboot和vue前后端分離開(kāi)發(fā)跨域登陸問(wèn)題,文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2019-09-09