Spring Boot集成starrocks快速入門Demo(適用場景)
1.什么是starrocks?
StarRocks 是新一代極速全場景 MPP (Massively Parallel Processing) 數(shù)據(jù)庫。StarRocks 的愿景是能夠讓用戶的數(shù)據(jù)分析變得更加簡單和敏捷。用戶無需經(jīng)過復(fù)雜的預(yù)處理,就可以用 StarRocks 來支持多種數(shù)據(jù)分析場景的極速分析。 StarRocks 架構(gòu)簡潔,采用了全面向量化引擎,并配備全新設(shè)計(jì)的 CBO (Cost Based Optimizer) 優(yōu)化器,查詢速度(尤其是多表關(guān)聯(lián)查詢)遠(yuǎn)超同類產(chǎn)品。 StarRocks 能很好地支持實(shí)時(shí)數(shù)據(jù)分析,并能實(shí)現(xiàn)對實(shí)時(shí)更新數(shù)據(jù)的高效查詢。StarRocks 還支持現(xiàn)代化物化視圖,進(jìn)一步加速查詢。 使用 StarRocks,用戶可以靈活構(gòu)建包括大寬表、星型模型、雪花模型在內(nèi)的各類模型。 StarRocks 兼容 MySQL 協(xié)議,支持標(biāo)準(zhǔn) SQL 語法,易于對接使用,全系統(tǒng)無外部依賴,高可用,易于運(yùn)維管理。StarRocks 還兼容多種主流 BI 產(chǎn)品,包括 Tableau、Power BI、FineBI 和 Smartbi。 StarRocks 是 Linux 基金會(huì)項(xiàng)目,采用 Apache 2.0 許可證,可在 StarRocks GitHub 存儲庫中找到(請參閱 StarRocks 許可證)。StarRocks(i)鏈接或調(diào)用第三方軟件庫中的函數(shù),其許可證可在 licenses-binary 文件夾中找到;和(ii)包含第三方軟件代碼,其許可證可在 licenses 文件夾中找到。
適用場景?
StarRocks 可以滿足企業(yè)級用戶的多種分析需求,包括 OLAP (Online Analytical Processing) 多維分析、定制報(bào)表、實(shí)時(shí)數(shù)據(jù)分析和 Ad-hoc 數(shù)據(jù)分析等。
OLAP 多維分析?
利用 StarRocks 的 MPP 框架和向量化執(zhí)行引擎,用戶可以靈活的選擇雪花模型,星型模型,寬表模型或者預(yù)聚合模型。適用于靈活配置的多維分析報(bào)表,業(yè)務(wù)場景包括:
用戶行為分析用戶畫像、標(biāo)簽分析、圈人高維業(yè)務(wù)指標(biāo)報(bào)表自助式報(bào)表平臺業(yè)務(wù)問題探查分析跨主題業(yè)務(wù)分析財(cái)務(wù)報(bào)表系統(tǒng)監(jiān)控分析 實(shí)時(shí)數(shù)據(jù)倉庫?
StarRocks 設(shè)計(jì)和實(shí)現(xiàn)了主鍵表,能夠?qū)崟r(shí)更新數(shù)據(jù)并極速查詢,可以秒級同步 TP (Transaction Processing) 數(shù)據(jù)庫的變化,構(gòu)建實(shí)時(shí)數(shù)倉,業(yè)務(wù)場景包括:
- 用戶行為分析
- 用戶畫像、標(biāo)簽分析、圈人
- 高維業(yè)務(wù)指標(biāo)報(bào)表
- 自助式報(bào)表平臺
- 業(yè)務(wù)問題探查分析
- 跨主題業(yè)務(wù)分析
- 財(cái)務(wù)報(bào)表
- 系統(tǒng)監(jiān)控分析
高并發(fā)查詢?
StarRocks 通過良好的數(shù)據(jù)分布特性,靈活的索引以及物化視圖等特性,可以解決面向用戶側(cè)的分析場景,業(yè)務(wù)場景包括:
- 廣告主報(bào)表分析
- 零售行業(yè)渠道人員分析
- SaaS 行業(yè)面向用戶分析報(bào)表
- Dashboard 多頁面分析
統(tǒng)一分析?
- 通過使用一套系統(tǒng)解決多維分析、高并發(fā)查詢、預(yù)計(jì)算、實(shí)時(shí)分析查詢等場景,降低系統(tǒng)復(fù)雜度和多技術(shù)棧開發(fā)與維護(hù)成本。
- 使用 StarRocks 統(tǒng)一管理數(shù)據(jù)湖和數(shù)據(jù)倉庫,將高并發(fā)和實(shí)時(shí)性要求很高的業(yè)務(wù)放在 StarRocks 中分析,也可以使用 External Catalog 和外部表進(jìn)行數(shù)據(jù)湖上的分析。
存算一體架構(gòu)
本地存儲為實(shí)時(shí)查詢提供了更低的查詢延遲。 作為典型的大規(guī)模并行處理 (MPP) 數(shù)據(jù)庫,StarRocks 支持存算一體架構(gòu)。在這種架構(gòu)中,BE 負(fù)責(zé)數(shù)據(jù)存儲和計(jì)算。直接訪問 BE 本地?cái)?shù)據(jù)允許本地計(jì)算,避免了數(shù)據(jù)傳輸和復(fù)制,從而提供超快的查詢和分析性能。該架構(gòu)支持多副本數(shù)據(jù)存儲,增強(qiáng)了集群處理高并發(fā)查詢的能力并確保數(shù)據(jù)可靠性。非常適合追求最佳查詢性能的場景。
2.環(huán)境搭建
采用docker搭建最簡單的測試環(huán)境
docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd --name quickstart starrocks/allin1-ubuntu
3.代碼工程
實(shí)驗(yàn)?zāi)康?/h3>
- 測試mysql創(chuàng)建 修改 插入刪除數(shù)據(jù)
- 用streamload導(dǎo)入數(shù)據(jù)
pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <artifactId>springboot-demo</artifactId> <groupId>com.et</groupId> <version>1.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>starrocks</artifactId> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-autoconfigure</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.48</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.3</version> </dependency> </dependencies> </project>
mysql連接
/** Copyright (c) 2021 Beijing Dingshi Zongheng Technology Co., Ltd. All rights reserved. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. **/ package com.et.starrocks.mysql; import java.sql.*; public class MysqlClient { public static void main(String[] args) { String host = "172.30.17.1"; //query_port in fe.conf String port = "9030"; String user = "root"; //password is empty by default String password = ""; //connect to starrocks Connection conn = null; try { conn = getConn(host, port, user, password, ""); } catch (Exception e) { System.out.println("connect to starrocks failed"); e.printStackTrace(); return; } System.out.println("connect to starrocks successfully"); //create statement Statement stmt = null; try { stmt = conn.createStatement(); } catch (SQLException e) { System.out.println("create statement failed"); e.printStackTrace(); closeConn(conn); return; } System.out.println("create statement successfully"); //create database try { stmt.execute("CREATE DATABASE IF NOT EXISTS db_test"); } catch (SQLException e) { System.out.println("create database failed"); e.printStackTrace(); closeStmt(stmt); closeConn(conn); return; } System.out.println("create database successfully"); //set db context try { stmt.execute("USE db_test"); } catch (SQLException e) { System.out.println("set db context failed"); e.printStackTrace(); closeStmt(stmt); closeConn(conn); return; } System.out.println("set db context successfully"); //create table try { stmt.execute("CREATE TABLE IF NOT EXISTS table_test(siteid INT, citycode SMALLINT, pv BIGINT SUM) " + "AGGREGATE KEY(siteid, citycode) " + "DISTRIBUTED BY HASH(siteid) BUCKETS 10 " + "PROPERTIES(\"replication_num\" = \"1\")"); } catch (Exception e) { System.out.println("create table failed"); e.printStackTrace(); closeStmt(stmt); closeConn(conn); return; } System.out.println("create table successfully"); //insert data try { stmt.execute("INSERT INTO table_test values(1, 2, 3), (4, 5, 6), (1, 2, 4)"); } catch (Exception e) { System.out.println("insert data failed"); e.printStackTrace(); closeStmt(stmt); closeConn(conn); return; } System.out.println("insert data successfully"); //query data try { ResultSet result = stmt.executeQuery("SELECT * FROM table_test"); System.out.println("data queried is :"); while (result.next()) { int siteid = result.getInt("siteid"); int citycode = result.getInt("citycode"); int pv = result.getInt("pv"); System.out.println("\t" + siteid + "\t" + citycode + "\t" + pv); } } catch (Exception e) { System.out.println("query data failed"); e.printStackTrace(); closeStmt(stmt); closeConn(conn); return; } //drop database /* try { stmt.execute("DROP DATABASE IF EXISTS db_test"); } catch (Exception e) { System.out.println("drop database failed"); e.printStackTrace(); closeStmt(stmt); closeConn(conn); return; }*/ System.out.println("drop database successfully"); closeStmt(stmt); closeConn(conn); } public static Connection getConn(String host, String port, String user, String password, String database) throws Exception { Class.forName("com.mysql.jdbc.Driver"); String url = "jdbc:mysql://" + host + ":" + port + "/" + database + "?user=" + user + "&password=" + password; return DriverManager.getConnection(url); } public static void closeConn(Connection conn) { try { conn.close(); System.out.println("conn closed"); } catch (Exception e) { System.out.println("close conn failed"); e.printStackTrace(); } } public static void closeStmt(Statement stmt) { try { stmt.close(); System.out.println("stmt closed"); } catch (Exception e) { System.out.println("close stmt failed"); e.printStackTrace(); } } }
streamload方式
package com.et.starrocks.streamload;// Copyright (c) 2021 Beijing Dingshi Zongheng Technology Co., Ltd. All rights reserved. // // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. import org.apache.commons.codec.binary.Base64; import org.apache.http.HttpHeaders; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpPut; import org.apache.http.entity.StringEntity; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.DefaultRedirectStrategy; import org.apache.http.impl.client.HttpClientBuilder; import org.apache.http.impl.client.HttpClients; import org.apache.http.util.EntityUtils; import java.io.IOException; import java.nio.charset.StandardCharsets; /** * This class is a java demo for starrocks stream load * * The pom.xml dependency: * * <dependency> * <groupId>org.apache.httpcomponents</groupId> * <artifactId>httpclient</artifactId> * <version>4.5.3</version> * </dependency> * * How to use: * * 1 create a table in starrocks with any mysql client * * CREATE TABLE `stream_test` ( * `id` bigint(20) COMMENT "", * `id2` bigint(20) COMMENT "", * `username` varchar(32) COMMENT "" * ) ENGINE=OLAP * DUPLICATE KEY(`id`) * DISTRIBUTED BY HASH(`id`) BUCKETS 20; * * * 2 change the StarRocks cluster, db, user config in this class * * 3 run this class, you should see the following output: * * { * "TxnId": 27, * "Label": "39c25a5c-7000-496e-a98e-348a264c81de", * "Status": "Success", * "Message": "OK", * "NumberTotalRows": 10, * "NumberLoadedRows": 10, * "NumberFilteredRows": 0, * "NumberUnselectedRows": 0, * "LoadBytes": 50, * "LoadTimeMs": 151 * } * * Attention: * * 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException * Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present * at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96) * at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) * at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) * at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) * at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) * at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) * *2 run this class more than once, the status code for http response is still ok, and you will see * the following output: * * { * "TxnId": -1, * "Label": "39c25a5c-7000-496e-a98e-348a264c81de", * "Status": "Label Already Exists", * "ExistingJobStatus": "FINISHED", * "Message": "Label [39c25a5c-7000-496e-a98e-348a264c81de"] has already been used.", * "NumberTotalRows": 0, * "NumberLoadedRows": 0, * "NumberFilteredRows": 0, * "NumberUnselectedRows": 0, * "LoadBytes": 0, * "LoadTimeMs": 0 * } * 3 when the response statusCode is 200, that doesn't mean your stream load is ok, there may be still * some stream problem unless you see the output with 'ok' message */ public class StarRocksStreamLoad { private final static String STARROCKS_HOST = "127.0.0.1"; private final static String STARROCKS_DB = "db_test"; private final static String STARROCKS_TABLE = "stream_test"; private final static String STARROCKS_USER = "root"; private final static String STARROCKS_PASSWORD = ""; private final static int STARROCKS_HTTP_PORT = 8040; private void sendData(String content) throws Exception { final String loadUrl = String.format("http://%s:%s/api/%s/%s/_stream_load", STARROCKS_HOST, STARROCKS_HTTP_PORT, STARROCKS_DB, STARROCKS_TABLE); final HttpClientBuilder httpClientBuilder = HttpClients .custom() .setRedirectStrategy(new DefaultRedirectStrategy() { @Override protected boolean isRedirectable(String method) { return true; } }); try (CloseableHttpClient client = httpClientBuilder.build()) { HttpPut put = new HttpPut(loadUrl); StringEntity entity = new StringEntity(content, "UTF-8"); put.setHeader(HttpHeaders.EXPECT, "100-continue"); put.setHeader(HttpHeaders.AUTHORIZATION, basicAuthHeader(STARROCKS_USER, STARROCKS_PASSWORD)); // the label header is optional, not necessary // use label header can ensure at most once semantics put.setHeader("label", "39c25a5c-7000-496e-a98e-348a264c81de1"); put.setEntity(entity); try (CloseableHttpResponse response = client.execute(put)) { String loadResult = ""; if (response.getEntity() != null) { loadResult = EntityUtils.toString(response.getEntity()); } final int statusCode = response.getStatusLine().getStatusCode(); // statusCode 200 just indicates that starrocks be service is ok, not stream load // you should see the output content to find whether stream load is success if (statusCode != 200) { throw new IOException( String.format("Stream load failed, statusCode=%s load result=%s", statusCode, loadResult)); } System.out.println(loadResult); } } } private String basicAuthHeader(String username, String password) { final String tobeEncode = username + ":" + password; byte[] encoded = Base64.encodeBase64(tobeEncode.getBytes(StandardCharsets.UTF_8)); return "Basic " + new String(encoded); } public static void main(String[] args) throws Exception { int id1 = 1; int id2 = 10; String id3 = "Simon"; int rowNumber = 10; String oneRow = id1 + "\t" + id2 + "\t" + id3 + "\n"; StringBuilder stringBuilder = new StringBuilder(); for (int i = 0; i < rowNumber; i++) { stringBuilder.append(oneRow); } stringBuilder.deleteCharAt(stringBuilder.length() - 1); String loadData = stringBuilder.toString(); StarRocksStreamLoad starrocksStreamLoad = new StarRocksStreamLoad(); starrocksStreamLoad.sendData(loadData); } }
以上只是一些關(guān)鍵代碼,所有代碼請參見下面代碼倉庫
代碼倉庫 https://github.com/Harries/springboot-demo
4.測試
MysqlClient
啟動(dòng)main方法,可以看到執(zhí)行成功
connect to starrocks successfully create statement successfully create database successfully set db context successfully create table successfully insert data successfully data queried is : 1 2 7 4 5 6 drop database successfully stmt closed conn closed
StarRocksStreamLoad
啟動(dòng)main方法,可以看到插入成功
20:51:47.521 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://127.0.0.1:8040] can be kept alive indefinitely 20:51:47.521 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://127.0.0.1:8040][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20] { "TxnId": 2, "Label": "39c25a5c-7000-496e-a98e-348a264c81de1", "Status": "Success", "Message": "OK", "NumberTotalRows": 10, "NumberLoadedRows": 10, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 109, "LoadTimeMs": 975, "BeginTxnTimeMs": 261, "StreamLoadPlanTimeMs": 342, "ReadDataTimeMs": 0, "WriteDataTimeMs": 106, "CommitAndPublishTimeMs": 259 }
5.引用
Spring Boot集成starrocks快速入門Demo | Harries Blog™
到此這篇關(guān)于Spring Boot集成starrocks快速入門Demo的文章就介紹到這了,更多相關(guān)Spring Boot集成starrocks內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!
相關(guān)文章
詳解MyBatisPlus如何實(shí)現(xiàn)分頁和查詢操作
這篇文章主要為大家詳細(xì)介紹了MyBatisPlus是如何實(shí)現(xiàn)分頁和查詢操作的,文中的示例代碼講解詳細(xì),對我們學(xué)習(xí)有一定的幫助,需要的可以參考一下2022-05-05IDEA2020.2.3中創(chuàng)建JavaWeb工程的完整步驟記錄
這篇文章主要給大家介紹了關(guān)于IDEA2020.2.3中創(chuàng)建JavaWeb工程的完整步驟,文中通過圖文介紹的非常詳細(xì),對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧2020-12-12JAVA正則表達(dá)式及字符串的替換與分解相關(guān)知識總結(jié)
今天給大家?guī)淼氖顷P(guān)于Java的相關(guān)知識總結(jié),文章圍繞著JAVA正則表達(dá)式及字符串的替換與分解展開,文中有非常詳細(xì)的介紹及代碼示例,需要的朋友可以參考下2021-06-06Java實(shí)戰(zhàn)在線選課系統(tǒng)的實(shí)現(xiàn)流程
讀萬卷書不如行萬里路,只學(xué)書上的理論是遠(yuǎn)遠(yuǎn)不夠的,只有在實(shí)戰(zhàn)中才能獲得能力的提升,本篇文章手把手帶你用java+SSM+jsp+mysql+maven實(shí)現(xiàn)一個(gè)在線選課系統(tǒng),大家可以在過程中查缺補(bǔ)漏,提升水平2021-11-11Java后端長時(shí)間無操作自動(dòng)退出的實(shí)現(xiàn)方式
這篇文章主要介紹了Java后端長時(shí)間無操作自動(dòng)退出的實(shí)現(xiàn)方式,具有很好的參考價(jià)值,希望對大家有所幫助。如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2022-01-01Java使用synchronized修飾方法來同步線程的實(shí)例演示
synchronized下的方法控制多線程程序中的線程同步非常方便,這里就來看一下Java使用synchronized修飾方法來同步線程的實(shí)例演示,需要的朋友可以參考下2016-06-06java 進(jìn)程是如何在Linux服務(wù)器上進(jìn)行內(nèi)存分配的
這篇文章主要介紹了java 進(jìn)程是如何在Linux服務(wù)器上進(jìn)行內(nèi)存分配的,幫助大家更好的理解和使用Java,感興趣的朋友可以了解下2020-11-11