欧美bbbwbbbw肥妇,免费乱码人妻系列日韩,一级黄片

Spring Boot集成starrocks快速入門Demo(適用場景)

 更新時(shí)間:2024年08月21日 14:45:07   作者:HBLOGA  
StarRocks 是新一代極速全場景 MPP (Massively Parallel Processing) 數(shù)據(jù)庫,StarRocks 的愿景是能夠讓用戶的數(shù)據(jù)分析變得更加簡單和敏捷,這篇文章主要介紹了Spring Boot集成starrocks快速入門Demo,需要的朋友可以參考下

1.什么是starrocks?

StarRocks 是新一代極速全場景 MPP (Massively Parallel Processing) 數(shù)據(jù)庫。StarRocks 的愿景是能夠讓用戶的數(shù)據(jù)分析變得更加簡單和敏捷。用戶無需經(jīng)過復(fù)雜的預(yù)處理,就可以用 StarRocks 來支持多種數(shù)據(jù)分析場景的極速分析。 StarRocks 架構(gòu)簡潔,采用了全面向量化引擎,并配備全新設(shè)計(jì)的 CBO (Cost Based Optimizer) 優(yōu)化器,查詢速度(尤其是多表關(guān)聯(lián)查詢)遠(yuǎn)超同類產(chǎn)品。 StarRocks 能很好地支持實(shí)時(shí)數(shù)據(jù)分析,并能實(shí)現(xiàn)對實(shí)時(shí)更新數(shù)據(jù)的高效查詢。StarRocks 還支持現(xiàn)代化物化視圖,進(jìn)一步加速查詢。 使用 StarRocks,用戶可以靈活構(gòu)建包括大寬表、星型模型、雪花模型在內(nèi)的各類模型。 StarRocks 兼容 MySQL 協(xié)議,支持標(biāo)準(zhǔn) SQL 語法,易于對接使用,全系統(tǒng)無外部依賴,高可用,易于運(yùn)維管理。StarRocks 還兼容多種主流 BI 產(chǎn)品,包括 Tableau、Power BI、FineBI 和 Smartbi。 StarRocks 是 Linux 基金會(huì)項(xiàng)目,采用 Apache 2.0 許可證,可在 StarRocks GitHub 存儲庫中找到(請參閱 StarRocks 許可證)。StarRocks(i)鏈接或調(diào)用第三方軟件庫中的函數(shù),其許可證可在 licenses-binary 文件夾中找到;和(ii)包含第三方軟件代碼,其許可證可在 licenses 文件夾中找到。

適用場景?

StarRocks 可以滿足企業(yè)級用戶的多種分析需求,包括 OLAP (Online Analytical Processing) 多維分析、定制報(bào)表、實(shí)時(shí)數(shù)據(jù)分析和 Ad-hoc 數(shù)據(jù)分析等。

OLAP 多維分析?

利用 StarRocks 的 MPP 框架和向量化執(zhí)行引擎,用戶可以靈活的選擇雪花模型,星型模型,寬表模型或者預(yù)聚合模型。適用于靈活配置的多維分析報(bào)表,業(yè)務(wù)場景包括:

用戶行為分析用戶畫像、標(biāo)簽分析、圈人高維業(yè)務(wù)指標(biāo)報(bào)表自助式報(bào)表平臺業(yè)務(wù)問題探查分析跨主題業(yè)務(wù)分析財(cái)務(wù)報(bào)表系統(tǒng)監(jiān)控分析 實(shí)時(shí)數(shù)據(jù)倉庫?

StarRocks 設(shè)計(jì)和實(shí)現(xiàn)了主鍵表,能夠?qū)崟r(shí)更新數(shù)據(jù)并極速查詢,可以秒級同步 TP (Transaction Processing) 數(shù)據(jù)庫的變化,構(gòu)建實(shí)時(shí)數(shù)倉,業(yè)務(wù)場景包括:

  • 用戶行為分析
  • 用戶畫像、標(biāo)簽分析、圈人
  • 高維業(yè)務(wù)指標(biāo)報(bào)表
  • 自助式報(bào)表平臺
  • 業(yè)務(wù)問題探查分析
  • 跨主題業(yè)務(wù)分析
  • 財(cái)務(wù)報(bào)表
  • 系統(tǒng)監(jiān)控分析

高并發(fā)查詢?

StarRocks 通過良好的數(shù)據(jù)分布特性,靈活的索引以及物化視圖等特性,可以解決面向用戶側(cè)的分析場景,業(yè)務(wù)場景包括:

  • 廣告主報(bào)表分析
  • 零售行業(yè)渠道人員分析
  • SaaS 行業(yè)面向用戶分析報(bào)表
  • Dashboard 多頁面分析

統(tǒng)一分析?

  • 通過使用一套系統(tǒng)解決多維分析、高并發(fā)查詢、預(yù)計(jì)算、實(shí)時(shí)分析查詢等場景,降低系統(tǒng)復(fù)雜度和多技術(shù)棧開發(fā)與維護(hù)成本。
  • 使用 StarRocks 統(tǒng)一管理數(shù)據(jù)湖和數(shù)據(jù)倉庫,將高并發(fā)和實(shí)時(shí)性要求很高的業(yè)務(wù)放在 StarRocks 中分析,也可以使用 External Catalog 和外部表進(jìn)行數(shù)據(jù)湖上的分析。

存算一體架構(gòu)

本地存儲為實(shí)時(shí)查詢提供了更低的查詢延遲。 作為典型的大規(guī)模并行處理 (MPP) 數(shù)據(jù)庫,StarRocks 支持存算一體架構(gòu)。在這種架構(gòu)中,BE 負(fù)責(zé)數(shù)據(jù)存儲和計(jì)算。直接訪問 BE 本地?cái)?shù)據(jù)允許本地計(jì)算,避免了數(shù)據(jù)傳輸和復(fù)制,從而提供超快的查詢和分析性能。該架構(gòu)支持多副本數(shù)據(jù)存儲,增強(qiáng)了集群處理高并發(fā)查詢的能力并確保數(shù)據(jù)可靠性。非常適合追求最佳查詢性能的場景。  

2.環(huán)境搭建

采用docker搭建最簡單的測試環(huán)境

docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd --name quickstart starrocks/allin1-ubuntu

3.代碼工程

實(shí)驗(yàn)?zāi)康?/h3>
  • 測試mysql創(chuàng)建 修改 插入刪除數(shù)據(jù)
  • 用streamload導(dǎo)入數(shù)據(jù)

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>springboot-demo</artifactId>
        <groupId>com.et</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>
    <artifactId>starrocks</artifactId>
    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-autoconfigure</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.48</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.3</version>
       </dependency>
    </dependencies>
</project>

mysql連接

/**
Copyright (c) 2021 Beijing Dingshi Zongheng Technology Co., Ltd. All rights reserved.
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
      "License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
**/
package com.et.starrocks.mysql;
import java.sql.*;
public class MysqlClient {
    public static void main(String[] args) {
        String host = "172.30.17.1";
        //query_port in fe.conf
        String port = "9030";
        String user = "root";
        //password is empty by default
        String password = "";
        //connect to starrocks
        Connection conn = null;
        try {
            conn = getConn(host, port, user, password, "");
        } catch (Exception e) {
            System.out.println("connect to starrocks failed");
            e.printStackTrace();
            return;
        }
        System.out.println("connect to starrocks successfully");
        //create statement
        Statement stmt = null;
        try {
            stmt = conn.createStatement();
        } catch (SQLException e) {
            System.out.println("create statement failed");
            e.printStackTrace();
            closeConn(conn);
            return;
        }
        System.out.println("create statement successfully");
        //create database
        try {
            stmt.execute("CREATE DATABASE IF NOT EXISTS db_test");
        } catch (SQLException e) {
            System.out.println("create database failed");
            e.printStackTrace();
            closeStmt(stmt);
            closeConn(conn);
            return;
        }
        System.out.println("create database successfully");
        //set db context
        try {
            stmt.execute("USE db_test");
        } catch (SQLException e) {
            System.out.println("set db context failed");
            e.printStackTrace();
            closeStmt(stmt);
            closeConn(conn);
            return;
        }
        System.out.println("set db context successfully");
        //create table
        try {
            stmt.execute("CREATE TABLE IF NOT EXISTS table_test(siteid INT, citycode SMALLINT, pv BIGINT SUM) " +
                    "AGGREGATE KEY(siteid, citycode) " +
                    "DISTRIBUTED BY HASH(siteid) BUCKETS 10 " +
                    "PROPERTIES(\"replication_num\" = \"1\")");
        } catch (Exception e) {
            System.out.println("create table failed");
            e.printStackTrace();
            closeStmt(stmt);
            closeConn(conn);
            return;
        }
        System.out.println("create table successfully");
        //insert data
        try {
            stmt.execute("INSERT INTO table_test values(1, 2, 3), (4, 5, 6), (1, 2, 4)");
        } catch (Exception e) {
            System.out.println("insert data failed");
            e.printStackTrace();
            closeStmt(stmt);
            closeConn(conn);
            return;
        }
        System.out.println("insert data successfully");
        //query data
        try {
            ResultSet result = stmt.executeQuery("SELECT * FROM table_test");
            System.out.println("data queried is :");
            while (result.next()) {
                int siteid = result.getInt("siteid");
                int citycode = result.getInt("citycode");
                int pv = result.getInt("pv");
                System.out.println("\t" + siteid + "\t" + citycode + "\t" + pv);
            }
        } catch (Exception e) {
            System.out.println("query data failed");
            e.printStackTrace();
            closeStmt(stmt);
            closeConn(conn);
            return;
        }
        //drop database
     /*   try {
            stmt.execute("DROP DATABASE IF EXISTS db_test");
        } catch (Exception e) {
            System.out.println("drop database failed");
            e.printStackTrace();
            closeStmt(stmt);
            closeConn(conn);
            return;
        }*/
        System.out.println("drop database successfully");
        closeStmt(stmt);
        closeConn(conn);
    }
    public static Connection getConn(String host, String port, String user, String password, String database) throws Exception {
        Class.forName("com.mysql.jdbc.Driver");
        String url = "jdbc:mysql://" + host + ":" + port + "/" + database + "?user=" + user + "&password=" + password;
        return DriverManager.getConnection(url);
    }
    public static void closeConn(Connection conn) {
        try {
            conn.close();
            System.out.println("conn closed");
        } catch (Exception e) {
            System.out.println("close conn failed");
            e.printStackTrace();
        }
    }
    public static void closeStmt(Statement stmt) {
        try {
            stmt.close();
            System.out.println("stmt closed");
        } catch (Exception e) {
            System.out.println("close stmt failed");
            e.printStackTrace();
        }
    }
}

streamload方式

package com.et.starrocks.streamload;// Copyright (c) 2021 Beijing Dingshi Zongheng Technology Co., Ltd. All rights reserved.
//
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.
import org.apache.commons.codec.binary.Base64;
import org.apache.http.HttpHeaders;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultRedirectStrategy;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
/**
 * This class is a java demo for starrocks stream load
 *
 * The pom.xml dependency:
 *
 *         <dependency>
 *             <groupId>org.apache.httpcomponents</groupId>
 *             <artifactId>httpclient</artifactId>
 *             <version>4.5.3</version>
 *         </dependency>
 *
 * How to use:
 *
 * 1 create a table in starrocks with any mysql client
 *
 * CREATE TABLE `stream_test` (
 *   `id` bigint(20) COMMENT "",
 *   `id2` bigint(20) COMMENT "",
 *   `username` varchar(32) COMMENT ""
 * ) ENGINE=OLAP
 * DUPLICATE KEY(`id`)
 * DISTRIBUTED BY HASH(`id`) BUCKETS 20;
 *
 *
 * 2 change the StarRocks cluster, db, user config in this class
 *
 * 3 run this class, you should see the following output:
 *
 * {
 *     "TxnId": 27,
 *     "Label": "39c25a5c-7000-496e-a98e-348a264c81de",
 *     "Status": "Success",
 *     "Message": "OK",
 *     "NumberTotalRows": 10,
 *     "NumberLoadedRows": 10,
 *     "NumberFilteredRows": 0,
 *     "NumberUnselectedRows": 0,
 *     "LoadBytes": 50,
 *     "LoadTimeMs": 151
 * }
 *
 * Attention:
 *
 * 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException
 *   Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present
 *     at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96)
 *     at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
 *     at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
 *     at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
 *     at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 *     at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
 *
 *2 run this class more than once, the status code for http response is still ok, and you will see
 *  the following output:
 *
 * {
 *     "TxnId": -1,
 *     "Label": "39c25a5c-7000-496e-a98e-348a264c81de",
 *     "Status": "Label Already Exists",
 *     "ExistingJobStatus": "FINISHED",
 *     "Message": "Label [39c25a5c-7000-496e-a98e-348a264c81de"] has already been used.",
 *     "NumberTotalRows": 0,
 *     "NumberLoadedRows": 0,
 *     "NumberFilteredRows": 0,
 *     "NumberUnselectedRows": 0,
 *     "LoadBytes": 0,
 *     "LoadTimeMs": 0
 * }
 * 3 when the response statusCode is 200, that doesn't mean your stream load is ok, there may be still
 *   some stream problem unless you see the output with 'ok' message
 */
public class StarRocksStreamLoad {
    private final static String STARROCKS_HOST = "127.0.0.1";
    private final static String STARROCKS_DB = "db_test";
    private final static String STARROCKS_TABLE = "stream_test";
    private final static String STARROCKS_USER = "root";
    private final static String STARROCKS_PASSWORD = "";
    private final static int STARROCKS_HTTP_PORT = 8040;
    private void sendData(String content) throws Exception {
        final String loadUrl = String.format("http://%s:%s/api/%s/%s/_stream_load",
                STARROCKS_HOST,
                STARROCKS_HTTP_PORT,
                STARROCKS_DB,
                STARROCKS_TABLE);
        final HttpClientBuilder httpClientBuilder = HttpClients
                .custom()
                .setRedirectStrategy(new DefaultRedirectStrategy() {
                    @Override
                    protected boolean isRedirectable(String method) {
                        return true;
                    }
                });
        try (CloseableHttpClient client = httpClientBuilder.build()) {
            HttpPut put = new HttpPut(loadUrl);
            StringEntity entity = new StringEntity(content, "UTF-8");
            put.setHeader(HttpHeaders.EXPECT, "100-continue");
            put.setHeader(HttpHeaders.AUTHORIZATION, basicAuthHeader(STARROCKS_USER, STARROCKS_PASSWORD));
            // the label header is optional, not necessary
            // use label header can ensure at most once semantics
            put.setHeader("label", "39c25a5c-7000-496e-a98e-348a264c81de1");
            put.setEntity(entity);
            try (CloseableHttpResponse response = client.execute(put)) {
                String loadResult = "";
                if (response.getEntity() != null) {
                    loadResult = EntityUtils.toString(response.getEntity());
                }
                final int statusCode = response.getStatusLine().getStatusCode();
                // statusCode 200 just indicates that starrocks be service is ok, not stream load
                // you should see the output content to find whether stream load is success
                if (statusCode != 200) {
                    throw new IOException(
                            String.format("Stream load failed, statusCode=%s load result=%s", statusCode, loadResult));
                }
                System.out.println(loadResult);
            }
        }
    }
    private String basicAuthHeader(String username, String password) {
        final String tobeEncode = username + ":" + password;
        byte[] encoded = Base64.encodeBase64(tobeEncode.getBytes(StandardCharsets.UTF_8));
        return "Basic " + new String(encoded);
    }
    public static void main(String[] args) throws Exception {
        int id1 = 1;
        int id2 = 10;
        String id3 = "Simon";
        int rowNumber = 10;
        String oneRow = id1 + "\t" + id2 + "\t" + id3 + "\n";
        StringBuilder stringBuilder = new StringBuilder();
        for (int i = 0; i < rowNumber; i++) {
            stringBuilder.append(oneRow);
        }
        stringBuilder.deleteCharAt(stringBuilder.length() - 1);
        String loadData = stringBuilder.toString();
        StarRocksStreamLoad starrocksStreamLoad = new StarRocksStreamLoad();
        starrocksStreamLoad.sendData(loadData);
    }
}

以上只是一些關(guān)鍵代碼,所有代碼請參見下面代碼倉庫

代碼倉庫 https://github.com/Harries/springboot-demo

4.測試

MysqlClient

啟動(dòng)main方法,可以看到執(zhí)行成功

connect to starrocks successfully
create statement successfully
create database successfully
set db context successfully
create table successfully
insert data successfully
data queried is :
 1 2 7
 4 5 6
drop database successfully
stmt closed
conn closed

StarRocksStreamLoad

啟動(dòng)main方法,可以看到插入成功

20:51:47.521 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://127.0.0.1:8040] can be kept alive indefinitely
20:51:47.521 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://127.0.0.1:8040][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
{
 "TxnId": 2,
 "Label": "39c25a5c-7000-496e-a98e-348a264c81de1",
 "Status": "Success",
 "Message": "OK",
 "NumberTotalRows": 10,
 "NumberLoadedRows": 10,
 "NumberFilteredRows": 0,
 "NumberUnselectedRows": 0,
 "LoadBytes": 109,
 "LoadTimeMs": 975,
 "BeginTxnTimeMs": 261,
 "StreamLoadPlanTimeMs": 342,
 "ReadDataTimeMs": 0,
 "WriteDataTimeMs": 106,
 "CommitAndPublishTimeMs": 259
}

5.引用

Architecture | StarRocks

Spring Boot集成starrocks快速入門Demo | Harries Blog™

到此這篇關(guān)于Spring Boot集成starrocks快速入門Demo的文章就介紹到這了,更多相關(guān)Spring Boot集成starrocks內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!

相關(guān)文章

最新評論