SpringBoot文件上傳接口并發(fā)性能調(diào)優(yōu)
前言
在一個項目現(xiàn)場,文件上傳接口(文件500K)QPS只有30,這個并發(fā)性能確實堪憂。此文記錄出坑過程。
問題一、InputStream按字節(jié)讀取效率低
// 讀取上傳的文件 Part part = request.getPart("data"); InputStream in = part.getInputStream(); ByteArrayOutputStream str=new ByteArrayOutputStream(); int k; byte[] file = null; while((k=in.read())!=-1){ str.write(k); } file = str.toByteArray(); str.close(); in.close();
/** * Reads the next byte of data from the input stream. The value byte is * returned as an {@code int} in the range {@code 0} to * {@code 255}. If no byte is available because the end of the stream * has been reached, the value {@code -1} is returned. This method * blocks until input data is available, the end of the stream is detected, * or an exception is thrown. * * <p> A subclass must provide an implementation of this method. * * @return the next byte of data, or {@code -1} if the end of the * stream is reached. * @throws IOException if an I/O error occurs. */ public abstract int read() throws IOException;
直接調(diào)用接口發(fā)現(xiàn)接口響應(yīng)確實比較慢,經(jīng)過排查是上述代碼in.read()
按字節(jié)讀取效率特別低。既然定位到問題了,換個方式,每次讀取8K數(shù)據(jù)。
byte[] buffer = new byte[8192]; int bytesRead; while ((bytesRead = in.read(buffer)) != -1) { str.write(buffer, 0, bytesRead); }
/** * Reads some number of bytes from the input stream and stores them into * the buffer array <code>b</code>. The number of bytes actually read is * returned as an integer. This method blocks until input data is * available, end of file is detected, or an exception is thrown. * * <p> If the length of <code>b</code> is zero, then no bytes are read and * <code>0</code> is returned; otherwise, there is an attempt to read at * least one byte. If no byte is available because the stream is at the * end of the file, the value <code>-1</code> is returned; otherwise, at * least one byte is read and stored into <code>b</code>. * * <p> The first byte read is stored into element <code>b[0]</code>, the * next one into <code>b[1]</code>, and so on. The number of bytes read is, * at most, equal to the length of <code>b</code>. Let <i>k</i> be the * number of bytes actually read; these bytes will be stored in elements * <code>b[0]</code> through <code>b[</code><i>k</i><code>-1]</code>, * leaving elements <code>b[</code><i>k</i><code>]</code> through * <code>b[b.length-1]</code> unaffected. * * <p> The <code>read(b)</code> method for class <code>InputStream</code> * has the same effect as: <pre><code> read(b, 0, b.length) </code></pre> * * @param b the buffer into which the data is read. * @return the total number of bytes read into the buffer, or * <code>-1</code> if there is no more data because the end of * the stream has been reached. * @exception IOException If the first byte cannot be read for any reason * other than the end of the file, if the input stream has been closed, or * if some other I/O error occurs. * @exception NullPointerException if <code>b</code> is <code>null</code>. * @see java.io.InputStream#read(byte[], int, int) */ public int read(byte b[]) throws IOException { return read(b, 0, b.length); }
如果JDK>=9,可以使用
readAllBytes
方法,更為便捷。內(nèi)部實現(xiàn)其實也是按照8K進行讀取的。
文件上傳接口通常僅對業(yè)務(wù)邏輯做處理,文件存儲往往會調(diào)用專門的存儲服務(wù)。有2種處理思路:1、接收到完整文件數(shù)據(jù),存儲至內(nèi)存中,然后調(diào)用存儲接口;2、用流的方式,一邊read ServletRequest#InputStream,一邊write 到存儲服務(wù)的Stream中。個人認為方式2更合理,節(jié)約內(nèi)存。
問題二、tomcat暫存性能瓶頸
接口采用multipart/form-data
方式上傳文件,tomcat接收到請求后會將請求內(nèi)容暫存至本地磁盤,目錄通常位于tomcat basedir目錄下,比如我本地路徑為{basedir}\work\Tomcat\localhost\ROOT
。受限于磁盤寫入速率瓶頸,限制了接口性能上限。
機械硬盤寫入速率預估100MB/s,則在千兆組網(wǎng)場景不存在性能瓶頸,如果是固態(tài)硬盤,則寫入速率更高。所以此項配置在2G以上組網(wǎng)才需考慮配置。
修改方法為修改sizeThreshold,默認值為0
。如下所示修改為1MB
,即內(nèi)容大于1MB才存入磁盤,小于直接存入內(nèi)存。
關(guān)于sizeThreshold,catalina包中處理邏輯為:如果對servlet做了配置,會使用配置的值。如果未配置,默認值為0。util包中DiskFileItemFactory默認值為10k。
servlet: multipart: file-size-threshold: 1MB
Tomcat中的相關(guān)處理邏輯,parseRequest
方法按照RFC 1867
規(guī)范對request進行處理。
// org.apache.tomcat.util.http.fileupload.disk.DiskFileItemFactory.java /** * <p>The default {@link org.apache.tomcat.util.http.fileupload.FileItemFactory} * implementation. This implementation creates * {@link org.apache.tomcat.util.http.fileupload.FileItem} instances which keep * their * content either in memory, for smaller items, or in a temporary file on disk, * for larger items. The size threshold, above which content will be stored on * disk, is configurable, as is the directory in which temporary files will be * created.</p> * * <p>If not otherwise configured, the default configuration values are as * follows:</p> * <ul> * <li>Size threshold is 10 KiB.</li> * <li>Repository is the system default temp directory, as returned by * {@code System.getProperty("java.io.tmpdir")}.</li> * </ul> * <p> * <b>NOTE</b>: Files are created in the system default temp directory with * predictable names. This means that a local attacker with write access to that * directory can perform a TOUTOC attack to replace any uploaded file with a * file of the attackers choice. The implications of this will depend on how the * uploaded file is used but could be significant. When using this * implementation in an environment with local, untrusted users, * {@link #setRepository(File)} MUST be used to configure a repository location * that is not publicly writable. In a Servlet container the location identified * by the ServletContext attribute {@code javax.servlet.context.tempdir} * may be used. * </p> * * <p>Temporary files, which are created for file items, will be deleted when * the associated request is recycled.</p> * * @since FileUpload 1.1 */ public class DiskFileItemFactory implements FileItemFactory { // ----------------------------------------------------- Manifest constants /** * The default threshold above which uploads will be stored on disk. */ public static final int DEFAULT_SIZE_THRESHOLD = 10240; }
// org.apache.tomcat.util.http.fileupload.disk.DiskFileItem.java /** * The threshold above which uploads will be stored on disk. */ private final int sizeThreshold; /** * Returns an {@link java.io.OutputStream OutputStream} that can * be used for storing the contents of the file. * * @return An {@link java.io.OutputStream OutputStream} that can be used * for storing the contents of the file. * */ @Override public OutputStream getOutputStream() { if (dfos == null) { final File outputFile = getTempFile(); dfos = new DeferredFileOutputStream(sizeThreshold, outputFile); } return dfos; }
// org.apache.tomcat.util.http.fileupload.DeferredFileOutputStream.java /** * An output stream which will retain data in memory until a specified * threshold is reached, and only then commit it to disk. If the stream is * closed before the threshold is reached, the data will not be written to * disk at all. * <p> * This class originated in FileUpload processing. In this use case, you do * not know in advance the size of the file being uploaded. If the file is small * you want to store it in memory (for speed), but if the file is large you want * to store it to file (to avoid memory issues). */ public class DeferredFileOutputStream extends ThresholdingOutputStream { /** * Constructs an instance of this class which will trigger an event at the * specified threshold, and save data to a file beyond that point. * The initial buffer size will default to 1024 bytes which is ByteArrayOutputStream's default buffer size. * * @param threshold The number of bytes at which to trigger an event. * @param outputFile The file to which data is saved beyond the threshold. */ public DeferredFileOutputStream(final int threshold, final File outputFile) { this(threshold, outputFile, null, null, null, ByteArrayOutputStream.DEFAULT_SIZE); } }
問題三、網(wǎng)絡(luò)帶寬瓶頸
對于常規(guī)企業(yè)內(nèi)部應(yīng)用,局域網(wǎng)環(huán)境下,至少能提供穩(wěn)定的千兆帶寬,常規(guī)業(yè)務(wù)接口不存在網(wǎng)絡(luò)帶寬瓶頸。但是對于文件上傳接口而言,即使是小文件上傳,接口并發(fā)高的場景帶寬消耗依然較大,可能是性能瓶頸。
以千兆帶寬為例,理論最大上傳速率=1000Mbps÷8=125MB/s理論最大上傳速率=1000Mbps÷8=125MB/s理論最大上傳速率=1000Mbps÷8=125MB/s,實際場景很難達到理論最大速率,按照100MB/s預估。500K:200QPS,1M:100QPS,2M:50QPS
問題解決思路整理
- client
指請求接口的客戶端 - nginx
作為反向代理服務(wù)器 - tomcat
web容器 - webserver
web服務(wù),比如springboot項目
排查過程可以根據(jù)由外向內(nèi)層層遞進的方式進行排查,當然也可采用經(jīng)驗判斷法,對最有可能出現(xiàn)性能瓶頸的webserver進行排查。
- 復現(xiàn)問題,在高負載場景請求接口復現(xiàn)問題或者使用Jmeter等工作做并發(fā)壓力測試。復現(xiàn)問題是解決問題的基礎(chǔ)。
- 查看接口請求耗時,對耗時結(jié)構(gòu)進行分析,比如Wating(TTFB)、Content Download耗時長,。比如Content Download耗時長,那就會首先懷疑帶寬。
- nginx性能較高,出現(xiàn)瓶頸概率低??赏ㄟ^查看nginx訪問日志,對比接口總耗時,如果耗時差異較大,就需要排查nginx本身性能、nginx與tomcat之間網(wǎng)絡(luò)。
- tomcat作為主流的web容器,影響性能的配置主要是maxThreads、maxConnections、堆內(nèi)存、垃圾回收。對于成熟的應(yīng)用開發(fā)團隊,會有相對合理的初始配置??赏ㄟ^查看tomcat訪問日志,對比webserver接口耗時,如果耗時差異較大,就需要排查tomcat自身性能問題。
- webserver中的業(yè)務(wù)處理邏輯,通常是接口總耗時占比最高的。優(yōu)先在controller入口和出口記錄日志,計算controller總耗時。如果確定是業(yè)務(wù)邏輯耗時長,再層層遞進排查縮小范圍,找到罪魁禍首。
測試性能匯總
測試環(huán)境
服務(wù)器主機、客戶機
測試環(huán)境所限,服務(wù)器主機、客戶機使用同一臺開發(fā)主機。操作系統(tǒng):windows10,CPU:Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz,內(nèi)存16G磁盤
RND512KQ1T1 Read1219.86Mb/s Write44.88Mb/sJmeter
400線程,60s拉起全部線程tomcat
tomcat9,做了如下配置
tomcat: threads: max: 400 max-connections: 10000 accept-count: 1000
jar啟動參數(shù)
配置了初始堆內(nèi)存java -Dfile.encoding=UTF-8 -jar .\xxx.jar -server -Xms4096m -Xmx9000m
測試結(jié)果
類型 | 平均響應(yīng)時間 ms | 吞吐量/s |
---|---|---|
原始狀態(tài) | 22081 | 0.18 |
優(yōu)化Byte[] | 3966 | 89 |
優(yōu)化file-size-threshold | 1203 | 265 |
基準-(form-data) | 1279 | 279 |
基準-(優(yōu)化file-size-threshold) | 109 | 2930 |
基準-空接口 | 28 | 12401 |
原始狀態(tài):現(xiàn)場報性能問題時的版本,性能太過炸裂,Jmeter線程數(shù)調(diào)整為4,測試上傳文件5KB
優(yōu)化Byte[]:優(yōu)化了從stream讀取存入優(yōu)化Byte[]方法,測試上傳文件5KB。此時網(wǎng)絡(luò)吞吐量45MB/s,生產(chǎn)環(huán)境服務(wù)器配置性能至少比當前測試機器高2倍,接口性能至少提高1倍,對于千兆組網(wǎng)場景無須進一步優(yōu)化,并發(fā)瓶頸是網(wǎng)絡(luò)帶寬
優(yōu)化file-size-threshold:優(yōu)化為>1MB文件才存入磁盤,測試場景文件全部讀入內(nèi)存,測試上傳文件5KB。此時網(wǎng)絡(luò)吞吐量已大于100MB/s
基準-(form-data):form-data配置簡單key參數(shù),不上傳文件,服務(wù)端接口直接返回簡單字符串。相當于默認情況下form-data參數(shù)類型接口的性能基準,性能瓶頸是磁盤寫入速率
基準-(優(yōu)化file-size-threshold):form-data配置簡單key參數(shù),不上傳文件,服務(wù)端接口直接返回簡單字符串,優(yōu)化為>1MB文件才存入磁盤??梢詫Ρ瓤闯龃疟P與內(nèi)存的速率差異
基準-空接口:普通的get無參接口,直接返回“hello”,作為當前配置環(huán)境下,tomcat接口性能極限
現(xiàn)場問題處理方案
經(jīng)過定位現(xiàn)場性能瓶頸是網(wǎng)絡(luò)
?,F(xiàn)場采用分布式架構(gòu),客戶端、服務(wù)端部署多個節(jié)點,客戶端通過本地回環(huán)地址調(diào)用服務(wù)端,降低網(wǎng)絡(luò)壓力。
原架構(gòu)
新架構(gòu)
以上就是SpringBoot文件上傳接口并發(fā)性能調(diào)優(yōu)的詳細內(nèi)容,更多關(guān)于SpringBoot接口性能調(diào)優(yōu)的資料請關(guān)注腳本之家其它相關(guān)文章!
相關(guān)文章
idea2022創(chuàng)建javaweb項目步驟(超詳細)
本文主要介紹了idea2022創(chuàng)建javaweb項目步驟,文中通過示例代碼介紹的非常詳細,對大家的學習或者工作具有一定的參考學習價值,需要的朋友們下面隨著小編來一起學習學習吧2023-07-07mybatis-plus如何修改日志只打印SQL語句不打印查詢結(jié)果
這篇文章主要介紹了mybatis-plus如何修改日志只打印SQL語句不打印查詢結(jié)果問題,具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教2024-06-06SpringBoot使用Redis單機版過期鍵監(jiān)聽事件的實現(xiàn)示例
在緩存的使用場景中經(jīng)常需要使用到過期事件,本文主要介紹了SpringBoot使用Redis單機版過期鍵監(jiān)聽事件的實現(xiàn)示例,文中通過示例代碼介紹的非常詳細,對大家的學習或者工作具有一定的參考學習價值,需要的朋友們下面隨著小編來一起學習學習吧2024-07-07