快捷導(dǎo)航

PyOD進(jìn)行異常值檢測(cè)使用實(shí)例

更新時(shí)間：2024年02月16日 12:30:24 作者：Deephub

異常值檢測(cè)各個(gè)領(lǐng)域的關(guān)鍵任務(wù)之一,PyOD是Python Outlier Detection的縮寫,可以簡(jiǎn)化多變量數(shù)據(jù)集中識(shí)別異常值的過程,在本文中,我們將介紹PyOD包,并通過實(shí)際給出詳細(xì)的代碼示例

PyOD簡(jiǎn)介

PyOD為異常值檢測(cè)提供了廣泛的算法集合，適用于有監(jiān)督和無(wú)監(jiān)督的場(chǎng)景。無(wú)論處理的是帶標(biāo)簽的數(shù)據(jù)還是未帶標(biāo)簽的數(shù)據(jù)，PyOD都提供了一系列技術(shù)來(lái)滿足特定需求。PyOD的突出特性之一是其用戶友好的API，使新手和有經(jīng)驗(yàn)的從業(yè)者都可以輕松的訪問它。

示例1：kNN

我們從一個(gè)簡(jiǎn)單的例子開始，利用k近鄰(kNN)算法進(jìn)行離群值檢測(cè)。

首先從PyOD導(dǎo)入必要的模塊

 from pyod.models.knn import KNN 
 from pyod.utils.data import generate_data
 from pyod.utils.data import evaluate_print

我們生成具有預(yù)定義離群率的合成數(shù)據(jù)來(lái)模擬異常值。

contamination = 0.1  # percentage of outliers
 n_train = 200  # number of training points
 n_test = 100  # number of testing points
 X_train, X_test, y_train, y_test = generate_data(
     n_train=n_train, n_test=n_test, contamination=contamination)

初始化kNN檢測(cè)器，將其與訓(xùn)練數(shù)據(jù)擬合，并獲得離群值預(yù)測(cè)。

clf_name = 'KNN'
 clf = KNN()
 clf.fit(X_train)

使用ROC和Precision @ Rank n指標(biāo)評(píng)估訓(xùn)練模型在訓(xùn)練和測(cè)試數(shù)據(jù)集上的性能。

print("\nOn Training Data:")
 evaluate_print(clf_name, y_train, clf.decision_scores_)
 print("\nOn Test Data:")
 evaluate_print(clf_name, y_test, clf.decision_function(X_test))

最后可以使用內(nèi)置的可視化功能可視化離群檢測(cè)結(jié)果。

from pyod.utils.data import visualize
 
 visualize(clf_name, X_train, y_train, X_test, y_test, clf.labels_,
           clf.predict(X_test), show_figure=True, save_figure=False)

這是一個(gè)簡(jiǎn)單的用法示例

示例2 模型集成

異常值檢測(cè)有時(shí)會(huì)受到模型不穩(wěn)定性的影響，特別是在無(wú)監(jiān)督的情況下。所以PyOD提供了模型組合技術(shù)來(lái)提高魯棒性。

import numpy as np
 from sklearn.model_selection import train_test_split
 from scipy.io import loadmat
 
 from pyod.models.knn import KNN
 from pyod.models.combination import aom, moa, average, maximization, median
 from pyod.utils.utility import standardizer
 from pyod.utils.data import generate_data
 from pyod.utils.data import evaluate_print
 
 X, y = generate_data(train_only=True)  # load data
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)
 
 # standardizing data for processing
 X_train_norm, X_test_norm = standardizer(X_train, X_test)
 
 n_clf = 20  # number of base detectors
 
 # Initialize 20 base detectors for combination
 k_list = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140,
             150, 160, 170, 180, 190, 200]
 
 train_scores = np.zeros([X_train.shape[0], n_clf])
 test_scores = np.zeros([X_test.shape[0], n_clf])
 
 print('Combining {n_clf} kNN detectors'.format(n_clf=n_clf))
 
 for i in range(n_clf):
     k = k_list[i]
 
     clf = KNN(n_neighbors=k, method='largest')
     clf.fit(X_train_norm)
 
     train_scores[:, i] = clf.decision_scores_
     test_scores[:, i] = clf.decision_function(X_test_norm)
 
 # Decision scores have to be normalized before combination
 train_scores_norm, test_scores_norm = standardizer(train_scores,
                                                     test_scores)
 # Combination by average
 y_by_average = average(test_scores_norm)
 evaluate_print('Combination by Average', y_test, y_by_average)
 
 # Combination by max
 y_by_maximization = maximization(test_scores_norm)
 evaluate_print('Combination by Maximization', y_test, y_by_maximization)
 
 # Combination by median
 y_by_median = median(test_scores_norm)
 evaluate_print('Combination by Median', y_test, y_by_median)
 
 # Combination by aom
 y_by_aom = aom(test_scores_norm, n_buckets=5)
 evaluate_print('Combination by AOM', y_test, y_by_aom)
 
 # Combination by moa
 y_by_moa = moa(test_scores_norm, n_buckets=5)
 evaluate_print('Combination by MOA', y_test, y_by_moa)

如果上面代碼提示錯(cuò)誤，需要安裝combo包