机器学习——模型评估方法

落日映苍穹つ 2024-04-18 06:54 84阅读 0赞

一：confusion\_matrix

from sklearn.metrics import confusion_matrix
    confusion = confusion_matrix(y_test, pred_logreg)
    print("Confusion matrix:\n{}".format(confusion))

![在这里插入图片描述][2019090310340112.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70]  
二：Accuracy  
![在这里插入图片描述][20190903103932462.png]  
三：Precision, recall, and f-score.（评估分类问题）  
（1）**precision**：positive predictive value (PPV).  
![在这里插入图片描述][20190903104029669.png]  
（2）**recall**： sensitivity, hit rate, or true positive rate (TPR).  
![在这里插入图片描述][20190903104044231.png]  
（3）**f-score**  
![在这里插入图片描述][20190903104757926.png]  
（4）classification\_report可以同时得到这三个值

from sklearn.metrics import classification_report
    print(classification_report(y_test, pred_most_frequent,
    target_names=["not nine", "nine"])) #nine是positive class

![在这里插入图片描述][20190903105341842.png]  
三：Precision-recall curves and ROC curves

from sklearn.metrics import precision_recall_curve
    precision, recall, thresholds = precision_recall_curve(
    y_test, svc.decision_function(X_test))

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 1]  
四：计算ROC曲线下的面积

from sklearn.metrics import average_precision_score
    ap_rf = average_precision_score(y_test, rf.predict_proba(X_test)[:, 1])
    ap_svc = average_precision_score(y_test, svc.decision_function(X_test))
    print("Average precision of random forest: {:.3f}".format(ap_rf))
    print("Average precision of svc: {:.3f}".format(ap_svc))

![在这里插入图片描述][20190903111442901.png]  
五：Receiver operating characteristics (ROC) and AUC（area under curve）  
![在这里插入图片描述][20190903111618757.png]

from sklearn.metrics import roc_curve
    fpr, tpr, thresholds = roc_curve(y_test, svc.decision_function(X_test))
    plt.plot(fpr, tpr, label="ROC Curve")
    plt.xlabel("FPR")
    plt.ylabel("TPR (recall)")
    # find threshold closest to zero
    close_zero = np.argmin(np.abs(thresholds))
    plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
    label="threshold zero", fillstyle="none", c='k', mew=2)
    plt.legend(loc=4)

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 2]  
六：计算ROC曲线下的面积  
**AUC用来评估不均衡数据集时效果较好**

from sklearn.metrics import roc_auc_score
    rf_auc = roc_auc_score(y_test, rf.predict_proba(X_test)[:, 1])
    svc_auc = roc_auc_score(y_test, svc.decision_function(X_test))
    print("AUC for Random Forest: {:.3f}".format(rf_auc))
    print("AUC for SVC: {:.3f}".format(svc_auc))

![在这里插入图片描述][20190903111955260.png]  
七：多分类问题的accuracy，confusion\_matrix  
（1）accuracy，confusion\_matrix

from sklearn.metrics import accuracy_score
    X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, random_state=0)
    lr = LogisticRegression().fit(X_train, y_train)
    pred = lr.predict(X_test)
    print("Accuracy: {:.3f}".format(accuracy_score(y_test, pred)))
    print("Confusion matrix:\n{}".format(confusion_matrix(y_test, pred)))

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 3]  
![在这里插入图片描述][20190903113323669.png]  
（2）使用classification\_report给每个类计算precision, recall,和f-score

print(classification_report(y_test, pred))

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 4]  
（3）计算多分类的mean f1-score  
\#三种average参数：macro, weighted, micro  
\#micro:若平均考虑每个样本（每个样本权重一样）  
\#macro:若平均考虑每个类

print("Micro average f1 score: {:.3f}".format(f1_score(y_test, pred, average="micro")))
    print("Macro average f1 score: {:.3f}".format(f1_score(y_test, pred, average="macro")))

![在这里插入图片描述][20190904094746555.png]  
八：回归问题  
大部分情况，使用[r2\_score][r2_score]就可以了，还有其他选择：neg\_mean\_absolute\_error，neg\_mean\_squared\_error…

from sklearn.metrics import r2_score
    y_true = [3, -0.5, 2, 7]
    y_pred = [2.5, 0.0, 2, 8]
    r2_score(y_true, y_pred)

九：使用AUC作为cross\_val\_score的评价函数

# default scoring for classification is accuracy
    print("Default scoring: {}".format(
    cross_val_score(SVC(), digits.data, digits.target == 9)))
    # providing scoring="accuracy" doesn't change the results
    explicit_accuracy = cross_val_score(SVC(), digits.data, digits.target == 9,
    scoring="accuracy")
    print("Explicit accuracy scoring: {}".format(explicit_accuracy))
    roc_auc = cross_val_score(SVC(), digits.data, digits.target == 9,
    scoring="roc_auc")
    print("AUC scoring: {}".format(roc_auc))

![在这里插入图片描述][20190904101209515.png]  
十：AUC用在GridSearchCV上

X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target == 9, random_state=0)
    # we provide a somewhat bad grid to illustrate the point:
    param_grid = {'gamma': [0.0001, 0.01, 0.1, 1, 10]}
    
    # using AUC scoring instead:
    grid = GridSearchCV(SVC(), param_grid=param_grid, scoring="roc_auc")
    grid.fit(X_train, y_train)
    print("\nGrid-Search with AUC")
    print("Best parameters:", grid.best_params_)
    print("Best cross-validation score (AUC): {:.3f}".format(grid.best_score_))
    print("Test set AUC: {:.3f}".format(
    roc_auc_score(y_test, grid.decision_function(X_test))))
    print("Test set accuracy: {:.3f}".format(grid.score(X_test, y_test)))

![在这里插入图片描述][20190904101414452.png]

[2019090310340112.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/c031e5b674db42f58f12d1fc52e9b0aa.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/281b1315a3ae4c51b0007171f8aa49ff.png
[20190903103932462.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/c3c57766c28e467393292a9ed9c7fa04.png
[20190903104029669.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/0db5bbb80d964c2689f04e14cdfc4772.png
[20190903104044231.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/6aea6c52c9164b38b3b04b1b22bac570.png
[20190903104757926.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/0cd37ad14590440c9bb005914d016420.png
[20190903105341842.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/596a997c7d704b6faa5e4e5004762305.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/175c47d904f245f3a718ba69639de044.png
[20190903111442901.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/ae6e9389be584962bcfbb0c48d2503be.png
[20190903111618757.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/14fb5f4be19d40929182046d35e8f01e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/1a58b5c7c9864c6ba58ef124fb909f18.png
[20190903111955260.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/2bb33aea80c7417f8fa7f7a92da4b4ff.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 3]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/198111a07c904eb09aa68a0b9b636bde.png
[20190903113323669.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/135543ae5a724cc69fde1f8a994146c5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p6bGRt_size_16_color_FFFFFF_t_70 4]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/ebc598c0a19c40ebbcdb4cf98cbfb412.png
[20190904094746555.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/5a623c0eb46346e7b95d9a11089c687f.png
[r2_score]: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score
[20190904101209515.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/ae3734dc0d104d85b173e4102b0a1da8.png
[20190904101414452.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/fc6aac817b444ea2b460c51bfb70dbdb.png