数据挖掘（数据预处理,特征工程）

我不是女神ヾ 2024-04-17 21:51 90阅读 0赞

1.StandardScaler  
确保处理后的特征均值为0，方差为1，但是不确保特征任何特定的最大，最小值

2.RobustScaler  
与StandardScaler类似，确保所有特征scale在同一规模，但是使用属性median，quartiles，  
忽视outliers的影响

3.MinMaxScaler  
将特征值范围scale到0,1

4.Normalizer  
将数据点映射到半径为1的圆；一般在数据的方向影响预测时使用

5.PCA  
进行特征选取;  
使用pca之前需要对数据scale

6.Non-negative matrix factorization（NMF）  
提取有用特征；  
使用必须确保数据是正值的

7.Binning, Discretization（将连续值转为离散值）  
对linear model有提升作用，对DT没用；  
当数据集很大，维度很高，有一些特征与输出有非线性关系时，有效；

bins = np.linspace(-3, 3, 11) #划分10个bins
    which_bin = np.digitize(X, bins=bins)#将X中的值分到所属的bin
    
    from sklearn.preprocessing import OneHotEncoder
    # transform using the OneHotEncoder
    encoder = OneHotEncoder(sparse=False)
    # encoder.fit finds the unique values that appear in which_bin
    encoder.fit(which_bin)
    # transform creates the one-hot encoding
    X_binned = encoder.transform(which_bin)
    print(X_binned[:5])
    
    reg = LinearRegression().fit(X_binned, y)

![在这里插入图片描述][20190901194828533.png]  
8.PolynomialFeatures(degree=2)  
多项式特征，一般只对线性模型，naive bayes有效, tree-based的模型能自行找到这种特征间的交互关系，不需要特意转换数据

9.考虑对数据进行log，exp转换

10.按百分比选取特征

from sklearn.feature_selection import SelectPercentile
    
    select = SelectPercentile(percentile=75)
    select.fit(train_x, train_y)
    # transform training set
    train_x_selected = select.transform(train_x)

11.利用模型选择特征

from sklearn.feature_selection import SelectFromModel
    from sklearn.ensemble import RandomForestClassifier
    select = SelectFromModel(
    RandomForestClassifier(n_estimators=100, random_state=42),
    threshold="median") #median选择一半的特征
    
    select.fit(X_train, y_train)
    X_train_l1 = select.transform(X_train)
    print("X_train.shape: {}".format(X_train.shape))
    print("X_train_l1.shape: {}".format(X_train_l1.shape))

![在这里插入图片描述][20190902181959864.png]  
12.recursive feature elimination (RFE)

from sklearn.feature_selection import RFE
    select = RFE(RandomForestClassifier(n_estimators=100, random_state=42),
    n_features_to_select=40) #选择40个特征
    select.fit(X_train, y_train)

13.使用Pearson Correlation选取特征

def cor_selector(X, y):
        cor_list = []
        # calculate the correlation with y for each feature
        for i in X.columns.tolist():
            cor = np.corrcoef(X[i], y)[0, 1]
            cor_list.append(cor)
        # replace NaN with 0
        cor_list = [0 if np.isnan(i) else i for i in cor_list]
        # feature name
        cor_feature = X.iloc[:,np.argsort(np.abs(cor_list))[-100:]].columns.tolist() #选择相关性最大的100个特征
        # feature selection? 0 for not select, 1 for select
        cor_support = [True if i in cor_feature else False for i in feature_name]
        return cor_support, cor_feature
    
    _, cor_feature = cor_selector(train_x, train_y['isFraud'])
    train_x = train_x[cor_feature]

14.使用Label Encoding将category类特征值转为数值型

from sklearn import preprocessing
    
    # Label Encoding
    for f in df_train.drop('isFraud', axis=1).columns:
        if df_train[f].dtype=='object' or df_test[f].dtype=='object': 
            lbl = preprocessing.LabelEncoder()
            lbl.fit(list(df_train[f].values) + list(df_test[f].values))
            df_train[f] = lbl.transform(list(df_train[f].values))
            df_test[f] = lbl.transform(list(df_test[f].values))

15.减少dataframe占用的存储

## Function to reduce the DF size
    def reduce_mem_usage(df, verbose=True):
        numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
        start_mem = df.memory_usage().sum() / 1024**2    
        for col in df.columns:
            col_type = df[col].dtypes
            if col_type in numerics:
                c_min = df[col].min()
                c_max = df[col].max()
                if str(col_type)[:3] == 'int':
                    if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                        df[col] = df[col].astype(np.int8)
                    elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                        df[col] = df[col].astype(np.int16)
                    elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                        df[col] = df[col].astype(np.int32)
                    elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                        df[col] = df[col].astype(np.int64)  
                else:
                    if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                        df[col] = df[col].astype(np.float16)
                    elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                        df[col] = df[col].astype(np.float32)
                    else:
                        df[col] = df[col].astype(np.float64)    
        end_mem = df.memory_usage().sum() / 1024**2
        if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))
        return df

[20190901194828533.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/8e8d3044c6f64ddc9be06262645d36a5.png
[20190902181959864.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/6dee8163fe074fb2b96e92f4367052cf.png