Spark ML 正则化标准化归一化 ---- spark 中的归一化

Myth丶恋晨 2022-09-09 04:38 240阅读 0赞

### 文章大纲 ###

*  spark 中的归一化
 *   *  MaxAbsScaler
     *  MinMaxScaler
 *  参考文献

--------------------

# spark 中的归一化 #

## MaxAbsScaler ##

*  http://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/feature/MaxAbsScaler.html

## MinMaxScaler ##

*  http://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/feature/MinMaxScaler.html

Rescale each feature individually to a common range \[min, max\] linearly using column summary  
statistics, which is also known as min-max normalization or Rescaling. The rescaled value for  
feature E is calculated as:  
 R e s c a l e d ( e i ) = e i − E m i n E m a x − E m i n ∗ ( m a x − m i n ) + m i n Rescaled(e\_i) = \\frac\{e\_i - E\_\{min\}\}\{E\_\{max\} - E\_\{min\}\} \* (max - min) + min Rescaled(ei)=Emax−Eminei−Emin∗(max−min)\+min

For the case (E\_\{max\} == E\_\{min\}), (Rescaled(e\_i) = 0.5 \* (max + min)).  
note ：

> Since zero values will probably be transformed to non-zero values, output of the  
> transformer will be DenseVector even for sparse input.

核心代码：主要就是计算 最大最小值

override def fit(dataset: Dataset[_]): MinMaxScalerModel = {
        
        transformSchema(dataset.schema, logging = true)
    
        val Row(max: Vector, min: Vector) = dataset
          .select(Summarizer.metrics("max", "min").summary(col($(inputCol))).as("summary"))
          .select("summary.max", "summary.min")
          .first()
    
        copyValues(new MinMaxScalerModel(uid, min.compressed, max.compressed).setParent(this))
      }

注意： 上面的计算方式， 我们发现只能支持Vector的形式，那么对于但一值的情况如何转换呢？

val temp_mean = df_num.select(functions.mean(df_num.col("features"))).collect()(0)
            println(temp_mean.getDouble(0))
           val Row(mean2: Vector) =Row(Vectors.dense(temp_mean.getDouble(0)))
    
       val df_num = spark.createDataFrame(Seq(
                (0, 0.5, -1.0),
                (1, 1.0, 1.0),
                (2, 10.0, 2.0),
                (3, 10.0, 0.0)
            )).toDF("id", "features","result")
            df.show()

--------------------

# 参考文献 #

系列文章：

*  [正则化、标准化、归一化基本概念简介][Link 1]
 *  [ spark 中的正则化][spark]
 *  [spark 中的标准化][spark 1]
 *  [spark 中的归一化][spark 2]
 *  [扩展spark 的归一化函数][spark 3]

spark 中的 特征相关内容处理的文档

*  http://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/feature/index.html

概念简介

*  https://blog.csdn.net/u014381464/article/details/81101551

参考：

*  https://segmentfault.com/a/1190000014042959
 *  https://www.cnblogs.com/nucdy/p/7994542.html
 *  https://blog.csdn.net/weixin\_34117522/article/details/88875270
 *  https://blog.csdn.net/xuejianbest/article/details/85779029

[Link 1]: https://blog.csdn.net/wangyaninglm/article/details/119728871
[spark]: https://season.blog.csdn.net/article/details/119975289
[spark 1]: https://season.blog.csdn.net/article/details/119975304
[spark 2]: https://blog.csdn.net/wangyaninglm/article/details/119975312
[spark 3]: https://blog.csdn.net/wangyaninglm/article/details/119975330