Spark实现ALS最小二乘法

野性酷女 2022-01-06 03:27 272阅读 0赞

## ALS算法流程： ##

*  **初始化数据集和Spark环境**
 *  **切分测试机和检验集**
 *  **训练ALS模型**
 *  **验证结果**
 *  **检验满足结果，直接推荐商品，否则继续训练ALS模型**

## 数据集的含义 ##

数据根式 用户ID，产品ID，评分

1,11,2
    1,12,3
    1,13,1
    1,14,0
    1,15,1
    2,11,2
    2,12,2
    2,13,2
    2,14,1
    2,15,4
    3,11,2
    3,12,3
    3,13,1
    3,14,0
    3,15,1
    4,11,1
    4,12,2
    4,13,2
    4,14,1
    4,15,4
    5,11,1
    5,12,2
    5,14,2
    5,14,2
    5,15,4

## 源码解析 ##

**ALS类的所有字段如下:**

@Since("0.8.0")
    class ALS private (
        private var numUserBlocks: Int,
        private var numProductBlocks: Int,
        private var rank: Int,
        private var iterations: Int,
        private var lambda: Double,
        private var implicitPrefs: Boolean,  使用显式反馈ALS变量或隐式反馈
        private var alpha: Double,    ALS隐式反馈变化率用于控制每次拟合修正的幅度
        private var seed: Long = System.nanoTime()
      ) extends Serializable with Logging {

**ALS.train方法:**

/**
      * Train a matrix factorization model given an RDD of ratings given by users to some products,
      * in the form of (userID, productID, rating) pairs. We approximate the ratings matrix as the
     * product of two lower-rank matrices of a given rank (number of features). To solve for these
      * features, we run a given number of iterations of ALS. This is done using a level of
     * parallelism given by `blocks`.
      *
      * @param ratings    RDD of (userID, productID, rating) pairs
     * @param rank       number of features to use  
     * @param iterations number of iterations of ALS (recommended: 10-20)
     * @param lambda     regularization factor (recommended: 0.01)
    * @param blocks     level of parallelism to split computation into  将并行度分解为等级
     * @param seed       random seed  随机种子
     */
     @Since("0.9.1")
     def train(
          ratings: RDD[Rating], //RDD序列由用户ID 产品ID和评分组成
          rank: Int,    //模型中的隐藏因子数目
          iterations: Int,  //算法迭代次数
          lambda: Double,  //ALS正则化参数
          blocks: Int,   //块
          seed: Long ): MatrixFactorizationModel = {
            new ALS(blocks, blocks, rank, iterations, lambda, false, 1.0, seed).run(ratings)
      }

**基于ALS算法的协同过滤推荐实例代码:**

import org.apache.spark.SparkConf
    import org.apache.spark.SparkContext
    import org.apache.spark.mllib.recommendation.Rating
    import org.apache.spark.mllib.recommendation.ALS
    
    /**
     * ALS最小二乘法
     */
    object CollaborativeFilter {
       /**
        * 程序入口
        */
      def main(args: Array[String]): Unit = {
        val conf=new SparkConf().setAppName("ALS").setMaster("local[*]")
        val context=new SparkContext(conf)
        //第一列位用户编号，第二列位产品编号，第三列的评分Rating为Double类型
        val data=context.textFile("D:\\opt\\data.txt")
        //处理数据
        val ratings=data.map(_.split(",") match{
           //数据集的转换
           case Array(user,item,rate) =>
             //将数据集转化为专用的Rating
             Rating(user.toInt,item.toInt,rate.toDouble)
         })
        //设置隐藏因子
         val rank=4
        //设置迭代次数
        val numIterations=3
        //进行模型训练
        val model =ALS.train(ratings,rank,numIterations,0.01)
       //为用户4推荐一个商品
       val rs=model.recommendProducts(4,1)
       //打印结果
       rs.foreach(println)
       //预测的评分结果
       val result:Double=rs(0).rating   
       val realilty=data.map(_.split(",") match {
          case Array(user,item,rate) =>
            Rating(user.toInt,item.toInt,rate.toDouble)
        }).map(num=>{
          if(num.user==4&&num.product==15)
            num.rating                                            //返回实际评分结果
          else
            0
        }).foreach(num=>{
          if(num!=0)
           println("预测的准确率为："+(1-(math.abs(result-num)/1)))
        })
       context.stop()
      }
    }

## 运行推荐结果: ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyMTY4MDg3_size_16_color_FFFFFF_t_70]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyMTY4MDg3_size_16_color_FFFFFF_t_70]: /images/20211228/42240e5ea7984cb79f05d1eb816e4642.png