推荐系统笔记——推荐引擎之实现基本的实时处理（二）

Love The Way You Lie 2022-03-18 11:17 258阅读 0赞

## 流式处理 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE0OTk3NDcz_size_16_color_FFFFFF_t_70][]

优势：低响应时延、动态效果丰富、用户体验效果好、转化率高

劣势：资源消耗严重、效果准确度有影响、效果起伏大且分析原因难

如何打造自己的流式处理系统：

• 设计场景和梳理需求（如视频网站场景、在线阅读小说的场景）  
• 制定流式处理规则（如点击流式处理规则、收藏流式处理规则）  
• 人工干预推荐结果（编辑推荐）

应用流式处理：重定向、类别关联、同类目推荐、人工干预推荐结果

流式处理实现：内存更新方式、一致性哈希算法、更新缓存机制

## 内存更新方式 ##

内存更新方式：map、list

## 一致性哈希算法 ##

参考博客：[https://blog.csdn.net/cywosp/article/details/23397179][https_blog.csdn.net_cywosp_article_details_23397179]

构造环形Hash空间，把数据通过Hash算法映射到环上，按照顺时针方向将所有对象存储到离自己最近的机器中。

考虑节点的添加删除以及平衡性。

![SouthEast][]

## 模拟实现流式处理 ##

Server

# -*- coding: utf-8 -*-
    import socket
    
    HOST,PORT = "", 8888
    
    listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    listen_socket.bind((HOST, PORT))
    listen_socket.listen(1)
    print("Serving HTTP on port %s ..."%PORT)
    
    def Processing(request):
        uid = request.split("key=")[1].split(" ")[0]
        return uid
    
    def gp(uid):
        click_action = {}  # key = uid, value = video_ids
        file = open("./logfile.txt", "r")
        for line in file.readlines():
            line = line.strip()
            ls = line.split("&")    # len(ls) = 8
            if ls[7] != "1":
                continue
            if ls[1] not in click_action.keys():
                click_action[ls[1]] = []
            click_action[ls[1]].append(ls[4])
        if uid in click_action.keys():
            return "&&".join(click_action[uid])
        
    comment_log = {}
    
    def log_process(request):
        print("xiao ming's log")
        print(request)
        ls = request.split("&")
        if ls[1] not in comment_log.keys():
            comment_log[ls[1]] = []
        comment_log[ls[1]].append(ls[4])
        for k,v in comment_log.items():
            print("now")
            print(k + "\t" + "&&".join(v))
        return "yes i got it"
    
    while True:
        client_connection, client_address = listen_socket.accept()
        request = client_connection.recv(1024)
        #http_response = gp(request)
        #http_response = log_process(request)
    
        client_connection.sendall(b'hi')
        #client_connection.sendall(http_responsei)
        client_connection.close()

Client

# -*- coding: utf-8 -*-
    import socket
    import os
    
    ss = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    ss.connect(('192.168.0.1',8888)) #改成自己电脑的ip
    ss.sendall(r"ebLfaG&five&Macintosh Chrome Safari&1.1.1.1&898923&苹果发布会&0&7")
    #os.system('sleep 1')
    ss.send(b'EOF')
    data=ss.recv(1024)
    print("server's response : %s "%data)
    ss.close()

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE0OTk3NDcz_size_16_color_FFFFFF_t_70]: /images/20220318/bcb971ca32794e6e9ab00bc54ecbb036.png
[https_blog.csdn.net_cywosp_article_details_23397179]: https://blog.csdn.net/cywosp/article/details/23397179
[SouthEast]: /images/20220318/e3cfbb8a3f464010911d26ac09064ae3.png