python threading库_python中threading和queue库实现多线程编程

秒速五厘米 2022-10-22 11:51 151阅读 0赞

摘要

本文主要介绍了利用python的 threading和queue库实现多线程编程，并封装为一个类，方便读者嵌入自己的业务逻辑。最后以机器学习的一个超参数选择为例进行演示。

多线程实现逻辑封装

实例化该类后，在.object\_func函数中加入自己的业务逻辑，再调用.run方法即可。

\# -\*- coding: utf-8 -\*-

\# @Time : 2021/2/4 14:36

\# @Author : CyrusMay WJ

\# @FileName: run.py

\# @Software: PyCharm

\# @Blog ：https://blog.csdn.net/Cyrus\_May

import queue

import threading

class CyrusThread(object):

def \_\_init\_\_(self,num\_thread = 10,logger=None):

"""

:param num\_thread: 线程数

:param logger: 日志对象

"""

self.num\_thread = num\_thread

self.logger = logger

def object\_func(self,args\_queue,max\_q):

while 1:

try:

arg = args\_queue.get\_nowait()

step = args\_queue.qsize()

self.logger.info("progress:\{\}\\\{\}".format(max\_q,step))

except:

self.logger.info("no more arg for args\_queue!")

break

"""

此处加入自己的业务逻辑代码

"""

def run(self,args):

args\_queue = queue.Queue()

for value in args:

args\_queue.put(value)

threads = \[\]

for i in range(self.num\_thread):

threads.append(threading.Thread(target=self.object\_func,args = args\_queue))

for t in threads:

t.start()

for t in threads:

t.join()

模型参数选择实例

\# -\*- coding: utf-8 -\*-

\# @Time : 2021/2/4 14:36

\# @Author : CyrusMay WJ

\# @FileName: run.py

\# @Software: PyCharm

\# @Blog ：https://blog.csdn.net/Cyrus\_May

import queue

import threading

import numpy as np

from sklearn.datasets import load\_boston

from sklearn.svm import SVR

import logging

import sys

class CyrusThread(object):

def \_\_init\_\_(self,num\_thread = 10,logger=None):

"""

:param num\_thread: 线程数

:param logger: 日志对象

"""

self.num\_thread = num\_thread

self.logger = logger

def object\_func(self,args\_queue,max\_q):

while 1:

try:

arg = args\_queue.get\_nowait()

step = args\_queue.qsize()

self.logger.info("progress:\{\}\\\{\}".format(max\_q,max\_q-step))

except:

self.logger.info("no more arg for args\_queue!")

break

\# 业务代码

C, epsilon, gamma = arg\[0\], arg\[1\], arg\[2\]

svr\_model = SVR(C=C, epsilon=epsilon, gamma=gamma)

x, y = load\_boston()\["data"\], load\_boston()\["target"\]

svr\_model.fit(x, y)

self.logger.info("score:\{\}".format(svr\_model.score(x,y)))

def run(self,args):

args\_queue = queue.Queue()

max\_q = 0

for value in args:

args\_queue.put(value)

max\_q += 1

threads = \[\]

for i in range(self.num\_thread):

threads.append(threading.Thread(target=self.object\_func,args = (args\_queue,max\_q)))

for t in threads:

t.start()

for t in threads:

t.join()

\# 创建日志对象

logger = logging.getLogger()

logger.setLevel(logging.INFO)

screen\_handler = logging.StreamHandler(sys.stdout)

screen\_handler.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(message)s')

screen\_handler.setFormatter(formatter)

logger.addHandler(screen\_handler)

\# 创建需要调整参数的集合

args = \[\]

for C in \[i for i in np.arange(0.01,1,0.01)\]:

for epsilon in \[i for i in np.arange(0.001,1,0.01)\] + \[i for i in range(1,10,1)\]:

for gamma in \[i for i in np.arange(0.001,1,0.01)\] + \[i for i in range(1,10,1)\]:

args.append(\[C,epsilon,gamma\])

\# 创建多线程工具

threading\_tool = CyrusThread(num\_thread=20,logger=logger)

threading\_tool.run(args)

运行结果

2021-02-04 20:52:22,824 - run.object\_func:31 - INFO - progress:1176219\\1

2021-02-04 20:52:22,824 - run.object\_func:31 - INFO - progress:1176219\\2

2021-02-04 20:52:22,826 - run.object\_func:31 - INFO - progress:1176219\\3

2021-02-04 20:52:22,833 - run.object\_func:31 - INFO - progress:1176219\\4

2021-02-04 20:52:22,837 - run.object\_func:31 - INFO - progress:1176219\\5

2021-02-04 20:52:22,838 - run.object\_func:31 - INFO - progress:1176219\\6

2021-02-04 20:52:22,841 - run.object\_func:31 - INFO - progress:1176219\\7

2021-02-04 20:52:22,862 - run.object\_func:31 - INFO - progress:1176219\\8

2021-02-04 20:52:22,873 - run.object\_func:31 - INFO - progress:1176219\\9

2021-02-04 20:52:22,884 - run.object\_func:31 - INFO - progress:1176219\\10

2021-02-04 20:52:22,885 - run.object\_func:31 - INFO - progress:1176219\\11

2021-02-04 20:52:22,897 - run.object\_func:31 - INFO - progress:1176219\\12

2021-02-04 20:52:22,900 - run.object\_func:31 - INFO - progress:1176219\\13

2021-02-04 20:52:22,904 - run.object\_func:31 - INFO - progress:1176219\\14

2021-02-04 20:52:22,912 - run.object\_func:31 - INFO - progress:1176219\\15

2021-02-04 20:52:22,920 - run.object\_func:31 - INFO - progress:1176219\\16

2021-02-04 20:52:22,920 - run.object\_func:39 - INFO - score:-0.01674283914287855

2021-02-04 20:52:22,929 - run.object\_func:31 - INFO - progress:1176219\\17

2021-02-04 20:52:22,932 - run.object\_func:39 - INFO - score:-0.007992354170952565

2021-02-04 20:52:22,932 - run.object\_func:31 - INFO - progress:1176219\\18

2021-02-04 20:52:22,945 - run.object\_func:31 - INFO - progress:1176219\\19

2021-02-04 20:52:22,954 - run.object\_func:31 - INFO - progress:1176219\\20

2021-02-04 20:52:22,978 - run.object\_func:31 - INFO - progress:1176219\\21

2021-02-04 20:52:22,984 - run.object\_func:39 - INFO - score:-0.018769934807246536

2021-02-04 20:52:22,985 - run.object\_func:31 - INFO - progress:1176219\\22

到此这篇关于python中threading和queue库实现多线程编程的文章就介绍到这了,更多相关python 多线程编程内容请搜索聚米学院以前的文章或继续浏览下面的相关文章希望大家以后多多支持聚米学院！

python threading库_python中threading和queue库实现多线程编程

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关 Python多线程：threading库使用示例

相关 Python多线程Thread

相关 Python基础知识——多线程threading和thread

相关 Python多线程编程---（1）threading 模块 Thread 类

相关 Python多线程编程threading类

相关 python多线程包_8.Python标准库_多线程与同步 (threading包)

相关 Python标准库08 多线程与同步 (threading包)

相关 python threading库_python中threading和queue库实现多线程编程

相关 Python多线程thread与threading实现

相关 python3多线程threading、_thread、Queue