php多进程上传,php多进程插入数据

港控/mmm° 2022-11-17 14:33 202阅读 0赞

个人在虚拟机centos7，单核，1G内存/\*\*

\* 模拟并发请求，10万次写入数据库

\* 拆分为10个进程，每个进程处理一万条插入

\*/

$total = 10000;

$num = 10;

$per = $total/$num;

$sql = '';

$child = '';

echo 'start '.microtime(true).PHP\_EOL;

for($i = 1; $i<= $num; $i++)

$pid = pcntl\_fork();

if($pid == -1) \{

die('fork error');

if($pid > 0) \{

//$id = pcntl\_wait($status,WNOHANG);

$child\[\] = $pid;

\} else if ($pid == 0) \{

$link = mysqli\_connect('localhost','root','root','yii2advanced');

$start = ($i-1)\*$per + 1;

$end = $start + $per;

for($j = $start; $j< $end; $j++)\{

$time = microtime(true);

$sql = 'insert pcntl\_test (rank,time) values ('.$j.','.$time.')';

mysqli\_query($link,$sql);

mysqli\_close($link);

$id = getmypid();

echo 'child '.$id.' finished '.microtime(true).PHP\_EOL;

exit(0);

while(count($child))\{

foreach($child as $k => $pid) \{

$res = pcntl\_waitpid($pid, $status, WNOHANG);

if ( -1 == $res || $res > 0) \{

unset($child\[$k\]);

echo 'end '.microtime(true).PHP\_EOL;

当$total=10000，$num = 10;执行结果如下:start 1491903371.5548

child 19860 finished 1491903417.2113

child 19857 finished 1491903417.6909

child 19864 finished 1491903417.7793

child 19855 finished 1491903417.8695

child 19859 finished 1491903417.9162

child 19861 finished 1491903418.0089

child 19856 finished 1491903418.0532

child 19863 finished 1491903418.0842

child 19862 finished 1491903418.1474

child 19858 finished 1491903418.4341

end 1491903418.4424

总时间为46.88759994506836秒

当$total=10000,$num = 100时，执行结果如下：start 1491904334.1735

child 20085 finished 1491904337.0712

child 20086 finished 1491904337.144

……

child 20262 finished 1491904341.5602

child 20264 finished 1491904341.5803

end 1491904341.5869

总时间为7.413399934768677

当$total=10000,$num = 1000时，执行结果如下：start 1491904562.0166

child 20282 finished 1491904562.1191

child 20277 finished 1491904562.1268

child 20279 finished 1491904562.1352

...

child 21586 finished 1491904576.6954

child 21582 finished 1491904576.7024

child 21584 finished 1491904576.7226

end 1491904576.7297

总时间为14.71310019493103，相比100个子进程，耗时更长了。进程切换太多，影响了了效率应该是原因之一。

当$total=100000 ,$num=100时，十万条记录，100个进程插入start 1491905670.2652

child 21647 finished 1491905725.4382

child 21651 finished 1491905725.4595

child 21642 finished 1491905725.5402

....

child 21810 finished 1491905729.7709

child 21812 finished 1491905729.8498

child 21811 finished 1491905729.9612

end 1491905729.9679

总时间为59.70270013809204

单进程插入1万条数据，耗时18秒，相对10个进程插入1万记录来说，耗时少些。

而单进程插入10万条记录，耗时187.40066790581，相对来说，是挺慢的了。三分钟。。。

不过，本人再fork 1000个进程，来插入10万记录时，成功的情况下36秒左右，也可能会出现错误，mysqli\_connection返回false，是不是连接数受限制了？

fork 一万个子进程，插入一百万数据，这时，出现连接错的情况就很多了。最后耗时360秒，数据表中插入了945300条记录,成功率94.53%。于是查看数据库的相关配置信息mysql> show global status like '%connect%';

\+-----------------------------------------------+---------------------+

| Variable\_name | Value |

\+-----------------------------------------------+---------------------+

| Aborted\_connects | 0 |

| Connection\_errors\_accept | 0 |

| Connection\_errors\_internal | 0 |

| Connection\_errors\_max\_connections | 628 |

| Connection\_errors\_peer\_address | 0 |

| Connection\_errors\_select | 0 |

| Connection\_errors\_tcpwrap | 0 |

| Connections | 16519 |

| Locked\_connects | 0 |

| Max\_used\_connections | 501 |

| Max\_used\_connections\_time | 2017-04-12 15:19:54 |

| Performance\_schema\_session\_connect\_attrs\_lost | 0 |

| Ssl\_client\_connects | 0 |

| Ssl\_connect\_renegotiates | 0 |

| Ssl\_finished\_connects | 0 |

| Threads\_connected | 4 |

\+-----------------------------------------------+---------------------+

mysql> show global variables like '%connect%';

\+-----------------------------------------------+--------------------+

| Variable\_name | Value |

\+-----------------------------------------------+--------------------+

| character\_set\_connection | utf8mb4 |

| collation\_connection | utf8mb4\_general\_ci |

| connect\_timeout | 10 |

| disconnect\_on\_expired\_password | ON |

| init\_connect | |

| max\_connect\_errors | 100 |

| max\_connections | 500 |

| max\_user\_connections | 0 |

| performance\_schema\_session\_connect\_attrs\_size | 512 |

\+-----------------------------------------------+--------------------+

修改 myqsql 配置文件，/etc/my.cnf

把max\_connections 改为10000，然后重启mysql

实际MySQL服务器允许的最大连接数16384；

结果然并卵，虚拟机好像挂了了。

并发量大的时候，问题就出在了连接mysql这里。

可以通过一个连接池来尝试解决该问题。

个人在虚拟机centos7，单核，1G内存/\*\*

\* 模拟并发请求，10万次写入数据库

\* 拆分为10个进程，每个进程处理一万条插入

\*/

$total = 10000;

$num = 10;

$per = $total/$num;

$sql = '';

$child = '';

echo 'start '.microtime(true).PHP\_EOL;

for($i = 1; $i<= $num; $i++)

$pid = pcntl\_fork();

if($pid == -1) \{

die('fork error');

if($pid > 0) \{

//$id = pcntl\_wait($status,WNOHANG);

$child\[\] = $pid;

\} else if ($pid == 0) \{

$link = mysqli\_connect('localhost','root','root','yii2advanced');

$start = ($i-1)\*$per + 1;

$end = $start + $per;

for($j = $start; $j< $end; $j++)\{

$time = microtime(true);

$sql = 'insert pcntl\_test (rank,time) values ('.$j.','.$time.')';

mysqli\_query($link,$sql);

mysqli\_close($link);

$id = getmypid();

echo 'child '.$id.' finished '.microtime(true).PHP\_EOL;

exit(0);

while(count($child))\{

foreach($child as $k => $pid) \{

$res = pcntl\_waitpid($pid, $status, WNOHANG);

if ( -1 == $res || $res > 0) \{

unset($child\[$k\]);

echo 'end '.microtime(true).PHP\_EOL;

当$total=10000，$num = 10;执行结果如下:start 1491903371.5548

child 19860 finished 1491903417.2113

child 19857 finished 1491903417.6909

child 19864 finished 1491903417.7793

child 19855 finished 1491903417.8695

child 19859 finished 1491903417.9162

child 19861 finished 1491903418.0089

child 19856 finished 1491903418.0532

child 19863 finished 1491903418.0842

child 19862 finished 1491903418.1474

child 19858 finished 1491903418.4341

end 1491903418.4424

总时间为46.88759994506836秒

当$total=10000,$num = 100时，执行结果如下：start 1491904334.1735

child 20085 finished 1491904337.0712

child 20086 finished 1491904337.144

……

child 20262 finished 1491904341.5602

child 20264 finished 1491904341.5803

end 1491904341.5869

总时间为7.413399934768677

当$total=10000,$num = 1000时，执行结果如下：start 1491904562.0166

child 20282 finished 1491904562.1191

child 20277 finished 1491904562.1268

child 20279 finished 1491904562.1352

...

child 21586 finished 1491904576.6954

child 21582 finished 1491904576.7024

child 21584 finished 1491904576.7226

end 1491904576.7297

总时间为14.71310019493103，相比100个子进程，耗时更长了。进程切换太多，影响了了效率应该是原因之一。

当$total=100000 ,$num=100时，十万条记录，100个进程插入start 1491905670.2652

child 21647 finished 1491905725.4382

child 21651 finished 1491905725.4595

child 21642 finished 1491905725.5402

....

child 21810 finished 1491905729.7709

child 21812 finished 1491905729.8498

child 21811 finished 1491905729.9612

end 1491905729.9679

总时间为59.70270013809204

单进程插入1万条数据，耗时18秒，相对10个进程插入1万记录来说，耗时少些。

而单进程插入10万条记录，耗时187.40066790581，相对来说，是挺慢的了。三分钟。。。

不过，本人再fork 1000个进程，来插入10万记录时，成功的情况下36秒左右，也可能会出现错误，mysqli\_connection返回false，是不是连接数受限制了？

\+-----------------------------------------------+---------------------+

| Variable\_name | Value |

\+-----------------------------------------------+---------------------+

| Aborted\_connects | 0 |

| Connection\_errors\_accept | 0 |

| Connection\_errors\_internal | 0 |

| Connection\_errors\_max\_connections | 628 |

| Connection\_errors\_peer\_address | 0 |

| Connection\_errors\_select | 0 |

| Connection\_errors\_tcpwrap | 0 |

| Connections | 16519 |

| Locked\_connects | 0 |

| Max\_used\_connections | 501 |

| Max\_used\_connections\_time | 2017-04-12 15:19:54 |

| Performance\_schema\_session\_connect\_attrs\_lost | 0 |

| Ssl\_client\_connects | 0 |

| Ssl\_connect\_renegotiates | 0 |

| Ssl\_finished\_connects | 0 |

| Threads\_connected | 4 |

\+-----------------------------------------------+---------------------+

mysql> show global variables like '%connect%';

\+-----------------------------------------------+--------------------+

| Variable\_name | Value |

\+-----------------------------------------------+--------------------+

| character\_set\_connection | utf8mb4 |

| collation\_connection | utf8mb4\_general\_ci |

| connect\_timeout | 10 |

| disconnect\_on\_expired\_password | ON |

| init\_connect | |

| max\_connect\_errors | 100 |

| max\_connections | 500 |

| max\_user\_connections | 0 |

| performance\_schema\_session\_connect\_attrs\_size | 512 |

\+-----------------------------------------------+--------------------+

修改 myqsql 配置文件，/etc/my.cnf

把max\_connections 改为10000，然后重启mysql

实际MySQL服务器允许的最大连接数16384；

结果然并卵，虚拟机好像挂了了。

并发量大的时候，问题就出在了连接mysql这里。

可以通过一个连接池来尝试解决该问题。

php多进程上传,php多进程插入数据

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关 php多进程处理

相关 php多进程处理

相关 PHP之多进程

相关 php多进程实现

相关 php多进程处理

相关 php多进程使用

相关 PHP多进程、PHP协程

相关 php使用多进程

相关 php多进程上传,php多进程插入数据

相关 php中实现多进程