Python测试进阶——（6）Bash脚本启动Python监控程序并传递PID

Myth丶恋晨 2021-11-26 16:31 210阅读 0赞

用HiBench执行Hadoop——Sort测试用例，进入 /HiBench-master/bin/workloads/micro/sort/hadoop 目录下，执行命令：

[root@node1 hadoop]# ./run.sh

执行后返回如下信息：

[root@node1 hadoop]# ./run.sh
    patching args=　　#enter_bench()
    Parsing conf: /home/cf/app/HiBench-master/conf/hadoop.conf
    Parsing conf: /home/cf/app/HiBench-master/conf/hibench.conf
    Parsing conf: /home/cf/app/HiBench-master/conf/spark.conf
    Parsing conf: /home/cf/app/HiBench-master/conf/workloads/micro/sort.conf
    probe sleep jar: /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/../../jars/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.14.2-tests.jar
    start HadoopSort bench
    hdfs rm -r: /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/bin/hadoop --config /etc/hadoop/conf.cloudera.yarn fs -rm -r -skipTrash hdfs://node1:8020/HiBench/Sort/Output
    Deleted hdfs://node1:8020/HiBench/Sort/Output　　#rmr_hdfs()
    hdfs du -s: /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/bin/hadoop --config /etc/hadoop/conf.cloudera.yarn fs -du -s hdfs://node1:8020/HiBench/Sort/Input
    Submit MapReduce Job: /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/bin/hadoop --config /etc/hadoop/conf.cloudera.yarn jar /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/../../jars/hadoop-mapreduce-examples-2.6.0-cdh5.14.2.jar sort -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -r 8 hdfs://node1:8020/HiBench/Sort/Input hdfs://node1:8020/HiBench/Sort/Output #run_hadoop_job()
    The job took 14 seconds.
    finish HadoopSort bench

发现HiBench执行Python监控程序脚本的命令为：

UID        PID  PPID  C STIME TTY          TIME CMD
    
    root     32614     1  0 16:02 pts/0    00:00:00 python2 /home/cf/app/HiBench-master/bin/functions/monitor.py HadoopSort 32331 /home/cf/app/HiBench-master/report/sort/hadoop/conf/../monitor.log /home/cf/app/H
    
    root     32621 32331  0 16:02 pts/0    00:00:00 python2 /home/cf/app/HiBench-master/bin/functions/execute_with_log.py /home/cf/app/HiBench-master/report/sort/hadoop/conf/../bench.log /opt/cloudera/parcels/CD

查看文件run.sh的内容：

17 current_dir=`dirname "$0"`
     18 current_dir=`cd "$current_dir"; pwd`
     19 root_dir=${current_dir}/../../../../../
     20 workload_config=${root_dir}/conf/workloads/micro/sort.conf
     21 . "${root_dir}/bin/functions/load_bench_config.sh"
     22 
     23 enter_bench HadoopSort ${workload_config} ${current_dir}
     24 show_bannar start
     25 
     26 rmr_hdfs $OUTPUT_HDFS || true
     27 
     28 SIZE=`dir_size $INPUT_HDFS`
     29 START_TIME=`timestamp`
     30 run_hadoop_job ${HADOOP_EXAMPLES_JAR} sort -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -r ${NUM_REDS} ${INPUT_HDFS} ${OUTPUT_HDFS}
     31 
     32 END_TIME=`timestamp`
     33 gen_report ${START_TIME} ${END_TIME} ${SIZE}
     34 show_bannar finish
     35 leave_bench

在文件run.sh中，发现 run\_hadoop\_job() 调用了 start\_monitor 方法：

查看 start\_monitor 方法的定义：

function start_monitor(){
        MONITOR_PID=`${workload_func_bin}/monitor.py ${HIBENCH_CUR_WORKLOAD_NAME} $$ ${WORKLOAD_RESULT_FOLDER}/monitor.log ${WORKLOAD_RESULT_FOLDER}/bench.log ${WORKLOAD_RESULT_FOLDER}/monitor.html ${SLAVES} &`
    #    echo "start monitor, got child pid:${MONITOR_PID}" > /dev/stderr
        echo ${MONITOR_PID}
    }

还有 stop\_monitor  方法的定义：

function stop_monitor(){
        MONITOR_PID=$1
        assert $1 "monitor pid missing"
    #    echo "stop monitor, kill ${MONITOR_PID}" > /dev/stderr
        kill ${MONITOR_PID}
    }

以及 execute\_withlog 方法的定义：

function execute_withlog () {
        CMD="$@"
        if [ -t 1 ] ; then          # Terminal, beautify the output.
            ${workload_func_bin}/execute_with_log.py ${WORKLOAD_RESULT_FOLDER}/bench.log $CMD
        else                        # pipe, do nothing.
            $CMD
        fi
    }

在 run.sh 中加入以下三行：

32 echo "PID of this script: $$"
     33 echo "PPID of this script: $PPID"
     34 echo "UID of this script: $UID"

文件的部分内容如下：

28 SIZE=`dir_size $INPUT_HDFS`
     29 START_TIME=`timestamp`
     30 run_hadoop_job ${HADOOP_EXAMPLES_JAR} sort -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -r ${NUM_REDS} ${INPUT_HDFS} ${OUTPUT_HDFS}
     31 
     32 echo "PID of this script: $$"
     33 echo "PPID of this script: $PPID"
     34 echo "UID of this script: $UID"
     35 
     36 END_TIME=`timestamp`
     37 gen_report ${START_TIME} ${END_TIME} ${SIZE}
     38 show_bannar finish
     39 leave_bench

执行后返回如下信息：

PID of this script: 32331
    PPID of this script: 18804
    UID of this script: 0

用 ps -ef 查看发现 18804 和 32331 分别对应如下进程：

UID        PID  PPID  C STIME TTY          TIME CMD
    root     18804 18802  0 15:42 pts/0    00:00:00 -bash
    root     32331 18804  0 16:02 pts/0    00:00:00 /bin/bash ./run_bak.sh

经查找，发现：

> 在bash中，子shell进程的PID存储在一个特殊的变量\\$\\$中。这个变量只读，你不可以在脚本中修改它。除了\\$\\$, bash shell还会导出其他的只读变量。比如，\\$PPID存储子shell父进程的ID（也就是主shell）。\\$UID存储了执行这个脚本的当前用户ID。上面输出中，PID每次执行都会变化。这个因为每次运行都会创建一个新的shell。另一方面，PPID每次都会一样只要你在同一个shell中运行。

分析 monitor.py 发现如下方法调用：

pid=os.fork()

分析 execute\_with\_log.py 发现如下方法调用：

proc = subprocess.Popen(" ".join(command_lines), shell=True, bufsize=1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

经过思考，利用 subprocess.Popen( ) 方法启动子进程并获取子进程返回值，子进程利用 \\$\\$ 变量获取PID并作为返回值传递给父进程，父进程可以启动监控程序记录子进程的运行数据。

参考：

[https://www.jb51.net/article/62370.htm][https_www.jb51.net_article_62370.htm]

[https://www.cnblogs.com/ratels/p/11039813.html][https_www.cnblogs.com_ratels_p_11039813.html]

[https://www.cnblogs.com/ratels/p/11070615.html][https_www.cnblogs.com_ratels_p_11070615.html]

[https://www.cnblogs.com/zhoug2020/p/5079407.html][https_www.cnblogs.com_zhoug2020_p_5079407.html]

转载于:https://www.cnblogs.com/ratels/p/11070262.html

[https_www.jb51.net_article_62370.htm]: https://www.jb51.net/article/62370.htm
[https_www.cnblogs.com_ratels_p_11039813.html]: https://www.cnblogs.com/ratels/p/11039813.html
[https_www.cnblogs.com_ratels_p_11070615.html]: https://www.cnblogs.com/ratels/p/11070615.html
[https_www.cnblogs.com_zhoug2020_p_5079407.html]: https://www.cnblogs.com/zhoug2020/p/5079407.html