【视频分类】training_extensions/action_recognition复现

旧城等待， 2023-01-02 14:26 193阅读 0赞

参考：

[https://github.com/openvinotoolkit/training\_extensions/tree/develop/pytorch\_toolkit/action\_recognition][https_github.com_openvinotoolkit_training_extensions_tree_develop_pytorch_toolkit_action_recognition]

# 0.环境 #

ubuntu16.04
    python3.6
    
    # pip安装
    torch==1.1.0
    torchvision==0.3.0
    numpy>=1.15.2
    onnx>=1.3.0
    opencv-python>=3.4.3.18
    pandas>=0.23.4
    tensorboardX>=1.4
    tqdm>=4.26.0
    pretrainedmodels>=0.7.4
    networkx==2.3
    
    
    # apt安装
    sudo apt-get install ffmpeg

# 1.数据准备 #

## （1）下载数据 ##

官网：[https://www.crcv.ucf.edu/data/UCF101.php][https_www.crcv.ucf.edu_data_UCF101.php],下面有两处是我们需要的。第一处是avi数据，第二处是分训练集与测试集的。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70][]

UCF101视频分类数据集：http://www.crcv.ucf.edu/datasets/human-actions/ucf101/UCF101.rar

新建data目录，解压UCF101.rar文件到其中：

apt-get install unrar
    unrar x UCF101.rar

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 1][]

下载第二处数据[https://www.crcv.ucf.edu/data/UCF101/UCF101TrainTestSplits-DetectionTask.zip][https_www.crcv.ucf.edu_data_UCF101_UCF101TrainTestSplits-DetectionTask.zip]。放到自己想放的地方，我是放到data目录下。

参考：[https://blog.csdn.net/qq\_41185868/article/details/108474259][https_blog.csdn.net_qq_41185868_article_details_108474259]

将utils/ucf101\_json.py文件中所有的：

data.ix
    替换为
    data.iloc

## （2）生成json文件 ##

运行生成json文件：

python utils/ucf101_json.py ./data/ucf-101/ucfTrainTestlist

![20210106145148843.png][]

生成的数据在./data/ucf-101/ucfTrainTestlist目录下。

## （3）视频转jpg ##

CUDA_VISIBLE_DEVICES="1" python utils/preprocess_videos.py --annotation_file ./data/ucf-101/ucfTrainTestlist/ucf101_01.json \
        --raw_dir ./data/ucf-101/UCF-101/ \
        --destination_dir ./data/data/UCF101_jpg \
        --video-size 480 \
        --video-format frames \
        --video-quality 1 \
        --threads 6
    
    CUDA_VISIBLE_DEVICES="1" python utils/preprocess_videos.py --annotation_file ./data/ucf-101/ucfTrainTestlist/ucf101_02.json \
        --raw_dir ./data/ucf-101/UCF-101/ \
        --destination_dir ./data/data/UCF101_jpg \
        --video-size 480 \
        --video-format frames \
        --video-quality 1 \
        --threads 6
    
    CUDA_VISIBLE_DEVICES="1" python utils/preprocess_videos.py --annotation_file ./data/ucf-101/ucfTrainTestlist/ucf101_03.json \
        --raw_dir ./data/ucf-101/UCF-101/ \
        --destination_dir ./data/data/UCF101_jpg \
        --video-size 480 \
        --video-format frames \
        --video-quality 1 \
        --threads 6

![20210106164808716.png][]

## （4）生成n\_frames ##

参考：[https://github.com/kenshohara/3D-ResNets-PyTorch/tree/CVPR2018][https_github.com_kenshohara_3D-ResNets-PyTorch_tree_CVPR2018]

创建生成n\_frames脚本文件，放到utils目录下：

# -*- coding: UTF-8 -*-
    '''
    @author: mengting gu
    @contact: 1065504814@qq.com
    @time: 2021/1/7 15:47
    @file: n_frames_ucf101_hmdb51.py
    @desc: 
    '''
    
    from __future__ import print_function, division
    import os
    import sys
    import subprocess
    
    def class_process(dir_path, class_name):
        class_path = os.path.join(dir_path, class_name)
        if not os.path.isdir(class_path):
            return
    
        for file_name in os.listdir(class_path):
            video_dir_path = os.path.join(class_path, file_name)
            image_indices = []
            for image_file_name in os.listdir(video_dir_path):
                if 'image' not in image_file_name:
                    continue
                image_indices.append(int(image_file_name[6:11]))
    
            if len(image_indices) == 0:
                print('no image files', video_dir_path)
                n_frames = 0
            else:
                image_indices.sort(reverse=True)
                n_frames = image_indices[0]
                print(video_dir_path, n_frames)
            with open(os.path.join(video_dir_path, 'n_frames'), 'w') as dst_file:
                dst_file.write(str(n_frames))
    
    
    if __name__=="__main__":
        dir_path = sys.argv[1]
        for class_name in os.listdir(dir_path):
            class_process(dir_path, class_name)

运行命令：

python utils/n_frames_ucf101_hmdb51.py ./data/data/UCF101_jpg

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 2][]

![20210107155502857.png][]

## （5）文件结构 ##

准备好数据后，文件结构：

.../
        data/ (root dir)
            data
            	UCF101_jpg/  (jpg files)
            ucf-101/
                UCF-101/ (video files)
                ucfTrainTestlist/ (annotation path)
                	classInd.txt
                	ucf101_01.json

# 2.训练与评估 #

如分类任务以imagenet的预训练模型一样，视频分类主要以kinetics训练的模型作为预训练模型。

## （1）准备预训练模型 ##

准备预训练模型：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 3][]

这里以ResNet34-VTN为例，下载对应的预训练模型，放到models下。

![2021010615395772.png][]

![20210106154024342.png][]

![20210106154118925.png][]

## （2）文件结构 ##

.../
        data/ (root dir)
            ...
    	    models/
    	    	resnet_34_vtn_rgbd_kinetics.pth  
    	    	se_resnext_101_32x4d_vtn_rgbd_ucf101_s1.pth

## （3）训练 ##

预训练模型：resnet\_34\_vtn\_rgbd\_kinetics.pth

CUDA_VISIBLE_DEVICES="0" python main.py --root-path ./data --result-path ./logs/ --dataset ucf101_1 --model resnet34_vtn_rgbdiff -b16 --lr 1e-5 --seq 16 --pretrain-path ./models/resnet_34_vtn_rgbd_kinetics.pth --video-path ./UCF101_jpg

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 4][]

## （4）评估 ##

评估使用的预训练模型：se\_resnext\_101\_32x4d\_vtn\_rgbd\_ucf101\_s1.pth

CUDA_VISIBLE_DEVICES="2" python main.py --root-path ./data --result-path ./logs/ --dataset ucf101_1 --model se-resnext101-32x4d_vtn_rgbdiff -b128 --lr 1e-5 --seq 16 --st 2 --no-mean-norm --no-std-norm --no-train --no-val --test --pretrain-path ./models/se_resnext_101_32x4d_vtn_rgbd_ucf101_s1.pth --video-path ./UCF101_jpg

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 5][]

![20210114171711853.png][]

## （5）参数 ##

使用什么模型与参数都是根据下图来的：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 3][]

\--dataset：ucf101或者kinetics；

\--model：与Model与Input有关，以性能最好的93.44%模型为例，se-resnext101-32x4d\_vtn\_rgbdiff；

\-b：batch\_size；

\--lr：学习率；

\--pretrain-path：就是Checkpoint中下载的模型；

\--video-path：如果不以默认的"./data/data/utf-101/jpeg2"，可以自己设置。

其中--model与--pretrain-path不对应是会出问题的。

训练se\_resnext\_101\_32x4d\_vtn\_rgbd，b=2大概需要7-8GB显存。

# 3.单帧图像推理 #

import sys
    import time
    from argparse import ArgumentParser
    from collections import deque
    from copy import deepcopy
    
    import cv2
    import numpy as np
    import torch
    import torch.nn.functional as F
    
    from action_recognition.model import create_model
    from action_recognition.options import add_input_args
    from action_recognition.spatial_transforms import (CenterCrop, Compose,
                                                       Normalize, Scale, ToTensor, MEAN_STATISTICS, STD_STATISTICS)
    from action_recognition.utils import load_state, generate_args
    import os
    
    TEXT_COLOR = (255, 255, 255)
    TEXT_FONT_FACE = cv2.FONT_HERSHEY_DUPLEX
    TEXT_FONT_SIZE = 1
    TEXT_VERTICAL_INTERVAL = 45
    NUM_LABELS_TO_DISPLAY = 2
    
    
    class TorchActionRecognition:
        def __init__(self, encoder, checkpoint_path, num_classes=400, **kwargs):
            # model_type = "{}_vtn".format(encoder)
            model_type = "{}".format(encoder)
            args, _ = generate_args(model=model_type, n_classes=num_classes, layer_norm=False, **kwargs)
            self.args = args
            self.model, _ = create_model(args, model_type)
    
            self.model = self.model.module
            self.model.eval()
            self.model.cuda()
    
            checkpoint = torch.load(str(checkpoint_path))
            load_state(self.model, checkpoint['state_dict'])
    
            self.preprocessing = make_preprocessing(args)
            self.embeds = deque(maxlen=(args.sample_duration * args.temporal_stride))
    
        def preprocess_frame(self, frame):
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            return self.preprocessing(frame)
    
        def infer_frame(self, frame):
            embedding = self._infer_embed(self.preprocess_frame(frame))
            self.embeds.append(embedding)
            sequence = self.get_seq()
            return self._infer_logits(sequence)
    
        def _infer_embed(self, frame):
            with torch.no_grad():
                frame_tensor = frame.unsqueeze(0).to('cuda')
                tensor = self.model.resnet(frame_tensor)
                tensor = self.model.reduce_conv(tensor)
                embed = F.avg_pool2d(tensor, 7)
            return embed.squeeze(-1).squeeze(-1)
    
        def _infer_logits(self, embeddings):
            with torch.no_grad():
                ys = self.model.self_attention_decoder(embeddings)
                ys = self.model.fc(ys)
                ys = ys.mean(1)
            return ys.cpu()
    
        def _infer_seq(self, frame):
            with torch.no_grad():
                result = self.model(frame.view(1, self.args.sample_duration, 3,
                                               self.args.sample_size, self.args.sample_size).to('cuda'))
            return result.cpu()
    
        def get_seq(self):
            sequence = torch.stack(tuple(self.embeds), 1)
            if self.args.temporal_stride > 1:
                sequence = sequence[:, ::self.args.temporal_stride, :]
    
            n = self.args.sample_duration
            if sequence.size(1) < n:
                num_repeats = (n - 1) // sequence.size(1) + 1
                sequence = sequence.repeat(1, num_repeats, 1)[:, :n, :]
    
            return sequence
    
    
    def make_preprocessing(args):
        return Compose([
            Scale(args.sample_size),
            CenterCrop(args.sample_size),
            ToTensor(args.norm_value),
            Normalize(MEAN_STATISTICS[args.mean_dataset], STD_STATISTICS[args.mean_dataset])
        ])
    
    
    def draw_rect(image, bottom_left, top_right, color=(0, 0, 0), alpha=1.):
        xmin, ymin = bottom_left
        xmax, ymax = top_right
    
        image[ymin:ymax, xmin:xmax, :] = image[ymin:ymax, xmin:xmax, :] * (1 - alpha) + np.asarray(color) * alpha
        return image
    
    
    def render_frame(frame, probs, labels):
        order = probs.argsort(descending=True)
    
        status_bar_coordinates = (
            (0, 0),  # top left
            (650, 25 + TEXT_VERTICAL_INTERVAL * NUM_LABELS_TO_DISPLAY)  # bottom right
        )
    
        draw_rect(frame, status_bar_coordinates[0], status_bar_coordinates[1], alpha=0.5)
    
        for i, imax in enumerate(order[:NUM_LABELS_TO_DISPLAY]):
            text = '{} - {:.1f}%'.format(labels[imax], probs[imax] * 100)
            text = text.upper().replace("_", " ")
            cv2.putText(frame, text, (15, TEXT_VERTICAL_INTERVAL * (i + 1)), TEXT_FONT_SIZE,
                        TEXT_FONT_FACE, TEXT_COLOR)
    
        return frame
    
    
    def run_demo(model, labels, input_path, save_path):
    
        fps = 30
        tick = time.time()
    
        for file in sorted(os.listdir(input_path)):
            frame = cv2.imread(os.path.join(input_path, file))
            if frame is None:
                print("图像为空:"+str(frame))
                break
            print("Now processing file : {}".format(file))
        # while video_cap.isOpened():
        #     ok, frame = video_cap.read()
        #
        #     if not ok:
        #         break
    
            logits = model.infer_frame(frame)
            probs = F.softmax(logits[0], dim=0)
            frame = render_frame(frame, probs, labels)
    
            tock = time.time()
            expected_time = tick + 1 / fps
            if tock < expected_time:
                delay = max(1, int((expected_time - tock) * 1000))
            tick = tock
    
            cv2.imwrite(os.path.join(save_path, file), frame)
            # cv2.imshow("demo", frame)
            # key = cv2.waitKey(delay)
            # if key == 27 or key == ord('q'):
            #     break
    
    
    def main():
        parser = ArgumentParser()
        parser.add_argument("--encoder", help="What encoder to use ", default='resnet34')
        parser.add_argument("--checkpoint", help="Path to pretrained model (.pth) file", required=True)
        parser.add_argument("--input-video", type=str, help="Path to input img or video", required=True)
        parser.add_argument("--save-path", type=str, help="Path to save img", required=True)
        parser.add_argument("--labels", help="Path to labels file (new-line separated file with label names)", type=str,
                            required=True)
        add_input_args(parser)
        args = parser.parse_args()
    
        with open(args.labels) as fd:
            labels = fd.read().strip().split('\n')
    
        extra_args = deepcopy(vars(args))
        input_data_params = set(x.dest for x in parser._action_groups[-1]._group_actions)
        for name in list(extra_args.keys()):
            if name not in input_data_params:
                del extra_args[name]
    
        input_path = args.input_video
        save_path = args.save_path
        try:
            model = TorchActionRecognition(args.encoder, args.checkpoint, num_classes=len(labels), **extra_args)
            # cap = cv2.VideoCapture(args.input_video)
            run_demo(model, labels, input_path, save_path)
        except Exception as error:
            print("an error occur : "+error)
    
    if __name__ == '__main__':
        sys.exit(main())

![202101142114089.png][]

准备一张图像与模型，运行以下命令：

CUDA_VISIBLE_DEVICES="0" python vtn_jpg_demo.py --encoder  resnet34_vtn --checkpoint ./data/models/resnet_34_vtn_rgb_ucf101_s1.pth --input-video ./data/ourdata/input --save-path ./data/ourdata/output --labels ./data/ucf-101/ucfTrainTestlist/classInd.txt

得到结果：

![20210114204657763.png][]

# 4.注意 #

1）出现各种各样关于模型的错误，很有可能就是调用的模型，与前面的模型名称不一致；

2）明明数据放在data对应目录下，调用模型错误，是因为代码内部设置了data/data+(video\_path)；

3）出现无n\_frames文件，请看我的1.4；

4）测试预训练模型，出现错误的可能：

数据准备时，即python utils/preprocess\_videos.py调用过程中默认使用的-q=4，图像质量最差的保存方式；

推理时，参数不对，多了或者少了，--no-mean-norm --no-std-norm。

注：这些都是花了很多时间的代价，还有一些记不起了。

[https_github.com_openvinotoolkit_training_extensions_tree_develop_pytorch_toolkit_action_recognition]: https://github.com/openvinotoolkit/training_extensions/tree/develop/pytorch_toolkit/action_recognition
[https_www.crcv.ucf.edu_data_UCF101.php]: https://www.crcv.ucf.edu/data/UCF101.php
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70]: /images/20221119/d470546f88104945a638fa880a0a8ccf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 1]: /images/20221119/5b280c24d7ff4a1e8ca09380075a93e9.png
[https_www.crcv.ucf.edu_data_UCF101_UCF101TrainTestSplits-DetectionTask.zip]: https://www.crcv.ucf.edu/data/UCF101/UCF101TrainTestSplits-DetectionTask.zip
[https_blog.csdn.net_qq_41185868_article_details_108474259]: https://blog.csdn.net/qq_41185868/article/details/108474259
[20210106145148843.png]: /images/20221119/a61e8b9005be4a00aecde46765c1ea36.png
[20210106164808716.png]: /images/20221119/45a7194c3a184efc8903713de7707b63.png
[https_github.com_kenshohara_3D-ResNets-PyTorch_tree_CVPR2018]: https://github.com/kenshohara/3D-ResNets-PyTorch/tree/CVPR2018
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 2]: /images/20221119/e993e2cff9314b55b754628a8be10a82.png
[20210107155502857.png]: /images/20221119/254e92b40a7b4cd9995b2b1ac7cf90b4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 3]: /images/20221119/e1ff856de6024155ba556012f993a84a.png
[2021010615395772.png]: /images/20221119/68a4fc7aaa494775b1d17e9f8b0a97eb.png
[20210106154024342.png]: /images/20221119/97897b6478a346baa651821ea4c6e400.png
[20210106154118925.png]: /images/20221119/febc00334e0a4c9599007e7be67ca798.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 4]: /images/20221119/dcc4256054f140b4927dc2794a62f8b6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70 5]: /images/20221119/a47937eb2d1944f5b07080ef10fbe17e.png
[20210114171711853.png]: /images/20221119/74b1f1c2aea9434db17c093b6445b131.png
[202101142114089.png]: /images/20221119/e672f046d0df4c2393eb3477ab11f24b.png
[20210114204657763.png]: /images/20221119/b5aa887b264440f18b16f188319d9ddf.png