OpenCV调用TensorFlow预训练模型

深藏阁楼爱情的钟 2022-05-23 07:13 389阅读 0赞

## OpenCV调用TensorFlow预训练模型 ##

【[尊重原创，转载请注明出处][Link 1]】[https://panjinquan.blog.csdn.net/article/details/80570120][https_panjinquan.blog.csdn.net_article_details_80570120]

强大OpenCV从自OpenCV 3.1版以来，dnn模块一直是opencv\_contrib库的一部分，在3.3版中，它被提到了主仓库中。新版OpenCV dnn模块目前支持Caffe、TensorFlow、Torch、PyTorch等深度学习框架。另外，新版本中使用预训练深度学习模型的API同时兼容C++和Python

OpenCV 3.3开始就提供了读取TensoFlow模型的接口了，不过现在能支持的模型并不多。目前，我测试成功只有两个，分别是object\_detection的“ssd\_mobilenet\_v1\_coco\_11\_06\_2017”和“ssd\_inception\_v2\_coco\_2017\_11\_17”预训练模型，其他模型报出各种各样的错误，反正我暂时还没有解决问题，各位神一样的网友若有新的进展，麻烦告知一声，哈哈~！

## 一、版本配置要求 ##

*  1. Windows 7 和 VS2015
 *  2. OpenCV3.3.1以上，本人使用的OpenCV 3.4.1，下载地址：*https://github.com/opencv/opencv/releases *
 *  3.至于TensoFlow，鄙人只是想测试已经训练好的模型，那就可以不安装TensoFlow了，偷懒一下~！

## 二、模型下载和配置 ##

1.   先用VS2015新建一个项目“opercv4tensorflow”，并在源文件目录下新建 一个“models”的文件夹，这个文件夹用于下载TensoFlow的模型。

2.   目前，只测试TensoFlow的object\_detection模块的“ssd\_mobilenet\_v1\_coco\_11\_06\_2017”和“ssd\_inception\_v2\_coco\_2017\_11\_17”两个预训练模型，其他的自己看着办吧：

*  [ssd\_mobilenet\_v1\_coco\_11\_06\_2017][ssd_mobilenet_v1_coco_11_06_2017]下载地址：

http://download.tensorflow.org/models/object\_detection/ssd\_mobilenet\_v1\_coco\_11\_06\_2017.tar.gz

*  [ssd\_inception\_v2\_coco\_2017\_11\_17][ssd_inception_v2_coco_2017_11_17]下载地址：

http://download.tensorflow.org/models/object\_detection/ssd\_inception\_v2\_coco\_2017\_11\_17.tar.gz

*  [COCO-trained models][]

当然了，你可以自己训练模型，也可以下载更多的预训练模型，这个可以到TensoFlow下载，下载地址是：  
    https://github.com/tensorflow/models/blob/master/research/object\_detection/g3doc/detection\_model\_zoo.md  
![70][]

Tensorflow模型的graph结构可以保存为.pb文件或者.pbtxt文件，或者.meta文件，其中只有.pbtxt文件是可读的。在OpenCV中，每个模型.pb文件，原则上应有一个对应的文本图形定义的.pbtxt文件，当然也可能没有，在opencv\_extra\\testdata\\dnn有些.pbtxt文件是可以对应找到，这个要看opencv会不会提供，当然，你厉害的话，可以自己按照网络定义结构写一份。

下表给出，目前我在object\_detection模块中测试成功.pb文件与.pbtxt文件的对应关系，更多的模型，还请网友下方留言，支援一下，我会及时更新博客内容：

<table> 
 <tbody> 
  <tr> 
   <td style="text-align:center;">序号</td> 
   <td>.pb文件：预训练模型</td> 
   <td>.pbtxt文件</td> 
   <td>备注</td> 
  </tr> 
  <tr> 
   <td style="text-align:center;">1</td> 
   <td><a href="http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz" rel="nofollow">ssd_mobilenet_v1_coco_11_06_2017</a></td> 
   <td>ssd_mobilenet_v1_coco.pbtxt</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="text-align:center;">2</td> 
   <td><a href="http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2017_11_17.tar.gz" rel="nofollow">ssd_inception_v2_coco_2017_11_17</a></td> 
   <td>ssd_inception_v2_coco_2017_11_17.pbtxt</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="text-align:center;">3</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="text-align:center;">4</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

3. 上面下载的TensoFlow模型解压后，里含有重要的二进制protobuf描述的.pb文件，我们还需要对应的protobuf格式文本图形定义的.pbtxt文件，这个就需要到opencv\_extra\\testdata\\dnn下载了

opencv\_extra下载地址：https://github.com/opencv/opencv\_extra/tree/master/testdata/dnn

把dnn文件夹中的“ssd\_inception\_v2\_coco\_2017\_11\_17.pbtxt”和“ssd\_mobilenet\_v1\_coco.pbtxt”也下载复制到项目的models中吧

![70 1][]

**三、使用opencv读取网络模型**

一切准备妥当了，下面就使用opencv C++（也可以是python的）实现读取TensoFlow训练好的网络模型。opencv dnn为我们提供TensoFlow的接口：readNetFromTensorflow，当然也有支持Caffe的接口：readNetFromCaffe。我们这里只考虑TensoFlow的模型，readNetFromTensorflow函数有多个重载函数：

/** @brief Reads a network model stored in <a href="https://www.tensorflow.org/">TensorFlow</a> framework's format.
          * @param model  path to the .pb file with binary protobuf description of the network architecture
          * @param config path to the .pbtxt file that contains text graph definition in protobuf format.
          *               Resulting Net object is built by text graph using weights from a binary one that
          *               let us make it more flexible.
          * @returns Net object.
          */
        CV_EXPORTS_W Net readNetFromTensorflow(const String &model, const String &config = String());

完整的测试代码：

#include<opencv2\opencv.hpp>
    #include<opencv2\dnn.hpp>
    #include <iostream>
    
    using namespace std;
    using namespace cv;
    
    const size_t inWidth = 300;
    const size_t inHeight = 300;
    const float WHRatio = inWidth / (float)inHeight;
    const char* classNames[] = { "background","face" };//这个需要根据训练的类别定义
    
    int main() {
    
    	Mat frame = cv::imread("2.jpg");
    	Size frame_size = frame.size();
    
    
    	String weights = "models\\ssd_inception_v2_coco_2017_11_17\\frozen_inference_graph.pb";
    	String prototxt = "models\\ssd_inception_v2_coco_2017_11_17.pbtxt";
    	dnn::Net net = cv::dnn::readNetFromTensorflow(weights, prototxt);
    
    	Size cropSize;
    	if (frame_size.width / (float)frame_size.height > WHRatio)
    	{
    		cropSize = Size(static_cast<int>(frame_size.height * WHRatio),
    			frame_size.height);
    	}
    	else
    	{
    		cropSize = Size(frame_size.width,
    			static_cast<int>(frame_size.width / WHRatio));
    	}
    
    	Rect crop(Point((frame_size.width - cropSize.width) / 2,
    		(frame_size.height - cropSize.height) / 2),
    		cropSize);
    
    
    	cv::Mat blob = cv::dnn::blobFromImage(frame, 1. / 255, Size(300, 300));
    	//cout << "blob size: " << blob.size << endl;
    
    	net.setInput(blob);
    	Mat output = net.forward();
    	//cout << "output size: " << output.size << endl;
    
    	Mat detectionMat(output.size[2], output.size[3], CV_32F, output.ptr<float>());
    
    	frame = frame(crop);
    	float confidenceThreshold = 0.20;
    	for (int i = 0; i < detectionMat.rows; i++)
    	{
    		float confidence = detectionMat.at<float>(i, 2);
    
    		if (confidence > confidenceThreshold)
    		{
    			size_t objectClass = (size_t)(detectionMat.at<float>(i, 1));
    
    			int xLeftBottom = static_cast<int>(detectionMat.at<float>(i, 3) * frame.cols);
    			int yLeftBottom = static_cast<int>(detectionMat.at<float>(i, 4) * frame.rows);
    			int xRightTop = static_cast<int>(detectionMat.at<float>(i, 5) * frame.cols);
    			int yRightTop = static_cast<int>(detectionMat.at<float>(i, 6) * frame.rows);
    
    			ostringstream ss;
    			ss << confidence;
    			String conf(ss.str());
    
    			Rect object((int)xLeftBottom, (int)yLeftBottom,
    				(int)(xRightTop - xLeftBottom),
    				(int)(yRightTop - yLeftBottom));
    
    			rectangle(frame, object, Scalar(0, 255, 0), 2);
    			String label = String(classNames[objectClass]) + ": " + conf;
    			int baseLine = 0;
    			Size labelSize = getTextSize(label, FONT_HERSHEY_SIMPLEX, 0.5, 1, &baseLine);
    			rectangle(frame, Rect(Point(xLeftBottom, yLeftBottom - labelSize.height),
    				Size(labelSize.width, labelSize.height + baseLine)),
    				Scalar(0, 255, 0), CV_FILLED);
    			putText(frame, label, Point(xLeftBottom, yLeftBottom),
    				FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 0, 0));
    		}
    	}
    	imshow("image", frame);
    	waitKey(0);
    	return 0;
    }

运行结果：

![70 2][]

## 四、常见的错误和坑 ##

# #

*  OpenCV(3.4.1) Error: Unspecified error (FAILED: fs.is\_open(). Can't open "models\\ssd\_inception\_v2\_coco\_2017\_1117\\frozen\_inference\_graph.pb") in cv::dnn::ReadProtoFromBinaryFile, file C:\\build\\master\_winpack-build-win64-vc14\\opencv\\modules\\dnn\\src\\caffe\\caffe\_io.cpp, line 1126

出现这个问题，很大的原因是你的模型路径，错鸟！！导致不能打开

*  OpenCV(3.4.1) Error: Unspecified error **(Const input blob for weights not found)** in cv::dnn::experimental\_dnn\_v4::\`anonymous-namespace'::TFImporter::getConstBlob, file C:\\build\\master\_winpack-build-win64-vc14\\opencv\\modules\\dnn\\src\\tensorflow\\tf\_importer.cpp, line 579

出现这个问题，可能是你的.pb模型文件与.pbtxt文件不对应

## 五、参考资料： ##

https://blog.csdn.net/xingchenbingbuyu/article/details/78416887

![70 3][]

[Link 1]: https://blog.csdn.net/guyuealian/article/details/80570120
[https_panjinquan.blog.csdn.net_article_details_80570120]: https://panjinquan.blog.csdn.net/article/details/80570120
[ssd_mobilenet_v1_coco_11_06_2017]: http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz
[ssd_inception_v2_coco_2017_11_17]: http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2017_11_17.tar.gz
[COCO-trained models]: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
[70]: /images/20220523/92eef4ab9b3e485da6d3fcdb0fbb8f54.png
[70 1]: /images/20220523/07352afb00e2424d8724851aa04dfcbf.png
[70 2]: /images/20220523/76288a102ad3458398ab56fd59c55ebf.png
[70 3]: /images/20220523/4acb921a67294ef2bdaf46810fbdf31b.png