[实现] 安装 TadasBaltrusaitis/OpenFace 并修改代码逻辑

蔚落 2022-02-26 13:35 333阅读 0赞

> 最近需要用 TadasBaltrusaitis/OpenFace 做一个 Demo，安装与修改代码的过程十分繁琐，所以以本文进行记录，给需要的人留个参考。  
> 链接：https://github.com/TadasBaltrusaitis/OpenFace

### 目录 ###

1.  **安装 OpenFace**
2.  **使用 OpenFace**
3.  **修改 OpenFace 代码逻辑**

### 1. **安装 OpenFace** ###

> Windows Installation：https://github.com/TadasBaltrusaitis/OpenFace/wiki/Windows-Installation  
> Ubuntu Installation：https://github.com/TadasBaltrusaitis/OpenFace/wiki/Unix-Installation  
> Model Download：https://github.com/TadasBaltrusaitis/OpenFace/wiki/Model-download

（建议先认真看完作者写的 Wiki，再进行安装，这样即使报错也可以心里有数）

OpenFace 在 Windows 下安装非常简单，只需要先将文件下载好（[链接][Link 1]），再执行脚本文件 download\_models.ps1，下载相应的模型文件即可。

如果是要在 Ubuntu 下安装 OpenFace，就得直接将 Github 上的项目文件下载下来，先执行脚本文件 download\_models.sh，下载相应的模型文件，再执行脚本文件 install.sh，进行项目的编译即可。这里需要提醒，由于 OpenFace 的依赖比较多，导致我在之前的环境下一直无法安装完成，直到重装了系统之后才安装完成，我个人感觉 OpenFacee 有点坑。

此外，我之前安装好后，运行时会报错 `Could not find CEN patch experts`，网上有博客指出去官网上下更多的的模型文件（[链接][Link 2]），但是我是通过重新执行脚本文件 install.sh 解决的。

**以下是网上所说的解决办法：**

下载完后放到对应的文件夹内，如果按照官网的路径保存 .dat 文件依然报错，其实是保存路径不对，应该：

*  把12个.mat文件存放到：/home/username/OpenFace/matlab\_version/models/cen下即可
 *  把4个.dat文件存放到：/home/username/OpenFace/build/lib/local/LandmarkDetector/model/patch\_experts即可

### 2. **使用 OpenFace** ###

在 Windows 下，OpenFace 可以使用 GUI 进行操作，分别是：OpenFaceDemo 和 OpenFaceOffline。其中，OpenFaceOffline 可供调整的选项更多，一般都使用该程序。

此外，OpenFace 在 Ubuntu 和 Windows 下都支持命令行操作，可执行文件分别是：FaceLandmarkImg, FaceLandmarkVid, FaceLandmarkVidMulti, FeatureExtraction了，关于命令行参数的详情可以查看：[Command line arguments][]。

### 3. **修改 OpenFace 代码逻辑** ###

在 Ubuntu 下修改 OpenFace 的代码逻辑后，需要在 build 文件夹下，执行`make`命令，重新进行项目的编译。

四个可执行文件的源码路径：

OpenFace-master\exe\FaceLandmarkImg\FaceLandmarkImg.cpp
    OpenFace-master\exe\FaceLandmarkVidFaceLandmarkVid.cpp
    OpenFace-master\exe\FaceLandmarkVidMulti\FaceLandmarkVidMulti.cpp
    OpenFace-master\exe\FeatureExtractionFeatureExtraction.cpp

以修改 FaceLandmarkVidMulti 的代码逻辑为例：

*  修改人脸数量限制设置

OpenFace-master\exe\FaceLandmarkVidMulti\FaceLandmarkVidMulti.cpp
    Line 129:	int num_faces_max = 4;//修改此处即可对人脸数量设置进行修改

*  修改检测置信度阈值设置

OpenFace-master\lib\local\Utilities\include\Visualizer.h
    Line 96:	double visualisation_boundary = 0.9;//当检测结果置信度低于阈值时，结果将不进行展示

*  限制程序展示 LandMark 结果

OpenFace-master\exe\FaceLandmarkVidMulti\FaceLandmarkVidMulti.cpp
    Line 357:	visualizer.SetObservationLandmarks(face_models[model].detected_landmarks, face_models[model].detection_certainty);//此语句用于展示 LandMark 结果
    Line 365:   open_face_rec.SetObservationLandmarks(face_models[model].detected_landmarks, face_models[model].GetShape(sequence_reader.fx, sequence_reader.fy, sequence_reader.cx, sequence_reader.cy),face_models[model].params_global, face_models[model].params_local, face_models[model].detection_certainty, face_models[model].detection_success);//此语句用于记录 LandMark 结果

*  修改 LandMark 可视化函数

OpenFace-master\lib\local\Utilities\src\Visualizer.cpp
    Line 170:    void Visualizer::SetObservationLandmarks(const cv::Mat_<float>& landmarks_2D, double confidence, const cv::Mat_<int>& visibilities) //修改该函数即可

如果想修改 AU 与 LandMark 的展示逻辑，作者给出了一些详细的信息：[Output Format][]，[Facial Action Coding System][]。

Code:

void Visualizer::SetObservationActionUnits_New(const std::vector<std::pair<std::string, double> >& au_intensities, const std::vector<std::pair<std::string, double> >& au_occurences, const cv::Mat_<float>& landmarks_2D, double confidence, const cv::Mat_<int>& visibilities)
    {
        
        if (au_intensities.size() > 0 || au_occurences.size() > 0)
        {
        
    
            std::set<std::string> au_names;
            std::map<std::string, bool> occurences_map;
            std::map<std::string, double> intensities_map;
    
            for (size_t idx = 0; idx < au_intensities.size(); idx++)
            {
        
                au_names.insert(au_intensities[idx].first);
                intensities_map[au_intensities[idx].first] = au_intensities[idx].second;
            }
    
            for (size_t idx = 0; idx < au_occurences.size(); idx++)
            {
        
                au_names.insert(au_occurences[idx].first);
                occurences_map[au_occurences[idx].first] = au_occurences[idx].second > 0;
            }
    
            const int AU_TRACKBAR_LENGTH = 400;
            const int AU_TRACKBAR_HEIGHT = 10;
    
            const int MARGIN_X = 185;
            const int MARGIN_Y = 10;
    
            const int nb_aus = (int) au_names.size();
    
            // Do not reinitialize
            if (action_units_image.empty())
            {
        
                action_units_image = cv::Mat(nb_aus * (AU_TRACKBAR_HEIGHT + 10) + MARGIN_Y * 2, AU_TRACKBAR_LENGTH + MARGIN_X, CV_8UC3, cv::Scalar(255, 255, 255));
            }
            else
            {
        
                action_units_image.setTo(255);
            }
    
            std::map<std::string, std::pair<bool, double>> aus;
    
            // first, prepare a mapping "AU name" -> { present, intensity }
            for (auto au_name : au_names)
            {
        
                // Insert the intensity and AU presense (as these do not always overlap check if they exist first)
                bool occurence = false;
                if (occurences_map.find(au_name) != occurences_map.end())
                {
        
                    occurence = occurences_map[au_name] != 0;
                }
                else
                {
        
                    // If we do not have an occurence label, trust the intensity one
                    occurence = intensities_map[au_name] > 1;
                }
                double intensity = 0.0;
                if (intensities_map.find(au_name) != intensities_map.end())
                {
        
                    intensity = intensities_map[au_name];
                }
                else
                {
        
                    // If we do not have an intensity label, trust the occurence one
                    intensity = occurences_map[au_name] == 0 ? 0 : 5;
                }
    
                aus[au_name] = std::make_pair(occurence, intensity);
            }
    	// then, build the graph
    	unsigned int idx = 0;
    	for (auto& au : aus)
    	{
        
    		std::string name = au.first;
    		bool present = au.second.first;
    		double intensity = au.second.second;
    
    		int offset = MARGIN_Y + idx * (AU_TRACKBAR_HEIGHT + 10);
    		std::ostringstream au_i;
    		au_i << std::setprecision(2) << std::setw(4) << std::fixed << intensity;
    		cv::putText(action_units_image, name, cv::Point(10, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.5, CV_RGB(present ? 0 : 200, 0, 0), 1, cv::LINE_AA);
    		cv::putText(action_units_image, AUS_DESCRIPTION.at(name), cv::Point(55, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.3, CV_RGB(0, 0, 0), 1, cv::LINE_AA);
    
    		if (present)
    		{
        
    			cv::putText(action_units_image, au_i.str(), cv::Point(160, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.3, CV_RGB(0, 100, 0), 1, cv::LINE_AA);
    			cv::rectangle(action_units_image, cv::Point(MARGIN_X, offset),
    				cv::Point((int)(MARGIN_X + AU_TRACKBAR_LENGTH * intensity / 5.0), offset + AU_TRACKBAR_HEIGHT),
    				cv::Scalar(128, 128, 128),
    				cv::FILLED);
    		}
    		else
    		{
        
    			cv::putText(action_units_image, "0.00", cv::Point(160, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.3, CV_RGB(0, 0, 0), 1, cv::LINE_AA);
    		}
    		idx++;
    	}
            // Draw the AU Land
            int n = landmarks_2D.rows / 2;
    	int thickness = (int)std::ceil(3.0* ((double)captured_image.cols) / 640.0);
            int thickness_2 = (int)std::ceil(1.0* ((double)captured_image.cols) / 640.0);
    
            std::vector<cv::Point> Points;
    
    	for (auto& au : aus)
            {
        
                std::string name = au.first;
                bool present = au.second.first;
                double intensity = au.second.second;
    
                if (present && intensity>=1.5)
                {
        
                    if (name=="AU01")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(21) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(21 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(22) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(22 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU02")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(18) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(18 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(25) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(25 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU04")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(19) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(19 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(21) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(21 + n) * (float)draw_multiplier));
                        cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(22) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(22 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(24) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(24 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU05")
                    {
        
    		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(37) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(37 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(38) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(38 + n) * (float)draw_multiplier));
                        cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(43) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(43 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(44) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(44 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU06")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(41) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(41 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(46) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(46 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU07")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(38) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(38 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(43) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(43 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU09")
                    {
        
    		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(27) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(27 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(28) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(28 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU10")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(31) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(31 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(35) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(35 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU12")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU14")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU15")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
                        Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
                    else if (name=="AU17")
                    {
        
    		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(8) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(8 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(57) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(57 + n) * (float)draw_multiplier));
    		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU20")
                    {
        
    		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(8) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(8 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(57) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(57 + n) * (float)draw_multiplier));
    		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU23")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(66) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(66 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(62) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(62 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                        
                    }
                    else if (name=="AU25")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(61) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(61 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(63) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(63 + n) * (float)draw_multiplier));
    		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(65) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(65 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(67) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(67 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU26")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(61) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(61 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(63) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(63 + n) * (float)draw_multiplier));
    		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(65) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(65 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(67) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(67 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU28")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(61) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(61 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(63) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(63 + n) * (float)draw_multiplier));
    		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(65) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(65 + n) * (float)draw_multiplier));
                        cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(67) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(67 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
    		    Points.push_back(featurePoint3);
    		    Points.push_back(featurePoint4);
                    }
                    else if (name=="AU45")
                    {
        
                        cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(38) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(38 + n) * (float)draw_multiplier));
                        cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(43) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(43 + n) * (float)draw_multiplier));
    		    Points.push_back(featurePoint1);
    		    Points.push_back(featurePoint2);
                    }
    
                }
            }
    	    int count = Points.size();
        	for (int i = 0; i < count;i++)
        	{
        
            	cv::circle(captured_image, Points[i], 1 * draw_multiplier, cv::Scalar(0, 0, 255), thickness, cv::LINE_AA, draw_shiftbits);
            	cv::circle(captured_image, Points[i], 1 * draw_multiplier, cv::Scalar(255, 0, 0), thickness_2, cv::LINE_AA, draw_shiftbits);
    		
        	}
    	
    
    
        }
    }

**参考资料：**

*  [OpenFace安装问题—Could not find CEN patch experts][OpenFace_Could not find CEN patch experts]
 *  [OpenFace][]

--------------------

**如果你看到了这篇文章的最后，并且觉得有帮助的话，麻烦你花几秒钟时间点个赞，或者受累在评论中指出我的错误。谢谢！**

**作者信息：**  
**知乎：[没头脑][Link 3]**  
**CSDN：[Code\_Mart][Code_Mart]**  
**Github：[Tao Pu][]**

[Link 1]: https://github.com/TadasBaltrusaitis/OpenFace/wiki/Windows-Installation
[Link 2]: https://github.com/TadasBaltrusaitis/OpenFace/wiki/Model-download
[Command line arguments]: https://github.com/TadasBaltrusaitis/OpenFace/wiki/Command-line-arguments
[Output Format]: https://github.com/TadasBaltrusaitis/OpenFace/wiki/Output-Format
[Facial Action Coding System]: https://github.com/TadasBaltrusaitis/OpenFace/wiki/Action-Units
[OpenFace_Could not find CEN patch experts]: https://blog.csdn.net/Will_Ye/article/details/82933621
[OpenFace]: https://blog.csdn.net/qq_14845119/article/details/53994607
[Link 3]: https://www.zhihu.com/people/putao537
[Code_Mart]: https://me.csdn.net/Code_Mart
[Tao Pu]: https://github.com/putao537