Deep Learning Enables You to Hide Screen when Your Boss is Approaching

一时失言乱红尘 2022-07-13 11:29 41阅读 0赞

## Introduction ##

When you are working, you have browsed information that is not relevant to your work, haven’t you?

I feel awkward when my boss is creeping behind. Of course, I can switch the screen in a hurry, but such behavior is suspicious, and sometimes I don’t notice him. So, in order to switch the screen without being suspected, I create a system that automatically recognizes that he is approaching to me and hides the screen.

Specifically, Keras is used to implement neural network for learning his face, a web camera is used to recognize that he is approaching, and switching the screen.

## Mission ##

The mission is to switch the screen automatically when my boss is approaching to me.  
The situation is as follows:  
![situation-1][]

It is about 6 or 7 meters from his seat to my seat. He reaches my seat in 4 or 5 seconds after he leaves his seat. Therefore, it is necessary to hide the screen during this time. There’s not much time!

## Strategy ##

Maybe you have various strategies, but my strategy is following.  
First, let the computer learn the face of the boss with deep learning. Then, set up a web camera at my desk and switch the screen when the web camera captures his face. It’s a perfect strategy. Let’s call this wonderful system Boss Sensor.

![strategy][]

### System Architecture ###

The simple system architecture of the Boss Sensor is as follows.

![process][]

*  Web camera take an image in real time.
 *  Learned model detect and recognize face for the taken image.
 *  If the recognition result is my boss, switch screen.

The following techniques are required to do above:

*  Taking face image
 *  Recognizing face image
 *  Switching screen

Let’s verify one by one, then integrate at the end.

## Taking Face Image ##

First of all, taking face image with webcam.  
This time, I used [BUFFALO BSW20KM11BK][] as webcam.  
![s\_DSC\_0081.JPG][s_DSC_0081.JPG]

You can also take image from the camera with the included software, but it is better to be able to take from the program because of considering the processing afterwards. Also, since face recognition is done in the subsequent processing, it is necessary to cut out only the face image. So, I use Python and OpenCV to take face image. Here’s the code for that:

*  [BossSensor/camera\_reader.py][BossSensor_camera_reader.py]

I was able to acquire a more clearly face image than I expected.  
![スクリーンショット 2016-09-15 9.49.32.png][2016-09-15 9.49.32.png]

## Recognizing Boss Face ##

Next, we use machine learning so that the computer can recognize the face of the boss.  
We need the following three steps:

1.  Collecting images
2.  Preprocessing images
3.  Building Machine Learning Model

Let’s take a look at these one by one.

### Collecting Images ###

First of all, I need to collect a large number of images for learning. As a collection method, I used the following:

*  Google image search
 *  Image collection on Facebook
 *  Taking video

Initially, I collected images from Web search and Facebook, but enough images did not gather. So, I took video using a video camera and decomposed video into a large number of images.

### Preprocessing Images ###

Well, I got a lot of images with faces, but the learning model can not be learned as it is. This is because the part not related to the face occupies a considerable part of the image. So we cut out only the face image.

I mainly used ImageMagick for extraction. You can get only face images by cutting out with ImageMagick.

*  [ImageMagick][]

A large number of face images gathered as follows:  
![s\_many\_bosses.jpg][s_many_bosses.jpg]

Perhaps I am the one who possesses the face image of the most boss in the world. I must have it more than his parents.

Now I’m ready for learning.

### Building Machine Learning Model ###

Keras is used to build convolutional neural network(CNN) and CNN is trained. TensorFlow is used for Keras’s back end. If you only recognize the face, you can call the Web API for image recognition like [Computer Vision API][] in Cognitive Services, but this time I decided to make it by myself considering real time nature.

The network has the following architecture. Keras is convenient because it can output the architecture easily.

____________________________________________________________________________________________________
    Layer (type)                     Output Shape          Param #     Connected to                     
    ====================================================================================================
    convolution2d_1 (Convolution2D)  (None, 32, 64, 64)    896         convolution2d_input_1[0][0]      
    ____________________________________________________________________________________________________
    activation_1 (Activation)        (None, 32, 64, 64)    0           convolution2d_1[0][0]            
    ____________________________________________________________________________________________________
    convolution2d_2 (Convolution2D)  (None, 32, 62, 62)    9248        activation_1[0][0]               
    ____________________________________________________________________________________________________
    activation_2 (Activation)        (None, 32, 62, 62)    0           convolution2d_2[0][0]            
    ____________________________________________________________________________________________________
    maxpooling2d_1 (MaxPooling2D)    (None, 32, 31, 31)    0           activation_2[0][0]               
    ____________________________________________________________________________________________________
    dropout_1 (Dropout)              (None, 32, 31, 31)    0           maxpooling2d_1[0][0]             
    ____________________________________________________________________________________________________
    convolution2d_3 (Convolution2D)  (None, 64, 31, 31)    18496       dropout_1[0][0]                  
    ____________________________________________________________________________________________________
    activation_3 (Activation)        (None, 64, 31, 31)    0           convolution2d_3[0][0]            
    ____________________________________________________________________________________________________
    convolution2d_4 (Convolution2D)  (None, 64, 29, 29)    36928       activation_3[0][0]               
    ____________________________________________________________________________________________________
    activation_4 (Activation)        (None, 64, 29, 29)    0           convolution2d_4[0][0]            
    ____________________________________________________________________________________________________
    maxpooling2d_2 (MaxPooling2D)    (None, 64, 14, 14)    0           activation_4[0][0]               
    ____________________________________________________________________________________________________
    dropout_2 (Dropout)              (None, 64, 14, 14)    0           maxpooling2d_2[0][0]             
    ____________________________________________________________________________________________________
    flatten_1 (Flatten)              (None, 12544)         0           dropout_2[0][0]                  
    ____________________________________________________________________________________________________
    dense_1 (Dense)                  (None, 512)           6423040     flatten_1[0][0]                  
    ____________________________________________________________________________________________________
    activation_5 (Activation)        (None, 512)           0           dense_1[0][0]                    
    ____________________________________________________________________________________________________
    dropout_3 (Dropout)              (None, 512)           0           activation_5[0][0]               
    ____________________________________________________________________________________________________
    dense_2 (Dense)                  (None, 2)             1026        dropout_3[0][0]                  
    ____________________________________________________________________________________________________
    activation_6 (Activation)        (None, 2)             0           dense_2[0][0]                    
    ====================================================================================================
    Total params: 6489634

The code is here:

*  [BossSensor/boss\_train.py][BossSensor_boss_train.py]

So far, I can recognize the boss when he appears on the camera.

## Switching Screen ##

Now, when learned model recognize the face of the boss, I need to change the screen. In this time, let’s display the image to pretend to work.  
I am a programmer so I prepared the following image.

![s\_pycharm.png][s_pycharm.png]

I only display this image.

Since I want to display the image in full screen, I use PyQt. Here’s the code for that:

*  [BossSensor/image\_show.py][BossSensor_image_show.py]

Now, everything is ready.

## Finished Product ##

Once we integrate the technologies we have verified, we are done. I actually tried it.

“My boss left his seat and he was approaching to my seat.”  
![standup.jpg][]

“OpenCV has detected the face and input the image into the learned model.”  
![approach.jpg][]

“The screen has switched by recognizing him! ヽ(‘ ∇‘ )ノ ﾜｰｲ”  
![s\_editor.jpg][s_editor.jpg]

## Source Code ##

You can download Boss Sensor from following link:

*  [BossSensor][]

Your star encourage me ｍ(\_ \_)ｍ

## Conclusion ##

In this time, I combined the real-time image acquisition from Web camera with face recognition using Keras to recognize my boss and hide the screen.

Currently, I detect the face with OpenCV, but since the accuracy of face detection in OpenCV seems not good, I’d like to try using Dlib to improve the accuracy. Also I would like to use my own trained face detection model.

Since the recognition accuracy of the image acquired from the Web camera is not good, I would like to improve it.

**Referenced from: [http://ahogrammer.com/2016/11/15/deep-learning-enables-you-to-hide-screen-when-your-boss-is-approaching/][http_ahogrammer.com_2016_11_15_deep-learning-enables-you-to-hide-screen-when-your-boss-is-approaching]**

**中文版：[http://mp.weixin.qq.com/s/K7aTjZUz0YHIeb5rWjnhfA][http_mp.weixin.qq.com_s_K7aTjZUz0YHIeb5rWjnhfA]**

[situation-1]: http://ahogrammer.com/wp-content/uploads/2016/11/situation-1.png
[strategy]: http://ahogrammer.com/wp-content/uploads/2016/11/strategy.png
[process]: http://ahogrammer.com/wp-content/uploads/2016/11/process-1024x449.png
[BUFFALO BSW20KM11BK]: https://www.amazon.co.jp/dp/B008AO4KXQ
[s_DSC_0081.JPG]: /images/20220713/93a82fb1be254a34af78aeeee39f0aae.png
[BossSensor_camera_reader.py]: https://github.com/Hironsan/BossSensor/blob/master/camera_reader.py
[2016-09-15 9.49.32.png]: /images/20220713/dca1212d698c4de79bc56fd5cd26b81d.png
[ImageMagick]: http://www.imagemagick.org/script/index.php
[s_many_bosses.jpg]: /images/20220713/6f0cfde4f5c54ababc4b9e9b6a67a337.png
[Computer Vision API]: https://www.microsoft.com/cognitive-services/en-us/computer-vision-api
[BossSensor_boss_train.py]: https://github.com/Hironsan/BossSensor/blob/master/boss_train.py
[s_pycharm.png]: /images/20220713/81b74b33294747378fa1173116b7dc72.png
[BossSensor_image_show.py]: https://github.com/Hironsan/BossSensor/blob/master/image_show.py
[standup.jpg]: /images/20220713/da3e33b8ff1a4825ad7cb225455876a5.png
[approach.jpg]: /images/20220713/4169c4085f4c40639f4ed8d7cbcd266a.png
[s_editor.jpg]: /images/20220713/6f4ebf21900c4c80b310b5d412ab4a4e.png
[BossSensor]: https://github.com/Hironsan/BossSensor
[http_ahogrammer.com_2016_11_15_deep-learning-enables-you-to-hide-screen-when-your-boss-is-approaching]: http://ahogrammer.com/2016/11/15/deep-learning-enables-you-to-hide-screen-when-your-boss-is-approaching/
[http_mp.weixin.qq.com_s_K7aTjZUz0YHIeb5rWjnhfA]: http://mp.weixin.qq.com/s/K7aTjZUz0YHIeb5rWjnhfA