深度学习图像配准 Image Registration: From SIFT to Deep Learning

刺骨的言语ヽ痛彻心扉 2022-11-17 10:20 269阅读 0赞

**Image Registration** is a fundamental step in **Computer Vision**. In this article, we present [**OpenCV**][OpenCV] **feature-based methods** before diving into **Deep Learning.**

--------------------

# What is Image Registration? #

Image registration isthe process of **transforming different images of one scene into the same coordinate system.** These images can be taken at different times (multi-temporal registration), by different sensors (multi-modal registration), and/or from different viewpoints. The spatial relationships between these images can be [**rigid**][rigid] (translations and rotations), [**affine**][affine] ([shears][] for example), [**homographies**][homographies], or complex **large deformations models**.

![3d0161528140f12dc4f9978b2a83834a.png][]

Image registration has a wide variety of applications: it is essential as soon as the task at hand requires comparing multiple images of the same scene. **It is very common in the field of medical imagery, as well as for satellite image analysis and optical flow.**

![032a1f8261d2ca1696e89a3983c6007d.png][]

In this article, we will focus on a few different ways to perform image registration between a reference image and a sensed image. We choose not to go into [iterative][] / [intensity][]\-based methods because they are less commonly used.

--------------------

# Traditional Feature-based Approaches #

Since the early 2000s, image registration has mostly used traditional **feature-based approaches**. These approaches are based on three steps:**Keypoint Detection and Feature Description, Feature Matching, and Image Warping.** In brief, we select points of interest in both images, associate each point of interest in the reference image to its equivalent in the sensed image and transform the sensed image so that both images are aligned.

![42b592d6eff1689432470c35a72e1fb2.png][]

Keypoint Detection and Feature Description

A **keypoint** is a point of interest. It defines what is important and distinctive in an image (corners, edges, etc). Each keypoint is represented by a **descriptor**: a feature vector containing the keypoints’ essential characteristics. A descriptor should be robust against image transformations (localization, scale, brightness, etc). Many algorithms perform **keypoint detection and feature description**:

*  [**SIFT**][SIFT] **(Scale-invariant feature transform)** is the original algorithm used for keypoint detection but it is not free for commercial use. The SIFT feature descriptor is invariant to uniform scaling, orientation, brightness changes, and partially invariant to affine distortion.
 *  [**SURF**][SURF] **(Speeded Up Robust Features)** is a detector and descriptor that is greatly inspired by SIFT. It presents the advantage of being several times faster. It is also patented.
 *  [**ORB**][ORB] **(Oriented FAST and Rotated BRIEF)** is a fast binary descriptor based on the combination of the [**FAST**][FAST] **(Features from Accelerated Segment Test)** keypoint detector and the [**BRIEF**][BRIEF] **(Binary robust independent elementary features)** descriptor. It is rotation invariant and robust to noise. It was developed in OpenCV Labs and it is an efficient and free alternative to SIFT.
 *  [**AKAZE**][AKAZE] **(Accelerated-KAZE)** is a sped-up version of [**KAZE**][KAZE]. It presents a fast multiscale feature detection and description approach for non-linear [scale spaces][]. It is both scale and rotation invariant. It is also free!

These algorithms are all available and easily usable in OpenCV. In the example below, we used the OpenCV implementation of AKAZE. The code remains roughly the same for the other algorithms: only the name of the algorithm needs to be modified.

import numpy as np
    import cv2 as cv
    
    img = cv.imread('image.jpg')
    gray= cv.cvtColor(img, cv.COLOR_BGR2GRAY)
    
    akaze = cv.AKAZE_create()
    kp, descriptor = akaze.detectAndCompute(gray, None)
    
    img=cv.drawKeypoints(gray, kp, img)
    cv.imwrite('keypoints.jpg', img)

![426cfa259eb158b7dac74b6a90493ddd.png][]

For more details on feature detection and description, you can check out this [OpenCV tutorial][] .

## Feature Matching ##

Once keypoints are identified in both images that form a couple, we need to associate, or “match”, keypoints from both images that correspond in reality to the same point. One possible method is *BFMatcher.knnMatch().* This matcher measures the distance between each pair of keypoint descriptors and returns for each keypoint its *k* best matches with the minimal distance.

We then apply a ratio filter to only keep the correct matches. In fact, to achieve a reliable matching, matched keypoints should be significantly closer than the nearest incorrect match.

import numpy as np
    import cv2 as cv
    import matplotlib.pyplot as plt
    
    img1 = cv.imread('image1.jpg', cv.IMREAD_GRAYSCALE)  # referenceImage
    img2 = cv.imread('image2.jpg', cv.IMREAD_GRAYSCALE)  # sensedImage
    
    # Initiate AKAZE detector
    akaze = cv.AKAZE_create()
    # Find the keypoints and descriptors with SIFT
    kp1, des1 = akaze.detectAndCompute(img1, None)
    kp2, des2 = akaze.detectAndCompute(img2, None)
    
    # BFMatcher with default params
    bf = cv.BFMatcher()
    matches = bf.knnMatch(des1, des2, k=2)
    
    # Apply ratio test
    good_matches = []
    for m,n in matches:
        if m.distance < 0.75*n.distance:
            good_matches.append([m])
            
    # Draw matches
    img3 = cv.drawMatchesKnn(img1,kp1,img2,kp2,good_matches,None,flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)
    cv.imwrite('matches.jpg', img3)

![fec2bbf7fedf9997caefdda4a333bca2.png][]

Check out this [documentation][] for other feature matching methods implemented in OpenCV.

## Image Warping ##

After matching at least four pairs of keypoints, we can transform one image relatively to the other one. This is called [***image*** ***warping***][image_ _warping]. Any two images of the same planar surface in space are related by a [**homography**][homographies]. Homographies are geometric transformations that have 8 free parameters and are represented by a 3x3 matrix. They represent any distortion made to an image as a whole (as opposed to local deformations). Therefore, to obtain the transformed sensed image, we compute the **homography matrix** and apply it to the sensed image.

To ensure optimal warping, we use the [RANSAC][] algorithm to detect outliers and remove them before determining the final homography. It is directly built-in OpenCV’s [*findHomography*][findHomography] method. There exist alternatives to the RANSAC algorithm such as [LMEDS][]:Least-Median robust method.

# Select good matched keypoints
    ref_matched_kpts = np.float32([kp1[m[0].queryIdx].pt for m in good_matches]).reshape(-1,1,2)
    sensed_matched_kpts = np.float32([kp2[m[0].trainIdx].pt for m in good_matches]).reshape(-1,1,2)
    
    # Compute homography
    H, status = cv.findHomography(ref_matched_kpts, sensed_matched_kpts, cv.RANSAC,5.0)
    
    # Warp image
    warped_image = cv.warpPerspective(img1, H, (img1.shape[1]+img2.shape[1], img1.shape[0]))
                
    cv.imwrite('warped.jpg', warped_image)

![34b2ae8ff12988d7c0a14520e83bc64c.png][]

If you are interested in more details about these three steps, OpenCV has put together a [series of useful tutorials][].

--------------------

# Deep Learning Approaches #

Most research nowadays in image registration concerns the use of **deep learning**. In the past few years, deep learning has allowed for **state-of-the-art performance in** **Computer Vision tasks** such as image classification, object detection, and segmentation. There is no reason why this couldn’t be the case for Image Registration.

## Feature Extraction ##

The first way deep learning was used for image registration was for **feature extraction**. **Convolutional neural networks**’ successive layers manage to **capture increasingly complex image characteristics** and **learn task-specific features**. Since 2014, researchers have applied these networks to the feature extraction step rather than SIFT or similar algorithms.

*  In 2014, Dosovitskiy et al. proposed a [generic feature learning method][]to train a convolutional neural network using only unlabeled data. The **genericity of these features** enabled them to be **robust to transformations**. These features, or descriptors, outperformed SIFT descriptors for matching tasks.
 *  In 2018, Yang et al. developed a [non-rigid registration method][] based on the same idea. They used layers of a **pre-trained VGG network** to generate a feature descriptor that keeps both convolutional information and localization capabilities. These descriptors also seem to outperform SIFT-like detectors, particularly in cases where SIFT contains many outliers or cannot match a sufficient number of feature points.![f609e83f4d4c180ae83d1c0188fc5d41.png][]

The code for this last paper can be found [here][]. While we were able to test this registration method on our own images within 15 minutes, the algorithm is approximatively 70 times slower than the SIFT-like methods implemented earlier in this article.

## Homography Learning ##

Instead of limiting the use of deep learning to feature extraction, researchers tried to use a neural network to directly **learn the geometric transformation to align two images**.

**Supervised Learning**  
In 2016, DeTone et al. published [Deep Image Homography Estimation][] that describes **Regression HomographyNet**, a VGG style model that **learns the homography relating two images**. This algorithm presents the advantage of learning the homography and the CNN model parameters simultaneously in an **end-to-end fashion**: no need for the previous two-stage process!

![9d29e12c09d7f61ae4c33cdbf957e845.png][]

The network produces eight real-valued numbers as an output. It is trained in a **supervised fashion** thanks to a **Euclidean loss** between the output and the **ground-truth homography**.

![416214cc1334da57e921a08edce6782d.png][]

Like any supervised approach, this homography estimation method **requires labeled pairs of data**. While it is easy to obtain the ground truth homographies for artificial image pairs, it is much more **expensive to do so on real data**.

**Unsupervised Learning**  
With this in mind, Nguyen et al. presented an [unsupervised approach to deep image homography estimation][]. They kept the same CNN but had to use a **new loss function adapted to the unsupervised approach**: they chose the photometric loss that **does not require a ground-truth label**. Instead, it computes the **similarity** between the reference image and the sensed transformed image.

![eca706452547e90d601865966ef2625b.png][]

Their approach introduces two new network structures: a Tensor Direct Linear Transform and a Spatial Transformation Layer. We will not go into the details of these components here, we can simply consider that these are used to obtain a transformed sensed image using the homography parameter outputs of the CNN model, that we then use to compute the photometric loss.

![8de8f4db8c91d17886fa529b292668e6.png][]

The authors claim that this unsupervised method obtains comparable or better accuracy and robustness to illumination variation than traditional feature-based methods, with faster inference speed. In addition, it has **superior adaptability** and performance compared to the supervised method.

## Other Approaches ##

**Reinforcement Learning**  
Deep reinforcement learning is gaining traction as a registration method for medical applications. As opposed to a pre-defined optimization algorithm, in this approach, we use a **trained agent to perform the registration**.

![f8c6b6069e4a5f2dc3bb702fe387d4cb.png][]

*  In 2016, Liao et al. were the first to use reinforcement learning for image registration. [Their method][] is based on a **greedy supervised algorithm for end-to-end training**. Its goal is to align the images by finding the **best sequence of motion actions**. This approach outperformed several state-of-the-art methods but it was **only used for rigid transformations**.
 *  Reinforcement Learning has also been used for more complex transformations. In [Robust non-rigid registration through agent-based action learning][], Krebs et al. apply an **artificial agent to optimize the parameters of a deformation model**. This method was evaluated on inter-subject registration of prostate MRI images and showed promising results in 2-D and 3-D.

**Complex Transformations**  
A **significant proportion of current research** in image registration concerns the field of **medical imagery**. Often times, the transformation between two medical images **cannot simply be described by a homography** matrix because of the **local** **deformations** of the subject (due to breathing, anatomical changes, etc.).More complex transformations models are necessary, such as diffeomorphisms that can be represented by displacement vector fields.

![be3861aed63076f3652d4de917014fef.png][]

Researchers have tried to use **neural networks to estimate these large deformation models** that have many parameters.

*  A first example is Krebs et al.’s Reinforcement Learning method mentioned just above.
 *  In 2017 De Vos et al. proposed the [DIRNet][]. It is a network that used a CNN to predict a grid of control points that are used to **generate the displacement vector field** to warp the sensed image according to the reference image.
 *  ![8baf780fff81f75cb4d0f558c9d2f3d7.png][]

*  [Quicksilver registration][] tackles a similar problem. Quicksilver uses a **deep encoder-decoder network** to **predict patch-wise deformations**directly on image appearance.

--------------------

We hope you enjoyed our article! Image registration is a vast field with numerous use cases. There is plenty of other fascinating research on this subject that we could not mention in this article, we tried to keep it to a few fundamental and accessible approaches. This [survey][] on deep learning in Medical Image Registration could be a good place to look for more information.

来源： [https://blog.sicara.com/image-registration-sift-deep-learning-3c794d794b7a][https_blog.sicara.com_image-registration-sift-deep-learning-3c794d794b7a]

转载于:https://www.cnblogs.com/jins-note/p/11203934.html

相关资源：[关于*图像**配**准*和*深度**学习*的一篇文章][Link 1]

[OpenCV]: https://opencv.org/
[rigid]: https://en.wikipedia.org/wiki/Rigid_transformation
[affine]: https://en.wikipedia.org/wiki/Affine_transformation
[shears]: https://en.wikipedia.org/wiki/Shear_mapping
[homographies]: https://docs.opencv.org/3.4.1/d9/dab/tutorial_homography.html
[3d0161528140f12dc4f9978b2a83834a.png]: /images/20221022/7607883540a34b268305dc5d1d84f672.png
[032a1f8261d2ca1696e89a3983c6007d.png]: /images/20221022/a753129fff7141d2b71c5fd29a486083.png
[iterative]: https://pdfs.semanticscholar.org/51fe/a461cf3724123c888cb9184474e176c12e61.pdf
[intensity]: http://elastix.isi.uu.nl/marius/downloads/2010_j_TMI.pdf
[42b592d6eff1689432470c35a72e1fb2.png]: /images/20221022/f560a47f06144683b15cc3bbc8e2efc8.png
[SIFT]: https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf
[SURF]: https://www.vision.ee.ethz.ch/~surf/eccv06.pdf
[ORB]: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.370.4395&rep=rep1&type=pdf
[FAST]: https://www.edwardrosten.com/work/rosten_2006_machine.pdf
[BRIEF]: https://www.cs.ubc.ca/~lowe/525/papers/calonder_eccv10.pdf
[AKAZE]: http://www.bmva.org/bmvc/2013/Papers/paper0013/paper0013.pdf
[KAZE]: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.304.4980&rep=rep1&type=pdf
[scale spaces]: https://en.wikipedia.org/wiki/Scale_space
[426cfa259eb158b7dac74b6a90493ddd.png]: /images/20221022/37eae1f4dcdc4f098ea9172fee45002b.png
[OpenCV tutorial]: https://docs.opencv.org/3.4/d7/d66/tutorial_feature_detection.html
[fec2bbf7fedf9997caefdda4a333bca2.png]: /images/20221022/faf553b4bcf74dfc9f768706662b3f1a.png
[documentation]: https://docs.opencv.org/trunk/dc/dc3/tutorial_py_matcher.html
[image_ _warping]: https://docs.opencv.org/3.0-beta/modules/cudawarping/doc/warping.html
[RANSAC]: https://en.wikipedia.org/wiki/Random_sample_consensus
[findHomography]: https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html?highlight=findhomography#findhomography
[LMEDS]: https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html?highlight=findhomography
[34b2ae8ff12988d7c0a14520e83bc64c.png]: /images/20221022/2e7bf9a6675142e6bb0aa3fa3ddb4076.png
[series of useful tutorials]: https://docs.opencv.org/3.1.0/db/d27/tutorial_py_table_of_contents_feature2d.html
[generic feature learning method]: https://arxiv.org/abs/1406.6909
[non-rigid registration method]: https://ieeexplore.ieee.org/document/8404075
[f609e83f4d4c180ae83d1c0188fc5d41.png]: /images/20221022/5fd8bb7923a14272a3d5eb23a0fe2b4e.png
[here]: https://github.com/yzhq97/cnn-registration
[Deep Image Homography Estimation]: https://arxiv.org/pdf/1606.03798.pdf
[9d29e12c09d7f61ae4c33cdbf957e845.png]: /images/20221022/ced521f8668a496da770bce00c5a6fae.png
[416214cc1334da57e921a08edce6782d.png]: /images/20221022/58eec4164d074c6b91f32bbaee1537ef.png
[unsupervised approach to deep image homography estimation]: https://arxiv.org/pdf/1709.03966.pdf
[eca706452547e90d601865966ef2625b.png]: /images/20221022/888babb190274ae2961eea659317bfde.png
[8de8f4db8c91d17886fa529b292668e6.png]: /images/20221022/09f7043bc31148aa8313fb602b2ec7f1.png
[f8c6b6069e4a5f2dc3bb702fe387d4cb.png]: /images/20221022/8eac2e7ef825407a83fdc2efe74c46e7.png
[Their method]: https://arxiv.org/pdf/1611.10336.pdf
[Robust non-rigid registration through agent-based action learning]: https://hal.inria.fr/hal-01569447/document
[be3861aed63076f3652d4de917014fef.png]: /images/20221022/a5981982e6e24b6e979ddf9dfdd927c6.png
[DIRNet]: https://arxiv.org/pdf/1704.06065.pdf
[8baf780fff81f75cb4d0f558c9d2f3d7.png]: /images/20221022/d1fde678d6144735a5e64d3864363f69.png
[Quicksilver registration]: https://www.researchgate.net/publication/315748621_Quicksilver_Fast_Predictive_Image_Registration_-_a_Deep_Learning_Approach
[survey]: https://arxiv.org/pdf/1903.02026.pdf
[https_blog.sicara.com_image-registration-sift-deep-learning-3c794d794b7a]: https://blog.sicara.com/image-registration-sift-deep-learning-3c794d794b7a
[Link 1]: https://download.csdn.net/download/qq_41934735/10557048?spm=1001.2101.3001.5697