14.8. Region-based CNNs (R-CNNs) — Dive into Deep Learning 1.0.3 documentation (2024)

14.8.1. R-CNNs¶

The R-CNN first extracts many (e.g., 2000) region proposals from theinput image (e.g., anchor boxes can also be considered as regionproposals), labeling their classes and bounding boxes (e.g., offsets).

(Girshick et al., 2014)

Then a CNN is used to perform forward propagation on each regionproposal to extract its features. Next, features of each region proposalare used for predicting the class and bounding box of this regionproposal.

Fig. 14.8.1 shows the R-CNN model. More concretely, the R-CNNconsists of the following four steps:

1. Perform selective search to extract multiple high-quality regionproposals on the input image(Uijlings et al., 2013). These proposedregions are usually selected at multiple scales with different shapesand sizes. Each region proposal will be labeled with a class and aground-truth bounding box.

2. Choose a pretrained CNN and truncate it before the output layer.Resize each region proposal to the input size required by thenetwork, and output the extracted features for the region proposalthrough forward propagation.

3. Take the extracted features and labeled class of each region proposalas an example. Train multiple support vector machines to classifyobjects, where each support vector machine individually determineswhether the example contains a specific class.

4. Take the extracted features and labeled bounding box of each regionproposal as an example. Train a linear regression model to predictthe ground-truth bounding box.

Although the R-CNN model uses pretrained CNNs to effectively extractimage features, it is slow. Imagine that we select thousands of regionproposals from a single input image: this requires thousands of CNNforward propagations to perform object detection. This massive computingload makes it infeasible to widely use R-CNNs in real-worldapplications.

14.8.2. Fast R-CNN¶

The main performance bottleneck of an R-CNN lies in the independent CNNforward propagation for each region proposal, without sharingcomputation. Since these regions usually have overlaps, independentfeature extractions lead to much repeated computation. One of the majorimprovements of the fast R-CNN from the R-CNN is that the CNN forwardpropagation is only performed on the entire image(Girshick, 2015).

Fig. 14.8.2 describes the fast R-CNN model. Its majorcomputations are as follows:

1. Compared with the R-CNN, in the fast R-CNN the input of the CNN forfeature extraction is the entire image, rather than individual regionproposals. Moreover, this CNN is trainable. Given an input image, letthe shape of the CNN output be\(1 \times c \times h_1 \times w_1\).

2. Suppose that selective search generates \(n\) region proposals.These region proposals (of different shapes) mark regions of interest(of different shapes) on the CNN output. Then these regions ofinterest further extract features of the same shape (say height\(h_2\) and width \(w_2\) are specified) in order to beeasily concatenated. To achieve this, the fast R-CNN introduces theregion of interest (RoI) pooling layer: the CNN output and regionproposals are input into this layer, outputting concatenated featuresof shape \(n \times c \times h_2 \times w_2\) that are furtherextracted for all the region proposals.

3. Using a fully connected layer, transform the concatenated featuresinto an output of shape \(n \times d\), where \(d\) dependson the model design.

4. Predict the class and bounding box for each of the \(n\) regionproposals. More concretely, in class and bounding box prediction,transform the fully connected layer output into an output of shape\(n \times q\) (\(q\) is the number of classes) and an outputof shape \(n \times 4\), respectively. The class prediction usessoftmax regression.

The region of interest pooling layer proposed in the fast R-CNN isdifferent from the pooling layer introduced in Section 7.5.In the pooling layer, we indirectly control the output shape byspecifying sizes of the pooling window, padding, and stride. Incontrast, we can directly specify the output shape in the region ofinterest pooling layer.

For example, let’s specify the output height and width for each regionas \(h_2\) and \(w_2\), respectively. For any region of interestwindow of shape \(h \times w\), this window is divided into a\(h_2 \times w_2\) grid of subwindows, where the shape of eachsubwindow is approximately \((h/h_2) \times (w/w_2)\). In practice,the height and width of any subwindow shall be rounded up, and thelargest element shall be used as output of the subwindow. Therefore, theregion of interest pooling layer can extract features of the same shapeeven when regions of interest have different shapes.

As an illustrative example, in Fig. 14.8.3, the upper-left\(3\times 3\) region of interest is selected on a \(4 \times 4\)input. For this region of interest, we use a \(2\times 2\) region ofinterest pooling layer to obtain a \(2\times 2\) output. Note thateach of the four divided subwindows contains elements 0, 1, 4, and 5 (5is the maximum); 2 and 6 (6 is the maximum); 8 and 9 (9 is the maximum);and 10.

Below we demonstrate the computation of the region of interest poolinglayer. Suppose that the height and width of the CNN-extracted featuresX are both 4, and there is only a single channel.

import torchimport torchvisionX = torch.arange(16.).reshape(1, 1, 4, 4)X

tensor([[[[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.], [12., 13., 14., 15.]]]])

from mxnet import np, npxnpx.set_np()X = np.arange(16).reshape(1, 1, 4, 4)X

[22:09:37] ../src/storage/storage.cc:196: Using Pooled (Naive) StorageManager for CPU

array([[[[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.], [12., 13., 14., 15.]]]])

Let’s further suppose that the height and width of the input image areboth 40 pixels and that selective search generates two region proposalson this image. Each region proposal is expressed as five elements: itsobject class followed by the \((x, y)\)-coordinates of itsupper-left and lower-right corners.

rois = torch.Tensor([[0, 0, 0, 20, 20], [0, 0, 10, 30, 30]])

rois = np.array([[0, 0, 0, 20, 20], [0, 0, 10, 30, 30]])

Because the height and width of X are \(1/10\) of the height andwidth of the input image, the coordinates of the two region proposalsare multiplied by 0.1 according to the specified spatial_scaleargument. Then the two regions of interest are marked on X asX[:, :, 0:3, 0:3] and X[:, :, 1:4, 0:4], respectively. Finallyin the \(2\times 2\) region of interest pooling, each region ofinterest is divided into a grid of sub-windows to further extractfeatures of the same shape \(2\times 2\).

torchvision.ops.roi_pool(X, rois, output_size=(2, 2), spatial_scale=0.1)

tensor([[[[ 5., 6.], [ 9., 10.]]], [[[ 9., 11.], [13., 15.]]]])

npx.roi_pooling(X, rois, pooled_size=(2, 2), spatial_scale=0.1)

array([[[[ 5., 6.], [ 9., 10.]]], [[[ 9., 11.], [13., 15.]]]])

14.8.3. Faster R-CNN¶

To be more accurate in object detection, the fast R-CNN model usuallyhas to generate a lot of region proposals in selective search. To reduceregion proposals without loss of accuracy, the faster R-CNN proposesto replace selective search with a region proposal network(Ren et al., 2015).

Fig. 14.8.4 shows the faster R-CNN model. Compared withthe fast R-CNN, the faster R-CNN only changes the region proposal methodfrom selective search to a region proposal network. The rest of themodel remain unchanged. The region proposal network works in thefollowing steps:

1. Use a \(3\times 3\) convolutional layer with padding of 1 totransform the CNN output to a new output with \(c\) channels. Inthis way, each unit along the spatial dimensions of the CNN-extractedfeature maps gets a new feature vector of length \(c\).

2. Centered on each pixel of the feature maps, generate multiple anchorboxes of different scales and aspect ratios and label them.

3. Using the length-\(c\) feature vector at the center of eachanchor box, predict the binary class (background or objects) andbounding box for this anchor box.

4. Consider those predicted bounding boxes whose predicted classes areobjects. Remove overlapped results using non-maximum suppression. Theremaining predicted bounding boxes for objects are the regionproposals required by the region of interest pooling layer.

It is worth noting that, as part of the faster R-CNN model, the regionproposal network is jointly trained with the rest of the model. In otherwords, the objective function of the faster R-CNN includes not only theclass and bounding box prediction in object detection, but also thebinary class and bounding box prediction of anchor boxes in the regionproposal network. As a result of the end-to-end training, the regionproposal network learns how to generate high-quality region proposals,so as to stay accurate in object detection with a reduced number ofregion proposals that are learned from data.

14.8.4. Mask R-CNN¶

In the training dataset, if pixel-level positions of object are alsolabeled on images, the mask R-CNN can effectively leverage suchdetailed labels to further improve the accuracy of object detection(He et al., 2017).

As shown in Fig. 14.8.5, the mask R-CNN is modified basedon the faster R-CNN. Specifically, the mask R-CNN replaces the region ofinterest pooling layer with the region of interest (RoI) alignmentlayer. This region of interest alignment layer uses bilinearinterpolation to preserve the spatial information on the feature maps,which is more suitable for pixel-level prediction. The output of thislayer contains feature maps of the same shape for all the regions ofinterest. They are used to predict not only the class and bounding boxfor each region of interest, but also the pixel-level position of theobject through an additional fully convolutional network. More detailson using a fully convolutional network to predict pixel-level semanticsof an image will be provided in subsequent sections of this chapter.

14.8.5. Summary¶

• The R-CNN extracts many region proposals from the input image, uses aCNN to perform forward propagation on each region proposal to extractit* features, then uses these features to predict the class andbounding box of this region proposal.

• One of the major improvements of the fast R-CNN from the R-CNN isthat the CNN forward propagation is only performed on the entireimage. It also introduces the region of interest pooling layer, sothat features of the same shape can be further extracted for regionsof interest that have different shapes.

• The faster R-CNN replaces the selective search used in the fast R-CNNwith a jointly trained region proposal network, so that the formercan stay accurate in object detection with a reduced number of regionproposals.

• Based on the faster R-CNN, the mask R-CNN additionally introduces afully convolutional network, so as to leverage pixel-level labels tofurther improve the accuracy of object detection.

14.8.6. Exercises¶

1. Can we frame object detection as a single regression problem, such aspredicting bounding boxes and class probabilities? You may refer tothe design of the YOLO model(Redmon et al., 2016).

2. Compare single shot multibox detection with the methods introduced inthis section. What are their major differences? You may refer toFigure 2 of Zhao et al. (2019).