Network Architecture

Network in Network

In Convolutional Nets, there is no such thing as “fully-connected layers”. There are only convolution layers with 1x1 convolution kernels and a full connection table. --Yann LeCun

Description of the operation: suppose that a conv layer which outputs an (N, F, H, W) shaped tensor where:

N is the batchsize
F is the number of conv filters
H, W are the spatial dimensions

Suppose the output is fed into a conv layer with F1 1x1 filters, zero padding and stride 1, the outputs will have shape (N, F1, H, W).

Quote from Google Inception: one big problem with the above modules, at least in this naive form is that even a modest number of 5x5 convolutions can be prohibitively expensive on top of a convolutional layer with a large number of filters. This leads to the second idea of the proposed architecture: judiciously applying dimension reductions and projections wherever the computational requirements would increase too much otherwise. This is based on the success of embeddings: even low dimensional embeddings might contain a lot of information about a relatively large image patch... 1x1 convolutions are used to compute reductions before the expensive 2x2 and 5x5 convolutions. Besides being used as reductions, they also include the use of rectified linear activation which make them dual-purpose.

An article on this subject.

HRNet

Architecture design idea is to maintain high-resolution representations through the whole process.

The above objective is achieved by (1) sequential multi-resolution subnetworks, (2) parallel multi-resolution subnetworks and (3) repeated multi-scale fusion .

On the contrary, we have high-to-low resolution networks, e.g. ResNet, VGG etc.

Note: Pay attention to the implementation of down sampling and upsampling. transposed convolution...

Attention Mechanism

Interpretability

  • (Gradient-based) Classification Activation Mapping

results matching ""

    No results matching ""