深度学习入门(7) - Video Understanding

Videos

Due to the memory constraints, we have to do some down-sampling

Raw video: long high fps

Training: short clips with low fps

testing: run model on different clips, averaging predictions

Single-frame CNN

train normal 2D CNN to classify frames independently!

easy but a very strong baseline!

Late fusion

take the time axis into account

(flatten / average pooling) concatenate the results of CNNs and feed to MLP to get a classification score

Problem: Hard to compare low-level motion between frames

Early Fusion

compare frames with very first conv layer after that normal 2D CNN

then pass a 2D CNN to get class score

Problem: only one layer of the temporal processing may be not enough

3D CNN (slow fusion network)

use 3D conv and 3D pooling operations

C3D: The VGG of 3D CNNs

Measuring Motion: Optical Flow

Optical Flow gives a displacement field F between image t and image t+1.

Two-Stream Networks

combine Spatial stream Convnet and Temporal stream Convnet to a large vector and get a class score

Train the two Convnets seperately and take averages in Test time

Modeling long-term temporal structure

we can extract features with CNN (2D / 3D) and use lstm in the time axis

trick: sometimes we don’t backprop to CNN to save memory (use them as a feature extractor)

Recurrent Convolutional Network

Spatio-Temporal Self-Attention (Nonlocal Block)

Inflating 2D Networks to 3D (I3D)

take a 2D CNN architecture and replace each 2D conv/pool with a 3D version

we can inflate the trained weighted as well, duplicate them in time

pre-train model on images -> inflate them into videos -> fine-tuning

We can use the same visualizing tricks

SlowFast Networks

Other aspects: Temporal Action Localization, Spatio-Temporal Detection