par 1 Introduction
1.1 Fundamentals of Neural Networks
1.1.1 Forward Computation: Neural Network as a Function
1.1.2 NeuralNetworkLearning
1.2 Matrix Representation of Neural Networks
1.3 The Development of Deep Learning and Its Factors
1.4 Appendix
1.4.1 Proof of the Backpropagation Formula
2 Fundamental Technologies Supporting Deep Learning
2.1 Enhancement of Stochastic Gradient Descent
2.1.1 Momentum Stochastic Gradient Descent (Momentum Method)
2.1.2 Adaptive Adjustment of Learning Rate
2.1.3 Momentum Method Adaptive Learning Rate
2.2 Dealing with the Gradient Vanishing/Diverging Problem
2.2.1 ReLU Function
2.3 Residual Connection
2.4 NormalizationofActivations
2.4.1 NecessityofNormalization
2.4.2 BatchNormalization
2.4.3 Layer Normalization
2.4.4 Instance Normalization
2.4.5 GroupNormalization
2.5 Appendix
2.5.1 Convergence Rate of Gradient Descent
2.5.2 Completeness of ReLU Functions
2.5.3 Subdifferential..
