deeplearning

Collection of my notes on neural network and practical examples

:notebook: NN and DL :notebook:

Theory

Logistic Regression as a Neural Network

1. Logistic Regression as a Neural Network
2. Cost Function
3. Gradient Descent
4. Computational Graph
5. Logistic Regression Gradient Descent
6. Examples of Gradient Descent

   Python and Vectorization

7. Vectorization
8. Vectorized Implementation of Logistic Regression
9. Computation of Vectorized Logistic Regressions Gradient
10. Broadcasting in Python
11. Numpy Vectors
12. Justification of Logistic Regression Cost Function

    Shallow Neural Network

13. Neural Networks Overview
14. Vectorizing Implementation of Neural Network
15. Activation Functions
16. Derivatives of Activation Functions
17. Gradient Descent for Neural Networks
18. Random Initialization

    Deep Neural Network

19. Deep L-layer neural network
20. Forward Propagation in a Deep Network
21. Getting matrix dimensions right
22. Why deep network?
23. Building blocks of deep neural networks
24. Forward and Backward Propagation
25. Parameters vs Hyperparameters
26. What does this have to do with the brain?

    Machine Learning Application

27. Train\Dev\Test sets
28. Bias and Variance Tradeoff
29. Basic Recipe for Machine Learning

    Neural Network Regularization

30. Regularization
31. Dropout Regularization
32. Other Regularization Methods

    Optimization problem

33. Normalizing Inputs
34. Vanishing Exploding Gradients
35. Weight Initialization for Deep Networks
36. Numerical approximation of gradients
37. Gradient Checking

    Optimization Algorithms

38. Mini Batch Gradient
39. Exponentially weighted averages
40. Bias correction in exponentially weighted averages
41. Gradient descent with momentum
42. Optimization Algorithms RMSprop
43. Optimization Algorithms Adam
44. Learning rate decay
45. Local optima

    Hyperparameter tuning

46. Tuning Process

    Batch Normalization

47. Normalizing activations in a network
48. Fitting Batch Norm into a neural network
49. Why does Batch Norm work?
50. Batch Norm at test time

Multi-class classification

1. Softmax Regression
2. Training a softmax classifier

Deeplearning Programming Framework

1. Deeplearning programming framework
2. TensorFlow

:notebook: ML in Practice :notebook:

ML Strategy

1. Why Machine Learning Strategy
2. Orthogonalization

   Goal Setting

3. Single number evaluation metric
4. Satisficing and Optimizing metric
5. Train/dev/test distributions
6. Size of dev/test data
7. When to change dev/test sets and metrics

Comparing to human-level performance

1. Why human-level performance?
2. Avoidable bias
3. Understanding human-level performance
4. Surpassing human-level performance
5. Improving your model performance

Error Analysis

1. Error analysis
2. Cleaning incorrectly labeled data
3. Build system quickly then iterate

Mismatched training and dev/test set

1. Training and testing on different distributions
2. Bias and Variance with mismatched data distributions
3. Addressing data mismatch

Learning from multiple tasks

1. Transfer learning
2. Multi-task learning

End-to-end deep learning

1. What is end to end deep learning
2. Whether to use end to end deep learning

:notebook: Convolutional Neural Nets :notebook:

Theory

1. Computer Vision
2. Edge Detection
3. Padding
4. Strided Convolution
5. Convolutions Over Volume
6. One Layer Convolutional Network
7. Simple Convolutional Network
8. Pooling Layer
9. CNN Example
10. Why Convolutions are useful?

    Deep convolutional models: case studies

11. LeNet-5, ALexNet, VGG-16
12. ResNets
13. Networks in Networks and 1x1 Convolutions
14. GoogLeNet: Inception Network

ConvNet in Practice

1. Data Augmentation
2. State of Computer Vision

Object Detectction

1. Object Lcoalization
2. Landmark Detection
3. Object Detection
4. Convolutional Implementation of Sliding Windows
5. YOLO algorithm
6. IoU
7. Non max suppression
8. Anchor Boxes
9. R-CNN

Special applications: Face recognition & Neural style transfer

1. Face Recognition
2. neural style transfer
3. Convolutional Networks in 1D and 3D

Sequence models

1. Sequence Data Examples

RNN

1. Notations
2. Recurrent Neural Network Model
3. Language model and sequence generation
4. Sampling novel sequences
5. GRU
6. LSTM
7. Bidirectional RNN

Word Embedding

1. Word Representation
2. Learning word embeddings
3. Nueral Network Language model
4. Word2Vec
5. GloVe
6. Debiasing word embedding

Sequence to Sequence model

1. Basic Model
2. Picking the most likely sentence
3. Beam Search Algorithm
4. Error Analysis in Beam Search
5. Attention Model
6. Speech Recognition

Generative Adversarial Nets

• GAN

• BiGAN

  Applied GAN

• GAN for anormaly detection
  • AnoGAN
  • EGBAD

Summation of Important Papers

• [Word2vec]
• [FastText]
• [Elmo]
• seq2seq
• Transformer
• Universal Language Model Fine-tuning for Text Classification
• [Tranformer XL]
• [Universal Sentence Encoder](https://stomioka.github.io/deeplearning/docs/
• [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding]
• [GPT2]
• [MN-DDN]
• [XLNet]

TensorFlow Basic

• Using Callbacks to control training
• Simple covnet
• MNIST classification example
• Convnet with TF
• Image augmentation

NLP in TensorFlow

• Sentiment Analysis
• Embeddings
• Sentiment Analysis - Embeddings
• IMDB Subwords 8K with Single Layer LSTM
• IMDB Subwords 8K with Multi Layer LSTM
• IMDB Subwords 8K with 1D Convolutional Layer
• Sarcasm with Bidirectional LSTM
• Sarcasm with 1D Convolutional Layer

IMDB Reviews with GRU (and optional LSTM and Conv1D)

Implementations in Python and R

Implementations in R:

1. Neural Network Classification
2. Simple 2 hidden layer network with Keras

Implementations in Python:

1. Building an image recognition algorithm using logistic regression with neural network mindset
2. Building a 2 layer neural network for binary classification problem
3. Building neural network utilities
4. Building a 2 layer neural network and deep learning neural network from scratch
5. Initialization
   • initialize_parameters_zeros
   • initialize_parameters_random
   • initialize_parameters_he
6. Regularization
   • compute_cost_with_regularization
   • backward_propagation_with_regularization
   • forward_propagation_with_dropout
   • backward_propagation_with_dropout
7. Gradient Checking
   • gradient_check
   • gradient_check_n
8. Optimization
   • update_parameters_with_gd
   • random_mini_batches
   • initialize_velocity
   • update_parameters_with_momentum
   • initialize_adam
   • update_parameters_with_adam
9. TensorFlow: Build neural network for multiclassification problem
10. Building Convnet with numpy
11. Building Convnet with TensorFlow
12. Building ResNets with Keras
13. AEDECOD and AETERM similarity with Universal Sentence Encoder with Transformer
14. Objects detection using pre-trained Yolo weights
15. Neural Style Transfer
16. Face Recognition
17. Character level language moodel - LSTM
18. Generate Text - LSTM
19. NLP tasks