My learnings from attending a 3 hour session on Deep Learning :

Code Links

GitHub repo: Workshop slides:


  • Everyone is doing ResNet
  • Transformers - Cutting edge models in NLP
  • - Lead by NVIDIA - Uses GPU
  • ArXiv - 25000 papers every year
  • AI’s compute > Moore’s law
  • Changes in TF2.0
  • No more graphs and sessions - Use Keras API + tf.function decorator
  • tf.contrib – Tensor flow add-ons
  • Eager execution
  • Functional API solves 99% of the usecases
  • Model Subclassing is possible - Autoencoder
    • Encodes in a dense vector
    • Decodes in to a transformed vector
    • Choose encoder or decoder separately
  • Standalone keras will be stopped development after 2.3
  • Send the data faster to GPU so that there is less time spent on sending data
  • is quick - Data flows efficiently to GPU
  • Idle time on GPU can be reduced
  • can help you do a parallel processing
  • shuffle, map, batch, prefetch - Datasets that cannot fit in RAM
  • Loading code is same -
  • Keras automatically runs on GPU
  • XLA - Little performance boost if you are using standard architectures
  • What parts of the model can be in 16 bit ? 32 bit ?
    • TPUs - Mixed Precision
  • What is the policy that keras should use for precision ?
  • Tensorflow lite works for mobile devices etc
  • XLA might not give a perf boost
  • Model that cannot be fit on one GPU can be split on multiple GPUs - Quite rare
  • Horovod - Train on one GPU script - Magically scale it to multiple GPUs
    • Takes control of optimizer and makes scaling possible
    • Automatically used relevant libraries to do it
  • tf.distribute allows you to distribute to multiple GPUs
    • You can set distribution strategy to place model on different GPUs


  1. I don’t have to parallelize my code at all. I use the GPU and do it one time series after another. Instead of trying to parallelize stuff. Keras automatically uses all cores on the machine
  2. Access to a bunch of notebooks that I can use to learn ML on TF2
  3. Broad understanding of
  4. Broad understanding of XLA and Automatic Mixed Precision
  5. Understanding of basic structure of colab
  6. Got inspired by the undergrad who had really put in a lot of effort in DL