Just imagine what you are losing just because you are not reading!

— Ernest Agyemang Yeboah

I am revisiting the book titled, Introduction to Statistical Learning with Applications in R, after 7 years. It was back in 2013, when I read through this book for the first time, and worked through the book. Needless to say R ecosystem has expanded greatly since then. I have done many projects in the field of data science and have grown a bit wiser. This blogpost summarizes my re-learnings from this fascinating book, that is a bridge to Elements of Statistical Learning, a book that is considered the bible of Statistical Learning

This blog post contains the main points from the paper titled, ESG Investments: Filtering versus Machine Learning approaches .

This blog post talks about some of my learnings from the book “Mathematical Notation”

This blog post summarizes the book titled “The Formula - The Science Behind Why People Succeed or Fail”

This blog post summarizes the book titled “Python Tricks - A Buffet of Awesome Python Features”

This blog post summarizes the book titled “You Look Like a Thing and I Love You”

The following post contains a summary of the book titled Treading on Python II by Matt Harrison

Programming Styles

• Python supports three types of programming paradigms
  • Imperative/Procedural
  • Object Oriented
  • Declarative/Functional

Iterator Protocol

• iter is a global built-in function that calls the object’s dunder method __iter__
• Writing a for loop based on iterators

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

test = [1, 2, 4]
for i in test:
    print(i)

iterator = iter(test)
while True:
    try:
        x = iterator.__next__()
        print(x)
    except StopIteration as e:
        break

• Each loop is converted in to byte code and this byte code is run by the interpreter
• The actual iterator is not the object that is being iterated. list and string have separate iterator objects to iterate upon them
• StringIO class implements the iterator protocol
• Iterator protocol defines the process of iterating the objects in a container utilizing the methods __iter__ and __next__

Iterable vs. Iterator

• What is an iterable ? An iterable is any object that allows iteration
  • This object must implement __iter__ method and must return an iterator object. This iterator object can be the same object or a completely different object
  • This object must also implement __next__ method
• Iterators are good for one pass over the values. This means that iterators are stateful
• range(10) returns an rangeiterator object that implements __iter__ and __next__ methods
• A class is called a self-iterator if its __iter__ method returns the same instance on which the dunder method has been invoked
• Most iterable objects are not self-iterators. They return a different object when their __iter__ method is invoked
• If the datatype is a self-iterator, then there could be problems in nested loops. Here is a nice example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

class Counter(object):
    def __init__(self, size):
        self.size = size
        self.start = 0
    def __iter__(self):
        return self
    def __next__(self):
        if self.start < self.size:
            self.start +=1
            return self.start
        raise StopIteration

x = Counter(2)
y = Counter(3)
for i in x:
    for j in y:
        print(i,j)

The above code does not work as desired as the iter returns the same instance and the inner loop goes through only once and never gets repeated. The solution to this problem is to make sure that the iter method returns a different object as compared to the original object on which the method was invoked

The following post contains a summary of the book titled Tiny Python Projects by Ken Youens-Clark

What did I learn from going through 2 hours of videos on Pytest ?

Python Testing 101

• pytest the most popular Python package for testing
• it is also a basis for rich ecosystem for testing plugins
• unittesting comes with Python. It is used to test the internals of core python. It is a good solid tool but there are a lot of api calls that one might have to learn
• The test should be divided in to three stages
  • Arrange : Setting up test, variables, data structure
  • Act : Execute the code on the above setting
  • Assert : Assert the way the code has run the test
• If you are building a user interface, you build a CLI tool and then keep adding additional tests in pytest scripts

Python Testing 202

• .’s are used to represent the number of functions tested in a test file
• tests can expect an Exception and pytest can be used to check whether the right exceptions are occurring in the code.
• What if you want to test more examples ?
• The first assertion that fails aborts the rest of assertions
• One can test a bunch of examples with parametrized feature
• method is a function attached to a class
• One can group the functionality in to a class
• Pytest incorporates fixtures that helps you incorporate the set up that goes along with testing a function
• fixtures should be a part of conftest.py
• pytest has a built-in fixtures
• pytest has a rich set of ecosystem that gives a set of variety of new fixtures
• fixtures for setup/reuse
• one can organize the test code in to classes

Takeaway

After working through the examples, I am now much more comfortable in going through the book on pytest. There is no doubt that I will be using all these things in the UOB project implementation

The following post contains a summary of the book titled Python Workout by Reuven M. Lerner

The following are the learnings from Hugging Face Interview in Oct 2019

• GPT2 from Open AI is impressive - Packaged in to Demo Application
• Conversational AI + Open Source package(Transformers)
• Half a million monthly active users
• Hard to good Deep Conversational AI
• Self starter - Was working in 2008 on ML and then moved on to do some software jobs
• I was curious to see what the number of downloads for various pre-trained models were. So, wrote a small Python program to get the downloads

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import torch
import pandas as pd
from bs4 import BeautifulSoup
import requests
url                = "https://huggingface.co/models"
response           = requests.get(url)
html_soup          = BeautifulSoup(response.text, 'html.parser')
models             = html_soup.find(class_ = 'models-list')
model_names=[]
for a in models.find_all('li'):
    model_names.append(a.a['href'])

model_names  = [m[1:] for m in model_names]
download_counts = []
for a in models.find_all('li'):
    count = a.find(class_='tooltip').text.strip().split("downloads")[0].strip()
    download_counts.append(count)
model_stats = pd.DataFrame({'model':model_names,'downloads_l30days':download_counts})
model_stats.head(20)

Here are the top 25 models as of [2020-07-06 Mon]

Watched a fantastic Ted Talk that highlighted the importance of gamifying learning

• What if you looked at your learning as similar to playing Super Mario
• You focus on princess and all the rest of the steps are your learnings on the way
• Life as a straight path is never a story worth telling
• Turn any learning process in to game and then things become super interesting
• Research that showed no penalty means increased attempts and better score
• Nobody gets disappointed when the italian plumber falls in to a ditch - They just learn that they need to be careful at that level, the next time they play
• 3 year effort - dart board moves based on how one throws the dart
• Redesign boring tasks to games

The following are my learning from the paper titled, Attention is all you need :

• Using RNN’s for language modeling has been particularly painful as they take long time to train and have problems with learning representational encodings all at once
• In Transformer architecture, the number of operations required to relate signals from two arbitrary input or output positions is a constant
• Self-attention is an attention mechanism relating different positions of a single sentence in order to compute a representation of a the sequence
• Transformer is a first transduction model relying entirely on self-attention to compute representations of its inputs and output without using sequence aligned RNNs or convolution networks
• Learnt about the relationship between Induction, Deduction and Transduction
  • Induction, derives the function from the given data, i.e. creates an approximating function
  • Deduction derives the values of the given functions for points of interest
  • Transduction derives the values of an unknown function for points of interest from the data

What did I learn from attention primer from Jay Allamar

What did I learn from CodeEmporium about BERT and Transfomers

What did I learn from transformer primer from Jay Allamar

This blog post summarizes the book titled “Information Theory”

The following are my learnings from the book .

The following are the takeaways from the course:

The following are the learnings from the podcast:

The following are the learnings from the podcast:

The following are the learnings from the podcast:

The following are the takeaways from Ian GoodFellow Interview.

Kyle Polich discusses the limitations of Bert Model.

This post summarizes the book, “Indistractable”.

My learnings from this article on parallel processing :

My learnings from attending a 3 hour session on Deep Learning :

The following is an excellent summary of Data Leakage in time series testing.

The following are my learnings from the Meetup talk by Prof Hyndman .

The following are my learnings from the Hyndman-MASE .

Kyle Polich discusses BERT. The following are my takeaways.

The following are my takeaways from the talk, Embedding work in NLP .

The following are some of the takeaways from this medium article.

In this brief post, I would like to pen down my thoughts on two aspects: Heuristics and Non-Intepretability of models.

Let’s look at word embedding matrix. If you take a bunch of words and want to build a learning algorithm, the first task is to convert the text in to a bunch of numbers. The two popular algorithms that have revolutionized the field of NLP are Skipgram method and CBOW method. Both involve learning a lower dimensional representation of the word. The dimensions are not interpretable as the dimensions are not unique. The fact that dimensions are not interpretable did not stop someone from developing fantastic applications. Suppose you are in foreign country and you are lost and want to check with someone the correct way to your destination: You flip open your phone, speak your native language and your phone translates the sentence to a foreign language (text/audio), and use it converse with strangers. The job gets done. Do you really care how the word embedding algo is working ? Not really. So, we don’t need to be hung up in intepretability for all applications. In trading for example, if the strategy makes money, you might not care too much about the interpretability of the strategy.

This post illustrates CNN LSTM Models that can be fit to simple time series data.

This post illustrates the early stopping callback in Keras.

This post illustrates generative LSTM via a plain vanilla LSTM.

This post creates a Bidirectional LSTM and learns a simple pattern in the sequence.

This post has two pop quizzes relating to the output of LSTM.

This post creates a plain vanilla LSTM and learns a simple pattern in the sequence.

This post creates a sequence to sequence LSTM and learns a simple pattern in the sequence.

This post creates a Stacked LSTM and learns a simple pattern in the sequence.

This post attempts to summarize the contents of the book Ultra learning by Scott H Young.

The following are my learnings from Jose course on NLP.

This post gives some of the learnings from the deliberate practice on spacy.

I was able to undersand the basic idea behind RNN after working through an example from the book by Antonio Gulli.

I was puzzled with the way LSTMs were used to do sentiment analysis. Finally the book by Antonio Gulli helped me understand the mechanics of the LSTM.

In this article, I will explain the way you can code a simple RNN that tracks a simple shift in the pattern, i.e a value from a normal distribution.

As compared to previous implementations where we had used OutputProjectionWrapper, this code does away with that component and does it more efficiently

Create Training and Validation Data

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import numpy as np
import re
from sklearn.model_selection import train_test_split
import tensorflow as tf
input_seed = 1234
time_steps = 24
n_samples = 100000
X = np.random.randint(1,30,n_samples*time_steps).reshape(n_samples, time_steps)
Y = np.apply_along_axis(lambda x : x + np.random.normal(3, 1, 1),1,X)
X = X.reshape(X.shape[0],X.shape[1],1)
Y = X.reshape(Y.shape[0],Y.shape[1],1)
np.random.seed(input_seed)
idx     = np.arange(len(X))
np.random.shuffle(idx)
X, Y    = X[idx,:,:], Y[idx,:,:]
X_train, X_valid, Y_train, Y_valid = train_test_split(X,Y,test_size=0.25, random_state = input_seed)

Set up the RNN Model in TensorFlow

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

tf.reset_default_graph()
hidden_units  = 32
tf_X          = tf.placeholder(tf.float32, shape=[None, time_steps, 1])
tf_Y          = tf.placeholder(tf.float32, shape=[None, time_steps, 1])
rnn_cell      = tf.contrib.rnn.BasicRNNCell(num_units= hidden_units, activation=tf.nn.relu)
outputs, states  =tf.nn.dynamic_rnn(rnn_cell, inputs = tf_X,
                                   dtype=tf.float32)

stacked_outputs = tf.reshape(outputs,[-1,hidden_units])
stacked_outputs = tf.contrib.layers.fully_connected(stacked_outputs, 1,activation_fn=None)
outputs = tf.reshape(stacked_outputs,[-1,time_steps,1])
loss           = tf.square(outputs - tf_Y)
total_loss     = tf.reduce_mean(loss)
learning_rate  = 0.001
optimizer      = tf.train.AdamOptimizer(learning_rate= learning_rate).minimize(loss=total_loss)
batch_size =1000
n_batches = int(X_train.shape[0]/batch_size)
epochs = 20
i      = 0

Train the Model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for e in range(epochs):
      idx     = np.arange(len(X_train))
      np.random.shuffle(idx)
      X_train, Y_train    = X_train[idx,:,:], Y_train[idx]
      for i in range(n_batches):
          x  = X_train[(i*batch_size):((i+1)*batch_size),:,:]
          y  = Y_train[(i*batch_size):((i+1)*batch_size),:,:]
          _, curr_loss = sess.run([optimizer, total_loss],
                                 feed_dict={tf_X:x, tf_Y:y})
      loss_val,output_val = sess.run([total_loss,outputs], feed_dict={tf_X:X_valid, tf_Y:Y_valid})
      print("Epoch:",str(e), " Loss:", loss_val)

The output from testing validation data is

In this article, I will explain the way you can code a simple RNN that tracks a simple shift in the pattern

Create Training and Validation Data

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import numpy as np
import re
from sklearn.model_selection import train_test_split
import tensorflow as tf
input_seed = 1234
time_steps = 24
n_samples = 100000
X = np.random.randint(1,30,n_samples*time_steps).reshape(n_samples, time_steps)
Y = np.apply_along_axis(lambda x : x + 10,1,X)
X = X.reshape(X.shape[0],X.shape[1],1)
Y = X.reshape(Y.shape[0],Y.shape[1],1)
np.random.seed(input_seed)
idx     = np.arange(len(X))
np.random.shuffle(idx)
X, Y    = X[idx,:,:], Y[idx,:,:]
X_train, X_valid, Y_train, Y_valid = train_test_split(X,Y,test_size=0.25, random_state = input_seed)

Set up the RNN Model in TensorFlow

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

tf.reset_default_graph()
hidden_units  = 32
tf_X          = tf.placeholder(tf.float32, shape=[None, time_steps, 1])
tf_Y          = tf.placeholder(tf.float32, shape=[None, time_steps, 1])
rnn_cell      = tf.contrib.rnn.OutputProjectionWrapper(tf.contrib.rnn.BasicRNNCell(
                                                        num_units= hidden_units,
                                                        activation=tf.nn.relu),
                                                    output_size = 1)
outputs, states  =tf.nn.dynamic_rnn(rnn_cell, inputs = tf_X,
                                   dtype=tf.float32)

loss           = tf.square(outputs - tf_Y)
total_loss     = tf.reduce_mean(loss)
learning_rate  = 0.001
optimizer      = tf.train.AdamOptimizer(learning_rate= learning_rate).minimize(loss=total_loss)
batch_size =1000
n_batches = int(X_train.shape[0]/batch_size)
epochs = 20
i      = 0

Train the Model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for e in range(epochs):
      idx     = np.arange(len(X_train))
      np.random.shuffle(idx)
      X_train, Y_train    = X_train[idx,:,:], Y_train[idx]
      for i in range(n_batches):
          x  = X_train[(i*batch_size):((i+1)*batch_size),:,:]
          y  = Y_train[(i*batch_size):((i+1)*batch_size),:,:]
          _, curr_loss = sess.run([optimizer, total_loss],
                                 feed_dict={tf_X:x, tf_Y:y})
          #print("Epoch:",str(e), "Batch:",str(i), "loss:",str(curr_loss))
      loss_val,output_val = sess.run([total_loss,outputs], feed_dict={tf_X:X_valid, tf_Y:Y_valid})
      print("Epoch:",str(e), " Loss:", loss_val)

The output from testing validation data is