The following is a nice example from Michael Hardy that shows how excessive focus on unbiasedness of an estimator leads to nonsense results :

A light source is at an unknown location µ somewhere in the unit disk D. A dart thrown at the disk strikes some random location U in the disk, casting a shadow at a point X on the boundary. The random variable U is uniformly distributed in the disk, i.e., the probability that it is within any particular region is proportional to the area of the region. The boundary is a translucent screen, so that an observer located outside of the disk can see the location X of the shadow, but cannot see where either the light source or the opaque object is. Given only that information—the location X of the shadow—the location µ of the light source must be guessed. What’s your estimate of µ, given X ?

A whistleblower( someone working in a hedge fund at Singapore) alleges that NSE had indulged in market manipulation to the tune of 100 crores over a period of three years. What are the allegations against NSE, Omnesys and others?

• Tick by Tick data was disseminated via TCP/IP instead of UDP for three years and there was no load balancer in place.This meant that the person who could connect to the server that had the least load and be first in the queue was at an advantage. The information on server loads was leaked out.
• The info of the exact start of time of the primary and backup servers was leaked out to Omnesys and an unnamed individual
• Massive Multi legged IOC orders were executed by those who were privy to the primary and backup server start times
• NSE provides certain special privileges to hedge funds and HFT shops based out in Singapore, who then use to game the system

Link to the whistleblower’s detailed note

Making beautiful and powerful music is a physical act - pulling, scraping, beating, and straining wood, steel and muscles. Music is not made. It is wrought. Sounds are wrung from wood and steel, Fingers pull metal. Horsehair scrapes gut. And out of the pulling, beating, scraping and straining comes that most intangible, that most beautiful and powerful of things . . . music.

This book is vastly different from the books that try to warn us against incorrect statistical arguments present in media and other mundane places. Instead of targeting newspaper articles, politicians, journalists who make errors in their reasoning, the author investigates research papers, where one assumes that scientists and researchers make flawless arguments, at least from stats point of view. The author points a few statistical errors, even in the pop science book, “How to lie with statistics?’’. This book takes the reader through the kind of statistics that one comes across in research papers and shows various types of flawed arguments. The flaws could arise because of several reasons such as eagerness to publish a new finding without thoroughly vetting the findings, not enough sample size, not enough statistical power in the test, inference from multiple comparisons etc. The tone of the author isn’t deprecatory. Instead he explains the errors in simple words. There is minimal math in the book and the writing makes the concepts abundantly clear even to a statistics novice. That in itself should serve as a good motivation for a wider audience to go over this 130 page book.

Via Zenpencils

Go placidly amid the noise and haste, and remember what peace there may be in silence.

As far as possible, without surrender, be on good terms with all persons. Speak your truth quietly and clearly; and listen to others, even to the dull and the ignorant, they too have their story. Avoid loud and aggressive persons, they are vexations to the spirit.

If you compare yourself with others, you may become vain and bitter; for always there will be greater and lesser persons than yourself. Enjoy your achievements as well as your plans. Keep interested in your own career, however humble; it is a real possession in the changing fortunes of time.

When physicists do mathematics, they don’t say they’re doing “number science”. They’re doing math. If you’re analyzing data, you’re doing statistics. You can call it data science or informatics or analytics or whatever, but it’s still statistics.

- Karl Broman

I was looking to code Metropolis-within-Gibbs sampler for a specific model and in that process, stumbled on to an online supplement for a work-in-progress book,``Handbook of Modern Item Response Theory’’.  The following document summarizes the main points from chapter-15 of the book. The document also contains WinBUGS code to estimate the parameters of 2PL model.

Building Metropolis Hastings sampler from scratch

“You never quit on your music. No matter what happens. Cuz anytime something bad happens to you, that’s the one place you can escape to and just let it go. I learned it the hard way”

- August Rush

Modeling Capture-Recapture Data via Bayes involves specifying the likelihood of the survival + encounter probabilities and the priors on the parameters. A natural extension of a plain vanilla model is to make the survival probabilities time dependent. The paper titled, “Autoregressive models for capture–recapture data: A Bayesian approach”, is about modeling the survival probabilities by taking various covariates such as time and random effects.

The following document contains a brief summary of the paper :

AR(2) estimation in Bayes from first principles involves writing an M-H sampler as full conditional distributions for some of the parameters does not belong to any standard form. However softwares like WinBUGS and JAGS make the life easy for a modeler. The following document contains some basic code to do Bayesian inference for AR(2) process using WinBUGS and JAGS

AR(2) parameter estimation in WinBUGS and JAGS

The paper, “Understanding the Metropolis-Hastings Algorithm” by Chib and Greenberg is a vintage paper that illustrates the nuts and bolts of the algorithm.  The following is a link to the document that summarizes the paper.

Summary of Metropolis Hastings Algorithm

Summary of the main points from the paper

The book serves a nice intro to Bayes theory for an absolute newbie. There is minimal math in the book. Whatever little math that’s mentioned, is accompanied by figures and text so that a newbie to this subject “gets” the basic philosophy of Bayesian inference. The book is a short one spanning 150 odd pages that can be read in a couple of hours.  The introductory chapter of the book comprises few examples that repeat the key idea of Bayes. The author says that he has deliberately chosen this approach so that a reader does not miss the core idea of the Bayesian inference which is,

Via boingboing:

Someone stole $90 million from a company I was involved in. I’m a poor judge of people. The company collapsed.

Some things I can’t learn. I tend to like people too much.

So it’s hard for me to be a good judge of people, no matter how much I try. So I find other people who are good at judging people and I ask them to help me.

Don’t force yourself to learn something if you don’t want to or it’s not a natural talent.

Link : A brief summary of the book

 Takeaway

“Write your code as though you are releasing it as a package” - This kind of thinking forces one to standardize directory structure, abandon adhoc scripts and instead code well thought out functions, and finally leverage the devtools functionality to write efficient, extensible and shareable code.

Via No Mobile Phones …

I’m not interested in daily chores. We have now swapped information for knowledge, which is not the same thing. I do not want to know. I’m not online. I don’t even have a computer.

- Brunello Cucinelli

The paper titled, “Discerning Information from Trade Data” by David Easley, Marcos Lopez de Prado, Maureen O’Hara, gives a Bayesian framework for trade classification. The most popular method for classifying a trade as buy/sell is via “tick test”. The authors introduced Bulk Volume Classification (BVC) and empirically test the performance of it vis-à-vis tick test. In this post, I will briefly summarize the paper :

**Introduction
**With the advent of HFT, the markets have changed completely. Order cancellations and modifications have shot up as compared to yesteryears. Execution side algos chop orders and send it across to the exchange and hence it is order flow rather than individual orders that relate to trade motivation. Also with strategies such as “persistent bidder”, where the aggressive trader uses limit orders to trade, the vital link between informed traders and aggressive traders is lost. All these implications severely undermine algos that infer trade direction from a single trade.

Via Gauri :

I had to read this paper again after ~1.5 years as I had forgotten the basic idea behind the classification. My understanding, this time was far better than the previous encounter. In this post, I will list down a few points form the paper

One can think of three ways to classify trades as “buy” or “sell” trades.

1. Tick test : Compare the current trade price to the previous trade price

The paper titled,”Spectra of some self-exciting and mutually exciting point processes”, is one of the most widely cited papers in marked point process literature. I guess this was the first paper that explored the complete covariance density function of point processes, and in particular, self exciting and mutually exciting processes. In the time series literature, the covariance of  a stationary process at various lags have special meaning. If you consider the generating function of the covariance at various lags and evaluate at a specific complex exponential, you arrive at population spectrum of the time series. The population spectrum thus obtained has several interesting applications.

The following note is motivated by the blog post, “Bitcoin Trade Arrival as Self-Exciting Process”. Since the author has shared the data and code, I wanted to check some of the numbers from the post. The author uses “ptproc” which is deprecated library and has been removed from CRAN. In this note I have used the trades dataset from the author’s github directory and fit a self-exciting model to the trade arrivals. My analysis shows that the data does not fit Hawkes process, contradicting the blog post conclusion. In fact when I looked at the code at github, I found the task of adding random milliseconds missing. The sanitized dataset used after randomized addition is double the size of the original trades data. Clearly there is something wrong with the data.

I have stumbled on to a few mini-projects that revolve around fitting univariate and bivariate Hawkes processes. In this post, I will briefly summarize the write ups :

High Frequency Trade Prediction with Bivariate Hawkes Process

The authors starts with a SDE for intensity process and formulate its solution as a univariate Hawkes process. A visual depiction of self-excited intensity process is obtained via simulation. The time change theorem is stated and a QQ plot of the compensator is shown to follow an exponential inter-arrival distribution. The same thing is repeated for bivariate mutually exciting Hawkes process. Expressions for the log-likelihood of the bivariate Hawkes process is stated and MLE results are shown on a simulated dataset so that the estimates can be compared to the true values. TAQ database is used to obtain tick data for DELL, YHOO and ORCL stock. Since the data is discretized in whole seconds, the timestamps that share the same second are uniformly redistributed in the overlapping second. Using Lee and Ready tick algo, the trades are categorized as buy or sell trades. Bivariate Hawkes process are fit to the buy and sell trades. This model is put to test on a strategy where 1) stocks are longed if buy intensity > 8 times sell intensity, 2) stocks are shorted if the ratio drops below 1/8.

The paper titled, “Self-Exciting Point Process Models of Civilian Deaths in Iraq”, deals with fitting point processes to civilian deaths from March 2003 to December 2007. In this post, I will summarize main points from the paper

Firstly, What is “Operation Iraqi Freedom” ? Here’s a wiki blurb

The 2003 invasion of Iraq lasted from 19 March to 1 May 2003 and signaled the start of the conflict that later came to be known as the Iraq War, which was dubbed Operation Iraqi Freedom by the United States. The invasion consisted of 21 days of major combat operations, in which a combined force of troops from the United States, the United Kingdom, Australia and Poland invaded Iraq and deposed the Ba’athist government of Saddam Hussein. The invasion phase consisted primarily of a conventionally fought war which concluded with the capture of the Iraqi capital of Baghdad by American forces.

The paper, written by Ioane Muni Toke and Fabrizio Pomponio, titled, Modeling Trades-Through in a Limit Order Book Using Hawkes Processes, uses Hawkes process to examine microstructure behavior.

This paper uses Multivariate Hawkes process to model trades-through. The best thing about this paper is that the authors have made the dataset available for the readers so that they can work through the numbers
and get a feel of model inference. The dataset is available at dataverse. I have used the dataset from the repository, crunched numbers and have managed to replicate most of the results in the paper. I hope this feature of “Reproducible Research” becomes more widespread and authors start disseminating their datasets along with their papers. In this blog post, I will summarize the main points of the paper.

Via TP (Techcrunch) :

Today, Silicon Valley is the hottest place for quants to be – though people with this skill set are often referred to now as data scientists. A similar confluence of factors — data, technology and algorithms — has combined to enable a new class of transformational opportunities. These opportunities are not limited to just financial services; they are showing up in every sector of the economy.

The volume and variety of data sources has exploded, with companies now regularly directly capturing all manner of user web and mobile traffic, e-commerce and real-world transactions, social profile information, location and even sensor data. In addition, there are vast pools of third-party data available through APIs for everything from advertising and beauty to yellow pages and ZIP codes.

The following document contains a brief summary of the paper titled, “Autoregressive Conditional Duration - A New Model for Irregularly Spaced Transaction Data” by Engle and Russell.

ACD-Summary

Takeaway :

The paper models the duration between transactions.With the ease of availability of HF data, there needs to be a model that captures irregularly spaced timestamps. It is obvious that neither a standard Poisson process nor a non-homogeneous Poisson process is going to be a good fit. The conditional intensity function has to depend on the past history of transaction times. The authors formulate a particular form for the conditional intensity function and explain the clustering phenomenon as well as some well known market microstructure patterns.

Gregory LaBerta, a Special Agent at FBI has come up with a 35 page document that gives a detailed description of the trades executed by Navinder Singh Sarao, who is accused of playing a major role in the “Flash crash” on May 6, 2010.

I am surprised with this new finding as it comes out of nowhere and that too after 5 looong years. For the last 5  years, there has been a debate; the majority view being that the whole HFT community played a big role in the “Flash crash”. One could see harsh criticism on the HFT guys by academicians, low+mid frequency traders, regulators etc. Michael Lewis book on HFT created furor in the entire investing community. Now after all the noise,  FBI thinks that it is ONE guy ( who sits at his parents house with a couple of terminals + toxic trading strategies) IS mainly responsible for “Flash crash”.  Is it believable? Can one guy really destabilize the highly liquid E-minis market ? We will know based on the outcome of his trial.

The paper, written by Yosihiko Ogata, titled, ``On Lewis Simulation Method for Point Process", gives a detailed procedure to simulate univariate and multivariate point processes. The following document contains the algo and necessary R code for simulating univariate Hawkes’ self exciting process. The good thing about this algo is that is based on “thinning” and hence it is not a computationally expensive task.

Simulation of univariate Hawkes via Thinning

T.Ozaki’s paper titled, “Maximum likelihood estimation of Hawkes’ self-exciting point processes”, deals with the univariate Hawkes’ process. The paper gives a detailed method to obtain the ML estimates of the process. In order to verify the ML estimates, the author simulates the process and compares the true vs. estimated parameters. I found a typo in one of the expressions for the gradient. In any case, R provides nlm function, that computes the gradient and hessian numerically. So, if one chooses to, one can slog out the expressions for gradient and hessian, and then feed in to nlm , else, you can allow the function to do the job numerically.

Today, I had a chance to see a play, “Kuchh Bhi Ho Sakta Hai”. It is a one person act and in this case, it is Anupam Kher, who narrates his own life story in a way that keeps the audience entertaining right through. There was not a single dull moment in the entire play. Thoroughly entertaining 2 hour act.

Homogeneous Poisson process (HPP) is easy to simulate in R as there are standard functions built in to the language. Simulating Non-Homogeneous Poisson process(NHPP) take a little bit of effort. There are many algorithms out there and I guess the best amongst them is based on “thinning”. The math behind thinning is explained in the paper,”Simulation of NonHomogeneous Poisson Process  by Thinning”. The paper, while explaining the method, also gives a laundry list of methods that can be used to simulate NHPP.

Via Barrons

The world has a way of making what was once controversial safe for consumption. For evidence, look no further than Lou Reed’s upcoming induction into the Rock and Roll Hall of Fame. The same holds true for Virtu Financial, which successfully completed its initial public offering last week.

That Virtu (ticker: VIRT) is now a publicly traded company is a marvel, considering the failure of its first attempt about a year ago. That coincided with the release of Michael Lewis’ book Flash Boys, which argued that high-frequency traders rigged the market in their favor. With so much venom being spewed in its direction, Virtu pulled its IPO.

Via The Moral Bucket List

If you live for external achievement, years pass and the deepest parts of you go unexplored and unstructured. You lack a moral vocabulary. It is easy to slip into a self-satisfied moral mediocrity. You grade yourself on a forgiving curve. You figure as long as you are not obviously hurting anybody and people seem to like you, you must be O.K. But you live with an unconscious boredom, separated from the deepest meaning of life and the highest moral joys. Gradually, a humiliating gap opens between your actual self and your desired self, between you and those incandescent souls you sometimes meet.

The paper written by Thierry Ane and Helyette Geman, titled, “Order Flow, Transaction Clock, and Normality of Asset Returns” explores the concept of changing the “calendar time” of the asset return process to recover normality.

The idea that, “Calendar time might not be an appropriate measure of time in the financial markets”, has been explored in the past. Mandelbrot in 1963 published a paper that introduced a class of stable processes that could recover normality. The stable processes were nothing but Brownian motion processes subordinated by another stable process. In 1973, Clark published a paper in which he introduced a subordinator based on trading volume. However the presence of these two classic papers has not spawned further research. The investigation of specific subordinators that might recover asset return normality has been sparse in the finance literature

A document containing brief summary of the book

Takeaway :

This book is a beautiful book that describes the math behind queueing systems. One learns a ton of math tools from this book, that can be used to analyze any system that has a queueing structure within it. The author presents the material in a highly enthusiastic tone with superb clarity. Thoroughly enjoyed going through the book.

Queueing theory is an area where one can find superb applications of various stochastic processes. Here is an adapted visual from Kleinrock that gives a 10,000 ft. view of the various processes. The visual below, at once, explains the reason for Poisson process being so special. It is a process that is an intersection of Semi Markov process, Markov process, Random Walk process, Renewal process and Pure Birth process.

It is in the sweet spot, so to speak, of six popular stochastic processes.

Does anything in nature despair except man? An animal with a foot caught in a trap does not seem to despair. It is too busy trying to survive. It is all closed in, to a kind of still, intense waiting. Is this a key? Keep busy with survival. Imitate the trees. Learn to lose in order to recover, and remember that nothing stays the same for long, not even pain, psychic pain. Sit it out. Let it all pass. Let it go.

A brief summary of the book

 Takeaway :

The book is definitely overpriced for what it delivers. Given that 50% of this book explains R basics, the title is not at all appropriate. Even the quant stuff that is covered in the remaining 50% of the book is laughably inadequate.

“Music is a proud, temperamental mistress. Give her the time and attention she deserves, and she is yours. Slight her and there will come a day when you call and she will not answer. So I began sleeping less to give her the time she needed.”

― Patrick Rothfusss

Via Frequentist inference only seems easy

Bayesian methods are not necessarily more painful that frequentist procedures. The Bayesian estimation procedure requires more from the user (the priors) and has an expensive and complicated convolution step to use the data to relate the priors to the posteriors (unless you are lucky enough to have something like the theory of conjugate distributions to hide this step). The frequentist estimation procedure seems to be as simple as “copy over your empirical observation as your estimate.” That is unless you have significant hidden state, constraints or discreteness (not the same as having priors). When you actually have to justify the frequentist inference steps (versus just benefiting from them) you find you have to at least imaging submitting every possible inference you could make as a set of variables and picking a minimax solution optimizing expected square-error over the unknown quantities while staying in the linear flat of unbiased solutions (itself a complicated check).

The central theme of this book is — major governments & central banks around the world have been waging a war on “gold” to keep its price low. The reason being, they want everyone to believe in their fiat currency, so that they can print away the money to solve their temporary problems.

The author strongly believes that by 2020, no amount of “managing gold price” tactics will work. Dollar will lose its supremacy and gold prices will skyrocket to represent its true value.

Monaco has the highest life expectancy at birth, 89.57 years as per 2014 estimate. One might think that the country has exceptional living conditions that have given rise to such a stellar life expectancy level.

However the actual reason for such a statistic is explained via, “Inspection Paradox”, a famous paradox from Renewal theory.

Link : On the Persistence of Bad Luck (and Good)

One of my friends sent me a paper titled, “Path to Purchase: A Mutually Exciting Point Process Model for Online Advertising and Conversion”.

Not knowing most of the marketing jargon in the paper, I had to read it slowly. After going through the paper, I  have realized that it is a superb application of point processes to measuring effectiveness of online advertising. In finance, there are tons of papers published that use Hawke’s process for modeling. I was clueless about its application in other domains.

This book is written by Amit, a self-proclaimed witness to Mother Divine!. In his current incarnation, he calls himself Om Swami and runs a website – www.omswami.com. This book talks about his journey from a successful tech entrepreneur(again a self-claimed one) to a spiritual guru.

I found the book to be total crap. This guy decides to take a spiritual path in 2009, renounces everything and heads to Varanasi, then to Himalayas and ends up meditating in a cave for over 150 days. In the cave, he claims to have seen the divine. In his famished state, he might as well have been hallucinating.

I have enjoyed reading Nicholas Carr’s previous book titled “The Shallows”, that discusses at length various ways in which Internet impacts our brain. In this book, the author takes “automation” as the subject and explores its human consequences. In this blog post, I will summarize the main points of various chapters.

Alert for Operators

This chapter opens with a terse notice issued by Federal Aviation Administration on Jan 2013. This note was addressed to all the US. Airlines, commercial air carriers and said,

In the last few decades, enormous computational speed has become accessible to many. Modern day desktop has good enough memory and processing speed that enables a data analyst to compute probabilities and perform statistical inference by writing computer programs. In such a context, this book can serve as a starting point to anyone who wishes to explore the subject of computational probability. This book has 21 puzzles that can be solved via simulation.

“Life is really simple, but we insist on making it complicated.”

- Confucius

In the case of a queueing model, it is very likely that service time distributions in a real life situation, do not have an exponential tail. This means that all the analytic solutions derived in any standard textbook are no longer applicable. If the server following a generic distribution, the expression linking the distributions such as Waiting time distribution, First passage time distributions, etc. and service time distributions is in the “Laplace Transform space”. This means that Laplace Transform of let’s say waiting time distribution is given as a function of Laplace transform of Service time distribution. For M/G/1, the waiting time distribution is given by the following expression :