This book is written by Shruti Jauhari, a noted Hindustani Classical Singer and a disciple of K.J.Yesudas. The book is an accessible introduction to the various elements of Hindustani Classical Music. To read a language, you need to know the alphabets and the grammar associated with it. In the same way, to listen to Hindustani music, you need to have a basic knowledge of Svar,Raag and Taal   One often comes across terms like bandish, gharana, alap, vilambit , taal, etc in the context of Hindustani Music. Unless one has some understanding of these terms, it is difficult to be a discerning listener.

It has been a long time since I have read any fiction book. So, thought of reading one, on this weekend to take a break from stats, programming and the usual routine. It took me about 9 hours to read the entire book and I must say that there was not a single instance during those 9 hours that I felt like taking a break. So, the book has a smooth flow of prose with just enough characters that you don’t lose the story or get bored anywhere. The plot has about 15 characters , including one played by the author himself, Somerset Maugham but not all 15 characters get the same footage(obviously) .The protagonist of the story is Laurence Larry Darrell who goes on a spiritual quest and the book tries to weave a story around that quest. The other 13 characters come and go in with varying periodicity through out the book. Besides Larry, the main characters in the story are Isabel who loves Larry but ends up marrying Gray Maturin, Elliott Templeton, Isabel’s uncle whose sole purpose in life is to socialize and be at parties, Sophie MacDonald, a poetess turned whore turned dope addict, whom Larry almost ends up getting married, but thanks to a Isabel’s devious plan, never does so.

I have started using the development version of IPython from github and the experience has been extremely pleasant. Writing code and doing interactive data analysis has become so much more easier in iPython Notebook.

Fernando Perez at Pycon 2012 says that,  very soon, there will Sphinx support, debugging support and other parallel computing support that will be built in to IPython. Will eagerly await the production version of IPython notebook. However development in whatever form it is in, is good enough to experiment and do quick data analysis. The fact that matplotlib graphs appear inline is something that I found really useful.

Austin Kleon’s book is essentially a blog post turned in to a book .The basic theme of the book is that in art as in other forms, you have to steal. The word steal has a negative connotation and hence the book differentiates between two kinds of thefts:

Basically theft in this sense means that you are free from the burden of trying to be completely original. Most of the book is common sense stuff but how often do we really follow commonsense ? 

There are a ton of relationships between various distributions. One typically understands them, remembers them based on the kind of stats work that one does. If one does a lot of Bayesian stuff, one tends to remember the conjugate priors and related distributions. If you are doing survival modeling, you tend to focus on specific distributions like weibull etc. If you are in to OR work, Erlang, gamma, beta distribution parameters are at your finger tips. Irrespective of the type of analysis that one does, it is always good to have a decent overview of various random variables and the connections between them. The connections that one must understand should typically encompass :

Via Lemire

Over 20 years ago, back when I was in high school, I went on a sailboat trip. I was so impressed that I decided to own a sailboat one day. I realized that a sailboat was expensive, and I guess I thought that owning a boat would not only be cool, it would be a symbol of my success.

How did it go? Today, not only do I now own two sailboats, I’m building a third one. Of course, they are radio-controlled boats, about 4 feet tall and 2 feet long. I find that I really like to design and build these little boats.

Pandas is a Python library developed by Wes McKinney. The USP of Pandas is that it provides “Data Frame” environment in Python.  Dataframe is one of the commonly used data structures in R. The first thing that pains you when you start coding in Python , after having worked in R is that – “there is no readily available data frame object” . Even though there is NumPy ndarray object, it is not as flexible and extensible as the dataframe in R.  Pandas library addresses this problem and provides DataFrame and a ton of associated functionality that can be used for data munging, data cleaning and interactive data exploration. If you look at random data cleaning code written using Pandas and R, they will look very similar. However pandas is like R’s dataframe on steroids. Pandas also has some preliminary graphing capabilities using matplotlib.I think as the library matures, it will be a default module in any data analyst’s toolkit.

Via Mint :

Announcing the long-awaited guidelines for so-called algorithmic trading, or “algo” as it’s popularly known, India’s capital market regulator on Friday insisted on a stringent and advanced risk management system to provide support for such trades as their speed and volume entail a higher risk. Automated execution logic enables the rapid execution of a large number of trade orders. Stock exchanges have been given three months to complete the approval process for brokers currently executing orders through algos.

For any number crunching work, be it making a report or developing a model or doing data diagnostics, visualization of the data is imperative. Be it univariate or multivariate data, visuals help us to look beyond the summary statistics or test statistics. For someone working in finance, there is an entire discipline of `Technical Trading’ where buy , sell , stop-loss decisions are made based on visuals. Whether one believes it or not is a different question altogether. Keeping Technical analysis side, there is an obvious need to look at data, be it histograms, density plots, contour plots, barplots, boxplots, etc. Tools that churn out these graphics are compulsory in any data analyst’s toolbox. My toolbox contains ggplot2, lattice and base-R. I had started using ggplot2 package, 4 years ago, and since then I have been using it regularly in my work. Since the output is usually publication ready, one of the real life situations where I had used ggplot2 visuals was in an annual newsletter to investors that reported their portfolio performance. I don’t think anybody cared about what the visuals were saying as long as the portfolio was making money. But using ggplot2 definitely lent a professional look to the newsletter.

Talent borrows, genius steals

- Oscar Wilde

Programming in Matlab / R exposes one to vectorized way of thinking. One doesn’t usually write loops often and one tends to think in terms of vectors,array, matrices etc. R for instance is designed to facilitate vectorized input and output. Almost all the functions in base R support vectorization. Most of the functions in the packages on CRAN are equipped to take vectorized input. In fact the program design itself makes vectorizing easy. Recycling rule in R for instance , sometimes makes your function automatically handle vectorized input, even though you never meant to handle such an input. For a R newbie , the fact that his/her function does much more than expected is a happy side effect. However once you have logged in decent number of hours in R, you realize that it is your responsibility that whatever code you write, it handles vectorized input . Hence these days, one of the first unit tests that I write is to check whether my code breaks down for vectorized input.

In the Apr 2012 issue of “Advanced Trading ” magazine , I found these points . quotes worth noting down :

• Exchanges have announced plans to roll out measures aimed squarely at eradicating the technique of
  quote-stuffing, a common strategy in which high-frequency traders rapidly fire off, and then cancel, millions of orders as a means to probe the direction in which the market is heading.

• Under Dodd-Frank, hedge funds and other private fund advisers that manage $150 million in assets or more will have to fill out Form PF, the brainchild of regulators who are looking to prevent seismic events like the demise of LTCM

In the Mar 2012 issue of “Traders” magazine , I found these points worth noting down :

• The FIX Protocol Ltd. Americas buy side working group has been trying to amend FIX Protocol to make it easier to track orders. It is considering the adoption of new FIX tags later this year

• Interview with Keith Ross, the early partner at GETCO

• HFT guys are incredibly good risk managers

• The good HFT firms very rarely have losing models or they lose at best $50 to $100 a trade, and these firms very rarely have losing trading days where they don’t make any money.

This book is cited as the classic reference for  Python programmers. Instead of diving in to Python as the title suggests, I did some ground work before going through this book. I went over Learning Python the Hard Way, Think Python, Python Visual Quick Start guide and understood Python 101 . Those three books gave me some confidence to go over this book that is supposedly for experienced programmers.

I had used LaTeX for a few documents many years ago, but the interfaces were so daunting that I never felt like writing a TeX document, unless it was a thesis or a assignment submission or some compulsory requirement. Needless to say MS Word happened to my ally , though a painful ally.  Then two things happened in my life simultaneously. One, a chance encounter with a TeX enthusiast , who suggested that I make a conscious decision to avoid MS Word for at least a month and try to use TeX irrespective of the pain that entails a beginner. If my experience was not good, I could always revert to MS Word.

This book gives a quick intro to Python and one does not need any prior programming knowledge to go over the contents. I quick read the book in an hour’s time and it served as a syntax F5er.

• Some of the GUI libraries mentioned that I need to explore are PyQT, PyGTK, wxPython, TkInter

• Need to check out Charming Python link mentioned in the book

• There is no char datatype in Python. Never realized this actually.

Via ET

Securities and Exchange Board of India’s chief UK Sinha reiterated on Friday that the regulator is considering imposing speed limits on high-frequency trading of listed Indian shares.

“Earlier, it was 200 micro second speed, then 20 micro second and then 8 micro seconds, and there are still demands to reduce it even further. At some stage, this has to stop. What we need to look at is whether it is serving any public good,” Sinha said at the International Bar Association Conference.

Thanks to Ravi, came to know about an old wired article on Netflix prize, where Gavin Potter used fundas from Behavioral economics to crack the problem

A deeper part of Potter’s strategy is based on the work of Amos Tversky and Nobel Prize winner Daniel Kahneman, pioneers of the science now called behavioral economics. This new field incorporates into traditional economics those features of human life that are lost when you think of a person as a rational machine, or as a list of numbers representing cinematic taste.

Dexy seems to offer an amazing solution to documentation that is Platform Independent. It’s a tool created by a Ana Nelson, an open source software developer,  who happens to have a PhD in economics.
Obviously a better output that star gazing and making predictions in economics , that anyway go wrong .

Dexy’s USP in Ana’s words :

There are many excellent tools for specific types of documentation, in particular many object-oriented languages have built-in tools for writing API reference documentation (e.g. JavaDocs, RDoc, pydoc), however these tools are very specialized and don’t tend to work well for other types of documentation, such as tutorials, nor do they deal well with the fact that projects these days almost always involve more than one programming language (it would be hard to find a project that didn’t include, at minimum, a few bash scripts or some JavaScript). Dexy lets you continue to use specialized tools for some parts of your documentation, and also gives you the flexibility to write other types of documentation in different formats (tutorials, user guides, lecture notes, presentation slides, posters, even blog posts).

In the Mar 2012 issue of “Advanced Trading” magazine , I found these points worth noting down :

• Gone are the days when you could take a prime brokerage account from Goldman/big Prime brokerage  and start your hedge fund. In Today’s world, launching a hedge fund is becoming harder than ever. You have to have the tech platform that talks to multiple primes , an ops team , at least two counter parties. Further compounding matters for new hedge funds are much more stringent compliance requirements
  than a decade ago. Under the Dodd-Frank Act, firms with at least $150 million in private assets under management will have to periodically file Form PF documents with the Securities and Exchange Commission. The Financial Stability Oversight Council will use those filings to gauge how much risk is flowing

I like books that explain things visually and this book falls in that category. I am reading this book after reading and understanding the basics of Python from “Think Python ” and “Learn Python the Hard Way ”. This book serves a nice visual recap of Python 101.  I have listed down some of the points in various chapters mainly to ruminate over the learning’s from the previous two books.

Prof. Hadley Wickham, the creator of ggplot2 and other useful packages like plyr, reshape etc. has one strong advice to R programmers – “Read other’s code”. This comes from a person who has developed 30 packages till date. We all have an immense urge to program, code up something, view the results, tweak our code to make it work etc. However pausing to read somebody else’s source code requires a certain amount of hard work, willingness to learn from others . In R particularly, where all the functions are documented really well, one hardly NEEDS to go in to the code. But that’s exactly what Hadley Wickham recommends.

If there is a lot of data parsing and cleaning that needs to done before modeling, I tend to follow one of the three paths :

• Path 1: Use Python to clean the data, export the data structure in to a file/database. Leave Python environment and move in to R to do the modeling.

• Path 2 :  Use Python to clean the data, Stay in Python environment and invoke R to do the modeling. Rpy is the go to module in this context.

Via TP:

What is Big Data? A meme and a marketing term, for sure, but also shorthand for advancing trends in technology that open the door to a new approach to understanding the world and making decisions. There is a lot more data, all the time, growing at 50 percent a year, or more than doubling every two years, estimates IDC, a technology research firm. It’s not just more streams of data, but entirely new ones. For example, there are now countless digital sensors worldwide in industrial equipment, automobiles, electrical meters and shipping crates. They can measure and communicate location, movement, vibration, temperature, humidity, even chemical changes in the air.

A popular ghazal based on Raag Yaman Kalyan  rendered by Rohini Ravada , Shankar Tucker

The connection between The Lady Tasting Tea and Statistics :

In the Feb 2012 issue of “Traders” magazine , I found these points worth noting down :

• A 300 page document, Volcker rule, signed into law in July 2010, prohibits U.S. banks from engaging in proprietary trading as of July 2014. This rule is similar to “asking you to switch off mobile phones” in the flight. The chance of electronic signal messing up with aircraft is 1 in million. For most of the passengers this does not make sense. But we still have been following this switch off iPods/electronic devices for the past 20 years. Volcker rule banning prop trading is supposedly a protection against trades that can pull down the entire system. Prop trading accounts for 11% of trading volume. So, who will fill in as liquidity providers ?

Alan Jacobs, the author of this book is an English Professor at Wheaton College, Illinois. Given his position as a professor, his students and other people often ask him, “What are the 10 best books on literature that every educated person must read", “Dear Prof, Can you suggest some books to read this summer?”, This book is written to answer all such questions. So one might think this book is basically a recommendation type / instructional / didactic guide to reading. Far from it, this 150 page long essay on reading at Whim, with no fixed pattern, with only one objective in mind, “Pleasure”.

Via BayesianBiologist   : An excellent explanation of the fallacy that most people have about, P-value

What we really want is the probability of hypotheses given our data (written as P(H | D) ), which we can obtain by applying Bayes rule.

What we get from a p-value is the probability of observing something as extreme or more than our data, under the null hypothesis ( written as P(x>=D | Ho) ).  Isn’t that awkward? No wonder it is so commonly misrepresented.

In the Feb 2012 issue of “Advanced Trading” magazine , I found these points worth noting down :

• 2011 was an extremely bad year for hedge funds, with Asia seeing the maximum number of fund closures. Hedge fund managers have quoted random volatility as a major reason for leaving business.

• The top performer in 2012 will be Commodity managers who have inside information about an underlying commodity — it is legal for them to utilize that, as opposed to equities, in which insider information is illegal.The managers who have fabulous networks and a really good understanding of supply and demand chains within the commodity space actually have a real advantage — more so than their equity counterparts.

This book contains most of the productivity hacks that one comes across in various articles/blogs/books.  In one sense, this book is a laundry list of hacks that one can try out to increase productivity. A big font size for the text and rich images scattered through out the book, makes it a coffee table book.

Some of the hacks that I found interesting are,

• On a daily basis, Try to use pen and paper at least a few minutes to work something out , be it a math problem or a back of the envelope calculation of something or simply draw images that capture whatever you are working on. Writing makes one’s relation to work intensely personal and more so using a pen and paper.

_“Unless one is happy, one cannot bestow happiness on others.
Happiness is born of Peace and can reign only when there is no disturbance.
__Disturbance is due to thoughts, which arise in the mind.
_When the (thinking) mind is absent there will be perfect Peace.”

-– Ramana Maharshi

Via Advanced Trading :

As regulators in the United States and Europe weigh the merits of new regulations to govern high-frequency trading, emerging markets have been methodically paving the way for the practice to expand within their borders.

In an interview with the BBC, Progress Software’s chief technology officer John Bates says said the practice of rapid-fire trading is quickly expanding in the so-called BRIC nations - Brazil, Russia, India and China.

http://video.ted.com/assets/player/swf/EmbedPlayer.swf

An old NY Times article on the surprising ubiquity of Zipf’s law :

Math and the City

One of the pleasures of looking at the world through mathematical eyes is that you can see certain patterns that would otherwise be hidden. This week’s column is about one such pattern. It’s a beautiful law of collective organization that links urban studies to zoology. It reveals Manhattan and a mouse to be variations on a single structural theme.

One tight slap on Times of India.

The preface to this book gives a reason behind the title for this book,

This year we learnt that there are many thousands of children across Britain who cannot read competently, that there are thousands who leave primary school unable to put together basic sentences. One in three teenagers reads only two books a year, or fewer, and one in six children rarely reads books outside of the classroom. Many parents do not read stories to their children, and many homes do not have books in them. Stories and poems, for these thousands of children, are not a source of enchantment or excitement. Books are are associated with school, or worse – they are associated with acute feelings of shame and frustration.

“Life is too short for a bad book or the right book at the wrong time.”

-– Tim Parks

I’m astounded by people who want to ‘know’ the universe when it’s hard enough to find your way around Chinatown.

-- Woody Allen

An old set of videos that I found extremely interesting.

Lex Van Dam, a hedge fund manager in London, picks 8 rookies and gives them a million dollars to manage for two months. This two hour long set of videos brings out so many human dynamics about trading that this can be a fantastic case study in any B school.  Will the 8 rookies survive 2 month period?Will they manage to make money ? Will they work as a team ? and many more questions get answered on the way.

In the introduction, the author tries to define HFT using the voices of leading HFT players. What are the characteristics of HFT? There is no agreement on a common definition of HFT, but there are some elements that one can at least attribute to HFT. They are

• Low latency trading.

• High turnover strategy.

• Trader goes home `flat’ with no open position.

The goal of any HFT firm broadly is to have a set of uncorrrelated trading strategies that has statistically more winners than losers for all the trading positions of the day. The introduction mentions the fundamental market driven factors that gave rise to HFT:

Thanks to my Sitar Sir, I came to know about a concert by Ustad Rais Khan this weekend.  Khan Sahab is from Mewati Gharana ( Indore) and plays in “Beenkar Baj Gayaki Ang”.

Before the concert began , he told the audience that he was running high fever and was having a severe body ache. He apologized in advance for any mistakes he might commit in the concert. He is 72 years old!. How many 72 year olds are there in this world who can actually perform in front of an audience ? I guess there is something in music that keeps a mind fresh and active. Despite his ill health, once Khan Sahab started playing , it looked like he entered another world. It was a delight to watch him play. Khan sahab performed for 2 hours in front of a jam packed auditorium. His son Farhan accompanied him on stage and was providing the speed component whereas Khan Sahab played meend, gammak , zamzama and other variations.

In the Jan 2012 issue of “Traders” magazine , I found these points worth noting down :

• In 2011, Knight Capital earned $172M from market making operations out of its entire on $517M

• According to a Buy Side survey, the top 5 providers of algo trading systems are Credit Suisse, Goldman, ITG, Bank of America and Merrill Lynch and UBS

• Aquaequities is a firm that focuses on proving block liquidity by aggregating from various mid tier specialized brokers. Very niche solution provider and making money.

The book starts off with a discussing ‘What’ and ‘Who’ aspects of High Frequency trading systems.

What is High Frequency Trading :

With the advent of electronic trading , the human market maker has been replaced by electronic market maker. The back end is a programmed strategy that does the market making. HFT’s DNA comprises two elements.First element is the trading strategy that is typically liquidity provision or intra day arb extraction. Second element of its DNA is the holding time that is usually a fraction of a second.

Via Number Sleuth :

All the gold in the world can be fit in to 3.42 Olympic Swimming pools.