The Flash Boys

[y66]

I admire the ingenuity of the guys who figured out how to run this scam. What makes this a cool story for me is how "nice guy" Brad Katsuyama from Toronto and his pals figure out what the scammers are up to and create a new exchange that levels the playing field. This is like an American western in which the good guys are all foreigners.

 

From John Gapper's review on ft.com:

 

Even Katsuyama comes round to the HFT villains towards the end. “I hate them a lot less than before we started,” Lewis quotes him as saying. “This is not their fault. I think most of them have just rationalised that the market is creating the inefficiencies and they are just capitalising on them. Really, it’s brilliant what they have done.”

 

Indeed, as Lewis explains, much of their behaviour was perfectly in line with regulations – the Securities and Exchange Commission deliberately tried to weaken the monopoly power of large exchanges to create more competition. It may not have appreciated the scale of what it would unleash – 13 public stock exchanges and more than 40 private “dark pools”.

 

And financial trading is not the only ecosystem that is highly complex and aggressive. “I would ask the question, ‘On the savannah, are the hyenas and the vultures the bad guys?’ ” says one of Katsuyama’s more dispassionate colleagues. “We have a boom in carcasses on the savannah. So what? It’s not their fault. The opportunity is there.”

[hrothgar]

Do I misunderstand the rules" In (1,0) the 1 is a distance of 1 from half of 0 (the other player's number), the 0 is a distance of 1/2 from half of 1 (the other player's number), so the 0 takes everything. If we change 1 to 0, they split.

 

 

As I understand the rules, playing 0 you win if the other player plays anything except 0, and you tie if he also plays 0. You can't do better than that. What am I missing?

 

This is what Art said a while back and when I read it I thought oh yeah, that's obviously it. I don't see why it isn't.

 

I tried to use the (0, 1) to display the combination in which Player A chose "0" and player B chose "1".

 

With this said and done, this is complete bullocks. I am apparently not at all awake this morning.

[kenberg]

I tried to use the (0, 1) to display the combination in which Player A chose "0" and player B chose "1".

 

With this said and done, this is complete bullocks. I am apparently not at all awake this morning.

 

Believe me, it happens.I once went into a class and began by saying "Some of you may have thought I said such and such last time. You may even have it in your notes that I said such and such. However I could not possibly have said such and such, so you are all suffering from a mass hallucination."

 

I was really worrying that I had missed something (Cherdano caught me on a very careless error a while back).

 

Anyway, go with 0.

[kenberg]

My favorite problem along these lines was told to me many years ago by Jack Feldman. You die and ascent upwards. At the entrance to heaven you are informed that contrary to what you may have been told on Earth, entrance into heaven has nothing to do with the life you have led. Rather it goes like this. God has a continuous probability distribution on the real numbers and, using it, he chooses two real numbers independently. "Continuous distribution" (as many of you know) means that the probability of the two numbers being the same is 0. The Guardian at the gate informs you of the first number that God picked. You are to guess, with your soul in the balance, whether the second number is larger or smaller than the first. You are not allowed to take Eternity thinking up your response. Guess right, you get a harp. Guess wrong and, well, you won't get cold.

 

You know nothing of the distribution other than it's continuity, but the first and the second number are randomly selected independently using the same distribution.

[ArtK78]

My favorite problem along these lines was told to me many years ago by Jack Feldman. You die and ascent upwards. At the entrance to heaven you are informed that contrary to what you may have been told on Earth, entrance into heaven has nothing to do with the life you have led. Rather it goes like this. God has a continuous probability distribution on the real numbers and, using it, he chooses two real numbers independently. "Continuous distribution" (as many of you know) means that the probability of the two numbers being the same is 0. The Guardian at the gate informs you of the first number that God picked. You are to guess, with your soul in the balance, whether the second number is larger or smaller than the first. You are not allowed to take Eternity thinking up your response. Guess right, you get a harp. Guess wrong and, well, you won't get cold.

 

You know nothing of the distribution other than it's continuity, but the first and the second number are randomly selected independently using the same distribution.

I suppose you guess lower if the first number is greater than zero and higher if the first number is less than zero.

 

It would seem that there are an infinite number of real numbers higher than the first number and an infinite number of real numbers lower than the first number, so it is a 50-50 guess. After all, infinite is infinite. But, unless I am mistaken (and I am pretty sure that I am not), the infinite number of real numbers higher than a negative number is greater than the infinite number of real numbers lower than a negative number, and vice versa for positive numbers. In other words, there are larger infinities and smaller infinities. Might as well go with the larger infinity.

 

I don't know if this rises to the level of a proof, but it would seem that if x is positive, then the number of real numbers greater than x is equal to the number of real numbers that are lower than -x. Since there are more than zero real numbers between x and -x, there are more real numbers lower than x than there are that are greater than x.

 

And, if x is negative, the rationale is the converse.

 

With my luck, the first number will be zero. Now you can flip a coin. :)

[kenberg]

I suppose you guess lower if the first number is greater than zero and higher if the first number is less than zero.

 

It would seem that there are an infinite number of real numbers higher than the first number and an infinite number of real numbers lower than the first number, so it is a 50-50 guess. After all, infinite is infinite. But, unless I am mistaken (and I am pretty sure that I am not), the infinite number of real numbers higher than a negative number is greater than the infinite number of real numbers lower than a negative number, and vice versa for positive numbers. In other words, there are larger infinities and smaller infinities. Might as well go with the larger infinity.

 

I don't know if this rises to the level of a proof, but it would seem that if x is positive, then the number of real numbers greater than x is equal to the number of real numbers that are lower than -x. Since there are more than zero real numbers between x and -x, there are more real numbers lower than x than there are that are greater than x.

 

And, if x is negative, the rationale is the converse.

 

With my luck, the first number will be zero. Now you can flip a coin. :)

 

 

Go with the flip a coin strategy, only do it better.

 

I wandered the neighborhood while trying to figure out how to show that no such strategy could exist. Once I started thinking right, it was easy.

 

Using 0 will often help, but God can be foxy and if he wishes to keep you out and if he knows your strategy (what's omniscience goo for if not here?) he can reduce your chances to 1/2.

 

The trouble with deciding based on the seen number being positive is that God's distribution may be supported on the positive numbers so that both of his numbers are certain to be positive. For example, the time you have to wait for your next phone call is a random number, sometimes modeled as an exponentially distributed random variable, but at any rate it won't be a negative number.

 

That's where the part about you knowing nothing about the distribution God has chosen comes into play.

 

Fight fire with fire. Don't use 0, use a number selected randomly from the standard normal distribution, or any distribution that, for any interval (a,b) with a<b, gives a positive probability of the chosen number lying in (a,b).

 

Now you see God's first number, and you see your own randomly chosen number. You make your guess that your number, and the unseen number, lie on the same side of God's first number.

 

Now suppose God's two numbers are a and b in some order and your number is c. If c lies above both a and b you will be right in your guess half the time. Similarly, you will be right half the time if c lies beneath both a and b. So that's 50-50 so far. But if c falls between a and b, you always guess right. If God shows you the smaller, c is larger than it so you guess correctly that the unseen number is larger. Similarly if God's first number is the larger.

 

So no mater what distribution God is using, your probability of success exceeds 1/2 by half the probability of c lying between a and b. Whatever this is, it isn't zero.

 

 

I saw this as a puzzle, but I understand this technique is actually used in some situations where not much is actually known. Any edge is an edge, even if you do not know how large an edge it is.

 

 

Maybe this is clear. Or maybe not. I think this falls under the heading of random algorithms so there no doubt is some web stuff on it.

[FM75]

I tried to use the (0, 1) to display the combination in which Player A chose "0" and player B chose "1".

 

With this said and done, this is complete bullocks. I am apparently not at all awake this morning.

 

PEBCAK - Problem Exists Between Chair And Keyboard. It is a common situation, especially experienced (and ultimately recognized) by programmers.

The clue here might have been 0, 1 instead of 0, 100.

[FM75]

I suppose you guess lower if the first number is greater than zero and higher if the first number is less than zero.

 

It would seem that there are an infinite number of real numbers higher than the first number and an infinite number of real numbers lower than the first number, so it is a 50-50 guess. After all, infinite is infinite. But, unless I am mistaken (and I am pretty sure that I am not), the infinite number of real numbers higher than a negative number is greater than the infinite number of real numbers lower than a negative number, and vice versa for positive numbers. In other words, there are larger infinities and smaller infinities. Might as well go with the larger infinity.

 

I don't know if this rises to the level of a proof, but it would seem that if x is positive, then the number of real numbers greater than x is equal to the number of real numbers that are lower than -x. Since there are more than zero real numbers between x and -x, there are more real numbers lower than x than there are that are greater than x.

 

And, if x is negative, the rationale is the converse.

 

With my luck, the first number will be zero. Now you can flip a coin. :)

 

The questions here are:

1) Does God want you to win?

2) Does God know your choice (is really omniscient even about the future)? It does not matter, if he does.

3) Are you properly informed about the conditions of the contest - (God's agent (perhaps) told you about the new rules...)

 

As we go down an interesting Game Theory Tangent - not worrying about the original topic...

[kenberg]

Yes, and I apologize for any thread wandering. Sometimes I can resist.

 

So on the original issue, where I should probably shut up because I don't understand diddly about it, I have mothing in principle against using technology to make a buck in the market. But I sometimes have this "uh oh" feeling when it all seems to be heading toward technical gee-whizzery and away from any concern withwhat the company actually does. It's day trading on steroids, if I understand it right.

 

I think it was Michael Rosenberg who, commenting on whether bidding or play was the most important part of bridge, said that he didn't know but that he wanted the play to be the most important. Maybe that's my thinking here. I want business fundamentals to rule. You can't always get what you want.

[barmar]

Ever since we started fashioning sharp rocks into spear tips, it has generally been the case that people and organizations that make more effective use of technology were more successful. In that respect, it seems wrong to deny the HFTers the right to use computers to take advantage of opportunities in the stock market.

 

On the other hand, there are a number of areas in modern life where we feel the need to rein in use of technology. The power of computers has gotten too good in some cases. Like the privacy violations that are possible with analysis of "big data".

 

It's a fine line, so it's difficult to know whether or not we should do something just because we can.

[hrothgar]

Ever since we started fashioning sharp rocks into spear tips, it has generally been the case that people and organizations that make more effective use of technology were more successful. In that respect, it seems wrong to deny the HFTers the right to use computers to take advantage of opportunities in the stock market.

 

Its so rare that one see's people seriously arguing "Might makes right"

Well done Barry!

 

(Good thing you aren't at Akamai any more, or I'd feel obliged to take your lunch money...)

[barmar]

Its so rare that one see's people seriously arguing "Might makes right"

Well, in this context I was thinking more along the lines of free markets, which also reward better technology. It's the age-old argument about laissez-faire capitalism.

[Mbodell]

Well, in this context I was thinking more along the lines of free markets, which also reward better technology. It's the age-old argument about laissez-faire capitalism.

 

Free markets don't really make sense in and of themselves completely abstracted from gov't. When you are talking about the specific rules of the market a vague notion of "free markets" isn't really enough. Adam Smith wealth of nations and all that still needs a government to enforce contracts and provide justice system, grant patents and copy rights, provide public goods such as infrastructure, provide national defense and regulate banking (emphasis added). Rightly or wrongly some sort of trading is also illegal (insider trading). There are also rules around price fixing and many, many others.

 

Unless of course by laissez-faire you mean something like "enforce all the rules on those other people that I like, need, and regularly benefit from - but have me be above the law/don't enforce any rules on me!".

[barmar]

I guess that's what I mean by having to figure out where to draw the line. Laws against insider trading define it as unfair that you can gain an advantage by having relationships that are not available to other investors -- there's nothing they can do to level the playing field with you. But just being "smarter" or developing/buying better technology is generally considered OK -- there's nothing inherently preventing a competitor from achieving the same thing; if they don't, they're just not trying hard enough, and they deserve to lose.

 

Sports are frequently running into this type of issue with regard to equipment. If someone comes up with a better tennis racket or golf ball, the opponents can also buy them, so the playing field is level. On the other hand, sometimes we let ideology or morality trump this philosophy: performance-enhancing drugs could be treated like better equipment, because there's nothing preventing all the competitors from taking them, but authorities have deemed that this is against the spirit of athletic competitions (probably it's tainted by association with other forms of drug abuse).

[barmar]

[This probably is more appropriate for some previous thread about the financial industry, but rather than dredge one up I'm posting it here.]

 

From the "plus ça change" department:

 

Scientific American has a column every month titled "50, 100, & 150 Years Ago", where they print a paragraph from a couple of notable articles from those old issues. This month they printed the following from the April 1864 edition:

Vultures Everywhere

"One of the most alarming signs of the times in which we live is the extraordinary and villainous speculations now rife in Wall street, in the shape of gold and other mining operations. Bogus companies are forming every day, whose foundations are as the ‘baseless fabric of a vision.’ We warn the people to beware of these swindlers—they should shun them as they would the gambling-hells of the city. These vile schemes are incubated and hatched in the region of the Stock Exchange, and are designed to entrap the innocent and unwary. Every one of them ought to be indicted by the Grand Jury, and the guilty swindlers sent to Sing Sing."

http://www.scientificamerican.com/article/50-years-ago-dyson-on-detecting-alien-life/

[FM75]

Free markets don't really make sense in and of themselves completely abstracted from gov't. When you are talking about the specific rules of the market a vague notion of "free markets" isn't really enough. Adam Smith wealth of nations and all that still needs a government to enforce contracts and provide justice system, grant patents and copy rights, provide public goods such as infrastructure, provide national defense and regulate banking (emphasis added). Rightly or wrongly some sort of trading is also illegal (insider trading). There are also rules around price fixing and many, many others.

 

Unless of course by laissez-faire you mean something like "enforce all the rules on those other people that I like, need, and regularly benefit from - but have me be above the law/don't enforce any rules on me!".

 

You have pretty much missed my point here - which is that stock trading - or selling merchandise online, auctioning off items on e-bay, etc. is a game. It is a multi-player game in which all participants know the rules, know what the penalties are for breaking them, etc. Whether the rules are set by a government, or by a private organization running the game is pretty much irrelevant.

 

A further point is that even if the optimal strategy is difficult to determine - as it may have been for some in my example - once the strategy is known, you don't have to understand why it works to use it.

 

Regardless of the mechanism of the game - it can be changed, by simply changing the rules - it must have players to be meaningful (successful), and those players must find it fair - or they won't play.

 

Think of Nash Equilibrium like a system of predators and prey - where the prey survive on grass. If the prey are excessively available, and predators are successful at eating and mating, they could reach a point where the prey are all eaten and the predators die. OTOH, if the predators are too rare, then the prey starves from lack of grass.

 

If you attempt to oversimplify the problem by looking at only the strategies of one side, you lose. If you fail to recognize that they are looking at your strategies, you lose.