Sometime art imitates life and sometime life imitates art.

Back in 02011, when I first heard about the correlation between sentiment in social media tweets and the prices on the stock market I was intrigued. An academic paper was published with data about the sentiment of brands on social media on a day to day basis with a correlation of the price of shares trending positively or negatively with a delay.

At the time I was deep in natural language parsing and trying to build and improve some of our techniques to get sentiment analysis by correctly guessing the feelings behind a corpus of text. This is by no means an easy task. Some of the best software out there wasn’t doing much better than a 50-50 coin flip.

This reminded me of an old professor. He told a story about a conference where some new AI software was unveiled and the team proudly announced it could predict the weather accurately 40% of the time. Everyone was impressed! Then someone stood up and said they could predict the weather 60% of the time. He said he would just use their software and negate the results. The moral of the story is to be sure you’re accurate more than 50% of the time, otherwise the inverse is better.

With sentiment analysis it isn’t easy to extract a good estimate of positive or negativity from something as short as 140 characters. One of the simplest techniques is to eschew AI and simply ask people. It is possible to setup a mechanical turk task or other crowdsourcing service to pay people 5-10 cents to rate some text either positive or negative on some scale. As 3 people to review the same text, take an average which tends to be a pretty good indicator. For a quick result with a low-volume of text, this is economical and more accurate than any piece of software.

Using software to determine sentiment is hard, for the simple reason of sarcasm and negative operators. The simplest approach is called “bag of words” which adds +1 point for every positive work in the text and -1 for every negative word, then after looking at every word you have a simple score. Something with 10 positive and 10 negative words could come out at zero, just like text with no matching sentiment terms. This method isn’t great, but it is fast, cheap and easy. What it fails to recognise are terms like “not happy”, which is negative, but instead the term “happy” would give the text +1. There are other techniques to improve on these methods. For instance, looking at two or three word combinations.

What “bag of words” and many other techniques can’t handle is sarcasm. The sentence, “I love my bank, I’ve been waiting in line for two hours, what great service” might be perceived as positive. 2 hours is a long time, all the terms are positive, but any human can spot this isn’t a good thing.

After searching for a company, product or term on twitter for mentions along with some sentiment, software would rank and rate the company as trending in a positive or negative direction. When the researchers conducted the tests there was a lag of something like 18-24h before the stock market would react in a similar manner. If people were bad mouthing a company on social media, the rest of the world caught-up a few days later and the perceived value of the company changed accordingly.

I am fascinated with prediction markets, and social media trends like this are a good lead indicator which people exploit for their own gains. It isn’t exactly insider knowledge, but more like the wisdom of crowds. Which ironically is what the stock market is too. A crowd of people speculating on the future success of a company. Social media just exposed the zeitgeist a small fraction sooner than the market can react. Consequently, as time goes by and people learn about this predictive power, it will close the loop and no longer be predictive.

All this talk of predictions, social media and market manipulation got me thinking about an interesting thought experiment or short story I’ve been kicking around for a few years now.

DARPA is a big sponsor of projects that revolve around military intelligence, equipment and other aspects of national defence. They have also had requests for proposals around social media for various known and unknown usages. One of which was to allow for government officials to manage several social media accounts simultaneously to stir-up vigour into a frenzy, much like how social media aided in the Egyptian revolution of 02011. The ethics of using online tools to tips a crowd into a mob is dubious, but much like everything in technology, optimisations allow a single person to do what used to take a hundred. All this played into my idea for an interesting short-story.

The story is set some time in the near-future, a newly elected president decides to take the USA into a more of an isolationist policy. Seeing an offensive armed service as not something the country wanted, their budget was slashed. This is where DARPA would dust off all the tools they had built for social media management and execute their long-time con.

For years they have been running shell companies which they have floated publicly on various stock markets. Putting all their social media tools into over-drive, they would begin to talk about how great their shell companies are, which in turn triggers plenty of the social media monitoring tools to enter into the echo chamber and also begin to talk and take notice of these shell companies. All the hype would drive-up the price of the shares. The government would then sell-off all their shares as the price was rising. A typical pump and dump scheme, but on a massively organised scale.

Market manipulation would be how the armed services could continue to operate as if their budget had not been cut. All the previous years spent building tools to allow them to continue to thrive at their current state or even continue to grow without direct government monetary support.

This was my idea for an interesting short story. I thought I could introduce a few colourful characters and walk through some examples of the tools they have built and theoretical others. A futuristic boiler-room style movie but with the shadowy government behind the schemes instead of rouge traders.

Recently the Washington post published an article entitled, As Wall Street relies more on technology, social media can tilt the markets. It goes through how social media hype is directly affecting stock prices and automatic trades execute deals. My short story idea isn’t an idea any more, it is a reality already happening. I might get around to writing it, but it seems a bit passé now.