A sunny Friday afternoon in the heart of Madrid’s Salamanca district. As you are walking down Maria de Molina Street, you feel hungry. Instinctively, you tweet “I’m starving!” and, instantly, you receive a €5 voucher for a veggie sandwich in a friendly place, within a five minute walk.
Behind the scenes your tweet was captured by sophisticated algorithms. Then, by integrating your credit transactions, social media activity, and your location (given away by your phone’s GPS) you were identified as a 29-year old vegetarian, with a craving for healthy food, who appreciates a friendly atmosphere and would pay up to €5 for a snack. After scanning the nearby cafes, the algorithm found the perfect match for you.
Our digital life
Data that is too big or too fast to process in typical systems is called Big Data. With nearly 6 billion people, i.e. 87% of the world population, using mobile phones and 2 billion using the Internet, anything from transactions, through social media, to search queries or even medical records is being tracked. While living our life, each one of us is leaving behind his/her own digital trail.
Furthermore, millions of networked street sensors, smart phones, or industrial equipment that measures and communicates location/ movement/ temperature etc., generate unprecedented amounts of data and rapidy fomulating our digital universe.
Among the most stunning data visualzation maps (above) that illustrate the global use of social media, Eric Fischer compared Flickr and Twitter usage. The color white indicates where people used both, blue is Twitter, and orange is Flickr.
How big is Big?
1.8 zettabytes of data, equivalent to 200 billion HD movies (that would take 47 million years to watch), was generated in 2011 and this figure is more than doubling every two years, IDC reports. The Economist reports that this industry is worth more than $100 billion and is growing at almost 10% a year.
Big Data is a disruptive trend
Companies are making every effort to build a customer-centric business. If only they could collect their customer data, mine it and transform it into actionable insights they could improve their bottom line. This ability will soon become vital for their growth and will drive their competitive advantage.
Companies will use Big Data for informed decision making, predictive analytics, sentiment analysis, or micro-segmentation.
A pioneer in that area is Wal-Mart’s customer tracking system that captures a total of 24 terabytes of transaction information from each of its 2,900 retail outlets, and analyzes sales, pricing, demographics and even weather info to customize the most efficient product selections and to determine the timing for discounts at particular stores.
Amazon exploits Big Data analytics to feed its recommendation engine (“you may also like”) and LinkedIn for its “People you may know” feature.
Another use of Big Data is sentiment analysis, i.e. detecting consumer attitudes towards brands, companies, or individuals. Start-ups such as Datasift or Gnip already analyze Google search queries, Facebook posts and Twitter messages to measure the public sentiment in real-time.
Global Pulse is a UN organization that analyzes social networks and text messages in the Third World countries to predict job losses or disease outbreaks in certain regions. Big data analysis, thus, serves them as a digital warning that signals where to send assistance programs.
Governments may use Big Data to combine different databases and improve public service. A Mckinsey report estimates that the EU public sector could save up to €300 billion in administrative costs by using big data.
Government agencies, such the NYPD, analyze historical arrest patterns and combine them with variables such as paydays, sporting events, weather and holidays in order to predict the places where a crime could occur and deploy in advance.
The Challenges of Big Data
There are technical challenges, privacy and security concerns, and a lack of talent in the Big Data arena.
Some of the technical challenges are related to capturing and analyzing different types of data (structured and unstructured), integrating different databases together and ensuring that out of the vast amount of data only the relevant one will be transferred to the decision makers so they can extract actionable insights from it.
Governments will need to create policies related to privacy, security and intellectual property as not many people are happy with the idea that their every move is digitized and analyzed, while companies want to make sure their data collecting is legal and not compromising them.
Finally, a lack of talent is another challenge. According to McKinsey, in the next 5 years in the US there will be a shortage of 140,000 to 190,000 people with analytical skills and 1.5 million managers and analysts skilled enough to make decisions based on the Big Data input.
Tell me what you think about Big Data and how will it affect our lives?
Use the comments below!