Big Data is the popular name for the field of data analytics, the process of sifting through massive amounts of information with large numbers of variables in split seconds to produce results that can be used make evidence-based decisions.
Think statistics blogger Nate Silver accurately predicting Obama's win in the 2012 presidential election, or IBM's Watson computer trouncing two human "Jeopardy" champs -- that's Big Data.
The bits of data generated by websites, social networks and smartphones apps are creating a "nuclear explosion" of information, as a recent Wired report called it. Combined with the power of relatively low-cost cloud-based software and digital storage, it's contributing to a global mountain of data that in 2013 is expected to reach an unprecedented 1.2 zettabytes, according to a report from Tableau Software. That's equal to 1 billion terabytes.
Companies and other organization are jumping into big data with abandon. It's being used by everyone from health insurance carriers to analyze bills for fraud to police departments to analyze past crime patterns to better predictwhere future thefts or robbers are most likely to occur.
The average Joe and Jane are benefiting because retailers and other companies are using big data to offer more personalized services. People have always put a high value on one-on-one interactions, and the experience that Big Data offers "simulates a parallel world online that's as personalized and trustworthy as offline," says Vikas Sabnani, chief data scientist with Glassdoor.com, the jobs and careers website. "Big data is the only way we can get there."
Big data, no price tag
Many big data-based services for shoppers, job seekers and other consumers are free.
Job hunters pay nothing to use Glassdoor to search through millions of job openings, see salary information for specific jobs and read reviews of what it's like to work at myriad companies.
Companies post the job openings. The rest of the information comes from reviews or data on salaries, bonuses and commissions that job hunters share with the site. Glassdoor compiles and analyzes all that data to provide job hunters with as many specifics as possible when they search on the site. If someone who's interested in software engineer jobs does a search on the term and then clicks on a job at Google, Glassdoor takes that as feedback that they want to work in the tech industry or at a large company like Google, and uses the information to show ratings and reviews of companies they might be interested in.
Glassdoor also lets job seekers link to their Facebook accounts to see if any of their friends or connections work at any of the companies. All of it is possible because Big Data doesn't take as much computing power or money as it would have only a short time ago, Sabnani says. "The cloud makes it much cheaper for us to process and scale as opposed to hosting it on our own servers."
Amazon is a long-time proponent of Big Data, which runs the recommendations engine on its online bookstore. Browse through the bookstore and you'll see recommendations pop up for novels or other books you might be interested in. The company analyzes stored data on books you've previously looked up on the site, and compares that with data from customers who've either looked up or bought the same books. Based on the information, the website shows you books that those customers have looked at or purchased that might appeal to you.
Not all big-data services are free. A recommendation service on Netflix is only available to people who pay the $7.99 monthly subscription. If they do, they can opt to use the online entertainment company's Taste Profile tool to rate films and TV shows with up to five stars. The company then compiles that data with billions of ratings from its other 36 million members to suggest other shows or films a subscriber might want to watch. The more shows or movies you watch and rate, the better the suggestions get. "It's the network effect," says Phil Simon, author of the just-published book, "Too Big to Ignore: The Business Case for Big Data."
Mind-boggling amounts of data
Companies such as Google, Twitter and Facebook are built on Big Data, Simon says. Google uses it in multiple information products, from its ubiquitous search engine to Google News Alerts, a news finding service that can be set up to monitor specific subjects and have updates sent to an email address on a real-time, daily or weekly basis.
Facebook uses Big Data to analyze what users "like" or share, and then uses that to sell ad space on a person's timeline or newsfeed. Simon, a long-time Rush devotee, likes the rock band's Facebook page and takes other actions that show he's a fan. If a music promoter was trying to fill seats at a Rush concert in Simon's hometown of Henderson, Nev., it wouldn't surprise him to see an ad for tickets on his Facebook page. "The amount of data Facebook has on people is mind boggling," he says.
Twitter’s Trends feature is also built on Big Data. When someone on Twitter writes a status update with a hashtag-labeled keyword signifying a topic the update is related to, and agrees to share their location, the company combines the information with other users' updates, hashtags and locations and shares what topics are trending in specific locations or throughout the service.