Monday, 15 Apr 2013 | 12:14 PM ET
By: Constance Gustke, Special to CNBC.com
Three years ago, Macy's was still relying on Excel spreadsheets to analyze reams of customer data. Enter Karem Tomak, the vice president of analytics at Macys.com, whose task was to bring the 155-year-old retailer and Macys.com into the age of big data.
To better understand customer-buying behavior and sharpen its promotions, Tomak set about installing analytics software. Macys.com now analyzes tens of millions of terabytes of information every day, including social media, store transactions and even Twitter feeds.
The results are compelling. Big data analytics has helped Macy's boost store sales by 10 percent, according to Tomak. "Data is now at our fingertips," he said.
For retailers like Macys, the big data revolution is seen as a key competitive advantage that can bolster razor-thin margins, streamline operations and move more goods off shelves—despite the pitfalls associated with a technology approach that lacks a long-term track record of return on investment.
Analysis of the information dug up by big data tools has become an essential for retailers, helping them sell more goods by analyzing behavior patters, signal hot products and even predict in-store sales based on weather patterns. It's also being executed in real time, rather than taking weeks or months to decipher.
Kroger CEO David Dillon has called big data analytics his "secret weapon" in fending off other grocery competitors. And Sterling Jewelers has harnessed customer analytics in its digital channels, increasing sales 49 percent in the past holiday season.
Retailers are moving quickly into big data, according to Jeff Kelly, lead big data analyst at Wikibon. Big retail chains such as Sears and Target have already invested heavily in reacting to market demand in real time, he said. That means goods can be priced dynamically as they become hot, or not. Similar products can be cross-sold within seconds to a customer paying at the cash register. Data analysis also allows for tighter control of inventory so items aren't overstocked.
Amazon.com, with an infrastructure that rivals that of Google, was, not surprisingly, early to the big data game. The company pioneered using data to make recommendations to customers based on their past buying behavior.
(Read more: Data Premium Rises in Health Care)
"It can even cross-correlate buying behavior between home and garden sales," Kelly said.
Amazon, part tech company and part retail giant, illustrates the technology-based pressure faced by legacy brick-and-mortar businesses and the critical role of data analysis in keeping them from falling behind.
"Retailers are seeing more pressure from dot-com company sites," said Matt Quinn, chief technology officer at Tibco, which makes big data analytics software. One retailer, for example, now processes customer data in three seconds, the amount of time it takes to swipe a credit card.
In fact, retailers are quickly outgrowing their database infrastructure. "Networks aren't fast enough to process big data, and there isn't enough memory," said Mark Beyer, a big data analyst at Gartner.
(Read more: Big Data Cashes in on Customer Feelings)
Adoption is occurring so quickly that retailers are inundated with data streams. Wal-Mart, for example, generates a million rows of transaction records every hour, according to Bank of America Merrill Lynch. The data isn't coming just from its in-store transaction records, but from social networks, smartphones, mobile devices, sensors and digital video recorders that can track customers' shopping habits.
The ultimate trick is marrying outside big data, such as weather reports or social media, to the inside data retailers already collect.
"Big data's challenge is that it's so varied," said Jill Puleri, global leader for IBM's retail consulting practice. "How do you make sense of this noise?"
For example, Twitter streams can produce keywords that are misinterpreted by machines compiling data. Coordinating online and brick-and-mortar stores is especially challenging, Puleri said.
Tomak admitted that connecting Macys.com customer analytics with its actual stores is a work in progress. The challenge is to refine the use of data analysis.
Retailers that aren't embracing big data as part of their arsenal "are behind the eight ball," Wikibon's Kelly said. "There's no time to spare."