Friday, 26 April 2013
The R
Thursday, 25 April 2013
Wednesday, 24 April 2013
Cordray defends consumer bureau big-data effort - Business - The Boston Globe
By Carter Dougherty
| BLOOMBERG NEWS APRIL 24, 2013
WASHINGTON — The Consumer Financial Protection Bureau’s data-collection program does not invade individuals’ privacy, is a vital regulatory tool, and mimics techniques in wide use by the private sector, the agency’s director said Tuesday.
‘‘The big banks know more about you than you know about yourself,’’ director Richard Cordray told the Senate Banking Committee. At a hearing, he faced criticism about the initiative.
The bureau is collecting data to improve its rule-writing and supervisory work, sometimes through purchasing it from credit bureaus like Experian. It is also requesting large amounts of information from major banks in areas such as checking account overdrafts and credit cards.
Senator Mike Johanns, Republican of Nebraska, highlighted potential violations of privacy.‘‘
‘‘To many people, this is going to sound downright creepy,’’ Johanns said. ‘‘People are going to be bothered by the fact that there’s this federal agency that’s collecting data on the behavior of people like you and me and everybody else who’s paying off a mortgage, who’s paying credit card bills every month.’’
Cordray said the bureau does not monitor people and guards data carefully. Consumers, though, do sometimes share personally identifiable information, as when they file complaints.
‘‘The notion that we’re tracking individual consumers or invading their privacy is quite wrong,’’ Cordray said.
Cordray emphasized the bureau needs this information to do the work, such as cost-benefit analysis, that Congress has ordered. He also said the private sector has fully embraced data analytics, often labeled ‘‘Big Data.’’
Republican Senator Michael Crapo of Idaho complained that he ‘‘specifically’’ asked the agency about its data collection last month, but learned more about the subject from an April 17 article published by Bloomberg News.
‘‘The responses I received downplayed the nature and extent of the issue,’’ Crapo said.
Crapo said that private-sector data collection is fundamentally different from the bureau’s activities. ‘‘Nobody in the private sector has the power that the federal government has,’’ Crapo said. ‘‘The power of the government is behind this data collection.’’
Cordray said the Federal Reserve sometimes purchases data for its own work on consumer finances. The goal is to establish a starting point for where the market stands, he said. ‘‘There are times where as we examine institutions, we have to begin by getting a baseline of data to calibrate what is going on,’’ he said.
In its first 21 months, the bureau has secured $425 million in relief for 6 million consumers wronged by financial service providers, Cordray said.
‘‘We also imposed penalties on the companies to deter such activity in the future,’’ Cordray said.
The bureau, created by the 2010 Dodd-Frank Act, has taken enforcement actions in areas including credit-card add-on products such as credit monitoring, and mortgage insurance.
Tuesday, 23 April 2013
With Big Data, Context is a Big Issue | Innovation Insights | Wired.com
- 04.23.13
In the war on noise, contextual applications serve as crucial ammunition.
The Age of Content
With all the hype around Big Data, we’ve become extremely proficient at collecting data – be it from enterprise systems, documents, social interactions, or e-mail and collaboration services. The expanding smorgasbord of data collection points are turning increasingly portable and personal, including mobile phones and wearable sensors, resulting in a data mining gold rush that will soon have companies and organizations accruing Yottabytes (10^24) of data.
To put things into perspective, 1 Exabyte (10^18) of data is created on the internet daily, amounting to roughly the equivalent of data in 250 million DVDs. Humankind produces in two days the same amount of data it took from the dawn of civilization until 2003 to generate, and as the Internet of Things become a reality and more physical objects become connected to the internet, we will enter the Brontobyte (10^27) Era.
So it’s all dandy that we’re getting better and better at sucking up every trace of potential information out there, but what do we do with these mountains of data? Move over Age of Content, enter the Age of Context.
The Age of Context
Big Data has limited value if not paired with its younger and more intelligent sibling, Context. When looking at unstructured data, for instance, we may encounter the number “31” and have no idea what that number means, whether it is the number of days in the month, the amount of dollars a stock increased over the past week, or the number of items sold today. Naked number “31” could mean anything, without the layers of context that explain who stated the data, what type of data is it, when and where it was stated, what else was going on in the world when this data was stated, and so forth. Clearly, data and knowledge are not the same thing.
Take the example of a company that has invested heavily in business intelligence (BI) software that organizes internal data. In an increasingly connected world, this company has not leveraged its data to its potential. Why not? Because the company’s internal data is isolated from the rest of the data universe including news, social media, blogs, and other relevant sources. What if you could join the dots between all these data sources and surface hidden connections?
Using an application that, unlike the average BI product, pulls in information from the entire data universe would help the company answer questions like “Why did our sales plummet last month?” as opposed to just “What happened to our sales figures last month?”
The company could further question, “Did our slip in sales have anything to do with the recent elections and uncertainty?” and “How can we make sure our sales do not slip next time there is a shift in the political landscape?” Identifying potential causality is key in spotting patterns that enable prediction.
For organizations and businesses to survive today, they have to contextualize their data. Just as a doctor diagnosing a patient with diabetes based on body temperature alone is incorrect, so is making business decisions derived from data out of context. A doctor needs to know about the patient’s age, lifestyle, diet, weight, family history, and more in order to make a probable and guarded diagnosis and prognosis. Contextualization is crucial in transforming senseless data into real information – information that can be used as actionable insights that enable intelligent corporate decision-making.
At the end of the day, our overworked minds want to be spoon-fed insights. We want key signals and tightly packaged summaries of relevant, intriguing information that will arm us with knowledge and augment our intelligence.
But how do we extract real intelligence from data?
Extract Real Intelligence From Data
Besides cross-referencing internal data with a plethora of other sources, we need algorithms to boil off the noise and extract the signals, or real human meaning, from the data. What do we mean? Let’s say you have a million tweets from New York City on the eve of the U.S. elections and rather than read them all, you want to know quickly how people are feeling based on these tweets.
To do this you must apply complex algorithms derived from machine learning, computational linguistics, and natural language processing to harvest the key words and corresponding emotions from the tweets. Then you get the key signals: Are people feeling anxious? Hopeful? Confident? Fearful? This is precisely what we do here at Augify: Paving the way for the future of understanding by surfacing signals from the noise in our cloud-based, algorithm-packed product. We wrap the signals in a dashboard of slickly designed, color-coded gauges and visualizations that enhance your understanding of key insights.
Getting Contextual
So it’s great that applications out there can gather and analyze data, detect human-based meaning from it, and visualize it all, but any application is limiting itself if it is only useful once you open the application and enter a query. We wanted to go beyond this. That’s why we decided to get contextual on you and increase our technology’s reach. We go beyond contextual data by developing contextual applications – we wanted to put data to good use by applying it to real-life situations in our daily lives.
The Don of ubiquitous computing, Mark Weiser, stated in 1991 that “the most profound technologies are those that disappear. They weave themselves into the fabric of our everyday life until they are indistinguishable from it,” much like the telephone or electricity. They have seeped into our surroundings, playing an integral role in our everyday lives.
Modern day examples include adaptive technologies such as Google Now, which tracks your online behavior and uses this data to predict the information that you will need, such as local traffic or weather updates. Similarly, “learning thermostat” Nest self-adjusts your home’s temperature based on your activity, saving energy usage and bills. Pervasive technologies mean they are everywhere and nowhere at the same time – an invisible layer listening to your actions in order to be more helpful.
Along the same vein, we believe in pervasive Augmented Intelligence, meaning that you can reap the benefits of Augify’s signal-detecting technology anywhere you go in the connected world. So even when you are working outside of the Augify application screen, be it writing an e-mail or reading a website, Augify’s algorithms will work silently and invisibly in the background, culling key insights from the text you are reading at that very moment and feeding them to you in a pop-up window.
Pervasive Augmented Intelligence
Imagine being able to save time reading long text by getting a synopsis of the key themes, topics, emotions, people, and entities in a news article? By seeing the emotions, sentiments, intentions, and credibility in someone’s Twitter feed, and by spotting the key influencers in their network? What if, as you typed an e-mail to your boss about a new enterprise software, you received pop-up recommendations of credible reference articles related to said software?
In other words, what if you could detect the important signals in any web content in real-time? With Augify’s contextual application, this is truly possible, enabling you to augment your intelligence anywhere your online activity takes you.
Augment Your Intelligence Anywhere Your Online Activity Takes You
The Age of Context demands that contextual data be applied to everyday situations in useful ways. How do we make use of this data? Since we’ve gotten good at collecting data, now it’s all about putting it into context and making sense out of it – mining for the nuggets of insights that answer the “So What?” question. Data is meaningless and even cumbersome without context – the key holistic and interpretive lens through which data is filtered and turned into real information.
Surprise: You’re using ‘Big Data’- MSN Money
Big Data is the popular name for the field of data analytics, the process of sifting through massive amounts of information with large numbers of variables in split seconds to produce results that can be used make evidence-based decisions.
Think statistics blogger Nate Silver accurately predicting Obama's win in the 2012 presidential election, or IBM's Watson computer trouncing two human "Jeopardy" champs -- that's Big Data.
The bits of data generated by websites, social networks and smartphones apps are creating a "nuclear explosion" of information, as a recent Wired report called it. Combined with the power of relatively low-cost cloud-based software and digital storage, it's contributing to a global mountain of data that in 2013 is expected to reach an unprecedented 1.2 zettabytes, according to a report from Tableau Software. That's equal to 1 billion terabytes.
Companies and other organization are jumping into big data with abandon. It's being used by everyone from health insurance carriers to analyze bills for fraud to police departments to analyze past crime patterns to better predictwhere future thefts or robbers are most likely to occur.
The average Joe and Jane are benefiting because retailers and other companies are using big data to offer more personalized services. People have always put a high value on one-on-one interactions, and the experience that Big Data offers "simulates a parallel world online that's as personalized and trustworthy as offline," says Vikas Sabnani, chief data scientist with Glassdoor.com, the jobs and careers website. "Big data is the only way we can get there."
Big data, no price tag
Many big data-based services for shoppers, job seekers and other consumers are free.
Job hunters pay nothing to use Glassdoor to search through millions of job openings, see salary information for specific jobs and read reviews of what it's like to work at myriad companies.
Companies post the job openings. The rest of the information comes from reviews or data on salaries, bonuses and commissions that job hunters share with the site. Glassdoor compiles and analyzes all that data to provide job hunters with as many specifics as possible when they search on the site. If someone who's interested in software engineer jobs does a search on the term and then clicks on a job at Google, Glassdoor takes that as feedback that they want to work in the tech industry or at a large company like Google, and uses the information to show ratings and reviews of companies they might be interested in.
Glassdoor also lets job seekers link to their Facebook accounts to see if any of their friends or connections work at any of the companies. All of it is possible because Big Data doesn't take as much computing power or money as it would have only a short time ago, Sabnani says. "The cloud makes it much cheaper for us to process and scale as opposed to hosting it on our own servers."
Amazon is a long-time proponent of Big Data, which runs the recommendations engine on its online bookstore. Browse through the bookstore and you'll see recommendations pop up for novels or other books you might be interested in. The company analyzes stored data on books you've previously looked up on the site, and compares that with data from customers who've either looked up or bought the same books. Based on the information, the website shows you books that those customers have looked at or purchased that might appeal to you.
Not all big-data services are free. A recommendation service on Netflix is only available to people who pay the $7.99 monthly subscription. If they do, they can opt to use the online entertainment company's Taste Profile tool to rate films and TV shows with up to five stars. The company then compiles that data with billions of ratings from its other 36 million members to suggest other shows or films a subscriber might want to watch. The more shows or movies you watch and rate, the better the suggestions get. "It's the network effect," says Phil Simon, author of the just-published book, "Too Big to Ignore: The Business Case for Big Data."
Mind-boggling amounts of data
Companies such as Google, Twitter and Facebook are built on Big Data, Simon says. Google uses it in multiple information products, from its ubiquitous search engine to Google News Alerts, a news finding service that can be set up to monitor specific subjects and have updates sent to an email address on a real-time, daily or weekly basis.
Facebook uses Big Data to analyze what users "like" or share, and then uses that to sell ad space on a person's timeline or newsfeed. Simon, a long-time Rush devotee, likes the rock band's Facebook page and takes other actions that show he's a fan. If a music promoter was trying to fill seats at a Rush concert in Simon's hometown of Henderson, Nev., it wouldn't surprise him to see an ad for tickets on his Facebook page. "The amount of data Facebook has on people is mind boggling," he says.
Twitter’s Trends feature is also built on Big Data. When someone on Twitter writes a status update with a hashtag-labeled keyword signifying a topic the update is related to, and agrees to share their location, the company combines the information with other users' updates, hashtags and locations and shares what topics are trending in specific locations or throughout the service.
Big Data’s Usability Problem - Bill Wise - Voices - AllThingsD
Bill Wise, CEO, Mediaocean | APRIL 23, 2013
Sen. Lindsay Graham just told Fox News that the reason the FBI never realized that Boston Marathon bombing suspect Tamerlan Tsarnaev went to Russia in 2011 is that “when he got on the Aeroflot plane, they misspelled his name, so it never went into the system that he actually went to Russia.” Meanwhile, the Reinhart-Rogoff paper that has been a catalyst for government austerity policies worldwide since 2010 has, in fact, accidentally left out several countries’ worth of critical data in Excel.
As one blogger sums up scathingly: “One of the core empirical points providing the intellectual foundation for the global move to austerity in the early 2010s was based on someone accidentally not updating a row formula in Excel.”
Taken together, these factors offer a critical lesson here about the power and limits of Big Data today. In both scenarios, data management tools (i.e., the FBI’s systems and Excel) were undone by fairly simple errors: In one situation, a misspelling; in another, a failure to code a spreadsheet properly. And in both scenarios, the results were dire — an awful tragedy, and a potentially misdirected government economic policy in the midst of a recession.
As someone who spends day and night thinking through data management and workflow, these two stories lead me to three observations:
- As a society, we’re hugely reliant on data management platforms for our most critical information.
- Our core data platforms often aren’t set up to handle human error, from basic coding flaws to spelling mistakes.
- The wealth of data in our data tools can mask that human error. Consider: TheReinhart-Rogoff study examined “new data on forty-four countries spanning about two hundred years” with “over 3,700 annual observations covering a wide range of political systems, institutions, exchange rate arrangements, and historic circumstances.”
In such a wide sea of data, a few lines of code can be very easy to overlook, even if they have strong ramifications for analysis.
There are lots of things to take away from these three points, but I’ll just focus on one: The promise of Big Data is that it can make everyday processes — from critical analyses to mundane tasks — work smarter through data intelligence. Ultimately, all that data management translates into an economy and society that lets machines handle the minutiae as humans think through the larger picture.
To a large extent, that vision is already here. But at the same time, more human/data interaction means a lot more room for error (and inefficiency) around increasingly critical data sets — which, as we’ve seen, can have very serious results. Which means that, if we want to make the reality of Big Data match the dream, we need to spend serious time around providing usability that guides human users in the best way to engage with the data, and automation that takes human interaction (and human error) out of the picture for a lot of the basic calculations and tasks — and for some of the complicated ones, too.
If Big Data can’t fit hand-in-glove with usability and workflow, a lot of the promise of big data will be empty data crunching. That’s not just a problem for getting where we want to be in the evolution of computing. It’s a situation that can lead to bad data management — which translates into bad economics and, sometimes, far worse.
What big data means for marketing decision making | Media Network | guardian.co.uk
David Lloyd | Tuesday 23 April 2013
"Not everything that counts can be counted, and not everything that can be counted counts." This quote, often attributed to Albert Einstein, originated before the concept of big data. Yet now, more than ever, we should reflect on what it means in the context of how our marketing decisions are made.
The first part of the quote reveals an inconvenient truth of analytics; as much as we strive for a world where everything is measurable and all decisions can be based on real-time, infallible data, this is not always possible. Take "social media ROI" for instance; a term that pervades marketing industry commentary. Often, the data to attribute sales revenue confidently simply isn't there and no amount of white papers and positive thinking can fill the gap.
In such cases, it's important to understand the options quickly along with the associated cost, commitment and accuracy. Be prepared not to do it at all. Yes, really. That may seem controversial and may be something that doesn't fit with cultures that only approve projects based on hard commercial benefits, but it is better to focus efforts on what you can do than stretch data beyond credibility.
Instead, focus on more readily available measures that indicate performance against objectives. Have I got good reach across my target audience compared with other channels? Does my target audience even use this social platform? How do they engage with my content compared with the content from competitors? You may find that some of these measures, once properly benchmarked and tracked, along with a good dose of marketing nous, will lead to a well-informed social media strategy.
The second part of the quote provides a counter-perspective yet is equally important. We are told that big data is all around us; that we must collect it or be hopelessly behind the curve and lose huge amounts of revenue. Almost every day a new vendor, seminar or white paper springs up promising the solution to our big data problems. If anything, big data (the campaign) is an unintentional mirror of big data (the real stuff) itself; it creates more noise, more pitfalls and requires more precision to navigate. The amount of data available does not necessarily correlate with its value – not everything that can be counted counts.
Yet the solution to such concerns is similar to the solution when you don't have enough data. Align the data you need to your strategic objectives, assess the options and make the decision accordingly. This could be big data, small data or somewhere in-between data. These principles hold true now, as they did when customer relationship management (CRM) was marketing's next big thing, and before the quote itself was conceived.
We live in an environment where certain parties obfuscate some of these truths, whether they do that to sell technology or simply amplify the hype to appear to be leading the conversation.
I'm aware that I'm focusing on the fallibility of data; we can't take for granted that data can drive every decision, we must use also our judgment. Well, I'm prepared to be honest about this as, despite the limitations of data, far more decisions are made by instinct alone when data could have improved those decisions. This gap is a huge opportunity and to close it, we just need the foresight to collect the right data and ensure it is used correctly by people with the right skills.
If you're a business focusing on the big data question and how it might revolutionise your marketing, I advise you to take stock of your objectives. Is there a clear connection between your strategy and the data? Are you only planning to collect the data you really need (regardless of size)? Do you have a clear roadmap and the expertise to steer the course? If you can't provide a confident yes to these questions, make sure you rectify that before spending considerable time and money on solutions. Otherwise you could find yourself another wreck stranded on the rocks.
Monday, 22 April 2013
Forbes India Magazine - Making Big Data Real
Much of the business world is consumed by and swimming in Big Data. Were you a casual observer you might even say that the majority of companies are now drowning in data. A considerable proportion of insiders in industries as diverse as financial services, mobile telephony and digital publishing has little to no idea which data are valuable or how to deploy them. The truth is in this in confusing world, where new vendors are popping up every day and there are few standards for measuring performance and return on investment, many marketers and proportionally more publishers are avoiding taking on the challenge of Big Data altogether.
 | |
The purpose of this article is to present a broad brush on the state of ad tech – broadly defined as technology-, math- and data-enabled marketing services – an admittedly complex marketplace, but one that can make the challenge of taking on the Big Data challenge less formidable. Another purpose is to highlight the business models and strategies of four leading companies from quite different walks of life – in terms of how they serve clients and the business models they employ – that are helping to reduce the complexity of big data by presenting cohesive and integrated enterprise solutions for marketers and publishers.
Whose data is it anyway?
Given what we know, or don’t know, about audience and media fragmentation, the reality – for those serving the Big Data market for marketers and publishers — is that real data, effectively deployed and interpreted, make for real insight.
Few marketing department leaders are sufficiently numerate to know which data are useful and how best to deploy data, as well as the supporting math to understand the value they are creating or destroying for their shareholders. This unacceptable state of affairs prevails inside big marketers like retailers, consumer packaged goods companies and financial services companies. It also applies to traditional and digitally native publishers who are seeking to “own” their readers, listeners and viewers by better understanding what the data about their behaviors, intent and loyalty are trying to tell them. Finally, the imperative to understand the data and to do the math especially goes for advertising agencies and other intermediaries (such as by type of business), the many that are clinging to any perch they can as their business continues to decline.
Ad agencies in particular have much to loose, since there are few barriers to entry for vendors offering significant reach and scale, and that are seeking to control specific links of the industry’s value chain (see the four examples below). Gone are the simpler but inherently conflicted days when agency clients were able to rely on the same agency to spend their money and then tell them the results they created. Now, leading marketers[i] use between five and ten data, math and tech vendors, none of which is an agency holding-company subsidiary.
Agencies have always behaved as if their clients’ customer data were their own. But marketers nowadays are increasingly viewing data as an asset – to be owned, mined and leveraged to maintain an edge in serving certain core audiences. For marketers, publishers and ad agencies, the legal ownership of the data or right to access them is the ultimate form of control over how, when, where and why an audience is engaged. Since many marketers, and even more publishers, would prefer to outsource the complicated work of audience management, ownership is the missing link in a lot of audience data-management strategies at the moment.
Big Data: A crowded market
The data-ownership issue frames the current status quo among ad tech vendors: there are just a few players – first, second- and third-party data[ii] providers and integrators, as they are known – that matter. So they can bob and weave with their clients’ emerging needs, each has a strong entrepreneurial bend. Each member of this short list also knows how to take a disorganized corner of a growing market and make a real company out of the opportunity.
The many vendors who provide services to marketers, publishers, and their supply chain intermediaries form what is loosely called the digital advertising marketplace[iii]. Current estimates value this market at just under $60-billion for 2012. With such a bounty, it’s no surprise that there are as many vendors as there are. Terry Kawaja, founder of Luma Partners in the San Francisco Bay area, has identified several clusters of vendors. (Please see the footnoted link to a series of visual depictions of the display, video, social, mobile, gaming and commerce marketplaces).
That there are dozens if not hundreds of these vendors — virtually all of them with venture backing — has created a malaise for their industry, clients and for the state of Big Data as a decision-making tool. This presents a quandary for entrepreneurs looking to enter the market with their next great algorithm, data integration tool or attribution (what happened when money was invested) insight. The same goes for the potential investors that are seeking to organize the marketplace by rolling up a few of the most differentiated and successful vendors to present a more integrated and cohesive offering to publishers and marketers.
Whose data is it anyway?
Given what we know, or don’t know, about audience and media fragmentation, the reality – for those serving the Big Data market for marketers and publishers — is that real data, effectively deployed and interpreted, make for real insight.
Few marketing department leaders are sufficiently numerate to know which data are useful and how best to deploy data, as well as the supporting math to understand the value they are creating or destroying for their shareholders. This unacceptable state of affairs prevails inside big marketers like retailers, consumer packaged goods companies and financial services companies. It also applies to traditional and digitally native publishers who are seeking to “own” their readers, listeners and viewers by better understanding what the data about their behaviors, intent and loyalty are trying to tell them. Finally, the imperative to understand the data and to do the math especially goes for advertising agencies and other intermediaries (such as by type of business), the many that are clinging to any perch they can as their business continues to decline.
Ad agencies in particular have much to loose, since there are few barriers to entry for vendors offering significant reach and scale, and that are seeking to control specific links of the industry’s value chain (see the four examples below). Gone are the simpler but inherently conflicted days when agency clients were able to rely on the same agency to spend their money and then tell them the results they created. Now, leading marketers[i] use between five and ten data, math and tech vendors, none of which is an agency holding-company subsidiary.
Agencies have always behaved as if their clients’ customer data were their own. But marketers nowadays are increasingly viewing data as an asset – to be owned, mined and leveraged to maintain an edge in serving certain core audiences. For marketers, publishers and ad agencies, the legal ownership of the data or right to access them is the ultimate form of control over how, when, where and why an audience is engaged. Since many marketers, and even more publishers, would prefer to outsource the complicated work of audience management, ownership is the missing link in a lot of audience data-management strategies at the moment.
Big Data: A crowded market
The data-ownership issue frames the current status quo among ad tech vendors: there are just a few players – first, second- and third-party data[ii] providers and integrators, as they are known – that matter. So they can bob and weave with their clients’ emerging needs, each has a strong entrepreneurial bend. Each member of this short list also knows how to take a disorganized corner of a growing market and make a real company out of the opportunity.
The many vendors who provide services to marketers, publishers, and their supply chain intermediaries form what is loosely called the digital advertising marketplace[iii]. Current estimates value this market at just under $60-billion for 2012. With such a bounty, it’s no surprise that there are as many vendors as there are. Terry Kawaja, founder of Luma Partners in the San Francisco Bay area, has identified several clusters of vendors. (Please see the footnoted link to a series of visual depictions of the display, video, social, mobile, gaming and commerce marketplaces).
That there are dozens if not hundreds of these vendors — virtually all of them with venture backing — has created a malaise for their industry, clients and for the state of Big Data as a decision-making tool. This presents a quandary for entrepreneurs looking to enter the market with their next great algorithm, data integration tool or attribution (what happened when money was invested) insight. The same goes for the potential investors that are seeking to organize the marketplace by rolling up a few of the most differentiated and successful vendors to present a more integrated and cohesive offering to publishers and marketers.
One might ask why this marketplace has not become better organized. The fact is that few funds or strategic investors such as private equity firms are interested in markets with low barriers to entry, namely a market with the many venture-backed vendors and companies that look more like features than companies.
While many believe marketing has become the corporate flag-bearer for Big Data, marketing is not and should not be its only or best home. Marketing is responsible for extracting insights on audience behaviors and intent that clarify company-wide lines of business and brand financial forecasts. Finance is responsible for incorporating these insights into the sensitivity analyses that allow board members to understand the specific near- and longer-term effects of alternative investments in marketing – allowing the prioritization of marketing investment alongside others. Technology is responsible for making the best decisions on vendor selection based on performance.
In an ideal world, the triumvirate of marketing, finance and technology is creating substantially new ways for companies to deal with current competitive challenges and for adjusting audience campaigns and other elements of ad spend while more precisely tuning long-term forecasts for shareholders and establishing performance standards for data vendors and integrators. In the real world, marketing, finance and technology are not departments that typically speak the same language or even get along much of the time. Marketing has long been saddled by the judgment of finance and tech departments that, as the owner of the their company’s revenue, they are important but essentially are also a cumbersome and ultimately irrelevant cost center.
Long the bugbear for companies trying to efficiently sort fact and fiction come budget time, the marketing function has lately been going through a substantial transformation. Much of the transformation has been led by the availability of audience and consumer data, by marketing and computer scientists able to parse and understand those data in a meaningful way, and the technology and computing cycles available to generate these insights in real time. Finally, there is a new willingness by c-suites, boards and other business leaders to look at their businesses differently[iv]. Financial services, healthcare, retail and several of their supply-chain companies have particularly benefited from deeper and more consistent insights on audience and customer behavior, intent and loyalty, and have used Big Data to gain a sustainable competitive edge.
All this change and open-mindedness, though, comes at a price. For old business methods to give way to new, old-business practitioners need to be able to establish business processes, standards of practice and performance metrics. The goal for companies that have not moved to do so would be to create the triumvirate of marketing, finance and tech departments referred to above. Marketers want standard performance metrics used across all their marketing campaigns.
Publishers want to understand their real cost per thousand (CPM) pricing floor (above which they make a profit on the content they are selling to marketers). Agencies, who buy publishers’ content and attach their clients’ advertising to it, desire to maintain the preeminence of their creative and workflow management capabilities and the fee streams from them. As such, they will continue to resist any substantial changes to the way they do business. But in the $60B online advertising world, a figure now representing over half of all advertising budgets, substantial change is needed. Zach Coelius, the CEO of Triggit (profiled below), puts the opportunity succinctly: “If you can stem the dramatic fragmentation of media and audience that has accompanied the explosion of screens and other venues where consumers access content, you can monetize the dramatic amount of data made available to marketers from these many new channels.”
A real and measurable opportunity to reinvent marketing for advertisers and publishers
There are just a few data and insight vendors who have not become captive to the entrenched needs of old practitioners or to the fad, for some, of Big Data. One of them, ThinkVine, based in Cincinnati, Ohio, has deconstructed Big Data. Calling the glut of data available a “huge distraction”, ThinkVine’s Founder Damon Ragusa speaks to his customers (not “clients”) about Smart Data and Big Insights. ThinkVine’s customers have come to appreciate the hands-on nature of the business model and the fact that beyond the technology and math behind the offering, the basic operating principle is to remain source-agnostic and to understand the utility of the data.
Another leader is Triggit. Based in San Francisco, Triggit exclusively serves FaceBook Exchange, on one level a global social media publisher that has moved substantially toward not only owning its audience data, and is finding ever-better ways to use it to make advertising investment decisions for its advertiser clients.
The third is Krux digital, another San Francisco, California company that has developed a self serve offering for enterprises and medium and smaller sized business as well.
The fourth is Merkel, which is based in the Columbia, Maryland. With ambitions of building a substantial growth trajectory and better organizing data serving the ad-tech world, Merkel Group companies present clients with a potentially significant opportunity to cut through the data “Babel.” Coming from an offline data orientation but firmly rooted in the digital data world, Merkel can promise marketers and publishers that they will have a real opportunity to have the same conversation about how best to serve mutual customers’ interests. It would be safe to say that Merkle and the others we have highlighted have changed the game for many of their clients. Clients have realized tenfold and even greater returns on their capital investments in ad campaigns and have enjoyed significantly greater reliability from multi-year forecasts.
Each of the companies we have examined offers entrepreneurs, their clients and other business leaders important lessons, to paraphrase Animal Mother’s taunt (a character in Stanley Kubrick’s film Full Metal Jacket[v]) on how to stop just talking about data and “walk the talk.” As stated earlier in this article, the reality of media and audience fragmentation for entrepreneurs serving the big data market for marketers and publishers is that real data – effectively deployed and interpreted – makes for real insight.
While many believe marketing has become the corporate flag-bearer for Big Data, marketing is not and should not be its only or best home. Marketing is responsible for extracting insights on audience behaviors and intent that clarify company-wide lines of business and brand financial forecasts. Finance is responsible for incorporating these insights into the sensitivity analyses that allow board members to understand the specific near- and longer-term effects of alternative investments in marketing – allowing the prioritization of marketing investment alongside others. Technology is responsible for making the best decisions on vendor selection based on performance.
In an ideal world, the triumvirate of marketing, finance and technology is creating substantially new ways for companies to deal with current competitive challenges and for adjusting audience campaigns and other elements of ad spend while more precisely tuning long-term forecasts for shareholders and establishing performance standards for data vendors and integrators. In the real world, marketing, finance and technology are not departments that typically speak the same language or even get along much of the time. Marketing has long been saddled by the judgment of finance and tech departments that, as the owner of the their company’s revenue, they are important but essentially are also a cumbersome and ultimately irrelevant cost center.
Long the bugbear for companies trying to efficiently sort fact and fiction come budget time, the marketing function has lately been going through a substantial transformation. Much of the transformation has been led by the availability of audience and consumer data, by marketing and computer scientists able to parse and understand those data in a meaningful way, and the technology and computing cycles available to generate these insights in real time. Finally, there is a new willingness by c-suites, boards and other business leaders to look at their businesses differently[iv]. Financial services, healthcare, retail and several of their supply-chain companies have particularly benefited from deeper and more consistent insights on audience and customer behavior, intent and loyalty, and have used Big Data to gain a sustainable competitive edge.
All this change and open-mindedness, though, comes at a price. For old business methods to give way to new, old-business practitioners need to be able to establish business processes, standards of practice and performance metrics. The goal for companies that have not moved to do so would be to create the triumvirate of marketing, finance and tech departments referred to above. Marketers want standard performance metrics used across all their marketing campaigns.
Publishers want to understand their real cost per thousand (CPM) pricing floor (above which they make a profit on the content they are selling to marketers). Agencies, who buy publishers’ content and attach their clients’ advertising to it, desire to maintain the preeminence of their creative and workflow management capabilities and the fee streams from them. As such, they will continue to resist any substantial changes to the way they do business. But in the $60B online advertising world, a figure now representing over half of all advertising budgets, substantial change is needed. Zach Coelius, the CEO of Triggit (profiled below), puts the opportunity succinctly: “If you can stem the dramatic fragmentation of media and audience that has accompanied the explosion of screens and other venues where consumers access content, you can monetize the dramatic amount of data made available to marketers from these many new channels.”
A real and measurable opportunity to reinvent marketing for advertisers and publishers
There are just a few data and insight vendors who have not become captive to the entrenched needs of old practitioners or to the fad, for some, of Big Data. One of them, ThinkVine, based in Cincinnati, Ohio, has deconstructed Big Data. Calling the glut of data available a “huge distraction”, ThinkVine’s Founder Damon Ragusa speaks to his customers (not “clients”) about Smart Data and Big Insights. ThinkVine’s customers have come to appreciate the hands-on nature of the business model and the fact that beyond the technology and math behind the offering, the basic operating principle is to remain source-agnostic and to understand the utility of the data.
Another leader is Triggit. Based in San Francisco, Triggit exclusively serves FaceBook Exchange, on one level a global social media publisher that has moved substantially toward not only owning its audience data, and is finding ever-better ways to use it to make advertising investment decisions for its advertiser clients.
The third is Krux digital, another San Francisco, California company that has developed a self serve offering for enterprises and medium and smaller sized business as well.
The fourth is Merkel, which is based in the Columbia, Maryland. With ambitions of building a substantial growth trajectory and better organizing data serving the ad-tech world, Merkel Group companies present clients with a potentially significant opportunity to cut through the data “Babel.” Coming from an offline data orientation but firmly rooted in the digital data world, Merkel can promise marketers and publishers that they will have a real opportunity to have the same conversation about how best to serve mutual customers’ interests. It would be safe to say that Merkle and the others we have highlighted have changed the game for many of their clients. Clients have realized tenfold and even greater returns on their capital investments in ad campaigns and have enjoyed significantly greater reliability from multi-year forecasts.
Each of the companies we have examined offers entrepreneurs, their clients and other business leaders important lessons, to paraphrase Animal Mother’s taunt (a character in Stanley Kubrick’s film Full Metal Jacket[v]) on how to stop just talking about data and “walk the talk.” As stated earlier in this article, the reality of media and audience fragmentation for entrepreneurs serving the big data market for marketers and publishers is that real data – effectively deployed and interpreted – makes for real insight.
Four Data Entrepreneurs @ Two Lessons Each
Merkle is a $300-million private company that serves the customer relationship management (CRM) marketplace. In business since 1988, when CEO David Williams acquired the company, Merkle has promoted a management ethos to grow its business organically as much as possible and keep the widely respected management team intact. In 2008, the eminent Technology Crossover Ventures fund invested $75-million in the company. Currently, the company produces higher-than-median earnings (before non-cash items) and is attracting the interest of a few well-known private equity funds. These funds are interested in creating a more interesting marketing business model by building a multi-company roll up of smaller tech-enabled data leaders.
“We’re not there yet,” said Craig Dempster, CMO, when discussing the often crazy world of valuations among Merkle’s Big (digital) Data competitors. “It is possible to build the $5-$10-billion company [in this marketplace], though” because the traditional incumbents (IBM, Microsoft, Oracle and SAP, in particular) have failed to organize the market as they have done for accounting, manufacturing, human resources management and other central business processes.
While Merkle behaves like an integrated advertising agency holding company and presents itself as a CRM vendor, there is more going on within their unique business model. Among the companies profiled here, Merkle is the odd duck: it has much greater revenues, is better funded and more self-sufficient from a funding perspective, and is increasingly emerging as one of those able to better organize the display – traditional online advertising – and other marketing channels. As such, it is the source of a lot of buzz from the investor and competitor sides of the playing field.
ThinkVine is certainly an original thinker of the first order. ThinkVine’s intellectual property is based on its software-as-a-service application of agent-based modeling – a methodology for predicting consumer behaviors using artificial consumers in a closed universe. ThinkVine was first to market with a variant that could deliver near-perfect accuracy. But what has accounted for their greater-than 35 percent year-over-year growth rate? Simple to say, much harder to do: to establish a defensible and unique business model, ThinkVine has chosen to remain agnostic with respect to the data types and sources required to power their model.
As recently as two years ago, driven by venture capital investors’ quests for defendable assets, all the players in ad-tech were considering owning uniquely generated or originated data feeds that would more precisely describe or predict consumer behaviors in digital media.
As recently as two years ago, driven by venture capital investors’ quests for defendable assets, all the players in ad-tech were considering owning uniquely generated or originated data feeds that would more precisely describe or predict consumer behaviors in digital media.
On the subject of barriers to entry in the marketplace, Triggit’s CEO Zach Coelius is emphatic: the barriers are “much higher than you think” since “data and automated buying is the future for all media.” He is more pointed on the subject of the role of the ad agency: they “put themselves in a tricky position of protecting their own margins before their customers’ – this blocks entrepreneurship”. About questions relating to the future of his marketplace, Coelius is unequivocal: “How could I be worried. I bet on data, fragmentation and on real time. This means I also bet against other players’, business models and existing players like agencies.
The industry suffers from a lack of organization
“When does a company offer a ‘feature’ and when is it a real business?”[vi]
This is the question posed frequently by buy-side interests. In other words, is the offer unique and does it solve a business problem or is it a nice-to-have service or unique data source? Guess which gets the renewal.
Many of the business models used in digital marketing channels are going the way of their counterparts who have become dinosaurs in the offline world. Those choosing to ride out the truly giant structural changes in the marketing and broader business processes that have put digital marketing decision makers out front have chosen their futures. The market is about to be forced to organize by the vendors offering distinctive services that solve business problems for publishers and marketers simultaneously and by the strategic investors seeking to create a more efficient marketplace. The winners and losers will be sorted, as they often are when a market achieves catharsis, by how real they are.
Sorting this way will allow Big Data vendors and their funders to complete the process of changing how marketing and publishing businesses think.
Reprint from Ivey Business Journal
“When does a company offer a ‘feature’ and when is it a real business?”[vi]
This is the question posed frequently by buy-side interests. In other words, is the offer unique and does it solve a business problem or is it a nice-to-have service or unique data source? Guess which gets the renewal.
Many of the business models used in digital marketing channels are going the way of their counterparts who have become dinosaurs in the offline world. Those choosing to ride out the truly giant structural changes in the marketing and broader business processes that have put digital marketing decision makers out front have chosen their futures. The market is about to be forced to organize by the vendors offering distinctive services that solve business problems for publishers and marketers simultaneously and by the strategic investors seeking to create a more efficient marketplace. The winners and losers will be sorted, as they often are when a market achieves catharsis, by how real they are.
Sorting this way will allow Big Data vendors and their funders to complete the process of changing how marketing and publishing businesses think.
Reprint from Ivey Business Journal
Source: http://forbesindia.com/article/ivey/making-big-data-real/34977/0
Phil Simon: Boston, Privacy, and Big Data
Phil Simon
The events last week were nothing short of horrible, although Bostonians and Americans together rejoiced on Friday as the second bombing suspect was captured. I for one shed a tear Friday night while watching citizens cheering police officers and government officials. You just don't see that every day.
Without question, the Brothers Tsarnaeva would not have been identified and apprehended so quickly-if at all-were it not for the scores of videos and pictures people willingly provided to authorities. You'll get no argument from me that we should have used every means at our disposal to apprehend these dangerous individuals.
Catching the bad guys is great, but in an ideal world federal, state, and local governments wouldalways prevent attacks like these from happening in the first place. Unfortunately, that's just not a likely scenario, although Big Data and powerful analytic tools should help us reduce the likelihood of (successful) future terrorist attacks.
In Too Big to Ignore, I address the far-reaching privacy and security implications of Big Data. Yes, good old-fashioned police work still matters, as events of last week proved. At the same time, though, government agencies and officials have at their disposal much better technologies and tools than even ten years ago. They could not realistically mine billions of call records, search emails, penetrate networks with geolocation data, etc. Today, these limitations cease to exist. Authorities can effectively use the data generated by suspected and real terrorists and criminals as a weapon. And here's where privacy implications start to matter.
The sticky ethical conundrum involves drawing the line. Should Internet service providers, cell phone carriers, and email services give government agencies user data? Should Amazon let the FBI and CIA in on who is buying what books? Should Google flag certain videos and report stats (along with individually identifiable information) to authorities?
I don't have the answers to these questions. According to one school of thought, no one has a right to know which videos people have watched, emails we've sent, and calls we've made. On the other end of the spectrum, if the government can use this information to prevent another Boston-like tragedy and you have nothing to hide, then isn't a little sacrifice of personal privacy in order?
In an era of Big Data, events like the Boston Marathon bombings make answering this question more important than ever.
Unlock the creativity of your security team with big data | Media Network | Guardian Professional
Mark Seward | Monday 22 April 2013
When we talk about data security, there's an increasingly popular adage that there are two types of companies: those that have been breached and those that don't know they've been breached.
The reason that the latter type of company exists is because attacks on enterprise infrastructure have become both more persistent and more sophisticated. In other words, unless the company has bullet-proof security processes and technology in place, it's almost inevitable that some form of attack will have penetrated the enterprise perimeter, whether the company realises it or not.
Once an attacker or piece of malware is inside the network, it can often lurk unseen among the mass of data that enterprise systems generate and trying to locate it, even if you're aware that an attack has taken place is extremely difficult.
That's why the new frontier of enterprise security is big data and statistical analysis specifically in machine data. Every interaction with a 'machine' – whether it's a website, mobile device, application server, corporate network, sensor or electronic tag, and whether it's automatically generated or a manual transaction – leaves a trail and a record.
In this new world of security, chief security officers (CSOs) and IT teams have to unlearn their over-reliance on traditional data protection technologies such as anti-virus software, firewalls and security information and event-management (SIEM) systems. Instead, the tools which might be most useful in the future include a statistics textbook, a subscription to Psychology Today, and the security professional's own brain.
Over the past few years, three major issues have come to dominate the current enterprise security landscape. First, the enterprise is now under pretty much constant attack, with common sense dictating that this increases the likelihood of a breach taking place. The attacker only has to be right once, while the IT team has to be right all the time. This being the case, relying on an attacker to alert you to their presence by tripping a specific rule that you've set in a SIEM clearly isn't a sufficient or reliable strategy.
Second, attackers understand that the pressure on IT teams has increased enormously as a result and with much of their attention and resources dedicated to protecting the perimeter, companies don't tend to do a great job at monitoring what's happening inside the network. As such, if they make it past the barricades, attackers can become very difficult to spot and can act with impunity.
Finally, and perhaps most worrying for the security industry as a whole, this constant bombardment has turned security into a reactive, administrative role, where team members are just responding to systems alerts rather than thinking more laterally about threats. Security is an exciting industry to be in, but too often, both seasoned professionals and new entrants aren't being challenged to be creative and come up with new ways to defend the enterprise – they're just doing what they are told to do by the tools they use.
So what's the solution to these problems? It's about having much greater oversight of everything that's happening inside the enterprise and developing operational intelligence. It's also about giving security professionals the right tools to quickly analyse and sift through enterprise data sets that include data generated through normal interaction with IT systems – right down to the machine data – in order to identify unusual patterns and abnormal behaviour in that data which might indicate that an attack is taking place.
It's here that statistics, psychology and old-fashioned brainpower really come into their own.
Big data analysis technologies exist that can help to identify possible anomalies, but it still requires human insight and intelligence to interpret what they might mean.
For example, there are giveaways that malware might be trying to compromise your network available in the machine data logs – these include URL strings that might be two to three times longer than normal, indicating the possible presence of command and control instructions attempting to launch a web protocol attack, or it might be the 'tell' of a network access password being entered 10 times faster than it's possible for a human to type.
In addition to these web protocol attacks, there are other examples of data logs that track human interactions with IT systems and facilities that might raise the security teams' suspicions. Why is a user repeatedly trying to access a file they don't have permission to view, or why has their ID card been used to enter the office when they're meant to be on holiday in the Bahamas?
Achieving this level of operational intelligence not only opens up new possibilities for how companies defend themselves against the myriad security threats that they face, but also re-engages the interest and creativity of the IT teams entrusted with this vital task.
The days of rules-based security engines looking for known threats are drawing to a close, as they're simply not built to handle the volume and sophistication of attacks today. To truly understand the nature of the threats they face, companies need to move beyond traditional approaches to security and delve deeper into the machine data being generated every second of every day.
Mark Seward is senior director of security and compliance at Splunk
Friday, 19 April 2013
James Grundvig: Making Big Data Small (and Accessible): Interview With SiSense CEO Amit Bendov
Two weeks after the GigaOm Conference on Structure Data in New York, Gerstein Fisher held its annual Real Talk series. The subject of the investment management firm's lecture followed GigaOm's footsteps, as if by design.
"Knowledge Redefined: How Technology is Changing the Way We Learn, What We Know, and How We Invest," centered on the lecture of David Weinberger, a Harvard researcher and bestselling author.
After Weinberger showed a pyramid with four levels of intelligence -- data, information, knowledge, and wisdom at the top -- to the packed Times Center auditorium, he went on a fast-paced tear explaining the eccentricities, broken paradigms, and counter-intuitive insights into three of the four levels.
He punted on the fourth giving up trying to explain, "What is wisdom?"
Like the CIA and its CTO Gus Hunt at the GigaOm Conference, Mr. Weinberger claimed that we should be collecting everything with respect to data. "We are living in a time of inclusion, not curating data. Today, we filter on the way out."
He went on to give an example that even data of reality TV shows on celebrities should be gathered, as what might be thought of nonsense one day, became a Harvard course on celebrities and their influence on fashion and culture last semester.
"Order doesn't scale in the big data era," he said. "Rows and columns don't scale. Agreements don't scale." Weinberger went through a few, often funny, examples of how humans always agree to disagree. "But messy data does scale. It gives meaning."
In mashing up all kinds of data, from polar opposite categories to data sets of different variety, velocity, and volume, the Harvard researcher drove home his point that we should no longer be excluding (censoring) data, but collecting all of it. As Mr. Hunt said at the GigaOm Conference, the CIA will collect and keep all its data storing it in a $600 million Amazon cloud to enable the intelligence agency "to connect the dots" in some unforeseen way for future events.
So how do we make sense of all the terabytes (TB) and petabytes of data that different companies, institutions, and industries are collecting and storing in the cloud today?
A Beacon into Making 'Sense' of Big Data
At the same GigaOm Conference, I came across the answer: SiSense.
In meeting Bruno Aziza, vice president of marketing at SiSense, I found an energetic man excited about the future of data. He believed that his firm, which just closed a $10 million Series B investment with Battery Ventures, has engineered the road out of the dark void of messy data.
Prism™, SiSense's flagship business intelligence (BI) software, is a technology that reinforces both the company's mantra -- "You Don't Have to be a Data Scientist" to mine big data -- and the 10x10x10 challenge for analytics.
Last month, SiSense partnered with AOL to make sense of a decade worth of stored big data in the CrunchBase ecosystem of venture capital and technology startups for free.
In test-driving a demo of CrunchBase through Prism, Mr. Aziza showed the average investment by round for various startups, from mobile to cloud computing. He pulled up different tables, joined them together, then ran the analytics without typing a single key.
"It's very simple to use," he said. "Call up pivot tables, index them. Ask if you want to count it. Auto-create and aggregate the data. With other solutions, when users make changes along the way accumulating evermore data, things breakdown. But that's when SiSense gets smarter."
Once users have the data, they want it in a simple, visual way. "They can share it, export it, publish it, and schedule distribution," Aziza said. "Using CrunchBase, people can mine fifty tables joined from different sources and different systems.
Not only is the BI software useful, but also its user-friendly as all this non-coder had to do was use a laptop glidepad to pull up data and join the tables.
SiSense's 10x10x10 challenge: "10 TB of data in 10 seconds on a $10K machine."
Without the heavy investment in legacy systems, SiSense has made the world of messy data accessible and affordable to the small-to-medium sizes businesses.
Interview with CEO Amit Bendov
In a follow up interview with SiSense CEO Amit Bendov, he made the point that his firm has solved the "Tower of Babel" problem of BI and data analytics of different software programs that don't speak to one another, conflicting pivot tables, and big untenable data. What seemed impossible -- at least for a reasonable price -- to "pull" key data from vast oceans and warehouses of unrelated data in 2012 appears to have turned the corner this year.
"Launching Prism last October, we set out to make data analytics simple with what feels like a monster of complicated models that cost millions to crunch and do it at a fraction of the cost," Bendov said. Probably in a fraction of the time, too.
SiSense's co-founders, Chief Product Officer Elad Israeli and CTO Eldad Farkash, both veterans of Israel's technology hub, started developing the BI software many years ago.
To put that database analytics model into commercial use, Amit Bendov came on board to accelerate growth of the company: "A couple of years ago our growth soared from 300% to 520% last year. We have been very aggressive in achieving our goals and currently we are one month ahead of our latest milestone," he said.
"We are in high demand to keep support people in New York City, where we will soon open a regional office. Our client-base there is growing tremendously as we offer unmatched power, speed, and simplicity compared to IBM, HP, and Oracle. We do a different job than they do. They're very complicated for non-technical people, who can't handle big data or computer storage. Our product is revolutionary," Bendov said.
"Who are your competitors?" I asked.
Without hesitation, Amit Bendov replied, "They are the lower end solutions, from Tableau (which just announced an IPO) to other ediscovery tools and software like Qliktech. We are cheaper to use. But the critical difference, the more users and larger portions of data in a database, the smarter Prism works."
Machine learning isn't new, but for big data that's hard to manage and impossible to pull key analytics easily and at a low cost, this could be an inflection point.
"Data connectivity and visualizing everything comes in a package, to scale. When I did a demo for a non-profit in Washington, DC, they were impressed that a non-techie CEO could demo the product and show them how easy it is to use," he said.
"We are thrilled to add Battery Ventures as an investor. We are experiencing exponential sales growth and incredible buzz in the market -- now it's time to add oil to the fire. Our biggest challenge right now is growing the sales force and the support teams quickly enough to keep up with the demand," said Bendov in a press release.
That isn't the only challenge for SiSense. Getting the word out that their software can add value to information from unthreaded, messy forms of data, and for businesses to make intelligent, actionable decisions may be just as big.
If it all works out, big data may not be as big and imposing as we once thought. That's hard to imagine as each day we send, share, and collect a trillion emails, tweets, millions of images and photographs, and other skynormous troves of data.
Source: www.huffingtonpost.com/james-grundvig/making-big-data-small-and_b_3099551.html
"Knowledge Redefined: How Technology is Changing the Way We Learn, What We Know, and How We Invest," centered on the lecture of David Weinberger, a Harvard researcher and bestselling author.
After Weinberger showed a pyramid with four levels of intelligence -- data, information, knowledge, and wisdom at the top -- to the packed Times Center auditorium, he went on a fast-paced tear explaining the eccentricities, broken paradigms, and counter-intuitive insights into three of the four levels.
He punted on the fourth giving up trying to explain, "What is wisdom?"
Like the CIA and its CTO Gus Hunt at the GigaOm Conference, Mr. Weinberger claimed that we should be collecting everything with respect to data. "We are living in a time of inclusion, not curating data. Today, we filter on the way out."
He went on to give an example that even data of reality TV shows on celebrities should be gathered, as what might be thought of nonsense one day, became a Harvard course on celebrities and their influence on fashion and culture last semester.
"Order doesn't scale in the big data era," he said. "Rows and columns don't scale. Agreements don't scale." Weinberger went through a few, often funny, examples of how humans always agree to disagree. "But messy data does scale. It gives meaning."
In mashing up all kinds of data, from polar opposite categories to data sets of different variety, velocity, and volume, the Harvard researcher drove home his point that we should no longer be excluding (censoring) data, but collecting all of it. As Mr. Hunt said at the GigaOm Conference, the CIA will collect and keep all its data storing it in a $600 million Amazon cloud to enable the intelligence agency "to connect the dots" in some unforeseen way for future events.
So how do we make sense of all the terabytes (TB) and petabytes of data that different companies, institutions, and industries are collecting and storing in the cloud today?
A Beacon into Making 'Sense' of Big Data
At the same GigaOm Conference, I came across the answer: SiSense.
In meeting Bruno Aziza, vice president of marketing at SiSense, I found an energetic man excited about the future of data. He believed that his firm, which just closed a $10 million Series B investment with Battery Ventures, has engineered the road out of the dark void of messy data.
Prism™, SiSense's flagship business intelligence (BI) software, is a technology that reinforces both the company's mantra -- "You Don't Have to be a Data Scientist" to mine big data -- and the 10x10x10 challenge for analytics.
Last month, SiSense partnered with AOL to make sense of a decade worth of stored big data in the CrunchBase ecosystem of venture capital and technology startups for free.
In test-driving a demo of CrunchBase through Prism, Mr. Aziza showed the average investment by round for various startups, from mobile to cloud computing. He pulled up different tables, joined them together, then ran the analytics without typing a single key.
"It's very simple to use," he said. "Call up pivot tables, index them. Ask if you want to count it. Auto-create and aggregate the data. With other solutions, when users make changes along the way accumulating evermore data, things breakdown. But that's when SiSense gets smarter."
Once users have the data, they want it in a simple, visual way. "They can share it, export it, publish it, and schedule distribution," Aziza said. "Using CrunchBase, people can mine fifty tables joined from different sources and different systems.
Not only is the BI software useful, but also its user-friendly as all this non-coder had to do was use a laptop glidepad to pull up data and join the tables.
SiSense's 10x10x10 challenge: "10 TB of data in 10 seconds on a $10K machine."
Without the heavy investment in legacy systems, SiSense has made the world of messy data accessible and affordable to the small-to-medium sizes businesses.
Interview with CEO Amit Bendov
In a follow up interview with SiSense CEO Amit Bendov, he made the point that his firm has solved the "Tower of Babel" problem of BI and data analytics of different software programs that don't speak to one another, conflicting pivot tables, and big untenable data. What seemed impossible -- at least for a reasonable price -- to "pull" key data from vast oceans and warehouses of unrelated data in 2012 appears to have turned the corner this year.
"Launching Prism last October, we set out to make data analytics simple with what feels like a monster of complicated models that cost millions to crunch and do it at a fraction of the cost," Bendov said. Probably in a fraction of the time, too.
SiSense's co-founders, Chief Product Officer Elad Israeli and CTO Eldad Farkash, both veterans of Israel's technology hub, started developing the BI software many years ago.
To put that database analytics model into commercial use, Amit Bendov came on board to accelerate growth of the company: "A couple of years ago our growth soared from 300% to 520% last year. We have been very aggressive in achieving our goals and currently we are one month ahead of our latest milestone," he said.
"We are in high demand to keep support people in New York City, where we will soon open a regional office. Our client-base there is growing tremendously as we offer unmatched power, speed, and simplicity compared to IBM, HP, and Oracle. We do a different job than they do. They're very complicated for non-technical people, who can't handle big data or computer storage. Our product is revolutionary," Bendov said.
"Who are your competitors?" I asked.
Without hesitation, Amit Bendov replied, "They are the lower end solutions, from Tableau (which just announced an IPO) to other ediscovery tools and software like Qliktech. We are cheaper to use. But the critical difference, the more users and larger portions of data in a database, the smarter Prism works."
Machine learning isn't new, but for big data that's hard to manage and impossible to pull key analytics easily and at a low cost, this could be an inflection point.
"Data connectivity and visualizing everything comes in a package, to scale. When I did a demo for a non-profit in Washington, DC, they were impressed that a non-techie CEO could demo the product and show them how easy it is to use," he said.
"We are thrilled to add Battery Ventures as an investor. We are experiencing exponential sales growth and incredible buzz in the market -- now it's time to add oil to the fire. Our biggest challenge right now is growing the sales force and the support teams quickly enough to keep up with the demand," said Bendov in a press release.
That isn't the only challenge for SiSense. Getting the word out that their software can add value to information from unthreaded, messy forms of data, and for businesses to make intelligent, actionable decisions may be just as big.
If it all works out, big data may not be as big and imposing as we once thought. That's hard to imagine as each day we send, share, and collect a trillion emails, tweets, millions of images and photographs, and other skynormous troves of data.
Source: www.huffingtonpost.com/james-grundvig/making-big-data-small-and_b_3099551.html
Will Big Data Mark the End of the Market Research Industry? | SmartData Collective
April 19, 2013
Big data is disrupting the world, society and all industries. One industry that will definitely notice these waves of change is the market research industry. The new big data startups, that are founded everywhere around the world, can be viewed as new, but improved, market research agencies. Consequently, the market research industry as we know it today will cease to exist.
Customers interacting with organisations provide valuable feedback for companies. With big data this feedback, whether provided via the website, via the call centre or by simply using the product, can be turned into market information for product improvements. With big data it is possible to turn every conversation recorded by a call centre into text, which could then be data mined for insights. This would provide very useful content about what and how consumers feel about the brand and the products.
Even more, consumer panels, panels of business managers or focus groups will not be necessary anymore in the world of big data. In an inter-connected world, where products have sensors and are connected to the internet, companies will know in real-time how people use their product, when they use it, for how long they use it and when things go wrong. Whenever a product needs to be improved or a new product needs to be developed, organisations can simply look at the real-time sensor data pouring into the organisation and understand the needs and wishes of their customers. If they want to understand how people think of their products, the company or what the sentiment of the brand is, they can simply connect to a big data startup and real-time data starts appearing on their screen, analysed and visualized, to be understood by everyone in the organisation. A market research organisation that requires a few weeks to create and perform a
questionnaire and another few weeks to analyse the results, is likely to be by-passed in this fast world.
In a blog post by Brigid Kilcoin it is noted though that market research agencies can actually thrive due to big data because market research organisations focus on the ‘why’. She writes that it is only in understanding the context of why something is happening, that an enterprise can drive meaningful change within its organization. According to Kiloin, market research organisations are there to provide an answer to this ‘why’ question. In a world of big data, however, understanding of the context is done automatically by the algorithms. Members of an organisation can use the analyses and visualizations made by the big data tools to under the context of why something is happening.
In a big data society, researchers are consequently not needed anymore to tell a compelling story using visualizations. There are great tools available on the market already that can make these visualizations automatically and deliver them in real-time to the decision-maker.
Gigi DeVault, from about.com, notes that “all theories of human behavior can be tossed out since data analysts no longer need to know why people do what they do. Human behavior can be tracked and measured with “‘unprecedented fidelity.”” Market research organisations will therefore have to adapt to the new rules of the game or they will be by-passed by big data startups who can do the work for market research agencies at a fraction of the costs and in real-time.
It will require market researchers to understand the power of Big Data and to become comfortable with the tools and techniques of Big Data. If they want to survive, they should start hiring big data scientists to be able to deliver the same kind of services as big data startups around the world already offer.
So yes, big data will mark the end of the market research industry as we know it. Unless, of course, they are able to adapt and pivot on time. Otherwise, the big data scientist will replace the market researcher and this will happen faster than one might expect.
Source: http://smartdatacollective.com/bigdatastartups/118931/will-big-data-mark-end-market-research-industry
Big data is disrupting the world, society and all industries. One industry that will definitely notice these waves of change is the market research industry. The new big data startups, that are founded everywhere around the world, can be viewed as new, but improved, market research agencies. Consequently, the market research industry as we know it today will cease to exist.
Customers interacting with organisations provide valuable feedback for companies. With big data this feedback, whether provided via the website, via the call centre or by simply using the product, can be turned into market information for product improvements. With big data it is possible to turn every conversation recorded by a call centre into text, which could then be data mined for insights. This would provide very useful content about what and how consumers feel about the brand and the products.
Even more, consumer panels, panels of business managers or focus groups will not be necessary anymore in the world of big data. In an inter-connected world, where products have sensors and are connected to the internet, companies will know in real-time how people use their product, when they use it, for how long they use it and when things go wrong. Whenever a product needs to be improved or a new product needs to be developed, organisations can simply look at the real-time sensor data pouring into the organisation and understand the needs and wishes of their customers. If they want to understand how people think of their products, the company or what the sentiment of the brand is, they can simply connect to a big data startup and real-time data starts appearing on their screen, analysed and visualized, to be understood by everyone in the organisation. A market research organisation that requires a few weeks to create and perform a
questionnaire and another few weeks to analyse the results, is likely to be by-passed in this fast world.
In a blog post by Brigid Kilcoin it is noted though that market research agencies can actually thrive due to big data because market research organisations focus on the ‘why’. She writes that it is only in understanding the context of why something is happening, that an enterprise can drive meaningful change within its organization. According to Kiloin, market research organisations are there to provide an answer to this ‘why’ question. In a world of big data, however, understanding of the context is done automatically by the algorithms. Members of an organisation can use the analyses and visualizations made by the big data tools to under the context of why something is happening.
In a big data society, researchers are consequently not needed anymore to tell a compelling story using visualizations. There are great tools available on the market already that can make these visualizations automatically and deliver them in real-time to the decision-maker.
Gigi DeVault, from about.com, notes that “all theories of human behavior can be tossed out since data analysts no longer need to know why people do what they do. Human behavior can be tracked and measured with “‘unprecedented fidelity.”” Market research organisations will therefore have to adapt to the new rules of the game or they will be by-passed by big data startups who can do the work for market research agencies at a fraction of the costs and in real-time.
It will require market researchers to understand the power of Big Data and to become comfortable with the tools and techniques of Big Data. If they want to survive, they should start hiring big data scientists to be able to deliver the same kind of services as big data startups around the world already offer.
So yes, big data will mark the end of the market research industry as we know it. Unless, of course, they are able to adapt and pivot on time. Otherwise, the big data scientist will replace the market researcher and this will happen faster than one might expect.
Source: http://smartdatacollective.com/bigdatastartups/118931/will-big-data-mark-end-market-research-industry
Intel CIO Kim Stevenson on big data, OpenStack, women in IT | ZDNet
By Larry Dignan for Between the Lines |
Intel CIO Kim Stevenson, who has been at the helm for a little more than a year, said OpenStack is the most useful cloud architecture for avoiding lock-in, outlined how the chip giant is using big data techniques, and talked capacity planning for her company.
Stevenson, formerly the vice president of global operations and services at Intel, has a dual role at the company. First, she's in charge of the IT operations for Intel globally and has the usual CIO headaches. And second, Stevenson is the guinea-pig-in-chief for Intel's products.
Intel's IT group is typically the "first implementer" for new hardware and software, said Stevenson. In other words, Intel eats its own dog food and gives product people honest feedback on what needs to happen. Stevenson quipped that the conversations from an IT customer perspective don't always go well.
Here are a few key highlights from a breakfast conversation in New York:
Big data. One area where Intel is the first implementer is on its Hadoop distribution. Like other large enterprises, Stevenson's group is looking toward big data and analytics to boost revenue, improve time to market and cut costs. Here are Stevenson's biggest takeaways on using big data internally.
Here's how the cloud fits in with Intel's key areas: Office applications, silicon design and manufacturing.
Beyond design and manufacturing, though, Stevenson requires that an application request has to prove why it needs physical resources. In other words, applications need to be virtualized or they won't get capacity.
Capacity planning. Stevenson said the goal of Intel is to have 15 days of capacity on hand. This 15 day rule means that if the typical usage occurs within Intel with the usual new requests, the company will be out of compute in 15 days. For Stevenson, compute capacity is her inventory. The 15-day capacity cushion is based on memory mostly since that's the first to go when resources are being taxed. Compute and storage are also watched. Stevenson acknowledges that the 15-day capacity best practice may not work for all companies and could even be shorter. "Some companies may have to convince the CFO that having 15 days of excess capacity sitting around is a good idea," said Stevenson. "We didn't have to convince our CFO of that."
Playing the cloud provider game. Stevenson said Intel has gone with OpenStack, a cloud architecture that's gaining momentum. Why? Intel uses multiple cloud providers working under a master services agreement and wants to hop between them to maximize performance and costs. "We use several cloud providers mostly in the U.S.," she said. "We've had no issues with SLAs, but we're only bursting. We're careful with what workloads we use in the public cloud." OpenStack enables Intel to use multiple providers and avoid lock-in, said Stevenson.
About those cloud brokers. I asked Stevenson what she thought of cloud brokers---companies that would manage providers to maximize savings. "Cloud brokers are transitory. That dog doesn't hunt as an independent business model," she said. Ultimately, the ability to hop between cloud providers will be built into an infrastructure like OpenStack and automated.
Women in IT. Finally, Stevenson talked a bit about women in IT and was asked about Sheryl Sandberg's Lean In: Women, Work, and the Will to Lead, which drew some fire over what the Facebook operating chief had to say. Stevenson said the book was largely on target and added that Sandberg had good advice for anyone in business not just women. Her main takeaway on women in IT is that they have to understand the communication differences in a male dominated industry. In other words, know how to navigate the subcultures such as sports chatter and international differences. Also use good context at the start of a business problem to navigate functional and relational communication styles.
Source: http://www.zdnet.com/intel-cio-kim-stevenson-on-big-data-openstack-women-in-it-7000014221/
Intel CIO Kim Stevenson, who has been at the helm for a little more than a year, said OpenStack is the most useful cloud architecture for avoiding lock-in, outlined how the chip giant is using big data techniques, and talked capacity planning for her company.
Stevenson, formerly the vice president of global operations and services at Intel, has a dual role at the company. First, she's in charge of the IT operations for Intel globally and has the usual CIO headaches. And second, Stevenson is the guinea-pig-in-chief for Intel's products.
Intel's IT group is typically the "first implementer" for new hardware and software, said Stevenson. In other words, Intel eats its own dog food and gives product people honest feedback on what needs to happen. Stevenson quipped that the conversations from an IT customer perspective don't always go well.
Here are a few key highlights from a breakfast conversation in New York:
Big data. One area where Intel is the first implementer is on its Hadoop distribution. Like other large enterprises, Stevenson's group is looking toward big data and analytics to boost revenue, improve time to market and cut costs. Here are Stevenson's biggest takeaways on using big data internally.
- Talent: "Whether developing or implementing big data, talent is in short supply," said Stevenson. She added that Intel is working university partnerships and programs to get talent, but it takes time to turn interns into big data visionaries. Intel is also training employees, but noted that the transition to big data can be difficult. "It's not easy for a database person who thinks in rows and columns to learn Hadoop," said Stevenson. "It's also difficult for business people because they usually want clean data."
- Time to market is the big data win: Intel's biggest goal with big data and analytics "is to reduce the product cycle time," said Stevenson. After all, Intel is entering the smartphone market and the product cycles are twice as fast as PCs. To speed up product development time, Intel looked at the post silicon validation process. This process revolves around making sure that all of the transistors---all 1 billion or so---are functioning properly. The data out of a silicon wafer equates to 750 terabytes. Intel deployed a machine learning model that isolates testing and will cut 25 percent of a five quarter product cycle period.
- Manufacturing: Big data is also used to cut costs in manufacturing by analyzing silicon wafers and how they are cut. The goal is to have less waste and more processors for better yields. Hadoop and Intel's analytics stack -- which consists of IBM's Netezza, Hadoop and proprietary algorithms -- logs information, finds errors and best practices and passes the data along to the next tool. If each step improves, there are better good die yields and lower unit costs, explained Stevenson.
- Marketing fraud: Big data is also used to detect fraud and errors in the chip maker's Intel Inside marketing. In a nutshell, Intel partners market Intel Inside and get reimbursed by the company on their marketing spend. Stevenson said that Intel used its machine learning engine to look at fraudulent claims, mistakes and incomplete reimbursement requests. Intel also looked to cut down on manual post audits. The return was $7 million in savings in a quarter and the expulsion of bad guys. Reimbursement time also improved.
Here's how the cloud fits in with Intel's key areas: Office applications, silicon design and manufacturing.
- Typical enterprise applications: According to Stevenson, 75 percent of Intel's typical office systems such as email and ERP are in the cloud in a private-public hybrid model. "In 2012, we were bursting to other Intel regions and now we burst out to other providers," she said. One public cloud use case is Will.I.Am. The artist can offer a free download on Intel.com and get a ton of Web traffic. That use case is perfect for the public cloud.
- Design: "Silicon design will never go out to the cloud. That's our core IP," said Stevenson. She added that no cloud service level agreement or chargeback would ever compensate for Intel's intellectual property being leaked. Instead, product design runs on a high performance computing grid that's internal.
- Manufacturing: Manufacturing is another area that won't be put into the cloud. The information is housed in small data centers near the manufacturing site and later aggregated.
Beyond design and manufacturing, though, Stevenson requires that an application request has to prove why it needs physical resources. In other words, applications need to be virtualized or they won't get capacity.
Capacity planning. Stevenson said the goal of Intel is to have 15 days of capacity on hand. This 15 day rule means that if the typical usage occurs within Intel with the usual new requests, the company will be out of compute in 15 days. For Stevenson, compute capacity is her inventory. The 15-day capacity cushion is based on memory mostly since that's the first to go when resources are being taxed. Compute and storage are also watched. Stevenson acknowledges that the 15-day capacity best practice may not work for all companies and could even be shorter. "Some companies may have to convince the CFO that having 15 days of excess capacity sitting around is a good idea," said Stevenson. "We didn't have to convince our CFO of that."
Playing the cloud provider game. Stevenson said Intel has gone with OpenStack, a cloud architecture that's gaining momentum. Why? Intel uses multiple cloud providers working under a master services agreement and wants to hop between them to maximize performance and costs. "We use several cloud providers mostly in the U.S.," she said. "We've had no issues with SLAs, but we're only bursting. We're careful with what workloads we use in the public cloud." OpenStack enables Intel to use multiple providers and avoid lock-in, said Stevenson.
About those cloud brokers. I asked Stevenson what she thought of cloud brokers---companies that would manage providers to maximize savings. "Cloud brokers are transitory. That dog doesn't hunt as an independent business model," she said. Ultimately, the ability to hop between cloud providers will be built into an infrastructure like OpenStack and automated.
Women in IT. Finally, Stevenson talked a bit about women in IT and was asked about Sheryl Sandberg's Lean In: Women, Work, and the Will to Lead, which drew some fire over what the Facebook operating chief had to say. Stevenson said the book was largely on target and added that Sandberg had good advice for anyone in business not just women. Her main takeaway on women in IT is that they have to understand the communication differences in a male dominated industry. In other words, know how to navigate the subcultures such as sports chatter and international differences. Also use good context at the start of a business problem to navigate functional and relational communication styles.
Source: http://www.zdnet.com/intel-cio-kim-stevenson-on-big-data-openstack-women-in-it-7000014221/
Subscribe to:
Posts (Atom)