Big Data Will Help Shape Your Market’s Next Big Winners

It seems that every week another vendor slaps “big data” into its marketing material – and it’s going to get worse. Should you look beyond the vendor hype and pay attention? Absolutely yes! Why? Because big data has the potential to shape your market’s next winners and losers.

At Forrester, we think clients must develop an intuitive understanding of big data by learning: 1) what is new about it; 2) what it is; and 3) how it will influence their market.

What is new about big data? We estimate that firms effectively utilize less than 5% of available data. Why so little? The rest is simply too expensive to deal with. Big data is new because it lets firms affordably dip into that other 95%. If two companies use data with the same effectiveness but one can handle 15% of available data and one is stuck at 5%, who do you think will win? The deal, however, is that big data is not like your traditional BI tools; it will require new processes and may totally redefine your approach to data governance.

So what is big data? In our CIO document on big data, we provide a deceptively simple answer: It is techniques and technologies that make handling data at extreme scale affordable. Put less formally, it provides affordable technology to people that understand how to get value from the 95% you are not thinking about right now. It’s also not just about big volume, either – other factors like velocity, variety, and variability are also important. In fact, when you just have one of these, like high volume, your existing technology may do the trick. It’s when you get several characteristics together that big data becomes attractive. Examples include customer call records, social graph data, point of sale transactions, and medical monitoring equipment data. Not only is it possible to collect a large volume of this stuff, but the data is usually is generated so fast that you need to constantly capture more of it to be valuable for some decisions.

How will big data influence your market? Every once in a while, a company will do something that rapidly elevates it to the top of its market. Walmart optimized its supply chain, drove down prices, and smacked its competition. Google, Yahoo, and Facebook used big data to deal with web scale search, content relevance, and social connections, and we see what happened to those markets. If you are not thinking about how to leverage big data to get the value from the other 95%, your competition is. In this information age, the firms that best turn information to their advantage will dominate their competition. And big data will play a big part in helping them do it.

Read the report “Expand Your Digital Horizon With Big Data.” In it, we help IT executives hone their intuition about the importance of big data and provide pragmatic advice that goes beyond just the technology.



Big data opens up the new world of exploration

Brian, great points. Another key one is that while traditional BI and DW are all about reporting and analysis (where you have specific questions in mind, and these questions require structures, such as data models), big data technology allows you to explore, ask questions without any preconcieved notions. It's a huge paradigm shift!

Thank Boris - I was Also Reminded To Share A Few Findings

Rather than update the post, here are a few of our findings from 19 interviews with subject matter experts and a survey of 60 clients with some knowledge of big data:
1. Three-quarters of firms are interested for big volume, but high velocity and variety were confirmed as important. What was interesting was that second to high volume was an interest for analysis driven BI requirements discovery. It's what Boris mentioned about - discovery of patterns and value that lead to requirements for traditional BI solutions.
2. About one third of firms use different project management and/or SDLC processes for big data, while about the same number aren't sure what approach they are going to use. Conclusion - your existing processes for solution delivery may not work because of the need for agility and business led solutions. If sucessful, your business is going to want control of big data tools, and we think IT should give it to them in the right circumstances.
3. Speaking of that, we found about 1 in 5 respondents indicating that big data was a pure business initiative with no IT support at all. For enterprises, we think IT has a role to play in big data, but it may not be as a solution delivery organization. You may just focus on putting the right tools and data in the hands of the business people who know what to do with it, especially if those business people are "data scientists".
4. Big data tools are still early release, consisting mostly of pre 1.0 releases of open source and vendors madly scrambling to incorporate big data into their tool sets. We think this is going to get better in the near future, but now just yet. Organizations diving into big data must be prepared, and really understand the business value that will make the pain worth it.
5. We identified five big data patterns - predictive analytics and maching learning, operational prediction, dirty ODSs, bulk transformation and extreme ETL, and lastly, stream and event processing. The last one is important as we include capabilities provided by Apache S4 and IBM Streams in the big data mix.
How to you respond to big data? we are recommending CIOs take a pragmatic approach that considers more than just the technology. Big data first requires the right people. It will also have a big impact on your data governance, regulatory and compliance measures. Lastly, CIO's need further develop trusted partnerships with their business to ensure their respective roles are clearly defined.

I'm skeptical

Sounds a lot like other fads. Conceptually: really neat. Practical: NOT. For every Google and WalMart, there are 1,000 firms just looking for the next foothold to inch up the mountain. It takes lots of time, money and people to exploit 3x as much data, and there are so many examples of lower hanging fruit.

In my last company, we were constantly striving to get more/better data to analyze lots of interesting things, make pricing more efficient, etc. But, quite frankly, we had HR issues, we couldn't hire, our compensation plans were a mess, our process was hideously inefficient and plagued with quality problems, we didn't have any training, the culture was self destructive, we were in the middle of a system transition that was 4 months behind schedule, we had a backlog of systems requests that stretched more than a year into the future, etc. And you know what? We were still making piles of money. There was so much low hanging fruit that we would spend the next 4 years just picking it up off the ground.

Far flung ideas like this feel like the realm of guys who write white papers, not guys who are making companies work.

Your competition just breathed a sigh of relief!

We are seeing lots of examples from non-web 2.0 companies that are achieving real world success using 'big data' techniques and technologies. Just met with several this week. Our Business Process role is about to release a set of case studies that provide some details.
Agree that the term is hyped, but we encourage our clients to see past the hype to what's fundamentally changed, and that is the problem that needs to be solved - the new problem is how to stay competitive in a world where data is changing too fast use only the techniques and technologies we know.
While skeptics are busy doubting, our data suggests that leaders are separating the hype from reality and executing pragmatic strategies to do more with more data. These companies will win, in our opinion.

The point is you need fewer HR resources to tap "Big Data"

When you use software automation to harness and make sense of "big data" (or if that term rubs you the wrong way) leverage large data sets, you will most likely be able to reduce the HR impact in terms of additional employees, because automation helps you eliminate most, if not all, of the manual processes and ad hoc systems you are accustomed to using.

Software automation lets you tap large data sets, in real-time, and analyze them in ways that haven't been possible before. All while getting a leg up on the competition and increasing productivity. This is just one of the benefits we bring to our channel-driven customers.

Panta rei

Current messy, expensive, low yield data processing technologies ( RDBMS) are couple of decades old - it is just matter of time before better, cheaper, more efficient ways of data processing arrive. Big Data movement coupled with Cloud technologies may just be it. No/slow adopters might fall behind irreversibly.