During a recent webinar on big data, several listeners wanted to know what the biggest stumbling blocks and reasons for failure were when it comes to big data projects, and what they could do to avoid them. Given the amount of resonance, in particular the top issue I cited, I thought I’d share it in this blog post. Please let me have your views and comments.
There are clearly many reasons why projects struggle or fail, and big data projects are no exception. What can put big data initiatives in a league of their own, though, is the level of (typically unrealistic) expectations often associated with “big data” technologies. Based on many conversations with clients, consultants, and conference delegates over the past couple of years, I find three key issues are being mentioned time and again. These are:
Not starting the project with a question
Underestimating the technical skills and expertise required
Why? What organization couldn’t benefit from making better decisions? Just ask the Obama campaign, which used sophisticated uplift modeling to target and influence swing voters. Or telecom firms that use predictive analytics to help prevent customer churn. Or police departments that use it to reduce crime. The list goes on and on and on. Virtually every organization could benefit from predictive analytics. Don’t confuse traditional business intelligence (BI) with predictive analytics. BI is about reports, dashboards, and advanced visualizations (which are still essential to every organization). Predictive is different. Predictive analytics uses machine learning algorithms on large and small data sets alike to predict outcomes. But predictive is not about absolutes; it doesn’t gaurentee an outcome. Rather, it’s about probabilities. For example, there is a 76% chance that this person will click on this display ad. Or there is a 63% chance that this customer will buy at a certain price. Or there is an 89% chance that this part will fail. Good stuff, but it’s hard to understand and harder to do. It’s worth it, though: Organizations that employ predictive analytics can dramatically reduce risk, disrupt competitors, and save tons of dough. Many are doing it now. More want to.
Few understand the what, why, and how of predictive analytics. Here’s a short, ordered reading list designed to get you up to speed super fast: