Business intelligence has gone through multiple iterations in the past few decades. While BI's evolution has addressed some of the technology and process shortcomings of the earlier management information systems, BI teams still face challenges. Enterprises are transforming only 40% of their structured data and 31% of their unstructured data into information and insights. In addition, 63% of organizations still use spreadsheet-based applications for more than half of their decisions. Many earlier and current enterprise BI deployments:
Have hit the limits of scalability.
Struggle to address rapid changes in customer and regulatory requirements.
Fail to break through waterfall's design limitations.
Suffer from mismatched business and technology priorities and languages.
When I think about data, I can't help but think about hockey. As a passionate hockey mom, it's hard to separate my conversations about data all week with clients from the practices and games I sit through, screaming encouragement to my son and his team (sometimes to the embarrassment of my husband!). So when I recently saw a documentary on the building of the Russian hockey team that our miracle US hockey team beat at the 1980 Olympics, the story of Anatoli Tarsov stuck with me.
Before the 1960s, Russia didn't have a hockey team. Then the Communist party determined that it was critical that Russia build one — and compete on the world stage. They selected Anatoli Tarsov to build the team and coach. He couldn't see films on hockey. He couldn't watch teams play. There was no reference on how to play the game. And yet, he built a world-class hockey club that not only beat the great Nordic teams but went on to crush the Canadian teams that were the standard for hockey excellence.
This is a lesson for us all when it comes to data. Do we stick with our standards and recipes from Inmon and Kimball? Do we follow check-box assessments from CMMI, DM-BOK, or TOGAF's information architecture framework? Do we rely on governance compliance to police our data?
Or do we break the rules and create our own that are based on outcomes and results? This might be the scarier path. This might be the riskier path. But do you want data to be where your business needs it, or do you want to predefine, constrain, and bias the insight?
China Unicom demonstrated its big data analytics platform, including customer analytics, during the Shanghai World Mobile Congress last week. Huawei is helping China Unicom’s Shanghai affiliate build a big data analytics platform that can collect and analyze customer demographics and operational and behavioral data. For instance, it can estimate a consumer’s monthly income based on annual mobile fees, know whether she is walking or driving, and what routes she regularly takes. Such data is unique and even more comprehensive than that generated by Internet service giants like Baidu, Alibaba, or Tencent. China Unicom will begin to leverage this data analytics platform to monetize data in several ways:
Retain customers. China Unicom can predict which high-value customers may be thinking of dropping its services and target marketing based on the customers’ context to retain them. For instance, the system can automatically send a targeted offering when a customer passes by a China Unicom office. In 2014, the telco performed A/B testing among 200,000 Unicom subscribers in Shanghai who were thinking of changing telecom service providers. This helped China Unicom retain RMB 10 million in revenue from those users receiving targeted marketing offerings from the system.
Enhance public security. China Unicom uses the platform to help the Shanghai city government to monitor people’s location in the city in real time. The system provides a real-time heat map and automatically sends an alert when it discovers that too many people are crowded into one area. This can help the government avoid accidents such as the one that occurred in the Waitan district of Shanghai last year.
It is easy to get ahead of ourselves with all the innovation happening with data and analytics. I wouldn't call it hype, as that would imply no value or competency has been achieved. But I would say that what is bright, shiny, and new is always more interesting than the ordinary.
And, to be frank, there is still a lot of ordinary in our data management world.
In fact, over the past couple of weeks, discussions with companies have uncommonly focused on the ordinary. This in some ways appeared to be unusual because questions focused on the basic foundational aspects of data management and governance — and for companies that I have seen talk publicly about their data management successes.
"Where do I clean the data?"
"How do I get the business to invest in data?"
"How do I get a single customer view of my customer for marketing?"
What this tells me is that companies are under siege by zombie data.
Data is living in our business under outdated data policies and rules. Data processes and systems are persisting single-purpose data. As data pros turn over application rocks and navigate through the database bogs to centralize data for analytics and virtualize views for new data capabilities, zombie data is lurching out to consume more of the environment, blocking other potential insight to keep the status quo.
The questions you and your data professional cohorts are asking, as illustrated above, are anything but basic. The fact that these foundational building blocks have to be assessed once again demonstrates that organizations are on a path to crush the zombie data siege, democratize data and insight, and advance the business.
Keep asking basic questions — if you aren't, zombie data will eventually take over, and you and your organization will become part of the walking dead.
You've probably heard about the Quantified Self (QS), a movement that aims to capture, analyze, and act upon data from the human body in the interest of better health, fitter athletes, and sharper minds. Today, QS is giving way to QW -- Quantified Workforce. A variety of technologies -- devices, software, services -- can quantify the health, fitness, mental acuity, timeliness, and collaboration of workers. Many of these services are ready for prime time, but present some challenges in implementing. These challenges aren't primarily technological; they're related to privacy, workers' rights, and human resources policies. Done right, though, quantifying the workforce can drive both top- and bottom- line growth in your company's business.
I've analyzed this trend in a new report, Smart Body, Smarter Workforce. Here are just a couple of examples of how quantifying the workforce can drive better business outcomes:
Lower the company's insurance rates. In January, 2014, Forrester predicted that insurance companies would offer lower rates to individuals who donned wearables -- and we are now seeing that response. In April, 2015, John Hancock announced an opportunity for buyers of its term and life insurance policies to earn up to 15% discount on their insurance rates by wearing a Fitbit, sharing the data with the company, and meeting certain activity levels.
In scanning through my O’Reilly Data Newsletter today, I noticed A Healthy Dose of Data, an MIT Sloan case study on the data and analytics culture at Intermountain, a healthcare network that runs 22 hospitals and 185 clinics. The study is definitely worth the read. It reviews the history of data use at Intermountain, which began way before the “big data” craze of recent years. In fact, it was back in the 1950s that one of the Intermountain cardiologists, Homer Warner, began to explore clinical data to understand why some heart patients experienced better outcomes than others. He went on to become known as the “father of medical informatics – the use of computer programs to analyze patient data to determine treatment protocols,” and with colleagues designed and launched their first decision-support tool.
The case study goes on to describe how Intermountain has cultivated a strong data and analytics culture. Over time – Rome was not built in a day, as they say – they established data maturity across the organization by investing in the capacity (new tools and technologies), developing the competencies (new skills and processes) and finally spreading the culture (awareness, understanding and best practices) of data and analytics. Their analytical approach brought results – fewer surgical infections, more effective use of antibiotics, less time in intensive care etc – contributing to lower costs, better medical outcomes, and overall patient satisfaction.
Big data and Hadoop (Yellow Elephants) are so synonymous that you can easily overlook the vast landscape of architecture that goes into delivering on big data value. Data scientists (Pink Unicorns) are also raised to god status as the only real role that can harness the power of big data -- making insights obtainable from big data as far away as a manned journey to Mars. However, this week, as I participated at the DGIQ conference in San Diego and colleagues and friends attended the Hadoop Summit in Belgium, it has become apparent that organizations are waking up to the fact that there is more to big data than a "cool" playground for the privileged few.
The perspective that the insight supply chain is the driver and catalyst of actions from big data is starting to take hold. Capital One, for example, illustrated that if insights from analytics and data from Hadoop were going to influence operational decisions and actions, you need the same degree of governance as you established in traditional systems. A conversation with Amit Satoor of SAP Global Marketing talked about a performance apparel company linking big data to operational and transactional systems at the edge of customer engagement and that it had to be easy for application developers to implement.
Hadoop distribution, NoSQL, and analytic vendors need to step up the value proposition to be more than where the data sits and how sophisticated you can get with the analytics. In the end, if you can't govern quality, security, and privacy for the scale of edge end user and customer engagement scenarios, those efforts to migrate data to Hadoop and the investment in analytic tools cost more than dollars; they cost you your business.
Often considered the poster child of digital transformation, APIs are proliferating at enterprises making industry-leading investments in mobile, IoT, and big data. As these initiatives mature, CIOs, CTOs, and heads of development are coming together with business leaders to manage and secure companywide use of APIs using API management solutions.
Forrester recently released a report that sizes and projects annual spending on API management solutions. We predict US companies alone will spend nearly $3 billion on API management over the next five years. Annual spend will quadruple by the end of the decade, from $140 million in 2014 to $660 million in 2020. International sales will take the global market over the billion dollar mark.
In interviewing vendors for this piece of research, we discovered a vast and fertile landscape of participants:
Startups have taken $430 million in venture funding, and so far have realized $335 million in acquisition value. In April 2015, pure-play vendor Apigee went IPO and currently trades at a valuation north of $400 million.
Beware of insights! Real danger lurks behind the promise of big data to bring more data to more people faster, better, and cheaper: Insights are only as good as how people interpret the information presented to them. When looking at a stock chart, you can't even answer the simplest question — "Is the latest stock price move good or bad for my portfolio?" — without understanding the context: where you are in your investment journey and whether you're looking to buy or sell. While structured data can provide some context — like checkboxes indicating your income range, investment experience, investment objectives, and risk tolerance levels — unstructured data sources contain several orders of magnitude more context. An email exchange with a financial advisor indicating your experience with a particular investment vehicle, news articles about the market segment heavily represented in your portfolio, and social media posts about companies in which you've invested or plan to invest can all generate much broader and deeper context to better inform your decision to buy or sell.
But defining the context by finding structures, patterns, and meaning in unstructured data is not a simple process. As a result, firms face a gap between data and insights; while they are awash in an abundance of customer and marketing data, they struggle to convert this data into the insights needed to win, serve, and retain customers. In general, Forrester has found that:
The problem is not a lack of data. Most companies have access to plenty of customer feedback surveys, contact center records, mobile tracking data, loyalty program activities, and social media feeds — but, alas, it's not easily available to business leaders to help them make decisions.
What’s the top imperative at your company? If it’s not a transformation to make the company more customer-focused, you’re making a mistake. Technology and economic forces have changed the world so much that an obsession with winning, serving, and retaining customers is the only possible response.
We’re in an era of persistent economic imbalances defined by erratic economic growth, deflationary fears, an oversupply of labor, and surplus capital hunting returns in a sea of record-low interest rates. This abundance of capital and labor means that the path from good idea to customer-ready product has never been easier, and seamless access to all of the off-the-shelf components needed for a startup fuels the rise of weightless companies, which further intensify competition.
Chastened by a weak economy, presented with copious options, and empowered with technology, consumers have more market muscle than ever before. The information advantage tips to consumers with ratings and review sites. They claim pricing power by showrooming. And the only location that matters is the mobile phone in their hand from which they can buy anything from anyone and have it delivered anywhere.
This customer-driven change is remaking every industry. Cable and satellite operators lost almost 400,000 video subscribers in 2013 and 2014 as customers dropped them for the likes of Netflix. Lending Club, an alternative to commercial banks, has facilitated more than $6 billion in peer-to-peer loans. Now that most B2B buyers would rather buy from a website than a salesperson, we estimate that 1 million B2B sales jobs will disappear in the coming years.