Business intelligence has gone through multiple iterations in the past few decades. While BI's evolution has addressed some of the technology and process shortcomings of the earlier management information systems, BI teams still face challenges. Enterprises are transforming only 40% of their structured data and 31% of their unstructured data into information and insights. In addition, 63% of organizations still use spreadsheet-based applications for more than half of their decisions. Many earlier and current enterprise BI deployments:
Have hit the limits of scalability.
Struggle to address rapid changes in customer and regulatory requirements.
Fail to break through waterfall's design limitations.
Suffer from mismatched business and technology priorities and languages.
In scanning through my O’Reilly Data Newsletter today, I noticed A Healthy Dose of Data, an MIT Sloan case study on the data and analytics culture at Intermountain, a healthcare network that runs 22 hospitals and 185 clinics. The study is definitely worth the read. It reviews the history of data use at Intermountain, which began way before the “big data” craze of recent years. In fact, it was back in the 1950s that one of the Intermountain cardiologists, Homer Warner, began to explore clinical data to understand why some heart patients experienced better outcomes than others. He went on to become known as the “father of medical informatics – the use of computer programs to analyze patient data to determine treatment protocols,” and with colleagues designed and launched their first decision-support tool.
The case study goes on to describe how Intermountain has cultivated a strong data and analytics culture. Over time – Rome was not built in a day, as they say – they established data maturity across the organization by investing in the capacity (new tools and technologies), developing the competencies (new skills and processes) and finally spreading the culture (awareness, understanding and best practices) of data and analytics. Their analytical approach brought results – fewer surgical infections, more effective use of antibiotics, less time in intensive care etc – contributing to lower costs, better medical outcomes, and overall patient satisfaction.
Beware of insights! Real danger lurks behind the promise of big data to bring more data to more people faster, better, and cheaper: Insights are only as good as how people interpret the information presented to them. When looking at a stock chart, you can't even answer the simplest question — "Is the latest stock price move good or bad for my portfolio?" — without understanding the context: where you are in your investment journey and whether you're looking to buy or sell. While structured data can provide some context — like checkboxes indicating your income range, investment experience, investment objectives, and risk tolerance levels — unstructured data sources contain several orders of magnitude more context. An email exchange with a financial advisor indicating your experience with a particular investment vehicle, news articles about the market segment heavily represented in your portfolio, and social media posts about companies in which you've invested or plan to invest can all generate much broader and deeper context to better inform your decision to buy or sell.
But defining the context by finding structures, patterns, and meaning in unstructured data is not a simple process. As a result, firms face a gap between data and insights; while they are awash in an abundance of customer and marketing data, they struggle to convert this data into the insights needed to win, serve, and retain customers. In general, Forrester has found that:
The problem is not a lack of data. Most companies have access to plenty of customer feedback surveys, contact center records, mobile tracking data, loyalty program activities, and social media feeds — but, alas, it's not easily available to business leaders to help them make decisions.
Retaining and delighting empowered customers requires continuous, technology-enabled innovation and improved customer insight (CI). The logic is simple in theory, but that doesn’t make it any easier to implement in practice.
In my recent report, entitled “Applying Customer Insight To Your Digital Strategy”, I highlight the top lessons learned from organizations in Asia Pacific (AP) that are successfully leveraging CI to fuel digital initiatives. It all starts by ensuring that data-driven decision-making is central to the digital strategy. With that in mind, I want to use this blog post to focus on two key lessons from the report:
Lesson One: Establish A Clear Mandate To Invest In Customer Analytics
Successful companies serve empowered customers in the way they want to be served, not the way the company wants to serve them. When building a mandate you should:
■ Expect natural tensions between various business stakeholders to arise. To secure buy-in from senior business decision-makers, start by illustrating the clear link between digital capabilities and data as a source of improved customer understanding. Identify measurable objectives and then link them to three to four scenarios that highlight where the biggest opportunities and risks exist. Continue to justify data-related investments by restating these scenarios at regular intervals.
Intel has made no secret of its development of the Xeon D, an SOC product designed to take Xeon processing close to power levels and product niches currently occupied by its lower-power and lower performance Atom line, and where emerging competition from ARM is more viable.
The new Xeon D-1500 is clear evidence that Intel “gets it” as far as platforms for hyperscale computing and other throughput per Watt and density-sensitive workloads, both in the enterprise and in the cloud are concerned. The D1500 breaks new ground in several areas:
It is the first Xeon SOC, combining 4 or 8 Xeon cores with embedded I/O including SATA, PCIe and multiple 10 nd 1 Gb Ethernet ports.
It is the first of Intel’s 14 nm server chips expected to be introduced this year. This expected process shrink will also deliver a further performance and performance per Watt across the entire line of entry through mid-range server parts this year.
Why is this significant?
With the D-1500, Intel effectively draws a very deep line in the sand for emerging ARM technology as well as for AMD. The D1500, with 20W – 45W power, delivers the lower end of Xeon performance at power and density levels previously associated with Atom, and close enough to what is expected from the newer generation of higher performance ARM chips to once again call into question the viability of ARM on a pure performance and efficiency basis. While ARM implementations with embedded accelerators such as DSPs may still be attractive in selected workloads, the availability of a mainstream x86 option at these power levels may blunt the pace of ARM design wins both for general-purpose servers as well as embedded designs, notably for storage systems.
The business has an insatiable appetite for data and insights. Even in the age of big data, the number one issue of business stakeholders and analysts is getting access to the data. If access is achieved, the next step is "wrangling" the data into a usable data set for analysis. The term "wrangling" itself creates a nervous twitch, unless you enjoy the rodeo. But, the goal of the business isn't to be an adrenalin junky. The goal is to get insight that helps them smartly navigate through increasingly complex business landscapes and customer interactions. Those that get this have introduced a softer term, "blending." Another term dreamed up by data vendor marketers to avoid the dreaded conversation of data integration and data governance.
The reality is that you can't market message your way out of the fundamental problem that big data is creating data swamps even in the best intentioned efforts. (This is the reality of big data's first principle of a schema-less data.) Data governance for big data is primarily relegated to cataloging data and its lineage which serve the data management team but creates a new kind of nightmare for analysts and data scientist - working with a card catalog that will rival the Library of Congress. Dropping a self-service business intelligence tool or advanced analytic solution doesn't solve the problem of familiarizing the analyst with the data. Analysts will still spend up to 80% of their time just trying to create the data set to draw insights.
There's never been a question on the advantages of open source software. Crowdsourcing, vendor independence, ability to see and in some cases control the source code, and lower costs are just a few benefits of open source software (OSS) and business model. Linux and Apache Hadoop are prime examples of successful OSS projects. It's a different story, however, when it comes to OSS BI. For years, OSS BI vendors struggled with growth because of:
The developer-centric nature of open source projects. The target audience for open source projects is developers, which means deals are mostly sealed by technology management. The industry, on the other hand, has gravitated toward business decision-makers within organizations over the last several years. However, business users are less interested in the opportunities that a collaborative open source community offers, and more concerned about ease of use and quick setup. Indeed, Forrester's research constantly finds evidence correlating business ownership as one of the key success factors for effective BI initiatives.
The battle of trying to apply traditional waterfall software development life-cycle (SDLC) methodology and project management to BI has already been fought — and largely lost. These approaches and best practices, which apply to most other enterprise applications, work well in some cases, as with very well-defined and stable BI capabilities like tax or regulatory reporting. Mission-critical, enterprise-grade BI apps can also have a reasonably long shelf life of a year or more. But these best practices do not work for the majority (anecdotally, about three-quarters) of BI initiatives, where requirements change much faster than these traditional approaches can support; by the time a traditional BI application development team rolls out what it thought was a well-designed BI application, it's too late. As a result, BI pros need to move beyond earlier-generation BI support organizations to:
Focus on business outcomes, not just technologies. Earlier-generation BI programs lacked an "outcomes first" mentality. Those programs employed bottom-up approaches that focused on the project management and technology first, leaving clients without the proper outcomes that they needed to manage the business; in other words, they created an insights-to-action gap. BI pros should use a top-down approach that defines key performance indicators, metrics, and measures that support the business' goals and objectives. They must resist the temptation to address technology and data needs before the business requirements.
When you hear the term fast data the first thought is probably the velocity of the data. Not unusual in the realm of big data where velocity is one of the V's everyone talked about. However, fast data encompasses more than a data characteristic, it is about how quickly you can get and use insight.
Working with Noel Yuhanna on an upcoming report on how to develop your data management roadmap, we found speed was a continuous theme to achieve. Clients consistently call out speed as what holds them back. How they interpret what speed means is the crux of the issue.
Technology management thinks about how quickly data is provisioned. The solution is a faster engine - in-memory grids like SAP HANA become the tool of choice. This is the wrong way to think about it. Simply serving up data with faster integration and a high performance platform is what we have always done - better box, better integration software, better data warehouse. Why use the same solution that in a year or two runs against the same wall?
The other side of the equation is that sending data out faster ignores what business stakeholders and analytics teams want. Speed to the business encompasses self-service data acquisition, faster deployment of data services, and faster changes. The reason, they need to act on the data and insights.
The right strategy is to create a vision that orients toward business outcomes. Today's reality is that we live in a world where it is no longer about first to market, we have to be about first to value. First to value with our customers, and first to value with our business capabilities. The speed at which insights are gained and ultimately how they are put to use is your data management strategy.
Last year I published a reasonably well-received research document on Hadoop infrastructure, “Building the Foundations for Customer Insight: Hadoop Infrastructure Architecture”. Now, less than a year later it’s looking obsolete, not so much because it was wrong for traditional (and yes, it does seem funny to use a word like “traditional” to describe a technology that itself is still rapidly evolving and only in mainstream use for a handful of years) Hadoop, but because the universe of analytics technology and tools has been evolving at light-speed.
If your analytics are anchored by Hadoop and its underlying map reduce processing, then the mainstream architecture described in the document, that of clusters of servers each with their own compute and storage, may still be appropriate. On the other hand, if, like many enterprises, you are adding additional analysis tools such as NoSQL databases, SQL on Hadoop (Impala, Stinger, Vertica) and particularly Spark, an in-memory-based analytics technology that is well suited for real-time and streaming data, it may be necessary to begin reassessing the supporting infrastructure in order to build something that can continue to support Hadoop as well as cater to the differing access patterns of other tools sets. This need to rethink the underlying analytics plumbing was brought home by a recent demonstration by HP of a reference architecture for analytics, publicly referred to as the HP Big Data Reference Architecture.