Ah, the good old days. The world used to be simple. ETL vendors provided data integration functionality, DBMS vendors data warehouse platforms and BI vendors concentrated on reporting, analysis and data visualization. And they all lived happily ever after without stepping on each others’ toes and benefiting from lucrative partnerships. Alas, the modern world of BI and data integration is infinitely more complex with multiple, often overlapping offerings from data integration and BI vendors. I see the following three major segments in the market of preparing data for BI:
Fully functional and highly scalable ETL platforms that are used for integrating analytical data as well as moving, synchronizing and replicating operational, transactional data. This is still the realm of tech professionals who use ETL products from Informatica, AbInitio, IBM, Oracle, Microsoft and others.
An emerging market of data preparation technologies that specialize mostly in integrating data for BI use cases and mostly run by business users. Notable vendors in the space include Alteryx, Paxata, Trifecta, Datawatch, Birst, and a few others.
Data preparation features built right into BI platforms. Most leading BI vendors today provide such capabilities to a varying degree.
For decades, firms have deployed applications and BI on independent databases and warehouses, supporting custom data models, scalability, and performance while speeding delivery. It’s become a nightmare to try to integrate the proliferation of data across these sources in order to deliver the unified view of business data required to support new business applications, analytics, and real-time insights. The explosion of new sources, driven by the triple-threat trends of mobile, social, and the cloud, amplified by partner data, market feeds, and machine-generated data, further aggravates the problem. Poorly integrated business data often leads to poor business decisions, reduces customer satisfaction and competitive advantage, and slows product innovation — ultimately limiting revenue.
Forrester’s latest research reveals how leading firms are coping with this explosion using data virtualization, leading us to release a major new version of our reference architecture, Information Fabric 3.0. Since Forrester invented the category of data virtualization eight years ago with the first version of information fabric, these solutions have continued to evolve. In this update, we reflect new business requirements and new technology options including big data, cloud, mobile, distributed in-memory caching, and dynamic services. Use information fabric 3.0 to inform and guide your data virtualization and integration strategy, especially where you require real-time data sharing, complex business transactions, more self-service access to data, integration of all types of data, and increased support for analytics and predictive analytics.
Information fabric 3.0 reflects significant innovation in data virtualization solutions, including:
“Figuring out how to think about the problem.” That’s what Albert Einstein said when asked what single event was most helpful in developing the Theory of Relativity. Application integration is a problem. A big problem. Not to mention data, B2B, and other domains of integration. As an industry analyst and solution architect, what I’m most interested in first is how to think about the problem.
Pop Quiz: The Goal of Integration
Which of the following statements best articulates the goal of integration strategy?
The goal of integration is to keep data in sync across two or more siloed applications.
The goal of integration is to improve business outcomes by achieving consistent, coherent, effective business operations.
The correct answer is B. Was that too easy? Apparently not, because most of the integration strategies I see are framed as if the answer were A. Most, but not all — and it’s the ones framed around B that I’m most interested in. Here’s the difference:
A-style integration centers on technology. It begins with data and business logic fractured across application silos, and then asks, “How can integration technologies make it easier to live with this siloed mess?”
B-style integration centers on business design. It begins with a businessperson’s view of well-oiled business operations: streamlined processes, consistent transactions, unified tools for each user role, purpose-built views of data, and the like. It designs these first — that is, it centers on business design — and then asks, “How can integration technologies give us coherent business operations despite our application silos?”
Last week, I had the pleasure of attending the Insight Innovation Exchange conference in Philadelphia. There were many vendors that offered solutions to many common challenges that market researchers face. One common theme I noticed was the challenge for market researchers to make sense of big data. Yes, big data has become something of a buzzword, but consumers are creating a lot more data and market researchers can thrive if they embrace it.
For some time now, Forrester has been writing about the importance of incorporating behavioral tracking insights to marketer researchers’ research mix. Don’t get me wrong — survey research is and will continue to be incredibly important for companies to gain insights on consumers. A survey can capture a variety of consumer behaviors, sentiments, and attitudes. In one survey, marketers can assess their market share and find out the profile of their customers and what they want. And survey research can help provide insight into the “why” — the reasoning behind the choices that consumers make — something that is not possible with behavioral data. However, survey research cannot detail granular activities due to respondent recall. Enter big data, and with it many possibilities for behavioral tracking. Yes, this is nothing new for customer intelligence professionals, who analyze customer transactions, online web tracking, and other consumer behaviors. But by combining survey and behavioral data, marketers get the best of both worlds: They get consumer profiles and psychographics, brand health metrics, and a detailed record of the actions that those consumers actually do.
Over the course of this year, I’ve spoken with many organizations that are continuing to expand their usage of information-as-a-service (sometimes called data services) to support new business requirements such as self-service customer portals, real-time BI, and single-version-of-the-truth. With the growing complexity of data, increasing volume of data, and exploding security challenges all driving demand, IaaS is poised to grow significantly in the coming years, especially as existing integration technologies are failing to meet these new requirements. What we see is that most organizations that have embraced an IaaS strategy over the years aren’t looking back; they’re continuing to expand its usage to support more requirements such as real-time data, creating data domains, improving the ability to securely deliver information, integration with unstructured data and external sources, various Web portals, and enterprise search.
Recently, my colleague Gene Leganza, who serves Enterprise Architecture Professionals, compiled the top 15 technology trends EA should watch over the next three years . One of the trends Gene highlighted is that information-as-a-service (IaaS) is finding a broader audience. I see more organizations continuing to show strong interest in IaaS, as evidenced by increasing inquiries, to help with growing data integration challenges that traditional solutions are not addressing. IaaS can significantly alter IT’s approach to its data management strategy and delivers a flexible framework to support transactional, BI, and real-time data.
Here are my top predictions for 2011 related to IaaS:
Consistently rated as one of the most popular features of Forrester Events, one-on-one meetings give you the opportunity to discuss the unique technology issues facing your organization with Forrester analysts. Business & Technology Leadership Forum attendees may schedule up to two 20-minute one-on-one meetings with the Forrester analysts of their choice, depending on availability. Registered attendees will be able to schedule one-on-one meetings starting on Monday September 15, 2008. Book early!