Instrumenting Your Enterprise for Maximum Predictive Power

By James Kobielus

Business is all about placing bets and knowing if the odds are in your favor.

As I noted in my most recent Forrester report, business success depends on your company being able to visualize likely futures and take appropriate actions as soon as possible. You must be able to predict future scenarios well enough to prepare plans and deploy resources so that you can seize opportunities, neutralize threats, and mitigate risks.

Clearly, predictive analytics can play a pivotal role in the day-to-day operation of your business. It can help you focus strategy and continually tweak plans based on actual performance and likely future scenarios. And, as I noted in a recent Forrester blog post, the technology can sit at the core of your service-oriented architecture (SOA) strategy as you embed predictive logic deeply into data warehouses, business process management platforms, complex event processing streams, and operational applications.

The grand promise of predictive analytics—still largely unrealized in most companies—is that it will become ubiquitous, guiding all decisions, transactions, and applications. For the technology to rise to that challenge, organizations must move toward a comprehensive advanced analytics strategy that integrates data mining, content analytics, and in-database analytics. Already, we’ve sketched out a vision of “Service-Oriented Analytics,” under which you break down silos among data mining and content analytics initiatives and leverage these pooled resources across all business processes.

You may agree that this is the right vision but have doubt about whether there is a practical, incremental roadmap for taking your company in that direction.  In fact there is, and it starts with re-assessing the core of most companies’ predictive analytics capability: your data mining tools. As you plan your predictive analytics initiatives, you should avoid the traditional approach of focusing on tactical, bottom-up project-specific requirements. You should also try not to shoehorn your requirements into the limited feature set of whatever modeling tool you currently happen to use.

To become a fully predictive enterprise, you will need to take both a top-down and bottom-up approach to your data mining initiatives. From the top-down, it’s all about building and integrating alternate models of how your business environment is likely to evolve internally and externally. In our recent report on advanced analytics, Boris Evelson, Leslie Owens, and I sketched out the many business processes that can be enriched by predictive analytics.

So how do you instrument your company to become more predictive? For starters, assess whether your analytics tools support the following capabilities for developing, validating, and deploying predictive models:

• Model multiple business scenarios: You should be able to build complex models of multiple, linked business scenarios across different business, process, and subject-area domains, using such key features as strategy maps, ensemble modeling , and champion-challenger modeling.
• Incorporate multiple information types into models: You should be able to develop models against multiple information types, including unstructured content and real-time event streams, while leveraging state-of-the-art algorithm in sentiment analysis and social network analysis.
• Leverage multiple statistical algorithms and approaches in models: You should be able to develop models using the widest, most sophisticated range of statistical and mathematical algorithms and approaches, including regression, constraint-based optimization, neural networks, genetic algorithms, and support vector machines.
• Apply multiple metrics of model quality and fitness: You should be able to score and validate model quality using multiple metrics and approaches, including quality scores, lift charts, goodness-of-fit charts, comparative model evaluation, and auto best-model selection.
• Employ multiple variable discovery and assessment approaches: You should be able to build and validate models using various approaches for variable discovery, profiling, and selection, including decision trees, feature selection, clustering, association rules, affinity analysis, and outlier analysis.

How is this different from predictive analytics as usual? Traditionally, most predictive modeling specialists focus on the latter three capabilities: statistical algorithms and approaches, model quality and fitness, and variable discovery assessment. Most models are built in narrowly scoped business or subject domains—such as customer analytics for marketing campaign management—and only against structured data sources (such as relational tables). Traditionally, few predictive analytics projects have entailed modeling of multiple business scenarios across diverse domains--such as sales, marketing, customer service, manufacturing, and supply chain-- though in the real world these business processes are often quite interconnected. Also, many data mining initiatives fail to incorporate information from unstructured sources—such as text in call-center logs—though this content may be as important as what comes relational databases and other structured sources.

It’s very important to build multi-scenario predictive models against complex information sets, but becoming a fully predictive enterprise demands much more. To instrument your organization for maximum predictive power, you should also tool your advanced analytics to support the following capabilities:

·         DW-integrated data preparation: To speed up and standardize the most time-consuming predictive modeling project tasks, you should be able to leverage your existing data warehouse, extract transform load, data quality, and metadata tools to support a full range of data preparation features. These features include the ability to discover, acquire, capture, profile, sample, collect, collate, aggregate, deduplicate, transform, correct, augment, and load analytical data sets.

·         Deep application and middleware integration: To deliver models deeply into whatever heterogeneous SOA-enabled platform you happen to use, your predictive analytics tool should deploy on and/or integrate with a wide range of enterprise applications, middleware, operating platforms, and hardware substrate. You should be able to deploy models seamlessly into your data warehouse, business intelligence, online analytical processing, data integration, complex event processing, data quality, master data management, and business process management environments. And to play well in your SOA, your predictive modeling tools should support application programming interfaces, languages, tools, and approaches such as Web services, Java, C++, and Visual Studio, as well as emerging languages such as SQL-MapReduce and R.

·         Consistent cross-domain model governance: To avoid fostering an unmanageable glut of myriad models, your predictive analytics solution should support a wide range of tools, features, and interfaces to support life-cycle governance of models created in diverse tools. At the very least, your tools should enable model check in/check-out, change tracking, version control, and collaborative development and validation of models. To realize this promise, it should support a full range of tools, standards, and interfaces for import and embedding of models from other tools, as well as export and sharing of models to other environments.

·         Flexible model deployment: To execute modeling functions--such as data preparation, regression, and scoring—on the widest range of data warehouses and other platforms, your tools should support in-database or embedded analytics. And to scale to the max, your predictive analytics tools should deploy models to massively parallel data warehouses, software-as-a-service environments, and cloud computing fabrics.  Your advanced analytics tools should also support development of application logic in open frameworks—such as MapReduce and Hadoop—to enable convergence of data mining and content analytics in the cloud.

·         Rich interactive visualization: To deliver their precious payload—actionable intelligence—your advanced analytics tools should support interactive visualization of models, data, and results. Ideally, you should be able to visualize all of this in your preferred business intelligence tool, or in the predictive modeling vendor’s integrated visualization layer. Of course, you have every right to expect the full range of visualization techniques--histograms, box plots, heat maps, etc.—regardless of who provides the visualization layer.

As you can see, this goes well beyond data mining as usual. Forrester has a slightly different perspective on the development of the predictive analytics market than you’re likely to get from other sources. We see a robust, flexible, SOA-enabled data mining tools as the centerpiece of advanced analytics for fully predictive enterprises. The competitive stakes are too great for businesses to take the traditional silo-mired approach when implementing this mission-critical technology.

What do you think?

BI, Analytics, And CEP: Some Fruitful Potential Follow-Ons From Software AG’s Acquisition Of IDS Scheer

 by James Kobielus

Yes, of course, Software AG is buying IDS Scheer primarily for the latter’s ARIS family of business process management (BPM) tools. I’ll leave it to my Forrester colleagues who focus on BPM--on both the IT and TI sides of the house--to call out the ramifications for Software AG’s positioning in that market.

But, believe it or not, this deal will also launch Software AG into the growing markets for business intelligence (BI), analytics, and complex event processing (CEP) solutions. We bet you didn’t realize that IDS Scheer has ARIS solutions in these fast growing markets, but in fact they do--and they’re continue to evolve those offerings.

It’s no surprise that IDS Scheer’s BI, analytics, and CEP offerings supplement and extend its BPM portfolio. Its CEP solution, ARIS Process Event Monitor, supports business activity monitoring (BAM). Its analytics offerings, ARIS Process Performance Management and ARIS Performance Dashboard, support visualization, dashboarding, scorecarding, drilldown, and alerting on process key performance indicators (KPIs), both historical and real-time. And its forthcoming BI offering, ARIS MashZone, will support self-service user development of reports, dashboards, and other views of process and business metrics.

IDS Scheer has little market share in these non-core segments. And the vendor is no immediate threat, by itself or under its future corporate parent, to the leaders in the BI, analytics, and CEP segments. Indeed, its forthcoming mashup-oriented BI offering only provides a subset of the features available from market leaders such as SAP Business Objects, IBM Cognos, and MicroStrategy. But the fact that Software AG will soon be able to provide its own offerings in those segments, rather than rely wholly on partners, represents an important step in its attempt to field a full service oriented architecture (SOA) solution stack.

As noted in a blog entry a year and a half ago, BI is the crown jewel in any comprehensive SOA solution portfolio. SOA suites cannot be considered feature-complete unless they incorporate a comprehensive range of BI features. This acquisition continues the ongoing SOA solution build-out strategy that motivated Software AG to acquire webMethods in 2007.

But it’s not clear yet whether Software AG plans to flesh out its BI, analytics, and CEP strategies going forward and thereby confront SAP, Oracle, IBM, Microsoft, and other SOA full-stack vendors head-on in these segments. It is also unclear how much effort or expense Software AG would incur in extricating the IDS Scheer offerings from the larger ARIS portfolio in order to make them more general-purpose and less BPM-centric. Nevertheless, Software AG will at the very least have a strong set of enabling technologies to support any such strategy in the near future.

What’s most exciting, and potentially differentiating, about the Software AG/IDS Scheer BI portfolio is the combination of CEP with mashup and an in-memory architecture to support truly real-time, interactive analytics. In other words, Software AG/IDS Scheer could take a page out of the book of another SOA full-stack vendor: TIBCO and its Spotfire product group. In doing so, Software AG/IDS Scheer would also be well-positioned to duke it out with SAP, IBM, Microsoft, and Oracle, all of which are beginning to emphasize in-memory CEP-enabled BI strategies. As we noted in a report from late 2008, in-memory architectures are coming to dominate the BI arena. Likewise, Forrester has called attention in to the growing adoption of CEP for truly real-time BI.

Whether Software AG capitalizes on the opportunity to expand its SOA solution stack into BI remains to be seen. Considering that it took Oracle more than a year to publicly declare how it will position BEA’s CEP and data federation technologies within its own SOA stack, we may have to wait a while before Software AG and IDS Scheer craft an equivalent roadmap--if they ever do.

But if they wait too long, the newly merging vendors may find that the dynamic SOA, BI, and CEP markets have passed them by.

BI Mashup Maturity Model? Oxymoron? Au Contraire Mon Frère!

By James Kobielus

In one of my recent tweets, I commented that Forrester has developed a maturity model for enterprise adoption of mashup-style, self-service development of business intelligence (BI) applications. Indeed, we have, and it will appear in my forthcoming Forrester report, “Mighty Mashups: Do-It-Yourself Business Intelligence for the New Economy.”

Another tweeter--an astute, but sadly, non-Forrester BI analyst--scoffed that “BI mashup maturity model” is an oxymoron. Respectfully, I must disagree. Enterprises are adopting self-service BI approaches for many reasons--principally, to cut costs in a tight economy, to unclog the development backlog, and to speed delivery of actionable, targeted intelligence to decision makers. Also, companies are providing users with BI tools to do interactive, deeply dimensional exploration of information pulled from enterprise data warehouses (EDW), marts, cubes, transactional applications, and other systems. Furthermore, organizations everywhere have adopted browser-oriented BI environments that leverage the full Web 2.0 interactivity and collaboration.

Sitting at the convergence of those trends is BI mashup, which Forrester sees as the new paradigm for truly pervasive decision-support systems. What throws off some people is the term “mashup,” which sometimes gets pigeonholed as simply referring to using, say, Google Maps to display geocoded performance metrics and sundry Internet-sourced data in a browser-based dashboard. Yes, BI mashup encompasses that approach to presenting and integrating diverse data, but its application is much broader.

Just as important, BI mashup is not bleeding-edge. Rather, BI mashup leverages the in-memory BI clients, semantic virtualization layers, data federation middleware, automated data discovery, and other next-generation BI tools and platforms.

No one vendor or user has yet put together an end-to-end BI environment that is entirely focused on mashup-style self-service development. However, Forrester sees the BI industry converging toward as mashup-oriented architecture over the coming 2-3 years. With that in mind, we sketched out a BI maturity model that encompasses the following four levels (the first 3 of which are represented in case studies in the upcoming report):

• Level 1: Lightweight presentation mashup against transactional applications: This basic maturity level is for companies that have no prior BI or EDW; have little in-house BI expertise; and are comfortable with allowing casual users to use their browsers to customize parameterized reports from data from packaged business applications.                                                                
• Level 2: Deep presentation mashup against EDW: This level is for organization that do have prior BI and centralized EDWs, but have an understaffed BI development group and/or  power users and data modelers urgently require the ability to mashup and explore historical and current data within sophisticated BI workspaces.
• Level 3: Full BI mashup in federated environment: This level is for organizations that have decentralized, dynamic data management environments, and have the expertise to design reusable, composite data services to seamlessly mashup internal and external information.
• Level 4: Full collaborative mashup with IT governance: This level is for organizations that want to encourage subject  matter experts and operational users to collaborate on analytics created through mashup, but who are also concerned that all mashups be controlled, governed, and monitored in accordance with enterprise policies and best practices.

As I said, it will take a few years before we see a substantial number of enterprise case studies that implement the pinnacle of collaborative mashup with tight governance. Nevertheless, when you follow the evolution of next-generation solution portfolios from leading BI vendors such as SAP, IBM, Microsoft, and others, it’s clear that self-service user-centric mashup, to varying degrees, is a core theme.

BI mashup has such a strong business case that we’re confident it’s more than simply a “down economy” theme. It will almost certainly grow in importance for information and knowledge management professionals as the economy improves.

Self-Service Business Intelligence Depends on Automated Data Discovery

 By James Kobielus

If you tuned into my Forrester teleconference yesterday, you heard me discuss the end-to-end infrastructure necessary to fully support mashup-style self-service business intelligence (BI).

One of the key features for BI mashup is automated source-data discovery, which spares information workers from having to find new data sources or fresh updates from existing sources. Instead, the user simply relies on the BI and back-end data virtualization infrastructure to perform these critical activities as ongoing background tasks. Once new sources and feeds are discovered, transformed to a common semantic model, and published to a BI-mashup registry, all the user needs to do is drag and drop them visually into their mashed-up reports, dashboards, and other analytics.

Automated discovery is not only key to BI mashup, but to trustworthy data as well, because it helps detect and remediate anomalies across disparate data sources. Only a few vendors on the market today provide strong features for automated source discovery. One of them is Composite Software, which recently released an appliance that performs these functions. Another is Exeros, which is the closest thing to an automated-data-discovery pure-play in the market today.

Or, rather, was the closest thing, until IBM announced this morning that it is acquiring Exeros. I’ve been following Exeros for several years and have long considered them a strong candidate for acquisition by a leading BI, data warehousing (DW), data integration (DI), or data quality (DQ) vendor. On IBM’s part, this acquisition makes great sense as a complement to its InfoSphere and Optim portfolios on the data management and governance side of the house.

It will also fit nicely with IBM’s Cognos portfolio as a key enabler, potentially, for BI self-service mashup. As I stated on my teleconference, some vendors are further ahead on putting together a completely mashup-enabling end-to-end BI solution, and Cognos is among them. You can download the teleconference slides from Forrester’s website, listen to my streaming audio, and/or wait for my forthcoming report for more in-depth thoughts on this topic.

Now the ball’s in IBM’s rivals’ courts regarding whether, when, and how they plan to add automated source discovery to their BI portfolios.

Inmon’s Vitriolic Slap At “Virtual Data Warehousing” Does Not Withstand Scrutiny

By James Kobielus

In a recent article, Bill Inmon incinerates a straw man concept that he refers to as “virtual data warehousing (DW).” For those unfamiliar with Inmon, he is generally considered the founder of DW as a data management discipline, has been at it since the 70s, and has more published books and articles to his name than most mortals. So he clearly may be considered an authority on the topic of DW.

But methinks Mr. Inmon doth protest too much on this “virtual DW” bugaboo, however defined (we’ll get to that in a moment). Also, he attacks this concocted notion with such emotional vehemence that it’s clear he considers it a threat to the centralized EDW paradigm upon which he has built his career and reputation.

For starters, his definition of this concept is oddly vague and questionably narrow: “a virtual data warehouse occurs when a query runs around to a lot of databases and does a distributed query.” Essentially, Inmon defines “virtual DW” as the ability to a) farm out a query to be serviced in parallel by two or more distributed databases, b) aggregate and join results from those databases, and c) deliver a unified result set to the requester.

That’s an important query pattern, but not the only one that should be supported under (pick your quasi-synonym) data federation, data virtualization, or enterprise information integration (EII) architectures. Inmon’s definition excludes the many federated queries that may only hit on a single database, with no joins and results aggregation, and with the EII fabric handling the necessary on-demand transformation from that source’s schema to an abstract semantic model.

Per my data federation report from last fall, Forrester has a broader perspective on the topic than does Mr. Inmon. Data federation is any on-demand approach that queries information objects from one or more sources; applies various integration functions to the results; maps the results to a source-agnostic semantic-abstraction model; and delivers the results to requesters. Nothing in the scoping of data federation necessarily requires the multi-source aggregation and joining that Inmon puts at the heart of “virtual DW.”

Putting Inmon’s narrow scoping of “virtual DW” behind us for the moment, let’s consider his chief objections to this approach. First, it requires the “analyst to integrate data” (as if that’s something analysts are ill-suited for or regard as some inordinate burden). Second, it consumes resources, experiences suboptimal performance, and “shuffles a lot of data around the system that otherwise would not need to be moved” (as if centralized DWs don’t consume resources, experience performance bottlenecks, and move data). Third, it is “limited to the [historical] data found in the [source] databases.” Fourth, it suffers from “no reconcilability of data...[hence] no single version of the truth for the corporation.”

It’s a fairly straightforward matter to dispatch these objections:

First, data integration--through ETL, EII, and other approaches--is a core job function for DW professionals, not some alien function outside their core competency.

Second, data federation is often the optimal approach for low-latency BI (just check out the case studies in my data federation and really urgent analytics reports). Federated environments can be tuned to provide top-notch performance and minimize source-system impacts when “shuffling” data around in a decentralized fabric.

Third, the source databases in a federation environment often include DWs, which, per their core function, usually manage a considerable amount of historical data. Once again, see my data federation report with discussion of case studies for a) Federation of Local DWs via Centralized EII Infrastructure and b) Federation of Dispersed EDW and ODS Data Into Siloed BI Environments.

Fourth, data federation is not totally incompatible with data reconciliation. In fact, federation environments can be architected for single version of the truth, data governance, and master data management. However, it can indeed be tricky to manage data quality in federated environments (see Rob Karel’s coverage of MDM and DQ for a deep dive on that issue).

My basic objection to Inmon’s line of discussion is that he treats data federation as mutually exclusive from the enterprise DW (EDW), when in fact they are highly complementary approaches, not just in theory but in real-world deployments. Yes, data federation can be deployed as an alternative to traditional EDWs, providing direct interactive access to online transactional processing (OLTP) data stores. However, data federation can also coexist with, extend, virtualize, and enrich EDWs, as well as other data-persistence nodes such operational data stores (ODS) and online analytical processing (OLAP) data marts. The case studies in the cited reports bear that out.

Inmon’s arguments are worth consideration. The centralized EDW model he touts is useful for illuminating some traditional best practices. But by no means can it do justice to the stubbornly heterogeneous, distributed, mixed-latency BI and DW requirements of most enterprises.

After So Many Years Of Ballyhoo, Semantic Web Still Searching For Killer App

By James Kobielus

Cynics might call Semantic Web a technology looking for a solution. And they might have a point.

Semantic Web refers to a long-running World Wide Web Consortium (W3C) initiative that is working toward an ambitious--some might say hopelessly Utopian--goal. At heart, it is a vision for how the World Wide Web should evolve to realize its full interoperability potential.

People vary widely in how they interpret the scope of the Semantic Web initiative. The tech industries are swarming with a wide range of projects, products, and tools that implement different variants of this vision. What vision is that? In the broadest sense, Semantic Web refers to an all-encompassing metadata, description, and policy layer that can, potentially, support universal, automatic, comprehensive end-to-end interoperability across every macro or micro entity—including data, components, services, applications, and services—on every conceivable level.

Whew!!! If that’s not the working definition of “pie in the sky” or “boil the ocean” (pick your metaphor), I don’t know what is. In fact, I’m hard-pressed to refer to Semantic Web as a definable market or solution segment. However, it’s not entirely vacuous.

For starters, organizations can implement W3C-developed semantic description standards—such as Resource Description Framework (RDF) and Web Ontology Language (OWL)--to make the meaning of content unambiguously comprehensible to services, applications, bots, and other automated components. Second, there is a reasonably robust market for “ontology” tools to support RDF/OWL-based modeling of application semantics. Finally, there is some incremental adoption of these tools and concepts in established IT segments, such as:

• Enterprise content management (ECM): Semantic approaches can support more powerful discovery, indexing, search, classification, commentary, and navigation across heterogeneous stores of unstructured and semi-structured content. Semantic search—driven by concepts, not mere text strings--is regarded by some as a primary Semantic Web application. Indeed, many Semantic Web vendors are primarily implementing the technology in search engines that leverage ontology-based concepts to improve search accuracy and reduce spurious hits.

• Enterprise information integration (EII):Semantic approaches enable consolidated viewing, query, and update of structured data that has been retrieved from diverse sources. Indeed, most commercial EII environments present an abstract semantic layer that mediates access to heterogeneous data, such as enterprise resource planning and customer relationship management applications, converging it all to a common presentation-side schema. A handful of those EII vendors have begun to support Semantic Web standards, primarily through third-party software plug-ins

• Enterprise service bus (ESB):Semantic approaches can facilitate multilayered application, process, and service interoperability across disparate environments. To date, there has been little production implementation of Semantic Web standards in the ESB arena, though some vendors have adopted semantics, ontologies, and RDF to describe the conceptual models implemented by application endpoints, agents, and intermediary nodes within ESB-like middleware approaches such as event stream processing.

But Semantic Web approaches are still on the periphery of these markets. 10+ years into its inception, Semantic Web still has no clear killer app. It’s not clear if or when that app will emerge.

Lean Information Management Strategies for Lean Times

By James Kobielus

When the going gets tough, the tough get lean, focused, and flexible. To help organizations survive the bad times and thrive in all climates, their information management initiatives must remain agile and adaptable.

If you feel your information management strategy is anything but lean, you’re not alone. Many organizations struggle to gain control over information infrastructures that have become too bloated, rigid, and slow to realign with new business drivers.

Lean information management practices are essential for corporate survival. They are far more than belt-tightening exercises. They also help you build analytic muscle for excelling in any business environment. Here are some basic pointers for keeping your information management strategy lean:

• Trim your information infrastructure of excess cost. Lean means you should cut excessive, budget-busting overhead from your information management environment. Careful cuts are best, because they optimize your existing operations without gutting the core information, analytics, and applications that underpin your core competencies. Silo, server, database, and application consolidation should be your principal approaches. Also, you should re-evaluate vendor-sourcing strategies and renegotiate licenses at more favorable terms. And you should investigate lower-cost alternatives, such as software-as-a-service, to address business intelligence, business performance solutions, enterprise data warehousing, master data management, enterprise content management, and other information management requirements.
• Fit information initiatives to key business imperatives. Lean also means you fit, focus, and fully align your information management initiatives to mission-critical business imperatives. Strategic alignment ensures that you leverage information assets across diverse application domains and business processes, rather than allow that intelligence to languish underutilized in silos. To sustain this approach, you should establish an information management framework, such as a Business Intelligence Solution Center, that enables ongoing collaboration between business and IT stakeholders. You should engage all key business and technical groups in information management planning discussions.
• Flex information architectures to changing circumstances. Finally, lean means maintaining an approach that is flexible and adaptable, able to shift course as your needs and environment change. In yoga terms, lean is all about building, toning, and stretching analytical muscle to keep it from tearing when you need to transition rapidly from one strategic alignment to the next. You need the flexibility to swing between centralized information management infrastructures and decentralized or federated environments. For end-to-end data management environments, Forrester has developed an architecture decision support tool that helps information managers to determine which of several topologies is best suited to their needs: centralized enterprise data warehouse, hub-and-spoke, independent data marts, data federation, and information-as-a-service.

Considered as a comprehensive strategy, these lean practices are true bloat-busters and recession-beaters. They allow organizations to deliver practical insights that address all pain points, even--especially!!!--within strict budgets.

Whatever Happened To EII?

By Rob Karel and James Kobielus

The integration software market is a busy place, with technologies such as ETL (extract, transform, and load), ESB (enterprise service bus), CDC (change data capture) and IC-BPMS (integration-centric business process management suites) crowding the landscape.  The technologies within this "integration acronym buffet" address a common requirement: they either physically or virtually deliver information from point A to point B with integrity.  (For more information on the variety of integration options available, see "2009 Update: Evaluating Integration Alternatives.") Some approaches, such as ETL and ESB, have developed into multi-billion dollar stand-alone markets.   But other integration alternatives, while valuable, haven't survived as well as a standalone market segment.   EII (enterprise information integration) is one example of this. 

In a nutshell, EII -- also often referred to as data federation -- is an umbrella term that arches over a collection of technologies and best practices for providing custom views into multiple data sources as a way of integrating data and content for real-time read and write access by applications.   Though integration professionals have been using EII for many years in niche applications, it has struggled to find the ideal scenarios in which it has a clear advantage over other integration approaches.   

Where enterprise data warehousing (EDW) is concerned, EII provides an on-demand, decentralized "virtual" alternative, but has not dislodged ETL's role in the centralized, batch-oriented EDW architectures that predominate throughout the corporate world. For real-time and operational business intelligence (BI) requirements, EII supports low-latency queries across distributed data environments, but it competes against other approaches such as trickle-feed ETL and CDC.  And for real-time read/write access across disparate databases, EII competes against more versatile approaches such as ESB and information as a service (IaaS).

In more recent years, EII has found adoption as an adjunct to enterprises' core EDW, BI, and transactional computing integration strategies--not as the main approach.  When they implement EII, information architects generally use this technique sparingly, and primarily for tactical, project-based requirements -- not as an enterprise standard.  So it’s no surprise that a stand-alone EII tools marketplace has failed to develop, or that most EII vendors have slowly disappeared via acquisition into larger BI, EDW, data integration, or database platforms such as Business Objects's, Red Hat's, and Sybase's acquisitions of Medience, MetaMatrix, and Avaki, respectively.   Meanwhile, the remaining EII pure-play vendors such as Composite Software have wisely shifted their solutions and messaging towards a broader focus on SOA (service-oriented architecture) and IaaS.   

Ironically, as the EII-branded market continues to consolidate and merge with other data management markets and essentially disappear as a recognized standalone segment, EII's earliest goal of enabling virtual data warehousing may now be poised to become reality.   As the complexities of today's business operations increasingly requires real-time decision making, real-time data warehousing options are once again being discussing by data architects across industries.  As Forrester analyst James Kobielus discussed in his research, "Federation: Sharpen Your Focus On Vast Constellations Of Data," multiple use cases have arisen where leveraging EII tools and techniques may be the best way to deliver information insights in a complex, heterogeneous environment.  And federated use cases go beyond data warehousing and business intelligence.  Architects designing solutions to manage complex enterprise content management (ECM) and master data management (MDM) strategies also consider federation as a legitimate means to bridge information siloes.

Does this mean the standalone EII market is ready for a comeback?  Not quite.  What it does mean is that EII may have finally found its place in the software world.  Vendors building data management solutions focusing on areas such as IaaS, SOA, BI, data integration, data warehousing, and MDM should continue to invest in seamlessly integrating EII-like federation capabilities as an embedded integration technique available to their customers.  An executive at a data integration software company recently shared with us his perception of how their customers have evolved to think about EII: "What was once called EII are really just a set of integrated access, abstraction, federation, and delivery capabilities behind (a) Virtual Data Federation as a complementary data integration tool in the DI project toolbox and (b) IaaS as an enterprise data virtualization architecture approach for the enterprise." So integration architects should take a step back and consider all of the options within the integration acronym buffet not as competing tools, but complementary techniques that must build the foundation for your organization’s cross-enterprise integration competency.

Monday’s Musings: Master Data Management - Do Styles of MDM Matter Anymore?

Three Architectural Styles Represent Different Technologies To Build MDM

In 2003, customer data hub (CDI), product information management, and master data management (MDM) vendors strived to differentiate themselves by architectural style.  Each approach had its advantages and disadvantages.  A religion about styles emerged overnight along with a hard core following.  Here's a quick recap (see Figure 1):

Figure 1.  The Three Architectural Styles of Master Data Management

The bottom line - choose a style that aligns with your project's business driver
While these approaches still exist, leading vendors such as D&B Purisma, IBM, Initiate Systems, Oracle, Oracle-Siebel, SAS DataFlux, and Siperian now have offerings in more than one style. This may make the question seem less relevant, however, its still important to understand the trade-offs while beginning your MDM journey.  In fact, it's best to align the style and approach based on your business driver.  Here's a high level summary:

• Cross-referenced registry delivers rapid results for operational efficiency business drivers. This approach is best suited for rapid implementation scenarios such as POC's that prove the value of master data.  Also valuable when data can not be stored on-site.
  Pro's: Rapid implementation without having to agree on a common enterprise data model.  Utilize existing source systems.
  Con's: Deduplication of source systems not addressed.  Data quality must be solved in each independent source system.
• Hybrid harmonized reference enables compliance and regulatory business drivers. This approach allows the best of both worlds, especially when moving to a transactional operational data store is not politically feasible and data governance and stewardship activities are just starting up.
  Pro's: Single master copy of reference data.  Uses links to access source system records.  Model allows data quality efforts to be applied to shared master  reference data.
  Con's: Synchronization with source systems can create some complexity if changes are not made in the hub.
• Transactional operational data store supports strategic business drivers.  This approach provides a long term path for how legacy applications utilize data.
  Pro's: Single master copy of data.  No fussing with latency or synchronization issues.  Minimal mapping issues.
  Con's: Requires an agreed upon common enterprise data model to be used by all applications.  History must be harmonized and requires extensive key mapping.  Assumes homogeneity and requires tons of ETL and dedupe.

Your POV.

Which MDM style are you deploying? What successes have you seen? Post your thoughts or send me a private email to rwang0@forrester.com.

Copyright © 2009 R Wang. All rights reserved.
Reposted from http://blog.softwareinsider.org

Event Report: Siperian Masters 2008 - Customers Confirm Multi-Entity MDM Trends

by Ray Wang

About 200 attendees were present as Ramon Chen, VP of Marketing, kicked off the event to the theme of adventurers and pioneers in MDM at the Bridgewater Marriott (New Jersey).   CEO, Peter Caswell, led the keynote session with a view on where Siperian has been, where Siperian is going, and then introduced the Ravi Jagannathan VP of Product Management and Manish Sood, Senior Director of Product Management. They presented Siperian's road map well into 2012.  Key announcements include:

• Ongoing expansion of the partner ecosystem and alliances.
• Announcement of semantic masters for unstructured data.
• Focus on easier to maintain GUI
• Continued availability of modular deployment options and other cost effective implementations
• New state management and work flow integration tie backs to the Lombardi BPM tools
• Visually appealing data governance dashboards.

In addition, a few key trends emerged from conversations with customers and partners:

• Most customers who had MDM projects also were embarked on SOA projects
• Pharma customers successfully proved ROI and justification despite being in SAP and Oracle "only" environments
• Availability of system integrator resources has improved.
• MDM projects need to be more pervasive and address innovation in order to gain long term political support.
• Many customers have reached what Forrester Research considers a Level 3 and 4 MDM maturity.
• Prospects continue to see Siperian as short listed vendors
• Many seek more innovation from their MDM systems and are beginning to branch out of their single data entity focus.

The bottom line

Siperian customers seem to be well ahead of the pioneering stage with MDM.    Customers we spoke to remain satisfied with their decisions and have been successful in proving existing value.  many customers have transcended past level 3 on the MDM maturity model.

Your POV

Do these trends jive with what you are seeing in MDM and CDI? Looking forward to hearing your thoughts.  Post a comment or privately reach out to me at rwang@forrester.com