For years I've been predicting that relational DBMS will run out of steam when it comes to effectively managing and manipulating very large, heterogeneous (structured + unstructured) data sets for business intelligence. First, RDBMS were never designed and optimized for unstructured data (not just XML, which is structured data in my definition, but truly unstructured text pages). Second, there's just too much overhead and cost in RDBMS for handling OLTP functions. The result: search index DBMS will be king in BI and DSS in the future.
Today’s announcements that Microsoft may be buying Yahoo came several weeks early. On May 17th I would’ve gone on the record at Forrester IT Forum in Nashville by saying the following, and I quote from my presentation paper: “DBMS/BI vendor may buy a search company, to address the trend of increasing importance of unstructured data in BI and to obtain an early leading position in the space. I know it should be Oracle or IBM, but it probably won’t, since these guys will never admit that their relational DBMS cannot do something. Microsoft is a more likely contender since they know they won’t leapfrog IBM or Oracle in relational DBMS and they could use this opportunity to stick one to Google too.”
I thought Microsoft would buy somebody like Fast Search, but I guess that was too small for them.
I attended a conference sponsored by Carnegie Mellon West; The Fisher IT Center at the Haas School of Business, UC Berkeley; the Software Industry Center at Carnegie Mellon University; and Services: Science, Management, and Engineering Program at UC Berkeley. The one-day event was held at the Microsoft Campus at Moffitt Field in Silicon Valley. The goal of this conference was to discuss where the software industry is going. Ten sessions including individual speakers and panels from university and business communicated the strong message that software is at a crossroads and will dramatically change in the future, and . . . the change has already begun. To access slides of the speaker presentations go to http://west.cmu.edu/sofcon/postcon.
The changes are around growth of software-as-a-service, new roles of services as a value- add to commoditized software, and new businesses and pricing models. The overwhelming consensus was that software-as-a-service is where the growth is today. Speakers pointed out some of the most successful companies in terms of generating revenue like WebEx, Amazon, Google — all service-based. At the same time they do not see companies that have built their business around software like Oracle, SAP, and Microsoft going “down-the-tube” just yet. In fact Oracle already has Oracle On-demand, a very successful service solution while supporting their enterprise installed customers. Companies that have these installed applications will not find it easy to change to a service-model, even it they wanted to. It requires architectural, economic and cultural changes and requires a ten-year time table to move from an installed software model to a services model. It seems much easier to start from the ground up like Salesforce.com.
Remember George Costanza from a Seinfeld episode where he was pulling his hair out about “the two worlds colliding”? He was agonizing over the world of his girlfriends and the world of his friends that should never mix. In my world, process and data, separate disciplines until recently, are now “colliding”. While some of the vendors have already been toying with the convergence of both disciplines (IBM, Oracle, SAP), today’s announcement by Tibco that it will acquire a Spotfire, is the first transaction that will merge a pureplay middleware vendor with a pureplay BI vendor (a convergence that Forrester’s been predicting for almost a year, please see our Business Intelligence Meets BPM In The Information Workplace research document. But by acquiring Spotfire, Tibco has actually achieved more than one goal.
Being efficient is no longer enough. Enterprises can no longer stay competitive just by squeezing more efficiencies from operational applications, including workflow, business process management (BPM) and business rules engine (BRE) — business intelligence applications are needed to become more effective. For example, while workflow and rules are be used to efficiently process a customer credit application, Business Intelligence analytics are needed to effectively segment customer population and extend the credit offer to a much more targeted customer segment for a better response, cross-sell/up-sell ratios.
The actual convergence of process and data. The other slant is the natural interdependency of process and data from two angles: a) one needs data to feed and enrich the business process and process rules, and b) an event (an alert, for example) triggered by a data condition has to go into a process so that it can be followed up and acted on.
At the AIIM show in mid April, Xerox Global Services gathered a number of information management industry pundits to talk about issues related to eDiscovery. The conversation shed light on myriad issues that organizations face to meet the requirements of the newly amended Federal Rules of Civil Procedure (FRCP). You can listen to a podcast of the discussion here. The major points of note:
I’ve recently conducted research on the issues of VLDB (very large databases) and how it affects BI, since the challenges of reporting and analyzing Gb data sets are very different from reporting and analyzing multi Tb data sets. Among many other interesting findings and conclusions I uncovered the following approaches to handling VLDB challenges as they relate to BI:
Generic solutions by DBMS vendors:
Share everything vs. share nothing architecture
Caching, in-memory databases
Specialized file systems
Indexing (bit-map vs. B-tree)
BI-specific solutions by BI and other vendors:
ROLAP and reporting tool specific SQL optimization
Alternate DBMS (such as search indexes and vector DBMS)
I predict that for the foreseeable future, spreadsheets will remain the most widely used enterprise application. The widespread adoption isn't hard to understand — spreadsheets are powerful and flexible, yet intuitive and easy to use and learn. Plus, ad hoc applications and spreadsheet models isolate users from constant reliance on IT and incur low costs. Since the early days of BI, spreadsheets have played a natural and major role in the BI process/architecture, including:
I've been in the BI business for over 25 years so I've seen many ups and downs in the BI cycle. We are definitely in the "up" cycle these days. I see two main reasons for it:
Enterprises can no longer stay competitive just by squeezing more efficiencies from operational applications – business intelligence applications are needed to become more effective.
Digital data (structured and unstructured) volumes are growing at 30% a year, and will be reaching zetabyte sizes by year 2010 – that’s a number with 21 zeros! Solid BI implementations will be critical to successfully turn that data into useful information.
Does anyone have any comments on where we are in the BI cycle and what are some of the more recent key drivers?
Adobe has put an end to much speculation, announcing that it will take Flex into the open source realm and make the source and documentation available under the Mozilla Public License. This move certainly ratchets up the battle between Flex and Microsoft’s Silverlight technology — both are used to create rich Internet apps. Microsoft has targeted Flash and Flex, so Adobe apparently has come to believe that open source is the best option for gaining ubiquity for Flex. Traditionally, Microsoft has had the edge with developers; Adobe with designers. Adobe is clearly hoping this move will shake that up.
Organization’s learning leaders hear the words “informal learning” or “eLearning 2.0” and think, “Oh my, now we have to change the way we provide training!” Yes, you may want to make some changes but, more importantly, you need to look at existing learning within your organization and determine what is training and what is education or development. I see two distinct types of learning that are both complementary, but also dramatically different. Today’s knowledge workers need both.
Training refers to the learning that employees access in order to do their job. This includes traditional mandated training for fields like accounting or pharmaceuticals. But a large percentage of training should be the “just-in-time” kind that gives the employees the information or knowledge refresher that they need to continue their work task. This informal learning is driven by the employee and is generally not tracked except to indicate the number of employees who have accessed the sites. Examples include online mentoring, clicking on the “just-in-time” learning related to the work topic for a three-to-five minute learning nugget, accessing the context-sensitive learning built into the application, or clicking on “expertise location” on the intranet to find a person in the organization who has the expertise to help. This kind of training or knowledge seeking requires a good search engine to find a document, PowerPoint, video, blog, wiki, etc. on the organization’s intranet site. A good practice is to make the five to ten-minute learning objects or course components searchable so an employee can find the exact part of a module or course that will provide the assistance they need.
The WCM vendor landscape has shifted again, as global information management solution vendor SDL has announced that it will acquire Tridion. SDL and Tridion are touting the complementary functionality of SDL’s translation management capabilities and Tridion’s WCM product, and the inevitable tighter integration between the two. The acquision certainly provides Tridion more stability, and gives the soon-to-be-christened SDL Tridion a strong Web content globalization story. This should also give Tridion leg up in the US; Tridion has opened two US offices over the past year or so in an effort to target the North American market, but SDL already has a bigger field organization there.
My take: this isn't an earthshaking announcement, but Tridion and SDL are stronger together than they are apart. For Tridion customers, the good news is that Tridion and SDL don’t have overlapping technologies, so this acquisition shouldn’t lead to any forseeable pain for future upgrades. And the Tridion management team will be in place for the next two years, lending some stability to the acquisition.
SDL and Tridion will be announcing a technology roadmap and integration timeline sometime in the next six weeks. Stay tuned.