One of the reasons for only a portion of enterprise and external (about a third of structured and a quarter of unstructured -) data being available for insights is a restrictive architecture of SQL databases. In SQL databases data and metadata (data models, aka schemas) are tightly bound and inseparable (aka early binding, schema on write). Changing the model often requires at best just rebuilding an index or an aggregate, at worst - reloading entire columns and tables. Therefore many analysts start their work from data sets based on these tightly bound models, where DBAs and data architects have already built business requirements (that may be outdated or incomplete) into the models. Thus the data delivered to the end-users already contains inherent biases, which are opaque to the user and can strongly influence their analysis. As part of the natural evolution of Business Intelligence (BI) platforms data exploration now addresses this challenge. How? BI pros can now take advantage of ALL raw data available in their enterprises by:
I’ll be brief, because I know you’re busy. If you’re a customer-obsessed marketer, you should plan to attend Forrester’s annual Forum designed just for you – MARKETING 2016. Join us and 600+ marketing leaders in New York City on April 26-27 as we dive deep into the issues that matter most to you. Our agenda this year is comprised of five sections:
1. Thriving In The Post-Digital Age: Led by our own VP/Principal Analyst Shar VanBoskirk, hear our latest research on what it takes to succeed as a marketing leader in a world where digital embeds in everything.
2. Customer Understanding: Sick of all the noise about big data? Join VP and Research Director Srividya Sridharan as she uncovers how to move from data, to insights, to business action.
3. Contextual Engagement: Principal Analyst Rusty Warner will be joined onstage by eBay and J&J as they discuss best practices in using situational context to drive deeper customer engagement.
4. The Leadership Question: Forrester’s Michelle Moorehead will moderate a superstar panel on the changing leadership role for CMOs.
5. The Power Of Trust: Principal Analyst Fatemeh Khatibloo will discuss how your ability to manage consumer privacy will be a key differentiator in building trust.
When I read articles like today's WSJ article on mutual funds exiting high tech startups and triangulate the content with Forrester client interactions over the last 12 to 18 months (and some rumors) I am now becoming convinced that there will be some Business Intelligence (BI) and analytics vendor shake ups in 2016. Even though according to our research enterprises are still only leveraging 20%-40% of their entire universe of data for insights and decisions, and 50%-80% of all BI/analytics apps are still done in spreadsheets, the market is over saturated with vendors. Just take a look at the 50+ vendors we track in our BI Vendor Landscape. IMHO we are nearing a saturating point where the buy side of the market cannot sustain so many sellers. Indeed we are already seeing a trend where large enterprises, which a couple of years ago had 10+ different BI platforms, today usually only deploy somewhere between 3 and 5. And, in case you missed it, we already saw what is surely to be a much bigger trend of BI/analytics M&A - SAP acquiring mobile BI vendor Roambi. Start hedging your BI vendor bets!
Rule #1. Don't just jump into creating a hefty enterprise wide Business Intelligence (BI)
Business intelligence and its next iteration, systems of insight (SOI), have moved to the top of BI pros' agendas for enterprise software adoption. Investment in BI tools and applications can have a number of drivers, both external (such as regulatory requirements or technology obsolescence) and internal (such as the desire to improve processes or speed up decision-making). However, putting together a BI business case is not always a straightforward process. Before embarking on a BI business case endeavor, consider that:
You may not actually need a business case. Determining whether a BI business case is necessary includes three main considerations. Is it an investment that the organization must make to stay in business, should consider because other investments are changing the organization's IT landscape, or wants to make because of expected business benefits?
A business sponsor does not obviate the need for a business case. It may be tempting to conclude that you can skip making a business case for BI whenever there is a strong push for investment from the business side, in particular when budget holders are prepared to commit money. Resist this impulse whenever possible: The resulting project will likely suffer from a lack of focus, and recriminations are likely to follow sooner or later.
With the incredible popularity of big data and Hadoop every Business Intelligence (BI) vendor wants to also be known as a "BI on Hadoop" vendor. But what they really can do is limited to a) querying HDFS data organized in HIVE tables using HiveQL or b) ingest any flat file into memory and analyze the data there. Basically, to most of the BI vendors Hadoop is just another data source. Let's now see what qualifies a BI vendor as a "Native Hadoop BI Platform". If we assume that all BI platforms have to have data extraction/integration, persistence, analytics and visualization layers, then "Native Hadoop/Spark BI Platforms" should be able to (ok, yes, I just had to add Spark)
Use Hadoop/Spark as the primary processing platform for MOST of the aforementioned functionality. The only exception is visualization layer which is not what Hadoop/Spark do.
Use distributed processing frameworks natively, such as
Generation of MapReduce and/or Spark jobs
Management of distributed processing framework jobs by YARN, etc
Note, generating Hive or SparkSQL queries does not qualify
Do declarative work in the product’s main user interface interpreted and executed on Hadoop/Spark directly. Not via a "pass through" mode.
Natively support Apache Sentry and Apache Ranger security
You've done all the right things by following your enterprise vendor selection methodology. You created an RFI and sent it out to all of the vendors on your "approved" list. You then filtered out the responses based on your requirements, and sent out a detailed RFP. You created a detailed scoring methodology, reviewed the proposals, listened to the in-person presentations, and filtered out everyone but the top respondents. But you still ended up with more than one. What do you do?
If you shortlisted two or more market leaders (see Forrester's latest evaluation) I would not agonize over who has better methodologies, reference architectures, training, project execution and risk management, etc. They all have top of the line capabilities in all of the above. Rather, I'd concentrate on the following specifics
The vendor who proposed more specific named individuals to the project, and you reviewed and liked their resumes, gets an edge over a vendor who only proposed general roles to be staffed at the time of the project kick off.
I joined Forrester recently as a senior forecast analyst on the ForecastView team focusing on business technology (BT) topics. What is ForecastView you ask? It’s a Forrester product that puts the numbers around our research reports by publishing a five-year quantitative outlook. To learn how our forecasts can help you with your investment decisions, read our ForecastView overview.
Our BT forecast team takes a look at cloud, security, IoT, business intelligence, marketing ad technology, Big Data, and other hot topics in the BT space. We launched our ForecastView BT bundle in 2015. In case you missed it, our three 2015 forecasts examined eCommerce platforms, cloud security, and API management. Some highlights:
Sizing The Cloud Security Market: Companies will spend $2 billion over the next five years to protect data in the cloud. We expect the market to grow at a staggering 40%+ CAGR over the next five years.
Modern application delivery leaders realize that their primary goal is to deliver value to the business and its customers faster. Most of the modern successful change frameworks, like Agile (in its various instantiations), Lean, and Lean Startup, which inspire developers and development shops, put metrics and measurement at the center of improvement and feedback loops. The objective of controlling and governing projects to meet vaguely estimated efforts but precisely defined budgets as well as unrealistic deadlines is no longer on the agenda of leading BT organizations.
The new objective of BT organizations is to connect more linearly the work that app dev teams do and the results they produce to deliver business outcomes. In this context, application development and delivery (AD&D) leaders need a new set of metrics that help them monitor and improve the value they deliver, based on feedback from business partners and customers.
Preproduction metrics. Leading organizations capture preproduction data on activities and milestones through productivity metrics, but they place a growing emphasis on the predictability of the continuous delivery pipeline, quality, and value.
Three of four architects strive to make their firms data driven. But well-meaning technology managers only deal with part of the problem: How to use technology to glean deeper, faster insight from more data -- and more cheaply. But consider that only 29% of architects say their firms are good at connecting analytics results to business outcome. This is a huge gap! And the problem is the ‘data driven’ mentality that never fights it’s way out of technology and to what firms care about - outcomes.
In 2016, customer-obsessed leaders will leapfrog their competition, and we will see a shift as firms seek to grow revenue and transform customer experiences. Insight will become a key competitive weapon, as firms move beyond big data and solve problems with data driven thinking.
Shift #1 - Data and analytics energy will continue drive incremental improvement
In 2016, the energy around data-driven investments will continue to elevate the importance of data and create incremental improvement in business performance. In 2016, Forrester predicts:
Chief data officers will gain power, prestige and presence...for now. But the long term viability of the role is unclear. Certain types of businesses, like digital natives, won’t benefit from appointing a CDO.
Machine learning will reduce the insight killer - time. Machine learning will replace manual data wrangling and data governance dirty work. The freeing up of time will accelerate data strategies.
You can't bring up semantics without someone inserting an apology for the geekiness of the discussion. If you're a data person like me, geek away! But for everyone else, it's a topic best left alone. Well, like every geek, the semantic geeks now have their day — and may just rule the data world.
It begins with a seemingly innocent set of questions:
"Is there a better way to master my data?"
"Is there a better way to understand the data I have?"
"Is there a better way to bring data and content together?"
"Is there a better way to personalize data and insight to be relevant?"
Semantics discussions today are born out of the data chaos that our traditional data management and governance capabilities are struggling under. They're born out of the fact that even with the best big data technology and analytics being adopted, business stakeholder satisfaction with analytics has decreased by 21% from 2014 to 2015, according to Forrester's Global Business Technographics® Data And Analytics Survey, 2015. Innovative data architects and vendors realize that semantics is the key to bringing context and meaning to our information so we can extract those much-needed business insights, at scale, and more importantly, personalized.