Data Quality A Thankless Task — Until A Crisis Arises

By Rob Karel.

Quality data in the enterprise is like breathable air — you don't truly appreciate it until it's gone. Many companies don't even bother to ask whether the customer, product, asset, or any other data it captures is actually complete, valid, and useful. Other companies leave the responsibility of standardizing, cleansing, and aggregating data from source systems to their IT developers, perhaps leveraging transformation capabilities within extract, transform, and load (ETL) tools to automate this hygiene process.

Then there are those companies that have felt enough data quality-induced pain such as wasted marketing costs or low call center productivity, and have invested in data quality software that allows for the advanced definition and maintenance of rules to standardize, cleanse, enrich, match, and merge. Once an investment in data quality software is made, companies hopefully have invested also in staffing at least a handful of data quality stewards or business analysts. These data quality professionals (DQPs) can translate requirements and perspectives of quality from the business stakeholders to technical requirements that can be implemented within the DQ software.

Unfortunately, these DQPs are usually treated like the red-headed stepchild — they are expected to do their jobs with minimal executive support, funding, strategic priority, or recognition. It is assumed that because an investment in data quality software was made, there should no longer be a data quality issue. This is of course absurd: The technology is only as good as the data governance and business ownership of the data rules and definitions. If the business does not actively participate and take responsibility for the quality of data, the technology will be nothing more than yet another IT expense.

These underappreciated DQPs often spend a large percentage of their time trying to convince management to let them do what they were hired to do — improve the quality of data — but can't move the quality needle far enough through their own grassroots efforts. That is why it's ironic and sad that often a DQPs best friend may just be a data quality crisis. When senior executives learn that they have failed a compliance audit, a customer is suing for violation of their privacy preferences, or they have made a very bad strategic business decision based on faulty data, the DQP is suddenly very popular and subsequently receives plenty of priority, funding, and support.

Thankfully there are many organizations today that do put significant investment and priority in data quality, especially those in highly regulated industries like financial services and healthcare. But for the rest of the pack, data quality is still just a nice to have. And to the data quality professionals working for these laggard organizations: Keep up the good work, your time will come!

Categories:

Comments

re: Data Quality A Thankless Task — Until A Crisis Arises

Hi Rob:I couldn't agree more with some of the comments you made regarding companies and their view of data quality.I've seen it at a few companies in my career (both B2B and B2C alike), where record management, data hygiene, etc.. are viewed as "an unfortunate cost".The end result reminds me of the refrain used in the old FRAM oil filter TV Commercials "you can pay me now, or you can pay me later".The wasted cost resulting from inefficient marketing ROI alone is staggering, not to mention the higher cost at the point of a campaign, to do 'ad hoc' manual scrubbing and merge-purge procedures........ all because a company thought they were saving a few bucks upfront, by not having a thorough process in place.Keep the good thoughts coming!Mark Poklar

re: Data Quality A Thankless Task — Until A Crisis Arises

Data Quality issues and the people addressing them will in the future have the attention of top management sooner and earlier. Look at Amazon. The data Quality issues in Amazons product catalog "transport" the issue of Data Quality directly to the desk of every Consumer ( example and details in this post: http://www.line-of-reasoning.com/issues/data-quality-issues-cost-you-tim... )