Frederick the Great reportedly said, “An army marches on its stomach.” In the same way, a BI project relies on the data it uses for reporting. Fundamental issues, like poor data quality, can set a course for failure before you’ve created a single report.
A Variety of Data Challenges
When customers start an embedded BI project, they tend to have a good understanding of what challenges they may face. When scoping requirements, they’ve already considered what data issues have the potential to cause problems.
And that’s generally a smart practice. There are so many ways data can derail your BI project. It could be something as simple as getting access to a data source. If you have to wait for someone in another department to grant you a connection string to their database, that’s going to eat up a lot of time at the start of the project.
More likely, the challenges will come from things like trying to report off of many diverse data sources – and these challenges go beyond simply connecting to them.
For example, you may need to query an application like Salesforce which requires its own connector. And then, on top of that, running queries against a data warehouse. Throw in a few Excel files and suddenly things start to get very complicated very quickly.
Multiple data sources aside, having a real understanding of the data can be a heavy lift. Consider finding the source of truth for a given field The data in database one’s AccountName field may be quite different than account.name on database two. Or it might even map to a completely different field, say customer_name on database two. Which of these is the ultimate true name field?
There may be different validation on these fields, depending on the application that actually writes to them. Perhaps the field is required on one database, and not on the other.
When it comes to reporting analytics on a field like this, outcomes could be quite different depending on how the databases are queried. If there is no data governance in effect, you can’t necessarily trust the data.
Data Quality Issues
Actual data quality is a whole other issue. To quote one redditor: “Data is less pretty than the customers think.” You are (hopefully) testing your implementation using a QA or sandbox environment, but that may not give you a view into poor quality or non-reporting-ready data that exists in your production system.
Alternately, a team trying to get a BI system in place may not have access to a rich set of test data. A tiny subset of data is never going to reveal realistic query speeds. Any query is going to be lightning-fast when your test database only contains 1000 rows.
It’s not unusual for an analytics project to require database changes. An OLTP database will likely be optimized for CRUD, not reporting. Optimization for analytics might require simply developing some reporting-friendly views, but it could snowball into larger updates, whether to individual fields or entire data sources.
Usually, we look to denormalize databases for OLAP purposes, but our team has seen examples of databases that have not been properly normalized in the first place. That may not have been a problem when writing transactions to the file, but needs to be fixed before you can create analytics on those data points.
If the database is a legacy one, it may not be supported as a reporting source. Our analytics platform, for example, no longer supports versions of MySQL 5.5 or older. Of course, if you are trying to report off of a database that has not been updated in over ten years, that in itself should throw up some red flags.
Finally, even with clean, up-to-date and optimized data sources, you can still have reporting-related issues. You may have a deep understanding of your data operationally. But reporting against it is a matter of seeing the proverbial forest from the trees, so you can extract a view of the data that will reveal trends and display them in a dashboard or report. That requires understanding the kind of reports your customers need, and the primary data points they are looking at. And of course, you also will need to help your application’s users understand their new reporting system – and appreciate the value it delivers.
Izenda: A Partnership Approach to BI
At Izenda, we’re committed to ensuring that each consultation, demo, and trial brings value to the organization. And we strive to set everyone up for success – whether or not we move forward. If your company is beginning a BI project and looking for guidance, feel free to schedule a call and demo with one of our BI professionals. Just follow the link below to get started.
