Building Datasets: Experts Need Not Apply
There are three reasons why it's hard to feed data to spend analysis systems:
- Data feeds from different sources have incompatible formats;
- Data feeds consist of files with imperfect relations between them that cause transactions to be omitted from the dataset;
- Data feeds need translation and repair before loading.
This requirement for "data experts" means that traditional spend analysis products are always sold "on a tether" — that is to say, they are useless unless services personnel are packaged right along with the product. So, vendor personnel typically perform refreshes of data; they build and maintain datasets; and they get involved whenever there is a change of any kind.
But the problems don't stop there. Traditional spend analysis systems face several more hurdles even after raw data are loaded:
- The "dataset build" process is complex. Creating new dimensions and defining hierarchies requires the same "experts" that were needed in the data loading process;
- The build process takes many hours, and errors created by imperfect relations may cause multiple re-starts of the many-hour process;
- Once built, the dataset must be "published" to a server, a process that requires IT personnel to accomplish.
We felt there had to be a better way.
The BIQ Difference
We designed BIQ so that business users can load their own data and build their own datasets. Dozens of BIQ business users accomplish this magic daily. How do we know this? Because BIQ doesn't offer any "services on a tether." If you asked us for them, we'd stare at you blankly. In fact, BIQ is so easy to use that many of our users never took advantage of a BIQ training class at all — they just picked up the manual and taught themselves.
The Eureka moment, for us, was when we stopped thinking about data in "database" terms. Real-world data is messy. So, BIQ can easily transform data from one format to another. Joins are imperfect. So, BIQ is very tolerant of poor joins. Transactions must never be omitted. So, BIQ never omits transactions under any circumstance (which gladdens every accountant's heart), it simply groups them intelligently under an "Other" category. And (you've probably already guessed this) — BIQ doesn't use traditional database technology to build its cubes.
Our build process is very fast — we average 2-3 minutes for each 1M transactions on an ordinary desktop PC. Many changes to BIQ datasets take place instantaneously (such as hierarchy and rules changes) — as opposed to the multi-hour re-publish rigmarole that legacy systems require, whenever the slightest alteration is made. With BIQ, there isn't any "re-publish" or "cooking" step — BIQ datasets are always good to go.
Defining new dimensions and hierarchies is also accomplished by business users. BIQ flattens related files automatically into a big, wide "virtual" record. Any column of this record can be defined as a dimension or as a measure, and hierarchical relationships can be established between columns — or externally, within index files (BIQ already understands the most common types of computer-generated hierarchy indexes). New columns can be created within the virtual record, as functions of other columns. And it's all done with point-and-click tools that are easy to use and easy to understand.