Extreme Data Management

Colin Hartley
CTO, DataGenic Ltd.

Backward and forward traceability; a small amount of text for what will essentially swamp your database with information - or will it?

Suppose I receive an assessment of Brent for the 1st May 2009 of $50, in a few days time, I want to see a report of everywhere that number is used: forward curves, reports, users viewing it, IPV, etc. If, further along, I see a report with the price for Brent on 1st May 2009 at $51, I will also want to see a complete trace back of that figure.

To be a responsible data provider, you need to be aware of who or what is using the data you provide and what they are using it for. To be a responsible data consumer, you need to have the ability to trace numbers back to see where they came from and what has happened to them on their journey.

As you can see, the major benefit of having this functionality is that a number is no longer a faceless digit, it has a history that is fully traceable both from where it came from and where it went.

This all adds up to the production of a lot of additional information for every single data point within a data management system and a hit on performance for every access (read and write) to data. With the ever decreasing cost of hardware, forever more powerful processors and capacious hard drives, this vision of data management utopia is not only possible, it is becoming mandatory for data quality aware users.

The challenge then, is making this tracking system as expeditious, practical and thorough as possible. It mustn't hinder the normal data management process by slowing it down or making it impractical to use, yet it must be thorough with the information that it stores.

The tracking information that is stored must be relatively easy to extract, but also secure from the possibility of tampering. After all, the main reason you are storing this information is so that you can blame someone for that dodgy report!

Please note, this functionality will be available in the next major release (v3.0) of Genic DataManager.