Simon JacksonCOO, DataGenic Ltd.
Working for a busy software and services business in the data management space, I have noticed that over the past year or so, there has been a deluge of RFI/RFP for 3rd party or internal replacement data management platform and decision support systems in the European energy space. So have people woken up to the importance of data? Or suddenly become aware of the costs of managing data in their current systems? Or is this a tipping point similar to the banking sector in the early 90’s where a number of business factors (cost, competitive advantage, regulation) suddenly made data consolidation projects essentially mandatory?
Although each RFI has specific requirements, I see a common thread of functional and non-functional requirements - forward curve construction and management, some level of managed services to outsource the basic collection and management of external market data, a centralised data repository, pro-active notifications of quality and operational issues, increased data validation, ease of data integration, ongoing scalability, redundancy and business continuity and of course a requirement for high performance and availability.
That is a long shopping list. And one that internal departments have a hard time filling. Data integrity in all meanings – completeness, range, accuracy, timeliness, provenance – is now enjoying a more prominent position on the stage than ever before and companies are looking for solutions that can provide the holy grail of a highly flexible, low latency technical and functional framework for the realisation of their corporate ‘version of data truth’ so they can make this platform accessible to all parts of the organisation and start capturing data once and using it many times – rather than the ‘capture data every time you want to use it’ approach that was common in the past.
Data Integrity is not confined to the corporate business level or data stewards who manage the day to day processes. Different parts of the organisation hold different perceptions of integrity that are subject to their own real-time business processes and business activities. Although a ‘single version of truth’ is the ultimate goal of an organisation, multi golden copy databases can exist that provide each level of the organisation their own ‘version of truth’.
There are many organisations pursuing the goal of creating the perfect platform, each with their own view of what is important. I have always had a clear idea of what is important – that you understand what each number in the system actually means, where it came from and where it has been and is going. Over the past year, DataGenic has greatly enhanced its toolkit and framework to address this new level of sophistication:
- Real-time multi-process (data, application and hardware) interactive dashboard that is user configurable.
- Data Statistics dashboard and analysis on any dataset.
- Data Alerts subscription to any workflow task, data correction, data validation and business rule (that highlights any ‘interesting’ data condition).
- The use of Artificial Intelligence as the underlying framework for rules-based fluid data validation, triggered on events (data arriving, process handover) or time (scheduled).
- Five levels of data tags to every data point in the database: colour coded, automated with custom views.
- Four Eyes Principle (FEP) data staging area for second person approval on any process and data validation.
- Data Set Package, allowing the true creation of golden copy databases.
- Data Federation allowing a seamless view and use of data from multiple geographically dispersed repositories.
- Data Provenance which allows any process used to create Data sets to be reproducible and analysable for defects.
- Data Equivalence providing a relationship between any number of data models in the data repository.
