DataGenic Blog

Commodity Data Management: Data Integrity and the Golden Copy Database

2011-09-08T16:12:00.010+01:00

Simon Jackson
COO, DataGenic Ltd.

Working for a busy software and services business in the data management space, I have noticed that over the past year or so, there has been a deluge of RFI/RFP for 3rd party or internal replacement data management platform and decision support systems in the European energy space. So have people woken up to the importance of data? Or suddenly become aware of the costs of managing data in their current systems? Or is this a tipping point similar to the banking sector in the early 90’s where a number of business factors (cost, competitive advantage, regulation) suddenly made data consolidation projects essentially mandatory?

Although each RFI has specific requirements, I see a common thread of functional and non-functional requirements - forward curve construction and management, some level of managed services to outsource the basic collection and management of external market data, a centralised data repository, pro-active notifications of quality and operational issues, increased data validation, ease of data integration, ongoing scalability, redundancy and business continuity and of course a requirement for high performance and availability.

That is a long shopping list. And one that internal departments have a hard time filling. Data integrity in all meanings – completeness, range, accuracy, timeliness, provenance – is now enjoying a more prominent position on the stage than ever before and companies are looking for solutions that can provide the holy grail of a highly flexible, low latency technical and functional framework for the realisation of their corporate ‘version of data truth’ so they can make this platform accessible to all parts of the organisation and start capturing data once and using it many times – rather than the ‘capture data every time you want to use it’ approach that was common in the past.

Data Integrity is not confined to the corporate business level or data stewards who manage the day to day processes. Different parts of the organisation hold different perceptions of integrity that are subject to their own real-time business processes and business activities. Although a ‘single version of truth’ is the ultimate goal of an organisation, multi golden copy databases can exist that provide each level of the organisation their own ‘version of truth’.

There are many organisations pursuing the goal of creating the perfect platform, each with their own view of what is important. I have always had a clear idea of what is important – that you understand what each number in the system actually means, where it came from and where it has been and is going. Over the past year, DataGenic has greatly enhanced its toolkit and framework to address this new level of sophistication:

Real-time multi-process (data, application and hardware) interactive dashboard that is user configurable.
Data Statistics dashboard and analysis on any dataset.
Data Alerts subscription to any workflow task, data correction, data validation and business rule (that highlights any ‘interesting’ data condition).
The use of Artificial Intelligence as the underlying framework for rules-based fluid data validation, triggered on events (data arriving, process handover) or time (scheduled).
Five levels of data tags to every data point in the database: colour coded, automated with custom views.
Four Eyes Principle (FEP) data staging area for second person approval on any process and data validation.
Data Set Package, allowing the true creation of golden copy databases.
Data Federation allowing a seamless view and use of data from multiple geographically dispersed repositories.
Data Provenance which allows any process used to create Data sets to be reproducible and analysable for defects.
Data Equivalence providing a relationship between any number of data models in the data repository.

Our markets clients don’t stand still, and neither do we!

Forward Curve Management...Why the Complexity?

2010-02-19T16:45:00.016Z

Richard Quigley
CEO, DataGenic Ltd.

Of all the issues and requirements that I come across in the industry, the winner by a country mile is forward curve management. This functional area concerns both buy and sell side organisations and is not restricted to a particular sector. Quite simply, if you are engaged in market trading activities that facilitate you buying, or selling forward, goods/services (physical or financial) the importance of forward curves is paramount for mark-to-market, risk exposure, key inputs for derivative pricing models, identification of future arbitrage opportunities and trader evaluation strategies.

What is difficult about forward curve management?

The calculation of a forward curve is not necessarily difficult in isolation. Any skilled Excel or VBA Excel user can take data and create some conditional expressions to form the calculation. However, the difficulty is accentuated when you have many different products (asset classes) that have different time horizons, different rollover and holiday calendars, different data integrity checks or different liquidity to name a few.

Some further factors that may compound the difficulties and complexities of forward curve construction and management for organisations are:

Forward curve methodology must adhere to compliance regulations - If you are a public listed company in the US or Europe, there is a regulatory requirement to ensure that any profit/loss forecasts use a robust, transparent and auditable methodology, which will require the forward curve for one of the key inputs.

Mitigate key-man risk - Many companies (large and small) still rely on 1 or 2 individuals to provide the forward curves to the Risk and Reporting System. Staff attrition, absence, etc., tend to upset the daily routine and reliability of the numbers.

Lack of any universal forward curve construction standards – Ask any two companies what their forward curve methodology is, for let’s say, a traded energy commodity, and you will get two different answers. Both can equally be correct as there is no defined right or wrong – just sensible, methodical and defined practice.

Forward Curve complexity – The characteristics of each market such as liquidity, volatility, market price sources or seasonality are different, and they require different modelling/calculation techniques.

Curve storage & views – A forward curve is no longer a time series and requires to be treated differently in a database environment to ensure that multiple curve views are available to the user. They may include Dated, Label, Tenor Point History, 3D History Surface.

It would be fair to say that most companies, big and small, struggle each day with forward curves, usually in an Excel environment, which comes with its own issues. We, at DataGenic, thought it was time to move away from Excel into a more stable, transparent, easy to use, auditable, easy to maintain application that can cope with any curve thrown at it!

Visit our website for further details on Genic CurveBuilder – it almost makes coffee for you in the morning!

datagenicgroup.com/our-products/genic-curvebuilder/features

2010: Forward Curves and beyond

2010-01-11T13:28:00.008Z

Richard Quigley
CEO, DataGenic Ltd.

A very happy new year to all our clients, partners and friends.

With 2009 now behind us and some embryonic optimism emanating from the media and global markets, this New Year has the potential to offer a more stable platform for business growth, than business consolidation in 2009 (for some). This was very evident in Q409 (traditional next year’s capex and opex budget setting time), with a notable number of clients with RFI/RFP’s for ETRM and data management solutions, with implementation starting Q1 and Q2 2010.

It is heartening for me to note that at least some organisations are placing value on data rather than treating it as an afterthought to the gilt-edged ETRM system. It is still not uncommon in the market to find organisations with multi-million pound - all singing and dancing - ETRM systems in place and yet are feeding this beast with an excel spreadsheet put together by a junior member of staff.

Fit for purpose software

Over the last 12 months, armed with the knowledge of the market requirements from our clients, our team has been working tirelessly to ensure our commitment through the release of Genic DataManager (GDM) 3.0 and two other major new optional module releases: Genic WorkFlow and Genic CurveBuilder.

As software releases go, GDM 3.0 is a major release for us, with a completely re-written front end in .NET, huge improvements in usability, as well as a host of new features. They include OLAP reporting, artificial intelligence, matrix management, scheduling, advanced formula editor, over 200 new formulas, reference data versioning, and many many more.

Genic WorkFlow has been used by our operations team for nearly a year, and we decided the software was road tested enough to provide to our clients. With visual processing mapping, workflow versioning, extensive auditing, full traceability and a host of other features, this latest optional module to GDM has been welcomed by our clients.

Genic CurveBuilder is certainly the most eagerly awaited of the 3 releases, offering a massive step-change over other 3rd party and client internally-built software for forward curve construction and management. Fully integrated with GDM and Genic WorkFlow, this module allows business users, with no IT experience, to build and manage complex forward curves. Furthermore, it offers deep audit logging, versioning and even a test area before you publish to the production application. This really is world-class designed software and I’m sure a demo of this application will prove what a world beater it is.

In November 2009, DataGenic won the prestigious Gold Award for Excellence in Energy Trading and Risk Management, which was a huge boost to the team’s effort over the past year. We will continue to develop innovative products, continuously listen to our clients and improve our support services in 2010. Our clients don’t stand still and neither do we.

Extreme Data Management

2009-05-01T12:05:00.006+01:00

Colin Hartley
CTO, DataGenic Ltd.

Backward and forward traceability; a small amount of text for what will essentially swamp your database with information - or will it?

Suppose I receive an assessment of Brent for the 1st May 2009 of $50, in a few days time, I want to see a report of everywhere that number is used: forward curves, reports, users viewing it, IPV, etc. If, further along, I see a report with the price for Brent on 1st May 2009 at $51, I will also want to see a complete trace back of that figure.

To be a responsible data provider, you need to be aware of who or what is using the data you provide and what they are using it for. To be a responsible data consumer, you need to have the ability to trace numbers back to see where they came from and what has happened to them on their journey.

As you can see, the major benefit of having this functionality is that a number is no longer a faceless digit, it has a history that is fully traceable both from where it came from and where it went.

This all adds up to the production of a lot of additional information for every single data point within a data management system and a hit on performance for every access (read and write) to data. With the ever decreasing cost of hardware, forever more powerful processors and capacious hard drives, this vision of data management utopia is not only possible, it is becoming mandatory for data quality aware users.

The challenge then, is making this tracking system as expeditious, practical and thorough as possible. It mustn't hinder the normal data management process by slowing it down or making it impractical to use, yet it must be thorough with the information that it stores.

The tracking information that is stored must be relatively easy to extract, but also secure from the possibility of tampering. After all, the main reason you are storing this information is so that you can blame someone for that dodgy report!

Please note, this functionality will be available in the next major release (v3.0) of Genic DataManager.

NY Fed Focuses on Data Management Practices

2009-04-17T16:01:00.009+01:00

Richard Quigley
CEO, DataGenic Ltd.

On the 23rd February 2009, a significant event happened in the Federal Reserve Bank of New York (NY Fed). It was not reported in our usual mainstream news, like CNN, BBC News etc.; and yet the significance of the event has not been lost by senior politicians on both sides of the Atlantic and data management practitioners alike - John Bottega, a reference data veteran and ex-Chief Data Officer for Citi's Institutional Client Group, was appointed as the as NY Fed's Chief Data Officer (CDO), Markets Division.

It's been suggested that this appointment is being linked with the change in administration in the US and President Barack Obama's focus on collaboration and transparency. However high up the food chain, this appointment is crucial and signals to the marketplace the importance of data management and its relationship to transparency and oversight.

So what are the responsibilities of a CDO? In general they are responsible for data strategy, policies, and data investments. Additionally, as Wikipedia advises, "Besides the revenue opportunities, acquisition strategy, and customer data policies, the Chief Data Officer is charged with explaining to executives, employees, and customers the strategic value of data and its important role as a business asset and revenue driver. This is in great contrast to the older view of data systems as back-end IT systems".

So what does this mean for other Central Banks? Given the current economic climate and the realisation of the inability of the world's central banks to regulate against excessive risk-taking in the financial system, this is the time to address the failings of the current data management practices, appoint a CDO and perhaps increase the data transparency that exists in the collection and sharing of data assets.

Are there lessons to be learned in the Energy Industry? John Bottega was the first appointed CDO in the finance industry, with many following in his footsteps. Data and data systems should be treated like any other strategic assets and not viewed simply as back-end IT systems.

Given the massive growth in data within the energy industry over the years, and especially with the impending Smart Metering Programs, it is paramount for organisations to provide focus and direction in this vital of assets - data! To even start thinking about data as a strategic asset is a step in the right direction - to actually appoint a CDO is the smartest of moves!

Asia 1st Anniversary

2009-03-13T14:16:00.022Z

Perlyn Per
Managing Director, Asia

To mark our successful one year anniversary operating in the Asian market, we hosted an evening cocktail party at the Swissotel, Stamford in Singapore, inviting industry peers in the energy, finance and public sectors and our data vendor partners to join in the celebration.

We are proud to share our success with our guests. Having been to many events over the years, I'm sure you share the same opinion that they tend to be very formal and even bordering on boring. We decided to be different, not for our sake but for our guests. Instead of a formal and boring presentation, we went for a quick walkthrough about what we do and took a 25 years Macallan whisky journey with our CEO, Richard Quigley who ended up showing a photo of himself at 19 years old. Boy...he had lots of hair back then!

We were very happy that many of our invited guests were able to attend despite their busy schedules. It is important to us that our guests felt it's worth the time and effort to be there with us... Everyone deserves a relaxing evening filled with good food and endless wine with loads of fun and laughter. To make the evening more exciting, we asked our guests to participate in a simple 'About DataGenic' quiz with the perspective of some cool prizes. Three of our prize winning guests went away happier that night!

1^st Prize
25 years old Macallan Whisky
Hoden Wan from Hypercube

2^nd Prize
Winner Aporgee Cross Pen
Wilson Wong from RBS Sempra Commodities

3^rd Prize
2001 Reserva Especial Rioja Red wine
Tang Chek Woo from Kim Eng Securities

We appreciated the compliments from our guests during the evening and indeed from the many emails we received and are pleased to hear that everyone enjoyed themselves. I must say, it makes our efforts all the more worthwhile. Our hope is that our guests take away the message that DataGenic is a fun company to partner with, and also a company that truly aims at delivering solutions that empower our clients and that address their business requirements.

As you are reading our blog now, I just want to thank you personally for taking the time to read. (Hey! Aren't you supposed to be working?)

Our very own professional DataGenic photographer, Jennee, has taken loads of candid shots. Warning: there is absolutely no censorship and no photo was edited... Viewer discretion is advised.

To view the photos, click here:
DataGenic Asia 1^st Anniversary Photo Gallery

Marketing dexterity will save the day

2009-02-27T15:14:00.008Z

Richard Quigley
CEO, DataGenic Ltd.

For anyone involved in the European energy industry, two weeks in February every year can easily become a blur. Endless meetings, parties, conferences are all part of the endurance for energy professionals at the two biggest events of the year: E-world (Essen, Germany) and IP Week (London, UK). I can still feel the effect of this 'marathon' even at the time of writing (some 2 weeks after)!

Although there is a degree of cross-over at the events, E-world remains very energy focused (power, gas, emissions) whilst IP Week concentrates for the most part on the biggest traded commodity on the planet: Oil. With the world in a deep recession for the foreseeable future, I was keen to immerse myself in both the buy and sell side of the business to understand the impact to our industry.

Understandably, you get the feeling of resilience and stubbornness that we can beat this unnecessary evil into submission and come out a stronger force. That was certainly evident at E-world which, impossibly, has gotten bigger than ever before (450+ exhibitors and 15000 delegates), and the exhibition stands seem to reach new heights, both physically and in marketing dexterity. Energy companies and vendors alike seem to be spending more on company promotion in this downturn, maybe trying to assure the market of their financial strength, credibility and future longevity. A friend of mine who works with one of the large European utilities said "my colleagues seem to be a lot more focused and friendly to our customers this year....before that they weren't so caring"! The statement was a bit 'tongue in cheek' but it certainly is a case that dominance and arrogance in the marketplace last year has been replaced with customer focus (client protection) and flexibility this year.

I certainly get the feeling from my discussions, that 2009 will be a tough year all round. On the buy side, all projects must provide beyond any reasonable doubt, a very tangible and indeed short-term ROI or have an operationally compelling argument to do so. The medium and long term projects seem to be on-hold (moth-balled) until conditions improve.

On the plus side from my point of view (sell-side), I consider these market conditions to be an opportunity to shine. Saving clients money, improving operational practices, increasing business responsiveness is what we do best - it's just a matter of using the same 'marketing dexterity' that was used by the exhibitors at the E-world show to highlight our uniqueness in the marketplace to our global audience.

Cut costs, not corners

2009-02-03T10:31:00.012Z

Richard Quigley
CEO, DataGenic Ltd.

As the global economy continues its relentless contraction, the costs of operating a company's services come under the microscope. Nothing focuses the CEO's mind better than the potential of business failure; a hostile takeover or simply losing market share.

In my many conversations over the past 17 years with traders, risk and IT management, there is one thing that I find bewildering and frustrating - the tacit acceptance of the old adage: "If it ain't broke, don't fix it." There have been countless times where the incumbent software package deployed is technically bereft, functionally poor and may even cost more than the nearest competitor. However, it may take a cleverer person than me to work out why the collective brain power and intellectual prowess of these talented individuals manifest itself into irrational exuberance?

Because a software package has "always been there" does not make it fit for today's challenges or tomorrow's unknowns. It doesn't make it effective or indeed efficient even in the right hands. Most conversations relating to why they won't buy another product tends to be around the "my staff don't have time to learn a new system", or "we find workarounds to the stuff we would have liked it to do", or "the account manager is great and does her best" or "Bob the trader likes it and he has a lot of power in the company". I've heard them all!

As the CEO of DataGenic, my mandate from the board is to ensure the future prosperity and profitability of our company. Thus, the challenge is fairly simple to me. Review the costs that are required to sustain this company in the tough period ahead and ensure that the services that we offer are value for money, whilst uncompromising in their richness and return on investment. As a services company in the data management arena, DataGenic is well positioned with our innate knowledge of software, utilities and services that are either low cost, or indeed free. We have ensured that our products can be offered to clients without additional Database or Web Server Licenses. We have also ensured that all our core development is undertaken in Bangalore, India, with exceptional quality control measures in place. Our low costs are passed on to our clients - in good times and in the bad.

So what have we achieved in 2008 and now offer to our clients for 2009?

Ensured our flagship product, Genic DataManager, is available in 3 Editions. Available now for a fraction of the Enterprise license costs for small to mid-size companies - who simply don't require all the bells and whistles.
Zero IT footprint, web-enabled desktop decision application for traders and data analysts for access to over 150 data sources from the world's leading exchanges, brokers and assessment providers.
Our managed data service, Genic DataHub, now offers clients the ability to simply download their licensed data directly to their own application or database server.
Full ASP/SaaS model offered to clients who simply understand that companies such as DataGenic have the ability to provide economies of scale with these services and give them a better deal than internal hosting. The added bonus is that system and data maintenance costs are shifted to the vendor (us) with a bespoke Service Level Agreement (SLA) in place to ensure minimal service requirements.
All software is architected for flexibility, scalability and performance. It is no longer acceptable that a system should be a 'black box' and comes with proprietary databases, and high level programming languages. It needs to be, and clients demand, Service Oriented Architecture (SOA).
On average over the last 3 years, 35% of our turnover is spent on Research and Development. At least 25% above the average for the industry and something we are proud of and our clients are enjoy.
Breaking the mould with our support offering: Our standard SLA provides client with a target resolution of 4 hours on Critical 1 issues.

My question to all CEO's, senior managers and business sponsors: Can you afford to accept the norm? Can you afford to accept that "If it ain't broke, don't fix it" in today's landscape? Isn't it time, we stop getting carried away with the latest 'management wisdom' that encourages the consolidation of vendors, thereby accepting mediocrity. The idea that your vendor partner of choice will develop in partnership with you and your business pace is a utopian view, not often realised in the real world. Encourage your managers to review the offerings from multiple vendors: small, medium and large size alike. You are after the best bang for your buck so why settle for second or third best?

You can cut costs in a global downturn but you don't have to cut corners!

An introduction

2009-01-27T12:49:00.003Z

This is the blog of DataGenic management. DataGenic an international specialist in the delivery of data management solutions and we'd like to share some of our thoughts and opinions with you here.