Owned by NKM National Public Utilities Co. Ltd., which has been in operation for over 160 years, NKM Földgázszolgáltató Co. is a company that provides its customers a framework of free market gas trade and universal service.
As a natural gas supplier of the entire Hungarian population, NKM Földgázeveató Zrt. Currently provides a continuous supply to more than 3.4 million customers. Thanks to its successful operation so far, NKM Földgázszolgáltató Co is the only player in the natural gas industry that has won the Superbrands, Business Superbrands and MagyarBrands awards.
Our work with NKM
Back in February 2018, NKM requested that we provide a data validation service on a 150 GB data set of gas consumption data. This data was provided by the external service provider E.ON.
As part of the international E.ON Group, E.ON Hungary is an important participant in the Hungarian energy industry. The Group's activities focus on providing complex and convenient energy services to customers, operating energy distribution networks (electricity and gas distribution networks), and renewable energies (including solar energy in particular).
Our service included the downloading of the data, and data transformation.
The downloaded data was in CSV format, and the receiving system was written in MSSQL. The data cleansing and transformation was laid out in their native system, and the entire data included 600,000 POD (point of delivery).
What We Did
To achieve success, we followed the steps below
- Specification of the data format
We had an effective meeting where we were able to talk to one of their experts, who could answer our questions on the required specifications of the reviewing architect
- Next, we created the data validation rules and systems.
- Then, we created the data cleansing scenario.
- Finally, we carried out the ETL process.
We were able to finish the whole data transformation in two days on our servers. Also, we used Amazon Cloud for the data processing after which we integrated the desired output right into NKM’s database.
Since then, we have been working side by side with NKM and even have a new contract starting February 2019.
Energy Forecast Ltd approached us and requested for a solar power production forecast. we were asked to prepare the variable features that can be used for the compilation of historical forecast for temperature, cloud coverage, wind speed, and wind direction.
For this we created a special crawler that collected data from:
- darksky: https://darksky.net
- windguru (here we are also subscribed for a premium package): https://www.windguru.cz
- windfinder: https://www.windfinder.com
- meteoblue: https://www.meteoblue.com
After gathering the data, we converted it into a unified, easily analyzable format. The target system was Postgre SQL.
Another big part of our data came from satellite images (weather.us). We automatically downloaded the images taken over the immediate past 5 years at 15 minutes intervals. After downloading them, we wrote an algorithm that was able to translate the data collected from the images into numerical numbers for easier analysis.
We delivered successfully. Since closing this project, we have kept a good relationship with Energy Forecast and have even worked together on several other projects similar to this one.
We worked together for about 4 months starting in March 2017 to create a basic system for real time data production.
We were already working for MVM’s market analysis department on a side project when their energy trading department, after hearing about our work, signed up too. They wanted us to create a system that can provide them real time data so as to enable them trade better.
You see before us, they were manually downloading all the data they needed, data such as weather, temperature, water level report, wind data and forecasts. Also, they were collecting exchange rates, stock prices, consumption needs, and electricity consumption data too. In order for them to have all these at hand, they spent more than one hour gathering these data from multiple sources once a day, at 8 a.m.
In addition to the time wasted gathering this info, another main problem they faced was that the data wasn’t comprehensive or unified. After gathering all the data, they still needed to process them according to source.
Our service helped them solve this problem.
What We Did
To achieve success, we followed the steps below:
- Survey of the existing architecture and systems – we discovered that they were using ORACLE based databases.
- Specification regarding the data set – after our meetings, we discovered that the below information was what they needed
- ENTSO-E - Database of the European Association of Transmission System Operators for Electricity, where system operators in European countries are constantly publishing planned and actual data on the electricity market
- Wattsight - Electricity production data, the successor of the former MKOnline database
- WSI - Historical and predicted weather data that are essential for more sophisticated energy market models
- JAO - Database of cross-border transmission capacities of the European electricity market
- ICE - Energy Market Trading Forecasts
- We integrated the data with their existing architecture and data configuration as they didn’t want it altered.
We created a data downloading solution that could update their database automatically every 5 minutes, as well as save the prehistoric data from the previous 3 hours. Thus, the loss of power plant capacity could be integrated into the traders' strategy in almost real time.
We downloaded nearly 1 GB of data a day, which we integrated with their existing systems.
Two Years Together
We've been working together ever since. In fact, in 2018, they ordered some more support service from us. We handle things such as updating and shipping changes in data sources API interface (server side), and the constant logging of data. On the service provider side, we automated the correction of incorrect data, such as duplication).
When you work with us, we always provide you dedicated personnel who will serve as a point of contact and handle all communication so that you are able to reach us whenever necessary - This was also the case here.
Right now, we are working together to add newer sources of data to the system.