The Business Need
A top information and analytics providers for the energy and commodities markets required data on changing prices, shipping, energy consumption, and oil transmission across many countries to be gathered from more than 400 government, trading websites and emails per day. Data thus collected would be further used for predictions in price trends, risk mitigation, and market analysis. Mobius provided a fully automated solution to aggregate all the commodity data points at real-time.
Challenges we faced
Since critical business decisions and insights relied on the pace at which information was being fed, on-time updates on various fluctuating data points had to be fetched with great accuracy and quality. Hence constant monitoring and tracking of information sources had to be performed, circumventing issues in site access and email extraction.
Pricing details pertaining to oil, gas, metals, and nuclear power were identified by the customer to be obtained from 400+ government and trading sites and 150 email tasks.
To achieve near real-time data dispense, an automated approach was adopted and scripts were deployed to gather and amass more than 3500 attributes in 400+ output files.
Data thus aggregated was then delivered via well-secured FTP server in .csv and .xls formats within 2 minutes from the time of update in the information sources.
Despite frequent site layout changes, the varying commodities data values were aggregated and delivered in less than 2 minutes of source update with a consistent and astounding accuracy of 99.8%.