The Business Need
Based out of USA, a major multinational intelligence and technology providers powering business professionals with industry-specific business insights required validation of huge volumes of entity business data like company address, phone numbers, fax, contact person details, SIC code etc.
The data collected had to be enriched with additional business-specific details aggregated from various sources and the information had to be maintained fresh and up-to-date. Mobius’ technical assistance was thus required in not just collecting, but also validating, enriching, and refreshing the business database of the multinational information and media firm.
Challenges we faced
Since the necessary information had to be collected from a plethora of sources each different in its layout, our crawlers had to be tweaked in order to make them site-specific.
The entire data aggregation process had to be automated to quickly pull data from third-party websites. Each entity had the possibility of being crawled multiple times from multiple sites and we had to put a process in place for deduplication of records.
How we solved the problem
We built site-specific crawlers to aggregate the exhaustive list of data points from various websites. Care was taken to retrieve phone numbers and contact names for all the given sites. Similar company data points found on multiple sites could skew results. So, a thorough process of deduplication was applied to the raw crawled output.
With the aggregation automated, manual quality checks were performed rigorously to ensure completeness and quality of the data sourced.
Additionally, Mobius also aggregated the annual sales & employee base wherever applicable giving an enriched database to the customer.
Mobius ensured that rich and fresh business records were ever present in the client’s database. A voluminous total of 350,000 records were delivered to the client at record time.