A fully automated ETL pipeline that drive data-driven decisions
About MakerDAO
Industry Context
MakerDAO dApp is an attractive source of high-quality business data like cryptocurrency rates, exchange rates, interest rates showing fluctuations in the market, etc. All this information may become very handy for effective decision making, if properly structured and presented. But this is a challenging process, since:
Data sourced from blockchain is highly technical and detailed.
Manually gathering crypto related data is a daunting task, highly prone to data loss and inaccuracy.
Collecting large datasets for building numerous reports and analytics requires specific skills, software, and financial costs.
The organization of data transport could be implemented using traditional means (through the use of an API) or the construction of an ETL (Extract, Transform, and Load) pipeline.
ETL scripts are flexible yet very powerful tools for collecting, structuring, and presenting blockchain data. In order to set up an ETL pipeline, one should have a solid knowledge of the software environment for executing the ETL scripts, databases, and blockchain operations.
Due to the sheer demand for this expertise, MakerDao turned to Unicsoft.
On the part of the Unicsoft team, a highly-skilled data scientist was committed to working on this project. He made a great effort to create an automated ETL pipeline:
01
Developed a system that takes blockchain data, integrates and processes it, and creates data repositories that are used for fast financial reporting.
02
Integrated the pipeline’s scripts with data sources and transformed the data into the formats required for storage in the client’s central database.
03
Synchronized work on the Flask server and APScheduler to avoid data duplication, improved the client’s scripts, and wrote a new method for loading data to the Postgres database.
04
Once the scripts were sorted out, analysis became easy and the entire process rolled along with end-to-end automation.
At the end of their collaboration with Unicsoft, MakerDAO got an ETL pipeline with a built-in web interface to monitor how it works and set up custom reports. It’s a solution that assists with extracting data from different sources, transforms it into the required format, and loads it into the client’s database. Today, the entire process is automated, such that it requires minimal support from the client’s side.
It’s a solution that assists with extracting data from different sources, transforms it into the required format, and loads it into the client’s database.
“They quickly reacted to our request and provided an interesting suite of candidates. During the cooperation period, they were always available and ready to help. Our overall experience was very good. Highly recommended.”
