Kiba — Loading Data To Buy & Defi

In my last post, I showed how I collected Earn information from Binance US’s website using Mechanize. Now with the data captured, how do I load it to a database that my website,, can use?

What is KIBA?

Kiba is an ETL framework written in ruby that makes it simple to:

  • Extract information from a source, such as a database or file
  • Transform that information
  • Load the updated information to a new destination (database, file, etc)

Source (aka Extract)

My page scraper for Binance retrieved information from their website and stored it in an array. To define a source in Kiba, I created a class called Binance which contains 2 methods, initialize and each.

My initialize method is basically the entire mechanize code that visits Binance’s website and stores the data in an array. The each method will read each element of the array and pass it downstream to the transform/destination parts of the ETL job defined by Kiba. Ideally, each element of the array should be a hash.


This part of the ETL job will allow us to update each “row” of data so that it can be loaded to the destination. The data we received from the scraper, we want to do 2 things:

  1. Put the name in all capital letters
  2. Add a new key called exchange with the value “BINANCE”

To do this, I created a new class called BinanceTransform.

When I run the ETL job, I can see that my extracted data did indeed get transformed as:


To load the newly transformed data into the Buy And Defi database, I created a new called LoadBinance which consists of three methods

  • initialize — open a connection to the database
  • write — insert/update the record in the appropriate table
  • close — close the database connection

Note: Because the my ETL job will run on a daily basis, my insert query in the write method includes an on conflict condition.

With all the classes defined, I can simply have Github run the Kiba job(s) and voila! 26 rows have been added to my database.

Screenshot of database (interface courtesy

My ETL job, get_binance, can now run on a daily basis and update the Buy And Defi database for any rate changes or new coins added on Binance’s Earn page.

In my next article, I’ll share how I configured a Github action to run the ETL job on a daily basis.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store