Running KIBA as a Github Action

Now that I have gone over how I built an ETL job using KIBA, let’s look at how I leverage Github Actions to run the jobs on a daily basis.

Setting Up Environments

As I continue to redesign buyanddefi.com, I want to set up different environments to test my etl code and other future feature. Fortunely, Bit.io’s free plan helps me to set up the 3 basic environments for future CI plans (local, staging, production). For local development, I’ll describe how I used the dotenv gem to manage my environment in a future article.

STAGING ENVIRONMENT (ETHEREUM)

For my staging environment, I want to run the etl job every time code gets merged to the develop branch. So, my workflow file starts off like this:

name: ETL Workflow On Ethereum
on:
  pull_request:
    branches:
      - develop
    types: [closed]

Now, I need to set up the environment variables for this workflow to connect to the Ethereum environment. On a paid plan of Github, there is an option called environments when you go the repository’s settings.

After creating the Ethereum environment, you will be provided a section to configure the environment variables. You are provided two options when setting up your environment variables, Environment Secrets and Environment Variables.

Based on the description provided, I decided to just set up 3 secrets and provide the following set up so that it is accessible from the workflow.

jobs:
  etl:
    runs-on: ubuntu-latest
    environment: ethereum
    env:
      DB: '${{ secrets.DB }}'
      DBPASS: '${{ secrets.DBPASS }}'
      DBUSER: '${{ secrets.DBUSER }}'
      SSLMODE: 'require'
      HOST: 'db.bit.io'
    steps:
      - uses: actions/checkout@v1
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.0' # Not needed with a .ruby-version file
          bundler-cache: true # runs 'bundle install' and caches installed gems automatically
      - run: bundle exec ruby etl.rb

The etl job will run on the ethereum environment (aka staging) with the environment variables defined from secrets I entered into the Github repository. In addition, two additional environment variables I’ve defined in the yaml file will be included (SSLMODE & HOST).

PRODUCTION ENVIRONMENT (BITCOIN)

The only difference in the Bitcoin (production) environment/workflow are two things:

1. The workflow is scheduled to run as a daily cron job
2. A new environment is set up for the production environment

name: ETL Workflow On Ethereum
on:
  schedule: ## Run job everyday at 1400 UTC
    - cron: "0 14 * * *"
jobs:
  etl:
    runs-on: ubuntu-latest
    environment: bitcoin
    env:
      DB: '${{ secrets.DB }}'
      DBPASS: '${{ secrets.DBPASS }}'
      DBUSER: '${{ secrets.DBUSER }}'
      SSLMODE: 'require'
      HOST: 'db.bit.io'
    steps:
      - uses: actions/checkout@v1
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.0' # Not needed with a .ruby-version file
          bundler-cache: true # runs 'bundle install' and caches installed gems automatically
      - run: bundle exec ruby etl.rb

Conclusion

From my two examples, you can now configure Github actions to run an ETL job when you either merge code to a branch or set it up to run on a daily basis. Feel free to revisit my previous article on how I built the ETL job using Kiba.