Skip to main content

dbt Integration

Overview​

Omni integrates closely with dbt (both dbt Core and Cloud). There are three core ways to use the dbt integration today (and more coming soon!):

  1. Schema refreshes to sync dbt changes
  2. Pull dbt context and metadata into Omni
  3. Author dbt models from Omni queries

Schema refreshes to sync dbt changes​

Schema refresh syncs downstream changes from dbt into your Omni model, pulling in field changes, new fields, new tables, etc. This eliminates the need to manually update the Omni model every time something changes in the layer below. To kick off a refresh, select 'Refresh Schema' from the IDE model menu. Learn more about schema refreshes here.

Pull dbt context and metadata into Omni​

Pulling metadata from dbt into Omni makes it easier for analytics engineers, analysts, and business users to share context about what fields and tables mean, ensuring they're using the right things in their queries (and making tracing logic throughout the data stack very easy!). Omni pulls the following information from dbt:

  • dbt model and field descriptions are brought into the corresponding Omni view files. Descriptions also become viewable in the workbook field picker
  • dbt SQL code is brought into the corresponding Omni view files. This makes it easy to trace logic between the tools
  • dbt dependencies are also brought into the corresponding Omni view files

Author dbt models from Omni queries​

Omni is great for both building analyses and data models. Every time you construct a query - select columns, aggregate, filter, add joins - you're creating the building blocks of a model. While not every query needs to be pushed down to dbt, when you do create a query where you want to promote the logic into dbt, you can easily do so from within Omni.

  • Within a workbook tab, there is an option under the Model menu to ‘Push to dbt’. This will open a new pull request in your dbt git repository with the SQL used to generate the query.
  • Once merged in dbt, the new model will show up inside Omni upon refreshing schema from the Omni IDE.

Looking ahead, we have plans to further flesh out this workflow- allowing users to trigger dbt runs from Omni, switch between dev and prod dbt schemas, and validate content against dbt changes.

Steps for Connecting dbt to Omni​

  1. Inside your Omni connection, fill out the dbt connection form:
    • Git SSH URL can be found from your git provider (make sure to use the SSH one)
    • Git branch is typically main or master
    • Target schema is the schema you used to configure dbt
      • If you're using dbt Cloud, you can find this under Deployment Environments > Prod > Settings > Deployment Credentials > Schema
      • If you're using dbt Core, you can often find this from the profiles.yml file under 'Schema' (for BigQuery, this is sometimes called 'dataset' instead)
    • Hit ‘Submit’ and Omni will provide you a public key

  1. Follow the instructions from dbt to add a deploy key to your git provider. If you want to use Omni’s ‘push to dbt’ integration, make sure you allow write access. You’ve now connected Omni to your dbt repo.

  2. Back on the Omni connection page, hit ‘Refresh Schema.’ This will sync the metadata from the dbt schema.yml file to your Omni model. The schema.yml file is what dbt uses to generate their documentation. Example dbt schema.yml file:

  1. dbt model descriptions will be added to the matching Omni view files. These are also exposed in Omni’s field picker
  1. If metadata changes in dbt, you’ll need to hit ‘refresh schema’ again in Omni (either from the connection page, or the model) to resync. We will be automating this in the future.