This is an example of using dbt to enrich data in BigQuery with LLM model.
- Python 3.11
- GCP project
Create a connection of Cloud Resource type in BigQuery:
Replace
YOUR_REGION
andYOUR_PROJECT_ID
with your own values
bq mk --connection --location=YOUR_REGION --project_id=YOUR_PROJECT_ID --connection_type=CLOUD_RESOURCE cloud_resources_connection
Please follow the instructions in remote_functions/call_llm_model/README.md
Create a virtual environment:
python -m venv venv
source venv/bin/activate
Install dependencies:
pip install -r requirements.txt
Setup dbt profile in ~/.dbt/profiles.yml
Test the connection:
dbt debug
Run dbt models:
Before running the models, please make sure that you have created a connection of Cloud Resource type in BigQuery and that you have created a Cloud Function
call_llm_model
in your GCP project.
dbt build
Special thanks to Piotr Chaberski for reviewing the code and providing valuable feedback.
This project is licensed under the MIT License - see the LICENSE file for details