Repository navigation
Commit b997361
feat: Add dbt integration for importing models as FeatureViews (#5827)
* feat: Add dbt integration for importing models as FeatureViews (#3335)
This PR implements the dbt-Feast integration feature requested in #3335,
enabling users to import dbt models as Feast FeatureViews.
## New CLI Commands
- `feast dbt list` - List dbt models available for import
- `feast dbt import` - Import dbt models as Feast objects
## Features
- Parse dbt manifest.json files to extract model metadata
- Map dbt types to Feast types (38 types supported)
- Generate Entity, DataSource, and FeatureView objects
- Support for BigQuery, Snowflake, and File data sources
- Tag-based filtering (--tag) to select specific models
- Code generation (--output) to create Python files
- Dry-run mode to preview changes before applying
## Usage Examples
```bash
# List models with 'feast' tag
feast dbt list -m target/manifest.json --tag feast
# Import models to registry
feast dbt import -m target/manifest.json -e driver_id --tag feast
# Generate Python file instead
feast dbt import -m target/manifest.json -e driver_id --output features.py
```
Closes #3335
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Address mypy and ruff lint errors in dbt integration
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Address ruff lint errors in dbt unit tests
Signed-off-by: yassinnouh21 <[email protected]>
* style: Format dbt files with ruff
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Remove unused dbt-artifacts-parser import and fix enum import
Signed-off-by: yassinnouh21 <[email protected]>
* feat: Use dbt-artifacts-parser for typed manifest parsing
- Add dbt-artifacts-parser as optional dependency (feast[dbt])
- Update parser to use typed parsing with fallback to raw dict
- Provides better support for manifest versions v1-v12
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Add graceful fallback for dbt-artifacts-parser validation errors
When parsing minimal/incomplete manifests (e.g., in unit tests),
dbt-artifacts-parser may fail validation. This change adds a graceful
fallback to use raw dict parsing when typed parsing fails.
Also updated test fixture with dbt_schema_version field.
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Skip dbt tests when dbt-artifacts-parser is not installed
Since dbt-artifacts-parser is an optional dependency, unit tests
should be skipped in CI when it's not installed.
Signed-off-by: yassinnouh21 <[email protected]>
* refactor: Simplify parser to rely solely on dbt-artifacts-parser
Removed manual/fallback dict parsing code. The parser now exclusively
uses dbt-artifacts-parser typed objects. Updated test fixtures to
create complete manifests that dbt-artifacts-parser can parse.
Signed-off-by: yassinnouh21 <[email protected]>
* ci: Add dbt-artifacts-parser to unit test dependencies
Install dbt-artifacts-parser in CI so dbt unit tests run instead
of being skipped.
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Address Copilot code review comments for dbt integration
- mapper.py: Fix Array element type check to use set membership instead
of incorrect isinstance() comparison
- codegen.py: Add safe getattr() with fallback for Array.base_type access
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Only add ellipsis to truncated descriptions
Signed-off-by: yassinnouh21 <[email protected]>
* style: Format dbt files with ruff
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Convert doctest examples to code blocks to avoid CI failures
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Add dbt-artifacts-parser to feast[ci] and update requirements
- Add dbt-artifacts-parser to pyproject.toml under feast[dbt] and feast[ci] extras
- Remove separate install step from unit_tests.yml workflow
- Update all requirements lock files
Addresses review feedback from @ntkathole.
Signed-off-by: YassinNouh21 <[email protected]>
Signed-off-by: yassinnouh21 <[email protected]>
* docs: Add dbt integration documentation
Add comprehensive documentation for the new dbt integration feature:
- Quick start guide with step-by-step instructions
- CLI reference for `feast dbt list` and `feast dbt import`
- Type mapping table for dbt to Feast types
- Data source configuration examples (BigQuery, Snowflake, File)
- Best practices for tagging, documentation, and CI/CD
- Troubleshooting section
Addresses review feedback from @franciscojavierarceo.
Signed-off-by: YassinNouh21 <[email protected]>
Signed-off-by: yassinnouh21 <[email protected]>
* docs: Add alpha warning to dbt integration documentation
Add prominent warning callout highlighting that the dbt integration is
an alpha feature with current limitations. This sets proper expectations
for users regarding:
- Supported data sources (BigQuery, Snowflake, File only)
- Single entity per model constraint
- Potential for breaking changes in future releases
Addresses feedback from PR #5827 review comments.
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Add dbt-artifacts-parser to CI_REQUIRED dependencies
Ensure dbt-artifacts-parser is installed in CI environments by adding
it to the CI_REQUIRED list in setup.py. This matches the dependency
already present in pyproject.toml and ensures CI tests for dbt
integration have access to the required parser library.
Addresses feedback from PR #5827 review comments.
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Add defensive Array.base_type handling with logging
Add logging and defensive attribute access for Array.base_type in code
generation to prevent potential AttributeError. While Array.__init__
always sets base_type, defensive programming with warnings provides:
- Protection against edge cases or future Array implementation changes
- Clear visibility when fallback occurs via logger.warning
- Consistent error handling across both usage sites
Changes:
- Add logging module and logger instance
- Update _get_feast_type_name() to use getattr with warning
- Update import tracking logic to use getattr with warning
- Add concise comments with examples (e.g., Array(String) -> base_type = String)
Addresses code review feedback from PR #5827.
Signed-off-by: yassinnouh21 <[email protected]>
* docs: Add comment explaining ImageBytes/PdfBytes exclusion
Add clarifying comment in type_map explaining why ImageBytes and
PdfBytes are not included in the dbt type mapping. While these types
exist in Feast, dbt manifests only expose generic BYTES type without
semantic information to distinguish between regular bytes, images, or
PDFs.
Example: A dbt model with image and PDF columns both appear as
'BYTES' in the manifest, making ImageBytes/PdfBytes types unmappable
from dbt artifacts.
Addresses feedback from PR #5827 review (franciscojavierarceo).
Signed-off-by: yassinnouh21 <[email protected]>
* fix: Move imports to top of file to resolve linter errors
- Fix E402 linter error in feast/dbt/codegen.py by moving imports before logger initialization
- Update requirements files to include dbt-artifacts-parser in pydantic dependency comments
Signed-off-by: yassinnouh21 <[email protected]>
---------
Signed-off-by: yassinnouh21 <[email protected]>
Signed-off-by: YassinNouh21 <[email protected]>
Co-authored-by: Francisco Javier Arceo <[email protected]>1 parent f6116f9 commit b997361
17 files changed
Lines changed: 2692 additions & 0 deletions
File tree
- docs
- how-to-guides
- sdk/python
- feast
- cli
- dbt
- requirements
- tests/unit/dbt
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| 79 | + | |
79 | 80 | | |
80 | 81 | | |
81 | 82 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
0 commit comments