Skip to content

Add fs.FS adapter for the filer interface#422

Merged
pietern merged 4 commits intomainfrom
filer-fs
Jun 2, 2023
Merged

Add fs.FS adapter for the filer interface#422
pietern merged 4 commits intomainfrom
filer-fs

Conversation

@pietern
Copy link
Copy Markdown
Contributor

@pietern pietern commented Jun 1, 2023

Changes

This enables the use of io/fs functions fs.Glob and fs.WalkDir with filers.

We can't use fs.FS as the standard interface instead of filer.Filer because:

  1. It was made for reading from filesystems only, not writing
  2. It doesn't take a context for the core functions

Therefore a wrapper will do.

Tests

  • Added unit tests to cover the adapter through a fake filer.
  • Manually ran fs.WalkDir against both WSFS and DBFS filers.

This enables use of fs.Glob and fs.WalkDir with filers.
libs/filer/fs.go Outdated

// Type that implements fs.File for a filer-backed fs.FS.
type fsFile struct {
fs *FS
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need FS here and not filer directly? If we want to abstract over filer and rely on FS, shall than define / use some interface?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the local FS struct, not the fs.FS interface. This struct contains both the filer and a context.Context. I agree this is a bit confusing and there's really no need to export the FS struct.

I renamed it to filerFs to unexport it and make it more clear it's a local type.

}

// If there are no more entries, return io.EOF.
if len(f.entries) == 0 {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we make it if f.entries == nil || len(f.entries) == 0? Otherwise if we let's say have n = 5 and len(f.entries) = 4, we will execute condition on line 132 which will set f.entries to nil and on the next run len(f.entries) will fail

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm, there is a test for this and f.entries = nil for slice just makes it empty slice

}

// If there are no more entries, return io.EOF.
if len(f.entries) == 0 {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm, there is a test for this and f.entries = nil for slice just makes it empty slice

@pietern pietern enabled auto-merge (squash) June 2, 2023 12:39
@pietern pietern merged commit 1c0d67f into main Jun 2, 2023
@andrewnester andrewnester deleted the filer-fs branch June 2, 2023 13:18
@pietern pietern mentioned this pull request Jun 12, 2023
pietern added a commit that referenced this pull request Jun 12, 2023
## Changes

CLI:
* Add directory tracking to sync
([#425](#425)).
* Add fs cat command for dbfs files
([#430](#430)).
* Add fs ls command for dbfs
([#429](#429)).
* Add fs mkdirs command for dbfs
([#432](#432)).
* Add fs rm command for dbfs
([#433](#433)).
* Add installation instructions
([#458](#458)).
* Add new line to cmdio JSON rendering
([#443](#443)).
* Add profile on `databricks auth login`
([#423](#423)).
* Add readable console logger
([#370](#370)).
* Add workspace export-dir command
([#449](#449)).
* Added secrets input prompt for secrets put-secret command
([#413](#413)).
* Added spinner when loading command prompts
([#420](#420)).
* Better error message if can not load prompts
([#437](#437)).
* Changed service template to correctly handle required positional
arguments ([#405](#405)).
* Do not generate prompts for certain commands
([#438](#438)).
* Do not prompt for List methods
([#411](#411)).
* Do not use FgWhite and FgBlack for terminal output
([#435](#435)).
* Skip path translation of job task for jobs with a Git source
([#404](#404)).
* Tweak profile prompt
([#454](#454)).
* Update with the latest Go SDK
([#457](#457)).
* Use cmdio in version command for `--output` flag
([#419](#419)).

Bundles:
* Check for nil environment before accessing it
([#453](#453)).

Dependencies:
* Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0
([#459](#459)).
* Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19
([#412](#412)).

Internal:
* Add Mkdir and ReadDir functions to filer.Filer interface
([#414](#414)).
* Add Stat function to filer.Filer interface
([#421](#421)).
* Add check for path is a directory in filer.ReadDir
([#426](#426)).
* Add fs.FS adapter for the filer interface
([#422](#422)).
* Add implementation of filer.Filer for local filesystem
([#460](#460)).
* Allow equivalence checking of filer errors to fs errors
([#416](#416)).
* Fix locker integration test
([#417](#417)).
* Implement DBFS filer
([#139](#139)).
* Include recursive deletion in filer interface
([#442](#442)).
* Make filer.Filer return fs.DirEntry from ReadDir
([#415](#415)).
* Speed up sync integration tests
([#428](#428)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants