Add fs ls command for dbfs by shreyas-goenka · Pull Request #429 · databricks/cli

shreyas-goenka · 2023-06-02T15:24:16Z

Changes

Adds fs ls command
Adds ability to define multiple templates

Tests

Manually and integration tests

shreyas-goenka · 2023-06-04T23:15:07Z

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore -l --output=json
[
  {
    "name":"babynames.csv",
    "is_directory":false,
    "size":278154,
    "last_modified":"2023-03-05T16:27:11+01:00"
  },
  {
    "name":"jars",
    "is_directory":true,
    "size":0,
    "last_modified":"2022-10-03T08:18:02+02:00"
  },
  {
    "name":"tables",
    "is_directory":true,
    "size":0,
    "last_modified":"2022-06-23T15:45:57+02:00"
  },
  {
    "name":"terraform-test",
    "is_directory":true,
    "size":0,
    "last_modified":"2023-03-26T20:18:06+02:00"
  }
]%

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore -l
FILE      278154 2023-03-05T15:27:11Z babynames.csv
DIRECTORY 0 2022-10-03T06:18:02Z jars
DIRECTORY 0 2022-06-23T13:45:57Z tables
DIRECTORY 0 2023-03-26T18:18:06Z terraform-test

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore
babynames.csv
jars
tables
terraform-test

shreyas-goenka · 2023-06-04T23:22:57Z

tests run: https://github.com/databricks/eng-dev-ecosystem/actions/runs/5171723068

cmd/fs/ls.go

andrewnester · 2023-06-05T07:56:13Z

cmd/fs/ls_output.go

+	ModTime time.Time `json:"last_modified"`
+}
+
+func toLsOutput(f fs.DirEntry) (*lsOutput, error) {


I would make it a method of fs.DirEntry instead, for better readability

We cannot define methods on structs defined in external packages right?

cmd/fs/ls.go

pietern

The change modifying cmdio to use a map of templates doesn't seem necessary. Please revert that and stick to calling RenderWithTemplate with an actual template.

libs/cmdio/render.go

cmd/fs/ls.go

libs/filer/dbfs_client.go

cmd/fs/ls_output.go

libs/cmdio/render.go

pietern · 2023-06-05T12:45:42Z

cmd/fs/ls.go

+			{{end}}
+			`))
+		}
+		return cmdio.Render(ctx, jsonDirEntries)


Please specify the template here as well. Now this command it taking two different approaches for rendering a template based on the mode which is weird to the uninitiated reader.

Is there a significance for storing templates in the annotations in the first place?

cc: @nfx

It has to be done for autogenerated commands because assignment of the annotations field happens in a different file from where the command is defined. Means a stable way to associate metadata with the command instance.

cmd/fs/ls.go

pietern

LGTM, final comment.

Thanks!

pietern · 2023-06-05T14:17:23Z

internal/ls_test.go

@@ -0,0 +1,104 @@
+package internal


Could you name this file fs_ls_test.go to disambiguate from (potential) other ls tests?

What are the other potential ls tests possible? did you have anything specific in mind?

workspace fs

shreyas-goenka · 2023-06-05T15:21:53Z

integration test run: https://github.com/databricks/eng-dev-ecosystem/actions/runs/5178919118

## Changes CLI: * Add directory tracking to sync ([#425](#425)). * Add fs cat command for dbfs files ([#430](#430)). * Add fs ls command for dbfs ([#429](#429)). * Add fs mkdirs command for dbfs ([#432](#432)). * Add fs rm command for dbfs ([#433](#433)). * Add installation instructions ([#458](#458)). * Add new line to cmdio JSON rendering ([#443](#443)). * Add profile on `databricks auth login` ([#423](#423)). * Add readable console logger ([#370](#370)). * Add workspace export-dir command ([#449](#449)). * Added secrets input prompt for secrets put-secret command ([#413](#413)). * Added spinner when loading command prompts ([#420](#420)). * Better error message if can not load prompts ([#437](#437)). * Changed service template to correctly handle required positional arguments ([#405](#405)). * Do not generate prompts for certain commands ([#438](#438)). * Do not prompt for List methods ([#411](#411)). * Do not use FgWhite and FgBlack for terminal output ([#435](#435)). * Skip path translation of job task for jobs with a Git source ([#404](#404)). * Tweak profile prompt ([#454](#454)). * Update with the latest Go SDK ([#457](#457)). * Use cmdio in version command for `--output` flag ([#419](#419)). Bundles: * Check for nil environment before accessing it ([#453](#453)). Dependencies: * Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0 ([#459](#459)). * Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 ([#412](#412)). Internal: * Add Mkdir and ReadDir functions to filer.Filer interface ([#414](#414)). * Add Stat function to filer.Filer interface ([#421](#421)). * Add check for path is a directory in filer.ReadDir ([#426](#426)). * Add fs.FS adapter for the filer interface ([#422](#422)). * Add implementation of filer.Filer for local filesystem ([#460](#460)). * Allow equivalence checking of filer errors to fs errors ([#416](#416)). * Fix locker integration test ([#417](#417)). * Implement DBFS filer ([#139](#139)). * Include recursive deletion in filer interface ([#442](#442)). * Make filer.Filer return fs.DirEntry from ReadDir ([#415](#415)). * Speed up sync integration tests ([#428](#428)).

shreyas-goenka added 3 commits June 2, 2023 17:24

Add fs ls command for dbfs

b5f5f11

added integration tests

a4879c0

lint

2706beb

shreyas-goenka requested review from andrewnester and pietern June 4, 2023 23:16

shreyas-goenka added 2 commits June 5, 2023 01:20

added test for ls on flie

6b667b5

Merge remote-tracking branch 'origin' into fs-ls

db953ac

shreyas-goenka added 2 commits June 5, 2023 01:34

added resolvedbfs path func

492382c

manually check for dbfs prefix

3dc4770

andrewnester reviewed Jun 5, 2023

View reviewed changes

cmd/fs/ls.go Outdated Show resolved Hide resolved

andrewnester reviewed Jun 5, 2023

View reviewed changes

cmd/fs/ls.go Outdated Show resolved Hide resolved

andrewnester reviewed Jun 5, 2023

View reviewed changes

cmd/fs/ls.go Outdated Show resolved Hide resolved

andrewnester requested changes Jun 5, 2023

View reviewed changes

pietern reviewed Jun 5, 2023

View reviewed changes

libs/cmdio/render.go Outdated Show resolved Hide resolved

cmd/fs/ls.go Outdated Show resolved Hide resolved

libs/filer/dbfs_client.go Outdated Show resolved Hide resolved

cmd/fs/ls_output.go Outdated Show resolved Hide resolved

pietern mentioned this pull request Jun 5, 2023

Add databricks fs ls command for WSFS #408

Closed

shreyas-goenka added 4 commits June 5, 2023 13:49

comments

19a2051

move sort outside

8050dc5

-

fc7fbbe

address comments

b048c86

pietern reviewed Jun 5, 2023

View reviewed changes

shreyas-goenka added 2 commits June 5, 2023 15:07

address comments 2

265e517

added preallocation of size

0882609

shreyas-goenka requested review from andrewnester and pietern June 5, 2023 13:10

initialize dbfs client at root

9160a6e

pietern approved these changes Jun 5, 2023

View reviewed changes

shreyas-goenka added 2 commits June 5, 2023 17:18

Merge remote-tracking branch 'origin' into fs-ls

503f727

-

993a545

Merge branch 'main' into fs-ls

e68aa27

shreyas-goenka merged commit 6ff0012 into main Jun 5, 2023

shreyas-goenka deleted the fs-ls branch June 5, 2023 15:41

pietern mentioned this pull request Jun 12, 2023

Release v0.100.3 #461

Merged

pietern mentioned this pull request Jun 15, 2023

Add unit test for bricks fs ls #50

Closed

andrewnester mentioned this pull request Jul 13, 2023

Add unit test for fs ls command #20

Closed

Conversation

shreyas-goenka commented Jun 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

shreyas-goenka commented Jun 4, 2023

Uh oh!

shreyas-goenka commented Jun 4, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Jun 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyas-goenka commented Jun 2, 2023 •

edited

Loading