Skip to content

Set notebook directory as current directory for pip installs from workspace fs#961

Merged
kartikgupta-db merged 5 commits intomainfrom
workflow-preamble
Dec 5, 2023
Merged

Set notebook directory as current directory for pip installs from workspace fs#961
kartikgupta-db merged 5 commits intomainfrom
workflow-preamble

Conversation

@kartikgupta-db
Copy link
Copy Markdown
Contributor

@kartikgupta-db kartikgupta-db commented Dec 3, 2023

When users use a relative path to install a notebook scoped library with pip (eg %pip install ../../foo) it can fail since cwd for shell commands can be an ephemeral directory in DBRs < 14.0.
For other shell commands, our preamble handles the cwd, but pip commands are moved to the beginning of the notebook, since they can lead to a kernel restart and hence loss of preamble. This means, they can't access the features of the full preamble and need an explicit os.chdir to set the correct path.

Changes

  • Add an explicit os.chdir(notebook_dir) before each %pip install.
  • Also remove code to pass source_file and project_root to the wrappers as notebook params (for notebook jobs) and cli args (for python file jobs). Now these parameters are hard coded into the wrapper. These are updated on every run since the wrapper is recreated on every run.

Tests

  • manual

Fixes #958

@kartikgupta-db kartikgupta-db merged commit eec87b8 into main Dec 5, 2023
@kartikgupta-db kartikgupta-db deleted the workflow-preamble branch December 5, 2023 11:53
@github-actions github-actions bot mentioned this pull request Dec 5, 2023
ilia-db added a commit that referenced this pull request Dec 5, 2023
## packages/databricks-vscode
## <small>1.2.4 (2023-12-05)</small>

* Send environment type (tests, prod) in telemetry events (#944)
([1ce3ef0](1ce3ef0)),
closes
[#944](#944)
* Set notebook directory as current directory for pip installs from
workspace fs (#961)
([eec87b8](eec87b8)),
closes
[#961](#961)
[#958](#958)
* Show error for a corrupted config file (#955)
([8b5a944](8b5a944)),
closes
[#955](#955)
* Update Databricks CLI to v0.210.1 (#957)
([c9864e1](c9864e1)),
closes
[#957](#957)



## packages/databricks-vscode-types
## <small>1.2.4 (2023-12-05)</small>

---------

Co-authored-by: releasebot <[email protected]>
Co-authored-by: Ilia Babanov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature request] set working directory when running as workflow

2 participants