feat: Add filesystem and projects API by gjreda · Pull Request #389 · refstudio/refstudio

gjreda · 2023-08-21T15:43:12Z

fixes #365

This PR adds APIs for both managing files and projects.

Running the server

cd python; poetry run uvicorn web:api --reload

Project API

Create a project

curl -X 'POST' \
  'http://127.0.0.1:8000/api/projects/' \
  -H 'accept: application/json' \
  -d '' -s | jq

{
  "63f40666-6aae-49c6-b375-be09ca3dd5bc": "/tmp/web-storage-url/user1/63f40666-6aae-49c6-b375-be09ca3dd5bc"
}

List projects

curl -X 'GET' \
  'http://127.0.0.1:8000/api/projects/' \
  -H 'accept: application/json' -s | jq

{
  "project1": "/tmp/web-storage-url/user1/project1",
  "project2": "/private/var/folders/4v/0m97cs6j7v53r6m3mwg340380000gn/T/pytest-of-greg/pytest-389/test_update_project_path_stora1/user1/project2",
  "d94ca1db-f251-48c3-aafe-67be329b6006": "/tmp/web-storage-url/user1/d94ca1db-f251-48c3-aafe-67be329b6006",
  "63f40666-6aae-49c6-b375-be09ca3dd5bc": "/tmp/web-storage-url/user1/63f40666-6aae-49c6-b375-be09ca3dd5bc"
}

Get project details

curl -X 'GET' \
  'http://127.0.0.1:8000/api/projects/63f40666-6aae-49c6-b375-be09ca3dd5bc' \
  -H 'accept: application/json' -s | jq

{
  "project_id": "63f40666-6aae-49c6-b375-be09ca3dd5bc",
  "project_path": "/tmp/web-storage-url/user1/63f40666-6aae-49c6-b375-be09ca3dd5bc",
  "filepaths": []
}

Delete a project

curl -X 'DELETE' \
  'http://127.0.0.1:8000/api/projects/63f40666-6aae-49c6-b375-be09ca3dd5bc' \
  -H 'accept: application/json'

{
  "status": "success",
  "message": "Project deleted",
  "project_id": "63f40666-6aae-49c6-b375-be09ca3dd5bc"
}

List project files

curl -X 'GET' \
  'http://127.0.0.1:8000/api/projects/63f40666-6aae-49c6-b375-be09ca3dd5bc/files' \
  -H 'accept: application/json'

Filesystem API

Create a file (this will respect directory segments)

curl -X 'PUT' \
  'http://127.0.0.1:8000/api/fs/b532a8c1-8bea-47e2-9f2f-0ac93e4da623/uploads%2Ftest.pdf' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@A Few Useful Things to Know about Machine Learning.pdf;type=application/pdf'

{
  "status": "success",
  "message": "File uploaded",
  "filepath": "/tmp/web-storage-url/user1/b532a8c1-8bea-47e2-9f2f-0ac93e4da623/uploads/test.pdf"
}

Read a file

curl -X 'GET' \
  'http://127.0.0.1:8000/api/fs/b532a8c1-8bea-47e2-9f2f-0ac93e4da623/uploads%2Ftest.pdf' \
  -H 'accept: application/json'

Delete a file

curl -X 'DELETE' \
  'http://127.0.0.1:8000/api/fs/b532a8c1-8bea-47e2-9f2f-0ac93e4da623/uploads%2Ftest.pdf' \
  -H 'accept: application/json'

{
  "status": "success",
  "message": "File deleted",
  "filepath": "/tmp/web-storage-url/user1/b532a8c1-8bea-47e2-9f2f-0ac93e4da623/uploads/test.pdf"
}

…emts

codecov · 2023-08-21T21:55:32Z

Codecov Report

Merging #389 (5249f2a) into main (fe1a3dc) will increase coverage by 0.11%.
Report is 1 commits behind head on main.
The diff coverage is 87.94%.

❗ Current head 5249f2a differs from pull request most recent head f026603. Consider uploading reports for the commit f026603 to get more accurate results

@@            Coverage Diff             @@
##             main     #389      +/-   ##
==========================================
+ Coverage   84.10%   84.21%   +0.11%     
==========================================
  Files         174      176       +2     
  Lines        9964    10135     +171     
  Branches     1097     1101       +4     
==========================================
+ Hits         8380     8535     +155     
- Misses       1573     1589      +16     
  Partials       11       11

Files Changed	Coverage Δ
python/sidecar/http.py	`84.68% <83.14%> (-3.96%)`	⬇️
python/sidecar/projects.py	`95.91% <95.91%> (ø)`
python/sidecar/settings.py	`100.00% <100.00%> (ø)`
python/web.py	`100.00% <100.00%> (ø)`

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

gjreda · 2023-08-21T22:13:48Z

+
+# Filesystem API
+# --------------
+@filesystem_api.put("/{project_id}/{filepath:path}")


@cguedes This will respect the filepath provided, so uploading a named foo.pdf to uploads/test.pdf will result in the appropriate directory being created and test.pdf being written there.

I think this is what we were discussing this morning, but let me know otherwise.

Yes, this is exactly what we want.

gjreda · 2023-08-21T22:16:07Z

+    project_path = projects.get_project_path(user_id, project_id)
+    filepath = project_path / filepath
+    try:
+        os.remove(filepath)


@cguedes I ended up deleting filesystem.py because once I cleaned this PR up, this line was literally the only one remaining that was actually used.

FastAPI gave us a lot for free with FileResponse and UploadFile.

gjreda · 2023-08-21T22:18:07Z

There was a lot of code to knock out today and there's probably some clean up necessary, but I think it's better to get the functionality in place first to unblock others. Any clean up can happen in subsequent PRs.

cguedes · 2023-08-22T09:23:18Z

There was a lot of code to knock out today and there's probably some clean up necessary, but I think it's better to get the functionality in place first to unblock others. Any clean up can happen in subsequent PRs.

Great work @gjreda I've just added a commit to ensure that the server path exists (/tmp/....) and some code cleanup for the paths.

Also, move the projects.json file under the user's folder instead of using projects_{user_id}.json.

There is still work to do, for example in the response payload for the project files (not sure if it should be a list of files/directories vs nested structure), but this def unblock the frontend work for the web.

danvk · 2023-08-23T14:08:21Z

+    }
+
+
+@filesystem_api.get("/{project_id}/{filepath:path}")


Is there anything that prevents reading from outside the project path here? Could filepath be something like ../../../some-other-folder or /etc/passwd? Just wondering about security here.

gjreda added 8 commits August 18, 2023 15:43

POC

f568c75

Merge branch 'main' into 365-create-pythonhttp-equivalent-of-filesyst…

f7e48df

…emts

project api is working

22d98af

everything seems to work

e36ec2e

remove unused dependency

1c62a87

Add tests for projects.py

e369c4e

Add http tests

f69ddf8

linter

059f1e9

gjreda changed the title ~~POC filesystem api~~ feat: Add filesystem and projects API Aug 21, 2023

fix tests

5249f2a

gjreda commented Aug 21, 2023

View reviewed changes

gjreda marked this pull request as ready for review August 21, 2023 22:16

gjreda commented Aug 21, 2023

View reviewed changes

Comment thread python/sidecar/http.py

gjreda requested a review from cguedes August 22, 2023 00:42

Ensure server's path storage directory

f026603

cguedes approved these changes Aug 22, 2023

View reviewed changes

cguedes merged commit 4516f86 into main Aug 22, 2023

cguedes deleted the 365-create-pythonhttp-equivalent-of-filesystemts branch August 22, 2023 09:23

This was referenced Aug 22, 2023

bug: python tests don't run in the git push hook #391

Closed

feat: Add python tests script and run python tests in pre-push #390

Merged

Get file ingestion running through HTTP #366

Closed

danvk reviewed Aug 23, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add filesystem and projects API#389

feat: Add filesystem and projects API#389
cguedes merged 10 commits into
mainfrom
365-create-pythonhttp-equivalent-of-filesystemts

gjreda commented Aug 21, 2023 •

edited

Loading

Uh oh!

codecov Bot commented Aug 21, 2023 •

edited

Loading

Uh oh!

gjreda Aug 21, 2023

Uh oh!

cguedes Aug 22, 2023

Uh oh!

gjreda Aug 21, 2023 •

edited

Loading

Uh oh!

gjreda commented Aug 21, 2023

Uh oh!

Uh oh!

cguedes commented Aug 22, 2023

Uh oh!

danvk Aug 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		}


		@filesystem_api.get("/{project_id}/{filepath:path}")

Uh oh!

Conversation

gjreda commented Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Running the server

Project API

Create a project

List projects

Get project details

Delete a project

List project files

Filesystem API

Create a file (this will respect directory segments)

Read a file

Delete a file

Uh oh!

codecov Bot commented Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gjreda Aug 21, 2023

Choose a reason for hiding this comment

Uh oh!

cguedes Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

gjreda Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gjreda commented Aug 21, 2023

Uh oh!

Uh oh!

cguedes commented Aug 22, 2023

Uh oh!

danvk Aug 23, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gjreda commented Aug 21, 2023 •

edited

Loading

codecov Bot commented Aug 21, 2023 •

edited

Loading

gjreda Aug 21, 2023 •

edited

Loading