Skip to content

Conversation

@shubham-up-47
Copy link
Contributor

@shubham-up-47 shubham-up-47 commented Jul 1, 2025

Moving the implementation of private methods UploadFileSimple and UploadFileResumable from client.cc file to connection_impl.cc file, so that tracing of both the upload file methods can be enabled (#12906),


This change is Reviewable

@shubham-up-47 shubham-up-47 requested review from a team as code owners July 1, 2025 14:02
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Jul 1, 2025
@shubham-up-47
Copy link
Contributor Author

Hi @ddelgrosso1 , Please see this approach here for creating the traces of UploadFile method, if this looks good, then i will go ahead and update the test files.

@shubham-up-47 shubham-up-47 marked this pull request as draft July 1, 2025 19:25
@ddelgrosso1
Copy link
Contributor

@shubham-up-47 I think overall the approach makes sense to me. You might want to run this by @scotthart as well to get a second, more informed opinion.

Copy link
Member

@scotthart scotthart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 8 files reviewed, 3 unresolved discussions


google/cloud/storage/internal/connection_impl.cc line 768 at r1 (raw file):

    os << __func__ << "(" << request << ", " << file_name
       << "): UploadFromOffset (" << upload_offset
       << ") is bigger than the size of file source (" << file_size << ")";

Let's update this and other usages of std::ostringstream being relocated in this PR to use absl::StrCat in this internal file.

Code quote:

    std::ostringstream os;
    os << __func__ << "(" << request << ", " << file_name
       << "): UploadFromOffset (" << upload_offset
       << ") is bigger than the size of file source (" << file_size << ")";

google/cloud/storage/client.h line 3547 at r1 (raw file):

  StatusOr<ObjectMetadata> UploadFileResumable(
      std::string const& file_name, internal::ResumableUploadRequest request);

The purpose of these helper functions is to move the code that doesn't rely on the templated types in UploadFileImpl in to the .cc file. Let's keep these functions but re-purpose them to contain the code added to UploadFileImpl.

Code quote:

  StatusOr<ObjectMetadata> UploadFileSimple(
      std::string const& file_name, std::size_t file_size,
      internal::InsertObjectMediaRequest request);

  StatusOr<ObjectMetadata> UploadFileResumable(
      std::string const& file_name, internal::ResumableUploadRequest request);

google/cloud/storage/internal/storage_connection.h line 118 at r1 (raw file):

  virtual StatusOr<std::unique_ptr<std::istream>> UploadFileResumable(
      std::string const& file_name,
      ResumableUploadRequest& request) = 0;

I'm concerned that adding new pure abstract methods to this class could break customers using a Mock or other derivation of this class. Instead, let's make these methods have a default implementation that returns a kUNIMPLEMENTED Status.

Code quote:

  virtual StatusOr<EmptyResponse> UploadFileSimple(
      std::string const& file_name, std::size_t file_size,
      InsertObjectMediaRequest& request) = 0;
  virtual StatusOr<std::unique_ptr<std::istream>> UploadFileResumable(
      std::string const& file_name,
      ResumableUploadRequest& request) = 0;

@shubham-up-47 shubham-up-47 marked this pull request as ready for review July 11, 2025 11:19
@codecov
Copy link

codecov bot commented Jul 11, 2025

Codecov Report

Attention: Patch coverage is 81.81818% with 58 lines in your changes missing coverage. Please review.

Project coverage is 92.92%. Comparing base (0819ec0) to head (894e71b).
Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
.../cloud/storage/internal/storage_connection_test.cc 18.84% 56 Missing ⚠️
google/cloud/storage/internal/connection_impl.cc 97.14% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #15245      +/-   ##
==========================================
- Coverage   92.93%   92.92%   -0.01%     
==========================================
  Files        2394     2396       +2     
  Lines      215564   215816     +252     
==========================================
+ Hits       200332   200553     +221     
- Misses      15232    15263      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@scotthart scotthart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4 of 11 files at r2, 6 of 11 files at r3, all commit messages.
Reviewable status: 10 of 15 files reviewed, 5 unresolved discussions (waiting on @shubham-up-47)


google/cloud/storage/internal/storage_connection.h line 118 at r3 (raw file):

      [[maybe_unused]] InsertObjectMediaRequest& request) {
    return Status(StatusCode::kUnimplemented, "unimplemented");
  }

There's not need to use maybe_unused here, simply omit the names of the parameters:

virtual StatusOr<std::unique_ptrstd::string> UploadFileSimple(
std::string const&,
std::size_t,
InsertObjectMediaRequest&) {
return Status(StatusCode::kUnimplemented, "unimplemented");
}

Code quote:

  virtual StatusOr<std::unique_ptr<std::string>> UploadFileSimple(
      [[maybe_unused]] std::string const& file_name,
      [[maybe_unused]] std::size_t file_size,
      [[maybe_unused]] InsertObjectMediaRequest& request) {
    return Status(StatusCode::kUnimplemented, "unimplemented");
  }

google/cloud/storage/client_object_test.cc line 263 at r3 (raw file):

  auto expected =
      storage::internal::ObjectMetadataParser::FromString(text).value();
  auto const contents = std::string{"some simple contents"};

When declaring simple variables like this, prefer not using auto, instead:

std::string const contents{"some simple contents"};

Here and elsewhere in this PR.

@shubham-up-47
Copy link
Contributor Author

Reviewable status: 0 of 8 files reviewed, 3 unresolved discussions

google/cloud/storage/internal/connection_impl.cc line 768 at r1 (raw file):

    os << __func__ << "(" << request << ", " << file_name
       << "): UploadFromOffset (" << upload_offset
       << ") is bigger than the size of file source (" << file_size << ")";

Let's update this and other usages of std::ostringstream being relocated in this PR to use absl::StrCat in this internal file.

Code quote:

    std::ostringstream os;
    os << __func__ << "(" << request << ", " << file_name
       << "): UploadFromOffset (" << upload_offset
       << ") is bigger than the size of file source (" << file_size << ")";

google/cloud/storage/client.h line 3547 at r1 (raw file):

  StatusOr<ObjectMetadata> UploadFileResumable(
      std::string const& file_name, internal::ResumableUploadRequest request);

The purpose of these helper functions is to move the code that doesn't rely on the templated types in UploadFileImpl in to the .cc file. Let's keep these functions but re-purpose them to contain the code added to UploadFileImpl.

Code quote:

  StatusOr<ObjectMetadata> UploadFileSimple(
      std::string const& file_name, std::size_t file_size,
      internal::InsertObjectMediaRequest request);

  StatusOr<ObjectMetadata> UploadFileResumable(
      std::string const& file_name, internal::ResumableUploadRequest request);

google/cloud/storage/internal/storage_connection.h line 118 at r1 (raw file):

  virtual StatusOr<std::unique_ptr<std::istream>> UploadFileResumable(
      std::string const& file_name,
      ResumableUploadRequest& request) = 0;

I'm concerned that adding new pure abstract methods to this class could break customers using a Mock or other derivation of this class. Instead, let's make these methods have a default implementation that returns a kUNIMPLEMENTED Status.

Code quote:

  virtual StatusOr<EmptyResponse> UploadFileSimple(
      std::string const& file_name, std::size_t file_size,
      InsertObjectMediaRequest& request) = 0;
  virtual StatusOr<std::unique_ptr<std::istream>> UploadFileResumable(
      std::string const& file_name,
      ResumableUploadRequest& request) = 0;
  1. Getting error while using absl::Strcat, since absl::StrCat can’t convert request to string format.
  2. Done, Keeping these functions and re-purposed them to contain the code added.
  3. Done, Keeping a default implementation which returns a kUNIMPLEMENTED status.

@shubham-up-47
Copy link
Contributor Author

Reviewed 4 of 11 files at r2, 6 of 11 files at r3, all commit messages.
Reviewable status: 10 of 15 files reviewed, 5 unresolved discussions (waiting on @shubham-up-47)

google/cloud/storage/internal/storage_connection.h line 118 at r3 (raw file):

      [[maybe_unused]] InsertObjectMediaRequest& request) {
    return Status(StatusCode::kUnimplemented, "unimplemented");
  }

There's not need to use maybe_unused here, simply omit the names of the parameters:

virtual StatusOrstd::unique_ptrstd::string UploadFileSimple( std::string const&, std::size_t, InsertObjectMediaRequest&) { return Status(StatusCode::kUnimplemented, "unimplemented"); }

Code quote:

  virtual StatusOr<std::unique_ptr<std::string>> UploadFileSimple(
      [[maybe_unused]] std::string const& file_name,
      [[maybe_unused]] std::size_t file_size,
      [[maybe_unused]] InsertObjectMediaRequest& request) {
    return Status(StatusCode::kUnimplemented, "unimplemented");
  }

google/cloud/storage/client_object_test.cc line 263 at r3 (raw file):

  auto expected =
      storage::internal::ObjectMetadataParser::FromString(text).value();
  auto const contents = std::string{"some simple contents"};

When declaring simple variables like this, prefer not using auto, instead:

std::string const contents{"some simple contents"};

Here and elsewhere in this PR.

  1. Done, was using may_be to pass code coverage, didn't know this solution, thanks for telling it.
  2. Done, will keep that thing in mind and avoid auto for simple variables.

Copy link
Member

@scotthart scotthart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 11 files at r3, 3 of 3 files at r4, all commit messages.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @ddelgrosso1 and @shubham-up-47)


google/cloud/storage/internal/storage_connection.h line 118 at r1 (raw file):

Previously, shubham-up-47 wrote…
  1. Getting error while using absl::Strcat, since absl::StrCat can’t convert request to string format.
  2. Done, Keeping these functions and re-purposed them to contain the code added.
  3. Done, Keeping a default implementation which returns a kUNIMPLEMENTED status.

re: ostringstream and absl::StrCat, let's not block this PR on that. Please create an internal cleanup issue to convert the ostringstream uses in storage internal to use absl::StrCat, we can address it in the future.

@shubham-up-47
Copy link
Contributor Author

Reviewed 2 of 11 files at r3, 3 of 3 files at r4, all commit messages.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @ddelgrosso1 and @shubham-up-47)

google/cloud/storage/internal/storage_connection.h line 118 at r1 (raw file):

Previously, shubham-up-47 wrote…
re: ostringstream and absl::StrCat, let's not block this PR on that. Please create an internal cleanup issue to convert the ostringstream uses in storage internal to use absl::StrCat, we can address it in the future.

Done, created it #15267

@shubham-up-47 shubham-up-47 merged commit 9bb18b8 into googleapis:main Jul 16, 2025
78 of 79 checks passed
@shubham-up-47 shubham-up-47 deleted the otel_in_upload_file branch July 16, 2025 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants