Skip to content

Log response body if config server returns non-JSON#2638

Merged
rkoster merged 1 commit intocloudfoundry:mainfrom
sap-contributions:config_server_better_exception_log
Nov 6, 2025
Merged

Log response body if config server returns non-JSON#2638
rkoster merged 1 commit intocloudfoundry:mainfrom
sap-contributions:config_server_better_exception_log

Conversation

@jochenehret
Copy link
Copy Markdown
Contributor

@jochenehret jochenehret commented Nov 5, 2025 •
edited
Loading

What is this change about?

Log full response body if a credential generation request to the config_server fails.

Please provide contextual information.

In a fresh bosh-bootloader environment, we often get stacktraces like this when creating the first deployment:

E, [2025-11-05T07:40:16.409932 #58382] [task:36] ERROR -- DirectorJobRunner: Config Server returned a NON-JSON body while generating value for '/bosh-maxime/cf/blobstore_admin_users_password' with type 'password'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/client.rb:440:in `rescue in generate_value'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/client.rb:437:in `generate_value'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/client.rb:475:in `generate_latest_version_id'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/client.rb:115:in `block in generate_values'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/client.rb:95:in `map'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/client.rb:95:in `generate_values'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/config_server/variables_interpolator.rb:62:in `generate_values'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/assembler.rb:170:in `generate_variables'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/assembler.rb:100:in `bind_models'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:99:in `block in prepare_deployment'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/event_log.rb:105:in `advance_and_track'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:87:in `prepare_deployment'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:42:in `block in perform'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/lock_helper.rb:7:in `block in with_deployment_lock'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/lock.rb:79:in `lock'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/lock_helper.rb:7:in `with_deployment_lock'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:34:in `perform'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:96:in `perform_job'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:32:in `block in run'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-common-0.0.0/lib/bosh/common/core_ext/kernel.rb:5:in `with_thread_name'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:32:in `run'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/base_job.rb:9:in `perform'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:35:in `block in perform'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:108:in `block in run'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:107:in `fork'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:107:in `run'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:30:in `perform'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/delayed_job-4.1.13/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/delayed_job-4.1.13/lib/delayed/lifecycle.rb:61:in `block in initialize'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/delayed_job-4.1.13/lib/delayed/lifecycle.rb:66:in `execute'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/delayed_job-4.1.13/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/delayed_job-4.1.13/lib/delayed/backend/base.rb:78:in `invoke_job'
/var/vcap/data/packages/director/d1658440292cb162b458a51c2eff0dab9cbe4fcf/gem_home/ruby/3.3.0/gems/delayed_job-4.1.13/lib/delayed/worker.rb:231:in `block (2 levels) in run'
/var/vcap/data/packages/director-ruby-3.3/a268bd86941dd75c530fd62935798e17bb941d78/lib/ruby/3.3.0/timeout.rb:186:in `block in timeout'

The second attempt usually succeeds. From the given logs, it's however unclear why the config server (i.e. the "credhub" instance running on the bosh director VM) fails.

What tests have you run against this PR?

None, adapted unit test.

How should this change be described in bosh release notes?

Doesn't need to be mentioned.

Does this PR introduce a breaking change?

No.

@jochenehret
Log response body if config server returns non-JSON
af6fa0a 
* in a fresh bbl environment we often get the error "Config Server returned a NON-JSON body while generating value for ..." without further information what went wrong
@github-project-automation github-project-automation bot moved this from Inbox to Pending Merge | Prioritized in Foundational Infrastructure Working Group Nov 5, 2025
@rkoster rkoster merged commit 2ce7f0c into cloudfoundry:main Nov 6, 2025
8 checks passed
@github-project-automation github-project-automation bot moved this from Pending Merge | Prioritized to Done in Foundational Infrastructure Working Group Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aramprice aramprice aramprice approved these changes

@rkoster rkoster rkoster approved these changes

Assignees

No one assigned

Labels

None yet

Projects

Foundational Infrastructure Working Group
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jochenehret @aramprice @rkoster