Skip to content

Add Parameters#expect to safely filter and require params#51674

Merged
dhh merged 4 commits intorails:mainfrom
martinemde:params-mandate
Sep 7, 2024
Merged

Add Parameters#expect to safely filter and require params#51674
dhh merged 4 commits intorails:mainfrom
martinemde:params-mandate

Conversation

@martinemde
Copy link
Contributor

@martinemde martinemde commented Apr 27, 2024

Motivation / Background

I've been hunting around trying to fix the problem with the default, recommended way of handling parameters in Rails.

user_params = params.require(:user).permit(:name, :age)

This is fine until someone using your app starts messing with the parameters and causing 500 errors by passing:

/path?user=string

This causes a NoMethodError because permit is called on "string".

The recommendation is not the best way. Instead, the first statement should be written like this:

user_params = params.permit(user: [:name, :age]).require(:user)

Detail

In this PR, I propose a new method on ActionController::Parameters that safely handles params filtering in one method, cuts down on ignorable errors caused by params tampering, and improves the specificity of param types.

params = ActionController::Parameters.new(user: { name: "Martin", age: 40 })
permitted = params.expect(user: %i[name age])
permitted.permitted?   # => true
permitted.has_key?(:name) # => true
permitted.has_key?(:age) # => true

In order to ensure consistent filtering of hashes vs arrays in params, expect introduces a new Array matching syntax using a double array [[:attr]]. (this example uses shorthand for all but the first example to make it more readable. The hashes returned are actually permitted ActionController::Parameter objects)

# Array is expected with `permit`
ActionController::Parameters.new(comments: { text: "Hello" }).permit(comments: [:text]).require(:comments)
# => #<ActionController::Parameters {"text"=>"Hello"} permitted: true>
ActionController::Parameters.new(comments: [{ text: "Hello" }]).permit(comments: [:text]).require(:comments)
# => [{"text"=>"Hello"}]
Parameters.new(comments: { text: "Hello" }).permit(comments: [[:text]]).require(:comments)
# => param is missing or the value is empty: comments (ActionController::ParameterMissing)
Parameters.new(comments: [{ text: "Hello" }]).permit(comments: [[:text]]).require(:comments)
# => [{"text"=>"Hello"}]

# Array is expected with `expect`
Parameters.new(comments: { text: "Hello" }).expect(comments: [[:text]])
# => param is missing or the value is empty: comments (ActionController::ParameterMissing)
Parameters.new(comments: [{ text: "Hello" }]).expect(comments: [[:text]])
# => [{"text"=>"Hello"}]

# Hash is expected with `permit`
Parameters.new(user: { name: "Martin" }).permit(user: [:name]).require(:user)
# => {"name"=>"Martin"}
Parameters.new(user: [{ name: "Martin" }]).permit(user: [:name]).require(:user)
# => [{"name"=>"Martin"}]

# Hash is expected with `expect`
Parameters.new(user: { name: "Martin" }).expect(user: [:name])
# => {"name"=>"Martin"}
Parameters.new(user: [{ name: "Martin" }]).expect(user: [:name])
# => param is missing or the value is empty: user (ActionController::ParameterMissing)

As you can see, permit will expect an array when the "double array" syntax is used, but defaults to the old behavior of allowing both arrays and hashes otherwise, maintaining backwards compatibility. expect on the other hand is strict in for both arrays and hashes.

This pull request also adds the expect! method. expect! raises an unhandled error in cases where expect would raise a handled error that would return a 400 bad request. The unhandled error can be used in cases where it is important to be alerted about a client using incorrect syntax (such as a private internal API where the client should be fixed).

Adds a 400.html template used to render the 400 bad request response.

Additional information

Related PRs and issues that address this problem:

Checklist

Before submitting the PR make sure the following are checked:

  • This Pull Request is related to one change. Unrelated changes should be opened in separate PRs.
  • Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex: [Fix #issue-number]
  • Tests are added or updated if you fix a bug or add a feature.
  • CHANGELOG files are updated for the changed libraries if there is a behavior change or additional feature. Minor bug fixes and documentation changes should not be included.

@rails-bot rails-bot bot added the actionpack label Apr 27, 2024
@martinemde martinemde changed the title Proof of concept documentation for ActionController::Parameters#mandate PoC & Documentation for new ActionController::Parameters#mandate Apr 27, 2024
@martinemde martinemde changed the title PoC & Documentation for new ActionController::Parameters#mandate PoC & Documentation for new Parameters#mandate Apr 27, 2024
@MatheusRich
Copy link
Contributor

@martinemde please add tests and a changelog entry

@MatheusRich
Copy link
Contributor

I wonder if we should just handle these NoMethodErrors differently. It feels weird to return a 500 when the error is the "the user's fault". Maybe we could return a 4XX somehow?

@rafaelfranca
Copy link
Member

Let's move on with this one. @dhh already reviewed the API and said it looks promising.

Can you write tests for it and prefer using it to require.permit in the guides?

@MatheusRich
Copy link
Contributor

Fixes #52463

@martinemde
Copy link
Contributor Author

I'll update the PR soon

@dari-us
Copy link
Contributor

dari-us commented Aug 2, 2024

Fixes #52463

Actually, if I'm not mistaken, fixes only half of it.
params.require(...).permit(...) will be fixed (as used all the time in StrongParameters)
params.fetch(:x, {}).permit(...) won't be fixed (first code snippet under StrongParameters#more_examples)

@martinemde
Copy link
Contributor Author

martinemde commented Aug 3, 2024

I'm pushing up work as I finish. Thanks for the feedback. Should have it done soon. (I'll squash when it's ready)

As I've been working on this I thought of an alternate name. What do you think?

params.expect(person: %i[name, age])

After typing mandate over and over, I'm not totally in love with it, but I don't feel strongly.

@dari-us
Copy link
Contributor

dari-us commented Aug 3, 2024

Seems good so far. I find expect sounds nicer, but less strict than mandate. When tasked with a name for this from scratch, I'd probably come up with expect.

However, seeing as mandate internally calls permit, it is safe to assume this does not play nicely with action_on_unpermitted_params...
According to (quite old) #26831, it would be fine to have this on a per-controller basis.
Or maybe, remove logging/raising from permit and instead implement permit_only_or_raise, and permit_only_or_log (quite verbose, but the idea is clear)

@martinemde
Copy link
Contributor Author

@dari-us maybe params.expect_only that would raise.

The existing configuration makes sense. Logging can be useful in dev but would be extra noise in most production applications. This implies to me that logging is an environment config and raising should be controlled by how the methods are called. Please correct me if you think otherwise.

If we added expect_only and permit_only (for a non-require version) that would allow the use case in #26831. These would be methods that would raise UnpermittedParameters. expect and permit would continue to log or raise depending on the global config, allowing an app that wants both to set globally, and use the _only methods selectively. I suggest doing this in a follow-up PR that I can certainly help with.

@martinemde
Copy link
Contributor Author

Regarding your comment @MatheusRich, I never got to respond.

I wonder if we should just handle these NoMethodErrors differently. It feels weird to return a 500 when the error is the "the user's fault". Maybe we could return a 4XX somehow?

This was what motivated me to create this. It's certainly wrong to return 500 when the client caused the problem, but once require(:key) returns the value of params[:key] we're in plain ruby land and a NoMethodError is bound to occur if you call permit next. I think we must just remove all uses or mentions of the pattern require(:key).permit(:attr) and switch to this new method. I aim to do that here to the extent possible.

permit(key: [:attr]).require(:key) is how rubygems.org now processes all params and it has removed an annoying class of 500s that used to get us paged occasionally. I will to convert to this once it's released.

@martinemde
Copy link
Contributor Author

martinemde commented Aug 3, 2024

I switched the name from mandate to expect. I used a single commit so we could revert if we decide not to go that route. I personally prefer it.

I'm doing one more pass on the guides to try to find non-obvious uses. Otherwise this is ready for code review (and please call out any documentation that you think should be fixed, if I missed it)

I'm working on fixing the test failures.

@dari-us
Copy link
Contributor

dari-us commented Aug 4, 2024

@martinemde Yeah, global config for logging and raising/ignoring via method call is fine by me and I think makes sense for the broad majority of apps.
What I mean in particular with "does not play nicely" is this case:

action_on_unpermitted_params = :raise
# route PATCH /users/1 => users#update
params.expect(user: [:name, :age]) # raises, b/c params contains unpermitted param :id

I think at this point, :raise basically kills your app and :log produces lots of noise. Also in a nested context like /groups/1/users, the create action also logs/raises.
To improve the situation, one would probably move the actual filtering that permit does into an internal method, say, apply_filters, and then expect could use that, so unpermitted_keys! is not called automatically.
Also, unpermitted_keys! needs to accept an argument to trigger :raise by hand and use global config as fallback.

On another note, sadly the use case how to replace params.fetch(:x, {}).permit(...) is not covered, but I guess this is out of scope for this PR. At least I don't know a good way how to incorporate it.

Also, I would love to make a followup PR to

  1. Account for action_on_unpermitted_params
  2. enable devs to :raise via *_only
  3. enable the params.fetch(:x, {}).permit(...) pattern (maybe this could be named accept and accept_only, as in, "it is accepted, but not expected").

Hopefully I can spare some time on that soon.

@martinemde
Copy link
Contributor Author

martinemde commented Aug 4, 2024

@dari-us, ok, I understand now what you mean. The fact of how permit works when applied to params "root" means it would consider any other params to be unpermitted. This explains something I was noticing on rubygems.org, which is that dev was logging more unpermitted params than usual.

Possible fix, but maybe too ugly: id, nested_id, table_params = params.expect(:id, :nested_id, table: [:attrs])
I think maybe your proposed solution is the best. Let's consider this a bug with this implementation until we can fix it, since it's not the expected outcome.

Edit: thinking ahead about expect_only, the way to make attributes required and permitted, while still raising on nested params that are not permitted, you would do: expect(user: {}).expect_only(:name, :age)

As for the other methods, I think you're right. The fetch.permit pattern could be allow(user: [:name, :age]) which would return {} on missing key. Then it uses the same pattern as expect but communicates that, just like many mocking frameworks, allows might or might not happen and that's ok. (I prefer allow over accept because accept sounds a lot like expect, just transpose a syllable).

@dhh dhh added this to the 8.0.0 milestone Sep 5, 2024
@dhh dhh merged commit e1d58cf into rails:main Sep 7, 2024
@dhh
Copy link
Member

dhh commented Sep 7, 2024

Excellent work, guys! This is a very nice upgrade.

We'll need to also upgrade the templates in the jbuilder gem, since they're used by default. Probably have to have a Rails version toggle where we use expect when 8.0+ and whatever is there before that. @martinemde want to take a stab at that too?

@bogdan
Copy link
Contributor

bogdan commented Sep 8, 2024

Nice change, silents some noise in bug notifications. Should there be a rubocop rule for it?

@dhh
Copy link
Member

dhh commented Sep 8, 2024

Nice change, silents some noise in bug notifications. Should there be a rubocop rule for it?

No. Don't need to nag people about this. They can upgrade to the new syntax as they see fit and new apps will automatically guide folks in that direction.

@p8
Copy link
Member

p8 commented Sep 8, 2024

Nice API!
I'm wondering if expect is a little too close to except, and could cause confusion and typo's?
Not sure if the previously mentioned allow is completely out of the picture?
Or maybe with, to counter the without alias?

@dari-us
Copy link
Contributor

dari-us commented Sep 9, 2024

@p8 I hardly see that as a problem, at least I cannot remember ever seeing except used with params.
allow sounds less strict, i.e. I wouldn't expect it to raise for missing root keys.
I would love to have allow as a replacement for params.fetch(:x, {}).permit(...) sometime, as this pattern appears in the docs, I often use it for "filter forms" and is not fixed by the expect API.

garrettblehm pushed a commit to garrettblehm/rails that referenced this pull request Sep 9, 2024
)

* `Parameters#expect` safely filters and requires params

* `Parameters#expect!` raises unhandled exception on missing params

* Add public/400.html file rendered on bad request

---------

Co-authored-by: dari-us <[email protected]>
@p8
Copy link
Member

p8 commented Sep 10, 2024

I can see non-native English speakers or people with dyslexia having problems with this - copying the guide documentation and wondering why things aren't working as expected.

As this API seems to be more strict than require and permit, maybe a verb that has a more strict meaning would be clearer?

For example constrain or contraints, which has overlap with routing contraints, which already does parameter restriction.

@dhh
Copy link
Member

dhh commented Sep 10, 2024

Appreciate the feedback here, but this bikesheed has already been painted, @p8. We're moving on to other bikesheeds now 😄

bquorning added a commit to zendesk/stronger_parameters that referenced this pull request Sep 13, 2024
bquorning added a commit to zendesk/stronger_parameters that referenced this pull request Sep 13, 2024
bquorning added a commit to zendesk/stronger_parameters that referenced this pull request Sep 13, 2024
@jeromedalbert
Copy link
Contributor

jeromedalbert commented Sep 14, 2024

I took a stab at updating the jbuilder templates here: rails/jbuilder#573.

Also fixed a minor Rubocop issue introduced by this PR here: #52928.

DanielaVelasquez pushed a commit to DanielaVelasquez/rails that referenced this pull request Oct 3, 2024
)

* `Parameters#expect` safely filters and requires params

* `Parameters#expect!` raises unhandled exception on missing params

* Add public/400.html file rendered on bad request

---------

Co-authored-by: dari-us <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StrongParameters default approach prone to 500: Recommended approach? Strong Params require with permit combination produce unexpected exception

9 participants