LWN: Comments on "Rust in the 6.2 kernel"

Rust in the 6.2 kernel

sammythesnake — Wed, 04 Jan 2023 18:07:10 +0000

If it were me doing those videos, I would certainly have aimed for a clearer choice, as well as one that's less piercing. I doubt I could cope with that voice for 8 hours(!)

It's worth noting that the (auto-generated) subtitles are usable, even if not perfect, so you could watch with the audio turned down, though that wouldn't work well with having it on in the background of some other task and dipping in when something piques interest.

I wonder if there's an audio filter that approximates the inverse of whatever was used to mask the voice in the first place - that might provide a more natural sounding option that at least *my* ears would find kinder.

For now, I'll stick to reading her excellent blog posts as and when they get lunk to here on LWN :-P

Illuminating background on Rust

sdalley — Fri, 16 Dec 2022 12:25:41 +0000

That's why people say that Rust is attempting to raise the abstraction in the programming language: you genuinely can offload some of your knowledge into the machine and hope that it would verify that everything is done correctly.

Thanks khim for that very illuminating link on the functional-programming origins of Rust!

Rust in the 6.2 kernel

farnz — Tue, 13 Dec 2022 11:14:13 +0000

Yes - apologies for being unclear. I think of reallocations as an optimized form of allocate, copy and deallocate, so it's implicit to me that reallocation can invalidate the original string.

At heart, this is the same problem as iterator invalidation. You have a reference to some underlying data, and changes to that underlying data can result in your reference no longer being valid. C++ has no compiler checks for reference validity, and relies on the programmer not getting it wrong; this makes some sense, since any check for reference validity is going to be conservative and thus will need overriding from time to time, but the history of "rely on the programmer not getting it wrong" suggests it's not a great decision.

Rust in the 6.2 kernel

oconnor663 — Mon, 12 Dec 2022 21:04:09 +0000

You (farnz) probably already know this but I think it's worth adding: Not only deallocated, but also reallocated. Calling .push_back() or .append() on the original string also potentially invalidates a string_view.

It could be totally fair to describe these as niche issues that don't affect most callers of string_view. But I don't agree with calling someone a "liar" because you (ncm) think it's niche. Or if these issues are new to you, great! None of us is ever done learning.

Rust in the 6.2 kernel

nybble41 — Sat, 03 Dec 2022 00:58:40 +0000

`Either` is one of the two combinators needed to represent arbitrary arithmetic data types, together with the zero/void (`!`) and one/unit (`()`) atomic types. `Either` represents the capacity for sums (enums). Pairs / tuples are their dual, representing product types (structs). They're both very abstract, but sometimes that's exactly what you want.

It's true that you could just create your own custom enum type instead (which would be isomorphic to some tree of nested `Either` types) but your custom type won't benefit from any functions the standard library might provide for working generically with values which might be "either" one type or another. For example, partitioning a list of `Either<A,B>` into a list of `A` and a list of `B`. I'm not as familiar with what Rust provides for working with `Either` values but in Haskell the `Either` type is a key component of the Arrow abstraction, a more powerful (albeit less popular) alternative to monads, where they are used to represent conditional processing (`ArrowChoice`). It also benefits from ready-made instances of `Bifunctor`, `Bifoldable`, and `Bitraversable`.

Rust in the 6.2 kernel

ejr — Fri, 02 Dec 2022 18:32:34 +0000

And I very much thank you for the work. I'm trying to understand Rust's advantages and disadvantages, and this step-wise approach is hugely helpful to me.

Rust in the 6.2 kernel

mlindner — Wed, 30 Nov 2022 12:07:37 +0000

I don't really understand the need of adding the `Either` type, it's just a very very thin wrapper around "enum" with functionality limited to only two types. I wonder why they added it.

Rust in the 6.2 kernel

gasche — Sat, 26 Nov 2022 14:21:52 +0000

Another type that is "too vague" is the type of pair `(A, B)` -- and there are not that many standard-library functions defined on it. But we use it all the time! Honestly I think that the criticism against `Either` here comes from a lack of familiarity, rather than a deep reason. We programmers tend to be more familiar with product types than with sum types; for sums the most well-known is `Option<A>` that is the sum (A + 1), and we still need to time to collectively get used to using anonymous sum types when they make sense.

Rust in the 6.2 kernel

atnot — Thu, 24 Nov 2022 01:48:14 +0000

> the benefits in terms of reusing other people's code (auxiliary support functions etc.) are higher than the cost of the less-specific names.

That is my problem with Either though. It's too vague to enable any reuse at all. I think a look at the associated functions of Option and Result in rust compared to the Either crate illustrate that quite clearly.

Option and Result have dozens of combinators each that compose usefully. You can turn Results into Options, Options into Results, Options into Iterators, Results of Options into Results, Iterators of Options into Options, Iterators of Results into Results...

Meanwhile on the Either side we get:
(https://docs.rs/either/1.8.0/either/enum.Either.html)
- A few map variations, duplicated for left and right of course
- Some forwarding of inner traits
- A bunch of methods that turn it back into the more useful Option and Result types
- Flip, that swaps the sides

This is not a swipe against the authors. It just shows you just can't really do very much with types that are completely unconstrained and semantically meaningless.

Rust in the 6.2 kernel

khim — Wed, 23 Nov 2022 17:17:05 +0000

> - The argument that "defining your own variant with domain-specific names etc." also applies as a criticism of both Option and Result, and the fact that actually we use Option and Result a lot shows that this criticism only goes so far.

How? Option and Result are domain-specific.

Option is the fix for the billion-dollar mistake: it handles the case where object may or may not be present.

Result is for the case where function may return “normal” result or “error result”. Open POSIX specifications (or practically any API specification) and you'll find many such functions.

> - There are generic functions for which Result could be used, but the more symmetric Either is more natural and thus a better API.

It's not enough to have one such function. One function is always better server with ad-hock type. You need series of functions which may share a common type. It's easy to imagine such for Options: lot's of data structures have “leaf nodes”. It even easier to do that with Result: almost all functions which deal with files or network may suffer from the same errors (it doesn't matter whether you are creating file or removing it, if you don't have permission to do that error would be the exact same one).

Just what class of functions do you have in mind where you may have symmetrical two choices and these are the same across the whole range of functions?

Yes, partition_map example makes sense (and that's exactly where itertools are using Either), but Rust is imperative language with for. It's really not clear how often do you even need such thing in Rust.

Rust in the 6.2 kernel

TheGopher — Tue, 22 Nov 2022 14:11:53 +0000

Agree. Having worked with either in scala I can say that the overhead of a dedicated struct/union/variant is minimal - and the readability is much higher! This is the wrong place to be lazy.

Rust in the 6.2 kernel

micka — Mon, 21 Nov 2022 21:16:20 +0000

Looking at the Either crate api, it’s awful. At least Result gives you a meaningful name for each variant (ok, err).
Either gives you a... left and a right. Which is which? If both have the same content type, how do you differentiate?
By the way, some people (like me) can’t tell the right from the left. Or rather can’t give them a name (but never will for example drive on the wrong side of the road). I had to work on codd which used an Either type in another language and could never manage to understand any of it even after multiple years.
I guess it’self OK as write-only code...

Rust in the 6.2 kernel

steveklabnik — Mon, 21 Nov 2022 17:06:53 +0000

A small amount of history here: Rust *did* include Either in its standard library long, long ago. https://doc.rust-lang.org/0.7/std/either.html

My recollection of its removal was a survey of usage, and Result was used instead by 99.9% of the existing code instead.

That said, it is true that the crates.io has high usage; it is the dependent of two *extremely* popular packages, itertools and rayon. I'm not aware of any movement to move it back into the standard library, though.

Rust in the 6.2 kernel

gasche — Sun, 20 Nov 2022 20:42:01 +0000

We've had this argument within the OCaml community, and actually I'm the person who ended up proposing 'either' for the standard library. I agree with your point that, very often, defining your own enum / variant type is the better approach, and also with your point that Option and Result already cover a lot of cases. But:

- The argument that "defining your own variant with domain-specific names etc." also applies as a criticism of both Option and Result, and the fact that actually we use Option and Result a lot shows that this criticism only goes so far. Sometimes there are common scenarios were having a enum of fixed shape with standard name is a good approach, and the benefits in terms of reusing other people's code (auxiliary support functions etc.) are higher than the cost of the less-specific names.

- There are generic functions for which Result could be used, but the more symmetric Either is more natural and thus a better API. (Result and Either are isomorphic so you can clearly always use one instead of the user, the question is how much of a conceptual mismatch this creates.) My main example in OCaml is

val partition_map : ('a -> ('b, 'c) either) -> 'a list -> 'b list * 'c list

It expects a function that, for any value of type 'a, will compute either a 'b or a 'c from it. Then it takes a list of elements of type 'a, and partitions it (using the function) into a list of 'b and a list of 'c. (Interestingly, this is an instance of a sort of generic operation that would split the 'a into an arbitrary sum of possible types, and return as many lists; but that function cannot be expressed easily in the OCaml type system, while 'partition_map' above can.)

Rust in the 6.2 kernel

atnot — Sun, 20 Nov 2022 14:57:08 +0000

> `Either` sounds like a generic type that would have its place in any "standard library for building stuff"

This is somewhat unrelated to the rest of your question, but I personally really don't agree. Rust core already has types for Option and Result, which are the common case for things that can be one of two types. But to me the Either type ends up simultaneously too broad and specific compared to just defining your own enum. The generic "left" and "right" names are cryptic and confusing, there is little commonality betwen users and it has worse type safety and error reporting. It also lacks extensibility by only offering two variants, making for a painful refactoring if you notice you do need three options after all. Lastly a lot of the time when a function can take multiple things, those things usually share some property which can perhaps be better represented by a trait.

Rust in the 6.2 kernel

gasche — Sun, 20 Nov 2022 06:28:51 +0000

> Rust type system *can* express logical proofs

You are changing what you mean by "express logical proofs" quite a bit from what the original poster said with "implement logical proofs", quote:

> With Calculus of Constructions, it is possible to implement logical proofs. This would imply, that Rust implementation needs to be within the logical proofs, for being robust.

With the Calculus of Constructions (or other similar logics), you can define types that correspond to interesting mathematical propositions, and then you can "implement" a proof of this proposition as a program fragment at this type. This is the "Curry-Howard" view of proving things using a typed lambda-calculus, it is what the Calculus of Propositions was designed for, and this is *not* something that is done in Rust.
(Of course, as any reasonably-powerful type system, it is possible through a lot of effort to encode something similar to this process for weaker notions of propositions, using for example singleton types and what not. This does not change the fact that claiming that Rust is related to the Calculus of Constructions is fundamentally nonsensical.)

Now you are talking about a much weaker (but still important/relevant) meaning of "logical proofs", which is: proof of safety guarantees guaranteed by the type system. The idea is not that you can define types to express mathematical properties of interest, but that each type come with behavioral guarantees that gives a property that each program fragment at this type must verify. (Working out precisely how to define these guarantees is the essence of the RustBelt project and Ralf Jung's thesis.) This has, again, nothing to do with the Calculus of Constructions -- well, this work was formulated in Coq, which is maybe how the crackpots above thought to claim a connection.

> and it's not possible to circumvent it (except for bugs in the compiler, of course)

and except for, you know, *unsafe*.

> Which basically means that it's enough to prove that there are no UBs in safe Rust, but not enough to prove much beyond that.

To be fair (I'm not trying to be critical here, and your point at least are informed and make sense), you can get more than the absence of UB when you look at programs at higher type. (For example I would expect that polymorphism gives you representation-independence properties that let you reason on whether some values remain "hidden" inside a module, or what API usage patterns are prevented by the types. Some of this stuff is standard in ML/Haskell grade type systems, and there are new Rust-specific tricks that we can play with lifetimes and the static discipline.)

> Who said it's “wrong?” Even Wikipedia's article on subject start with the exact difference between pure and impure functional programming is a matter of controversy sentence for crying out loud!

There is disagreement on the finer details (and sometimes the word is used in a completely different context, "pure lambda calculus" means something else), but there is no disagreement on the fact that ML-family languages are *impure* functional programming languages; they allow for unrestricted non-termination but also mutable state, exceptions... Anyone in the field agrees that ML-family languages are *not* purely functional.

Rust in the 6.2 kernel

khim — Sat, 19 Nov 2022 22:45:59 +0000

I think you are talking past each other.

> Even if the Rust system *could* in theory express logical proofs (Coq or Agda can, for example), it would be entirely possible to write code that contains bugs in the language (at less precise types).

Rust type system can express logical proofs and it's not possible to circumvent it (except for bugs in the compiler, of course). That's precisely what the Ralf's Phd thesis is about.

Now, the weird part: Rust doesn't contain full-blown dependent types system which can be used pervasively, it's complicated system is centered on lifetimes and soundness.

Which basically means that it's enough to prove that there are no UBs in safe Rust, but not enough to prove much beyond that.

That's still significantly different property from what C/C++ have. And very practically useful.

As for how all that is related to functional programming… it's, basically, impossible to say.

> it starts with a definition of "purely functional programming language" that is wrong?! (ML is not a purely functional programming language.)

Who said it's “wrong?” Even Wikipedia's article on subject start with the exact difference between pure and impure functional programming is a matter of controversy sentence for crying out loud!

The big issue here is that reasonable people define “pureness” differently and then arrive at different conclusions.

Basically the best I can say about that link is… I couldn't say if he's even right or wrong because he talks about things which have not single “proper” definition.

Rust in the 6.2 kernel

gasche — Sat, 19 Nov 2022 21:46:37 +0000

> Your #rude filter seem to be off.

Maybe? I find it rather perplexing to see important technical ideas of our field being cargo-culted around. Why would someone name-drop the Calculus of Constructions (a rather technical topic that is mostly of experts interest) if they clearly don't know what it is? (Otherwise they couldn't claim that Rust is closely related, which is grossly wrong.) It may be that some language communities or discussion spaces are used to this kind of pseudo-technical discourse, but I'm not,
and it hurts.

(I have tried to be short and factual in my post above, which I certainly did not intend to be insulting or deprecating.)

The message I was replying can be decomposed as follows:

> There is some information, that Rust language is near Calculus of Constructions,

This information is wrong, the Rust language is nowhere near the Calculus of Constructions. Rust has a strong type system with polymorphism, sure, but that's about it (so do many other languages). The characteristic feature of the Calculus of Constructions is its very powerful pi-types / dependent abstractions, which are completely absent from Rust -- or most programming languages.

> With Calculus of Constructions, it is possible to implement logical proofs.

This is true.

> This would imply, that Rust implementation needs to be within the logical proofs,
for being robust.

I don't know what this is supposed to mean, but my best guess is that there is a fundamental misunderstanding here. Even if the Rust system *could* in theory express logical proofs (Coq or Agda can, for example), it would be entirely possible to write code that contains bugs in the language (at less precise types).

The blog post that is being cited in the message ( https://www.subarctic.org/is_rust_a_purely_functional_pro... ) is similarly fundamentally wrong. Out of the five subsections, exactly 2 are correct ("What is the Calculus of Constructions" is essentially correct, and "So is Rust purely functional" is arguably correct), the 3 other contain gross mistakes. I mean, this blog post is titled "Is Rust a Purely Functional Programming Language?", and it starts with a definition of "purely functional programming language" that is wrong?! (ML is not a purely functional programming language.)

Rust in the 6.2 kernel

hummassa — Sat, 19 Nov 2022 15:58:10 +0000

Your #rude filter seem to be off.
The poster was making a good point (as does the linked page) that the dependent typing usage possible using Rust traits can be used to construct purely (or semi-purely) funcional Rust programs, that can be checked via automation. The correlation to the kernel is that the same tooling possible with Rust is not possible with C++ or C.

Rust in the 6.2 kernel

amacater — Sat, 19 Nov 2022 15:07:09 +0000

It is, of course, possible that this content is effectively generated as anime-style content and a synthetic voice to preserve someone's
identity for some reason. There aren't many folk contributing to the kernel or other projects entirely pseudonymously but there are
a few, usually to prevent an obvious conflict of interest - maybe to preserve the ability to work on something as their own rather than as their employers' work - or to prevent themselves being marginalised for some other reason.

The synthetic-sounding voice *is* annoying but the content is useful.

Disclaimer: I only watched a few short segments of the video but I could certainly see that the voice would grate after a while.

Rust in the 6.2 kernel

Wol — Sat, 19 Nov 2022 13:37:54 +0000

Or they don't do email interviews and it has to be a video ... if they're a vlogger they might insist ...

Anyways, I consider it useful information to know that their videos are unwatchable ... there might be better ways to do it (I don't see how in this instance) but just telling the truth is not being nasty. Yes it can be told in a nasty way, but here it was just to inform ...

Cheers,
Wol

Rust in the 6.2 kernel

khim — Sat, 19 Nov 2022 11:19:26 +0000

Rust doesn't have Either in it's stdlib. There are either crate with really featureful Either, and it's widely used, but from my understanding Linux kernel doesn't like to depend on external crates.

Rust in the 6.2 kernel

gasche — Sat, 19 Nov 2022 11:06:50 +0000

I can see many reasons why kernel-side programming is a different enough environment to need its own stdlib, and there is nothing fundamentally wrong with that. But on the other hand, I would expect some principles about how to grow this kernel-stdlib. (Some people have experience growing standard libraries for languages/environments and some idea are probably worth porting over from the start.)

For example (this is just a random idea, not a suggestion or anything) Rust-in-kernel could have decided to minimize the diff with outside-kernel Rust code by sticking to subsets of standard APIs or well-known third party packages whenever possible, or taking other steps to avoid divergence and reuse existing design choices. I see no trace of such a process in the [Either patchset]([email protected]/">https://lwn.net/ml/linux-kernel/20221110164152.26136-28-o...), which does not make any mention of pre-existing Either code in Rust outside the kernel.

Rust in the 6.2 kernel

Wol — Sat, 19 Nov 2022 09:42:44 +0000

Bear in mind the kernel does not use the standard C stdlib either. Could they be re-writing the Rust stdlib for the exact same reason, namely (a) the stdlib breaks kernel invariants, and (b) the stdlib does not take into account that hardware breaks stdlib invariants?

Cheers,
Wol

Rust in the 6.2 kernel

gasche — Sat, 19 Nov 2022 09:02:02 +0000

`Either` sounds like a generic type that would have its place in any "standard library for building stuff" for the Rust language, not particularly related to kernel programming. Do I correctly understand that the Rust-in-kernel people are not using any of the existing Rust stdlib, and are in fact reinventing their own stdlib? If so, then what are the mechanisms in place to ensure the ongoing quality and consistency of the design?

Rust in the 6.2 kernel

gasche — Sat, 19 Nov 2022 08:56:04 +0000

I know about the Calculus of Construction, and I can tell you that this is mostly wrong. There is no obvious relation between Rust and the Calculus of Construction (which is about dependent types, no linear types), and the subarctic blog you cite is wrong. This also has no relevance to the Linux kernel or string types whatsoever.

Rust in the 6.2 kernel

eean — Sat, 19 Nov 2022 06:23:24 +0000

they're a vtuber so it actually isn't that bad, ha. it's cool that people are making content like this for different audiences.

Rust in the 6.2 kernel

ssokolow — Fri, 18 Nov 2022 22:09:26 +0000

Unless "you" was being used in the broader sense of "anyone who chooses to interview her", rather than LWN specifically.

Rust in the 6.2 kernel

gspr — Fri, 18 Nov 2022 22:00:39 +0000

Has LWN done a lot of voice interviews? No. The comment clearly brought up something completely irrelevant in order to be mean.

Rust in the 6.2 kernel

zdzichu — Fri, 18 Nov 2022 22:00:04 +0000

Which is, again, not relevant when discussing _writing_and article. It should ever be here on LWN, which we all consider a civilised place.

Rust in the 6.2 kernel

beagnach — Fri, 18 Nov 2022 21:15:40 +0000

It's not actually... the complaint is about a very high-pitched machine-generated fake voice that the developer of the M1 GPU driver adds to their YouTube videos. So not a personal attach.

I also find it unbearable.

Example:
https://www.youtube.com/watch?v=XG9xUxIf73o

Rust in the 6.2 kernel

khim — Fri, 18 Nov 2022 18:02:38 +0000

If someone claims that use of std::string_view doesn't lead to the problems then you can safely say that it's another crop of the “just don't do any mistakes and then C works fine… oh, and don't upgrade the compiler ever because these evil guys make it break it my programs”, just with C++ theme.

Here is discussion about dangers of std::string_view on the Core Guidelines site, here is article with more arguments and there are more, but they all are, obviously, wrong, because admitting that they are right means years of investment in C++ are in jeopardy.

Just leave these guys alone. It's the same thing as with systemd introduction: there would be lots of complains and there would be holdouts and yet it would happen in the classic planck's principle way:

> An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning.

Rust in the 6.2 kernel

farnz — Fri, 18 Nov 2022 17:05:32 +0000

It does need to be kept track of - you need to ensure that the underlying string is not deallocated before the string view is deallocated.

For the very specific case of just passing a string view of a string you own down a call stack, there's no issues, but as soon as the string view relates to a string whose lifespan is not determined purely by the enclosing scope (e.g. because you put the string view in a heap-allocated data structure), you have a lifespan tracking issue to worry about.

Rust in the 6.2 kernel

ncm — Fri, 18 Nov 2022 16:48:48 +0000

In fact, std::string_view validity does not need to be "kept track of". Passed down a call chain, it remains valid throughout, with no phony "complication". When you need to lie to make your case, it tells us all we need to know about your case. Hype reliant on spurious denigration of other languages adds no value here.

Rust in the 6.2 kernel

mathstuf — Fri, 18 Nov 2022 16:11:15 +0000

By "standard" do you mean the `log` crate? I believe some of its guarantees are around flushing and such, so `PR_CONT` might be tougher to implement using the APIs set down in the crate itself.

Rust in the 6.2 kernel

Lumia — Fri, 18 Nov 2022 15:24:15 +0000

What an unpleasant thing to say.

Rust in the 6.2 kernel

corbet — Fri, 18 Nov 2022 14:26:17 +0000

There's an awful lot of Rust infrastructure that has been posted in the past, including complete drivers; it's just not being pushed upstream yet. I've been fairly deliberately looking closely at the code as it heads toward the mainline just because it breaks the problem down into manageable pieces. The whole Rust-for-Linux patch set is a fair amount to absorb all at once.

Rust in the 6.2 kernel

xav — Fri, 18 Nov 2022 14:23:36 +0000

I wonder why they didn't use the standard log::info, log:debug etc. (and just added the missing levels) ?

Rust in the 6.2 kernel

xav — Fri, 18 Nov 2022 14:21:22 +0000

It is yes, but AFAIK it's an oversight and could be replaced soon with a thin pointer - this is alluded to here for example: https://github.com/m-ou-se/rfcs/blob/c-str-literal/text/3...

Rust in the 6.2 kernel

Fri, 18 Nov 2022 13:57:21 +0000

This is a mean thing you're going entirely out of the way to say, particularly given how entirely unrelated it is to the original request for her to _write_ a guest post.

If you choose to post again here, I personally would appreciate it if you would choose to stay remotely on topic and to be a nicer person :-D.