Maybe it is possible to reduce the amount of JS code inside the HTML...
Maybe you can create an object:

{
collapsable: {
onClick: ....,
onKeyDown: ...,
style: ...,
},
...
}

and spread it on this span?
I am just thinking out loud, you can skip it

I prefer implicit attributes (I've explained that recently), and moreover - the ESLint would shout about that.

this can be a single csv with a class that rotates it when it is active

<RiArrowUpSLine className={`arrow-up {expanded ? 'arrow-up--expanded' : ''}`} />

.arrow-up--expanded {
  transform: rotate(180deg);
}

instead of a single collapsable prop I would also add isExpanded: boolean as an initial value, that way we can control if the section is collapsed or not for both states when it is collapsable or not collapsable, so something like:

const Subsection = ({
  title,
  isExpanded,
  collapsable,
  children,
}) => ...

...
Subsection.defaultProps = {
  isExpanded: true,
  collapsable: true,
};

const [expanded, setExpanded] = useState(isExpanded);

prevent any toggle action if section is not collapsible, maybe name disabled fro this prop sound better but collapsable should also be good.

const triggerCollapse = () => {
  if (collapsable) {
   setExpanded(!expanded);
  }
};

Currently we are fine, because the onClick method would be null if a subsection is not collapseable, but I will add it just in case we would refactor something in the future.

I've added additional prop expandedByDefault as you suggested, to make it more reusable in different scenarios. I didn't decide to go for isExpanded, not to mix isExpanded with expanded, as it might cause confusions.