The effects that impose disadvantage on ability checks and saves for each of the particular abilities exist on the spell but aren't associated with the activity. I'm going to open a PR for this in a moment.