1) Sure, that's fair. It's definitely not precise. You couldn't do anything properly scientific in this way I don't think. It'd really have to be more like humanities research, where there's not only variability, but sometimes outright disagreement. (Though as I say that, I do wonder if there'd be some sort of rough "central limit theorem" with this, where if you have large enough samples, every model built in broadly similar ways would converge-ish. But who knows.)
2) I could see "different models do different things" also be related to the topic modeling stuff you're describing too. Even if LLMs (and AI generally, mostly) wasn't fundamentally probabilistic, you can always get different results by asking questions in slightly different ways, training the models differently, using slightly different models, and so on. So even if one company had a standard approach for how they do it, it's almost more cultural. The research analogy might still work there: Give 20 interviews to one research team; they'll give you X back. Give 20+5 to the same team, you'll probably get Xish. Give 20 to a different team, who knows? You could get something entirely different.
1) Sure, that's fair. It's definitely not precise. You couldn't do anything properly scientific in this way I don't think. It'd really have to be more like humanities research, where there's not only variability, but sometimes outright disagreement. (Though as I say that, I do wonder if there'd be some sort of rough "central limit theorem" with this, where if you have large enough samples, every model built in broadly similar ways would converge-ish. But who knows.)
2) I could see "different models do different things" also be related to the topic modeling stuff you're describing too. Even if LLMs (and AI generally, mostly) wasn't fundamentally probabilistic, you can always get different results by asking questions in slightly different ways, training the models differently, using slightly different models, and so on. So even if one company had a standard approach for how they do it, it's almost more cultural. The research analogy might still work there: Give 20 interviews to one research team; they'll give you X back. Give 20+5 to the same team, you'll probably get Xish. Give 20 to a different team, who knows? You could get something entirely different.