For a moment I thought you might touch on the mystical and necessary element of belief to direct inquiry.
On a jog the metaphor of water came to me: in the Oceans of Data, there is a Sea of Information that contains Observable Truth. We can observe by putting out rain buckets (experiments) to collect water for our Data Lake. The circumstances for collecting data is an exercise in belief. In science it is the belief in a model from which we generate the hypotheses. That is an exercise in judgement. The bronze water we put into the Data Lake has contaminant artifacts of the experiment including elements of bias in the observers willingness to perceive.
In the context of your story, the unstructured video data is the qualitative data which leads us closer to the Truth or Sea of Information. From which we can also glean quantitative data once we have some basis for a model.
And then there are the red herrings such as Survivorship Bias (https://en.wikipedia.org/wiki/Survivorship_bias). In the patching and reinforcing of the bullet holes from planes that returned (rather than when they didn't return), perhaps interviewing the target demographics who aren't there would have more useful especially if the pool of people interviewed was small or not representative. This is baked into our belief about what can be True and our ability to observe
Others have said this more completely and eloquently. Thank you for the article.
Yeah, so there are two things about this that I think are particularly interesting. First, on things like survivorship bias (and sampling biases, and all that), a lot of qualitative research is particularly vulnerable to them, because the samples are necessarily pretty small. But I do wonder if people could do this sort of qualitative research on much much larger samples, could you solve a lot of those problems? You couldn't entirely, obviously, as those biases still very much exist in data problems too. But it's somewhat of a different class of problem.
And second, for that same reason (samples are small), our default assumption tends to be that qualitative data needs to lead to quantitative data, which is where the truth is. But I'm not sure that's not not just a really strong association we've created. Qualitative data is typically small samples; quantitative data isn't; so we assume the latter is more true. But if the samples of the first were bigger, I'm not sure we wouldn't say it's just as "true."
Both of your points lead to a sort of Chicken and the Egg, Which came First? question. Most definitions of Qualitative are vague to the point of uselessness, so let's use Qualitative Chemistry vs Quantitative Chemistry. Qualitative Chem is the identification of What a compound is. Quantitative is How much do we have. I don't see any reason that we cannot Quantify Qualitative data. It is a different inquiry for Truth. To me this is more along the lines of teasing out cause-and-effect rather than the quantitative mechanics of the observable signal.
The LLM and the web may be a way to do cursory Qualitative research per your notion of wider Qualitative data. How to normalize against the various self-deceptive cognitive biases would be fun. Again, it requires that initial inquiry, the "hunch", i.e. belief in order to start the search in order to see it. Confirmation Bias manifest. =) Testing the Null Hypothesis becomes a bit more fuzzy, but it seems tractable.
One point I suppose I should make is that really just applies to, like, making business decisions or "soft" sciences like that. If you're trying to decide, "What is the best thing for this bar?," I think there is probably more truth - ie, information that reveals the best possible thing to do - in the interviews than most of the data. But if you're doing real science like chemistry, the quantitative stuff is what matters.
For a moment I thought you might touch on the mystical and necessary element of belief to direct inquiry.
On a jog the metaphor of water came to me: in the Oceans of Data, there is a Sea of Information that contains Observable Truth. We can observe by putting out rain buckets (experiments) to collect water for our Data Lake. The circumstances for collecting data is an exercise in belief. In science it is the belief in a model from which we generate the hypotheses. That is an exercise in judgement. The bronze water we put into the Data Lake has contaminant artifacts of the experiment including elements of bias in the observers willingness to perceive.
In the context of your story, the unstructured video data is the qualitative data which leads us closer to the Truth or Sea of Information. From which we can also glean quantitative data once we have some basis for a model.
And then there are the red herrings such as Survivorship Bias (https://en.wikipedia.org/wiki/Survivorship_bias). In the patching and reinforcing of the bullet holes from planes that returned (rather than when they didn't return), perhaps interviewing the target demographics who aren't there would have more useful especially if the pool of people interviewed was small or not representative. This is baked into our belief about what can be True and our ability to observe
Others have said this more completely and eloquently. Thank you for the article.
Cheers,
Joe
Yeah, so there are two things about this that I think are particularly interesting. First, on things like survivorship bias (and sampling biases, and all that), a lot of qualitative research is particularly vulnerable to them, because the samples are necessarily pretty small. But I do wonder if people could do this sort of qualitative research on much much larger samples, could you solve a lot of those problems? You couldn't entirely, obviously, as those biases still very much exist in data problems too. But it's somewhat of a different class of problem.
And second, for that same reason (samples are small), our default assumption tends to be that qualitative data needs to lead to quantitative data, which is where the truth is. But I'm not sure that's not not just a really strong association we've created. Qualitative data is typically small samples; quantitative data isn't; so we assume the latter is more true. But if the samples of the first were bigger, I'm not sure we wouldn't say it's just as "true."
Both of your points lead to a sort of Chicken and the Egg, Which came First? question. Most definitions of Qualitative are vague to the point of uselessness, so let's use Qualitative Chemistry vs Quantitative Chemistry. Qualitative Chem is the identification of What a compound is. Quantitative is How much do we have. I don't see any reason that we cannot Quantify Qualitative data. It is a different inquiry for Truth. To me this is more along the lines of teasing out cause-and-effect rather than the quantitative mechanics of the observable signal.
The LLM and the web may be a way to do cursory Qualitative research per your notion of wider Qualitative data. How to normalize against the various self-deceptive cognitive biases would be fun. Again, it requires that initial inquiry, the "hunch", i.e. belief in order to start the search in order to see it. Confirmation Bias manifest. =) Testing the Null Hypothesis becomes a bit more fuzzy, but it seems tractable.
Onward.
Cheers,
Joe
One point I suppose I should make is that really just applies to, like, making business decisions or "soft" sciences like that. If you're trying to decide, "What is the best thing for this bar?," I think there is probably more truth - ie, information that reveals the best possible thing to do - in the interviews than most of the data. But if you're doing real science like chemistry, the quantitative stuff is what matters.