I think the usual way we juggle this quant vs qual thing is that qual studies are just micro-samples.. but the math still works out that if 5 of the last 7 people you talked too ALL mention the same issue, you have a really big issue. Even more strongly so if everyone mentioned it without any prompting. This technique won't let you optimize at the far margins where humans suck at articulating complicated relationships.
So maybe the new generative AI stuff that does seem to do reasonably okay-ish at "summarize this" can help knock that tedium down. Which means us quant folk have best find other ways to be useful
That seems right to me. Data collection is still a problem, but if that was normalized a bit (instead of an NPS survey, click a button to record a 30 second audio review and get paid $10 bucks immediately), it feels you could almost start to run those sorts of qual studies at scale. And if you can do that, it seems like a lot of the "reading the tea leaves" elements that are currently inherent in that sort of work would go away.
A hypothetical "summarize these 50k open ended responses" machine would be really interesting... and I'm pretty sure a number of people are testing out the idea right now... My suspicion is that we're going to find a new boundary where 'just ask people' methodology fails and you need the structure of quantitative count data.... gonna think on this one
My main question with that is, does that allow you to "parse" text data (eg, is this a comment about an iphone?; what's the sentiment of this comment; etc) or can you aggregate is as well (eg, summarize all of these comments; of this set of comments, find me the most representative one)?
The first one seems useful but not *that* novel, whereas the second one is something I've never seen before.
Now imagine being able to do this at global scale, across virtually all human domains. For example, imagine having access to TikTok's data warehouse.
That aside, I propose that your "magical query" technique has a lot more utility than you realize...it is possible to think in this form by simply starting off by *consciously* invoking an omniscient Oracle. In a sense, this is almost exactly what human consciousness is composed of, the difference being with consciousness, the omniscient Oracle is the subconscious mind, which unlike the conscious mind, one has no control over (including the Oracle within it).
Now imagine building a cluster of networked humans that can think in this *and other* advanced forms. Then, imagine what one could do with this power, which one could focus like a laser on any issue (say: war, economics, democracy, geopolitics, etc).
That omniscient Oracle seems like basically what GPT is becoming? It obviously struggles with some stuff and isn't omniscient in a lot of ways and all of that, but it's basically 1) a giant database of everything ever written for which you can 2) write kind of generic queries like
SELECT summarize(themes) FROM books WHERE author = 'shakespeare'
It's not literally that, and you can't be that precise, and all those sorts of things, but if you squint, that seems roughly how it works?
> That omniscient Oracle seems like basically what GPT is becoming?
*Kiiiinda*....but this is a bit different than what I am thinking.
Regardless: ChatGPT and others will be what they are, and ~everyone will have access to "it" (which overlooks some people will have access to non-neutered versions, in addition to entirely novel models not available to the public).
And while, best case, if these things really do turn out to become highly beneficial to humanity, everyone always overlooks one problem (the main one): we are still stuck with all of the biological LLM's running loose on the planet, and if one thinks that a (neutered) silicon-based LLM will be able to coordinate these maniacs (especially when some of them secretly have their finger on the scale(s)), I think they are going to be severely disappointed.
> SELECT summarize(themes) FROM books WHERE author = 'shakespeare'
> It's not literally that, and you can't be that precise, and all those sorts of things, but if you squint, that seems roughly how it works?
Very much agree.....but, you only get what it has to give....and what it has to give is a function of both what it was designed to give, as well as what it is allowed to give (I assume you realize reps from the various 3 letter agencies will be well embedded within OpenAI in some manner by this point - it would be dereliction of (mostly undocumented) duty to do otherwise).
Having all of that power, plus something similarly powerful (that also addresses the biological AI problem, and is beyond the control of bad actors) seems like basic prudent gameplay strategy to me. God knows humanity needs someone on their side for a change.
WeтАЩll also have to grapple with one very messy issue that cloud computing can ignore: AI is opinionated. Though todayтАЩs cloud providers have tremendous power, itтАЩs almost entirely economic. Adam Selipsky and Thomas Kurian can extract rents, but EC2 and Google Compute Engine canтАЩt outright manipulate us
Public AI providers can do both. If nudging Facebook users towards more positive or negative content can change their emotions, imagine the effect of public AI providers turning up the temperature on their core models. That single parameter could control how polite or rude we are to each other in billions of emails and text messages. Other parameters could turn every companyтАЩs support staff into agents of chaos, or embed political bias in every generated piece of text.
ItтАЩs a terrifying amount of powerтАФfar bigger than Elon Musk controlling our Twitter feeds, far more direct than TikTok putting its thumb on its algorithmic scales, and far more precise than RussiaтАЩs disinformation campaigns. And I have no idea what to do about it.
Oh, no disagreement here, other than (as I imagine you know) you're being reductive....there's all these risks, and many others...there are essentially an infinite amount of possibilities. And if you look at humanity's (particularly that of our Dear Leaders and The Experts) gong show of a performance during COVID, a rather minor event on a historic/absolute basis, or during also silly things like the Fake News / Censorship / Russian Propaganda drama (or is this "threat" still ongoing? lol) meme war, what are the odds we're going to transition into the AI world without major turbulence, if not a meltdown? And I think most people don't appreciate with detail what you're getting at: if Facebook/Twitter/Reddit/etc, platforms that have a textbox, some clicky arrows/pictures and a few buttons, can cause all the problems they do, what might AI which can borderline ~think manage to do, even leaving aside that multiple bad actors are going to be playing various games behind the scenes (while telling a different story to the public, as is always the case).
> And I have no idea what to do about it.
Well as luck would have it, I have been working on this general problem for quite some time. I suspect part of the problem is that you mainly/entirely experience reality from your first person in-game character perspective, like this:
Take a top down "god view" approach (say, how a World of Warcraft administrator has to do, watching all the action & nonsense the in-game characters get up to) and things become much simpler, at least with practice. Think of humans as semi-intelligent, semi-conscious agents in a video game, with all activity, *each individual action* powered by this:
...according to all the borderline insane training each agent has received (which they each then proceed to misinterpret in various hilarious ways), and the whole thing starts to make a lot of sense. Remove yourself from the system, start with a completely blank slate, add planet earth and a couple billion of agents, and move forward thinking based on first principles, making no silly errors along the way. What is going on emerges, clearly and simply.
What to do about all this though....well, that's another story. Maybe what the world needs is a competitor (to literally everything) platform? I think once you understand the field well enough, it becomes less a question of how could this succeed, and more like *how could this possibly fail*?
That'd cost a lot of money though, and I have approximately none.
I think the usual way we juggle this quant vs qual thing is that qual studies are just micro-samples.. but the math still works out that if 5 of the last 7 people you talked too ALL mention the same issue, you have a really big issue. Even more strongly so if everyone mentioned it without any prompting. This technique won't let you optimize at the far margins where humans suck at articulating complicated relationships.
So maybe the new generative AI stuff that does seem to do reasonably okay-ish at "summarize this" can help knock that tedium down. Which means us quant folk have best find other ways to be useful
That seems right to me. Data collection is still a problem, but if that was normalized a bit (instead of an NPS survey, click a button to record a 30 second audio review and get paid $10 bucks immediately), it feels you could almost start to run those sorts of qual studies at scale. And if you can do that, it seems like a lot of the "reading the tea leaves" elements that are currently inherent in that sort of work would go away.
A hypothetical "summarize these 50k open ended responses" machine would be really interesting... and I'm pretty sure a number of people are testing out the idea right now... My suspicion is that we're going to find a new boundary where 'just ask people' methodology fails and you need the structure of quantitative count data.... gonna think on this one
Fair, I could imagine things getting kinda weird if anything like this actually worked.
We have actually built this machine :)
Flexor.ai
What does "delivering textual data into the hands of every data practitioner" mean though?
We enable running sql queries from your warehouse/bi tool
That run on your entire textual data source.
For example:
SELECT
*
FROM (
SELECT
EMAIL_CONTENT,
EMAIL_TIMESTAMP,
FLEX(FLEXOR_ID, 'Is the client flagging an error on an invoice?') AS invoice_error,
FROM
Emails_table
)
WHERE
invoice_error_prediction = true
My main question with that is, does that allow you to "parse" text data (eg, is this a comment about an iphone?; what's the sentiment of this comment; etc) or can you aggregate is as well (eg, summarize all of these comments; of this set of comments, find me the most representative one)?
The first one seems useful but not *that* novel, whereas the second one is something I've never seen before.
Now imagine being able to do this at global scale, across virtually all human domains. For example, imagine having access to TikTok's data warehouse.
That aside, I propose that your "magical query" technique has a lot more utility than you realize...it is possible to think in this form by simply starting off by *consciously* invoking an omniscient Oracle. In a sense, this is almost exactly what human consciousness is composed of, the difference being with consciousness, the omniscient Oracle is the subconscious mind, which unlike the conscious mind, one has no control over (including the Oracle within it).
Now imagine building a cluster of networked humans that can think in this *and other* advanced forms. Then, imagine what one could do with this power, which one could focus like a laser on any issue (say: war, economics, democracy, geopolitics, etc).
That omniscient Oracle seems like basically what GPT is becoming? It obviously struggles with some stuff and isn't omniscient in a lot of ways and all of that, but it's basically 1) a giant database of everything ever written for which you can 2) write kind of generic queries like
SELECT summarize(themes) FROM books WHERE author = 'shakespeare'
It's not literally that, and you can't be that precise, and all those sorts of things, but if you squint, that seems roughly how it works?
> That omniscient Oracle seems like basically what GPT is becoming?
*Kiiiinda*....but this is a bit different than what I am thinking.
Regardless: ChatGPT and others will be what they are, and ~everyone will have access to "it" (which overlooks some people will have access to non-neutered versions, in addition to entirely novel models not available to the public).
And while, best case, if these things really do turn out to become highly beneficial to humanity, everyone always overlooks one problem (the main one): we are still stuck with all of the biological LLM's running loose on the planet, and if one thinks that a (neutered) silicon-based LLM will be able to coordinate these maniacs (especially when some of them secretly have their finger on the scale(s)), I think they are going to be severely disappointed.
> SELECT summarize(themes) FROM books WHERE author = 'shakespeare'
> It's not literally that, and you can't be that precise, and all those sorts of things, but if you squint, that seems roughly how it works?
Very much agree.....but, you only get what it has to give....and what it has to give is a function of both what it was designed to give, as well as what it is allowed to give (I assume you realize reps from the various 3 letter agencies will be well embedded within OpenAI in some manner by this point - it would be dereliction of (mostly undocumented) duty to do otherwise).
Having all of that power, plus something similarly powerful (that also addresses the biological AI problem, and is beyond the control of bad actors) seems like basic prudent gameplay strategy to me. God knows humanity needs someone on their side for a change.
Yeah, so in this post (https://benn.substack.com/p/tribal-accelerationism), there was a third section that I cut that was related to this. Basically, another interpretation of the OpenAI board firing Altman was it shows how OpenAI is a company that is controlled entirely by a very tiny group of people who can do very irrational things, in secret, without any explation. In this case, the irrational thing they did was intentionally step on a rake, but it certainly could've been something more nefarious. And really, if we take anything away from this at all, it should be that - that very large tech companies have the capability to concentrate enormous amounts of power in the hands of a half-dozen people. And for foundational model providers, that seems particularly fraught. From here https://benn.substack.com/p/the-public-imagination#:~:text=We%E2%80%99ll%20also%20have,do%20about%20it.:
WeтАЩll also have to grapple with one very messy issue that cloud computing can ignore: AI is opinionated. Though todayтАЩs cloud providers have tremendous power, itтАЩs almost entirely economic. Adam Selipsky and Thomas Kurian can extract rents, but EC2 and Google Compute Engine canтАЩt outright manipulate us
Public AI providers can do both. If nudging Facebook users towards more positive or negative content can change their emotions, imagine the effect of public AI providers turning up the temperature on their core models. That single parameter could control how polite or rude we are to each other in billions of emails and text messages. Other parameters could turn every companyтАЩs support staff into agents of chaos, or embed political bias in every generated piece of text.
ItтАЩs a terrifying amount of powerтАФfar bigger than Elon Musk controlling our Twitter feeds, far more direct than TikTok putting its thumb on its algorithmic scales, and far more precise than RussiaтАЩs disinformation campaigns. And I have no idea what to do about it.
Oh, no disagreement here, other than (as I imagine you know) you're being reductive....there's all these risks, and many others...there are essentially an infinite amount of possibilities. And if you look at humanity's (particularly that of our Dear Leaders and The Experts) gong show of a performance during COVID, a rather minor event on a historic/absolute basis, or during also silly things like the Fake News / Censorship / Russian Propaganda drama (or is this "threat" still ongoing? lol) meme war, what are the odds we're going to transition into the AI world without major turbulence, if not a meltdown? And I think most people don't appreciate with detail what you're getting at: if Facebook/Twitter/Reddit/etc, platforms that have a textbox, some clicky arrows/pictures and a few buttons, can cause all the problems they do, what might AI which can borderline ~think manage to do, even leaving aside that multiple bad actors are going to be playing various games behind the scenes (while telling a different story to the public, as is always the case).
> And I have no idea what to do about it.
Well as luck would have it, I have been working on this general problem for quite some time. I suspect part of the problem is that you mainly/entirely experience reality from your first person in-game character perspective, like this:
https://qualiacomputing.com/2022/12/28/cartoon-epistemology-by-steven-lehar-2003/
Take a top down "god view" approach (say, how a World of Warcraft administrator has to do, watching all the action & nonsense the in-game characters get up to) and things become much simpler, at least with practice. Think of humans as semi-intelligent, semi-conscious agents in a video game, with all activity, *each individual action* powered by this:
https://i.imgur.com/wiFCZsZ.jpg
...according to all the borderline insane training each agent has received (which they each then proceed to misinterpret in various hilarious ways), and the whole thing starts to make a lot of sense. Remove yourself from the system, start with a completely blank slate, add planet earth and a couple billion of agents, and move forward thinking based on first principles, making no silly errors along the way. What is going on emerges, clearly and simply.
What to do about all this though....well, that's another story. Maybe what the world needs is a competitor (to literally everything) platform? I think once you understand the field well enough, it becomes less a question of how could this succeed, and more like *how could this possibly fail*?
That'd cost a lot of money though, and I have approximately none.