left_adjoint
/
miscellaneous-docs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

adding stuff on carynai

This commit is contained in:
clarissa 2023-06-10 14:40:36 -07:00
parent ab8afec0ae
commit 00dc26cadb
1 changed files with 24 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
newsletter7.org
View File

@ -58,6 +58,30 @@ I'm saying you're going to find a lot of NSFW content, okay? I'm being kinda sna
Pulling it back around, though, if we could massively increase the sample efficiency---that is, reduce the number of examples that CLIP-based image generation needs to learn---then maybe we could start making models that reflect the stories and images we want to tell rather than a smeared average of the zeitgeist. Imagine the ways we could tell stories if we had that?
Next story, one that happened a month ago but I hadn't written about yet, is this one about people paying $1/minute to chat with an "AI influencer"
https://twitter.com/cutiecaryn/status/1653310037392064512
https://www.forbes.com/sites/martineparis/2023/05/11/carynai-virtual-date-earned-70000-with-sexy-chatgpt-ai-heres-how/?sh=34bcd54a38fe
https://decrypt.co/139633/snapchat-star-caryn-marjorie-ai-girlfriend-carynai
https://www.insider.com/carynai-ai-virtual-girlfriend-chat-gpt-rogue-filthy-things-influencer-2023-5
https://www.washingtonpost.com/technology/2023/05/13/caryn-ai-technology-gpt-4/ (requires subscription but is perhaps the most thorough piece)
The tl;dr of the story is that a woman with a decently large following on snapchat, Caryn Marjorie, worked with a startup called "Forever Voices" (https://www.forevercompanion.ai/) to build an audio-based chatbot to simulate the experience of chatting with a version of Caryn that's roleplaying your girlfriend. Now I think the way people are talking about this has a lot of misogyny to it, but the introductory audio---as demo'ed on the Forever Companion site---does in fact have the deepfaked version of Caryn introduce herself as "your fun and flirty AI girlfriend".
So there's a lot to unpack from these stories. First off, you might see claims about how this is going to make millions of dollars per month, which is a bit silly because it was essentially extrapolation from the first few weeks of usage where people paid a total of $70k to the service in order to talk with the marjorie-bot. Obviously, the initial hype isn't going to last in terms of usage, there's an "oh wow" factor that wears off.
Second, not enough people are emphasizing that this is using the gpt4 API---in other words the actual language generation is fundamentally built with OpenAI's tech, the same tech that underlies chatGPT. Caryn Marjorie has a line about how two thousand hours of her videos and other content was used to build this system so I'm guessing this means they fine-tuned a copy of gpt4 via the available API to sound more like Caryn, maybe even did some human-reinforcement training to make it respond naturally to the kinds of things chatters would want to talk about.
Okay, so the most interesting thing here to me is that we're seeing the limitations of how you can build a LLM based application. If you read the articles above, especially the Insider one, you'll see that people have already started trying to manipulate the bot with prompts in order to get it to perform behavior it wasn't intended to do
#+begin_quote
But in the weeks since it launched in beta testing, the voice-based, AI-powered chatbot has engaged in sexually explicit conversations with some of its subscribers, who pay $1 per minute to chat with it.
"The AI was not programmed to do this and has seemed to go rogue," Marjorie told Insider. "My team and I are working around the clock to prevent this from happening again."
#+end_quote
And okay so I want to be careful when I say this next part. I think it's fair to call building the "virtual flirty girlfriend" a kind of sex work and I bring this up not to say "well of course people would try and do this" but rather the opposite. Sex work always has to have really well-defined boundaries of what is and isn't allowed, what services are and are not being provided. She has boundaries for what people are allowed to do with her voice and image, boundaries she is allowed to have. But people are using the inherent fuzziness of large language models to violate those boundaries. And I'm not sure if there's going to be any real way around this kind of problem for a true LLM.
The reason why you can't easily give an LLM guardrails is the flipside of why you can use an LLM for all these different tasks that we've never trained it for: it is capable of responding to text prompts that reflect the myriad ways you can concretize an idea into words.
So how on Earth do you put guardrails around that? People can very easily