left_adjoint
/
miscellaneous-docs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Adding more newsletter 8

This commit is contained in:
clarissa 2023-06-22 10:01:12 -07:00
parent 0e2a0de1b9
commit 1468056503
1 changed files with 15 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
newsletter8.org
View File

@ -1,15 +1,12 @@
* Outline
+ local portland news https://www.wweek.com/news/city/2023/06/17/ai-call-taker-will-begin-taking-over-police-non-emergency-phone-lines-next-week/
+ https://arxiv.org/abs/2306.08997 (https://dair-community.social/@emilymbender/110556470340507955)
+ https://adrianroselli.com/2023/06/no-ai-will-not-fix-accessibility.html
+ https://vulcan.io/blog/ai-hallucinations-package-risk
+ https://www.opendemocracy.net/en/beyond-trafficking-and-slavery/how-big-tech-and-ai-are-putting-trafficking-survivors-at-risk/
+ https://arxiv.org/abs/2305.00118
+ https://arstechnica.com/information-technology/2023/06/nature-bans-ai-generated-art-from-its-153-year-old-science-journal/
+ https://lcamtuf.substack.com/p/llms-are-better-than-you-think-at
+ https://www.npr.org/2023/06/08/1181097435/desantis-campaign-shares-apparent-ai-generated-fake-images-of-trump-and-fauci
+ https://simonwillison.net/2023/Jun/4/closed-model-training/
+ https://www.npr.org/sections/health-shots/2023/05/31/1179244569/national-eating-disorders-association-phases-out-human-helpline-pivots-to-chatbo , https://www.vice.com/en/article/qjvk97/eating-disorder-helpline-disables-chatbot-for-harmful-responses-after-firing-human-staff
+ https://shkspr.mobi/blog/2023/02/how-much-of-ais-recent-success-is-due-to-the-forer-effect/
+ https://medium.com/@emilymenonbender/thought-experiment-in-the-national-library-of-thailand-f2bf761a8a83
+ https://arxiv.org/abs/2305.07759
@ -27,8 +24,21 @@ Apparently we're moving to a automatic transcription -> natural language process
umm
we understand, right, exactly who the people this will fail are, right? Automatic speech recognition is much more likely to fail for people who speak english with an accent that isn't "midwestern american english", which my gut tells me are the people who are more likely to need help navigating something like a non-emergency services help line in the first place.
we understand, right, exactly who the people this will fail are, right? Automatic speech recognition is much more likely to fail for people who speak english with any kind of accent outside of a very narrow subset, which my gut tells me are the people who are more likely to need help navigating something like a non-emergency services help line in the first place.
So I'm not thrilled about this, especially since it's apparently had serious problems in development that kept it from being rolled out before now. This is, I think, a bad kind of automation: it's not processing information for a person to review it's trying to make decisions in our place.
Also, speaking of help lines no matter what this help line can't be as bad as the eating disorder helpline that fired its staff and tried to replace them with a chatbot!
Also, speaking of help lines no matter what this help line can't be as bad as the eating disorder helpline that fired its staff and tried to replace them with a chatbot! Here's [[https://www.npr.org/sections/health-shots/2023/05/31/1179244569/national-eating-disorders-association-phases-out-human-helpline-pivots-to-chatbo][a story about them making the change]] and then here's [[https://www.vice.com/en/article/qjvk97/eating-disorder-helpline-disables-chatbot-for-harmful-responses-after-firing-human-staff][a story about how it failed and had to be disabled in just a matter of days]]. Now as the first story points out, this wasn't an LLM it was a more classicly designed chatbot, but the lessons here are basically the same: for the love of Minsky stop trying to automate things that are delicate, contextual, situations and if you're going to do it have an actual human review the work first.
The fact that this eating disorder chatbot failed isn't the interesting part, to me, but rather that so little thought is /still/ going into the testing and deployment of these systems. Were I empress of AI deployment my first decree would be that you have to test any natural language processing against the machinations of a troll-brained 12 y/o. My second, unfunded, decree would be that you should really just pay actually good QA people to try and break your chatbots and natural language interfaces: treat it like bug-finding in video games where you're trying the absolutely most asinine things possible to break the system and produce unexpected behavior.
In actual LLM related news though this was definitely food for thought:
https://vulcan.io/blog/ai-hallucinations-package-risk
So as backstory for those who aren't programmers: for most programming languages of any significant popularity there are tools that help you find and install libraries other people have written for the language. By "libraries" we basically just mean software that helps simplify some task so you don't have to build a solution from scratch. For example, to run a web server you're going to want libraries that let you send messages over the internet, that implement the HTTP protocol that the web uses, that keep track of cookies or handle user account creation if needed, &c. If every programmer had to build all of that from scratch every time we'd literally never get anything done. Downloading and installing libraries can be kind of a pain though because libraries depend on other libraries which depend on other libraries which dot dot dot and so we have tools like =pip= in Python, =npm= for JavaScript, or =cabal= for Haskell that in principle should handle all of those dependencies for you. You just need to know the name of the library and if it's been uploaded to the central repository for the language then it will be downloaded as well as everything it needs.
Okay so now to explain the attack: the idea is that chatGPT can potentially generate code that has fake calls to non-existent libraries in response to queries, much like how it can generate citations to papers that don't exist when there isn't actually a high-confidence answer learned from the corpus. So enterprising attackers could potentially try asking code generation questions, note down the false libraries recommended, then quickly create malware with the names of these non-existent libraries and upload those to =pip= or =npm= or whatever. So then, later, when someone performs a similar query and gets the non-existent library recommended to them they'll try to install it and rather than it not existing it will appear to work and the attacker wins.
I don't really know how likely this is to work in practice, but it's an attack vector that had never even occurred to me before and that's really interesting. Again, I'm not actually sure what the defense against such an attack is. I mostly expect it might fail because I don't know how often you'll actually get the exact same fake library recommended. But does it need to work all the time? In a world where tons of developers are using the same LLM to generate code snippets, maybe it only needs to work 1/10000 times to be worth it.