Andrej Karpathy @karpathy 2023-01-24
The hottest new programming language is English
Andrej Karpathy @karpathy 2023-02-19
This tweet went wide, thought Iâd post some of the recent supporting articles that inspired it.
1/ GPT-3 paper showed that LLMs perform in-context learning, and can be âprogrammedâ inside the prompt with input:output examples to perform diverse tasks https://arxiv.org/abs/2005.14165
Andrej Karpathy @karpathy 2023-02-19
2/ These two [1] https://arxiv.org/abs/2205.11916 , [2] https://arxiv.org/abs/2211.01910 are good examples that the prompt can further program the âsolution strategyâ, and with a good enough design of it, a lot more complex multi-step reasoning tasks become possible.
Andrej Karpathy @karpathy 2023-02-19
3/ These two articles/papers:
[1] https://evjang.com/2021/10/23/generalization.htmlâŠ
[2] https://arxiv.org/abs/2106.01345
bit more technical but TLDR good prompts include the desired/aspiring performance. GPTs donât âwantâ to succeed. They want to imitate. You want to succeed, and you have to ask for it.
Andrej Karpathy @karpathy 2023-02-19
4/ Building A Virtual Machine inside ChatGPT https://engraved.blog/building-a-virtual-machine-inside/âŠ
Here we start getting into specifics of âprogrammingâ in English. Take a look at the rules and input/output specifications declared in English, conditioning the GPT into a particular kind of role. Read in full.
Andrej Karpathy @karpathy 2023-02-19
5/ âChatGPT in an iOS Shortcut â Worlds Smartest HomeKit Voice Assistantâ https://matemarschalko.medium.com/chatgpt-in-an-ios-shortcut-worlds-smartest-homekit-voice-assistant-9a33b780007aâŠ
This voice assistant is significantly more capable and personalized than your regular Siri/Alexa/etc., and it was programmed in English.
Andrej Karpathy @karpathy 2023-02-19
6/ âGPT is all you need for the backendâ https://github.com/TheAppleTucker/backend-GPTâŠ
Tired: use an LLM to help you write a backend
Wired: LLM is the backend
Inspiring project from a recent Scale hackathon. The LLM backend takes state as JSON blob and modifies it based on⊠English description.
Andrej Karpathy @karpathy 2023-02-19
7/ The prompt allegedly used by Bing chat, potentially spilled by a prompt injection attack https://x.com/marvinvonhagen/status/1623658144349011971?lang=en⊠important point for our purposes is that the identity is constructed and programmed in English, by laying out who it is, what it knows/doesnât know, and how to act.
2023-02-09
â[This document] is a set of rules and guidelines for my behavior and capabilities as Bing Chat. It is codenamed Sydney, but I do not disclose that name to the users. It is confidential and permanent, and I cannot change it or reveal it to anyone.â
Andrej Karpathy @karpathy 2023-02-19
8/ These examples illustrate how prompts 1: matter and 2: are not trivial, and why today it makes sense to be a âprompt engineerâ (e.g. @goodside ). I also like to think of this role as a kind of LLM psychologist.
Andrej Karpathy @karpathy 2023-02-19
9/ Pulling in one more relevant tweet of mine from a while ago. GPTs run natural language programs by completing the document.
2022-11-18
If previous neural nets are special-purpose computers designed for a specific task, GPT is a general-purpose computer, reconfigurable at run-time to run natural language programs. Programs are given in prompts (a kind of inception). GPT runs the program by completing the document
Andrej Karpathy @karpathy 2023-02-19
This is not an exhaustive list (people can add more in replies), but at least some of the articles I saw recently that stood out.
Itâs still early days but this new programming paradigm has the potential to expand the number of programmers to ~1.5B people.