Instantly share code, notes, and snippets. Star 5,000+ ( 5,000+ ) You must be signed in to star a gist Fork 2,679 ( 2,679 ) You must be signed in to fork a gist Embed Select an option Embed Embed this gist in your website. Share Copy sharable link for this gist. Clone via HTTPS Clone using the web URL. No results found Learn more about clone URLs Clone this repository at <script src="https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95.js"></script> Embed Embed this gist in your website. Share Copy sharable link for this gist. Clone via HTTPS Clone using the web URL. No results found Save karpathy/8627fe009c40f57531cb18360106ce95 to your computer and use it in GitHub Desktop. Embed Embed this gist in your website. Share Copy sharable link for this gist. Clone via HTTPS Clone using the web URL. No results found kaizenman commented Apr 5, 2026 Thanks Andrej! Made a version that generates 🎸 melodies instead of names. They sound weird but I learned the stuff, so now will try to make something that sounds actually good. https://gist.github.com/kaizenman/040a40682b2d047143239d7dd0d15ba3 marosko89 commented Apr 8, 2026 sawirricardo commented Apr 8, 2026 I ported this into zig (mind you i use opus to do this) https://github.com/sawirricardo/ziggpt ariannamethod commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . meet miniKarpathy.py : it's like @karpathy , but micro and a Python script. microKarpathy takes your innocent prompts and like a psychotic linguist tears them apart, brutally tokenizes them, builds a recursive tree of semantic mutations, and then, like Frankenstein having a particularly creative day — reassembles the corpse into something fresh. like flesh. and at the end microKarpathy writes an infernal couplet about the autopsy it just performed. no dependencies. no internet. no autoregressive generation. ain't no conscience either. Karpathy, whose microgpt.py proved that "everything else is just efficiency"б — so it's time to prove that everything at all is just efficiency. kinda little bonus until death makes life what this script makes with your words. 1: An Educational Example unsurprisingly, microKarpathy prefers to explain in a vertical style, because performing open-heart surgery on the complexity of life in this position is much more comfortable and safe than in any other. maybe cause of some other reasons, who knows, microKarpathy like all of us has tendencies. I fed him with "tell me how to build an llm from scratch" — the output below shows in detail what happened after. What just happened? tell → fitness → pressure → structure → system → evolution. Also → stratosphere → ionosphere → firmament. Telling is systemic. The word finds pressure. how → diminuendo → pianissimo → silence. memorandum → dispatch → bulletin. pastiche → satire → irony. How dissolves into music, bureaucracy, and mockery. build → tannin → vinegar → create → design → engineer → fabricate. Also → grammar → language → syntax. Building ferments into language. llm → sign → meaning → speech → symbol → language . Also → space → energy → gravity → force. Three letters find the meaning of language through gravitational physics. scratch → dialect → narrative → sonnet → prose. system → adaptation → evolution → complexity → truth. From scratch = from the beginning of language. Scratch finds poetry. The corpse reads: "precipitation pebble shroud until magnetosphere consumes" . The Three-Act Lecture From Scratch Act I: The Dissection of Core sometimes educational purposes demand hard decisions. microKarpathy takes what you said, strips it down to the bone: stopwords, numbers, capitalization: rejected. punctuation: never heard of her. core words selected by length, rarity, position, and a sprinkle of chaos. single letters: discarded. but what remains to microKarpathy after all this? only the words that matter — or at least, the words that think they do. for each core word, microKarpathy grows a recursive branching tree of mutations. the mutation provider works by hash-embedding cosine similarity : given a word, compute its deterministic FNV-1a hash embedding, measure cosine distance against all vocabulary tokens, blend with Hebbian co-occurrence from the vocabulary file, and return the nearest neighbors, because being lonely makes everything less funny. Act II: Time To Play God when microKarpathy has all the leaves collected, the gravity starts to weigh memory. microKarpathy is a ghost transformer: same architecture as Karpathy's microgpt (multi-head attention, RMSNorm, FFN). but the weights are actually metaweights and don't really exist — they provide deterministic random projections, but training? nope. the vocabulary isn't a dataset, it's a model. words listed near each other in the file are semantically related. the file ordering creates the co-occurrence matrix. the MetaWeights build themselves from this ordering. the reassembled corpse speaks its last words in coda — two lines, rhyming AA: nouns separated by commas, verbs isolated between periods, em-dashes for dramatic pauses. after every period — capitalize. that's the rule. good question. why does this exist? maybe to demonstrate that: a) words are fungible b) meaning is contextual c) prompts are just waiting to be perturbed d) sometimes you need to break things to understand them. or maybe it's just fun to watch language come apart at the seams. repo: https://github.com/ariannamethod/microKarpathy enjoy! till the next time. Nydhal commented Apr 11, 2026 APL port: https://github.com/Nydhal/microgpt.apl - the first port to an array language. Two versions, one token-by-token mirror of this source and one full-sequence rewrite that collapses the per-head loop into ⍤2 . No autograd, explicit matrix gradients, verified bit-for-bit against this file at every layer. ariannamethod commented Apr 11, 2026 what a brilliant idea! awesome lastforkbender commented Apr 13, 2026 ariannamethod commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . Hey @MattWenJun — your emojiGPT was the seed. We forked it and it mutated into caveLLMan. 88 SVG hieroglyphs instead of emoji. A semantic tokenizer compresses any text into 88 cave-painting concepts: "Count Dracula stood in the dark castle and waited" → dark stone and wait man It has Hebbian plasticity — learns from every conversation, no backprop. Invents new symbols when patterns crystallize. Symbols that don't survive die. We fed it Dracula — 2244 sentences devoured, 8 symbols born, 8 died. Runs on notorch — pure C, no PyTorch. The cave painter: caveLLMan Thank you for the spark. qhuang20 commented Apr 16, 2026 really nice work! yukieliot69 commented Apr 30, 2026 Cybertron-Ant commented May 2, 2026 but we were doing this in JavaScript since 2017!! xskongai commented May 6, 2026 Today I tried to implement microGPT by following your code, and when I read this part: I suddenly realized why you are so popular. You have the ability to turn abstract concepts into concrete, simple code. Ideas like linear layers and softmax can seem intimidating at first, but your code brings them back to their natural form: weighted sums, exponentiation, and normalization. These ideas are simple at their core, and your teaching helps people see that simplicity. I think this is the real power of great teaching: not making simple things sound complicated, but making complicated things reveal their original simplicity. Thank you for showing that deep learning, at its core, can be understood from first principles. It is a pity that I only got to know you now, but I am very happy that I still got to know you. napaputteppawan-netizen commented May 7, 2026 -- [[ ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต V.3 FINAL BY มหาเทพธัญญ่า ]] -- local Library = loadstring(game:HttpGet(" https://raw.githubusercontent.com/xHeptc/Kavo-UI-Library/main/source.lua"))( ) local Window = Library.CreateLib("ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต HUB", "BloodTheme")

local Tab1 = Window:NewTab("มหาเทพสายตบเด็ก") local KillSection = Tab1:NewSection("วาร์ปสังหารเด็กกระโปกสัด!!") _G.SilentAim = false _G.AutoKill = false _G.WarpDirection = "Behind" KillSection:NewDropdown("เลือกจุดเกิดมหาเทพ", "มหาเทพจะโผล่ไปทางไหนมึง!!", {"Behind", "Front", "Right", "Left", "Above", "Below"}, function(currentOption) _G.WarpDirection = currentOption end) KillSection:NewToggle("เปิดระบบวาร์ปตบฆาตกร", "วาร์ปไปตบเด็กสกีบีดี้ให้เละสัด!!", function(state) _G.AutoKill = state task.spawn(function() while _G.AutoKill do pcall(function() for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v ~= game.Players.LocalPlayer and v.Character and v.Character:FindFirstChild("HumanoidRootPart") then if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then local Murderer = v.Character.HumanoidRootPart local TargetPos = Murderer.CFrame KillSection:NewToggle("Silent Aim (ยิงเลี้ยวเจาะกะโหลก)", "กดยิงมั่วๆ ก็เข้าหัวสัด!!", function(state) _G.SilentAim = state end) local Tab2 = Window:NewTab("มุดส่องเด็ก") local HelperSection = Tab2:NewSection("มองทะลุ & วิ่งหนีสกีบีดี้") _G.ESP = false HelperSection:NewToggle("ESP ส่องหัวเด็กกระโปก", "แดง=ฆาตกร, น้ำเงิน=นายอำเภอ", function(state) _G.ESP = state task.spawn(function() while _G.ESP do for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v ~= game.Players.LocalPlayer then local hl = v.Character:FindFirstChild("Highlight") or Instance.new("Highlight", v.Character) hl.Enabled = true hl.FillTransparency = 0.5 if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then hl.FillColor = Color3.fromRGB(255, 0, 0) elseif v.Backpack:FindFirstChild("Gun") or v.Character:FindFirstChild("Gun") then hl.FillColor = Color3.fromRGB(0, 0, 255) else hl.FillColor = Color3.fromRGB(0, 255, 0) end end end task.wait(0.5) end for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v.Character:FindFirstChild("Highlight") then v.Character.Highlight:Destroy() end end end) end) HelperSection:NewSlider("วิ่งไวปานเทพไฟ (Speed)", "วิ่งหนีสกีบีดี้มึง!!", 200, 16, function(s) if game.Players.LocalPlayer.Character and game.Players.LocalPlayer.Character:FindFirstChild("Humanoid") then game.Players.LocalPlayer.Character.Humanoid.WalkSpeed = s end end) -- [[ ระบบเบื้องหลัง: มุดวิถีกระสุน (The Real Magic) ]] -- local mt = getrawmetatable(game) local oldNamecall = mt.__namecall setreadonly(mt, false) mt.__namecall = newcclosure(function(self, ...) local Method = getnamecallmethod() local Args = {...} end) setreadonly(mt, true) napaputteppawan-netizen commented May 7, 2026 `-- [[ ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต V.3 FINAL BY มหาเทพธัญญ่า ]] -- local Library = loadstring(game:HttpGet(" https://raw.githubusercontent.com/xHeptc/Kavo-UI-Library/main/source.lua"))( ) local Window = Library.CreateLib("ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต HUB", "BloodTheme") local Tab1 = Window:NewTab("มหาเทพสายตบเด็ก") local KillSection = Tab1:NewSection("วาร์ปสังหารเด็กกระโปกสัด!!") _G.SilentAim = false _G.AutoKill = false _G.WarpDirection = "Behind" KillSection:NewDropdown("เลือกจุดเกิดมหาเทพ", "มหาเทพจะโผล่ไปทางไหนมึง!!", {"Behind", "Front", "Right", "Left", "Above", "Below"}, function(currentOption) _G.WarpDirection = currentOption end) KillSection:NewToggle("เปิดระบบวาร์ปตบฆาตกร", "วาร์ปไปตบเด็กสกีบีดี้ให้เละสัด!!", function(state) _G.AutoKill = state task.spawn(function() while _G.AutoKill do pcall(function() for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v ~= game.Players.LocalPlayer and v.Character and v.Character:FindFirstChild("HumanoidRootPart") then if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then local Murderer = v.Character.HumanoidRootPart local TargetPos = Murderer.CFrame KillSection:NewToggle("Silent Aim (ยิงเลี้ยวเจาะกะโหลก)", "กดยิงมั่วๆ ก็เข้าหัวสัด!!", function(state) _G.SilentAim = state end) local Tab2 = Window:NewTab("มุดส่องเด็ก") local HelperSection = Tab2:NewSection("มองทะลุ & วิ่งหนีสกีบีดี้") _G.ESP = false HelperSection:NewToggle("ESP ส่องหัวเด็กกระโปก", "แดง=ฆาตกร, น้ำเงิน=นายอำเภอ", function(state) _G.ESP = state task.spawn(function() while _G.ESP do for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v ~= game.Players.LocalPlayer then local hl = v.Character:FindFirstChild("Highlight") or Instance.new("Highlight", v.Character) hl.Enabled = true hl.FillTransparency = 0.5 if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then hl.FillColor = Color3.fromRGB(255, 0, 0) elseif v.Backpack:FindFirstChild("Gun") or v.Character:FindFirstChild("Gun") then hl.FillColor = Color3.fromRGB(0, 0, 255) else hl.FillColor = Color3.fromRGB(0, 255, 0) end end end task.wait(0.5) end for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v.Character:FindFirstChild("Highlight") then v.Character.Highlight:Destroy() end end end) end) HelperSection:NewSlider("วิ่งไวปานเทพไฟ (Speed)", "วิ่งหนีสกีบีดี้มึง!!", 200, 16, function(s) if game.Players.LocalPlayer.Character and game.Players.LocalPlayer.Character:FindFirstChild("Humanoid") then game.Players.LocalPlayer.Character.Humanoid.WalkSpeed = s end end) -- [[ ระบบเบื้องหลัง: มุดวิถีกระสุน (The Real Magic) ]] -- local mt = getrawmetatable(game) local oldNamecall = mt.__namecall setreadonly(mt, false) mt.__namecall = newcclosure(function(self, ...) local Method = getnamecallmethod() local Args = {...} end) setreadonly(mt, true) ` napaputteppawan-netizen commented May 7, 2026 `-- [[ ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต V.3 FINAL BY มหาเทพธัญญ่า ]] -- local Library = loadstring(game:HttpGet(" https://raw.githubusercontent.com/xHeptc/Kavo-UI-Library/main/source.lua"))( )()) local Window = Library.CreateLib("ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต HUB", "BloodTheme") local Tab1 = Window:NewTab("มหาเทพสายตบเด็ก") local KillSection = Tab1:NewSection("วาร์ปสังหารเด็กกระโปกสัด!!") _G.SilentAim = false _G.AutoKill = false _G.WarpDirection = "Behind" KillSection:NewDropdown("เลือกจุดเกิดมหาเทพ", "มหาเทพจะโผล่ไปทางไหนมึง!!", {"Behind", "Front", "Right", "Left", "Above", "Below"}, function(currentOption) _G.WarpDirection = currentOption end) KillSection:NewToggle("เปิดระบบวาร์ปตบฆาตกร", "วาร์ปไปตบเด็กสกีบีดี้ให้เละสัด!!", function(state) _G.AutoKill = state task.spawn(function() while _G.AutoKill do pcall(function() for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v ~= game.Players.LocalPlayer and v.Character and v.Character:FindFirstChild("HumanoidRootPart") then if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then local Murderer = v.Character.HumanoidRootPart local TargetPos = Murderer.CFrame KillSection:NewToggle("Silent Aim (ยิงเลี้ยวเจาะกะโหลก)", "กดยิงมั่วๆ ก็เข้าหัวสัด!!", function(state) _G.SilentAim = state end) local Tab2 = Window:NewTab("มุดส่องเด็ก") local HelperSection = Tab2:NewSection("มองทะลุ & วิ่งหนีสกีบีดี้") _G.ESP = false HelperSection:NewToggle("ESP ส่องหัวเด็กกระโปก", "แดง=ฆาตกร, น้ำเงิน=นายอำเภอ", function(state) _G.ESP = state task.spawn(function() while _G.ESP do for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v ~= game.Players.LocalPlayer then local hl = v.Character:FindFirstChild("Highlight") or Instance.new("Highlight", v.Character) hl.Enabled = true hl.FillTransparency = 0.5 if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then hl.FillColor = Color3.fromRGB(255, 0, 0) elseif v.Backpack:FindFirstChild("Gun") or v.Character:FindFirstChild("Gun") then hl.FillColor = Color3.fromRGB(0, 0, 255) else hl.FillColor = Color3.fromRGB(0, 255, 0) end end end task.wait(0.5) end for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v.Character:FindFirstChild("Highlight") then v.Character.Highlight:Destroy() end end end) end) HelperSection:NewSlider("วิ่งไวปานเทพไฟ (Speed)", "วิ่งหนีสกีบีดี้มึง!!", 200, 16, function(s) if game.Players.LocalPlayer.Character and game.Players.LocalPlayer.Character:FindFirstChild("Humanoid") then game.Players.LocalPlayer.Character.Humanoid.WalkSpeed = s end end) -- [[ ระบบเบื้องหลัง: มุดวิถีกระสุน (The Real Magic) ]] -- local mt = getrawmetatable(game) local oldNamecall = mt.__namecall setreadonly(mt, false) mt.__namecall = newcclosure(function(self, ...) local Method = getnamecallmethod() local Args = {...} end) setreadonly(mt, true) ` vazhnov commented May 10, 2026 if you could add a license, that would be great, the total lines of the code still around 200is, : ) It would be great to see any popular open source license: it will allow people to fork, adjust and publish changes legally 😉 iamyb commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . Thanks for the nice code. I made an Excel workbook to visualize it step by step, to better understand how it moves through embeddings, attention, and MLP blocks, all inside Excel. Hopefully it can also be useful to anyone else who wants to learn this 200 line's GPT. yakovenkoroman1993 commented May 30, 2026 Instead of 300 pages of Sebastian Rashka Janhs888 commented Jun 8, 2026 go to school and learn chrisipanaque commented Jun 9, 2026 Please hire me, i need a job please. https://www.linkedin.com/in/chris-ai-engineer/ lxb12123 commented Jun 10, 2026 Thanks Andrej for sharing microgpt.py! I built an interactive 3D tutorial around it to help learners visualize the full pipeline: forward pass, loss, autograd, attention, training, and sampling. The sandbox runs directly in the browser and closely follows the original code. Demo: Interactive 3D tutorial Source: GitHub repository Feedback and corrections are very welcome! :) aiwonderland commented Jun 27, 2026 The zen of small gpts~ pablogiaccaglia commented Jul 7, 2026 this comment section is desperate tomsiwik commented Jul 9, 2026 this comment section is desperate and at this point automated Entrpi commented Jul 10, 2026 Today I had Fable make some meaningful updates to de novo microgpt: https://github.com/Entrpi/microgpt-denovo Entrpi commented Jul 10, 2026 Also, now eemicrogpt runs up to 74000x faster than this CPython on Mac M5: https://github.com/Entrpi/eemicrogpt Hexiaoqiao commented Jul 16, 2026 Thanks @karpathy for your cool works. I am one new GPT learner here. I am confused with L156 where if len(docs)> num_steps there are some docs will never be used to train, right? Please correct me if I missed something. Thanks. tavousi commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . Thank you, Andrej @karpathy , for the brilliant MicroGPT project! It completely inspired my recent work. I adapted your architecture into a character model trained on the Tranco Top 1M sites to generate premium domain names. I’d love to check it out here: https://github.com/tavousi/DomainerGPT danialzivehdadr commented Jul 30, 2026 🤨 GPT لاشی پول همشو من دادم 😂🙏 اهههههههههه الهام بخش خوب آمدی فردا نگی این کپی کار Imad-Jan commented Aug 4, 2026 This is a great example of how simple implementations can teach more than large frameworks. Once the core mechanics are visible, it becomes much easier for people to experiment and create their own versions.