Simon Willison's Recent LinkedIn Posts

Tom Scott just wrapped up an astonishing ten year streak of weekly YouTube videos. I wrote about that, and added some thoughts about the formidable power of escalating streaks

Today, we're joined by Simon Willison, independent researcher and creator of Datasette, to discuss the ways software developers and engineers can take advantage of large language models (LLMs) to boost their productivity. We dig into Simon’s own workflows and how he uses popular models like ChatGPT and Anthropic’s Claude to write and test hundreds of lines of code while out walking his dog. We review Simon’s favorite prompting and debugging techniques, his strategies for sidestepping the limitations of contemporary models, how he uses Claude’s Artifacts feature for rapid prototyping, his thoughts on the use and impact of vision models, the role he sees for open source models and local LLMs, and much more. 🎧 / 🎥 Listen or watch the full episode on our page: https://twimlai.com/go/701.

Here are detailed notes (as an annotated presentation) to accompany the video of a talk I gave last week on using Large Language Models on the command-line https://lnkd.in/gRcEwxji

I built a ChatGPT plugin to answer questions about data hosted in Datasette #data #chatgpt

I gave a talk for the GitHub OCTO (Office of the CTO, love that acronym) speaker series about some of the work I've been doing with my open source Datasette project to create a personal data warehouse. Here's the talk video along with annotated screenshots, links and notes:

I gave a couple of AI-related keynotes recently, both of which are now published on my blog accompanied by detailed annotated transcripts and slides. The first was a keynote at PyCon US 2024 called "Imitation Intelligence" - it discusses LLM topics that are most relevant to the Python community, and could be useful if you're looking to catch up on the field generally https://lnkd.in/dBcwpz-d The second was a keynote at the AI Engineer World's Fair, where I talked about the latest key developments in the field, in particular the fact that OpenAI's GPT-4 is no longer the only model in its class. https://lnkd.in/dPy_RwPd

In this week’s episode, Simon Willison, the creator of Datasette, an open-source data exploration and publishing tool, joins Nikita Roy to kick off a two part special episode. They discuss recent events at OpenAI and the latest features announced at OpenAI’s first developer conference earlier this month. Key highlights of the episode include: 1️⃣ Exploring New Capabilities from OpenAI: Simon explains the latest features from OpenAI, including the GPT-4 Turbo model for processing large documents, Whisper 3 for enhanced speech-to-text, creating GPTs (a customizable version of ChatGPT), an Assistant API for chatbots, GPT Vision API for image analysis, and the implications of an OpenAI GPT store for the industry. 2️⃣ Mitigating Security Risks in Generative AI Apps: Simon discusses "prompt injection," a term he is known for coining. This security vulnerability in large language models (LLMs) allows attackers to create prompts that manipulate model responses, leading to unauthorized access or data manipulation. Addressing this issue is crucial for the safe development of certain generative AI applications. 3️⃣ Leveraging Small Language Models: Simon explores the potential of small language models for journalists to use on their devices for analyzing sensitive data reducing dependence on external platforms. Simon's new open-source LLM tool simplifies experimenting with models like Meta’s LLaMA model on a local level. Stay tuned for the second part of this episode, coming out next week, where Simon shares how he's building open source data journalism tools to empower journalists. 🎧 Check out this episode on your preferred podcast platform: https://lnkd.in/eWSPFJez ✉️ Newsroom Robots now has a newsletter! Sign up: https://lnkd.in/efD52CdX

The GPT-4 Turbo with Vision API was released to general availability today. Datasette Extract is my first product to use it - it's a plugin for Datasette that adds the ability to load unstructured text and images into database tables, using GPT-4 Turbo to clean the data up to match a table schema Here's a video demo of the feature in action (3m43s) https://lnkd.in/gcN5fZfc

I interviewed Rajiv Sinclair about his team's new project, VERDAD - an outstanding piece of data journalism that tracks 48 US talk radio stations (many in Spanish), transcribes the audio and uses Gemini 1.5 to help identify potential misinformation

I collected together detailed notes on today's OpenAI announcement - they called it "structured outputs in the API" but there's a lot of depth to it, including a new model release which is half the price of the previous GPT-4o, meaning it's now OpenAI's cheapest model for processing image inputs (cheaper even than GPT-4o-mini). https://lnkd.in/g-duQxW5

My attempt to debunk a common misconception about how LLMs work Training is not the same as chatting: ChatGPT and other LLMs don’t remember everything you say

I wrote about prompt injection attacks against AI personal assistant applications built on top of ChatGPT and other Large Language Models Building a secure AI personal assistant is a fiendishly difficult problem - I propose an approach that might help The Dual LLM pattern for building AI assistants that can resist prompt injection https://lnkd.in/gqkkhB66

Wrote up some notes on the new Qwen2.5-Coder-32B model, which is the first model I've run on my own Mac (64GB M2) that appears to be highly competent at writing code https://lnkd.in/gdpQdkkz

I'm using Advent of Code to learn Rust this year, as an experiment in AI-assisted learning using Copilot and ChatGPT. I wrote up my progress so far - I think using these AI tools as a super-smart-and-super-dumb teaching assistant is a really promising pattern

OpenAI's ChatGPT has a lot of momentum, but the competition is really heating up (from proprietary *and* open source solutions). Simon Willison catches us up on all the action:

I wrote about the AI trust crisis: I'm increasingly seeing people flat out not believe AI companies when they say "we won't train models on your private data", which is a big problem. We need these companies to earn back our trust!

My notes on OpenAI's new GPT-4o (the "o" is for "omni") model release. Short version: it's not a huge leap in "intelligence" over GPT-4, but the multi-model audio support, the ability to output images (which appears a whole lot more sophisticated than DALL-E) and the drop in price are all very interesting new capabilities.

There's so much useful material in this course about LLMs, RAG and related concepts I wrote up my own talk here: Language models on the command-line https://lnkd.in/gRcEwxji

I put together some notes on today's release by Anthropic of Claude "Computer Use" mode - which lets you pass screenshots of a computer desktop to the model such that it can reply with coordinates for where to click and what to type to drive that computer. Anthropic also provide a convenient Docker container you can try out which runs an Ubuntu GUI and configures the model to drive it based on your prompts.

Wrote up some thoughts on the most interesting pieces of the Apple Intelligence announcements: - On-device models that can outsource to Apple’s servers with cryptographic guarantees of privacy are really interesting - Restricting image generation to three approved styles (sketch, illustration and animation) avoids a whole host of complicated ethical concerns - App Intents look like a minefield for prompt injection Thoughts on the WWDC 2024 keynote on Apple Intelligence: https://lnkd.in/g6ztZGku Plus some follow-up notes on Apple's Private Cloud Compute: https://lnkd.in/g2XPTAqB and on the details of their new foundation models: https://lnkd.in/gcN8NjzZ

JustHTML is a really interesting example of agentic coding in action - building a new HTML parsing library for Python by using coding agents to brute force the existing 9,200 test HTML5 conformance suite

I've been having fun building web apps using my iPhone for a while now, and this morning I took on my most ambitious project yet: scraping the schedule for a conference (Open Sauce 2025 in San Francisco) and turning that into a web app with an "export to calendar" button that works with Apple and Google Calendars I used OpenAI Codex (in the ChatGPT iPhone app) to scrape the schedule (I'm calling this "vibe scraping") and turn it into JSON, then Claude Artifacts to build the finished app Full details plus the prompts and transcripts here: https://lnkd.in/gs98EzUg

I collected notes on OpenAI's announcement that an unreleased and unnamed model of theirs scored at a gold medal level in this year's International Mathematics Olympiad - without the model using any tools (Python, Mathematica etc). This is a genuinely impressive and surprising result. https://lnkd.in/gr5FtxBq OpenAI's Noam Brown: > When you work at a frontier lab, you usually know where frontier capabilities are months before anyone else. But this result is brand new, using recently developed techniques. It was a surprise even to many researchers at OpenAI. Today, everyone gets to see where the frontier is.

I've had preview access to GPT-5 for a couple of weeks, so I have a lot to say about it. Here's my first post, focusing just on core characteristics, pricing (it's VERY competitively priced) and interesting details from the GPT-5 system card

I see a lot of complaints about untested AI slop in pull requests. Submitting those is a dereliction of duty as a software engineer: Your job is to deliver code you have proven to work

Claude Artifacts is the feature of https://claude.ai/ where you can prompt it to create interactive HTML+JavaScript applications right there in the Claude app I've been using it a LOT. Here's everything I've built with Claude Artifacts in just the past week - 14 different tools: https://lnkd.in/g-KrMfcn

I wrote up extensive notes to accompany a talk I gave last week trying to summarize everything we have figured out about Large Language Models (the tech behind ChatGPT et al) over the last few years. The video of the talk is available too.

To round off the year, I pulled together notes on all of the things that we figured out about AI in 2023. We figured out a lot of stuff!

Things we learned about LLMs in 2024 Here's the table of contents for my review of 2024 in Large Language Models - I summarized the key themes and pivotal moments from a very busy year https://lnkd.in/gDY4pNeH

Anthropic: A postmortem of three recent issues Anthropic had a very bad month in terms of model reliability: Between August and early September, three infrastructure bugs intermittently degraded Claude's response quality. We've now resolved these issues and want to explain what happened. [...] To state it plainly: We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone. [...] We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation. I'm really glad Anthropic are publishing this in so much detail. Their reputation for serving their models reliably has taken a notable hit. I hadn't appreciated the additional complexity caused by their mixture of different serving platforms: We deploy Claude across multiple hardware platforms, namely AWS Trainium, NVIDIA GPUs, and Google TPUs. [...] Each hardware platform has different characteristics and requires specific optimizations. It sounds like the problems came down to three separate bugs which unfortunately came along very close to each other. Anthropic also note that their privacy practices made investigating the issues particularly difficult: The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes. Our own privacy practices also created challenges in investigating reports. Our internal privacy and security controls limit how and when engineers can access user interactions with Claude, in particular when those interactions are not reported to us as feedback. This protects user privacy but prevents engineers from examining the problematic interactions needed to identify or reproduce bugs. The code examples they provide to illustrate a TPU-specific bug show that they use Python and JAX as part of their serving layer. Tags: python , ai , postmortem , generative-ai , llms , anthropic , claude

Chinese AI lab Z.ai (previously called Zhipu AI) released two new MIT licensed open weight LLMs yesterday - GLM-4.5 and GLM-4.5 Air - and they are very impressive. Here are my initial impressions formed against their hosted models: https://lnkd.in/gQg-9WdM ... and then I got a 3 bit quantized MLX version running on my Mac - using 48GB out of my 64GB total RAM - and it wrote me a working space invaders clone in a JavaScript from a single prompt! https://lnkd.in/gepM72jt These local models have got really good now. This is the same laptop I used to run the original LLaMA >2 years ago.

Over the past two years I've built more than 150 little "HTML tools" - single page interactive HTML+JavaScript utilities that do one useful thing. Almost all of them were vibe-coded with the assistance of Claude or ChatGPT or Gemini or Claude Code or Codex CLI I wrote about the patterns I've discovered along the way that work really well for this kind of development - things like storing state in URLs and localStorage, loading dependencies from CDNs and avoiding React-induced build steps

I published my third annual roundup of the last twelve months in LLMs. This one has 26 sections, starting with reasoning models and coding agents and working through Chinese open weight models, vibe coding, pelicans riding bicycles, coding on my phone and much more:

I wrote about "Designing agentic loops" - a new key skill that's needed to get the most out of coding agents like Anthropic's Claude Code and OpenAI's Codex CLI. A surprisingly large number of difficult programming challenges can be attacked using a brute-force approach if you figure out the right success criteria and provide access to the necessary tools in a carefully constructed environment. Includes tips on safely running these tools in YOLO mode (where you don't have to approve every action they take):

Leaked Google document: “We Have No Moat, And Neither Does OpenAI” The most interesting thing I've read recently about LLMs - a purportedly leaked document from a researcher at Google talking about the huge strategic impact open source models are having

GitHub released Spark yesterday, their take on the prompt-to-app pattern. Give it a prompt and it builds a full React application with optional simple server-side key/value storage and even the ability to run custom LLM prompts. I'm impressed by it. I used Spark itself to build an unofficial documentation app that reverse-engineered its own system prompt to learn more about how it worked. Detailed notes here: https://lnkd.in/gcsti5bd

📣📣📣 Exciting news! PyBay 2023 is in two weeks, and we're extending the $25 discount on tickets until September 30th. Grab your discounted ticket now and be part of the Python excitement! 🚀 https://lnkd.in/gt_HPiqn

I wrote up some thoughts on effective software engineering practices for medium to large teams. Here's the TLDR version - more details on each one in my article. #softwareengineering - Documentation in the same repo as the code - Mechanisms for creating test data - Rock solid database migrations - Templates for new projects and components - Automated code formatting - Tested, automated process for new development environments - Automated preview environments

I'm on Canadian radio this morning! I was interviewed by CBC Day 6 on the subject of prompt engineering It's a seven minute segment at the start of the show - you can listen online here: https://lnkd.in/gT3DigUh Or read my annotated transcript on my blog: https://lnkd.in/gvStCqgE

I wrote about what I consider to be "The Perfect Commit" - a commit that bundles together the implementation, tests, updated documentation and a link to an accompanying issue thread