The Deep Feed

For months, the industry has operated on a diet of 'vibe checks'. A developer tries a new model, finds it writes a decent email or a clean function, and declares it a winner. This approach is dangerous. It ignores the statistical variance and the specific failures that emerge under pressure. When Anthropic released Sonnet 5, the temptation was to simply feel its way through a few prompts. Instead, a more disciplined approach was required: building a repeatable, objective evaluation harness.

The How I AI Bench

To move beyond anecdotal evidence, a custom evaluation harness—the How I AI Bench—was constructed. This wasn't a simple automated script, but a hybrid system. It combined human scoring (70%) with LLM-as-judge scoring (30%). The logic is sound: humans are better at detecting the subtle 'soul' or personality of an agent, while LLMs are efficient at checking technical correctness against a rubric. By running 64 generations blind against competitors like GPT-5.5 and Gemini 3 Pro, the true performance delta becomes visible.

Trusting a single scoring method is a recipe for delusion; you need the friction between human intuition and machine logic.

The results of such a test reveal that model performance is highly task-specific. A model that excels at generating a Product Requirements Document (PRD) might fail miserably at maintaining a consistent agentic personality during a long-running task. We see a divergence between 'intelligence' as a general concept and 'utility' as a functional metric. Sonnet 5, for instance, shows specific strengths in prototype generation that distinguish it from the more generalist approaches of its peers.

The era of the 'magic' model is over. We are entering the era of the 'evaluated' model. For any agency or product team, the competitive advantage will not come from knowing which model is 'better' in a general sense, but from knowing which model is mathematically superior for your specific, repeatable workflows.

The traditional Product Manager has spent years as a professional coordinator. They sit in the middle of engineering, design, and stakeholders, ensuring everyone is aligned and the roadmap is clear. This role is dying. As AI tools become capable of handling the administrative and connective tissue of product development, the PM who only coordinates will find themselves redundant. The new PM must be a builder.

Ascending the Rungs

Leverage in the AI era can be categorised into three distinct ladders: Personal, Product, and Systems. Personal leverage is about your own output—using AI to draft documents or research topics faster. Product leverage is about the speed of shipping—using AI to prototype real code or query databases without waiting for an engineer. Systems leverage is the highest form, where you build repeatable, automated workflows that allow AI to complete multi-step tasks with minimal oversight.

The most successful PMs are moving from people-management to agent-management.

To reach the top of these ladders, a PM needs more than just a ChatGPT subscription. They need technical literacy. This means understanding how to use tools like Cursor for coding, how to interact with Model Context Protocol (MCP) for data querying, and how to set up automated evaluations to ensure the quality of AI-generated work. The gap between 'non-technical' and 'technical' is being bridged by these very tools.

This shift does not mean PMs need to become software engineers. It means they need to become 'AI-native builders'. They need to know how to leverage an agent to run a test, how to use a coding assistant to verify a hypothesis, and how to build systems that scale their impact far beyond their own hours in a day.

Mathematics is currently the fastest-moving frontier for AI. While LLMs have made strides in language, the formal, logical rigor required for high-level mathematics presents a different kind of challenge. Yet, we are seeing AI discover new proofs and solve complex problems. This isn't just a win for mathematicians; it is a window into the nature of intelligence itself. If an AI can master the abstract structures of math, what does that imply for the rest of the cognitive world?

The Spiky Frontier

Progress in AI is not a smooth upward curve; it is 'spiky'. There are specific domains where models perform at superhuman levels, such as certain types of geometry or combinatorics, while remaining unable to handle basic reasoning in other areas. This spikiness suggests that intelligence is not a single, monolithic capability, but a collection of specialized skills that can be mastered independently. The 'aha' moment of AGI will likely not be a single event, but a series of these spikes appearing in sequence.

AI is not just solving problems; it is revealing the fractal nature of intelligence.

One of the most significant tensions lies in the 'verification loop'. A mathematical breakthrough is only useful if humans can understand and verify it. If an AI produces a proof for the Riemann hypothesis that is ten thousand pages long and relies on logic no human can follow, has it actually contributed to human knowledge? We face a future where AI might expand the boundaries of what is true, while simultaneously shrinking the boundaries of what is understandable.

The implication for students and professionals is profound. The value of rote learning and standard problem-solving is evaporating. In a world where AI can bridge the gaps between disparate mathematical fields, the human role shifts toward curation, the formulation of the right questions, and the conceptual synthesis of what the machines discover.

When a new model is released, the headline usually focuses on performance: 'Faster, smarter, better'. But for the developers building on these APIs, the real story is in the tokenization. Anthropic's release of Sonnet 5 includes a significant change to the underlying tokenizer. While the nominal price per million tokens remains identical to Sonnet 4.6, the way text is broken down into those tokens has changed radically.

The Tokenization Tax

Testing shows that the same input text now produces approximately 30% to 40% more tokens than before. For English text, this represents a roughly 1.4x increase in effective cost. For Python code, the increase is about 1.28x. This is a classic case of hidden inflation. The 'price' is the same, but the 'purchasing power' of your dollar has been slashed. For companies running massive-scale inference, this 30% delta is the difference between a profitable product and a money-losing one.

A price freeze is not a discount if the underlying unit of measurement has shrunk.

This change also highlights a strategic shift. By optimizing the tokenizer for certain languages or structures, providers can influence how different types of users consume their services. It also suggests that as models become more complex, the overhead of processing them grows, and the cost of that complexity is being passed directly to the consumer through the math of the tokenizer.

One of the greatest friction points in working with coding agents is the 'black box' problem. An agent tells you it has completed a task, but you have no way to verify the result without manually running the code and navigating the interface. This creates a cycle of constant manual checking. The solution is to force the agent to produce a visual record of its work—a video demo.

Automated Storyboarding

The new 'shot-scraper video' command enables this by using a YAML-based storyboard. This allows a developer—or more importantly, an AI agent—to define a sequence of actions: click this, wait for that, fill in this field. The tool then uses Playwright to execute these actions and record a high-quality MP4 video. This transforms the agent's output from a mere text block of code into a verifiable, visual artifact.

A coding agent that can record its own demo is an agent that can be trusted.

The power of this approach lies in the feedback loop. If an agent can read its own `--help` documentation, it can construct its own storyboards. It can write the code, run the server, execute the video recording, and then present the video alongside the code. This reduces the cognitive load on the human developer, who can now 'watch' the agent's work rather than just reading its logs.

We are moving toward a workflow where the primary interaction with software isn't typing commands, but reviewing demonstrations. The ability for agents to provide these visual proofs is a critical step in the transition from simple code completion to fully autonomous software engineering.