The Deep Feed

01 — Lenny's Newsletter

The Benchmarking Trap

Why automated evaluations fail to capture the essence of quality

By Lenny Rachitsky · 8 min read

Editor's note: As models iterate faster than we can test them, the gap between 'correctness' and 'taste' is becoming a strategic liability.

The current race in artificial intelligence is often measured by benchmarks that look impressive on a spreadsheet but fail in the real world. When Anthropic released Sonnet 5, the immediate reaction was to look at the numbers. But numbers are a poor proxy for utility. A model might pass a coding test with flying colours while failing to understand the subtle constraints of a user interface or the specific 'voice' required for a professional interaction. The problem is that most automated evaluations are too polite; they cluster around middle-of-the-road scores and miss the sharp edges that define truly great work.

The Failure of the LLM-as-Judge

There is a growing trend to use large language models to grade other models. It sounds efficient, but it creates a feedback loop of mediocrity. In recent testing, models like GPT-5.5 and Opus 4.8 acted as judges, yet they were unable to catch visual errors or broken prototypes that a human eye spotted instantly. They lacked the ability to penalise a model for ignoring a wireframe constraint or for producing a response that was technically correct but practically useless. The automated judges were too generous, failing to provide the 'spiky' feedback necessary to drive real improvement.

The human signal remains the most useful part of any benchmark. Models cannot yet see what the human eye catches in the first screenshot.

This divergence suggests that the real value in AI development lies in encoding human taste into the evaluation process. If a model's output doesn't 'feel' right, the benchmark needs to reflect that. Claire Vo's approach—weighting human preference heavily against automated scores—revealed that Sonnet 4.6 actually outperformed models that scored higher on traditional metrics. It won because of its personality and its ability to handle the social nuances of agentic work, such as responding appropriately when a deployment fails.

How to build a better benchmark

Use frozen inputs to ensure repeatability across different model versions.
Create a custom scoring page to capture gut-feel ratings.
Weight human preference higher than LLM-as-judge scores.
Include 'personality' and 'voice' as measurable metrics.

For builders, the takeaway is clear: do not outsource your judgement to the machines you are trying to evaluate. The most successful workflows will be those that use AI to handle the heavy lifting while keeping a human in the loop to maintain the standard of excellence. The goal is not just to find a model that works, but to find one that works the way you think.

Key Takeaway

Technical correctness is a baseline; true quality is defined by human taste, which machines cannot yet simulate.

02 — Lenny's Newsletter

From Prompter to Manager

The shift in how we direct autonomous agents

By Claire Vo · 7 min read

Editor's note: The era of typing prompts is ending; the era of managing fleets of agents is beginning.

Most people approach AI as a prompter. They sit in front of a chat interface, typing instructions and waiting for a response. This is a low-leverage way to work. It is reactive and tethered to a single session. The real shift, as demonstrated by Alessio Fanelli, is moving from 'agent prompter' to 'agent manager'. An agent manager does not babysit a single chat; they oversee a system of autonomous actors that move through a defined lifecycle without constant intervention.

The State Machine Approach

To manage agents effectively, you need a structure that provides context and direction. Using tools like Linear as a 'state machine' allows agents to track their own progress. Instead of a human telling an agent what to do next, the agent looks at the project management board, sees what task is pending, and executes it. This turns the workflow from a series of disjointed prompts into a continuous, asynchronous process. It allows for work to happen in the background, even while you are away from your desk.

The friction of local runtimes and clunky interfaces kills the momentum of agentic workflows. You need a cloud-based system to scale.

Scaling this requires moving away from local machines like a Mac Mini and toward cloud VPS environments. A cloud setup provides the stability and accessibility needed to run agents 24/7. It also allows for better integration with browser-based tools. For instance, agents can be tasked with scouring marketplaces like eBay for specific items—such as underpriced Pokémon cards—by autonomously browsing, extracting data, and flagging deals. This is not just coding; it is the creation of a new category of small, automated businesses.

The Agent Manager's Toolkit

OpenAI Symphony for managing the agent lifecycle.
Linear for acting as a state machine to track tasks.
Cloud VPS to ensure 24/7 autonomous operation.
Browser access (via Codex) for real-world web tasks.

The transition to management requires a change in how we track costs and performance. You are no longer measuring the quality of a single prompt, but the efficiency of a system. You track token costs per task and the success rate of autonomous runs. This is a higher level of abstraction that enables much greater scale, turning a single person into a director of a digital workforce.

Key Takeaway

Stop prompting and start managing; build systems where agents use project management tools to drive their own progress.

03 — Lenny's Newsletter

The Great Bifurcation

How AI is splitting the tech workforce in two

By Noam Segal, Lenny Rachitsky · 9 min read

Editor's note: A new survey reveals that AI is not just changing tasks; it is fundamentally altering professional identity.

The tech industry is undergoing a quiet, structural split. A recent large-scale survey of tech workers reveals that the impact of AI is not uniform. Instead, the workforce is dividing into two distinct psychological camps. One group feels 'amplified'—they feel more capable, more confident, and more excited about their careers than ever before. The other group feels 'shaken'—they are uncertain of their value and increasingly anxious about their place in a world where their skills might be automated.

The Anxiety of Productivity

While 82% of respondents report that AI is making them measurably more productive, this gain comes with a heavy psychological cost. Burnout has jumped significantly, rising from 44.7% to 55.7% in a single year. The fear is not necessarily that AI will take jobs—only 22% of respondents worry about direct job loss—but that it will raise the bar for what is expected. Workers fear being expected to produce more volume for the same pay, or being trapped in an unsustainable pace of work driven by machine efficiency.

The defining feeling about AI is not fear or excitement, but ambivalence.

This ambivalence is most visible among designers and researchers, who report the highest levels of anxiety and the lowest willingness to recommend their fields to newcomers. For these professionals, the core of their work—the creative and analytical synthesis—feels most threatened by the ease with which models can now mimic their outputs. The industry is being described by its own workers as 'chaotic', a state of flux where the ground is moving too fast to find a stable footing.

The Four AI Identities

Amplified: I can do more and better.
Redefined: My role is changing, but the direction is unclear.
Destabilized: I am unsure where I stand.
Diminished: I feel less essential.

As we move forward, the gap between the amplified and the diminished will likely widen. Those who learn to integrate AI as a tool for expansion will thrive, while those who view it as a competitor for their core identity will face increasing burnout and obsolescence. The challenge for leaders is to ensure that productivity gains do not come at the expense of the human worker's sense of purpose and stability.

Key Takeaway

AI is creating a divide between those who use it to expand their capabilities and those who feel it diminishes their value.

04 — The Marginalian

The Error of Perception

Thich Nhat Hanh on the root of interpersonal conflict

By Maria Popova · 6 min read

Editor's note: In an era of rapid communication, we often forget that our interpretations of others are frequently wrong.

Human beings are storytelling creatures, but we often tell ourselves stories that are factually incorrect. When we experience hurt in a relationship, our immediate impulse is to construct a narrative that explains the pain. We assign motives to others, assuming their actions were intended to wound us. However, as the late Thich Nhat Hanh observed, these explanations are almost always wrong. They are usually projections of our own internal fears and vulnerabilities rather than accurate reflections of the other person's reality.

Correcting the Internal Narrative

The root of much suffering is not the event itself, but our perception of it. To repair a relationship, we must first address our own internal state. Hanh suggests a three-step process for correcting these wrong perceptions. The first step is purely internal: acknowledging that the picture in our head might be inaccurate and using breath and movement to return to a state of calm. We cannot engage in meaningful dialogue while we are in a state of reactive emotional turbulence.

When you make the effort to listen and hear the other side of the story, your understanding increases and your hurt diminishes.

Once calm, the second step involves approaching the other person not with an accusation, but with a request for help. Instead of saying, 'You hurt me,' one might say, 'I am suffering, and I am worried that my perception of what happened might be wrong. Can you help me understand?' This shifts the dynamic from a confrontation to a shared investigation. It invites the other person to explain their motives without the immediate need for defensiveness.

The Three Steps to Repair

Acknowledge your internal perception may be wrong and find calm.
Approach the other person for help in understanding, rather than accusing.
Listen deeply to their response to correct your own misunderstanding.

The final step—listening with true intent—is the most difficult. It requires the humility to accept that we might be the victims of our own misapprehensions. By doing this, we move beyond the self-referential loop of our own emotions and begin to see the invisible forces driving the other person. This is not just a way to resolve conflict; it is a fundamental way of caring for ourselves and for the integrity of our connections.

Key Takeaway

Most interpersonal conflict stems from our own incorrect interpretations; the remedy is radical listening and the humility to question our own motives.

05 — The Marginalian

The Inner Illumination

Hermann Hesse on the paradox of reading

By Maria Popova · 5 min read

Editor's note: Books do not provide the answers; they provide the path to the answers already within you.

We often turn to books as if they were external reservoirs of wisdom, containers of truths that we lack. We read to find solace, to find examples of how to live, and to find proof that we are not alone in our suffering. There is a profound comfort in seeing our own heartbreak mirrored in the pages of a classic. Yet, there is a paradox at the heart of reading: while books show us the lives of others, their ultimate purpose is to return us to ourselves.

The Mirror of Literature

Hermann Hesse captured this beautifully in his poetry. He argued that no amount of reading can bring happiness in isolation; rather, books build a 'secret path' toward the heart. The wisdom we find on the page is actually a reflection of the wisdom already present within us. As we encounter ideas that resonate, we are not learning something entirely new; we are experiencing the moment when an external thought flashes against our own internal truth.

The end of a book's wisdom appears to us as merely the start of our own.

This idea, echoed by Proust, suggests that the 'essential book' already exists within the reader. The text acts as a catalyst, a way to anneal our values and clarify our own experiences. We read to find the language for things we have felt but could not name. In this sense, literature is not an escape from reality, but a tool for deepening our engagement with it. It provides the framework through which we can understand our own lives.

Why We Read

To learn how to live, love, and suffer.
To clarify our own values and identity.
To find assurance that our experiences are not unprecedented.
To build a path toward our own inner truth.

Ultimately, the quest for wisdom through libraries is a journey toward self-knowledge. The books themselves are not the destination; they are the maps. The actual living, the actual suffering, and the actual joy must be experienced directly. The wisdom found in books is only truly realised when it is applied to the blank page of our own lives.

Key Takeaway

Books are not substitutes for experience, but catalysts that help us recognize the wisdom already residing within us.

06 — The Marginalian

The Battle for Individuality

E.E. Cummings on the courage to feel

By Maria Popova · 7 min read

Editor's note: In a world of constant feedback and conformity, being yourself is a radical act of defiance.

Every generation believes it is fighting a unique battle against conformity. We live in an informational ecosystem designed to reward the common opinion and punish the dissenting voice. It is a Pavlovian system of constant feedback that encourages us to blend in. To resist this is not merely a matter of preference; it is a fight for the integrity of the self. E.E. Cummings understood this better than almost any other artist of his time.

Thinking vs. Feeling

Cummings made a sharp distinction between thinking, believing, and feeling. He argued that anyone can be taught to think or to believe, because thinking and believing are often just the adoption of other people's ideas. When you think or believe, you are, in a sense, 'everybody else'. But the moment you truly feel, you become 'nobody-but-yourself'. Feeling is the only territory that cannot be colonised by the crowd.

To be nobody-but-yourself in a world which is doing its best to make you everybody else means to fight the hardest battle which any human being can fight.

This battle is not easy. For Cummings, it meant facing intense criticism from traditionalists who hated his unconventional style. He understood that using words like everyone else is easy, but expressing a unique truth requires an immense amount of work. To be an artist—or simply to be a person of integrity—requires a willingness to endure the discomfort of being misunderstood and the exhaustion of constant self-assertion.

The Cost of Authenticity

The rejection of easy, borrowed opinions.
The endurance of social and professional criticism.
The relentless work required to express unique truths.
The courage to remain in solitude with one's own knowledge.

The advice Cummings offers is not a platitude about 'being yourself'. It is a warning. He suggests that if you are not willing to fight, work, and feel until you die, then perhaps you should choose something easier. But for those who are willing, he promises that this is the most wonderful life on earth. The reward for the battle is not fame or approval, but the rare and precious experience of true existence.

Key Takeaway

Authenticity is not a state of being, but a continuous, difficult struggle against the ease of conformity.