Why Greatness Cannot Be Planned — Engineering Lessons
Kenneth Stanley and Joel Lehman's book "Why Greatness Cannot Be Planned" fundamentally changed how I think about engineering and innovation. Their core thesis - that objectives can actually prevent us from achieving ambitious goals - seems counterintuitive, but after seeing it play out repeatedly in my work at NVIDIA and in robotics research, I'm convinced they're onto something profound.
The Paradox of Objectives
The book's central insight is the "objective paradox": the more ambitious your goal, the less likely you are to achieve it by pursuing it directly. This happens because truly innovative solutions require discovering unexpected stepping stones that don't obviously lead to the goal.
Stepping Stones, Not Milestones
In traditional project management, we set milestones - predetermined checkpoints on the way to our goal. But innovation doesn't work this way. Instead, we need to think in terms of stepping stones - interesting discoveries that open new possibilities, even if we can't see where they lead.
A Personal Example: The Maxine SDK
When we started integrating Triton Inference Server with the Maxine SDK, our objective was simple: improve inference throughput. But by following interesting stepping stones rather than a rigid plan, we discovered something more valuable:
- First stepping stone: Dynamic batching revealed patterns in video frame processing we hadn't noticed
- Second stepping stone: These patterns suggested a new caching strategy
- Third stepping stone: The caching strategy enabled real-time style transfer we hadn't thought possible
- Unexpected destination: A completely new feature set that became more valuable than the original optimization
If we had rigidly pursued just "10x throughput improvement," we would have missed these discoveries.
The Novelty Search Principle
Stanley and Lehman propose "novelty search" as an alternative to objective-based search. Instead of optimizing for a specific goal, you optimize for discovering new and different solutions. This approach has profound implications for engineering.
Applied to Machine Learning
In my work with vision-language models, I've found that the most interesting breakthroughs come from exploring "what's different" rather than "what's better":
# Traditional approach: optimize for accuracy
for epoch in range(num_epochs):
loss = compute_loss(model, data, target_accuracy)
optimize(loss)
# Novelty-driven approach: explore diverse solutions
for epoch in range(num_epochs):
novelty_score = compute_novelty(model, previous_models)
quality_score = compute_quality(model, data)
combined_score = balance(novelty_score, quality_score)
optimize(combined_score)The second approach often discovers unexpected model architectures that wouldn't emerge from pure accuracy optimization.
Open-Ended Evolution in Engineering
The book's concept of "open-ended evolution" suggests that the most innovative systems are those that can continuously generate new and interesting possibilities without a fixed endpoint.
Building Open-Ended Systems
Consider these principles when designing systems:
- Composability over Monoliths: Small, combinable pieces create more stepping stones
- Experimentation over Optimization: Reserve 20% of resources for exploring "interesting" rather than "optimal"
- Diverse Metrics over Single KPIs: Track multiple dimensions of "interesting" not just one measure of "good"
- Serendipity over Predictability: Design systems that can surprise you
The Exploration vs. Exploitation Dilemma
Every engineer faces this dilemma: do you exploit known good solutions or explore new possibilities? The book suggests a radical answer: in ambitious projects, err heavily on the side of exploration.
Practical Implementation
Here's how I've applied this in practice:
- Sprint Planning: Allocate one "stepping stone" task per sprint - something interesting with unclear value
- Code Reviews: Ask "what did we learn?" not just "does it work?"
- Architecture Decisions: Choose designs that preserve optionality over those that optimize current requirements
- Performance Tuning: Try weird optimizations that "shouldn't" work - sometimes they reveal new principles
When Objectives Are Still Useful
The book doesn't argue against all objectives. Short-term, well-understood problems benefit from goal-directed approaches. The key is recognizing when you're in explored vs. unexplored territory:
| Use Objectives When: | Use Stepping Stones When: |
|---|---|
| • Problem is well-understood | • Problem space is novel |
| • Solution path is clear | • Many approaches might work |
| • Optimization is the goal | • Innovation is the goal |
| • Resources are limited | • Learning is valued |
The Cultural Challenge
The biggest obstacle to applying these ideas isn't technical - it's cultural. Most organizations are built around objectives, KPIs, and measurable outcomes. Advocating for exploration without clear goals can seem irresponsible.
Making the Case
Here's how to introduce these ideas:
- Start Small: Run a "stepping stone Friday" where people work on interesting-but-unclear projects
- Document Serendipity: Keep a log of valuable discoveries that weren't planned
- Reframe Failures: Present unsuccessful explorations as "collecting stepping stones"
- Celebrate the Unexpected: Publicly recognize discoveries that came from non-goal-directed exploration
Personal Reflections
Working on robotics at CMU and now on AI systems at NVIDIA, I've seen how the most impactful work often comes from following curiosity rather than requirements. My work on SLAM algorithms improved not when I focused on "better localization accuracy" but when I explored "what interesting things can we do with uncertainty estimates?"
The DARPA SubT challenge taught me this lesson viscerally. Teams that rigidly pursued "map the underground environment" often failed. Teams that collected interesting capabilities - robust communication, adaptive exploration, failure recovery - succeeded even when their mapping wasn't optimal.
Conclusion: Engineering as Exploration
Stanley and Lehman's insights challenge us to think of engineering not as building toward predetermined goals, but as exploring a space of possibilities. The greatest innovations - from the internet to neural networks - emerged not from careful planning but from following interesting stepping stones.
As engineers, we're trained to solve problems efficiently. But perhaps our greatest contribution comes not from solving the problems we're given, but from discovering problems we didn't know existed. Sometimes, the best way to achieve greatness is to stop trying to achieve it, and instead focus on doing something interesting.
The next time you're stuck on a hard problem, try stepping back from the objective. Look for what's interesting, what's different, what's surprising. You might not reach your original goal, but you might discover something far more valuable.