Where Does the Task Actually End?

A Moment of Levity

---

I watched a humanoid robot run the 100-meter dash recently.

It was remarkable.

The robot accelerated down the track and crossed the finish line faster than the fastest human ever has.

Then came the interesting part.

It kept going.

Some of the robots crashed into the protective wall. One broke apart. Another produced sparks. In one race, a robot even caught fire after reaching the end of the track.

My first reaction was probably the same as everyone else’s.

That was impressive.

My second thought was different.

What exactly was the task?

If the task was:

Run 100 meters as fast as possible.

then the robot succeeded spectacularly.

It crossed the finish line.

The clock stopped.

Mission accomplished.

Stopping afterward was apparently another task for another day.

There is something funny about that.

But there is also an important lesson hiding inside it.

We often talk about whether a system can perform a task without spending enough time defining where the task actually begins and ends.

Consider the difference between these two instructions:

Run 100 meters as fast as possible.

and:

Run 100 meters as fast as possible, recognize that the finish is approaching, transition out of maximum acceleration, decelerate while maintaining balance, stop safely, and remain functional afterward.

Those sound like versions of the same task.

They aren’t.

The second contains a much larger set of requirements.

The robot must not only generate speed.

It must know where it is in the task.

It must anticipate what comes next.

It must transition between behaviors.

And the transition itself must remain under control.

This matters because a benchmark can only measure what we ask it to measure.

If we reward only the time required to cross a line, a system optimized to cross that line quickly may be doing exactly what we asked of it.

The mistake would be ours if we looked at that result and concluded:

This robot has mastered running.

Perhaps it has mastered one important part of running.

But accelerating is not stopping.

Crossing a finish line is not recovering afterward.

A performance metric can reveal extraordinary capability in one dimension while telling us very little about the capabilities immediately beside it.

We see versions of this problem everywhere.

A robot may pick up an object successfully but not know what to do when the object begins to slip.

A self-driving system may navigate an ordinary road beautifully but struggle when construction unexpectedly changes the lane.

A medical test may accurately measure a number while telling us very little about the decision we actually need to make.

The measurement is not necessarily wrong.

We may simply have asked it a smaller question than we realized.

That is why one of the simplest questions can also be one of the most important:

What is the task?

And once we think we have answered it, perhaps we should ask one more:

Where does the task actually end?

Because sometimes the finish line is not really the finish.

And sometimes the most revealing part of a demonstration is what happens after the metric says the task is over.

I’ve been wondering…

How often do we mistake success on a benchmark for mastery of the larger task the benchmark was meant to represent?