Doing research in robotics

Russ Tedrake

Image credit: Boston Dynamics

Q: Autonomous or Driver-controlled?

DARPA Robotics Competition


Robots are dancing and starting to do parkour, but...


what about loading the dishwasher?

The Machine Learning Revolution

(in robotics; in a few slides)

What's my rule?

What's my rule?


ImageNet: 14 Million labeled images

Released in 2009

Transfer learning

Something we couldn't have expected...


(Pre-)Training on ImageNet makes it easier to "learn" to recognize other objects

A sample annotated image from the COCO dataset

"Self-supervised learning"

Example: Text completion

No extra "labeling" of the data required!

Now robots are starting to manipulate objects they've never seen before...

and the programmers never considered / tested!

More complex tasks...

Going out into the real world...

"some interesting failure cases"

We still need "common sense"

Put the plate in the dishwasher...

  • and don't break the plate,
  • throw any mugs out the kitchen window,
  • ....

"Foundation models" in machine learning

Example: Image + Text


Humans have put lots of captioned images on the web


"Foundation models" in machine learning

Example: Dall-E 2

"A painting of a professor giving a talk at a robotics competition kickoff"



"Foundation models" in machine learning

Example: Dall-E 2

"An FTC competition robot with green plants growing from its metal body"



"Foundation models" in machine learning

Some of the big questions: 

  • Have humans put enough random information on the web for robots to learn what they need to know about operating the world?
  • As we get more and more robots, how should they pool their experience?
  • How do we balance robot learning / exploration with safety?

We need your ideas, better theories/algorithms/robots, even more scalable machine learning systems, ...

Do you love robotics?

What can you do right now?

  • Programming => Software engineering
  • Physics
  • Math (it's extremely important!)
    • Calculus
    • Linear Algebra (the foundations of machine learning)
  • Machine learning tutorials online are becoming very accessible
  • FIRST robotics!

Work hard and good luck this season!

FTC Kickoff 2022

By russtedrake

FTC Kickoff 2022

Fall 2022 FTC Northeastern Region Kickoff (at NHS)

  • 532