We integrated ChatGPT with the Spot Robot
We integrated ChatGPT with the Spot Robot
A quick overview about why and how we built the integration and what will happen next.
This week's issue is brought to you by Cleanlab, which provides automated solutions to turn unreliable data into reliable models/analytics. Use AI to find and fix common issues in real-world images, text, or tabular datasets with Cleanlab Studio. This no-code app lets you quickly improve your dataset, assess its quality, and deploy more robust ML models auto-trained on the improved data with one click. Try it now!
A few weeks ago, this video went mega-viral on Twitter:
It’s a short, fun story showing the work we did to integrate ChatGPT with Spot.
The post got a ton of attention, to the point I lost count of how many major publications reached out for interviews or asked for permission to repost the video.
Most went the lazy route and painted the video as a step toward the apocalypse. Honestly, I don’t blame them: it’s an easy way to sell clicks, but it shows how little they care about researching, understanding, and informing their readers.
For everyone else, I wanted to summarize why and how we built the integration and some ideas of where I’d like to go next.
Why ChatGPT?
Two reasons.
First, we wanted to make the interaction with people more natural. Before integrating with ChatGPT, we built a simple interface to ask Spot to follow specific commands, like move left, walk back, or sit. But it was clunky.
ChatGPT solved that problem.
Here is an example prompt asking ChatGPT to “simplify” the request into a more structured command:
With a prompt like this, a user can say, “Get away from me,” and ChatGPT will transcribe it as “MOVE BACK.”
This makes the interaction feel magic!
But there’s a better reason we wanted to use ChatGPT.
Spot manages a ton of information. Every mission requires long configuration files, and at the end of the mission, Spot returns with tons of data. This information is hard to understand for anyone that’s not technical.
ChatGPT became the interface for the user to ask questions about the previous mission (the data we captured) and the next mission (the configuration information.)
The following prompt should give you an overall idea of how we made it work:
ChatGPT is pretty good at reading JSON files, so it didn’t take long to have something that makes the interaction with the robot quick and more natural.
The things we used
We used Whisper to transcribe the audio we captured from the onboard microphone. Google’s Text-to-Speech takes the robot responses and turns them into audio that we play through the speakers.
The rest is ChatGPT, the Spot SDK to communicate with the robot, and some glue code to assemble everything.
Finishing the demo wasn’t hard. Making this production-ready will take much longer.
What’s next?
Right now, we can ask Spot to do things.
That’s awesome but boring.
Here is a much better idea: Instead of giving Spot instructions to accomplish a task, we could give it a goal and let it devise the necessary tasks to get there.
That'd be sick!
Stay tuned.
I’ve made $600,000 in Upwork.
I'm a Top Rated Plus freelancer with a 100% Job Success rate. I sent 79 proposals and turned them into 19 completed jobs. That's a 24% acceptance rate, the highest I've seen on the platform.
I realize many people can’t even get a job in Upwork, so I’m recording a course to teach you how I do it:
How to organize your profile to attract attention.
How to structure proposals to get an interview.
How to succeed in the interview and get a contract.
No fluff.
I'll show you what works for me and why I do things that way.
If you are interested, you can preorder the course right here. The release date is June 15. If you are part of the ml.school community, you will get this course for free, so you don’t need to buy it.