AI for Robotics

This book approaches robotics from a deep learning perspective. Artificial intelligence (AI) has transformed many fields, including robotics. This book shows you how to reimagine decades-old robotics problems as AI problems and is a handbook for solving problems using modern techniques in an era of large foundation models. The book begins with an introduction to general-purpose robotics, how robots are modeled, and how physical intelligence relates to the movement of building artificial general intelligence, while giving you an overview of the current state of the field, its challenges, and where we are headed. The first half of this book delves into defining what the problems in robotics are, how to frame them as AI problems, and the details of how to solve them using modern AI techniques. First, we look at robot perception and sensing to understand how robots perceive their environment, and discuss convolutional networks and vision transformers to solve robotics problems such as segmentation, classification, and detection in two and three dimensions. The book then details how to apply large language and multimodal models for robotics, and how to adapt them to solve reasoning and robot control. Simulation, localization, and mapping and navigation are framed as deep learning problems and discussed with recent research. Lastly, the first part of this book discusses reinforcement learning and control and how robots learn via trial and error and self-play. The second part of this book is concerned with applications of robotics in specialized contexts. You will develop full stack knowledge by applying the techniques discussed in the first part to real-world use cases. Individual chapters discuss the details of building robots for self-driving, industrial manipulation, and…