Simultaneous localization and mapping — helping robots find their way

Simultaneous localization and mapping is one of the key problems for mobile robots.

A mobile robot may need to map its initially completely unfamiliar environment with the help of its sensors and, at the same time, determine its own position on the map it is creating. The map and localization data that the robot generates using SLAM are an important basis for navigation but may be also used for other purposes.

One example is the planning of movements: the map serves as a basis to find a collision-free path from the starting point to the desired destination. To follow this path in a controlled manner, the motion controller of a robot needs to estimate its position precisely and reliably at all times.

Depending on the main type of sensor the SLAM system uses, we also call this “visual SLAM” or “LiDAR SLAM,” for example. When a combination of different sensors is used, we refer to it as “multimodal SLAM.” Furthermore, almost every robot uses additional sensors such as odometry or inertial sensors.

As a fundamental robotic technology, SLAM has been a focal point of research at Bosch since the inception of robotics research. The remarkable achievements of the SLAM pipeline developed at Bosch Research can be traced back to the collaborative efforts with Bosch Engineering GmbH (BEG) in 2016. At that time, BEG was seeking a dependable and precise localization solution for the intralogistics robot prototype “Autobod,” which was the predecessor of today’s ACTIVE Shuttle shown in the picture above. Through numerous collaborative tests and data recording sessions conducted across various facilities, we were able to gather the necessary data to identify the optimal technological options for LiDAR SLAM. As development progressed, our SLAM portfolio expanded to encompass a diverse range of robots and use cases, ranging from compact consumer robots to large-scale automated container transport vehicles.

Today, the team has developed a versatile, robust, and efficient SLAM pipeline for LiDAR (2D and 3D), radar, and visual SLAM. And while the team is constantly working on making the system more accurate and supporting more applications, there are also two research topics that are more orthogonal to that: lifelong SLAM and semantic SLAM.

SLAM algorithms were initially developed in lab environments that were limited in scope and not subject to changes. To use SLAM in real-world use cases, additional challenges need to be overcome. In particular, a robot may spend a long time inside an environment, revisiting the same locations over and over again.

Over time, the environment may change, for example, in an outdoor setting: parked cars may change their location, plants may grow, and different seasons may affect how an environment looks. One key aspect of lifelong SLAM is to refresh the map in order to reflect changes of the environment such that the map stays up to date.

A big challenge in lifelong SLAM is the amount of collected data. If a robot traverses an environment for hours, days, weeks, moths, or even years, huge amounts of measurements are collected. Storing all these measurements and incorporating them into the map is impractical due to limitations in the memory and computing power available. Thus, the lifelong SLAM system has to carefully remove old and redundant data, while making sure that no important information is lost. For this purpose, we employ a pruning algorithm that removes data in areas of the map where the SLAM graph is very dense.

How this can work is illustrated in the figure above: given a stream of sensor data, geometric SLAM accurately tracks the sensor pose and creates a geometric map, for example containing point clouds. Semantic SLAM can additionally be enabled, assuming there is a DNN that can perform object detection, panoptic segmentation, or both.

Detected objects are clustered and merged into hypotheses, which are in turn tracked and turned into landmarks that have their pose and extents optimized within the SLAM graph optimization. The result is a list of robustly and accurately mapped objects that can be used as navigation goals, among other things.

Panoptic segmentation provides per-pixel semantic and instance labels that can in turn be transferred to the point cloud of a camera or LiDAR sensor. We employ two levels of label voting, both on a local and a global scale, to end up with a consistent panoptic point cloud map. This can be used as an alternative way to extract relevant objects without an explicit detector or to selectively remove or highlight obstacles of certain types.

Currently, at least two major challenges remain: one is making these systems generalize to a wide range of domains with often limited access to training data. A specific factory floor can contain quite different objects compared to the data available during training, a problem where foundation models can help. Making these systems work efficiently on affordable hardware is a second major challenge — and we do not only trust in Moore’s law to solve this.

Recent Publications

[i] Panoptic hierarchical Semantic SLAM: Submission to the Embodied AI Robotic Vision Scene Understanding Challenge at CVPR 2022

[ii] Detecting Invalid Map Merges in Lifelong SLAM

[iii] Geometry-based Graph Pruning for Lifelong SLAM

[iv] When Geometry is not Enough: Using Reflector Markers in Lidar SLAM

[v] Better Lost in Transition Than Lost in Space: SLAM State Machine

[vi] Plug-and-Play SLAM: A Unified SLAM Architecture for Modularity and Ease of Use

[vii] SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstruction

[viii] Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction

[ix] Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships