Learning visual models using a knowledge graph as a trainer

Therefore, the main challenge for CV in autonomous driving is the enormous variety of contexts in which an autonomous car must function safely. For example, it must be able to reliably detect traffic signs under many different lighting and weather conditions.

In collaboration with the University of Trier, we have developed a knowledge-infused approach for learning visual models. Knowledge-infused machine learning combines the best of both worlds: explicit and contextualized domain knowledge (symbolic AI) and data-driven machine learning methods (sub-symbolic AI).

Specifically, our solution uses a knowledge graph neural network (KG-NN), in which the training of an NN is guided by prior knowledge that is invariant across domains and represented in the form of a knowledge graph (KG). The prior knowledge is first encoded in the KG with respective concepts and their relationships. Figure 2 illustrates an excerpt of the road signs KG (RSKG). The RSKG contains high-level semantic relationships between the road signs of the dataset. For instance, the stop sign (RS_STOP) has the shape Octagon, the type Prohibitory, and the countries: Germany (DE) and China (CN).

In the second step, the KG is transformed into a vector space via a knowledge graph embedding (KGE) method. Since deep learning methods operate in high dimensional vector space, we use this space to interact with the NN and to infuse prior knowledge.

A conventional NN is then trained with the supervision of the KGE. Using a contrastive loss function, the KG-NN learns to adapt its visual embedding space and thus its weights according to the knowledge graph embedding.

We evaluate our KG-NN approach on a visual transfer learning tasks using road sign recognition datasets from Germany and China. The results show that an NN trained with a knowledge graph as a trainer outperforms a conventional NN particulary in transfer learning scenarios, in which the real-world data differs from the training data.