Robo-CSK-Organizer: Commonsense Knowledge to Organize Detected Objects for Multipurpose Robots

The expanding role of multipurpose robots necessitates the development of transparent, intelligent systems capable of handling complex, context-driven operations. Traditional robotic arms, while proficient in assembly tasks, often struggle with nuanced decision-making in dynamic environments. Our work addresses this challenge by introducing Robo-CSK-Organizer (see Fig. 1), a system that demonstrates Explainable AI (XAI) via Commonsense Knowledge (CSK) to enhance the decision-making capabilities of robots. Imbibing CSK through the ConceptNet knowledge base, this system aims to address the opaque-box issue in AI (typically found in pure deep learning systems), characterized by a lack of decision-making clarity — a growing concern in the realm of robotics where precision and trust are paramount [32].

The Robo-CSK-Organizer system is proposed to address the limitations of current AI systems, which often fall short in tasks requiring deep contextual awareness. Traditional AI models may often struggle with differentiating between similar objects in varied contexts, such as distinguishing a child’s toy from a pet’s toy, or understanding the appropriate placement of objects in household settings. Robo-CSK-Organizer bridges this gap by adequately deploying commonsense knowledge (CSK) to enhance the organization of detected objects for task classification in multiple avenues.

Note that CSK differs from encyclopedic knowledge [20], [16] (where AI systems far surpass humans). “Common sense” is naturally found in humans who acquire it inherently at birth, enhance it with further growth, and use it for intuitive reasoning. Machines on the other hand are not endowed with CSK by default and hence can often find it challenging to conduct reasoning intuitively unless pre-programmed with rigorous training [9]. For instance, it is very easy and in fact rather obvious for a human to know that the “door” of a refrigerator should not remain open (except while placing things in it or taking them out) [4]. Conversely, the “door” of an office can certainly be open and is often ajar - a related fact also quite obvious to humans. Such knowledge is very subtle and is thus considered really simplistic or too common! Yet it can often be crucial in decision-making. Therefore, the role of CSK can be vital in modern-day AI systems as noticed significantly in the literature [33], [1].

Robo-CSK-Organizer precisely addresses this concept of CSK in AI. More specifically, it employs a robust semantic network from a classical knowledge base, namely ConceptNet [26], hence enabling robots to make more well-informed decisions based on contextual cues. This is in line with the logic of harnessing commonsense knowledge to augment machine intelligence [29]. It thrives on the adequate extraction and compilation of CSK from a knowledge base, which is a non-trivial task [23], and can be crucial in AI applications.

The system diagram of Robo-CSK-Detector, illustrative of its functioning approach, appears in Fig. 2 Its salient features are as described below.

1. 1.

   Resolving ambiguity in object categorization: Classification of objects, such as deciding whether a pear belongs in the kitchen or the garden, highlights the ambiguity inherent in object categorization today. This issue extends to the difficulty of providing comprehensive labeled training examples for every possible object arrangement and is effectively achieved by Robo-CSK-Organizer by harnessing CSK in a task-relevant manner.

2. 2.

   Maintaining consistency in object placement: Ensuring consistency in placing objects is crucial, particularly to build trust among users (e.g. helpful in human-robot collaboration) as well as among other robots in a multi-robot environment. Robots must reason with basic commonsense as well as domain knowledge and manage contextual variations to in order to ensure reliable object placement. This is well-achieved by Robo-CSK-Organizer with clear reasoning paths.

3. 3.

   Depicting task relevance and adaptability: Robots demonstrating adaptability in prioritizing tasks based on context, e.g. choosing between gardening and culinary activities, is vital. This is especially with respect to probabilistic uncertainties in sensing and navigation. Robo-CSK-Organizer handles this very well due to its systematic approach guided by a well-grounded knowledge base.

4. 4.

   Fostering explainability in AI systems: A critical aspect of XAI (Explainable AI) is ensuring that AI systems are not just intelligent but also comprehensible and interpretable. Robo-CSK-Organizer excels in explainability, surpassing other systems (e.g. ChatGPT-based organizers) by adequately harnessing CSK due to which decision-making processes are more transparent and understandable.

In Fig. 3, Robo-CSK-Organizer’s utilization of ConceptNet for object categorization in a kitchen setting is visually depicted. This figure illustrates the decision-making pathway for categorizing a pear, among other items. Specifically, it shows 3 potential paths from “kitchen” to “pear”. The system selects the path with the highest “AtLocation” edge weight (in this case, 7.21), indicating the common location of food in a kitchen. This path is further delineated by linking “apple” to “food” (via a “RelatedTo” edge) and subsequently connecting “apple” to “pear”. This logical sequence leads Robo-CSK-Detector to place the pear in the kitchen, exemplifying the system’s reasoning process.

Designed with modules for object detection, context recognition, and semantic analysis, Robo-CSK-Organizer not only classifies objects but also interprets their appropriate placement within various contexts. This approach significantly enhances the transparency and interpretability of AI decisions, addressing the critical challenge of explainability in AI systems. It can thus be considered analogous to systems that delve into aspects such as spatial commonsense [10], [11], [22] for object recognition and training of autonomous systems. For instance, if there is an error, its traceability is highly facilitated in Robo-CSK-organizer (versus deep learning based systems, e.g, those using ChatGPT for training). Hence, Robo-CSK-Organizer can help robots learn from their mistakes and correct themselves, thus getting better in their performance. Moreover, the XAI contribution of Robo-CSK-Organizer is helpful when humans and robots work together, i.e. for human-robot collaboration, as humans are able to understand the actions of the robots much better, along with the reasons behind the robots’ decisions. The same logic applies to numerous robots working together. This enhances trust in the realm of robotics. All these facets are vital, especially with the growing prevalence of multipurpose robots, heading towards next-generation advancements.

The main functioning of Robo-CSK-Organizer, focusing on its reasoning, is outlined in Algorithm 1 here.

This algorithm that highlights the main functioning of Robo-CSK-Organizer operates on 2 primary inputs: a video feed $\mathcal{V}$ and the ConceptNet knowledge base $\mathcal{C}$. Its goal is to sort detected objects into their appropriate contexts. Initially, the robot’s vision system $\mathcal{R}$, implemented using Detectron2, is initialized. Thereafter, the context bins are scanned and recognized using BLIP ($\mathcal{B}$), with the contexts stored in a CSV (comma-separated variable) format ($\mathcal{S}_{\text{csv}}$). For each frame $f$ in the video feed $\mathcal{V}$, objects are detected and labeled as $\mathcal{D}$. Each detected object $o$ is then checked against $\mathcal{C}$ to determine its context $\mathcal{K}_{o}$. If this context matches that of a bin in $\mathcal{B}$, the object is placed in the corresponding bin. The process continues for each object in the frame, and the annotated frame can be displayed as needed.

This algorithm is implemented into the Robo-CSK-Organizer system using Python and is integrated with a robotic arm in our laboratory, namely, the CRoSS Lab (Collaborative Robotics and Smart Systems Lab at our university). Robo-CSK-Organizer is then executed using various real-world objects. Details of its execution are mentioned next in the respective parts of its system demonstration.

In order to demonstrate the efficacy of Robo-CSK-Organizer, it is compared with a baseline task organizer that uses the well-known ChatGPT for guidance. We thus present the following.

Object Detection: Both the systems, our Robo-CSK-Organizer and the ChatGPT baseline, use DETIC (”DEtector with Image) ensure a broad evaluation spectrum. Specific context groups, particularly domestic locations (e.g. kitchen, garden, pantry, dining room) are chosen for evaluation. These contexts are relevant to the selected object categories and provided with a controlled environment for testing. An advanced extension of Detectron2 for object detection, which contains over 21,000 classes [31] [35] is used here to provide choices of classes for object organization. Each object is queried against each system (Robo-CSK-Organizer / ChatGPT) 10 times, asking it to organize the object into one of the provided contexts. Responses from both systems are recorded for each iteration, and the most frequent context is identified as the predominant choice for object placement,

Context Recognition: BLIP (Bootstrapping Language-Image Pre-training) is employed to identify contexts such as the kitchen or office, and thus generate room captions for enhanced clarity [19]. Note that the usage of such software can be helpful in a variety of applications, e.g. image personalization via text by harnessing diffusion models [14]. The hardware foundation for both systems (i.e. Robo-CSK-Organizer and the ChatGPT baseline) includes a Franka Emika robotic arm [12] with an Intel Realsense D435i camera [15], integrated with ROS. Robo-CSK-Organizer works with this hardware, and incorporates CSK-based reasoning (See Algorithm 1). This is the key to addressing the opaque-box issue in AI, aiming for clearer and more transparent object sorting.

Note that the pivotal distinction between the 2 systems lies in the functioning approach for sorting objects into relevant contexts. While ChatGPT relies solely on prior training with deep learning, Robo-CSK-Organizer applies commonsense knowledge due to which it can be more adept in successfully handling first-time scenarios as well. More details appear next.

Robo-CSK-organizer: It harnesses a classical knowledge base called ConceptNet [26] for semantic insights and commonsense reasoning. It infers object locations using metrics known as edge weight and degree of separation, prioritizing paths based on these factors. While ConceptNet is chosen for its user-friendly interface and clear path logic, we claim that other relevant CSK knowledge bases can also be used.

ChatGPT baseline: A ChatGPT-trained organizer is used as a baseline; it relies on generative pre-trained transformer models for its decision-making, processing text-based inputs to infer object locations and categorizations. This approach, though adept in language processing, does not integrate a structured commonsense knowledge base. Consequently, the ChatGPT-based organizer’s decisions are more influenced by pre-trained patterns in textual data rather than explicit semantic relationships and intuitive logical reasoning. This can affect consistency and transparency in decision-making in complex or ambiguous scenarios, notably (but not limited to) those encountered for the first time.

In our comprehensive evaluation, we conduct experiments to assess the performance of Robo-CSK-Organizer and the Chat-GPT baseline across various contexts. The key aspects of these experiments are focused on ambiguity resolution, consistency, task-relevance adaptability, and explainability. A summary of our exhaustive experimentation is presented below.

This research presents significant contributions to the field of AI and robotics, particularly in the development and application of commonsense knowledge for task classification in multipurpose robots. Robo-CSK-Organizer, as a pioneering system, stands out in several key areas when compared to existing systems (e.g. a baseline organizer using ChatGPT).

The integration of AI and robotics, particularly in domestic environments, has seen a variety of innovative approaches. These methodologies have significantly contributed to the field by enhancing robots’ decision-making processes, adaptability, and interaction with their environment.

One approach focuses on using ConceptNet and Google search data for object categorization in domestic robotics, particularly for tidy-up services. This method effectively groups objects into functional categories, thereby aiding robots in more intuitive object handling [25]. Another study explores the use of large language models (LLMs) like GPT-3.5 as a repository of CSK for task planning. This demonstrates the potential of language models in enriching the robotic decision-making process [34]. Furthermore, paradigms based on the classical neural networks have been adapted to many contexts, ranging from machine translation [7] in text with recurrent neural networks (RNNs) to object recognition in multifaceted scenarios with computer vision models, e.g. VGG-16 [24] and ResNet-101 [13]. The issue of extracting cultural commonsense knowledge and its usefulness in enhancing chatbots has been addressed through a novel approach called CANDLE [21] with interesting real-world impacts.

Advances in visual commonsense reasoning introduce the R2C engine [33] to enhance object recognition, anchoring natural language descriptions in visual data. CSK-Detector [3] is an innovative system for object detection in domestic robotics, leveraging CSK from the Dice knowledge base [2]; it reduces the need for extensive image annotation.

The incorporation of CSK from the OMICS database using Description Logic has also been discussed. This integration enables robots to perform more nuanced tasks, showcasing the potential for more context-aware robotics [18]. Furthermore, the application of CSK in human-robot collaborative tasks has been highlighted, especially in robot action planning for assembly tasks, emphasizing the enhancement of cooperative interactions [5]. Its mathematical modeling insights along with core applications in smart manufacturing have been elaborated [6] as well, emphasizing the crucial role of commonsense reasoning.

Additionally, semantic task planning for service robots in dynamic, open-world environments has been explored. This method leverages natural language understanding and semantic reasoning, addressing the challenges posed by ever-changing environments [8]. The combination of non-monotonic logical reasoning and incomplete CSK with inductive learning to guide deep learning in robotics is another innovative approach. This integration offers a unique perspective on the convergence of CSK and advanced learning techniques [27]. Much of this work builds upon semantic advances over the years [28], [30] that help in managing knowledge and conducting predictive analysis.

These diverse methodologies underscore the importance of CSK in improving the functionality and intelligence of robotic systems, especially in domestic settings. They have advanced the field by demonstrating effectiveness in task planning, human-robot interaction, and environmental adaptation.

Building on these foundations, the Robo-CSK-Organizer system represents a significant advancement in the practical application of CSK in robotics. Unlike the existing systems, Robo-CSK-Organizer harnesses CSK in real-world settings for object organization in task-based classification, which is particularly beneficial for multipurpose robots. Its ability to resolve ambiguities and maintain consistency in object placement, adaptability to diverse task classifications, and contributions to explainable AI (XAI) set it apart from the current methodologies. This system not only categorizes and understands objects in various contexts but also intelligently organizes them, demonstrating a novel and practical application of CSK in enhancing the efficiency and functionality of multipurpose robotic systems.

Robo-CSK-Organizer is a system proposed in this paper that effectively demonstrates XAI via CSK for object organization, mitigating the challenges of non-transparent AI. Our early experiments open up areas for enhancement, e.g. decision paths from a CSK source such as ConceptNet. For instance, misplacement of high heels in the kitchen can be due to semantic overlap with stiletto as heel / knife. When this is clearly explained, potential improvements can be made. In our work, these can include refining relationships (e.g. RelatedTo) so as to provide better contextual accuracy, and using average weights in the knowledge base, not just the weight of the first edge in order to provide more robustness. Additionally, we are committed to refining the algorithmic logic used to identify the most optimal paths based on CSK.

Impacts of Robo-CSK-Organizer are highlighted here.

1. 1.

   Puts forth our objective to quantify enhancements that CSK brings to the reliability and transparency of AI

2. 2.

   Elevates the efficacy of robotic decision-making to bring it closer to human cognition

3. 3.

   Fosters a broader academic dialogue on commonsense in robots for better interpretation, trust, and explainability

4. 4.

   Can be useful in next-generation multipurpose robots, and in human-robot collaboration due to higher clarity.

5. 5.

   Can lead to energy savings due to more efficient learning, thus positively impacting sustainable AI.

6. 6.

   Well-mounted on an AI-robotics bridge, particularly that of explainable AI and multipurpose robotics.

This paper thus offers our modest contributions to both AI and robotics. We anticipate fruitful, long-lasting impacts.

The commonsense knowledge project thrived on a research visit by Dr. Aparna Varde at the Max Planck Institute for Informatics, Saarbrucken, Germany, with further work at Montclair. This work is supported in part by the National Science Foundation under Grants CMMI-2138351 and CNS-2117308. Our experiments are conducted in the CRoSS (Collaborative Robotics and Smart Systems) Lab at Montclair, of which Dr. Weitian Wang is the Director. We thank CESAC (Clean Energy and Sustainability Analytics Center) at Montclair, of which Dr. Aparna Varde is an Associate Director.