Neuro-Symbolic Predicates for robot planning and dealing with complex tasks

In one of our first "A path towards AGI" posts we discussed Neuro-symbolic systems: https://x.com/TheTuringPost/status/1819530581232722420

Here's a new example of their implementation??

Neuro-Symbolic Predicates (NSPs) are smart rules that help robots think by combining visual perception (neural) with logical rules (symbolic). With NSPs robots can easier plan and tackle complex tasks.

NSPs use programming basics (conditions, loops) and can connect with VLMs that understand images and text.

2 types of NSPs:

? Primitive NSPs: Directly interact with what the robot can see or feel. A primitive NSP might ask the VLM if the robot is holding something or if the gripper is open.

? Derived NSPs: Depend on other NSPs rather than direct observations. For example, they determine if an object is on a plate by checking if it's on another object that is on the plate.

To make sure NSPs are precise, researchers use 2 key strategies:

? Context awareness: Knowing the robot’s last action and previous view helps clarify current situation.

? Object labels: Every object has a unique ID, like "block2". This allows the robot to check items specifically and avoid confusion.

For task planning, robots use High-Level Actions (HLAs). They are general steps that predict what will happen after each action.

HLAs use primitive NSPs for the current task state, while derived NSPs adapt to changes without explicit steps in the final plan.

Robots learn by exploring freely (without human examples) their environment and trying to complete tasks using HLAs, tracking successes and failures to build a useful data set.

3 methods help VLMs suggest useful new NSPs:

? Discrimination: Finds conditions that indicate when a skill will work or fail.

? Transition modeling: Identifies what changes before or after a skill is used, helping the robot predict outcomes.

? Unconditional generation: Suggests new abstract rules based on existing ones, helping to create broader logical connections.

Paper: https://arxiv.org/pdf/2410.23156