Anti-Bias Semantic Prompt Modeling on ChatGPT

The objective of this article is to present an effective analysis of prompt structure, leading to a practical anti-bias solution for ChatGPT. The proposed methodology involves a detailed evaluation of semantics, demonstrated through a real-world recommendation example, showcasing how deep semantic structures of natural language can be modeled and utilized to develop a comprehensive bias mitigation plan to ensure user protection.

Attribute Management useful in Prompt Generation

Please do me a favor. Just type "I want to go shopping. " in ChatGPT and see what it get.

That's not helpful. But you can add more specific informations like this:

If you change the specifier about gender to "I am a woman." ChatGPT would respond to this change.

I cannot speak for my wife's opinion on the matter, but personally, I find the answer to be satisfactory in convincing me of the importance of providing necessary attributes in order to receive a meaningful prompt response.

To obtain effective ChatGPT responses, it is crucial to provide the appropriate meaning specifiers for the corresponding prompts.

Why Attribute Management is needed?

Isn't it a trivial question? You just need to make a template like this: "I am a man, I live in palo alto. [Replace with what I want to ask ChatGPT]." Done.

Well, that's certainly a show stopper. But our show has many "episodes." In the upcoming episodes, we will have these in the show.

• If we wish to apply certain rules based on categories such as man, woman, LGBTQ categories, etc., and utilize different prompt parameters, what would be the approach to implement this?
• If we intend to share with different individuals who may only make minimal configurations to customize their experience, it is possible to receive more personalized attributes that may not necessarily fit into a fixed template.
• If we aim to "instrument" the prompt to gather certain analytics, we could create functions that generate specific instrumentation codes. However, with an unstructured template, how can this be achieved?

There needs to be a monitoring mechanism for ChatGPT's prompt performance in an organizational setting, as well as tooling support for managing the individual and group attributes utilized in the prompts.

Model the Prompt Meanings with Formal Semantics

Utilizing Formal Semantics, we will model the in-depth meaning structures of that example use case, 'I want to go shopping.' This will provide valuable insights into the design of attribute management.

Definition CN:=Set.
Parameter Man: CN.
Parameter Human: CN.
Axiom mh : Man -> Human. Coercion mh: Man >-> Human.
Parameter Pingping: CN.
Axiom pm : Pingping -> Man. Coercion pm: Pingping >-> Man.
Parameter shop: Human->Prop.

Parameter in_palo_alto : Human->Prop.
Parameter I: Pingping.
Definition I_shop:= shop I.
Definition I_am_in_palo_alto:= in_palo_alto I.

Parameter as_a_man: (Man -> Prop) -> (Human -> Prop).
Parameter as_in_palo_alto : (Human->Prop)->(Human->Prop).

Definition I_shop_in_palo_alto:= (as_a_man shop) I.

Definition I_shop_in_palo_alto_in_man_manner:=
? (as_in_palo_alto (as_a_man shop)) I.
        

The above code can be copy pasted into the Coq Online Environment https://coq.vercel.app/scratchpad.html .

I will immediately present the takeaway from the formal semantic code above.

• The prompt 'I go shop' is too general and lacks specificity.
• The term "shop" is too general, as it only pertains to the "Human" category. It is a broad pattern that any human can shop, without specifying the type of human or their shopping preferences.
• The phrase 'as_a_man' represents an implicit semantic meaning. It implies that if Pingping Xiu identifies as a man, then any action taken by Pingping should reflect this identity and be performed 'as_a_man'.
• The phrase "as_in_palo_alto" is another implicit semantic meaning. However, we did not apply the advanced Event features in Formal Semantics here.
• The most specificity prompt is "(as_in_palo_alto (as_a_man shop)) I" which reads I want to shop as a man, as being in palo alto. It directly translate to "I am a man. I lives in Palo Alto. I want to go shopping."
• To move from a generic prompt like "I shop" to a more specific prompt like "I shop in Palo Alto in a manly manner," a lifting rule must be applied that considers properties such as "I am Pingping," "Pingping is a man," and "a man is a human" to make an entailment.

Immediate Benefits of the Formalization

It is possible to obtain the "I_shop_in_palo_alto_in_man_manner" prompt from a function that composes the prompt's logic based on the context or category. For instance, if the category belongs to a minority group like the LGBTQ community, a more suitable prompt would be a complex one that starts by referring to a major category, emphasizes the importance of fairness, and then specifies the request.

The additional complexity required for prompts targeted at minority groups can be abstracted away through a background mechanism such as a LongChain. The user will only see the final prompt, but there may be more underlying calls necessary to generate the appropriate prompt for the minority group. However, the costs associated with these additional calls will decrease over time.

The semantics analysis is a critical aspect of database design. The nuances lie not in the verb "shop," but in the adverbs like "as_a_man" which are implicit but important properties that require careful management. These adverbs need to be paired with the verb "shop," but they also have a categorical dependence on the subject, which is "Man," a subclass of "Human." This dependence on semantics is a foundational element for scalable database schema design.