Why we cannot allow invalid formations of patterns in semantics representation

In the Large Language Model era, few-shot learning is a new norm (See https://arxiv.org/abs/2302.07842 AUGMENTED LANGUAGE MODELS: A SURVEY).

In a sense, we see a paradigm shift from traditional classification-based learning to few-shot learning. In classification-based learning, one has to prepare a handful of examples under each class, and when there is only one interested class, one has to "fake" a negative class to form a multi-class classification problem. But we see in LLM era, few-shot learning has the convenience to be the new norm of learning method.

In few-shot learning, the "target" is a semantic meaningful text. Unlike classification-based learning, where each class label does not carry an internal structure, few-shot learning does allow the target to be a natural language text.

Now, if we want to "normalize" that, by modeling each target text with a semantic structure, and fine-tuning to predict that semantic structure, we may enable the org-specific logic. We also save costs.

However, we must make sure that semantic structure is sound, in terms of not allowing an invalid pattern to occur.

An invalid pattern, if it is allowed within a semantic framework, would be disastrous. To see why remember that a semantic framework by nature is a logic framework. And, the most important property of a logical framework is, no contradiction.

If an invalid pattern is allowed due to a design flaw of the semantic framework, then the first thing that happens is, there are universal rules within that semantic framework that can interact with that invalid pattern to generate some knowledge about some valid pattern, and that "knowledge" is fake, for the invalid pattern does not carry a real-world meaning. Therefore, we may produce some fake knowledge that contradicts some valid knowledge. And that's the end of the logic framework.

Takeaway

• In LLM, expressing the target's semantic structure is an important step toward solving bias and extending the logic capability.
• Such semantic structure should have selection restrictions, which is a fundamental capability of a linguistic framework to rule out invalid patterns.