Using ChatGPT, Embeddings, and HyDE to Improve Search Results
Using ChatGPT, Embeddings, and HyDE to Improve Search Results
Introduction
Staying ahead of the competition is imperative in today's fast-paced business environment. An efficient search engine that can provide accurate information to your customers or employees can make all the difference. However, building and maintaining a robust search engine can be a challenge. In this dev notebook, let’s explore how ChatGPT, Embeddings, and HyDE can help you improve your search results.
Combining ChatGPT with retrieval and re-ranking methods, you can achieve accurate, relevant, and fast search results that will set you apart from your competitors. This approach also facilitates seamless integration with existing search engines, making it an ideal way to improve search engine performance for businesses of all sizes. As CTOs, CIOs, engineering managers, and software engineers, you have much to gain by implementing this approach and making your search engine more efficient.
Overview
ChatGPT is a deep learning model that we will use to rank content returned from a search engine on a hypothetical answer. It creates a hypothetical document based on a given query and then embeddings for this hypothetical answer. The system then sorts the results using the dot calculation to get the cosine similarities to score and sort articles from the search service. Articles are ranked by relevance as determined by the cosine similarity of their embeddings versus the hypothetical answer. Using this approach, ChatGPT can assist in improving search systems.
There are two ways to retrieve information using GPT: Mimicking Human Navigation and Retrieval with embedding. Retrieval with Embeddings can be done by calculating embeddings for query results and an ideal user answer. Then, the most related content, as measured by cosine similarity to the hypothetical answer (HyDE), is retrieved. Combining these approaches and drawing inspiration from re-ranking methods can improve search accuracy.
One of the key benefits of this approach is its ability to be implemented on top of any existing search systems. Combining this with your current system can improve your search engine's accuracy and speed. This approach can be applied to search engines that use Elasticsearch, Solr, or any custom search engine application.
Dive In
This dev notebook shows how ChatGPT employs the Hypothetical Document Embeddings HyDE to rank content on a hypothetical answer. We use the HyDE by having ChatGPT create a hypothetical document based on a given query.
Then we use ChatGPT to create embeddings hypothetical answer, then sort the results using the dot calculation to get the cosine_similarities and rank the articles by relevance as determined by the cosine_similarities of their embeddings vs. the hypothetical answer.
Looking for relevant information can be a challenging task. This dev notebook adapts a ChatGPT cookbook for improved search to Java, then breaks the steps down and explains each step. In this developer notebook, we explore a way to improve existing search systems with various AI techniques that will help us. We explore a way to filter through the data using various AI techniques to help us put our data through Boot camp so the data becomes information.
There are two ways to retrieve information using GPT:
Combining these approaches and drawing inspiration from re-ranking methods gives us a simple pattern to improve search. This approach can be implemented on top of any existing search systems.
Steps:
This technique is fast and can be added to a search function you already have without managing a vector database. We will use the News API as our search engine stand-in to show how this fits together. Since our examples are all in Java, we use JAI, the Java Open AI API client lib .
?? News API - search engine for searching the news
News API is a simple and easy-to-use API that returns JSON metadata for headlines and articles from various news sources. It provides live breaking news headlines and articles from over 70,000 news sources and blogs worldwide, including CNN, BBC News, The New York Times, and more. The API is free for non-commercial purposes and offers a range of endpoints to query news data, including top headlines, everything, sources, and more.
To use the News API, you must sign up for an API key, a unique identifier that allows you to access the API. Once you have an API key, you can request HTTP to the API endpoints to retrieve news data in JSON format. The API supports a range of parameters that allow you to filter and sort news data based on various criteria, such as language, country, category, and more.
Main Method is the main flow.
Let’s dive right in.
I created a Java class named WhoWonUFC290, incorporating question-answering functionality using a search API and re-ranking techniques. The question we will answer is “Who won Main card fights in UFC 290? Tell us who won. Are there any new champions? Where are they from?”
The example aims to demonstrate a method of augmenting existing search systems with AI techniques to improve information retrieval.
Notice it covers all of the steps we talked about thus far. Here is the main method of doing all the steps. We will break this down step by step.
public static void main(String... args) {
try {
// Generating queries based on user question
String queriesJson = jsonGPT(QUERIES_INPUT.replace("{USER_QUESTION}", USER_QUESTION));
List<String> queries = JsonParserBuilder.builder().build().parse(queriesJson)
.getObjectNode().getArrayNode("queries")
.filter(node -> node instanceof StringNode)
.stream().map(Object::toString).collect(Collectors.toList());
// Generating a hypothetical answer and its embedding
var hypotheticalAnswer = hypotheticalAnswer();
System.out.println(hypotheticalAnswer);
var hypotheticalAnswerEmbedding = embeddings(hypotheticalAnswer);
// Searching news articles based on the queries
List<ArrayNode> results = queries.subList(0, 10).stream()
.map(WhoWonUFC290::searchNews).collect(Collectors.toList());
// Extracting relevant information from the articles
List<ObjectNode> articles = results.stream().map(arrayNode ->
arrayNode.map(on -> on.asCollection().asObject()))
.flatMap(List::stream).collect(Collectors.toList());
// Extracting article content and generating embeddings for each article
List<String> articleContent = articles.stream().map(article ->
String.format("%s %s %s", article.getString("title"),
article.getString("description"), article.getString("content").substring(0, 100)))
.collect(Collectors.toList());
List<float[]> articleEmbeddings = embeddings(articleContent);
// Calculating cosine similarities between the hypothetical answer embedding and article embeddings
List<Float> cosineSimilarities = articleEmbeddings.stream()
.map(articleEmbedding -> dot(hypotheticalAnswerEmbedding, articleEmbedding))
.collect(Collectors.toList());
// Creating a set of scored articles based on cosine similarities
Set<ScoredArticle> articleSet = IntStream.range(0,
Math.min(cosineSimilarities.size(), articleContent.size()))
.mapToObj(i -> new ScoredArticle(articles.get(i), cosineSimilarities.get(i)))
.collect(Collectors.toSet());
// Sorting the articles based on their scores
List<ScoredArticle> sortedArticles = new ArrayList<>(articleSet);
Collections.sort(sortedArticles, (o1, o2) -> Float.compare(o2.getScore(), o1.getScore()));
// Printing the top 5 scored articles
sortedArticles.subList(0, 5).forEach(s -> System.out.println(s));
// Formatting the top results as JSON strings
String formattedTopResults = String.join(",\\n", sortedArticles.stream().map(sa -> sa.getContent())
.map(article -> String.format(Json.niceJson("{'title':'%s', 'url':'%s', 'description':'%s', 'content':'%s'}\\n"),
article.getString("title"), article.getString("url"), article.getString("description"),
getArticleContent(article))).collect(Collectors.toList()).subList(0, 10));
// Generating the final answer with the formatted top results
String finalAnswer = jsonGPT(ANSWER_INPUT.replace("{USER_QUESTION}", USER_QUESTION)
.replace("{formatted_top_results}", formattedTopResults));
System.out.println(finalAnswer);
} catch (Exception e) {
e.printStackTrace();
}
}
The main method does the following, which we will explain in detail in the next few sections:
This is the basics of the technique to sort and rank search results (articles) using ChatGPT based on the relevance of their content to a hypothetical answer. It utilizes embeddings and cosine similarities to determine the similarity between the articles and the hypothetical answer. The result is a list of top-ranked articles that can be used to provide an informative answer to the user's question.
Please note that the code references certain variables (QUERIES_INPUT, USER_QUESTION, hypotheticalAnswer, jsonGPT, embeddings, etc.) and methods (searchNews, dot, getArticleContent) that are not shown in the provided code snippet. We will show these as we cover the basics.
The general flow of the app
Queries Generation
This part of the example generates queries based on the user's question using a JSON template (QUERIES_INPUT) and the USER_QUESTION variable.
public class WhoWonUFC290 {
private static final String QUERIES_INPUT = "Generate an array of search queries that are " +
"relevant to this question.\\n" +
"Use a variation of related keywords for the queries, trying to be as general as possible.\\n" +
"Include as many queries as you can think of, including and excluding terms.\\n" +
"For example, include queries like ['keyword_1 keyword_2', 'keyword_1', 'keyword_2'].\\n" +
"Be creative. The more queries you include, the more likely you are to find relevant results.\\n" +
"\\n" +
"User question: {USER_QUESTION}\\n" +
"\\n" +
"Format: {{\\"queries\\": [\\"query_1\\", \\"query_2\\", \\"query_3\\"]}}";
private static final String USER_QUESTION = "Who won Main card fights in UFC 290? Tell us who won. " +
"Are there any new champions? Where are they from?";
public static void main(String... args) {
try {
// Generating queries based on user question
String queriesJson = jsonGPT(QUERIES_INPUT.replace("{USER_QUESTION}", USER_QUESTION));
List<String> queries = JsonParserBuilder.builder().build().parse(queriesJson)
.getObjectNode().getArrayNode("queries")
.filter(node -> node instanceof StringNode)
.stream().map(Object::toString).collect(Collectors.toList());
Let’s walk through this:
Since we will use jsonGPT a lot, let’s specify how the jsonGPT is implemented. jsonGPT calls GPT and returns a JSON string.
The provided code snippet defines a method named jsonGPT that interacts with the OpenAI API to generate responses in a chat-based manner. Here's a breakdown of the code:
public static String jsonGPT(String input) {
final var client = OpenAIClient.builder().setApiKey(System.getenv("OPENAI_API_KEY")).build();
final var chatRequest = ChatRequest.builder()
.addMessage(Message.builder().role(Role.SYSTEM)
.content("All output shall be JSON").build())
.addMessage(Message.builder().role(Role.USER).content(input).build())
.build();
final var chat = client.chat(chatRequest);
if (chat.getResponse().isPresent()) {
return chat.getResponse().get().getChoices().get(0)
.getMessage().getContent();
} else {
System.out.println("" + chat.getStatusCode().orElse(666) + "" + chat.getStatusMessage().orElse(""));
throw new IllegalStateException();
}
}
The jsonGPT method sends a chat request to the OpenAI API and retrieves the response. It takes an input string as a parameter, sends the input to the API, and returns the content of the first choice in the response as a string.
Please note that the code snippet does not show the imports or the class where the jsonGPT method resides. Additionally, it assumes the availability of the OpenAI API client and relevant dependencies. We are using JAI, the Java Open AI API client lib . The full code listing is below and this example is included with JAI .
Sample Queries Generated by ChatGPT
UFC 290 Main card winners
Who were the winners in Main card fights UFC 290?
UFC 290 Main card champions
List of winners in UFC 290 Main card fights
UFC 290 Main card results
UFC 290 Main card winners nationality
UFC 290 Main card champions country
Who won the fights in Main card of UFC 290?
Tell me the winners of Main card fights in UFC 290
UFC 290 Main card winners and their countries
UFC 290 Main card fight results
New champions in UFC 290 Main card fights
Who are the winners in Main card of UFC 290?
UFC 290 Main card winners list
UFC 290 Main card champions nationality
Hypothetical Answer Generation
This part of the example uses ChatGPT to generate a hypothetical answer and then calculate its embedding. The example integrates with the OpenAI API to generate a hypothetical answer using a chat-based language model. It replaces placeholders in a template with the user's question and retrieves the generated answer as a response. The hypothetical answer embedding is later used to filter articles we retrieve from the new search.
public class WhoWonUFC290 {
private static final String HA_INPUT ="Generate a hypothetical answer to the user's question. " +
"This answer will be used to rank search results. \\n" +
"Pretend you have all the information you need to answer, but don't use any actual facts. " +
"Instead, use placeholders\\n" +
"like NAME did something, or NAME said something at PLACE. \\n" +
"\\n" +
"User question: {USER_QUESTION}\\n" +
"\\n" +
"Format: {{\\"hypotheticalAnswer\\": \\"hypothetical answer text\\"}}";
private static final String USER_QUESTION = "Who won Main card fights in UFC 290? Tell us who won. " +
"Are there any new champions? Where are they from?";
public static String jsonGPT(String input) {
final var client = OpenAIClient.builder().setApiKey(System.getenv("OPENAI_API_KEY")).build();
final var chatRequest = ChatRequest.builder()
.addMessage(Message.builder().role(Role.SYSTEM).content("All output shall be JSON").build())
.addMessage(Message.builder().role(Role.USER).content(input).build())
.build();
final var chat = client.chat(chatRequest);
if (chat.getResponse().isPresent()) {
return chat.getResponse().get().getChoices().get(0).getMessage().getContent();
} else {
System.out.println("" + chat.getStatusCode().orElse(666) + "" + chat.getStatusMessage().orElse(""));
return "";
}
}
public static String hypotheticalAnswer() {
final var input = HA_INPUT.replace("{USER_QUESTION}",
USER_QUESTION );
return jsonGPT(input);
}
public static void main(String... args) {
try {
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
var hypotheticalAnswer = hypotheticalAnswer();
System.out.println(hypotheticalAnswer);
var hypotheticalAnswerEmbedding = embeddings(hypotheticalAnswer);
...
The above example represents a Hypothetical Answer Generation functionality within the WhoWonUFC290 class. It generates a hypothetical answer and its embedding. Here's a breakdown of the code:
hypotheticalAnswer
There were some thrilling battles in the Main card fights of UFC 290.
NAME emerged victorious in the first fight, showcasing impressive
skills and defeating their opponent with a spectacular knockout.
In the second fight, NAME came out on top,
displaying dominant grappling techniques that led to a submission victory.
In the third fight, NAME showed incredible striking abilities,
outclassing their opponent and securing a unanimous decision win.
However, the winners of the Main card fights hailed from diverse backgrounds.
NAME, the winner of the first fight, is from PLACE1. NAME2,
who triumphed in the second fight, hails from PLACE2. Lastly,
the winner of the third fight, NAME3, is from PLACE3.
?? Text embedding
Text embedding is a technique used in natural language processing to represent words, phrases, or documents as vectors of numerical values. These vectors suit various machine learning algorithms, including clustering, classification, and information retrieval. The embedding process maps each word, phrase, or document to a vector in a high-dimensional space, where the distance between vectors represents the degree of semantic similarity between the corresponding words, phrases, or documents. The embedding process is usually performed using deep neural networks trained on large collections of text data. These networks learn to map similar words or phrases to similar vectors in the high-dimensional space, allowing for efficient representation and processing of text data.
ChatGPT’s Open AI API has an embedding API to get text embeddings from your text. Here is an example of getting an embedding which we will use later to filter the articles returned from the new search.
public class WhoWonUFC290 {
...
public static float[] embeddings(String input) {
return embeddings(List.of(input)).get(0);
}
public static List<float[]> embeddings(List<String> input) {
final var client = OpenAIClient.builder()
.setApiKey(System.getenv("OPENAI_API_KEY")).build();
var embedding = client.embedding(EmbeddingRequest.builder()
.model("text-embedding-ada-002").input(input).build());
if (embedding.getResponse().isPresent()) {
return embedding.getResponse().get().getData().stream()
.map(Embedding::getEmbedding).collect(Collectors.toList());
} else {
System.out.println("" + embedding.getStatusCode().orElse(666) + "" + embedding.getStatusMessage().orElse(""));
throw new IllegalStateException();
}
}
...
public static void main(String... args) {
try {
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
var hypotheticalAnswer = hypotheticalAnswer();
System.out.println(hypotheticalAnswer);
var hypotheticalAnswerEmbedding = embeddings(hypotheticalAnswer);
Let’s break these down a bit.
The embeddings methods utilize the OpenAI API client to generate embeddings for input strings. The embeddings capture the semantic representations of the text. The code handles both single and multiple input strings, returning the corresponding embeddings as float arrays or a list of float arrays.
News Article Search
Next, we need some results to filter. We will use the News Search. The example searches for news articles based on the generated queries using the searchNews method as follows.
public class WhoWonUFC290 {
...
public static void main(String... args) {
try {
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
List<ArrayNode> results = queries.subList(0, 10).stream()
.map(WhoWonUFC290::searchNews).collect(Collectors.toList());
...
This part of the example focuses on the step of searching news articles based on the generated queries. It uses Java’s stream function and the map operation to apply the searchNews method to each query. The resulting articles are collected into a list of ArrayNode objects stored in the results variable.
The searchNews method is defined as follows:
public class WhoWonUFC290 {
...
public static ArrayNode searchNews(final String query) {
final var end = Instant.now();
final var start = end.minus(java.time.Duration.ofDays(5));
return searchNews(query, start, end, 5);
}
public static ArrayNode searchNews(final String query, final Instant start, final Instant end, final int pageSize) {
System.out.println(query);
try {
String url = "<https://newsapi.org/v2/everything?q=>" +
URLEncoder.encode(query, StandardCharsets.UTF_8)
+ "&apiKey=" + System.getenv("NEWS_API_KEY")
+ "&language=en" + "&sortBy=relevancy"
+ "&from=" + dateStr(start) + "&to=" + dateStr(end)
+ "&pageSize=" + pageSize;
HttpClient httpClient = HttpClient.newHttpClient();
HttpResponse<String> response = httpClient.send(
HttpRequest.newBuilder().uri(URI.create(url))
.GET().setHeader("Content-Type", "application/json").build(),
HttpResponse.BodyHandlers.ofString());
if (response.statusCode() >= 200 && response.statusCode() < 299) {
return JsonParserBuilder.builder().build().parse(response.body()).atPath("articles").asCollection().asArray();
} else {
throw new IllegalStateException(" status code " + response.statusCode() + " " + response.body());
}
} catch (Exception ex) {
throw new IllegalStateException(ex);
}
}
This part of the example introduces the searchNews method within the WhoWonUFC290 class. This method searches news articles based on a given query and a specified time range. Here's a breakdown of the code:
The searchNews method enables searching news articles based on a query and a time range. It interacts with the News API by constructing the appropriate URL, sending an HTTP request, and handling the response. The method parses the JSON response and returns the "articles" section as an ArrayNode.
Please note that the code relies on certain environment variables such as "NEWS_API_KEY" and utilizes classes and methods (JsonParserBuilder, URLEncoder, HttpClient, HttpResponse, etc.).
Relevant Information Extraction
Next, we need to extract relevant information from the retrieved articles (title, description, and partial content) that we can use to create embeddings to score the articles.
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
List<ObjectNode> articles = results.stream().map(arrayNode ->
arrayNode.map(on -> on.asCollection().asObject()))
.flatMap(List::stream).collect(Collectors.toList());
// Extracting article content and generating embeddings for each article
List<String> articleContent = articles.stream().map(article ->
String.format("%s %s %s", article.getString("title"),
article.getString("description"),
article.getString("content")
.substring(0, 100)))
.collect(Collectors.toList());
After generating queries, a hypothetical answer, and searching for news articles, the code extracts relevant information from the retrieved articles and generates embeddings for each article. Here's a breakdown of the code:
领英推荐
Embeddings Generation
Next, we need to generate embeddings for each article's content.
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
...
// Extracting article content and generating embeddings for each article
List<String> articleContent = articles.stream().map(article ->
String.format("%s %s %s", article.getString("title"),
article.getString("description"), article.getString("content").substring(0, 100)))
.collect(Collectors.toList());
List<float[]> articleEmbeddings = embeddings(articleContent);
After generating queries, a hypothetical answer, searching for news articles, and extracting relevant information from the articles, the example proceeds with extracting article content and generating embeddings for each article. Here's a breakdown of the code:
The code focuses on extracting article content from the relevant information obtained from the news articles. It generates a list of formatted strings that include the title, description, and a substring of the content. Additionally, it calls the embeddings method to generate embeddings for the article content strings, resulting in a list of float arrays representing the embeddings.
Just so you know, the provided example snippet is a continuation of the previous example, and we will continue to the next step, the cosine similarities calculation.
?? Cosine Similarities Calculation
Cosine similarity measures the similarity between two vectors in a multi-dimensional space. It measures the cosine of the angle between two vectors and provides a value between -1 and 1. A value of 1 indicates that the vectors are pointing in the same direction, 0 indicates that the vectors are orthogonal (perpendicular), and -1 indicates that the vectors are pointing in opposite directions.
The cosine similarity is widely used in various fields, including information retrieval, natural language processing, recommendation systems, and machine learning. It is particularly useful when comparing documents or texts represented as vectors in a high-dimensional space.
To calculate the cosine similarity between two vectors, you can use the following formula:
cosine similarity = (A · B) / (||A|| * ||B||)
Here, A and B represent the two vectors being compared, "·" denotes the dot product of the vectors, and "|| ||" denotes the Euclidean norm or magnitude of a vector.
In natural language processing, cosine similarity is often used for document clustering, retrieval, and sentence similarity tasks. By representing documents or sentences as numerical vectors (e.g., using techniques like TF-IDF or word embeddings), cosine similarity can identify related or similar pieces of text.
Cosine similarity is a similarity measure between two non-zero vectors in an inner product space. It quantifies how similar the directions of two vectors are, regardless of their magnitudes. This concept is essential in many data analysis and machine learning areas, as it quantifies similarity between vectors and identifies relationships in high-dimensional spaces.
The cosine similarity is a popular metric in text analysis, used to measure the similarity of documents. In text analysis, each word is assigned a different coordinate, and a document is represented by the vector of the number of occurrences of each word in the document. Cosine similarity then gives a useful measure of how similar two documents are likely to be in terms of their subject matter and independently of the length of the documents.
For example, consider the following two documents:
Document 1: The quick brown fox jumps over the lazy dog. Document 2: The dog saw the fox jump over the lazy brown fence.
The cosine similarity between two documents would be high if they contain the same words in the same order, such as "quick", "brown", "fox", "jumps", "over", and "lazy". This metric can also be used to measure the similarity of groups of objects, such as products, customers, or users.
Cosine similarity is a simple and effective metric that can be used to measure the similarity of two or more vectors. It has several advantages, such as being independent of the magnitudes of the vectors and being a popular metric in machine learning algorithms like recommendation systems and clustering algorithms.
However, cosine similarity has some disadvantages. It does not consider the order of the elements in the vectors and is not sensitive to differences in the magnitudes of the elements in the vectors.
Overall, cosine similarity is a versatile and effective metric that can be used in text analysis and other types of data analysis to measure similarity between vectors.
Cosine Similarities Calculation
Next, calculate cosine similarities between the hypothetical answer embedding and the article embeddings.
public static float dot(float[] v1, float[] v2) {
assert v1.length == v2.length;
float result = 0;
for (int i = 0; i < v1.length; i++) {
result += v1[i] * v2[i];
}
return result;
}
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
...
// Extracting article content and generating embeddings for each article
...
List<float[]> articleEmbeddings = embeddings(articleContent);
// Calculating cosine similarities between the hypothetical answer embedding and article embeddings
List<Float> cosineSimilarities = articleEmbeddings.stream()
.map(articleEmbedding ->
dot(hypotheticalAnswerEmbedding, articleEmbedding))
.collect(Collectors.toList());
After generating queries, a hypothetical answer, searching for news articles, extracting relevant information from the articles, and generating embeddings for each article, the code proceeds with calculating cosine similarities between the hypothetical answer embedding and the article embeddings, which we will later use to score and sort the articles. Here's a breakdown of the code:
The example focuses on calculating the cosine similarities between the hypothetical answer embedding and the embeddings of the articles. It utilizes the dot method to compute the dot product and maps the resulting similarities to a list of Float objects.
Just so you know, the provided code snippet is a continuation of the previous example, and we are off to the next section, where we explain the article score and sort.
Scored Article Creation
Now let’s create a set of scored articles based on the cosine similarities and eliminate duplicates.
public class WhoWonUFC290 {
public static class ScoredArticle {
private final ObjectNode content;
private final float score;
public ScoredArticle(ObjectNode content, float score) {
this.content = content;
this.score = score;
}
public ObjectNode getContent() {
return content;
}
public float getScore() {
return score;
}
@Override
public String toString() {
return "ScoredArticle{" +
"content='" + content.getString("title") + '\\'' +
", score=" + score +
'}';
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof ScoredArticle)) return false;
ScoredArticle that = (ScoredArticle) o;
return Float.compare(that.score, score) == 0 &&
Objects.equals(content.getString("title"), that.content.getString("title"));
}
@Override
public int hashCode() {
return Objects.hash(content.getString("title"), score);
}
}
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
...
// Extracting article content and generating embeddings for each article
...
List<float[]> articleEmbeddings = embeddings(articleContent);
// Calculating cosine similarities between the hypothetical answer embedding and article embeddings
List<Float> cosineSimilarities = ...
// Creating a set of scored articles based on cosine similarities
Set<ScoredArticle> articleSet = IntStream.range(0,
Math.min(cosineSimilarities.size(), articleContent.size()))
.mapToObj(i -> new ScoredArticle(articles.get(i), cosineSimilarities.get(i)))
.collect(Collectors.toSet());
// Sorting the articles based on their scores
List<ScoredArticle> sortedArticles = new ArrayList<>(articleSet);
Collections.sort(sortedArticles, (o1, o2) -> Float.compare(o2.getScore(), o1.getScore()));
// Printing the top 5 scored articles
sortedArticles.subList(0, 5).forEach(s -> System.out.println(s));
After generating queries, a hypothetical answer, searching for news articles, extracting relevant information from the articles, generating embeddings for each article, and calculating cosine similarities, the code proceeds with creating a set of scored articles, sorting them based on their scores, and printing the top 5 scored articles. Here's a breakdown of the code:
This part of the example focuses on creating a set of scored articles, sorting them based on their scores, and printing the top 5 scored articles. It utilizes the ScoredArticle class to encapsulate the content and filter duplicates. Then it scores each article and leverages the Collections.sort method to perform the sorting operation.
The score is basically how similar the article is to the ideal answer.
Just so you know, the provided code snippet is a continuation of the previous example, and now we are down to sort based on the score.
Articles Sorting
Next, the example sorts the articles based on their scores, which we calculated in the previous step.
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
...
// Extracting article content and generating embeddings for each article
...
List<float[]> articleEmbeddings = embeddings(articleContent);
// Calculating cosine similarities between the hypothetical answer embedding and article embeddings
List<Float> cosineSimilarities = ...
// Creating a set of scored articles based on cosine similarities
Set<ScoredArticle> articleSet = ...
// Sorting the articles based on their scores
List<ScoredArticle> sortedArticles = new ArrayList<>(articleSet);
Collections.sort(sortedArticles,
(o1, o2) -> Float.compare(o2.getScore(), o1.getScore()));
// Printing the top 5 scored articles
sortedArticles.subList(0, 5).forEach(s -> System.out.println(s));
After generating queries, a hypothetical answer, searching for news articles, extracting relevant information from the articles, generating embeddings for each article, calculating cosine similarities, creating a set of scored articles, and sorting them based on their scores, the code proceeds with printing the top 5 scored articles. Here's a breakdown of the code:
The code focuses on printing the top 5 scored articles. It utilizes the subList method to extract the desired sublist from the sorted list and the forEach method to iterate over the sublist and print each scored article.
Top Results Formatting
Next, the example formats the top results as JSON strings that we can embed in our final question to get the final answer.
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
...
// Extracting article content and generating embeddings for each article
...
List<float[]> articleEmbeddings = embeddings(articleContent);
// Calculating cosine similarities between the hypothetical answer embedding and article embeddings
List<Float> cosineSimilarities = ...
// Creating a set of scored articles based on cosine similarities
Set<ScoredArticle> articleSet = ...
// Sorting the articles based on their scores
List<ScoredArticle> sortedArticles = new ArrayList<>(articleSet);
Collections.sort(sortedArticles,
(o1, o2) -> Float.compare(o2.getScore(), o1.getScore()));
...
// Formatting the top results as JSON strings
String formattedTopResults = String.join(",\\n",
sortedArticles.stream()
.map(sa -> sa.getContent())
.map(article -> String.format(
Json.niceJson("{'title':'%s', 'url':'%s', 'description':'%s', 'content':'%s'}\\n"),
article.getString("title"),
article.getString("url"), article.getString("description"),
getArticleContent(article)
))
.collect(Collectors.toList()).subList(0, 10));
After generating queries, a hypothetical answer, searching for news articles, extracting relevant information from the articles, generating embeddings for each article, calculating cosine similarities, creating a set of scored articles, and sorting them based on their scores, the code proceeds with formatting the top results as JSON strings so we can embed them into our final question. Here's a breakdown of the code:
The code focuses on formatting the top results as JSON strings using the stream function, map operation, and String.format. It collects the transformed strings into a list and joins them using String.join.
Just so you know, the provided code snippet continues the previous example, and now let's cover the final part. The grand finale! The final answer!
Final Answer Generation
Next, we generate the final answer using a JSON template (ANSWER_INPUT) with the user's question and the formatted top results.
After generating queries, a hypothetical answer, searching for news articles, extracting relevant information from the articles, generating embeddings for each article, calculating cosine similarities, creating a set of scored articles, sorting them based on their scores, and formatting the top results as JSON strings, the example proceeds with generating the final answer using a ANSWER_INPUT template.
public class WhoWonUFC290 {
private static final String ANSWER_INPUT ="Generate an answer to the user's question " +
"based on the given search results. \\n" +
"TOP_RESULTS: {formatted_top_results}\\n" +
"USER_QUESTION: {USER_QUESTION}\\n" +
"\\n" +
"Include as much information as possible in the answer. Reference the " +
"relevant search result urls as markdown links";
public static void main(String... args) {
...
// Generating queries based on user question
...
// Generating a hypothetical answer and its embedding
...
// Searching news articles based on the queries
...
// Extracting relevant information from the articles
...
// Extracting article content and generating embeddings for each article
...
List<float[]> articleEmbeddings = embeddings(articleContent);
// Calculating cosine similarities between the hypothetical answer embedding and article embeddings
List<Float> cosineSimilarities = ...
// Creating a set of scored articles based on cosine similarities
Set<ScoredArticle> articleSet = ...
// Sorting the articles based on their scores
List<ScoredArticle> sortedArticles = ...
...
// Formatting the top results as JSON strings
String formattedTopResults = ...
// Generating the final answer with the formatted top results
String finalAnswer = jsonGPT(ANSWER_INPUT
.replace("{USER_QUESTION}", USER_QUESTION)
.replace("{formatted_top_results}", formattedTopResults));
System.out.println(finalAnswer);
Here's a breakdown of the code:
The code focuses on generating the final answer by replacing the placeholders in the JSON template with the actual values and passing it to the jsonGPT method.
Final Answer
{
"answer": "In the main card fights of UFC 290, the winners were:\\n\\n1.
Alexander Volkanovski from Australia defeated Yair Rodriguez
from Mexico by TKO (punches) in Round 3.\\n2. Alexandre Pantoja
from Brazil defeated Brandon Moreno from Mexico by split decision.
\\n\\nAs a result, Alexander Volkanovski retained the
featherweight championship title. He is from Australia.",
"links": [
{
"title": "Alexander Volkanovski And the Real Winners and Losers from UFC 290",
"url": "<https://bleacherreport.com/articles/10082051-alexander-volkanovski-and-the-real-winners-and-losers-from-ufc-290>"
}
]
}
Conclusion
In summary, leveraging ChatGPT, Embeddings, and HyDE to enhance search results is an effective strategy for swiftly retrieving accurate information and increasing customer and employee satisfaction. By combining ChatGPT with retrieval and re-ranking methods, businesses can achieve precise, relevant, and expedient search results, setting themselves apart from competitors. Furthermore, this approach seamlessly integrates with existing search engines, making it an ideal solution for improving performance across organizations of all sizes. As CTOs, CIOs, Engineering Managers, and Software Engineers, implementing this approach will yield substantial benefits, elevating the efficiency of your search engine.
Please check out JAI, the Java Open AI API client lib , and if you liked this article, give it a star on GitHub.
To see complete code listings with a bit more syntax coverage and to download the example, go here:
Follow up links