How To Turn Your Blog Posts Into An AI Powered Chat App
Written by: Justin Hunter
Republished from here: https://bit.ly/4iDArks
I previously wrote about how to turn docs and knowledge bases into an AI-powered chat app, but I realized the app was missing key context found in the blog posts we publish. Many of our blog posts are technical tutorials which provide rich context that will be useful for any developer wanting to build on Pinata.
In this tutorial, I will show you how we leveraged our Ghost CMS and Pinata’s Vector Storage to build an AI-powered chat app that has deep understanding of Pinata’s blog. You can modify this to accommodate basically any blog that either provides an API that returns HTML or a blog that has markdown files.
Getting Started
To follow along with this tutorial, you will need a Pinata account. You can sign up free here. You will also need to make sure you have Node.js installed on your machine and a good text editor (I prefer VS Code, but there are many out there).
I’m going to use Bun for the scripting part of this tutorial because, in my opinion, it’s the fastest and easiest way to get Typescript support out of the box with no configuration headaches. Follow Bun’s guides for installation to ensure you have the most up-to-date instructions, but this is the command as of today:
curl -fsSL <https://bun.sh/install> | bash
Once you have Bun installed, we need to initialize our project. From your command line, create a new directory:
mkdir blog-crawler && cd blog-crawler
Then, you can run the following command:
bun init
Respond to the prompts, and you will have a simple Bun project set up and ready to modify. We’ll come back to this, but now we want to get our application set up. In a new command line window, let’s create the new project.
npx create-next-app ai-blog-chat
Vectorizing the blog
In order for this to work, we need to create vector embeddings of all of our blog posts. This works best when using markdown format, so we will need to make sure the blog posts are all in markdown.
The Pinata blog is hosted on Ghost, so we need to use Ghost’s API to get all of our posts and convert them to markdown. Then, we need to vectorize the posts, store the vector embeddings, and store the original markdown content. Fortunately, Pinata makes the vectorization and storage process simple with Pinata Vector Storage. You can check out the docs here.
Now, we’re going to need some things before we can write our script. First, create a Pinata API key. Follow this guide and create an admin key. Once you have the API key JWT, create a .env.local file in the root of your blog-crawler project. Make sure that file is added to .gitignore if it’s not already in there. Add your JWT like this:
PINATA_JWT=YOUR_JWT_HERE
Next, we need to create a Group in Pinata. Pinata Groups act as a handy way to organize files, but they are also the built-in way Pinata manages vector indices. In the Pinata web app, go to the Groups link. Click on the Private tab at the top of the page, then create a new Private Group. Once the Group is created, open it up.
There will be nothing inside yet, but we need the Group ID which is in the URL. It will be the last part of the URL. Grab that and add it to your .env.local file like this:
GROUP_ID=YOUR_GROUP_ID
Finally, we need to get the content key from Ghost to be able to read the posts from the Ghost API. From your Ghost account, go to settings, then scroll down until you find Integrations on the left.
You will need to create a custom integration:
Once you’ve done that, the content api key will be displayed. Grab that key and add it you your .env.local file like this:
GHOST_KEY=YOUR_CONTENT_KEY
You will need one more thing, but this isn’t for your environment variables. The Ghost API works by using your Ghost blog’s URL. You’ll need to know this (not the custom domain for your blog, but the domain that looks like yourname.ghost.io).
Ok, let’s write some code!
But first, let’s install some dependencies. Run the following command:
bun add p-limit pinata turndown
We’ll use each of these dependencies as we build out our blog crawler. Now, open your project in your text editor and find the src/index.ts file. Replace the contents with this:
import plimit from "p-limit";
import { PinataSDK } from "pinata";
import TurndownService from "turndown";
import fs from "fs";
export interface GhostPostsResponse {
posts: GhostPost[];
meta: {
pagination: GhostPagination;
};
}
export interface GhostPost {
id: string;
uuid: string;
title: string;
slug: string;
html: string;
comment_id: string;
feature_image: string | null;
featured: boolean;
visibility: string;
created_at: string;
updated_at: string;
published_at: string | null;
custom_excerpt: string | null;
codeinjection_head: string | null;
codeinjection_foot: string | null;
custom_template: string | null;
canonical_url: string | null;
url: string;
excerpt: string;
reading_time: number;
access: boolean;
comments: boolean;
og_image: string | null;
og_title: string | null;
og_description: string | null;
twitter_image: string | null;
twitter_title: string | null;
twitter_description: string | null;
meta_title: string | null;
meta_description: string | null;
email_subject: string | null;
frontmatter: string | null;
feature_image_alt: string | null;
feature_image_caption: string | null;
}
export interface GhostPagination {
page: number;
limit: number;
pages: number;
total: number;
next: number | null;
prev: number | null;
}
const crawlBlogPosts = async () => {
try {
let allPosts: GhostPost[] = [];
let hasMore = true;
let nextPage;
while (hasMore) {
try {
let url = `https://${YOUR_GHOST_URL}/blog/ghost/api/content/posts?key=${GHOST_KEY}`;
if (nextPage) {
url = url + `&page=${nextPage}`;
}
const res = await fetch(url);
const data: GhostPostsResponse = await res.json();
allPosts = [...allPosts, ...data.posts];
if (!data.meta.pagination.next) {
hasMore = false;
} else {
nextPage = data.meta.pagination.next;
}
} catch (error) {
throw error;
}
}
for (const post of allPosts) {
try {
const markdown = convertHtmlToMarkdown(post.html);
console.log(`Writing blog post ${post.title} as markdown`);
const safeTitle = post.title
.replace(/[^a-zA-Z0-9-_ ]/g, "")
.replace(/\\s+/g, "-")
.toLowerCase();
fs.writeFileSync(`./blog_posts/${safeTitle}.md`, markdown);
} catch (error) {
console.log(error);
throw error;
}
}
} catch (error) {
console.log(error);
throw error;
}
};
We’ve added the necessary types to fetch our Ghost blog posts and we’ve written a script that will do the following:
Go ahead and create that folder in the root of your project directory. We could run this function now, but let’s build out the end-to-end solution and run it all at once.
Below the code you just wrote, add the following:
const pinata = new PinataSDK({
pinataJwt: process.env.PINATA_JWT,
pinataGateway: "",
});
const loadDir = async () => {
try {
const dir = fs.readdirSync("./blog_posts");
return dir;
} catch (error) {
console.error("Error reading directory:", error);
throw error;
}
};
export function convertHtmlToMarkdown(html: string): string {
const turndownService = new TurndownService();
return turndownService.turndown(html);
}
const processFile = async (file: any) => {
try {
if (file.includes(".md")) {
const blob = new Blob([fs.readFileSync(`./blog_posts/${file}`)]);
const fileData = new File([blob], file, { type: "text/plain" });
await pinata.upload.private.file(fileData).group(process.env.GROUP_ID).vectorize();
console.log("Vectorized file name:", file);
fs.unlinkSync(`./blog_posts/${file}`);
}
} catch (error) {
console.error(`Error processing file ${file}:`, error);
process.exit(1);
}
};
const uploadAndVectorize = async () => {
try {
const files = await loadDir();
const limit = plimit(3);
const fileProcessingPromises = files.map((file: any) =>
limit(() => processFile(file))
);
await Promise.all(fileProcessingPromises);
console.log("All files processed successfully.");
} catch (error) {
console.error("Error in main process:", error);
process.exit(1);
}
};
We’re doing a few things here, but maybe it’s best to work bottom-up. The uploadAndVectorize function called the loadDir function first. This loads our blog-posts directory. We then set a concurrency limit using p-limit. This allows us to speed up the entire process while not overloading our resources. Next, we create an array of fileProcessingPromises by mapping over the files from the directory and running each through the processFile function.
The processFile function is where the magic happens. We read each file from disk then upload and vectorize the file in a single function through the Pinata SDK:
await pinata.upload.file(fileData).group(process.env.GROUP_ID).vectorize();
Pinata’s SDK makes vector embedding so simple compared to other solutions.
We then remove each file from the directory because we don’t need to store the original markdown anymore. This also makes it easier to pick back up where we left off if there is an error during the upload process.
When the process is complete, all of the blog posts have been uploaded to Private IPFS through Pinata and vector embeddings have been created from them.
Now, all we need to do is wire up these functions to run together and call the start script. Add the following at the bottom of your index.ts file:
领英推荐
async function startProcess() {
try {
await crawlBlogPosts();
await uploadAndVectorize();
console.log("Done!");
process.exit(0);
} catch(error) {
console.log(error);
process.exit(1);
}
}
startProcess();
Now, from your command line run bun run index.ts. This will kick off the process and when everything is complete, you will have the vector embeddings and raw files necessary for building the AI chat app to interact with your posts.
Let’s build the app now!
Building the app
We already set up the basic Next.js app to start our project. So, in your command line, switch to the Next.js project window. We need to install some dependencies and add environment variables similar to our blog crawler script.
Let’s install the Pinata SDK. From the root of you project directory and install Pinata:
npm i pinata
Just like before, we need to save our Pinata JWT as an environment variable. Create a new file in the root of the project called .env. In that file, add the following:
PINATA_JWT=YOUR_JWT_HERE
Paste your JWT in and save the file.
Our app will have a client and a backend. The backend will consist of a Next.js serverless function. We’ll start with the backend, then we’ll work on the interface.
In your project directory, create an api folder inside the app folder. Next, add a chat folder inside the api folder. And finally, add a file called router.ts inside the chat folder. Before we write the code for this API route, we’ll need to install one more dependency.
Our app can make use of any LLM (Large Language Model), especially if the LLM has an API compatible with the OpenAI API. For simplicity, we will just use the OpenAI API. You’ll need to have an OpenAI developer account, and you will need to get an OpenAI API key. Follow this guide to do so.
When you have your OpenAI API key, open your .env file and add it in like so:
OPENAI_API_KEY=YOUR_OPEN_AI_KEY
Now, let’s create a helper file that will make our lives easier when using the Pinata SDK. Create a pinata.ts file in the root of the app folder. Similar to how we set up our upload script, we’re going to be using the Pinata SDK. However, we are going to export it so that it’s available throughout our app.
Add the following to your pinata.ts file:
const { PinataSDK } = require("pinata");
export const pinata = new PinataSDK({
pinataJwt:
process.env.PINATA_JWT,
pinataGateway: process.env.PINATA_GATEWAY_URL
});
export const GROUP_ID = process.env.GROUP_ID;
Now, we could use the OpenAI SDK, and if we wanted more flexibility, we would. But we’re focused on a chat app, and we want to be able to stream the AI model’s responses back to the user interface easily. So, we’re going to make use of an open source tool created by the Vercel team called ai. Install that like this:
npm i @ai-sdk/openai ai
Let’s start writing our API route.
Inside your app/api/chat/route.ts file add the following:
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
import { GROUP_ID, pinata } from "@/app/pinata";
import { NextResponse, NextRequest } from "next/server";
export async function POST(request: NextRequest) {
try {
const body = await request.json();
const { messages } = body;
const text = messages[messages.length - 1].content;
const data = await pinata.private.files.queryVectors({
groupId: GROUP_ID,
query: text,
returnFile: true,
});
const result = streamText({
model: openai('gpt-4o'),
system: `Please only use the following to help inform your response and do not make up additional information: ${data.data}`,
messages,
});
return result.toTextStreamResponse({
headers: {
'Content-Type': 'text/event-stream',
},
});
} catch (error) {
console.log(error);
return NextResponse.json({ message: "Server error" }, { status: 500 });
}
}
Let’s take a look at what’s going on in this file. First, we’re importing the Pinata SDK and our Group ID. In our POST request, we parse the request body and extract the messages array. This array will be provided by the frontend and it will match the expected format for OpenAI chat completions.
We need to extract the user’s actual query from the messages array so we can use it in our vector search, so we do that by grabbing the content property of the last message in the array.
Next, we start working some magic. Using Pinata’s SDK, we can use the query from the user to search our vector embeddings and find the best match. Because Pinata has built vector storage and search to be file-first, we cut out a lot of boilerplate code and latency. With other services, you’d have to:
With Pinata’s vector storage, you just do the following:
We are setting the returnFile property to true with our vector search, which means we’ll get the raw data associated with top match. We provide that data as part of our prompt context to get the best response from the AI model.
Finally, we use the ai library to help us stream the response from OpenAI back to the client. Now, before we start building the application interface, let’s test this. Fire up your app by running:
npm run dev
Then, from the terminal, let’s try a curl request with a text input like so:
curl --location '<https://localhost:3000/api/chat>' \\
--header 'Content-Type: application/json' \\
--data '{
"messages": [
{
"role": "user",
"content": "How can I launch and NFT collection?"
}
]
}'
You should see a response output in the terminal that answers your question. Pretty cool, right?
Time to build our UI. Open up the page.tsx file found in your app folder in the project directory. Replace everything in there with the following:
"use client";
import { useChat } from "ai/react";
import { useRef, useEffect } from "react";
import { User, Bot } from "lucide-react";
export default function Page() {
const { messages, input, handleInputChange, handleSubmit, error } = useChat({
streamProtocol: "text",
});
// Ref to track the last message for auto-scrolling
const bottomRef = useRef(null);
useEffect(() => {
if (bottomRef.current) {
bottomRef.current.scrollIntoView({ behavior: "smooth" });
}
}, [messages]);
return (
<div className="w-screen min-h-screen bg-green flex flex-col">
<h1 className="text-center text-2xl font-bold p-4">Ask Pinnie</h1>
{/* Scrollable Message Container */}
<div className="bg-white rounded-md w-3/4 max-w-[1200px] m-auto flex-grow overflow-y-auto p-4 shadow-lg">
{messages.map((m) => (
<div key={m.id} className="whitespace-pre-wrap">
{m.role === "user" ? (
<div className="w-full flex justify-end my-2">
<span className="w-3/4 bg-purple text-white rounded-md p-2 mr-2">
{m.content}
</span>
<User />
</div>
) : (
<div className="w-full flex justify-start my-2">
<Bot />
<span className="w-3/4 bg-gray-200 text-black rounded-md p-2 ml-2">
{m.content}
</span>
</div>
)}
</div>
))}
{/* Dummy element for auto-scroll */}
<div ref={bottomRef} />
</div>
{/* Input Bar */}
<form
onSubmit={handleSubmit}
className="w-screen bg-white border-t p-4 shadow fixed bottom-0"
>
<input
autoFocus
className="w-full max-w-[1200px] p-2 border outline-none rounded shadow-lg m-auto block text-black"
value={input}
placeholder="Ask Pinnie something..."
onChange={handleInputChange}
/>
</form>
</div>
);
}
This single component is doing a lot of heavy lifting in fewer than 100 lines of code. For a little flare, we’re using icons from lucide-react, so you’ll need to install that library like this:
npm i lucide-react
We’re using the ai library on the frontend here as well. Specifically, we’re using built-in React hooks to handle the streaming response from our backend. We have some UI flare to make this look like a chat interface, and we also have an auto-scroll feature to scroll to the most recent message.
And that’s it. We have a chat app.
Go ahead and load https://localhost:3000 in your browser and take a look. You should see something like this:
Conclusion
Blogs are full of rich information that create incredible context for LLMs. By using Pinata’s Vector Storage, you can support a retrieval augmented generation (RAG) flow for your AI chat apps to increase the context.
Get creative, have fun, and enjoy building RAG apps with Pinata!