Insights from Initial Implementation of the Personal Knowledge Vault:

The personal knowledge vault makes it easy to capture, store, curate and retrieve knowledge now matter where or how its stored. Its like reading glasses for your mind.

The Challenge:

Accessing the knowledge that we need through the course of our work or in our personal lives has become incredibly complex. Whether its knowledge we have created, knowledge we need to decide or knowledge we are synthesizing for others, what used to be a relatively straightforward process now involves navigating a complex ecosystem of walled gardens of clouds and apps all competing for our data.

Each of these Apps has its own authentication, user interface, unique ways of describing and storing knowledge making it incredibly for the average user to work with, much less someone facing functional memory impairment.

The challenge is the same regardless of whether you are a single person, or part of a large organization, you have to do a lot of work to first find the right place to store the files you are working on, and even more work to find them again.

While major device and cloud providers are taking steps to address this complexity, it is still being done in a way designed to lock users into their version of the walled garden: great if you happen to exist entirely in their ecosystem but exacerbates the complexity as soon as you step outside the fence. Apps and cloud hosts design their systems explicitly to pull and retain data in the system in question.

The App Lock in pattern is compounded by the fundamental way many apps force the user to store files: having the user create a name for the file, and navigating to the 'correct' folder to store the file in.

The net of this is that the basic process of creating, storing and retrieving knowledge is still far too complex for the average user, much less someone who is having challenges with their functional memory.

This proliferation of applications and the lack of a useful pattern for storing and curating knowledge more effectively highlights the urgent need for AI to simplify user interfaces and streamline interactions for knowledge storage, curation and retrieval.

AI can play a crucial role in addressing this complexity by providing more intuitive and seamless user experiences.

The Research:

The Personal Knowledge Vault is a research project whose objective is to explore the application of AI to radically simplify users’ ability to find, discover and work with their personal knowledge while ensuring security and privacy. I am developing prototype AI Knowledge Agents that can automatically store and retrieve Knowledge Assets without the user having to deal with the complexity of where and how to store them.

I have a basic Proof of Concept (POC) now operational using a GTP in ChatGPT that is connected to a custom, cloud hosted, secured API backed by a knowledge graph whose goal is to help me verify the hypothesis:

“AI Agents will make it easy for a user to create, store and retrieve basic Knowledge Assets such as Articles, Documents and Notes by performing the capture, categorization and curation of Knowledge Assets for the user.”

The Proof of Concept:

Given the user in question is working on writing their memoires, I determined a useful test to validate the approach was to focus on a few basic operations that are typically involved with this exercise:

• Capture notes from sources such as books, handwritten notes or using voice transcription
• Add articles found on the internet to the knowledge vault
• Retrieve the captured knowledge by asking questions
• Retrieve word documents the user had created through the course of their career stored in the Knowledge Vault by asking questions
• Direct the Knowledge Agents to perform research and generate summarized versions of the research that could be then used for inclusion in the memoires

In exploring a new user experience for our proof of concept, I opted for an AI-based conversational approach rather than a traditional user interface. This shift aims to address the increasing complexity and difficulty users face with the conventional desktop metaphor, which has become burdened by the proliferation of thousands of apps, each with intricate interfaces, file folders, and filenames. This system demands users to navigate significant complexity just to create and store documents.

The complications we face in our current knowledge creation workflow stem from the technological constraints present when the desktop metaphor was first commercialized in the 1980s. Back then, the interface of apps, folders, and icons offered a basic Metaphorical Interface that translated machine commands into familiar actions, mimicking objects on a 1970s office desk. This interface, with its apps, icons, file folders, and pointer devices, was revolutionary when it emerged from the Xerox Alto onto personal computers and later onto mobile devices. However, over time, it has become cumbersome and challenging to manage.

AI offers a paradigm shift, enabling natural interactions with machines through questions and answers rather than forcing users to learn complex metaphors or cryptic commands. Unlike traditional databases, large language models (LLMs) facilitate these interactions. The primary challenge lies in ensuring that the AI delivers results that align with user expectations, effectively bridging the gap between human intent and machine response.

Based on these requirements I developed a Proof of Concept using a GTP in ChatGPT that is connected to a custom, cloud hosted, secured API backed by a knowledge graph to anchor the context in the user's knowledge sources.

Proof of Concept Setup:

In order to create a viable Proof of Concept system with sufficient capabilities to perform a basic test of the hypothesis, I set up chatGPT Teams edition on the user's systems including their mobile devices and laptop using the native chatGPT apps. The GPT was published to a private link, which I then registered into their chatGPT instance using the chatGPT teams workspace functionality which allows the administrator to control which GPTs people on the team have access to.

To facilitate exploring the basic concept of using contextual questions to retrieve knowledge without forcing the user to know explicitly where the knowledge was stored or what the file container was named, I uploaded all of the user's word documents to a cloud hosted file store. I linked this file store to an agent which was then given explicit permissions to scan through the library and enumerate the documents, which were then decomposed into a Graph Document structure that is stored in a Neo4j database.

I developed a python based API using fastAPI that provides a security layer, and endpoints designed to cover the desired interactions. LangChain is used to handle the interfacing between OpenAI GPT4o model and the Neo4j knowledge graph. Long running processes such as chunking documents into Page->Child structures, generating questions and summaries are performed using python's Celery Async library. Langgraph is used to develop the AI agent workflows that can research a topic provided by scanning the Private Knowledge Vault, a public knowledge base (Tavily in this case), curate the knowledge identified by ranking it against the topic for relevancy, and then use the items selected to write an article on the topic.

All of the actions taken by the Langgraph workflow are captured in the graph for studying and eventually improving the interaction and workflow patterns.

The GPT is interfaced into the API using chatGPT's openAPI 3.1 API mapping specification. This allows interaction requests by the user to be mapped to specific Actions.

The python API is hosted on Digital Ocean in a docker container. Images and files uploaded to chatGPT for text extraction are captured and downloaded by the API into a Minio S3 compliant storage container also hosted in the Digital Ocean instance.

Initial Testing:

Based on the initial testing, the user can much more fluidly and rapidly do the following:

• Capture notes from sources such as books, handwritten notes or using voice transcription
• Add articles found on the internet to the knowledge vault
• Direct the Knowledge Agents to perform research and generate summarized versions of the research
• Retrieve Knowledge Assets by asking questions
• Retrieve word documents stored in the Knowledge Vault by asking questions relating to the subject being exploring
• Capture and store memories

Having the conversational user interface allows the user to interact in a simpler and more natural way that is possible through searching through file folders, trying to remember where certain documents were stored or scanning through websites to try to find a bit of knowledge related to an event or time that they recalled.

Capturing notes by using the voice mode is much simpler than having to open a laptop, open a note or app such as Microsoft word. When the user had a memory they wanted to capture, they would simply open chatGPT, and then dictate the note and ask the GPT to store it.

Rather than manually transcribing handwritten documents, the user is able to take a photograph of them, ask chatGPT to read the text and store it as a note. The same thing can be done with books or other physical media sources the user is seeking to reference in the context of writing.

This makes it quick and easy to capture critical memories or other traditionally difficult to capture bits of knowledge without having to use complex apps, ocr scanners, or even the need to open a computer. This is particularly important if a memory happens to come up during a conversation or other event that happens to trigger recall. The person is able to easily capture the memory in that moment.

Most importantly, the process is not impeded by the user needing to create a file name, or determine the appropriate location to store the note, article or file. The AI does this for them automatically.

With respect to the writing process, the user was able to retrieve the knowledge through interactions by asking questions in the GPT.

The Interaction triggers an action that uses GraphRAG type pattern that uses cosine similarity chunk retrieval to pull back the appropriate page from the document containing the chunk. The results are synthesized into a response, and the response is returned to chatGPT along with all the sources. The Action is conditioned to provide the Answer along with the sources in a nicely formatted list.

In the next post, I'll show an example of a full interaction using my version of the Personal Knowledge Vault and associated GPT that I use for capturing all the articles and knowledge I read on the subject of AI and Advanced Analytics. I also identified challenges, limitations and gaps during the testing process as well, which I will document in a subsequent post as well.

Stay Tuned!