Alex Bloss
The realm of artificial intelligence is undergoing a thrilling transformation thanks to GPT4All . This comprehensive ecosystem is tailor-made for crafting and deploying personalized, large language models (LLMs). Unlike traditional setups reliant on potent Graphics Processing Units (GPUs) or cloud systems, GPT4All operates efficiently on regular computers equipped with standard CPUs (Central Processing Units) supporting Advanced Vector Extensions (AVX) or AVX2 instructions — essentially, your everyday laptop or PC! This marks a substantial departure from the previous reliance on specialized AI chips and cloud-based solutions.
The goal of GPT4All sounds simple, but it’s highly ambitious: to offer top-notch assistant-style language models to everyone, individuals and businesses alike, with no limitations on how they’re used or shared. These GPT4All models are compact (compared to their commercial siblings), ranging from 3GB to 8GB in size, and can easily fit into existing open-source software setups.
Nomic AI, the company responsible for GPT4All, ensures that this software ecosystem adheres to rigorous standards of quality and security. Additionally, the company aims to create a platform that empowers users to effortlessly train and deploy their own personalized on-edge LLMs.
Now, let’s delve deeper into the GPT4All ecosystem and its various components.
If you come across GPT4All for the first time, your attention might be drawn to the Chat Client, allowing you to effortlessly run LLMs on your local PC. However, GPT4All offers much more than that. It comprises a comprehensive ecosystem composed of various components.
The GPT4All ecosystem is open-source and includes:
In this first article, we will mainly focus on the GPT4All Chat Client, which is the most accessible component and offers a great way to get started with GPT4All. But be assured that this won’t be the last article about the fascinating possibilities the GPT4All ecosystem offers.
The installation of GPT4All Chat Client is a simple process, requiring only a few steps. The first step is to download the GPT4All installer for your platform.
After running the installer, you will be prompted to select the installation directory. The installer will automatically create the directory if it does not exist yet.
After the installation completes, you can start GPT4All by clicking on the Desktop icon.
If you want to run it on the console, just run the chat command in the bin-folder in the
installation directory.
In the UI you will be prompted to select the model you wish to use. The chosen model will be downloaded and installed automatically. You can find a complete list of the supported models and the current performance benchmarks on the GPT4All website (scroll down to “Performance Benchmarks”).
It’s crucial to note that these compact models can be sizable; for instance, the 4-bit quantized Falcon-Model necessitates a 3.8 GB download—patience is key.
Wait until the download is completed - then you should be able to use the model. Please note that sometimes, a downloaded model is not selectable right away - so you may have to restart GPT4All Chat to be able to use it.
After that you should be able to chat with the selected local model. Just type in your question and press enter. GPT4All will then generate an answer to your question.
You may notice that generating answers to your questions requires significant resources, which varies depending on the selected model. Thus, your patience will be essential. This provides an insight into the significant computational resources needed to generate responses to your queries using ChatGPT, which is powered by an extensive model like GPT-4. However, this won’t be apparent to you, as the computations are performed on highly specialized and more potent hardware located in the cloud.
Unfortunately, as of now, GPT4All does not support GPU usage. Hence, the performance and speed of the model primarily rely on the power of your CPU and the number of CPU cores you have set up in the GPT4All application settings.
Nevertheless, it is absolutely fascinating to see how well even the “minified” quantized models can answer your questions.
And it gets even better.
The LocalDocs Beta Plugin by GPT4All enables you to “chat with your documents”, i.e. you can ask questions and gain new insights in your data. What’s particularly elegant is that you have full control over which documents the model can access, and your data doesn’t leave your computer.
LocalDocs maintains an index of the documents in designated folders, allowing the language model to use snippets of documents to provide context when responding to your queries. However, it cannot answer general metadata queries or summarize a single document. The plugin supports various document-types, including txt, doc, docx, pdf, xls, xlsx, csv, ppt, and more.
To use this plugin, you should create a folder on your local computer containing the documents you want the language model to have access to during the chat session. Proceed to copy your documents into this designated folder. Subsequently, set up your folder as a collection by navigating to GPT4All settings, clicking on the ‘plugins’ section, and including the path to your local documents folder while specifying a name for the collection. Activate the ‘show references’ checkbox to visualize the document snippets utilized by the language model.
Don’t forget to activate the collection for the chat by clicking on the database icon and tick the checkbox next to the collections you want to use in your chat.
When you initiate a chat session with the selected LLM, GPT4All will use the documents in the selected collections as a knowledge base to answer your questions.
How does this work under the hood?
The approach employed by the LocalDocs plugin is known as Retrieval Augmented Generation, and it involves two key steps: retrieval and generation. Initially, the technique retrieves relevant information based on a given query, and then it employs this acquired information to craft a response.
In order to enhance the context of the LLM’s responses, LocalDocs extracts pertinent sections from each document within the activated collections, using the user’s current question as a guide. These extracted document chunks serve to impart the content knowledge from your documents, enabling the LLM to produce responses that are informative and insightful. The retrieval involves using “classical” pre-deep-learning n-gram and TF-IDF based methods to decide which document chunks should be used as context. This guarantees comparable quality with embedding based retrieval approaches but magnitudes faster to ingest data.
It is crucial to emphasize that although LocalDocs enables the LLM to incorporate additional local data without resorting to internet queries, the LLM might still generate responses that could be erroneous or “hallucinated,” drawn from its own internal knowledge corpus. So please check the generated content before using it for important tasks.
Chatting with your documents locally is a great feature, but wouldn’t it be even better if you could use your local LLM in your own applications? This is where GPT4All’s “Server Mode” comes in. It allows you to establish a local API endpoint dedicated to your personal LLM. This endpoint can be utilized for local software development and to integrate the LLM into your own applications.
To use this feature you enable the API-Server in the Application-Settings and GPT4All will
start a OpenAI-compatible API-endpoint on localhost:4891.
You can easily test the completions-service of the API using curl:
curl "http://localhost:4891/v1/completions" -H "Content-Type: application/json" \
-H "Authorization: Bearer any" \
-d '{ "model": "gpt4all-j-v1.3-groovy", "prompt": "Hello!", "max_tokens": 256, "temperature": 0.7 }'
To see which models are available simply run:
curl "http://localhost:4891/v1/models" -H "Content-Type: application/json" -Hmodels -H "Authorization: Bearer any"
You can use the API programmatically in your own applications, too, e.g. using a simple HTTPClient or a sophisticated framework like LangChain .
Please note: during our tests with LangChain, we often got empty responses from the API and we’re still investigating why this happened. In a future article we will show you how to use the API in your own applications.
As an added bonus, GPT4All offers Bindings for multiple programming languages. These enable you to seamlessly integrate LLMs into your own applications, harnessing the capabilities of local LLMs without the hassle to interact with any API endpoint. However, this topic deserves a dedicated article.
GPT4All is a promising project that aims to democratize the use of LLMs by making them available to everyone. The project is still in its infancy, but it is already very usable and shows great potential. The team behind GPT4All is very active and continuously improves the software. So, if you are interested in LLMs, you should definitely check out GPT4All.
Written By
Alex Bloss - Head of attempto-Lab. Creative Mind, Innovator, Critical Thinker, Developer, Hands-on Architect, CTO, Trend Scout and Curious Explorer with more than twenty years of experience.
Alex BlossWhat impact will AI have on the work of agile teams in software development? Will we be supplemented by tools or will Agent Smith take over? In the following article, we strive to get an overview of current developments and trends in AI and how agent-based workflows could change the way we work.
Adrien UngerDiscover how Angular 15's shift from class-based to functional route guards changes the game for developers, offering a more flexible approach to securing applications. However, this transition brings its own set of challenges, especially when it comes to unit testing.