‘Chat with RTX’ AI chatbot runs locally on your PC using Nvidia GPUs

“Chat with RTX uses retrieval-augmented generation (RAG), Nvidia TensorRT-LLM software and Nvidia RTX acceleration to bring generative AI capabilities to local, GeForce-powered Windows PCs. Users can quickly, easily connect local files on a PC as a dataset to an open-source large language model like Mistral or Llama 2, enabling queries for quick, contextually relevant answers,” reads the press release.

As the name suggests, Chat with RTX taps into the tensor cores on GeForce RTX 30 and 40 series graphics cards under the hood. This specialised hardware accelerates the intense matrix math operations required for neural network-based AI. Consequently, responses feel snappy compared to a cloud API.

Local processing also aids privacy since users’ data never leaves their PC. However, the initial download is substantial at 40GB due to all the AI model files. Additionally, the Python server uses around 3GB of RAM when active. Of course, since the model is run locally, the response times should also depend on the GPU model being used.

In testing, Chat with RTX proved capable of handling .txt, .pdf, .doc, and other formats. It can even digest YouTube captions to find mentions of search terms or produce video summaries. This functionality could bolster research and analysis workflows.

Despite its capabilities, Chat with RTX remains in the early stages. The tool lacks any context or memory between questions. There are also some odd quirks – it saves JSON files after indexing folders which could quickly clutter storage.

For now, Chat with RTX provides an exciting glimpse at the future of AI-assisted computing. As the software matures, local chatbots like this could become a staple way of augmenting personal devices.