IBM Granite + AnythingLLM local RAG on a laptop
IBM's Granite community recipe for running a basic local RAG over a PDF (e.g. a budget document) on your laptop.
Solo2-10 peopleDocument search / chat with filesUse existing hardwareSensitivity: High (client/confidential)
Hardware
| Item | Approx. price (as of 2026-10-16) |
|---|---|
| Existing laptop | Not stated in source |
Source prices are as stated by the source. Retailer offers: prices checked 2026-10-16, may change. Some links may be affiliate links; we may earn a commission at no extra cost to you. This never changes what we list.
Software stack
- Ollama
- AnythingLLM
Models used
- granite4.2:8b
Good at
- Clear screenshots-driven tutorial
- Shows before/after effect of adding documents
Weak at
- Recipe notes answers need prompt engineering to get exactly what you want
Rough total cost
Uses existing hardware; software is free.
Getting started
- Install Ollama and pull granite4.2:8b.
- Install AnythingLLM and select Ollama + the Granite model under LLM settings.
- Create a new workspace.
- Upload your PDF, move it to the workspace and click Save and Embed.
- Ask questions and refine prompts.
Source & credit
How to use Granite/AnythingLLM/Ollama for a Basic Local RAG — IBM Granite Community (GitHub)
Summarized from the source; quotes are the source author's words. Check the original before buying.