Spring AI with RAG and Vector Search
Spring AI RAG vector search integration brings the power of retrieval-augmented generation to the Java ecosystem. Spring AI 1.0, released in early 2026, provides first-class support for connecting language models with your proprietary data through vector databases, making it straightforward to build AI features in existing Spring Boot applications. Crucially, it follows the conventions Spring developers already know — starters, auto-configuration, and dependency injection — so the learning curve sits on the RAG concepts rather than the plumbing.
This guide walks you through building a production RAG pipeline — from ingesting documents and generating embeddings to querying vector stores and augmenting LLM prompts with relevant context. If your team already runs Spring Boot in production, Spring AI is the fastest path to adding intelligent search and Q&A capabilities.
Understanding the RAG Architecture
RAG solves a fundamental limitation of language models: they only know what they were trained on. By retrieving relevant documents from your data and injecting them into the prompt, you get accurate answers grounded in your specific content. Equally important, RAG gives you a citation trail — because you know exactly which chunks fed the answer, you can show sources and let users verify claims, which is often a hard requirement in enterprise settings.
RAG Pipeline Flow:
1. INGESTION (offline)
Documents → Chunking → Embedding Model → Vector Database
2. RETRIEVAL (runtime)
User Query → Embedding → Vector Similarity Search → Top-K Documents
3. AUGMENTATION (runtime)
System Prompt + Retrieved Documents + User Query → LLM → Response
Setting Up Spring AI with PGVector
<!-- pom.xml dependencies -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-pgvector-store-spring-boot-starter</artifactId>
</dependency>
# application.yml
spring:
ai:
openai:
api-key: ${OPENAI_API_KEY}
chat:
options:
model: gpt-4o
temperature: 0.3
embedding:
options:
model: text-embedding-3-small
vectorstore:
pgvector:
dimensions: 1536
index-type: HNSW
distance-type: COSINE_DISTANCE
datasource:
url: jdbc:postgresql://localhost:5432/aiapp
Notice the `dimensions: 1536` value — it must exactly match the embedding model’s output size, or PGVector will reject inserts. The `text-embedding-3-small` model emits 1536-dimensional vectors, so the two stay in lockstep. Likewise, the HNSW index type trades a slower build and more memory for fast approximate nearest-neighbor search at query time, which is the right default for a read-heavy Q&A workload. Choosing the wrong distance metric is a common, silent mistake: cosine distance suits normalized text embeddings, so pairing `COSINE_DISTANCE` with OpenAI embeddings is the standard recommendation.
Document Ingestion Pipeline
@Service
public class DocumentIngestionService {
private final VectorStore vectorStore;
private final TokenTextSplitter textSplitter;
public DocumentIngestionService(VectorStore vectorStore) {
this.vectorStore = vectorStore;
this.textSplitter = new TokenTextSplitter(
800, // chunk size (tokens)
200, // overlap (tokens)
5, // min chunk size
10000, // max chunk size
true // keep separators
);
}
public void ingestDocuments(List<Resource> resources) {
var documentReader = new TikaDocumentReader(resources);
var documents = documentReader.get();
// Split into chunks with metadata
var chunks = textSplitter.apply(documents);
// Add custom metadata for filtering
chunks.forEach(chunk -> {
chunk.getMetadata().put("source", chunk.getMetadata()
.getOrDefault("source", "unknown"));
chunk.getMetadata().put("ingested_at",
Instant.now().toString());
});
// Store embeddings in PGVector
vectorStore.add(chunks);
log.info("Ingested {} chunks from {} documents",
chunks.size(), documents.size());
}
// Ingest from various sources
public void ingestPDF(String path) {
ingestDocuments(List.of(new FileSystemResource(path)));
}
public void ingestWebPage(String url) {
var reader = new JsoupDocumentReader(url);
var docs = textSplitter.apply(reader.get());
vectorStore.add(docs);
}
}
Why Chunk Size and Overlap Matter
The two numbers passed to `TokenTextSplitter` — chunk size and overlap — quietly determine retrieval quality more than almost any other knob. Chunks that are too large dilute the embedding: a 2,000-token chunk covering three subtopics produces a vector that represents none of them sharply, so similarity search misses. Conversely, chunks that are too small fragment a coherent idea across several entries, and the one chunk you retrieve lacks the surrounding context the LLM needs to answer. The 800-token size with 200-token overlap shown above is a sensible starting point for prose documentation; the overlap ensures a sentence that straddles a boundary still appears intact in at least one chunk. For highly structured content like API references or tables, splitting on natural boundaries (headings, rows) usually beats a fixed token count. In practice teams treat chunking as an empirical tuning problem and re-index when they change the strategy.
Building the RAG Query Service
@Service
public class RagQueryService {
private final ChatClient chatClient;
private final VectorStore vectorStore;
public RagQueryService(ChatClient.Builder builder,
VectorStore vectorStore) {
this.chatClient = builder
.defaultSystem("""
You are a helpful assistant that answers questions
based on the provided context. If the context doesn't
contain relevant information, say so honestly.
Always cite which document your answer comes from.
""")
.build();
this.vectorStore = vectorStore;
}
public String query(String userQuestion) {
// Retrieve relevant documents
var searchRequest = SearchRequest.query(userQuestion)
.withTopK(5)
.withSimilarityThreshold(0.7);
var relevantDocs = vectorStore.similaritySearch(searchRequest);
// Build context from retrieved documents
String context = relevantDocs.stream()
.map(doc -> "Source: %s
Content: %s".formatted(
doc.getMetadata().get("source"),
doc.getContent()))
.collect(Collectors.joining("
---
"));
// Augmented prompt with retrieved context
return chatClient.prompt()
.user(u -> u.text("""
Context:
{context}
Question: {question}
Answer based on the context above:
""")
.param("context", context)
.param("question", userQuestion))
.call()
.content();
}
}
The `withSimilarityThreshold(0.7)` call is your guard against the most embarrassing RAG failure mode: confidently answering from irrelevant context. When a user asks something your corpus simply doesn’t cover, vector search still returns its five nearest neighbors — they’re just not close. The threshold filters those out so the system prompt’s “say so honestly” instruction has a chance to fire, instead of the model hallucinating an answer from loosely related noise. Tune the threshold per corpus, because the absolute similarity scores depend on your embedding model and document style.
Advanced Patterns
Metadata Filtering
// Filter by metadata during retrieval
var searchRequest = SearchRequest.query(question)
.withTopK(5)
.withFilterExpression(new FilterExpressionBuilder()
.eq("department", "engineering")
.and(b -> b.gte("ingested_at", "2026-01-01"))
.build());
Metadata filtering is what separates a toy demo from a production system. In a real application, the same vector store holds documents that different users are allowed to see — engineering docs versus HR records, or one tenant’s data versus another’s. By attaching `department`, `tenant_id`, or access-control labels at ingestion time and filtering on them at query time, you enforce authorization inside the retrieval step itself, so a user can never be handed context they shouldn’t read. PGVector pushes these predicates into the SQL query, combining the metadata filter with the vector search efficiently rather than fetching everything and filtering in Java.
Streaming Responses
@GetMapping(value = "/api/ask", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamAnswer(@RequestParam String question) {
var context = retrieveContext(question);
return chatClient.prompt()
.user(buildPrompt(context, question))
.stream()
.content();
}
Streaming matters for perceived performance. A grounded answer over several retrieved documents can take a few seconds to generate fully, and users tolerate that far better when tokens appear progressively rather than after a long blank pause. Because Spring AI exposes the stream as a reactive `Flux` over server-sent events, it slots directly into Spring WebFlux, and the browser’s native `EventSource` consumes it without extra libraries. Just remember that retrieval still happens up front, synchronously — only the generation phase streams.
When NOT to Use RAG
RAG adds complexity and latency. Skip it when the LLM already knows the answer (general knowledge), when data changes faster than you can re-index, or when exact keyword search (Elasticsearch) is sufficient. Consequently, evaluate whether a simple prompt or fine-tuned model would serve your use case better. A common middle ground is hybrid search — combining vector similarity with traditional BM25 keyword matching — because pure vector search struggles with exact identifiers, product codes, and rare proper nouns that embeddings smear together. For a broader treatment of these architectural choices, see the related RAG architecture patterns guide and the vector database comparison.
Key Takeaways
Spring AI RAG brings intelligent search to Java applications with minimal boilerplate. PGVector integration means you can add vector search to your existing PostgreSQL database instead of standing up and operating a separate vector store — one fewer system to back up, monitor, and secure. As a result, start with a small document corpus, measure retrieval quality before tuning the LLM, and expand as you validate the approach with real users.
Related Reading
External Resources
In conclusion, the Spring AI RAG vector search stack is an essential capability for modern Java applications that must answer questions over proprietary data. By applying the chunking, thresholding, and metadata-filtering patterns covered in this guide, you can build more robust, scalable, and maintainable systems. Start with the fundamentals, iterate on your implementation, and continuously measure retrieval quality to ensure you are getting the most value from these approaches.