RAG & Vector Search Integration

Enterprise AI-native mobile applications implement Retrieval-Augmented Generation (RAG) to provide grounded, factual responses based on private user documents or domain knowledge. RAG combines Vector Embeddings, Vector Databases (sqlite-vec, ObjectBox, or cloud stores), and LLM prompt augmentation.

---

1. Retrieval-Augmented Generation (RAG) Pipeline

 [ DOCUMENT INGESTION ]                                [ USER QUERY RAG PIPELINE ]
┌───────────────────────┐                             ┌───────────────────────┐
│ User PDF / Documents  │                             │ User Query ("Specs?") │
└───────────┬───────────┘                             └───────────┬───────────┘
            │                                                     │
            ▼ Chunk Text                                          ▼ Generate Query Embedding
┌───────────────────────┐                             ┌───────────────────────┐
│ Text Chunks (500 char)│                             │ Vector [0.12, -0.45..]│
└───────────┬───────────┘                             └───────────┬───────────┘
            │                                                     │
            ▼ Embedding API                                       ▼ Vector Search (Cosine Similarity)
┌───────────────────────┐                             ┌───────────────────────┐
│ Vector Database       │ ◄────────────────────────── │ Nearest Text Chunks   │
│ (sqlite-vec / ObjectBox)                            └───────────┬───────────┘
└───────────────────────┘                                         │
                                                                  ▼ Augment Prompt
                                                      ┌───────────────────────┐
                                                      │ LLM Prompt + Context  │ ──► Fact-Grounded Response
                                                      └───────────────────────┘

---

2. Generating & Searching Vector Embeddings

1. Vector Embeddings

A Vector Embedding represents semantic text meaning as a high-dimensional array of floating-point numbers (e.g. 768 dimensions for Gemini text-embedding-004). Mathematically, words with similar meanings reside near each other in vector space.

2. Cosine Similarity Formula

Measures the cosine of the angle between two vector embeddings ($A$ and $B$):

CosineSimilarity(A, B) = (A . B) / (||A|| * ||B||)

• A value of 1.0 represents identical semantic meaning; 0.0 represents zero semantic similarity.

double calculateCosineSimilarity(List<double> vectorA, List<double> vectorB) {
  double dotProduct = 0.0;
  double normA = 0.0;
  double normB = 0.0;

  for (int i = 0; i < vectorA.length; i++) {
    dotProduct += vectorA[i] * vectorB[i];
    normA += vectorA[i] * vectorA[i];
    normB += vectorB[i] * vectorB[i];
  }

  return dotProduct / (sqrt(normA) * sqrt(normB));
}

---

3. On-Device vs. Cloud Vector Databases

Feature	On-Device Vector Search (sqlite-vec, ObjectBox)	Cloud Vector DB (Pinecone, Qdrant, Vertex AI)
Privacy	100% On-Device (Private documents stay local)	Transmitted over network
Offline Support	100% Offline	Online network required
Scale Limit	~50,000 document vectors (Mobile RAM/Flash)	Billions of vectors
Latency	Sub-5ms search	50ms - 200ms network latency

---

4. Prompt Context Augmentation

Once relevant document chunks are retrieved from the vector store, inject them into the LLM prompt context:

Future<String> answerUserQueryWithRAG(String query) async {
  // 1. Generate embedding for query
  final queryVector = await embeddingService.getEmbedding(query);

  // 2. Retrieve top-3 nearest text chunks from local vector store
  final List<String> retrievedChunks = await vectorStore.searchNearest(queryVector, limit: 3);

  // 3. Construct Augmented Prompt
  final String augmentedPrompt = '''
You are a helpful assistant. Answer the user's question using ONLY the provided context below.
If the context does not contain the answer, reply "I do not have enough information to answer."

--- CONTEXT ---
${retrievedChunks.join('\n\n')}

--- USER QUESTION ---
$query
''';

  // 4. Generate grounded LLM response
  final response = await geminiModel.generateContent([Content.text(augmentedPrompt)]);
  return response.text!;
}

---

5. Trade-offs & Production Considerations

• Chunking Strategy: Chunking text too coarsely (e.g. 5,000 characters per chunk) dilutes embedding precision. Chunking too finely (e.g. 50 characters) loses context. Ideal chunk size for mobile RAG is 300-800 characters with 10% overlap.
• Vector Index Overhead: Storing 768-dimensional float vectors consumes memory (~3KB per vector). Indexing 10,000 documents consumes ~30MB of storage. Use quantization for local vector stores.

// Augmented RAG Prompt Structure
final prompt = 'Context:\n$relevantChunks\n\nQuestion: $query';

// ❌ Error: Anti-Pattern: Dumping a 100-page raw PDF text into an LLM prompt exhausts token limits and increases costs!

// ✅ Safe: Safe Pattern: Chunk document, store vector embeddings, and retrieve top-3 relevant chunks via RAG