vector database

_{scroll ↓ to Resources}

Note

• Vector databases are not databases, but search engines
• A vector database indexes and stores vector embedding, for fast search and optimized storage
• Provides the ability to compare multiple things (semantically) at the same time
• Helps machine learning models remember past data better, making them more useful for search, recommendations, and text generation
• Currently, all modern vector databases contain swiss-knife-level of instrument set to perform vector search, data storage, similarity measurement, reranking, etc.

Vendors

• See Vector databases | OpenAI Cookbook
  • Pinecone
  • Qdrant
  • Chroma
  • Faiss
  • LanceDB
  • Milvus
  • pgvector with Postgres
  • Azure-native
    • Azure AI search
    • Cosmos DB
    • Azure Databricks Mosaic AI Vector Search

How to choose a vector database

• see Resources
• support for role-based access control, multi-tenancy isolation
• having multiple embeddings per document
• algorithmic details
  • sparse algorithms beyond BM25: SPLADE
  • automatic switching between ANN and brute force search
    • can also be implemented manually in code using if n_docs<N: switch to full search
• self-hosted version
• Costs:
  • free tier
  • 50k\500k\1m… vectors with $d=1024$
• Performance:
  • latency
  • queries per second
• community and forward support (in case of open-source)

Resources

• Vector databases (1): What makes each one different? | The Data Quarry
• Vector databases (2): Understanding their internal…
• Vector databases (3): Not all indexes are created …
• Vector databases (4): Analyzing the trade-offs • T…
• A VectorDB Doesn’t Actually Work the Way You Think It Does
• What is a Vector Database? | A Comprehensive Vector Database Guide | Elastic
• Vector DB Comparison

---

Links to this File

table file.inlinks, filter(file.outlinks, (x) => !contains(string(x), ".jpg") AND !contains(string(x), ".pdf") AND !contains(string(x), ".png")) as "Outlinks" from [[]] and !outgoing([[]])  AND -"Changelog"