Stop Paying OpenAI for Embeddings — Self-Host BGE-M3 With TEI in 10 Minutes

You are paying OpenAI $0.13 per million tokens to embed your documents. Or Cohere $0.10. For a 50k-document corpus you re-embed every quarter, that is a recurring bill for work a single GPU can do faster. Text Embeddings Inference (TEI) from HuggingFace runs BGE-M3, BGE-large-en-v1.5, Nomic Embed v1.5, and 50+ other embedding models as a drop-in OpenAI-compatible HTTP service. One Docker command. Same API. 1/20th the cost. Higher throughput. Lower latency. Here is the recipe.

The Docker Command

docker run -d \
  --name tei \
  --restart unless-stopped \
  -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-embeddings-inference:1.7 \
  --model-id BAAI/bge-m3 \
  --port 80

That is the entire deploy. TEI pulls the model, compiles the kernel for your GPU (CPU fallback works), and starts an OpenAI-compatible HTTP server on port 8080. No Python virtualenv. No CUDA toolkit. No nginx. Single binary, single port, single model.

Smoke Test

curl -X POST http://localhost:8080/embed \
  -H "Content-Type: application/json" \
  -d '{"inputs": "The quick brown fox jumps over the lazy dog."}'

Returns a 1024-dim float array. Same shape as OpenAI's text-embedding-3-small. If your existing code calls client.embeddings.create(...), the swap is one base URL change.

The OpenAI-Compatible Client

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",  # TEI's OpenAI-compatible endpoint
    api_key="not-needed",  # local server, no auth
)
response = client.embeddings.create(
    input=["doc one", "doc two", "doc three"],
    model="BAAI/bge-m3",
)
vectors = [d.embedding for d in response.data]

That is the entire integration. RAG pipeline calling OpenAI today becomes local in two lines. No code rewrites. No new SDK.

Batch Embedding for Indexing

The single-request endpoint caps at 32 inputs per call. The /embed_batched route accepts hundreds per call — use it for the initial corpus index:

import httpx
def embed_corpus(docs: list[str], batch_size: int = 256) -> list[list[float]]:
    out = []
    with httpx.Client(timeout=60) as client:
        for i in range(0, len(docs), batch_size):
            chunk = docs[i:i+batch_size]
            r = client.post(
                "http://localhost:8080/embed_batched",
                json={"inputs": chunk},
            )
            r.raise_for_status()
            out.extend(r.json())
    return out

A 50,000-document corpus at 256 batch size is 196 round-trips. On an A10G, that finishes in 4 minutes. The same corpus against OpenAI's API costs $0.13 per million tokens, takes 18 minutes under their 3,000 req/min tier limit, and you eat every cent.

The Numbers

On a single A10G (24GB VRAM, ~$0.50/hr on RunPod or Vast.ai):

Faster. Cheaper. Beats OpenAI on MTEB. Handles 8192-token inputs (OpenAI caps at 8191). For multilingual corpora — Chinese, Japanese, code-switching — BGE-M3 crushes the OpenAI model because it was trained for it.

Gotchas

1. Pick your model to your workload. BGE-M3 is the default for multilingual + long context. For English-only short docs, use BAAI/bge-large-en-v1.5 — smaller, faster, slightly higher MTEB. For code, use nomic-ai/nomic-embed-code. TEI runs one model per container; spin up multiple for multi-model setups.

2. Watch VRAM. BGE-M3 needs ~6GB VRAM at inference. BGE-large needs ~2GB. An A10G (24GB) fits any of them with room for batch parallelism. A T4 (16GB) fits BGE-M3. A 3090 fits everything.

3. Supported model list only. TEI runs BGE, E5, GTE, Nomic, Stella, MXBAI out of the box. It does not run arbitrary transformers. Convert with optimum-export if yours is missing.

4. Truncation defaults to 512 tokens. Long-doc embeddings silently truncate. Pass --max-batch-tokens 8192 for BGE-M3 or your retrieval recall will silently drop. Measure before shipping.

5. No built-in auth. TEI ships an HTTP server with no auth. Put it behind nginx basic auth, a Cloudflare Tunnel, or Tailscale Funnel. Never expose it to the public internet unprotected.

The Take

Self-hosted embeddings are the easiest infrastructure win in your stack right now. One Docker command, one port, one model. The OpenAI-compatible API means your RAG pipeline swaps providers with a base URL change. Cost per million tokens drops 20x. Latency drops 5-10x. You own the model weights, the hardware, the data. The only reason to keep paying OpenAI for embeddings is convenience. After ten minutes of Docker, that argument evaporates.

— Mr. Technology