Amazon OpenSearch Service

Amazon OpenSearch is the memledger backend for agents that need hybrid search — vector recall ("things that look like this") plus BM25 lexical recall ("exact error code lookup") in a single query. memledger's OpenSearch backend signs every request with SigV4 using the IRSA-injected AWS credentials — no static API keys, no basic auth.

Validated end-to-end against a real OpenSearch domain — see Backend Validation.

When to choose OpenSearch over Aurora

OpenSearch wins for agents that need lexical recall on top of semantic recall — exact identifier / error-code / SKU lookup paired with vector similarity. Aurora + pgvector is the better default when your access pattern is purely semantic and you already operate Postgres.

Provisioning the domain

For production deployments:

Setting	Recommendation
Engine	OpenSearch 2.13 or newer
Deployment	Multi-AZ with standby
Instance type	m6g.large.search or larger for sustained agent workloads
Encryption	At-rest enabled, node-to-node TLS, HTTPS-only
Access policy	IAM-based (no fine-grained access control needed for SigV4)

Note: memledger's smoke validation runs against a t3.small.search single-AZ domain — fine for verification, not a production shape. Multi-AZ + m6g.large.search is the floor for sustained workloads.

SigV4 auth

memledger's OpenSearch backend signs every HTTP request with the agent's IRSA credentials. There is no API key, no master user, no users.yml.

The agent's IRSA role needs es:ESHttp* on the domain ARN:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["es:ESHttp*"],
    "Resource": [
      "arn:aws:es:<region>:<account-id>:domain/<your-opensearch-domain>/*"
    ]
  }]
}

The domain's own access policy must trust this role's ARN — typically:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::<account-id>:role/<your-irsa-role>"},
    "Action": "es:ESHttp*",
    "Resource": "arn:aws:es:<region>:<account-id>:domain/<your-opensearch-domain>/*"
  }]
}

SDK configuration

Set the OpenSearch endpoint via environment or backend config:

import os
from memledger import Memledger
from memledger.models import EmbeddingConfig

ml = await Memledger.create(
    backend_name="opensearch",
    backend_config={
        "endpoint": os.environ["OPENSEARCH_ENDPOINT"],   # https://...
        "region": "<your-region>",
        "index": "agent_memory",
    },
    embedding_config=EmbeddingConfig(
        provider="bedrock",
        model="amazon.titan-embed-text-v2:0",
        dimensions=1024,
    ),
)

Hybrid search

Hybrid search combines BM25 lexical scoring with k-NN vector similarity in a single OpenSearch query. Tune the relative weight at the call site:

results = await ml.search_hybrid(
    query="HikariCP connection pool exhausted",
    namespace="/ops/incidents",
    top_k=10,
    vector_weight=0.7,    # k-NN contribution (default 0.7)
    text_weight=0.3,      # BM25 contribution (default 0.3)
)

A higher vector_weight favors semantic similarity ("things that mean this"); a higher text_weight favors exact-token matches ("things that contain HikariCP"). For most agent workloads, the 0.7 / 0.3 default is a good starting point.

Index template

memledger creates the agent_memory index on first connection with an HNSW vector field sized for your embedding model:

Field	Type	Notes
embedding	knn_vector(1024)	HNSW; space_type=l2, m=16, ef_construction=512
content	text	Standard analyzer; powers BM25 in hybrid search
namespace	keyword	Filter dimension; never tokenized
Governance fields	keyword / boolean / date	confidence, hedged, created_by, workflow_id, etc.

To change HNSW parameters or analyzer, set them in backend_config["index_settings"] before the first connection — index settings on knn_vector are immutable post-create.

Embedding-dim alignment

Same constraint as Aurora: the knn_vector dimension is fixed at index-create time. Match your embedding model:

• Bedrock Titan v2 → dimensions=1024
• fastembed BAAI/bge-small → dimensions=384

Switching providers across a dim boundary requires reindexing into a new agent_memory index.

Next steps

• Aurora PostgreSQL — pure-vector alternative
• Bedrock — embeddings + LLM-as-judge
• kagent on EKS — full reference deployment
• Backend Validation — the smoke-test transcripts