Elasticsearch

--- name: Elasticsearch description: Query and index Elasticsearch with proper mappings, analyzers, and search patterns. metadata: {"clawdbot":{"emoji":"🔍","requires":{"anyBins":["curl"]},"os":["linux","darwin","win32"]}} ---

Mapping Mistakes

• Always define explicit mappings—dynamic mapping guesses wrong (first "123" makes field integer, later "abc" fails)
• text for full-text search, keyword for exact match/aggregations—using text for IDs breaks filters
• Can't change field type after indexing—must reindex to new index with correct mapping
• Set dynamic: "strict" to reject unmapped fields—catches typos in field names

Text vs Keyword

• text is analyzed (tokenized, lowercased)—"Quick Brown" matches search for "quick"
• keyword is exact bytes—"Quick Brown" only matches exactly "Quick Brown"
• Need both? Use multi-field: "title": { "type": "text", "fields": { "raw": { "type": "keyword" }}}
• Sort/aggregate on title.raw, search on title

Query vs Filter Context

• Query context calculates relevance score—expensive, use for search ranking
• Filter context is yes/no—cacheable, use for exact conditions (status, date ranges)
• Combine: bool.must for scoring, bool.filter for filtering without scoring
• Range queries on dates/numbers almost always belong in filter, not query

Analyzers

• standard analyzer lowercases and removes punctuation—fine for most text
• keyword analyzer keeps exact string—use for codes, SKUs, emails
• Language analyzers (english) stem words—"running" matches "run"
• Test analyzer with _analyze endpoint before indexing—surprises in production hurt

Nested vs Object

• Object type flattens arrays—{"tags": [{"key":"a","val":1}, {"key":"b","val":2}]} becomes tags.key: [a,b], tags.val: [1,2]
• Flattened loses association—query key=a AND val=2 incorrectly matches above
• Use nested type to preserve object boundaries—requires nested query wrapper
• Nested is expensive—avoid for high-cardinality arrays

Pagination Traps

• from + size limited to 10,000 hits—deep pagination fails
• search_after for deep pagination—requires consistent sort, typically _id
• Scroll API for bulk export—keeps point-in-time view, but ties up resources
• Don't use scroll for user pagination—search_after is correct choice

Bulk Operations

• Never index documents one-by-one—use _bulk API, 5-15MB batches
• Bulk format: newline-delimited JSON, action line then document line
• Check response for partial failures—bulk can succeed overall with individual doc errors
• Set refresh=false during bulk loads—refresh after batch completes

Performance

• _source: false with stored_fields if you don't need full document—reduces I/O
• Use filter for cacheable conditions—Elasticsearch caches filter results
• Avoid leading wildcards (*term)—forces full scan; use reverse field for suffix search
• profile: true shows query execution breakdown—find slow clauses

Sharding

• Shard size 10-50GB optimal—too small = overhead, too large = slow recovery
• Number of shards fixed at creation—can't reshard without reindexing
• Replicas for read throughput and availability—set based on query load
• Start with 1 shard for small indices—over-sharding kills performance

Index Management

• Use index templates—new indices get consistent mappings and settings
• Use aliases for zero-downtime reindexing—point alias to new index after reindex
• ILM (Index Lifecycle Management) for time-series—auto-rollover, delete old indices
• Close unused indices to free memory—closed index uses no heap

Aggregations

• terms agg needs keyword field—text fields fail or give garbage
• Default size: 10 on terms agg—increase to get all buckets, or use composite
• Cardinality is approximate (HyperLogLog)—exact count requires scanning all docs
• Nested aggs require nested wrapper—matches nested query pattern

Common Errors

• "cluster_block_exception"—disk > 85%, cluster goes read-only; clear disk, reset with _cluster/settings
• "version conflict"—concurrent update; retry with retry_on_conflict or use optimistic locking
• "circuit_breaker_exception"—query uses too much memory; reduce aggregation scope
• Mapping explosion from dynamic fields—set index.mapping.total_fields.limit and use strict mapping