While working at Sweet on the Incident feature—specifically a new capability around detection and response—we were building functionality for a customer who deals with a high volume of incidents. They needed better clarity into each incident, with richer context and more structured data to power advanced filtering and insights.

To support this, we started adding more detailed metadata to incidents—things like raw event payloads, detection signals, and enrichment data. That’s when I ran into a classic Elasticsearch issue:

illegal_argument_exception: Limit of total fields [1000] has been exceeded.

It happened because I was pushing full JSON objects into the documents, and Elasticsearch's dynamic mapping tried to index every nested key as a separate field. Each property—even ones we never planned to search on—counted toward the 1,000-field limit. Eventually, the index became unusable, and documents were rejected.

It was a frustrating issue to debug at first, but also a valuable lesson: when working with complex or nested data, it's important to define mappings explicitly and disable dynamic mapping for parts of the document that don't need to be individually indexed. That change helped us scale the incident data model without running into Elasticsearch's field limits—and ultimately gave the customer the clarity they needed. 

But why did this happen in the first place? To understand the cause - and how to prevent it - let's look at the main subject of this post: index mappings. Think of mappings as Elasticsearch's version of a database schema. Just like a relational database has tables with defined columns and data types, Elasticsearch uses mappings to control the structure and types of your fields. This schema-like approach gives you fine-grained control over how data is stored and searched, but it can also introduce pitfalls if you let it grow unchecked. Let's dive into what mappings are, why they matter, and how to avoid blowing up your cluster with too many fields.

What Is an Index Mapping?

In Elasticsearch, an index mapping is like an SQL schema - it defines:

• What fields exist
• The [data types] (`text`, `keyword`, `date`, `integer`, etc.)
• Analyzers and search behavior

Mappings help Elasticsearch store and search data efficiently. Even if you never explicitly define one, Elasticsearch will create one for you. This automatic mapping creation is what enables the dynamic behavior we'll explore next, but it also introduces the potential for the "too many fields" problem.

Automatic (Dynamic) Mapping

By default, mappings are created dynamically.[1]  

That means whenever you send a new JSON document to an index, Elasticsearch will inspect its structure and infer the appropriate types for each new field. For example, if you send a document like this:

{  "user": "Lior",  "createdAt": "2025-06-20"}

Elasticsearch will do the following under the hood:

• Treat the `user` field as a `text` field and also add a `keyword` subfield, allowing you to both search its contents and perform exact matches or aggregations.
• Treat the `createdAt` field as a `date`, so it can be sorted and used in range queries.

This hands-off behavior is incredibly convenient - especially when you want to iterate quickly or explore data without worrying about schema upfront.

However, this convenience comes at a cost. If your data contains unpredictable or highly variable shapes - for example, if you're pushing entire nested JSON objects into one field - every new key that appears will be automatically added to the mapping. Imagine one document has a user object like:

"user": { "firstName": "Lior", "lastName": "Pollak" }

And then the next one has:

"user": { "firstName": "Lior", "lastName": "Pollak", "middleName": "Amit" }

Suddenly, a new field (`middleName`) is created in the mapping. Over time, as variations pile up, the number of distinct fields grows - and so does the mapping size.

When this happens repeatedly at scale - especially if you ingest user-generated or unstructured data - your index can accumulate hundreds or thousands of unique fields before you realize it. This can lead to a nasty surprise when you hit the `index.mapping.total_fields.limit` (default is 1,000), and further document indexing fails.

That's why dynamic mapping is a double-edged sword. It can speed up early-stage development or work well for stable, well-known data shapes. But without guardrails like explicit mappings or disabled dynamic mapping for certain paths, it can lead to mapping explosion, reduced performance, and index failures.

Manually Creating Mappings

While dynamic mapping is great for getting up and running quickly, explicitly defining your mappings up front is the safest way to control your data structure and prevent surprises. This is even recommended by elastic themselves for production use cases.[1]  

With manual mappings, you declare exactly which fields your documents will have and what types they should be. This lets you plan your index design for long-term stability.

Here's a practical example in TypeScript using the official `@elastic/elasticsearch` client:

import { Client } from "@elastic/elasticsearch";‍
const client = new Client({ node: "http://localhost:9200" });‍
await client.indices.create({  
	index: "my_index",  
    body: {    
    	mappings: {      
        	properties: {        
            	user: { type: "keyword" },        
                createdAt: { type: "date" },        
                message: { type: "text" },      
            },    
        }, 	
    },
});

Here, we explicitly control:

• The field names (`user`, `createdAt`, `message`)
• The field types (`keyword`, `date`, `text`)
• How those fields will be handled for searching and sorting

By creating this mapping manually, you gain several clear advantages:

• Fields and types are explicit - your team and tools know what to expect.
• No uncontrolled dynamic mappings - you won't accumulate unpredictable fields as data shapes change.
• You won't accidentally exceed field limits - the index will only accept the fields you define, preventing mapping explosion.
• Easier optimization - you can choose the most efficient types (e.g. using keyword instead of text if you only need exact matches).
• Clear upgrade paths - when you need to add new fields, you do it consciously by updating the mapping.

And while this may feel more rigid at first, explicit mappings pay off as your data scales and your queries become more complex. They also help ensure consistency across different indices or environments - making your entire Elasticsearch setup easier to maintain over time.

Note: Once a field is mapped in an index, its type cannot be changed - you can only add new fields. If you need to change the type of an existing field, you must create a new index and reindex your data. If you want to change the name of an existing field, you need to use an [`alias`] field to create an alternate field name, since renaming a field would invalidate data already indexed under the old field name.

Conclusion

Index mappings give Elasticsearch its speed and search power - but they require careful control. Getting them right up front can save you a ton of headaches as your data grows.

Here's a quick overview of the trade-offs:

Feature   --> Benefit  --> Limitation   

• Explicit mappings --> Predictable types and search behavior --> Requires up-front planning 
• Dynamic mappings  --> Easy to prototype and ingest data quickly, up-and-running format --> Can cause index bloat and too-many-fields

As you can see, explicit mappings offer the most control and scalability, especially in production systems where stability is key. Dynamic mappings can be handy during early development or prototyping - just remember that they may introduce surprises if left unchecked.

When starting a new project:

• Design mappings up front for stable and scalable indices.
• Use flattened types if you need to store highly dynamic, semi-structured data without exploding your mappings.
• Avoid letting dynamic mapping run amok - uncontrolled field creation can hit the `index.mapping.total_fields.limit` and cause indexing errors.

By keeping these considerations in mind, you'll make sure your indices stay lean, mean, and lightning-fast⚡- and you'll spend more time building features and less time firefighting mapping errors.

[`alias`]: https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/field-alias

‍

This blog was originally posted on Lior Pollak’s blog and features feedback from Shahaf Cohen. To learn more about the author, check out Lior’s Linkedin.