442 lines
15 KiB
Markdown
442 lines
15 KiB
Markdown
# Elasticsearch Output Plugin
|
|
|
|
This plugin writes metrics to [Elasticsearch][elasticsearch] via HTTP using the
|
|
[Elastic client library][client_lib]. The plugin supports Elasticsearch
|
|
releases from v5.x up to v7.x.
|
|
|
|
⭐ Telegraf v0.1.5
|
|
🏷️ datastore, logging
|
|
💻 all
|
|
|
|
[elasticsearch]: https://www.elastic.co
|
|
[client_lib]: http://olivere.github.io/elastic/
|
|
|
|
## Elasticsearch indexes and templates
|
|
|
|
### Indexes per time-frame
|
|
|
|
This plugin can manage indexes per time-frame, as commonly done in other tools
|
|
with Elasticsearch.
|
|
|
|
The timestamp of the metric collected will be used to decide the index
|
|
destination.
|
|
|
|
For more information about this usage on Elasticsearch, check [the
|
|
docs][1].
|
|
|
|
[1]: https://www.elastic.co/guide/en/elasticsearch/guide/master/time-based.html#index-per-timeframe
|
|
|
|
### Template management
|
|
|
|
Index templates are used in Elasticsearch to define settings and mappings for
|
|
the indexes and how the fields should be analyzed. For more information on how
|
|
this works, see [the docs][2].
|
|
|
|
This plugin can create a working template for use with telegraf metrics. It uses
|
|
Elasticsearch dynamic templates feature to set proper types for the tags and
|
|
metrics fields. If the template specified already exists, it will not overwrite
|
|
unless you configure this plugin to do so. Thus you can customize this template
|
|
after its creation if necessary.
|
|
|
|
Example of an index template created by telegraf on Elasticsearch 5.x:
|
|
|
|
```json
|
|
{
|
|
"order": 0,
|
|
"template": "telegraf-*",
|
|
"settings": {
|
|
"index": {
|
|
"mapping": {
|
|
"total_fields": {
|
|
"limit": "5000"
|
|
}
|
|
},
|
|
"auto_expand_replicas" : "0-1",
|
|
"codec" : "best_compression",
|
|
"refresh_interval": "10s"
|
|
}
|
|
},
|
|
"mappings": {
|
|
"_default_": {
|
|
"dynamic_templates": [
|
|
{
|
|
"tags": {
|
|
"path_match": "tag.*",
|
|
"mapping": {
|
|
"ignore_above": 512,
|
|
"type": "keyword"
|
|
},
|
|
"match_mapping_type": "string"
|
|
}
|
|
},
|
|
{
|
|
"metrics_long": {
|
|
"mapping": {
|
|
"index": false,
|
|
"type": "float"
|
|
},
|
|
"match_mapping_type": "long"
|
|
}
|
|
},
|
|
{
|
|
"metrics_double": {
|
|
"mapping": {
|
|
"index": false,
|
|
"type": "float"
|
|
},
|
|
"match_mapping_type": "double"
|
|
}
|
|
},
|
|
{
|
|
"text_fields": {
|
|
"mapping": {
|
|
"norms": false
|
|
},
|
|
"match": "*"
|
|
}
|
|
}
|
|
],
|
|
"_all": {
|
|
"enabled": false
|
|
},
|
|
"properties": {
|
|
"@timestamp": {
|
|
"type": "date"
|
|
},
|
|
"measurement_name": {
|
|
"type": "keyword"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"aliases": {}
|
|
}
|
|
|
|
```
|
|
|
|
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
|
|
|
|
### Example events
|
|
|
|
This plugin will format the events in the following way:
|
|
|
|
```json
|
|
{
|
|
"@timestamp": "2017-01-01T00:00:00+00:00",
|
|
"measurement_name": "cpu",
|
|
"cpu": {
|
|
"usage_guest": 0,
|
|
"usage_guest_nice": 0,
|
|
"usage_idle": 71.85413456197966,
|
|
"usage_iowait": 0.256805341656516,
|
|
"usage_irq": 0,
|
|
"usage_nice": 0,
|
|
"usage_softirq": 0.2054442732579466,
|
|
"usage_steal": 0,
|
|
"usage_system": 15.04879301548127,
|
|
"usage_user": 12.634822807288275
|
|
},
|
|
"tag": {
|
|
"cpu": "cpu-total",
|
|
"host": "elastichost",
|
|
"dc": "datacenter1"
|
|
}
|
|
}
|
|
```
|
|
|
|
```json
|
|
{
|
|
"@timestamp": "2017-01-01T00:00:00+00:00",
|
|
"measurement_name": "system",
|
|
"system": {
|
|
"load1": 0.78,
|
|
"load15": 0.8,
|
|
"load5": 0.8,
|
|
"n_cpus": 2,
|
|
"n_users": 2
|
|
},
|
|
"tag": {
|
|
"host": "elastichost",
|
|
"dc": "datacenter1"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Timestamp Timezone
|
|
|
|
Elasticsearch documents use RFC3339 timestamps, which include timezone
|
|
information (for example `2017-01-01T00:00:00-08:00`). By default, the Telegraf
|
|
system's configured timezone will be used.
|
|
|
|
However, this may not always be desirable: Elasticsearch preserves timezone
|
|
information and includes it when returning associated documents. This can cause
|
|
issues for some pipelines. In particular, those that do not parse retrieved
|
|
timestamps and instead assume that the timezone returned will always be
|
|
consistent.
|
|
|
|
Telegraf honours the timezone configured in the environment variable `TZ`, so
|
|
the timezone sent to Elasticsearch can be amended without needing to change the
|
|
timezone configured in the host system:
|
|
|
|
```sh
|
|
export TZ="America/Los_Angeles"
|
|
export TZ="UTC"
|
|
```
|
|
|
|
If Telegraf is being run as a system service, this can be configured in the
|
|
following way on Linux:
|
|
|
|
```sh
|
|
echo TZ="UTC" | sudo tee -a /etc/default/telegraf
|
|
```
|
|
|
|
## OpenSearch Support
|
|
|
|
OpenSearch is a fork of Elasticsearch hosted by AWS. The OpenSearch server will
|
|
report itself to clients with an AWS specific-version (e.g. v1.0). In reality,
|
|
the actual underlying Elasticsearch version is v7.1. This breaks Telegraf and
|
|
other Elasticsearch clients that need to know what major version they are
|
|
interfacing with.
|
|
|
|
Amazon has created a [compatibility mode][3] to allow existing Elasticsearch
|
|
clients to properly work when the version needs to be checked. To enable
|
|
compatibility mode users need to set the `override_main_response_version` to
|
|
`true`.
|
|
|
|
On existing clusters run:
|
|
|
|
```json
|
|
PUT /_cluster/settings
|
|
{
|
|
"persistent" : {
|
|
"compatibility.override_main_response_version" : true
|
|
}
|
|
}
|
|
```
|
|
|
|
And on new clusters set the option to true under advanced options:
|
|
|
|
```json
|
|
POST https://es.us-east-1.amazonaws.com/2021-01-01/opensearch/upgradeDomain
|
|
{
|
|
"DomainName": "domain-name",
|
|
"TargetVersion": "OpenSearch_1.0",
|
|
"AdvancedOptions": {
|
|
"override_main_response_version": "true"
|
|
}
|
|
}
|
|
```
|
|
|
|
[3]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html#rename-upgrade
|
|
|
|
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
|
|
|
|
In addition to the plugin-specific configuration settings, plugins support
|
|
additional global and plugin configuration settings. These settings are used to
|
|
modify metrics, tags, and field or create aliases and configure ordering, etc.
|
|
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
|
|
|
|
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
|
|
|
|
## Secret-store support
|
|
|
|
This plugin supports secrets from secret-stores for the `username`,
|
|
`password` and `auth_bearer_token` option.
|
|
See the [secret-store documentation][SECRETSTORE] for more details on how
|
|
to use them.
|
|
|
|
[SECRETSTORE]: ../../../docs/CONFIGURATION.md#secret-store-secrets
|
|
|
|
## Configuration
|
|
|
|
```toml @sample.conf
|
|
# Configuration for Elasticsearch to send metrics to.
|
|
[[outputs.elasticsearch]]
|
|
## The full HTTP endpoint URL for your Elasticsearch instance
|
|
## Multiple urls can be specified as part of the same cluster,
|
|
## this means that only ONE of the urls will be written to each interval
|
|
urls = [ "http://node1.es.example.com:9200" ] # required.
|
|
## Elasticsearch client timeout, defaults to "5s" if not set.
|
|
timeout = "5s"
|
|
## Set to true to ask Elasticsearch a list of all cluster nodes,
|
|
## thus it is not necessary to list all nodes in the urls config option
|
|
enable_sniffer = false
|
|
## Set to true to enable gzip compression
|
|
enable_gzip = false
|
|
## Set the interval to check if the Elasticsearch nodes are available
|
|
## Setting to "0s" will disable the health check (not recommended in production)
|
|
health_check_interval = "10s"
|
|
## Set the timeout for periodic health checks.
|
|
# health_check_timeout = "1s"
|
|
## HTTP basic authentication details.
|
|
## HTTP basic authentication details
|
|
# username = "telegraf"
|
|
# password = "mypassword"
|
|
## HTTP bearer token authentication details
|
|
# auth_bearer_token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
|
|
|
|
## Index Config
|
|
## The target index for metrics (Elasticsearch will create if it not exists).
|
|
## You can use the date specifiers below to create indexes per time frame.
|
|
## The metric timestamp will be used to decide the destination index name
|
|
# %Y - year (2016)
|
|
# %y - last two digits of year (00..99)
|
|
# %m - month (01..12)
|
|
# %d - day of month (e.g., 01)
|
|
# %H - hour (00..23)
|
|
# %V - week of the year (ISO week) (01..53)
|
|
## Additionally, you can specify a tag name using the notation {{tag_name}}
|
|
## which will be used as part of the index name. If the tag does not exist,
|
|
## the default tag value will be used.
|
|
# index_name = "telegraf-{{host}}-%Y.%m.%d"
|
|
# default_tag_value = "none"
|
|
index_name = "telegraf-%Y.%m.%d" # required.
|
|
|
|
## Optional Index Config
|
|
## Set to true if Telegraf should use the "create" OpType while indexing
|
|
# use_optype_create = false
|
|
|
|
## Optional TLS Config
|
|
# tls_ca = "/etc/telegraf/ca.pem"
|
|
# tls_cert = "/etc/telegraf/cert.pem"
|
|
# tls_key = "/etc/telegraf/key.pem"
|
|
## Use TLS but skip chain & host verification
|
|
# insecure_skip_verify = false
|
|
|
|
## Template Config
|
|
## Set to true if you want telegraf to manage its index template.
|
|
## If enabled it will create a recommended index template for telegraf indexes
|
|
manage_template = true
|
|
## The template name used for telegraf indexes
|
|
template_name = "telegraf"
|
|
## Set to true if you want telegraf to overwrite an existing template
|
|
overwrite_template = false
|
|
## If set to true a unique ID hash will be sent as sha256(concat(timestamp,measurement,series-hash)) string
|
|
## it will enable data resend and update metric points avoiding duplicated metrics with different id's
|
|
force_document_id = false
|
|
|
|
## Specifies the handling of NaN and Inf values.
|
|
## This option can have the following values:
|
|
## none -- do not modify field-values (default); will produce an error if NaNs or infs are encountered
|
|
## drop -- drop fields containing NaNs or infs
|
|
## replace -- replace with the value in "float_replacement_value" (default: 0.0)
|
|
## NaNs and inf will be replaced with the given number, -inf with the negative of that number
|
|
# float_handling = "none"
|
|
# float_replacement_value = 0.0
|
|
|
|
## Pipeline Config
|
|
## To use a ingest pipeline, set this to the name of the pipeline you want to use.
|
|
# use_pipeline = "my_pipeline"
|
|
## Additionally, you can specify a tag name using the notation {{tag_name}}
|
|
## which will be used as part of the pipeline name. If the tag does not exist,
|
|
## the default pipeline will be used as the pipeline. If no default pipeline is set,
|
|
## no pipeline is used for the metric.
|
|
# use_pipeline = "{{es_pipeline}}"
|
|
# default_pipeline = "my_pipeline"
|
|
#
|
|
# Custom HTTP headers
|
|
# To pass custom HTTP headers please define it in a given below section
|
|
# [outputs.elasticsearch.headers]
|
|
# "X-Custom-Header" = "custom-value"
|
|
|
|
## Template Index Settings
|
|
## Overrides the template settings.index section with any provided options.
|
|
## Defaults provided here in the config
|
|
# template_index_settings = {
|
|
# refresh_interval = "10s",
|
|
# mapping.total_fields.limit = 5000,
|
|
# auto_expand_replicas = "0-1",
|
|
# codec = "best_compression"
|
|
# }
|
|
```
|
|
|
|
### Permissions
|
|
|
|
If you are using authentication within your Elasticsearch cluster, you need to
|
|
create a account and create a role with at least the manage role in the Cluster
|
|
Privileges category. Otherwise, your account will not be able to connect to
|
|
your Elasticsearch cluster and send logs to your cluster. After that, you need
|
|
to add "create_indice" and "write" permission to your specific index pattern.
|
|
|
|
### Required parameters
|
|
|
|
* `urls`: A list containing the full HTTP URL of one or more nodes from your
|
|
Elasticsearch instance.
|
|
* `index_name`: The target index for metrics. You can use the date specifiers
|
|
below to create indexes per time frame.
|
|
|
|
``` %Y - year (2017)
|
|
%y - last two digits of year (00..99)
|
|
%m - month (01..12)
|
|
%d - day of month (e.g., 01)
|
|
%H - hour (00..23)
|
|
%V - week of the year (ISO week) (01..53)
|
|
```
|
|
|
|
Additionally, you can specify dynamic index names by using tags with the
|
|
notation ```{{tag_name}}```. This will store the metrics with different tag
|
|
values in different indices. If the tag does not exist in a particular metric,
|
|
the `default_tag_value` will be used instead.
|
|
|
|
### Optional parameters
|
|
|
|
* `timeout`: Elasticsearch client timeout, defaults to "5s" if not set.
|
|
* `enable_sniffer`: Set to true to ask Elasticsearch a list of all cluster
|
|
nodes, thus it is not necessary to list all nodes in the urls config option.
|
|
* `health_check_interval`: Set the interval to check if the nodes are available,
|
|
in seconds. Setting to 0 will disable the health check (not recommended in
|
|
production).
|
|
* `username`: The username for HTTP basic authentication details (eg. when using
|
|
Shield).
|
|
* `password`: The password for HTTP basic authentication details (eg. when using
|
|
Shield).
|
|
* `manage_template`: Set to true if you want telegraf to manage its index
|
|
template. If enabled it will create a recommended index template for telegraf
|
|
indexes.
|
|
* `template_name`: The template name used for telegraf indexes.
|
|
* `overwrite_template`: Set to true if you want telegraf to overwrite an
|
|
existing template.
|
|
* `force_document_id`: Set to true will compute a unique hash from as
|
|
sha256(concat(timestamp,measurement,series-hash)),enables resend or update
|
|
data without ES duplicated documents.
|
|
* `float_handling`: Specifies how to handle `NaN` and infinite field
|
|
values. `"none"` (default) will do nothing, `"drop"` will drop the field and
|
|
`replace` will replace the field value by the number in
|
|
`float_replacement_value`
|
|
* `float_replacement_value`: Value (defaulting to `0.0`) to replace `NaN`s and
|
|
`inf`s if `float_handling` is set to `replace`. Negative `inf` will be
|
|
replaced by the negative value in this number to respect the sign of the
|
|
field's original value.
|
|
* `use_optype_create`: If set, the "create" operation type will be used when
|
|
indexing into Elasticsearch, which is needed when using the Elasticsearch
|
|
data streams feature.
|
|
* `use_pipeline`: If set, the set value will be used as the pipeline to call
|
|
when sending events to elasticsearch. Additionally, you can specify dynamic
|
|
pipeline names by using tags with the notation ```{{tag_name}}```. If the tag
|
|
does not exist in a particular metric, the `default_pipeline` will be used
|
|
instead.
|
|
* `default_pipeline`: If dynamic pipeline names the tag does not exist in a
|
|
particular metric, this value will be used instead.
|
|
* `headers`: Custom HTTP headers, which are passed to Elasticsearch header
|
|
before each request.
|
|
|
|
## Known issues
|
|
|
|
Integer values collected that are bigger than 2^63 and smaller than 1e21 (or in
|
|
this exact same window of their negative counterparts) are encoded by golang
|
|
JSON encoder in decimal format and that is not fully supported by Elasticsearch
|
|
dynamic field mapping. This causes the metrics with such values to be dropped in
|
|
case a field mapping has not been created yet on the telegraf index. If that's
|
|
the case you will see an exception on Elasticsearch side like this:
|
|
|
|
```json
|
|
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_state_exception","reason":"No matching token for number_type [BIG_INTEGER]"}},"status":400}
|
|
```
|
|
|
|
The correct field mapping will be created on the telegraf index as soon as a
|
|
supported JSON value is received by Elasticsearch, and subsequent insertions
|
|
will work because the field mapping will already exist.
|
|
|
|
This issue is caused by the way Elasticsearch tries to detect integer fields,
|
|
and by how golang encodes numbers in JSON. There is no clear workaround for this
|
|
at the moment.
|