Adding upstream version 1.34.4.
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
parent
e393c3af3f
commit
4978089aab
GPG key ID:
FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions
368
plugins/outputs/opensearch/README.md
Normal file
368
plugins/outputs/opensearch/README.md
Normal file
|
|
@ -0,0 +1,368 @@
|
|||
# OpenSearch Output Plugin
|
||||
|
||||
This plugin writes metrics to a [OpenSearch][opensearch] instance via HTTP.
|
||||
It supports OpenSearch releases v1 and v2 but future comparability with 1.x is
|
||||
not guaranteed and instead will focus on 2.x support.
|
||||
|
||||
> [!TIP]
|
||||
> Consider using the existing Elasticsearch plugin for 1.x.
|
||||
|
||||
⭐ Telegraf v1.29.0
|
||||
🏷️ datastore, logging
|
||||
💻 all
|
||||
|
||||
[opensearch]: https://opensearch.org/
|
||||
|
||||
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
|
||||
|
||||
In addition to the plugin-specific configuration settings, plugins support
|
||||
additional global and plugin configuration settings. These settings are used to
|
||||
modify metrics, tags, and field or create aliases and configure ordering, etc.
|
||||
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
|
||||
|
||||
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
|
||||
|
||||
## Configuration
|
||||
|
||||
```toml @sample.conf
|
||||
# Configuration for OpenSearch to send metrics to.
|
||||
[[outputs.opensearch]]
|
||||
## URLs
|
||||
## The full HTTP endpoint URL for your OpenSearch instance. Multiple URLs can
|
||||
## be specified as part of the same cluster, but only one URLs is used to
|
||||
## write during each interval.
|
||||
urls = ["http://node1.os.example.com:9200"]
|
||||
|
||||
## Index Name
|
||||
## Target index name for metrics (OpenSearch will create if it not exists).
|
||||
## This is a Golang template (see https://pkg.go.dev/text/template)
|
||||
## You can also specify
|
||||
## metric name (`{{.Name}}`), tag value (`{{.Tag "tag_name"}}`), field value (`{{.Field "field_name"}}`)
|
||||
## If the tag does not exist, the default tag value will be empty string "".
|
||||
## the timestamp (`{{.Time.Format "xxxxxxxxx"}}`).
|
||||
## For example: "telegraf-{{.Time.Format \"2006-01-02\"}}-{{.Tag \"host\"}}" would set it to telegraf-2023-07-27-HostName
|
||||
index_name = ""
|
||||
|
||||
## Timeout
|
||||
## OpenSearch client timeout
|
||||
# timeout = "5s"
|
||||
|
||||
## Sniffer
|
||||
## Set to true to ask OpenSearch a list of all cluster nodes,
|
||||
## thus it is not necessary to list all nodes in the urls config option
|
||||
# enable_sniffer = false
|
||||
|
||||
## GZIP Compression
|
||||
## Set to true to enable gzip compression
|
||||
# enable_gzip = false
|
||||
|
||||
## Health Check Interval
|
||||
## Set the interval to check if the OpenSearch nodes are available
|
||||
## Setting to "0s" will disable the health check (not recommended in production)
|
||||
# health_check_interval = "10s"
|
||||
|
||||
## Set the timeout for periodic health checks.
|
||||
# health_check_timeout = "1s"
|
||||
## HTTP basic authentication details.
|
||||
# username = ""
|
||||
# password = ""
|
||||
## HTTP bearer token authentication details
|
||||
# auth_bearer_token = ""
|
||||
|
||||
## Optional TLS Config
|
||||
## Set to true/false to enforce TLS being enabled/disabled. If not set,
|
||||
## enable TLS only if any of the other options are specified.
|
||||
# tls_enable =
|
||||
## Trusted root certificates for server
|
||||
# tls_ca = "/path/to/cafile"
|
||||
## Used for TLS client certificate authentication
|
||||
# tls_cert = "/path/to/certfile"
|
||||
## Used for TLS client certificate authentication
|
||||
# tls_key = "/path/to/keyfile"
|
||||
## Send the specified TLS server name via SNI
|
||||
# tls_server_name = "kubernetes.example.com"
|
||||
## Use TLS but skip chain & host verification
|
||||
# insecure_skip_verify = false
|
||||
|
||||
## Template Config
|
||||
## Manage templates
|
||||
## Set to true if you want telegraf to manage its index template.
|
||||
## If enabled it will create a recommended index template for telegraf indexes
|
||||
# manage_template = true
|
||||
|
||||
## Template Name
|
||||
## The template name used for telegraf indexes
|
||||
# template_name = "telegraf"
|
||||
|
||||
## Overwrite Templates
|
||||
## Set to true if you want telegraf to overwrite an existing template
|
||||
# overwrite_template = false
|
||||
|
||||
## Document ID
|
||||
## If set to true a unique ID hash will be sent as
|
||||
## sha256(concat(timestamp,measurement,series-hash)) string. It will enable
|
||||
## data resend and update metric points avoiding duplicated metrics with
|
||||
## different id's
|
||||
# force_document_id = false
|
||||
|
||||
## Value Handling
|
||||
## Specifies the handling of NaN and Inf values.
|
||||
## This option can have the following values:
|
||||
## none -- do not modify field-values (default); will produce an error
|
||||
## if NaNs or infs are encountered
|
||||
## drop -- drop fields containing NaNs or infs
|
||||
## replace -- replace with the value in "float_replacement_value" (default: 0.0)
|
||||
## NaNs and inf will be replaced with the given number, -inf with the negative of that number
|
||||
# float_handling = "none"
|
||||
# float_replacement_value = 0.0
|
||||
|
||||
## Pipeline Config
|
||||
## To use a ingest pipeline, set this to the name of the pipeline you want to use.
|
||||
# use_pipeline = "my_pipeline"
|
||||
|
||||
## Pipeline Name
|
||||
## Additionally, you can specify a tag name using the notation (`{{.Tag "tag_name"}}`)
|
||||
## which will be used as the pipeline name (e.g. "{{.Tag "os_pipeline"}}").
|
||||
## If the tag does not exist, the default pipeline will be used as the pipeline.
|
||||
## If no default pipeline is set, no pipeline is used for the metric.
|
||||
# default_pipeline = ""
|
||||
```
|
||||
|
||||
### Required parameters
|
||||
|
||||
* `urls`: A list containing the full HTTP URL of one or more nodes from your
|
||||
OpenSearch instance.
|
||||
* `index_name`: The target index for metrics. You can use the date format
|
||||
|
||||
For example: "telegraf-{{.Time.Format \"2006-01-02\"}}" would set it to
|
||||
"telegraf-2023-07-27". You can also specify metric name (`{{ .Name }}`), tag
|
||||
value (`{{ .Tag \"tag_name\" }}`), and field value
|
||||
(`{{ .Field \"field_name\" }}`).
|
||||
|
||||
If the tag does not exist, the default tag value will be empty string ""
|
||||
|
||||
## Permissions
|
||||
|
||||
If you are using authentication within your OpenSearch cluster, you need to
|
||||
create an account and create a role with at least the manage role in the Cluster
|
||||
Privileges category. Otherwise, your account will not be able to connect to your
|
||||
OpenSearch cluster and send logs to your cluster. After that, you need to
|
||||
add "create_index" and "write" permission to your specific index pattern.
|
||||
|
||||
## OpenSearch indexes and templates
|
||||
|
||||
### Indexes per time-frame
|
||||
|
||||
This plugin can manage indexes per time-frame, as commonly done in other tools
|
||||
with OpenSearch. The timestamp of the metric collected will be used to decide
|
||||
the index destination. For more information about this usage on OpenSearch,
|
||||
check [the docs][1].
|
||||
|
||||
[1]: https://opensearch.org/docs/latest/
|
||||
|
||||
### Template management
|
||||
|
||||
Index templates are used in OpenSearch to define settings and mappings for
|
||||
the indexes and how the fields should be analyzed. For more information on how
|
||||
this works, see [the docs][2].
|
||||
|
||||
This plugin can create a working template for use with telegraf metrics. It uses
|
||||
OpenSearch dynamic templates feature to set proper types for the tags and
|
||||
metrics fields. If the template specified already exists, it will not overwrite
|
||||
unless you configure this plugin to do so. Thus you can customize this template
|
||||
after its creation if necessary.
|
||||
|
||||
Example of an index template created by telegraf on OpenSearch 2.x:
|
||||
|
||||
```json
|
||||
{
|
||||
"telegraf-2022.10.02" : {
|
||||
"aliases" : { },
|
||||
"mappings" : {
|
||||
"properties" : {
|
||||
"@timestamp" : {
|
||||
"type" : "date"
|
||||
},
|
||||
"disk" : {
|
||||
"properties" : {
|
||||
"free" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"inodes_free" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"inodes_total" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"inodes_used" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"total" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"used" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"used_percent" : {
|
||||
"type" : "float"
|
||||
}
|
||||
}
|
||||
},
|
||||
"measurement_name" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"type" : "keyword",
|
||||
"ignore_above" : 256
|
||||
}
|
||||
}
|
||||
},
|
||||
"tag" : {
|
||||
"properties" : {
|
||||
"cpu" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"type" : "keyword",
|
||||
"ignore_above" : 256
|
||||
}
|
||||
}
|
||||
},
|
||||
"device" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"type" : "keyword",
|
||||
"ignore_above" : 256
|
||||
}
|
||||
}
|
||||
},
|
||||
"host" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"type" : "keyword",
|
||||
"ignore_above" : 256
|
||||
}
|
||||
}
|
||||
},
|
||||
"mode" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"type" : "keyword",
|
||||
"ignore_above" : 256
|
||||
}
|
||||
}
|
||||
},
|
||||
"path" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"type" : "keyword",
|
||||
"ignore_above" : 256
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"settings" : {
|
||||
"index" : {
|
||||
"creation_date" : "1664693522789",
|
||||
"number_of_shards" : "1",
|
||||
"number_of_replicas" : "1",
|
||||
"uuid" : "TYugdmvsQfmxjzbGRJ8FIw",
|
||||
"version" : {
|
||||
"created" : "136247827"
|
||||
},
|
||||
"provided_name" : "telegraf-2022.10.02"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
[2]: https://opensearch.org/docs/latest/opensearch/index-templates/
|
||||
|
||||
### Example events
|
||||
|
||||
This plugin will format the events in the following way:
|
||||
|
||||
```json
|
||||
{
|
||||
"@timestamp": "2017-01-01T00:00:00+00:00",
|
||||
"measurement_name": "cpu",
|
||||
"cpu": {
|
||||
"usage_guest": 0,
|
||||
"usage_guest_nice": 0,
|
||||
"usage_idle": 71.85413456197966,
|
||||
"usage_iowait": 0.256805341656516,
|
||||
"usage_irq": 0,
|
||||
"usage_nice": 0,
|
||||
"usage_softirq": 0.2054442732579466,
|
||||
"usage_steal": 0,
|
||||
"usage_system": 15.04879301548127,
|
||||
"usage_user": 12.634822807288275
|
||||
},
|
||||
"tag": {
|
||||
"cpu": "cpu-total",
|
||||
"host": "opensearhhost",
|
||||
"dc": "datacenter1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"@timestamp": "2017-01-01T00:00:00+00:00",
|
||||
"measurement_name": "system",
|
||||
"system": {
|
||||
"load1": 0.78,
|
||||
"load15": 0.8,
|
||||
"load5": 0.8,
|
||||
"n_cpus": 2,
|
||||
"n_users": 2
|
||||
},
|
||||
"tag": {
|
||||
"host": "opensearhhost",
|
||||
"dc": "datacenter1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Known issues
|
||||
|
||||
Integer values collected that are bigger than 2^63 and smaller than 1e21 (or in
|
||||
this exact same window of their negative counterparts) are encoded by golang
|
||||
JSON encoder in decimal format and that is not fully supported by OpenSearch
|
||||
dynamic field mapping. This causes the metrics with such values to be dropped in
|
||||
case a field mapping has not been created yet on the telegraf index. If that's
|
||||
the case you will see an exception on OpenSearch side like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"root_cause": [
|
||||
{"type": "mapper_parsing_exception", "reason": "failed to parse"}
|
||||
],
|
||||
"type": "mapper_parsing_exception",
|
||||
"reason": "failed to parse",
|
||||
"caused_by": {
|
||||
"type": "illegal_state_exception",
|
||||
"reason": "No matching token for number_type [BIG_INTEGER]"
|
||||
}
|
||||
},
|
||||
"status": 400
|
||||
}
|
||||
```
|
||||
|
||||
The correct field mapping will be created on the telegraf index as soon as a
|
||||
supported JSON value is received by OpenSearch, and subsequent insertions
|
||||
will work because the field mapping will already exist.
|
||||
|
||||
This issue is caused by the way OpenSearch tries to detect integer fields,
|
||||
and by how golang encodes numbers in JSON. There is no clear workaround for this
|
||||
at the moment.
|
||||
Loading…
Add table
Add a link
Reference in a new issue