influx/telegraf
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.34.4.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-05-24 07:26:29 +02:00
parent e393c3af3f
commit 4978089aab
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

146
plugins/outputs/amqp/README.md Normal file
View file

@ -0,0 +1,146 @@
# AMQP Output Plugin
This plugin writes to an Advanced Message Queuing Protocol v0.9.1 broker.
A prominent implementation of this protocol is [RabbitMQ][rabbitmq].
> [!NOTE]
> This plugin does not bind the AMQP exchange to a queue.
For an introduction check the [AMQP concepts page][amqp_concepts] and the
[RabbitMQ getting started guide][rabbitmq_getting_started].
⭐ Telegraf v0.1.9
🏷️ messaging
💻 all
[amqp_concepts]: https://www.rabbitmq.com/tutorials/amqp-concepts.html
[rabbitmq]: https://www.rabbitmq.com
[rabbitmq_getting_started]: https://www.rabbitmq.com/getstarted.html
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
## Secret-store support
This plugin supports secrets from secret-stores for the `username` and
`password` option.
See the [secret-store documentation][SECRETSTORE] for more details on how
to use them.
[SECRETSTORE]: ../../../docs/CONFIGURATION.md#secret-store-secrets
## Configuration
```toml @sample.conf
# Publishes metrics to an AMQP broker
[[outputs.amqp]]
## Brokers to publish to. If multiple brokers are specified a random broker
## will be selected anytime a connection is established. This can be
## helpful for load balancing when not using a dedicated load balancer.
brokers = ["amqp://localhost:5672/influxdb"]
## Maximum messages to send over a connection. Once this is reached, the
## connection is closed and a new connection is made. This can be helpful for
## load balancing when not using a dedicated load balancer.
# max_messages = 0
## Exchange to declare and publish to.
exchange = "telegraf"
## Exchange type; common types are "direct", "fanout", "topic", "header", "x-consistent-hash".
# exchange_type = "topic"
## If true, exchange will be passively declared.
# exchange_passive = false
## Exchange durability can be either "transient" or "durable".
# exchange_durability = "durable"
## Additional exchange arguments.
# exchange_arguments = { }
# exchange_arguments = {"hash_property" = "timestamp"}
## Authentication credentials for the PLAIN auth_method.
# username = ""
# password = ""
## Auth method. PLAIN and EXTERNAL are supported
## Using EXTERNAL requires enabling the rabbitmq_auth_mechanism_ssl plugin as
## described here: https://www.rabbitmq.com/plugins.html
# auth_method = "PLAIN"
## Metric tag to use as a routing key.
## ie, if this tag exists, its value will be used as the routing key
# routing_tag = "host"
## Static routing key. Used when no routing_tag is set or as a fallback
## when the tag specified in routing tag is not found.
# routing_key = ""
# routing_key = "telegraf"
## Delivery Mode controls if a published message is persistent.
## One of "transient" or "persistent".
# delivery_mode = "transient"
## Static headers added to each published message.
# headers = { }
# headers = {"database" = "telegraf", "retention_policy" = "default"}
## Connection timeout. If not provided, will default to 5s. 0s means no
## timeout (not recommended).
# timeout = "5s"
## Optional TLS Config
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
## Optional Proxy Configuration
# use_proxy = false
# proxy_url = "localhost:8888"
## If true use batch serialization format instead of line based delimiting.
## Only applies to data formats which are not line based such as JSON.
## Recommended to set to true.
# use_batch_format = false
## Content encoding for message payloads, can be set to "gzip" to or
## "identity" to apply no encoding.
##
## Please note that when use_batch_format = false each amqp message contains only
## a single metric, it is recommended to use compression with batch format
## for best results.
# content_encoding = "identity"
## Data format to output.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
# data_format = "influx"
```
### Routing
If `routing_tag` is set, and the tag is defined on the metric, the value of the
tag is used as the routing key. Otherwise the value of `routing_key` is used
directly. If both are unset the empty string is used.
Exchange types that do not use a routing key, `direct` and `header`, always use
the empty string as the routing key.
Metrics are published in batches based on the final routing key.
### Proxy
If you want to use a proxy, you need to set `use_proxy = true`. This will
use the system's proxy settings to determine the proxy URL. If you need to
specify a proxy URL manually, you can do so by using `proxy_url`, overriding
the system settings.

334
plugins/outputs/amqp/amqp.go Normal file
View file

@ -0,0 +1,334 @@
//go:generate ../../../tools/readme_config_includer/generator
package amqp
import (
"bytes"
_ "embed"
"errors"
"fmt"
"strings"
"time"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/plugins/common/proxy"
"github.com/influxdata/telegraf/plugins/common/tls"
"github.com/influxdata/telegraf/plugins/outputs"
)
//go:embed sample.conf
var sampleConfig string
const (
DefaultURL = "amqp://localhost:5672/influxdb"
DefaultAuthMethod = "PLAIN"
DefaultExchangeType = "topic"
DefaultRetentionPolicy = "default"
DefaultDatabase = "telegraf"
)
type externalAuth struct{}
func (*externalAuth) Mechanism() string {
return "EXTERNAL"
}
func (*externalAuth) Response() string {
return "\000"
}
type AMQP struct {
URL string `toml:"url" deprecated:"1.7.0;1.35.0;use 'brokers' instead"`
Brokers []string `toml:"brokers"`
Exchange string `toml:"exchange"`
ExchangeType string `toml:"exchange_type"`
ExchangePassive bool `toml:"exchange_passive"`
ExchangeDurability string `toml:"exchange_durability"`
ExchangeArguments map[string]string `toml:"exchange_arguments"`
Username config.Secret `toml:"username"`
Password config.Secret `toml:"password"`
MaxMessages int `toml:"max_messages"`
AuthMethod string `toml:"auth_method"`
RoutingTag string `toml:"routing_tag"`
RoutingKey string `toml:"routing_key"`
DeliveryMode string `toml:"delivery_mode"`
Database string `toml:"database" deprecated:"1.7.0;1.35.0;use 'headers' instead"`
RetentionPolicy string `toml:"retention_policy" deprecated:"1.7.0;1.35.0;use 'headers' instead"`
Precision string `toml:"precision" deprecated:"1.2.0;1.35.0;option is ignored"`
Headers map[string]string `toml:"headers"`
Timeout config.Duration `toml:"timeout"`
UseBatchFormat bool `toml:"use_batch_format"`
ContentEncoding string `toml:"content_encoding"`
Log telegraf.Logger `toml:"-"`
tls.ClientConfig
proxy.TCPProxy
serializer telegraf.Serializer
connect func(*ClientConfig) (Client, error)
client Client
config *ClientConfig
sentMessages int
encoder internal.ContentEncoder
}
type Client interface {
Publish(key string, body []byte) error
Close() error
}
func (*AMQP) SampleConfig() string {
return sampleConfig
}
func (q *AMQP) SetSerializer(serializer telegraf.Serializer) {
q.serializer = serializer
}
func (q *AMQP) Init() error {
var err error
q.config, err = q.makeClientConfig()
if err != nil {
return err
}
q.encoder, err = internal.NewContentEncoder(q.ContentEncoding)
if err != nil {
return err
}
return nil
}
func (q *AMQP) Connect() error {
var err error
q.client, err = q.connect(q.config)
return err
}
func (q *AMQP) Close() error {
if q.client != nil {
return q.client.Close()
}
return nil
}
func (q *AMQP) routingKey(metric telegraf.Metric) string {
if q.RoutingTag != "" {
key, ok := metric.GetTag(q.RoutingTag)
if ok {
return key
}
}
return q.RoutingKey
}
func (q *AMQP) Write(metrics []telegraf.Metric) error {
batches := make(map[string][]telegraf.Metric)
if q.ExchangeType == "header" {
// Since the routing_key is ignored for this exchange type send as a
// single batch.
batches[""] = metrics
} else {
for _, metric := range metrics {
routingKey := q.routingKey(metric)
if _, ok := batches[routingKey]; !ok {
batches[routingKey] = make([]telegraf.Metric, 0)
}
batches[routingKey] = append(batches[routingKey], metric)
}
}
first := true
for key, metrics := range batches {
body, err := q.serialize(metrics)
if err != nil {
return err
}
body, err = q.encoder.Encode(body)
if err != nil {
return err
}
err = q.publish(key, body)
if err != nil {
// If this is the first attempt to publish and the connection is
// closed, try to reconnect and retry once.
var aerr *amqp.Error
if first && errors.As(err, &aerr) && errors.Is(aerr, amqp.ErrClosed) {
q.client = nil
err := q.publish(key, body)
if err != nil {
return err
}
} else if q.client != nil {
if err := q.client.Close(); err != nil {
q.Log.Errorf("Closing connection failed: %v", err)
}
q.client = nil
return err
}
}
first = false
}
if q.sentMessages >= q.MaxMessages && q.MaxMessages > 0 {
q.Log.Debug("Sent MaxMessages; closing connection")
if err := q.client.Close(); err != nil {
q.Log.Errorf("Closing connection failed: %v", err)
}
q.client = nil
}
return nil
}
func (q *AMQP) publish(key string, body []byte) error {
if q.client == nil {
client, err := q.connect(q.config)
if err != nil {
return err
}
q.sentMessages = 0
q.client = client
}
err := q.client.Publish(key, body)
if err != nil {
return err
}
q.sentMessages++
return nil
}
func (q *AMQP) serialize(metrics []telegraf.Metric) ([]byte, error) {
if q.UseBatchFormat {
return q.serializer.SerializeBatch(metrics)
}
var buf bytes.Buffer
for _, metric := range metrics {
octets, err := q.serializer.Serialize(metric)
if err != nil {
q.Log.Debugf("Could not serialize metric: %v", err)
continue
}
buf.Write(octets)
}
body := buf.Bytes()
return body, nil
}
func (q *AMQP) makeClientConfig() (*ClientConfig, error) {
clientConfig := &ClientConfig{
exchange: q.Exchange,
exchangeType: q.ExchangeType,
exchangePassive: q.ExchangePassive,
encoding: q.ContentEncoding,
timeout: time.Duration(q.Timeout),
log: q.Log,
}
switch q.ExchangeDurability {
case "transient":
clientConfig.exchangeDurable = false
default:
clientConfig.exchangeDurable = true
}
clientConfig.brokers = q.Brokers
if len(clientConfig.brokers) == 0 {
clientConfig.brokers = []string{q.URL}
}
switch q.DeliveryMode {
case "transient":
clientConfig.deliveryMode = amqp.Transient
case "persistent":
clientConfig.deliveryMode = amqp.Persistent
default:
clientConfig.deliveryMode = amqp.Transient
}
if len(q.Headers) > 0 {
clientConfig.headers = make(amqp.Table, len(q.Headers))
for k, v := range q.Headers {
clientConfig.headers[k] = v
}
} else {
// Copy deprecated fields into message header
clientConfig.headers = amqp.Table{
"database": q.Database,
"retention_policy": q.RetentionPolicy,
}
}
if len(q.ExchangeArguments) > 0 {
clientConfig.exchangeArguments = make(amqp.Table, len(q.ExchangeArguments))
for k, v := range q.ExchangeArguments {
clientConfig.exchangeArguments[k] = v
}
}
tlsConfig, err := q.ClientConfig.TLSConfig()
if err != nil {
return nil, err
}
clientConfig.tlsConfig = tlsConfig
dialer, err := q.TCPProxy.Proxy()
if err != nil {
return nil, err
}
clientConfig.dialer = dialer
var auth []amqp.Authentication
if strings.EqualFold(q.AuthMethod, "EXTERNAL") {
auth = []amqp.Authentication{&externalAuth{}}
} else if !q.Username.Empty() || !q.Password.Empty() {
username, err := q.Username.Get()
if err != nil {
return nil, fmt.Errorf("getting username failed: %w", err)
}
defer username.Destroy()
password, err := q.Password.Get()
if err != nil {
return nil, fmt.Errorf("getting password failed: %w", err)
}
defer password.Destroy()
auth = []amqp.Authentication{
&amqp.PlainAuth{
Username: username.String(),
Password: password.String(),
},
}
}
clientConfig.auth = auth
return clientConfig, nil
}
func connect(clientConfig *ClientConfig) (Client, error) {
return newClient(clientConfig)
}
func init() {
outputs.Add("amqp", func() telegraf.Output {
return &AMQP{
Brokers: []string{DefaultURL},
ExchangeType: DefaultExchangeType,
AuthMethod: DefaultAuthMethod,
Headers: map[string]string{
"database": DefaultDatabase,
"retention_policy": DefaultRetentionPolicy,
},
Timeout: config.Duration(time.Second * 5),
connect: connect,
}
})
}

160
plugins/outputs/amqp/amqp_test.go Normal file
View file

@ -0,0 +1,160 @@
package amqp
import (
"testing"
"time"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/stretchr/testify/require"
"github.com/influxdata/telegraf/config"
)
type MockClient struct {
PublishF func() error
CloseF func() error
PublishCallCount int
CloseCallCount int
}
func (c *MockClient) Publish(string, []byte) error {
c.PublishCallCount++
return c.PublishF()
}
func (c *MockClient) Close() error {
c.CloseCallCount++
return c.CloseF()
}
func NewMockClient() Client {
return &MockClient{
PublishF: func() error {
return nil
},
CloseF: func() error {
return nil
},
}
}
func TestConnect(t *testing.T) {
tests := []struct {
name string
output *AMQP
errFunc func(t *testing.T, output *AMQP, err error)
}{
{
name: "defaults",
output: &AMQP{
Brokers: []string{DefaultURL},
ExchangeType: DefaultExchangeType,
ExchangeDurability: "durable",
AuthMethod: DefaultAuthMethod,
Headers: map[string]string{
"database": DefaultDatabase,
"retention_policy": DefaultRetentionPolicy,
},
Timeout: config.Duration(time.Second * 5),
connect: func(_ *ClientConfig) (Client, error) {
return NewMockClient(), nil
},
},
errFunc: func(t *testing.T, output *AMQP, err error) {
cfg := output.config
require.Equal(t, []string{DefaultURL}, cfg.brokers)
require.Empty(t, cfg.exchange)
require.Equal(t, "topic", cfg.exchangeType)
require.False(t, cfg.exchangePassive)
require.True(t, cfg.exchangeDurable)
require.Equal(t, amqp.Table(nil), cfg.exchangeArguments)
require.Equal(t, amqp.Table{
"database": DefaultDatabase,
"retention_policy": DefaultRetentionPolicy,
}, cfg.headers)
require.Equal(t, amqp.Transient, cfg.deliveryMode)
require.NoError(t, err)
},
},
{
name: "headers overrides deprecated dbrp",
output: &AMQP{
Headers: map[string]string{
"foo": "bar",
},
connect: func(_ *ClientConfig) (Client, error) {
return NewMockClient(), nil
},
},
errFunc: func(t *testing.T, output *AMQP, err error) {
cfg := output.config
require.Equal(t, amqp.Table{
"foo": "bar",
}, cfg.headers)
require.NoError(t, err)
},
},
{
name: "exchange args",
output: &AMQP{
ExchangeArguments: map[string]string{
"foo": "bar",
},
connect: func(_ *ClientConfig) (Client, error) {
return NewMockClient(), nil
},
},
errFunc: func(t *testing.T, output *AMQP, err error) {
cfg := output.config
require.Equal(t, amqp.Table{
"foo": "bar",
}, cfg.exchangeArguments)
require.NoError(t, err)
},
},
{
name: "username password",
output: &AMQP{
URL: "amqp://foo:bar@localhost",
Username: config.NewSecret([]byte("telegraf")),
Password: config.NewSecret([]byte("pa$$word")),
connect: func(_ *ClientConfig) (Client, error) {
return NewMockClient(), nil
},
},
errFunc: func(t *testing.T, output *AMQP, err error) {
cfg := output.config
require.Equal(t, []amqp.Authentication{
&amqp.PlainAuth{
Username: "telegraf",
Password: "pa$$word",
},
}, cfg.auth)
require.NoError(t, err)
},
},
{
name: "url support",
output: &AMQP{
URL: DefaultURL,
connect: func(_ *ClientConfig) (Client, error) {
return NewMockClient(), nil
},
},
errFunc: func(t *testing.T, output *AMQP, err error) {
cfg := output.config
require.Equal(t, []string{DefaultURL}, cfg.brokers)
require.NoError(t, err)
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
require.NoError(t, tt.output.Init())
err := tt.output.Connect()
tt.errFunc(t, tt.output, err)
})
}
}

146
plugins/outputs/amqp/client.go Normal file
View file

@ -0,0 +1,146 @@
package amqp
import (
"context"
"crypto/tls"
"errors"
"fmt"
"math/rand"
"net"
"time"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/common/proxy"
)
type ClientConfig struct {
brokers []string
exchange string
exchangeType string
exchangePassive bool
exchangeDurable bool
exchangeArguments amqp.Table
encoding string
headers amqp.Table
deliveryMode uint8
tlsConfig *tls.Config
timeout time.Duration
auth []amqp.Authentication
dialer *proxy.ProxiedDialer
log telegraf.Logger
}
type client struct {
conn *amqp.Connection
channel *amqp.Channel
config *ClientConfig
}
// newClient opens a connection to one of the brokers at random
func newClient(config *ClientConfig) (*client, error) {
client := &client{
config: config,
}
p := rand.Perm(len(config.brokers))
for _, n := range p {
broker := config.brokers[n]
config.log.Debugf("Connecting to %q", broker)
conn, err := amqp.DialConfig(
broker, amqp.Config{
TLSClientConfig: config.tlsConfig,
SASL: config.auth, // if nil, it will be PLAIN taken from url
Dial: func(network, addr string) (net.Conn, error) {
return config.dialer.DialTimeout(network, addr, config.timeout)
},
})
if err == nil {
client.conn = conn
config.log.Debugf("Connected to %q", broker)
break
}
config.log.Debugf("Error connecting to %q - %v", broker, err.Error())
}
if client.conn == nil {
return nil, errors.New("could not connect to any broker")
}
channel, err := client.conn.Channel()
if err != nil {
return nil, fmt.Errorf("error opening channel: %w", err)
}
client.channel = channel
err = client.DeclareExchange()
if err != nil {
return nil, err
}
return client, nil
}
func (c *client) DeclareExchange() error {
if c.config.exchange == "" {
return nil
}
var err error
if c.config.exchangePassive {
err = c.channel.ExchangeDeclarePassive(
c.config.exchange,
c.config.exchangeType,
c.config.exchangeDurable,
false, // delete when unused
false, // internal
false, // no-wait
c.config.exchangeArguments,
)
} else {
err = c.channel.ExchangeDeclare(
c.config.exchange,
c.config.exchangeType,
c.config.exchangeDurable,
false, // delete when unused
false, // internal
false, // no-wait
c.config.exchangeArguments,
)
}
if err != nil {
return fmt.Errorf("error declaring exchange: %w", err)
}
return nil
}
func (c *client) Publish(key string, body []byte) error {
// Note that since the channel is not in confirm mode, the absence of
// an error does not indicate successful delivery.
return c.channel.PublishWithContext(
context.Background(),
c.config.exchange, // exchange
key, // routing key
false, // mandatory
false, // immediate
amqp.Publishing{
Headers: c.config.headers,
ContentType: "text/plain",
ContentEncoding: c.config.encoding,
Body: body,
DeliveryMode: c.config.deliveryMode,
})
}
func (c *client) Close() error {
if c.conn == nil {
return nil
}
err := c.conn.Close()
if err != nil && !errors.Is(err, amqp.ErrClosed) {
return err
}
return nil
}

87
plugins/outputs/amqp/sample.conf Normal file
View file

@ -0,0 +1,87 @@
# Publishes metrics to an AMQP broker
[[outputs.amqp]]
## Brokers to publish to. If multiple brokers are specified a random broker
## will be selected anytime a connection is established. This can be
## helpful for load balancing when not using a dedicated load balancer.
brokers = ["amqp://localhost:5672/influxdb"]
## Maximum messages to send over a connection. Once this is reached, the
## connection is closed and a new connection is made. This can be helpful for
## load balancing when not using a dedicated load balancer.
# max_messages = 0
## Exchange to declare and publish to.
exchange = "telegraf"
## Exchange type; common types are "direct", "fanout", "topic", "header", "x-consistent-hash".
# exchange_type = "topic"
## If true, exchange will be passively declared.
# exchange_passive = false
## Exchange durability can be either "transient" or "durable".
# exchange_durability = "durable"
## Additional exchange arguments.
# exchange_arguments = { }
# exchange_arguments = {"hash_property" = "timestamp"}
## Authentication credentials for the PLAIN auth_method.
# username = ""
# password = ""
## Auth method. PLAIN and EXTERNAL are supported
## Using EXTERNAL requires enabling the rabbitmq_auth_mechanism_ssl plugin as
## described here: https://www.rabbitmq.com/plugins.html
# auth_method = "PLAIN"
## Metric tag to use as a routing key.
## ie, if this tag exists, its value will be used as the routing key
# routing_tag = "host"
## Static routing key. Used when no routing_tag is set or as a fallback
## when the tag specified in routing tag is not found.
# routing_key = ""
# routing_key = "telegraf"
## Delivery Mode controls if a published message is persistent.
## One of "transient" or "persistent".
# delivery_mode = "transient"
## Static headers added to each published message.
# headers = { }
# headers = {"database" = "telegraf", "retention_policy" = "default"}
## Connection timeout. If not provided, will default to 5s. 0s means no
## timeout (not recommended).
# timeout = "5s"
## Optional TLS Config
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
## Optional Proxy Configuration
# use_proxy = false
# proxy_url = "localhost:8888"
## If true use batch serialization format instead of line based delimiting.
## Only applies to data formats which are not line based such as JSON.
## Recommended to set to true.
# use_batch_format = false
## Content encoding for message payloads, can be set to "gzip" to or
## "identity" to apply no encoding.
##
## Please note that when use_batch_format = false each amqp message contains only
## a single metric, it is recommended to use compression with batch format
## for best results.
# content_encoding = "identity"
## Data format to output.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
# data_format = "influx"