influx/telegraf
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.34.4.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-05-24 07:26:29 +02:00
parent e393c3af3f
commit 4978089aab
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

103
plugins/inputs/cloud_pubsub_push/README.md Normal file
View file

@ -0,0 +1,103 @@
# Google Cloud PubSub Push Input Plugin
This plugin listens for messages sent via an HTTP POST from
[Google Cloud PubSub][pubsub] and expects messages in Google's Pub/Sub
_JSON format_. The plugin allows Telegraf to serve as an endpoint of push
service.
Google's PubSub service will __only__ send over HTTPS/TLS so this plugin must be
behind a valid proxy or must be configured to use TLS by setting the `tls_cert`
and `tls_key` accordingly.
Enable mutually authenticated TLS and authorize client connections by signing
certificate authority by including a list of allowed CA certificate file names
in `tls_allowed_cacerts`.
⭐ Telegraf v1.10.0
🏷️ cloud, messaging
💻 all
[pubsub]: https://cloud.google.com/pubsub
## Service Input <!-- @/docs/includes/service_input.md -->
This plugin is a service input. Normal plugins gather metrics determined by the
interval setting. Service plugins start a service to listen and wait for
metrics or events to occur. Service plugins have two key differences from
normal plugins:
1. The global or plugin specific `interval` setting may not apply
2. The CLI options of `--test`, `--test-wait`, and `--once` may not produce
output for this plugin
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
## Configuration
```toml @sample.conf
# Google Cloud Pub/Sub Push HTTP listener
[[inputs.cloud_pubsub_push]]
## Address and port to host HTTP listener on
service_address = ":8080"
## Application secret to verify messages originate from Cloud Pub/Sub
# token = ""
## Path to listen to.
# path = "/"
## Maximum duration before timing out read of the request
# read_timeout = "10s"
## Maximum duration before timing out write of the response. This should be
## set to a value large enough that you can send at least 'metric_batch_size'
## number of messages within the duration.
# write_timeout = "10s"
## Maximum allowed http request body size in bytes.
## 0 means to use the default of 524,288,00 bytes (500 mebibytes)
# max_body_size = "500MB"
## Whether to add the pubsub metadata, such as message attributes and
## subscription as a tag.
# add_meta = false
## Max undelivered messages
## This plugin uses tracking metrics, which ensure messages are read to
## outputs before acknowledging them to the original broker to ensure data
## is not lost. This option sets the maximum messages to read from the
## broker that have not been written by an output.
##
## This value needs to be picked with awareness of the agent's
## metric_batch_size value as well. Setting max undelivered messages too high
## can result in a constant stream of data batches to the output. While
## setting it too low may never flush the broker's messages.
# max_undelivered_messages = 1000
## Set one or more allowed client CA certificate file names to
## enable mutually authenticated TLS connections
# tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
## Add service certificate and key
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
```
This plugin assumes you have already created a PUSH subscription for a given
PubSub topic.
## Metrics
## Example Output

286
plugins/inputs/cloud_pubsub_push/cloud_pubsub_push.go Normal file
View file

@ -0,0 +1,286 @@
//go:generate ../../../tools/readme_config_includer/generator
package cloud_pubsub_push
import (
"context"
"crypto/subtle"
_ "embed"
"encoding/base64"
"encoding/json"
"io"
"net/http"
"sync"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/internal"
common_tls "github.com/influxdata/telegraf/plugins/common/tls"
"github.com/influxdata/telegraf/plugins/inputs"
)
//go:embed sample.conf
var sampleConfig string
var once sync.Once
// defaultMaxBodySize is the default maximum request body size, in bytes.
// if the request body is over this size, we will return an HTTP 413 error.
const (
// 500 MB
defaultMaxBodySize = 500 * 1024 * 1024
defaultMaxUndeliveredMessages = 1000
)
type PubSubPush struct {
ServiceAddress string
Token string
Path string
ReadTimeout config.Duration
WriteTimeout config.Duration
MaxBodySize config.Size
AddMeta bool
Log telegraf.Logger
MaxUndeliveredMessages int `toml:"max_undelivered_messages"`
common_tls.ServerConfig
telegraf.Parser
server *http.Server
acc telegraf.TrackingAccumulator
ctx context.Context
cancel context.CancelFunc
wg *sync.WaitGroup
mu *sync.Mutex
undelivered map[telegraf.TrackingID]chan bool
sem chan struct{}
}
// message defines the structure of a Google Pub/Sub message.
type message struct {
Atts map[string]string `json:"attributes"`
Data string `json:"data"` // Data is base64 encoded data
}
// payload is the received Google Pub/Sub data. (https://cloud.google.com/pubsub/docs/push)
type payload struct {
Msg message `json:"message"`
Subscription string `json:"subscription"`
}
func (*PubSubPush) SampleConfig() string {
return sampleConfig
}
func (p *PubSubPush) SetParser(parser telegraf.Parser) {
p.Parser = parser
}
// Start starts the http listener service.
func (p *PubSubPush) Start(acc telegraf.Accumulator) error {
if p.MaxBodySize == 0 {
p.MaxBodySize = config.Size(defaultMaxBodySize)
}
if p.ReadTimeout < config.Duration(time.Second) {
p.ReadTimeout = config.Duration(time.Second * 10)
}
if p.WriteTimeout < config.Duration(time.Second) {
p.WriteTimeout = config.Duration(time.Second * 10)
}
tlsConf, err := p.ServerConfig.TLSConfig()
if err != nil {
return err
}
p.server = &http.Server{
Addr: p.ServiceAddress,
Handler: http.TimeoutHandler(p, time.Duration(p.WriteTimeout), "timed out processing metric"),
ReadTimeout: time.Duration(p.ReadTimeout),
TLSConfig: tlsConf,
}
p.ctx, p.cancel = context.WithCancel(context.Background())
p.wg = &sync.WaitGroup{}
p.acc = acc.WithTracking(p.MaxUndeliveredMessages)
p.sem = make(chan struct{}, p.MaxUndeliveredMessages)
p.undelivered = make(map[telegraf.TrackingID]chan bool)
p.mu = &sync.Mutex{}
p.wg.Add(1)
go func() {
defer p.wg.Done()
p.receiveDelivered()
}()
p.wg.Add(1)
go func() {
defer p.wg.Done()
if tlsConf != nil {
if err := p.server.ListenAndServeTLS("", ""); err != nil && err != http.ErrServerClosed {
p.Log.Errorf("listening and serving TLS failed: %v", err)
}
} else {
if err := p.server.ListenAndServe(); err != nil {
p.Log.Errorf("listening and serving TLS failed: %v", err)
}
}
}()
return nil
}
func (*PubSubPush) Gather(telegraf.Accumulator) error {
return nil
}
// Stop cleans up all resources
func (p *PubSubPush) Stop() {
p.cancel()
p.server.Shutdown(p.ctx) //nolint:errcheck // we cannot do anything if the shutdown fails
p.wg.Wait()
}
func (p *PubSubPush) ServeHTTP(res http.ResponseWriter, req *http.Request) {
if req.URL.Path == p.Path {
p.authenticateIfSet(p.serveWrite, res, req)
} else {
p.authenticateIfSet(http.NotFound, res, req)
}
}
func (p *PubSubPush) serveWrite(res http.ResponseWriter, req *http.Request) {
select {
case <-req.Context().Done():
res.WriteHeader(http.StatusServiceUnavailable)
return
case <-p.ctx.Done():
res.WriteHeader(http.StatusServiceUnavailable)
return
case p.sem <- struct{}{}:
break
}
// Check that the content length is not too large for us to handle.
if req.ContentLength > int64(p.MaxBodySize) {
res.WriteHeader(http.StatusRequestEntityTooLarge)
return
}
if req.Method != http.MethodPost {
res.WriteHeader(http.StatusMethodNotAllowed)
return
}
body := http.MaxBytesReader(res, req.Body, int64(p.MaxBodySize))
bytes, err := io.ReadAll(body)
if err != nil {
res.WriteHeader(http.StatusRequestEntityTooLarge)
return
}
var payload payload
if err = json.Unmarshal(bytes, &payload); err != nil {
p.Log.Errorf("Error decoding payload %s", err.Error())
res.WriteHeader(http.StatusBadRequest)
return
}
sDec, err := base64.StdEncoding.DecodeString(payload.Msg.Data)
if err != nil {
p.Log.Errorf("Base64-decode failed %s", err.Error())
res.WriteHeader(http.StatusBadRequest)
return
}
metrics, err := p.Parse(sDec)
if err != nil {
p.Log.Debug(err.Error())
res.WriteHeader(http.StatusBadRequest)
return
}
if len(metrics) == 0 {
once.Do(func() {
p.Log.Debug(internal.NoMetricsCreatedMsg)
})
}
if p.AddMeta {
for i := range metrics {
for k, v := range payload.Msg.Atts {
metrics[i].AddTag(k, v)
}
metrics[i].AddTag("subscription", payload.Subscription)
}
}
ch := make(chan bool, 1)
p.mu.Lock()
p.undelivered[p.acc.AddTrackingMetricGroup(metrics)] = ch
p.mu.Unlock()
select {
case <-req.Context().Done():
res.WriteHeader(http.StatusServiceUnavailable)
return
case success := <-ch:
if success {
res.WriteHeader(http.StatusNoContent)
} else {
res.WriteHeader(http.StatusInternalServerError)
}
}
}
func (p *PubSubPush) receiveDelivered() {
for {
select {
case <-p.ctx.Done():
return
case info := <-p.acc.Delivered():
<-p.sem
p.mu.Lock()
ch, ok := p.undelivered[info.ID()]
if !ok {
p.mu.Unlock()
continue
}
delete(p.undelivered, info.ID())
p.mu.Unlock()
if info.Delivered() {
ch <- true
} else {
ch <- false
p.Log.Debug("Metric group failed to process")
}
}
}
}
func (p *PubSubPush) authenticateIfSet(handler http.HandlerFunc, res http.ResponseWriter, req *http.Request) {
if p.Token != "" {
if subtle.ConstantTimeCompare([]byte(req.FormValue("token")), []byte(p.Token)) != 1 {
http.Error(res, "Unauthorized.", http.StatusUnauthorized)
return
}
}
handler(res, req)
}
func init() {
inputs.Add("cloud_pubsub_push", func() telegraf.Input {
return &PubSubPush{
ServiceAddress: ":8080",
Path: "/",
MaxUndeliveredMessages: defaultMaxUndeliveredMessages,
}
})
}

264
plugins/inputs/cloud_pubsub_push/cloud_pubsub_push_test.go Normal file
View file

@ -0,0 +1,264 @@
//go:build !windows
package cloud_pubsub_push
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/agent"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/logger"
"github.com/influxdata/telegraf/models"
"github.com/influxdata/telegraf/plugins/parsers/influx"
"github.com/influxdata/telegraf/testutil"
)
func TestServeHTTP(t *testing.T) {
tests := []struct {
name string
method string
path string
body io.Reader
status int
maxsize int64
expected string
fail bool
full bool
}{
{
name: "bad method get",
method: "GET",
path: "/",
status: http.StatusMethodNotAllowed,
},
{
name: "post not found",
method: "POST",
path: "/allthings",
status: http.StatusNotFound,
},
{
name: "post large date",
method: "POST",
path: "/",
status: http.StatusRequestEntityTooLarge,
body: strings.NewReader(
`{"message":{"attributes":{"deviceId":"myPi","deviceNumId":"2808946627307959","deviceRegistryId":"my-registry",` +
`"deviceRegistryLocation":"us-central1","projectId":"conference-demos","subFolder":""},` +
`"data":"dGVzdGluZ0dvb2dsZSxzZW5zb3I9Ym1lXzI4MCB0ZW1wX2M9MjMuOTUsaHVtaWRpdHk9NjIuODMgMTUzNjk1Mjk3NDU1MzUxMDIzMQ==",` +
`"messageId":"204004313210337","message_id":"204004313210337","publishTime":"2018-09-14T19:22:54.587Z",` +
`"publish_time":"2018-09-14T19:22:54.587Z"},"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
},
{
name: "post valid data",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusNoContent,
body: strings.NewReader(
`{"message":{"attributes":{"deviceId":"myPi","deviceNumId":"2808946627307959","deviceRegistryId":"my-registry",` +
`"deviceRegistryLocation":"us-central1","projectId":"conference-demos","subFolder":""},` +
`"data":"dGVzdGluZ0dvb2dsZSxzZW5zb3I9Ym1lXzI4MCB0ZW1wX2M9MjMuOTUsaHVtaWRpdHk9NjIuODMgMTUzNjk1Mjk3NDU1MzUxMDIzMQ==",` +
`"messageId":"204004313210337","message_id":"204004313210337","publishTime":"2018-09-14T19:22:54.587Z",` +
`"publish_time":"2018-09-14T19:22:54.587Z"},"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
},
{
name: "fail write",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusServiceUnavailable,
body: strings.NewReader(
`{"message":{"attributes":{"deviceId":"myPi","deviceNumId":"2808946627307959","deviceRegistryId":"my-registry",` +
`"deviceRegistryLocation":"us-central1","projectId":"conference-demos","subFolder":""},` +
`"data":"dGVzdGluZ0dvb2dsZSxzZW5zb3I9Ym1lXzI4MCB0ZW1wX2M9MjMuOTUsaHVtaWRpdHk9NjIuODMgMTUzNjk1Mjk3NDU1MzUxMDIzMQ==",` +
`"messageId":"204004313210337","message_id":"204004313210337","publishTime":"2018-09-14T19:22:54.587Z",` +
`"publish_time":"2018-09-14T19:22:54.587Z"},"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
fail: true,
},
{
name: "full buffer",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusServiceUnavailable,
body: strings.NewReader(
`{"message":{"attributes":{"deviceId":"myPi","deviceNumId":"2808946627307959","deviceRegistryId":"my-registry",` +
`"deviceRegistryLocation":"us-central1","projectId":"conference-demos","subFolder":""},` +
`"data":"dGVzdGluZ0dvb2dsZSxzZW5zb3I9Ym1lXzI4MCB0ZW1wX2M9MjMuOTUsaHVtaWRpdHk9NjIuODMgMTUzNjk1Mjk3NDU1MzUxMDIzMQ==",` +
`"messageId":"204004313210337","message_id":"204004313210337","publishTime":"2018-09-14T19:22:54.587Z",` +
`"publish_time":"2018-09-14T19:22:54.587Z"},"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
full: true,
},
{
name: "post invalid body",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusBadRequest,
body: strings.NewReader(`invalid body`),
},
{
name: "post invalid data",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusBadRequest,
body: strings.NewReader(
`{"message":{"attributes":{"deviceId":"myPi","deviceNumId":"2808946627307959","deviceRegistryId":"my-registry",` +
`"deviceRegistryLocation":"us-central1","projectId":"conference-demos","subFolder":""},"data":"not base 64 encoded data",` +
`"messageId":"204004313210337","message_id":"204004313210337","publishTime":"2018-09-14T19:22:54.587Z",` +
`"publish_time":"2018-09-14T19:22:54.587Z"},"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
},
{
name: "post invalid data format",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusBadRequest,
body: strings.NewReader(
`{"message":{"attributes":{"deviceId":"myPi","deviceNumId":"2808946627307959","deviceRegistryId":"my-registry",` +
`"deviceRegistryLocation":"us-central1","projectId":"conference-demos","subFolder":""},` +
`"data":"bm90IHZhbGlkIGZvcm1hdHRlZCBkYXRh","messageId":"204004313210337","message_id":"204004313210337",` +
`"publishTime":"2018-09-14T19:22:54.587Z","publish_time":"2018-09-14T19:22:54.587Z"},` +
`"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
},
{
name: "post invalid structured body",
method: "POST",
path: "/",
maxsize: 500 * 1024 * 1024,
status: http.StatusBadRequest,
body: strings.NewReader(
`{"message":{"attributes":{"thing":1},"data":"bm90IHZhbGlkIGZvcm1hdHRlZCBkYXRh"},` +
`"subscription":"projects/conference-demos/subscriptions/my-subscription"}`,
),
},
}
for _, test := range tests {
wg := &sync.WaitGroup{}
req, err := http.NewRequest(test.method, test.path, test.body)
require.NoError(t, err)
rr := httptest.NewRecorder()
pubPush := &PubSubPush{
Log: testutil.Logger{},
Path: "/",
MaxBodySize: config.Size(test.maxsize),
sem: make(chan struct{}, 1),
undelivered: make(map[telegraf.TrackingID]chan bool),
mu: &sync.Mutex{},
WriteTimeout: config.Duration(time.Millisecond * 10),
}
pubPush.ctx, pubPush.cancel = context.WithCancel(t.Context())
if test.full {
// fill buffer with fake message
pubPush.sem <- struct{}{}
}
p := &influx.Parser{}
require.NoError(t, p.Init())
pubPush.SetParser(p)
dst := make(chan telegraf.Metric, 1)
ro := models.NewRunningOutput(&testOutput{failWrite: test.fail}, &models.OutputConfig{}, 1, 1)
pubPush.acc = agent.NewAccumulator(&testMetricMaker{}, dst).WithTracking(1)
wg.Add(1)
go func() {
defer wg.Done()
pubPush.receiveDelivered()
}()
wg.Add(1)
go func(d chan telegraf.Metric) {
defer wg.Done()
for m := range d {
ro.AddMetric(m)
ro.Write() //nolint:errcheck // test will fail anyway if the write fails
m.Accept()
}
}(dst)
ctx, cancel := context.WithTimeout(req.Context(), time.Duration(pubPush.WriteTimeout))
req = req.WithContext(ctx)
pubPush.ServeHTTP(rr, req)
require.Equal(t, test.status, rr.Code, test.name)
if test.expected != "" {
require.Equal(t, test.expected, rr.Body.String(), test.name)
}
pubPush.cancel()
cancel()
close(dst)
wg.Wait()
}
}
type testMetricMaker struct{}
func (*testMetricMaker) Name() string {
return "TestPlugin"
}
func (tm *testMetricMaker) LogName() string {
return tm.Name()
}
func (*testMetricMaker) MakeMetric(metric telegraf.Metric) telegraf.Metric {
return metric
}
func (*testMetricMaker) Log() telegraf.Logger {
return logger.New("test", "test", "")
}
type testOutput struct {
// if true, mock a write failure
failWrite bool
}
func (*testOutput) Connect() error {
return nil
}
func (*testOutput) Close() error {
return nil
}
func (*testOutput) Description() string {
return ""
}
func (*testOutput) SampleConfig() string {
return ""
}
func (t *testOutput) Write(_ []telegraf.Metric) error {
if t.failWrite {
return errors.New("failed write")
}
return nil
}

51
plugins/inputs/cloud_pubsub_push/sample.conf Normal file
View file

@ -0,0 +1,51 @@
# Google Cloud Pub/Sub Push HTTP listener
[[inputs.cloud_pubsub_push]]
## Address and port to host HTTP listener on
service_address = ":8080"
## Application secret to verify messages originate from Cloud Pub/Sub
# token = ""
## Path to listen to.
# path = "/"
## Maximum duration before timing out read of the request
# read_timeout = "10s"
## Maximum duration before timing out write of the response. This should be
## set to a value large enough that you can send at least 'metric_batch_size'
## number of messages within the duration.
# write_timeout = "10s"
## Maximum allowed http request body size in bytes.
## 0 means to use the default of 524,288,00 bytes (500 mebibytes)
# max_body_size = "500MB"
## Whether to add the pubsub metadata, such as message attributes and
## subscription as a tag.
# add_meta = false
## Max undelivered messages
## This plugin uses tracking metrics, which ensure messages are read to
## outputs before acknowledging them to the original broker to ensure data
## is not lost. This option sets the maximum messages to read from the
## broker that have not been written by an output.
##
## This value needs to be picked with awareness of the agent's
## metric_batch_size value as well. Setting max undelivered messages too high
## can result in a constant stream of data batches to the output. While
## setting it too low may never flush the broker's messages.
# max_undelivered_messages = 1000
## Set one or more allowed client CA certificate file names to
## enable mutually authenticated TLS connections
# tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
## Add service certificate and key
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"