influx/telegraf
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.34.4.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-05-24 07:26:29 +02:00
parent e393c3af3f
commit 4978089aab
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

148
plugins/parsers/avro/README.md Normal file
View file

@ -0,0 +1,148 @@
# Avro Parser Plugin
The `Avro` parser creates metrics from a message serialized with Avro.
The message is supposed to be encoded as follows:
| Bytes | Area | Description |
| ----- | ---------- | ------------------------------------------------ |
| 0 | Magic Byte | Confluent serialization format version number. |
| 1-4 | Schema ID | 4-byte schema ID as returned by Schema Registry. |
| 5- | Data | Serialized data. |
The metric name will be set according the following priority:
1. Try to get metric name from the message field if it is set in the
`avro_measurement_field` option.
2. If the name is not determined, then try to get it from
`avro_measurement` option as the static value.
3. If the name is still not determined, then try to get it from the
schema definition in the following format `[schema_namespace.]schema_name`,
where schema namespace is optional and will be added only if it is specified
in the schema definition.
In case if the metric name could not be determined according to these steps
the error will be raised and the message will not be parsed.
## Configuration
```toml
[[inputs.kafka_consumer]]
## Kafka brokers.
brokers = ["localhost:9092"]
## Topics to consume.
topics = ["telegraf"]
## Maximum length of a message to consume, in bytes (default 0/unlimited);
## larger messages are dropped
max_message_len = 1000000
## Avro data format settings
data_format = "avro"
## Avro message format
## Supported values are "binary" (default) and "json"
# avro_format = "binary"
## URL of the schema registry which may contain username and password in the
## form http[s]://[username[:password]@]<host>[:port]
## NOTE: Exactly one of schema registry and schema must be set
avro_schema_registry = "http://localhost:8081"
## Path to the schema registry certificate. Should be specified only if
## required for connection to the schema registry.
# avro_schema_registry_cert = "/etc/telegraf/ca_cert.crt"
## Schema string; exactly one of schema registry and schema must be set
#avro_schema = '''
# {
# "type":"record",
# "name":"Value",
# "namespace":"com.example",
# "fields":[
# {
# "name":"tag",
# "type":"string"
# },
# {
# "name":"field",
# "type":"long"
# },
# {
# "name":"timestamp",
# "type":"long"
# }
# ]
# }
#'''
## Measurement field name; The meauserment name will be taken
## from this field. If not set, determine measurement name
## from the following 'avro_measurement' option
# avro_measurement_field = "field_name"
## Measurement string; if not set, determine measurement name from
## schema (as "<namespace>.<name>")
# avro_measurement = "ratings"
## Avro fields to be used as tags; optional.
# avro_tags = ["CHANNEL", "CLUB_STATUS"]
## Avro fields to be used as fields; if empty, any Avro fields
## detected from the schema, not used as tags, will be used as
## measurement fields.
# avro_fields = ["STARS"]
## Avro fields to be used as timestamp; if empty, current time will
## be used for the measurement timestamp.
# avro_timestamp = ""
## If avro_timestamp is specified, avro_timestamp_format must be set
## to one of 'unix', 'unix_ms', 'unix_us', or 'unix_ns'. It will
## default to 'unix'.
# avro_timestamp_format = "unix"
## Used to separate parts of array structures. As above, the default
## is the empty string, so a=["a", "b"] becomes a0="a", a1="b".
## If this were set to "_", then it would be a_0="a", a_1="b".
# avro_field_separator = "_"
## Define handling of union types. Possible values are:
## flatten -- add type suffix to field name (default)
## nullable -- do not modify field name but discard "null" field values
## any -- do not modify field name and set field value to the received type
# avro_union_mode = "flatten"
## Default values for given tags: optional
# tags = { "application": "hermes", "region": "central" }
```
### `avro_format`
This optional setting specifies the format of the Avro messages. Currently, the
parser supports the `binary` and `json` formats with `binary` being the default.
### `avro_timestamp` and `avro_timestamp_format`
By default the current time at ingestion will be used for all created
metrics; to set the time using the Avro message you can use the
`avro_timestamp` and `avro_timestamp_format` options together to set the
time to a value in the parsed document.
The `avro_timestamp` option specifies the field containing the time
value. If it is not set, the time of record ingestion is used. If it
is set, the field may be any numerical type: notably, it is *not*
constrained to an Avro `long` (int64) (which Avro uses for timestamps in
millisecond or microsecond resolution). However, it must represent the
number of time increments since the Unix epoch (00:00 UTC 1 Jan 1970).
The `avro_timestamp_format` option specifies the precision of the timestamp
field, and, if set, must be one of `unix`, `unix_ms`, `unix_us`, or
`unix_ns`. If `avro_timestamp` is set, `avro_timestamp_format` must be
as well.
## Metrics
One metric is created for each message. The type of each field is
automatically determined based on the schema.

340
plugins/parsers/avro/parser.go Normal file
View file

@ -0,0 +1,340 @@
package avro
import (
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jeremywohl/flatten/v2"
"github.com/linkedin/goavro/v2"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/metric"
"github.com/influxdata/telegraf/plugins/parsers"
)
// If SchemaRegistry is set, we assume that our input will be in
// Confluent Wire Format
// (https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#wire-format) and we will load the schema from the registry.
// If Schema is set, we assume the input will be Avro binary format, without
// an attached schema or schema fingerprint
type Parser struct {
MetricName string `toml:"metric_name"`
SchemaRegistry string `toml:"avro_schema_registry"`
CaCertPath string `toml:"avro_schema_registry_cert"`
Schema string `toml:"avro_schema"`
Format string `toml:"avro_format"`
Measurement string `toml:"avro_measurement"`
MeasurementField string `toml:"avro_measurement_field"`
Tags []string `toml:"avro_tags"`
Fields []string `toml:"avro_fields"`
Timestamp string `toml:"avro_timestamp"`
TimestampFormat string `toml:"avro_timestamp_format"`
FieldSeparator string `toml:"avro_field_separator"`
UnionMode string `toml:"avro_union_mode"`
DefaultTags map[string]string `toml:"tags"`
Log telegraf.Logger `toml:"-"`
registryObj *schemaRegistry
}
func (p *Parser) Init() error {
switch p.Format {
case "":
p.Format = "binary"
case "binary", "json":
// Do nothing as those are valid settings
default:
return fmt.Errorf("unknown 'avro_format' %q", p.Format)
}
switch p.UnionMode {
case "":
p.UnionMode = "flatten"
case "flatten", "nullable", "any":
// Do nothing as those are valid settings
default:
return fmt.Errorf("unknown avro_union_mode %q", p.Format)
}
if (p.Schema == "" && p.SchemaRegistry == "") || (p.Schema != "" && p.SchemaRegistry != "") {
return errors.New("exactly one of 'schema_registry' or 'schema' must be specified")
}
switch p.TimestampFormat {
case "":
p.TimestampFormat = "unix"
case "unix", "unix_ns", "unix_us", "unix_ms":
// Valid values
default:
return fmt.Errorf("invalid timestamp format '%v'", p.TimestampFormat)
}
if p.SchemaRegistry != "" {
registry, err := newSchemaRegistry(p.SchemaRegistry, p.CaCertPath)
if err != nil {
return fmt.Errorf("error connecting to the schema registry %q: %w", p.SchemaRegistry, err)
}
p.registryObj = registry
}
return nil
}
func (p *Parser) Parse(buf []byte) ([]telegraf.Metric, error) {
var schema string
var codec *goavro.Codec
var err error
var message []byte
message = buf[:]
if p.registryObj != nil {
// The input must be Confluent Wire Protocol
if buf[0] != 0 {
return nil, errors.New("first byte is not 0: not Confluent Wire Protocol")
}
schemaID := int(binary.BigEndian.Uint32(buf[1:5]))
schemastruct, err := p.registryObj.getSchemaAndCodec(schemaID)
if err != nil {
return nil, err
}
schema = schemastruct.Schema
codec = schemastruct.Codec
message = buf[5:]
} else {
// Check for single-object encoding
magicBytes := int(binary.BigEndian.Uint16(buf[:2]))
expectedMagic := int(binary.BigEndian.Uint16([]byte("c301")))
if magicBytes == expectedMagic {
message = buf[10:]
// We could in theory validate the fingerprint against
// the schema. Maybe later.
// We would get the fingerprint as int(binary.LittleEndian.Uint64(buf[2:10]))
} // Otherwise we assume bare Avro binary
schema = p.Schema
codec, err = goavro.NewCodec(schema)
if err != nil {
return nil, err
}
}
var native interface{}
switch p.Format {
case "binary":
native, _, err = codec.NativeFromBinary(message)
case "json":
native, _, err = codec.NativeFromTextual(message)
default:
return nil, fmt.Errorf("unknown format %q", p.Format)
}
if err != nil {
return nil, err
}
// Cast to string-to-interface
codecSchema, ok := native.(map[string]interface{})
if !ok {
return nil, fmt.Errorf("native is of unsupported type %T", native)
}
m, err := p.createMetric(codecSchema, schema)
if err != nil {
return nil, err
}
return []telegraf.Metric{m}, nil
}
func (p *Parser) ParseLine(line string) (telegraf.Metric, error) {
metrics, err := p.Parse([]byte(line))
if err != nil {
return nil, err
}
if len(metrics) != 1 {
return nil, errors.New("line contains multiple metrics")
}
return metrics[0], nil
}
func (p *Parser) SetDefaultTags(tags map[string]string) {
p.DefaultTags = tags
}
func (p *Parser) flattenField(fldName string, fldVal map[string]interface{}) map[string]interface{} {
// Helper function for the "nullable" and "any" p.UnionModes
// fldVal is a one-item map of string-to-something
ret := make(map[string]interface{})
if p.UnionMode == "nullable" {
_, ok := fldVal["null"]
if ok {
return ret // Return the empty map
}
}
// Otherwise, we just return the value in the fieldname.
// See README.md for an important warning about "any" and "nullable".
for _, v := range fldVal {
ret[fldName] = v
break // Not really needed, since it's a one-item map
}
return ret
}
func (p *Parser) flattenItem(fld string, fldVal interface{}) (map[string]interface{}, error) {
sep := flatten.SeparatorStyle{
Before: "",
Middle: p.FieldSeparator,
After: "",
}
candidate := make(map[string]interface{})
candidate[fld] = fldVal
var flat map[string]interface{}
var err error
// Exactly how we flatten is decided by p.UnionMode
if p.UnionMode == "flatten" {
flat, err = flatten.Flatten(candidate, "", sep)
if err != nil {
return nil, fmt.Errorf("flatten candidate %q failed: %w", candidate, err)
}
} else {
// "nullable" or "any"
typedVal, ok := candidate[fld].(map[string]interface{})
if !ok {
// the "key" is not a string, so ...
// most likely an array? Do the default thing
// and flatten the candidate.
flat, err = flatten.Flatten(candidate, "", sep)
if err != nil {
return nil, fmt.Errorf("flatten candidate %q failed: %w", candidate, err)
}
} else {
flat = p.flattenField(fld, typedVal)
}
}
return flat, nil
}
func (p *Parser) createMetric(data map[string]interface{}, schema string) (telegraf.Metric, error) {
// Tags differ from fields, in that tags are inherently strings.
// fields can be of any type.
fields := make(map[string]interface{})
tags := make(map[string]string)
// Set default tag values
for k, v := range p.DefaultTags {
tags[k] = v
}
// Avro doesn't have a Tag/Field distinction, so we have to tell
// Telegraf which items are our tags.
for _, tag := range p.Tags {
flat, flattenErr := p.flattenItem(tag, data[tag])
if flattenErr != nil {
return nil, fmt.Errorf("flatten tag %q failed: %w", tag, flattenErr)
}
for k, v := range flat {
sTag, stringErr := internal.ToString(v)
if stringErr != nil {
p.Log.Warnf("Could not convert %v to string for tag %q: %v", data[tag], tag, stringErr)
continue
}
tags[k] = sTag
}
}
var fieldList []string
if len(p.Fields) != 0 {
// If you have specified your fields in the config, you
// get what you asked for.
fieldList = p.Fields
} else {
for k := range data {
// Otherwise, that which is not a tag is a field
if _, ok := tags[k]; !ok {
fieldList = append(fieldList, k)
}
}
}
// We need to flatten out our fields. The default (the separator
// string is empty) is equivalent to what streamreactor does.
for _, fld := range fieldList {
flat, err := p.flattenItem(fld, data[fld])
if err != nil {
return nil, fmt.Errorf("flatten field %q failed: %w", fld, err)
}
for k, v := range flat {
fields[k] = v
}
}
var schemaObj map[string]interface{}
if err := json.Unmarshal([]byte(schema), &schemaObj); err != nil {
return nil, fmt.Errorf("unmarshalling schema failed: %w", err)
}
if len(fields) == 0 {
// A telegraf metric needs at least one field.
return nil, errors.New("number of fields is 0; unable to create metric")
}
// If measurement field name is specified in the configuration
// take value from that field and do not include it into fields or tags
name := ""
if p.MeasurementField != "" {
sField := p.MeasurementField
sMetric, err := internal.ToString(data[sField])
if err != nil {
p.Log.Warnf("Could not convert %v to string for metric name %q: %s", data[sField], sField, err.Error())
} else {
name = sMetric
}
}
// Now some fancy stuff to extract the measurement.
// If it's set in the configuration, use that.
if name == "" {
// If field name is not specified or field does not exist and
// metric name set in the configuration, use that.
name = p.Measurement
}
separator := "."
if name == "" {
// Try using the namespace defined in the schema. In case there
// is none, just use the schema's name definition.
nsStr, ok := schemaObj["namespace"].(string)
// namespace is optional
if !ok {
separator = ""
}
nStr, ok := schemaObj["name"].(string)
if !ok {
return nil, fmt.Errorf("could not determine name from schema %s", schema)
}
name = nsStr + separator + nStr
}
// Still don't have a name? Guess we should use the metric name if
// it's set.
if name == "" {
name = p.MetricName
}
// Nothing? Give up.
if name == "" {
return nil, errors.New("could not determine measurement name")
}
var timestamp time.Time
if p.Timestamp != "" {
rawTime := fmt.Sprintf("%v", data[p.Timestamp])
var err error
timestamp, err = internal.ParseTimestamp(p.TimestampFormat, rawTime, nil)
if err != nil {
return nil, fmt.Errorf("could not parse '%s' to '%s'", rawTime, p.TimestampFormat)
}
} else {
timestamp = time.Now()
}
return metric.New(name, tags, fields, timestamp), nil
}
func init() {
parsers.Add("avro",
func(defaultMetricName string) telegraf.Parser {
return &Parser{MetricName: defaultMetricName}
})
}

183
plugins/parsers/avro/parser_test.go Normal file
View file

@ -0,0 +1,183 @@
package avro
import (
"os"
"path/filepath"
"testing"
"github.com/linkedin/goavro/v2"
"github.com/stretchr/testify/require"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/plugins/inputs"
"github.com/influxdata/telegraf/plugins/inputs/file"
"github.com/influxdata/telegraf/plugins/parsers/influx"
"github.com/influxdata/telegraf/testutil"
)
func TestCases(t *testing.T) {
// Get all test-case directories
folders, err := os.ReadDir("testcases")
require.NoError(t, err)
// Make sure testdata contains data
require.NotEmpty(t, folders)
// Set up for file inputs
inputs.Add("file", func() telegraf.Input {
return &file.File{}
})
for _, f := range folders {
fname := f.Name()
testdataPath := filepath.Join("testcases", fname)
configFilename := filepath.Join(testdataPath, "telegraf.conf")
expectedFilename := filepath.Join(testdataPath, "expected.out")
expectedErrorFilename := filepath.Join(testdataPath, "expected.err")
t.Run(fname, func(t *testing.T) {
// Get parser to parse expected output
testdataParser := &influx.Parser{}
require.NoError(t, testdataParser.Init())
var expected []telegraf.Metric
if _, err := os.Stat(expectedFilename); err == nil {
var err error
expected, err = testutil.ParseMetricsFromFile(expectedFilename, testdataParser)
require.NoError(t, err)
}
// Read the expected errors if any
var expectedErrors []string
if _, err := os.Stat(expectedErrorFilename); err == nil {
var err error
expectedErrors, err = testutil.ParseLinesFromFile(expectedErrorFilename)
require.NoError(t, err)
require.NotEmpty(t, expectedErrors)
}
// Set up error catching
var acc testutil.Accumulator
var actualErrors []string
var actual []telegraf.Metric
// Configure the plugin
cfg := config.NewConfig()
err := cfg.LoadConfig(configFilename)
require.NoError(t, err)
for _, input := range cfg.Inputs {
require.NoError(t, input.Init())
if err := input.Gather(&acc); err != nil {
actualErrors = append(actualErrors, err.Error())
}
}
require.ElementsMatch(t, actualErrors, expectedErrors)
actual = acc.GetTelegrafMetrics()
// Process expected metrics and compare with resulting metrics
testutil.RequireMetricsEqual(t, expected, actual, testutil.IgnoreTime())
})
}
}
const benchmarkSchema = `
{
"namespace": "com.benchmark",
"name": "benchmark",
"type": "record",
"version": "1",
"fields": [
{"name": "value", "type": "float", "doc": ""},
{"name": "timestamp", "type": "long", "doc": ""},
{"name": "tags_platform", "type": "string", "doc": ""},
{"name": "tags_sdkver", "type": "string", "default": "", "doc": ""},
{"name": "source", "type": "string", "default": "", "doc": ""}
]
}
`
func BenchmarkParsing(b *testing.B) {
plugin := &Parser{
Format: "json",
Measurement: "benchmark",
Tags: []string{"tags_platform", "tags_sdkver", "source"},
Fields: []string{"value"},
Timestamp: "timestamp",
TimestampFormat: "unix",
Schema: benchmarkSchema,
}
require.NoError(b, plugin.Init())
benchmarkData, err := os.ReadFile(filepath.Join("testcases", "benchmark", "message.json"))
require.NoError(b, err)
b.ResetTimer()
for n := 0; n < b.N; n++ {
//nolint:errcheck // Benchmarking so skip the error check to avoid the unnecessary operations
plugin.Parse(benchmarkData)
}
}
func TestBenchmarkDataBinary(t *testing.T) {
plugin := &Parser{
Measurement: "benchmark",
Tags: []string{"tags_platform", "tags_sdkver", "source"},
Fields: []string{"value"},
Timestamp: "timestamp",
TimestampFormat: "unix",
Schema: benchmarkSchema,
}
require.NoError(t, plugin.Init())
benchmarkDir := filepath.Join("testcases", "benchmark")
// Read the expected valued from file
parser := &influx.Parser{}
require.NoError(t, parser.Init())
expected, err := testutil.ParseMetricsFromFile(filepath.Join(benchmarkDir, "expected.out"), parser)
require.NoError(t, err)
// Re-encode the benchmark data from JSON to binary format
jsonData, err := os.ReadFile(filepath.Join(benchmarkDir, "message.json"))
require.NoError(t, err)
codec, err := goavro.NewCodec(benchmarkSchema)
require.NoError(t, err)
native, _, err := codec.NativeFromTextual(jsonData)
require.NoError(t, err)
benchmarkData, err := codec.BinaryFromNative(nil, native)
require.NoError(t, err)
// Do the actual testing
actual, err := plugin.Parse(benchmarkData)
require.NoError(t, err)
testutil.RequireMetricsEqual(t, expected, actual, testutil.SortMetrics())
}
func BenchmarkParsingBinary(b *testing.B) {
plugin := &Parser{
Measurement: "benchmark",
Tags: []string{"tags_platform", "tags_sdkver", "source"},
Fields: []string{"value"},
Timestamp: "timestamp",
TimestampFormat: "unix",
Schema: benchmarkSchema,
}
require.NoError(b, plugin.Init())
// Re-encode the benchmark data from JSON to binary format
jsonData, err := os.ReadFile(filepath.Join("testcases", "benchmark", "message.json"))
require.NoError(b, err)
codec, err := goavro.NewCodec(benchmarkSchema)
require.NoError(b, err)
native, _, err := codec.NativeFromTextual(jsonData)
require.NoError(b, err)
benchmarkData, err := codec.BinaryFromNative(nil, native)
require.NoError(b, err)
for n := 0; n < b.N; n++ {
//nolint:errcheck // Benchmarking so skip the error check to avoid the unnecessary operations
plugin.Parse(benchmarkData)
}
}

134
plugins/parsers/avro/schema_registry.go Normal file
View file

@ -0,0 +1,134 @@
package avro
import (
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"os"
"sync"
"time"
"github.com/linkedin/goavro/v2"
)
type schemaAndCodec struct {
Schema string
Codec *goavro.Codec
}
type schemaRegistry struct {
url string
username string
password string
cache map[int]*schemaAndCodec
client *http.Client
mu sync.RWMutex
}
const schemaByID = "%s/schemas/ids/%d"
func newSchemaRegistry(addr, caCertPath string) (*schemaRegistry, error) {
var client *http.Client
var tlsCfg *tls.Config
if caCertPath != "" {
caCert, err := os.ReadFile(caCertPath)
if err != nil {
return nil, err
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
tlsCfg = &tls.Config{
RootCAs: caCertPool,
}
}
client = &http.Client{
Transport: &http.Transport{
TLSClientConfig: tlsCfg,
MaxIdleConns: 10,
IdleConnTimeout: 90 * time.Second,
},
}
u, err := url.Parse(addr)
if err != nil {
return nil, fmt.Errorf("parsing registry URL failed: %w", err)
}
var username, password string
if u.User != nil {
username = u.User.Username()
password, _ = u.User.Password()
}
registry := &schemaRegistry{
url: u.String(),
username: username,
password: password,
cache: make(map[int]*schemaAndCodec),
client: client,
}
return registry, nil
}
// Helper function to make managing lock easier
func (sr *schemaRegistry) getSchemaAndCodecFromCache(id int) (*schemaAndCodec, error) {
// Read-lock the cache map before access.
sr.mu.RLock()
defer sr.mu.RUnlock()
if v, ok := sr.cache[id]; ok {
return v, nil
}
return nil, fmt.Errorf("schema %d not in cache", id)
}
func (sr *schemaRegistry) getSchemaAndCodec(id int) (*schemaAndCodec, error) {
v, err := sr.getSchemaAndCodecFromCache(id)
if err == nil {
return v, nil
}
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf(schemaByID, sr.url, id), nil)
if err != nil {
return nil, err
}
if sr.username != "" {
req.SetBasicAuth(sr.username, sr.password)
}
resp, err := sr.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
var jsonResponse map[string]interface{}
if err := json.NewDecoder(resp.Body).Decode(&jsonResponse); err != nil {
return nil, err
}
schema, ok := jsonResponse["schema"]
if !ok {
return nil, errors.New("malformed response from schema registry: no 'schema' key")
}
schemaValue, ok := schema.(string)
if !ok {
return nil, fmt.Errorf("malformed response from schema registry: %v cannot be cast to string", schema)
}
codec, err := goavro.NewCodec(schemaValue)
if err != nil {
return nil, err
}
retval := &schemaAndCodec{Schema: schemaValue, Codec: codec}
// Lock the cache map before update.
sr.mu.Lock()
defer sr.mu.Unlock()
sr.cache[id] = retval
return retval, nil
}

1
plugins/parsers/avro/testcases/bad-timestamp-format/expected.err Normal file
View file

@ -0,0 +1 @@
could not instantiate parser: invalid timestamp format 'unix_ps'

0
plugins/parsers/avro/testcases/bad-timestamp-format/expected.out Normal file
View file

0
plugins/parsers/avro/testcases/bad-timestamp-format/message.avro Normal file
View file

29
plugins/parsers/avro/testcases/bad-timestamp-format/telegraf.conf Normal file
View file

@ -0,0 +1,29 @@
[[ inputs.file ]]
files = ["./testcases/bad-timestamp-format/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]
avro_timestamp = "timestamp"
avro_timestamp_format = "unix_ps"
avro_schema = '''
{
"type":"record",
"name":"Value",
"namespace":"com.example",
"fields":[
{
"name":"tag",
"type":"string"
},
{
"name":"field",
"type":"long"
},
{
"name":"timestamp",
"type":"long"
}
]
}
'''

1
plugins/parsers/avro/testcases/benchmark/expected.out Normal file
View file

@ -0,0 +1 @@
benchmark,source=myhost,tags_platform=python,tags_sdkver=3.11.5 value=5.0 1653643421000000000

7
plugins/parsers/avro/testcases/benchmark/message.json Normal file
View file

@ -0,0 +1,7 @@
{
"timestamp": 1653643421,
"value": 5,
"source": "myhost",
"tags_platform": "python",
"tags_sdkver": "3.11.5"
}

25
plugins/parsers/avro/testcases/benchmark/telegraf.conf Normal file
View file

@ -0,0 +1,25 @@
[[ inputs.file ]]
files = ["./testcases/benchmark/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "benchmark"
avro_tags = ["tags_platform", "tags_sdkver", "source"]
avro_fields = ["value"]
avro_timestamp = "timestamp"
avro_timestamp_format = "unix"
avro_schema = '''
{
"namespace": "com.benchmark",
"name": "benchmark",
"type": "record",
"version": "1",
"fields": [
{"name": "value", "type": "float", "doc": ""},
{"name": "timestamp", "type": "long", "doc": ""},
{"name": "tags_platform", "type": "string", "doc": ""},
{"name": "tags_sdkver", "type": "string", "default": "", "doc": ""},
{"name": "source", "type": "string", "default": "", "doc": ""}
]
}
'''

4
plugins/parsers/avro/testcases/config-both/expected.err Normal file
View file

@ -0,0 +1,4 @@
could not instantiate parser: exactly one of 'schema_registry' or 'schema' must be specified

0
plugins/parsers/avro/testcases/config-both/expected.out Normal file
View file

0
plugins/parsers/avro/testcases/config-both/message.avro Normal file
View file

28
plugins/parsers/avro/testcases/config-both/telegraf.conf Normal file
View file

@ -0,0 +1,28 @@
[[ inputs.file ]]
files = ["./testcases/config-both/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]
avro_schema_registry = "https://localhost:8081"
avro_schema = '''
{
"type":"record",
"name":"Value",
"namespace":"com.example",
"fields":[
{
"name":"tag",
"type":"string"
},
{
"name":"field",
"type":"long"
},
{
"name":"timestamp",
"type":"long"
}
]
}
'''

2
plugins/parsers/avro/testcases/config-neither/expected.err Normal file
View file

@ -0,0 +1,2 @@
could not instantiate parser: exactly one of 'schema_registry' or 'schema' must be specified

0
plugins/parsers/avro/testcases/config-neither/expected.out Normal file
View file

0
plugins/parsers/avro/testcases/config-neither/message.avro Normal file
View file

5
plugins/parsers/avro/testcases/config-neither/telegraf.conf Normal file
View file

@ -0,0 +1,5 @@
[[ inputs.file ]]
files = ["./testcases/config-neither/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]

1
plugins/parsers/avro/testcases/enum/expected.out Normal file
View file

@ -0,0 +1 @@
sensors,name=temperature value_int=42i,status="OK"

7
plugins/parsers/avro/testcases/enum/message.json Normal file
View file

@ -0,0 +1,7 @@
{
"name": "temperature",
"value": {
"int": 42
},
"status": "OK"
}

41
plugins/parsers/avro/testcases/enum/telegraf.conf Normal file
View file

@ -0,0 +1,41 @@
[[ inputs.file ]]
files = ["./testcases/enum/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "sensors"
avro_tags = ["name"]
avro_fields = ["value", "status"]
avro_field_separator = "_"
avro_schema = '''
{
"type": "record",
"name": "Metric",
"fields": [
{
"name": "name",
"type": "string"
},
{
"name": "value",
"type": [
"null",
"int",
"string"
]
},
{
"name": "status",
"type": {
"type": "enum",
"name": "Status",
"symbols": [
"UNKNOWN",
"OK",
"FAILURE"
]
}
}
]
}
'''

1
plugins/parsers/avro/testcases/json-array/expected.out Normal file
View file

@ -0,0 +1 @@
array,name=pi data_0=3,data_1=3.0999999046325684,data_2=3.140000104904175,data_3=3.1410000324249268 1682509200092000

5
plugins/parsers/avro/testcases/json-array/message.json Normal file
View file

@ -0,0 +1,5 @@
{
"statistics_collection_time": 1682509200092,
"data": [ 3, 3.1, 3.14, 3.141 ],
"name": "pi"
}

24
plugins/parsers/avro/testcases/json-array/telegraf.conf Normal file
View file

@ -0,0 +1,24 @@
[[ inputs.file ]]
files = ["./testcases/json-array/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "array"
avro_tags = ["name"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_fields = ["data"]
avro_field_separator = "_"
avro_schema = '''
{
"namespace": "constants",
"name": "classical",
"type": "record",
"version": "1",
"fields": [
{"name": "name", "type": "string"},
{"name": "data", "type": "array", "items": "float"},
{"name": "statistics_collection_time", "type": "long"}
]
}
'''

1
plugins/parsers/avro/testcases/json-format/expected.out Normal file
View file

@ -0,0 +1 @@
Switch,switch_wwn=10:00:50:EB:1A:0B:84:3A up_time=1166984904i,cpu_utilization=14.0,memory_utilization=20.0 1682509200092000

7
plugins/parsers/avro/testcases/json-format/message.json Normal file
View file

@ -0,0 +1,7 @@
{
"switch_wwn": "10:00:50:EB:1A:0B:84:3A",
"statistics_collection_time": 1682509200092,
"up_time": 1166984904,
"cpu_utilization": 14.0,
"memory_utilization": 20.0
}

25
plugins/parsers/avro/testcases/json-format/telegraf.conf Normal file
View file

@ -0,0 +1,25 @@
[[ inputs.file ]]
files = ["./testcases/json-format/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "Switch"
avro_tags = ["switch_wwn"]
avro_fields = ["up_time", "cpu_utilization", "memory_utilization"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_schema = '''
{
"namespace": "com.brocade.streaming",
"name": "fibrechannel_switch_statistics",
"type": "record",
"version": "1",
"fields": [
{"name": "switch_wwn", "type": "string", "doc": "WWN of the Physical Switch."},
{"name": "statistics_collection_time", "type": "long", "doc": "Epoch time when statistics is collected."},
{"name": "up_time", "type": "long", "doc": "Switch Up Time (in hundredths of a second)"},
{"name": "cpu_utilization", "type": "float", "default": 0, "doc": "CPU Utilization in %"},
{"name": "memory_utilization", "type": "float", "default": 0, "doc": "Memory Utilization in %"}
]
}
'''

1
plugins/parsers/avro/testcases/measurement_name_from_message/expected.out Normal file
View file

@ -0,0 +1 @@
cpu_load,Server=test_server Value=18.7 1694526986671

1
plugins/parsers/avro/testcases/measurement_name_from_message/message.avro Normal file
View file

@ -0,0 +1 @@
ÞæîšÑbtest_server33333³2@cpu_load

30
plugins/parsers/avro/testcases/measurement_name_from_message/telegraf.conf Normal file
View file

@ -0,0 +1,30 @@
[[ inputs.file ]]
files = ["./testcases/measurement_name_from_message/message.avro"]
data_format = "avro"
avro_measurement_field = "Measurement"
avro_tags = [ "Server" ]
avro_fields = [ "Value" ]
avro_schema = '''
{
"type": "record",
"name": "TestRecord",
"fields": [
{
"name": "ServerTs",
"type": "long"
},
{
"name": "Server",
"type": "string"
},
{
"name": "Value",
"type": "double"
},
{
"name": "Measurement",
"type": "string"
}
]
}
'''

1
plugins/parsers/avro/testcases/no-timestamp-format/expected.out Normal file
View file

@ -0,0 +1 @@
measurement,tag=test_tag field=19i,timestamp=1664296121000000i 1664296121000000

1
plugins/parsers/avro/testcases/no-timestamp-format/message.avro Normal file
View file

@ -0,0 +1 @@
test_tag&€<>¿±äêô

28
plugins/parsers/avro/testcases/no-timestamp-format/telegraf.conf Normal file
View file

@ -0,0 +1,28 @@
[[ inputs.file ]]
files = ["./testcases/no-timestamp-format/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]
avro_timestamp = "timestamp"
avro_schema = '''
{
"type":"record",
"name":"Value",
"namespace":"com.example",
"fields":[
{
"name":"tag",
"type":"string"
},
{
"name":"field",
"type":"long"
},
{
"name":"timestamp",
"type":"long"
}
]
}
'''

1
plugins/parsers/avro/testcases/supplied_timestamp/expected.out Normal file
View file

@ -0,0 +1 @@
measurement,tag=test_tag field=19i,timestamp=1664296121000000i 1664296121000000

1
plugins/parsers/avro/testcases/supplied_timestamp/message.avro Normal file
View file

@ -0,0 +1 @@
test_tag&€<>¿±äêô

28
plugins/parsers/avro/testcases/supplied_timestamp/telegraf.conf Normal file
View file

@ -0,0 +1,28 @@
[[ inputs.file ]]
files = ["./testcases/supplied_timestamp/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]
avro_timestamp = "timestamp"
avro_timestamp_format = "unix_us"
avro_schema = '''
{
"type":"record",
"name":"Value",
"namespace":"com.example",
"fields":[
{
"name":"tag",
"type":"string"
},
{
"name":"field",
"type":"long"
},
{
"name":"timestamp",
"type":"long"
}
]
}
'''

1
plugins/parsers/avro/testcases/supplied_timestamp_fields_specified/expected.out Normal file
View file

@ -0,0 +1 @@
measurement,tag=test_tag field=19i,timestamp=1664296121000000i 1664296121000000

1
plugins/parsers/avro/testcases/supplied_timestamp_fields_specified/message.avro Normal file
View file

@ -0,0 +1 @@
test_tag&€<>¿±äêô

29
plugins/parsers/avro/testcases/supplied_timestamp_fields_specified/telegraf.conf Normal file
View file

@ -0,0 +1,29 @@
[[ inputs.file ]]
files = ["./testcases/supplied_timestamp_fields_specified/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]
avro_fields = [ "field", "timestamp"]
avro_timestamp = "timestamp"
avro_timestamp_format = "unix_us"
avro_schema = '''
{
"type":"record",
"name":"Value",
"namespace":"com.example",
"fields":[
{
"name":"tag",
"type":"string"
},
{
"name":"field",
"type":"long"
},
{
"name":"timestamp",
"type":"long"
}
]
}
'''

1
plugins/parsers/avro/testcases/supplied_timestamp_fields_unspecified/expected.out Normal file
View file

@ -0,0 +1 @@
measurement,tag=test_tag field=19i 1664296121000000

1
plugins/parsers/avro/testcases/supplied_timestamp_fields_unspecified/message.avro Normal file
View file

@ -0,0 +1 @@
test_tag&€<>¿±äêô

23
plugins/parsers/avro/testcases/supplied_timestamp_fields_unspecified/telegraf.conf Normal file
View file

@ -0,0 +1,23 @@
[[ inputs.file ]]
files = ["./testcases/supplied_timestamp_fields_unspecified/message.avro"]
data_format = "avro"
avro_measurement = "measurement"
avro_tags = [ "tag" ]
avro_fields = [ "field" ]
avro_schema = '''
{
"type":"record",
"name":"Value",
"namespace":"com.example",
"fields":[
{
"name":"tag",
"type":"string"
},
{
"name":"field",
"type":"long"
}
]
}
'''

1
plugins/parsers/avro/testcases/union-any/expected.out Normal file
View file

@ -0,0 +1 @@
Switch,switch_wwn=10:00:50:EB:1A:0B:84:3A statistics_collection_time=1682509200092i,up_time=1166984904i,cpu_utilization=11i,memory_utilization=20.0 1682509200092000

11
plugins/parsers/avro/testcases/union-any/message.json Normal file
View file

@ -0,0 +1,11 @@
{
"switch_wwn": "10:00:50:EB:1A:0B:84:3A",
"statistics_collection_time": 1682509200092,
"up_time": 1166984904,
"cpu_utilization": {
"int": 11
},
"memory_utilization": {
"float": 20.0
}
}

26
plugins/parsers/avro/testcases/union-any/telegraf.conf Normal file
View file

@ -0,0 +1,26 @@
[[ inputs.file ]]
files = ["./testcases/union-any/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "Switch"
avro_tags = ["switch_wwn"]
avro_fields = ["up_time", "cpu_utilization", "memory_utilization", "statistics_collection_time"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_union_mode = "any"
avro_schema = '''
{
"namespace": "com.brocade.streaming",
"name": "fibrechannel_switch_statistics",
"type": "record",
"version": "1",
"fields": [
{"name": "switch_wwn", "type": "string", "doc": "WWN of the Physical Switch."},
{"name": "statistics_collection_time", "type": "long", "doc": "Epoch time when statistics is collected."},
{"name": "up_time", "type": "long", "doc": "Switch Up Time (in hundredths of a second)"},
{"name": "cpu_utilization", "type": ["null", "float", "int"], "default": null, "doc": "CPU Utilization in %"},
{"name": "memory_utilization", "type": ["null", "float"], "doc": "Memory Utilization in %"}
]
}
'''

1
plugins/parsers/avro/testcases/union-array/expected.out Normal file
View file

@ -0,0 +1 @@
array,name=pi data_0=3,data_1=3.0999999046325684,data_2=3.140000104904175,data_3=3.1410000324249268 1682509200092000

5
plugins/parsers/avro/testcases/union-array/message.json Normal file
View file

@ -0,0 +1,5 @@
{
"statistics_collection_time": 1682509200092,
"data": [ 3, 3.1, 3.14, 3.141 ],
"name": "pi"
}

25
plugins/parsers/avro/testcases/union-array/telegraf.conf Normal file
View file

@ -0,0 +1,25 @@
[[ inputs.file ]]
files = ["./testcases/union-array/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "array"
avro_tags = ["name"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_fields = ["data"]
avro_union_mode = "any"
avro_field_separator = "_"
avro_schema = '''
{
"namespace": "constants",
"name": "classical",
"type": "record",
"version": "1",
"fields": [
{"name": "name", "type": "string"},
{"name": "data", "type": "array", "items": "float"},
{"name": "statistics_collection_time", "type": "long"}
]
}
'''

1
plugins/parsers/avro/testcases/union-nullable-tag/expected.out Normal file
View file

@ -0,0 +1 @@
Switch,switch_wwn=10:00:50:EB:1A:0B:84:3A,some_union_in_a_tag=some_value statistics_collection_time=1682509200092i,up_time=1166984904i,memory_utilization=20.0 1682509200092000

14
plugins/parsers/avro/testcases/union-nullable-tag/message.json Normal file
View file

@ -0,0 +1,14 @@
{
"some_union_in_a_tag": {
"string": "some_value"
},
"switch_wwn": "10:00:50:EB:1A:0B:84:3A",
"statistics_collection_time": 1682509200092,
"up_time": 1166984904,
"cpu_utilization": {
"null": null
},
"memory_utilization": {
"float": 20.0
}
}

27
plugins/parsers/avro/testcases/union-nullable-tag/telegraf.conf Normal file
View file

@ -0,0 +1,27 @@
[[ inputs.file ]]
files = ["./testcases/union-nullable-tag/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "Switch"
avro_tags = ["switch_wwn", "some_union_in_a_tag"]
avro_fields = ["up_time", "cpu_utilization", "memory_utilization", "statistics_collection_time"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_union_mode = "nullable"
avro_schema = '''
{
"namespace": "com.brocade.streaming",
"name": "fibrechannel_switch_statistics",
"type": "record",
"version": "1",
"fields": [
{"name": "some_union_in_a_tag", "type": ["null", "string"], "default": null, "doc": "Some union that is used in a tag"},
{"name": "switch_wwn", "type": "string", "doc": "WWN of the Physical Switch."},
{"name": "statistics_collection_time", "type": "long", "doc": "Epoch time when statistics is collected."},
{"name": "up_time", "type": "long", "doc": "Switch Up Time (in hundredths of a second)"},
{"name": "cpu_utilization", "type": ["null","float"], "default": null, "doc": "CPU Utilization in %"},
{"name": "memory_utilization", "type": ["null", "float"], "default": null, "doc": "Memory Utilization in %"}
]
}
'''

1
plugins/parsers/avro/testcases/union-nullable/expected.out Normal file
View file

@ -0,0 +1 @@
Switch,switch_wwn=10:00:50:EB:1A:0B:84:3A statistics_collection_time=1682509200092i,up_time=1166984904i,memory_utilization=20.0 1682509200092000

11
plugins/parsers/avro/testcases/union-nullable/message.json Normal file
View file

@ -0,0 +1,11 @@
{
"switch_wwn": "10:00:50:EB:1A:0B:84:3A",
"statistics_collection_time": 1682509200092,
"up_time": 1166984904,
"cpu_utilization": {
"null": null
},
"memory_utilization": {
"float": 20.0
}
}

26
plugins/parsers/avro/testcases/union-nullable/telegraf.conf Normal file
View file

@ -0,0 +1,26 @@
[[ inputs.file ]]
files = ["./testcases/union-nullable/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "Switch"
avro_tags = ["switch_wwn"]
avro_fields = ["up_time", "cpu_utilization", "memory_utilization", "statistics_collection_time"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_union_mode = "nullable"
avro_schema = '''
{
"namespace": "com.brocade.streaming",
"name": "fibrechannel_switch_statistics",
"type": "record",
"version": "1",
"fields": [
{"name": "switch_wwn", "type": "string", "doc": "WWN of the Physical Switch."},
{"name": "statistics_collection_time", "type": "long", "doc": "Epoch time when statistics is collected."},
{"name": "up_time", "type": "long", "doc": "Switch Up Time (in hundredths of a second)"},
{"name": "cpu_utilization", "type": ["null","float"], "default": null, "doc": "CPU Utilization in %"},
{"name": "memory_utilization", "type": ["null", "float"], "default": null, "doc": "Memory Utilization in %"}
]
}
'''

1
plugins/parsers/avro/testcases/union/expected.out Normal file
View file

@ -0,0 +1 @@
Switch,switch_wwn=10:00:50:EB:1A:0B:84:3A statistics_collection_time=1682509200092i,up_time=1166984904i,memory_utilization_float=20.0 1682509200092000

11
plugins/parsers/avro/testcases/union/message.json Normal file
View file

@ -0,0 +1,11 @@
{
"switch_wwn": "10:00:50:EB:1A:0B:84:3A",
"statistics_collection_time": 1682509200092,
"up_time": 1166984904,
"cpu_utilization": {
"null": null
},
"memory_utilization": {
"float": 20.0
}
}

26
plugins/parsers/avro/testcases/union/telegraf.conf Normal file
View file

@ -0,0 +1,26 @@
[[ inputs.file ]]
files = ["./testcases/union/message.json"]
data_format = "avro"
avro_format = "json"
avro_measurement = "Switch"
avro_tags = ["switch_wwn"]
avro_fields = ["up_time", "cpu_utilization", "memory_utilization", "statistics_collection_time"]
avro_timestamp = "statistics_collection_time"
avro_timestamp_format = "unix_ms"
avro_field_separator = "_"
avro_schema = '''
{
"namespace": "com.brocade.streaming",
"name": "fibrechannel_switch_statistics",
"type": "record",
"version": "1",
"fields": [
{"name": "switch_wwn", "type": "string", "doc": "WWN of the Physical Switch."},
{"name": "statistics_collection_time", "type": "long", "doc": "Epoch time when statistics is collected."},
{"name": "up_time", "type": "long", "doc": "Switch Up Time (in hundredths of a second)"},
{"name": "cpu_utilization", "type": ["null", "float"], "default": null, "doc": "CPU Utilization in %"},
{"name": "memory_utilization", "type": ["null", "float"], "default": null, "doc": "Memory Utilization in %"}
]
}
'''