1
0
Fork 0

Adding upstream version 1.34.4.

Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-05-24 07:26:29 +02:00
parent e393c3af3f
commit 4978089aab
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions

View file

@ -0,0 +1,151 @@
# Apache IoTDB Output Plugin
This plugin writes metrics to an [Apache IoTDB][iotdb] instance, a database
for the Internet of Things, supporting session connection and data insertion.
⭐ Telegraf v1.24.0
🏷️ datastore
💻 all
[iotdb]: https://iotdb.apache.org
## Getting started
Before using this plugin, please configure the IP address, port number,
user name, password and other information of the database server,
as well as some data type conversion, time unit and other configurations.
Please see the [configuration section](#configuration) for an example
configuration.
## Metric Translation
IoTDB uses a different data format for metric data than telegraf. It is
important to note that depending on the metrics being written, the translation
may be lossy. This plugin translates to IoTDB format in the following ways:
### Unsigned Integers
IoTDB currently **DOES NOT support unsigned integer**.
There are three available options of converting uint64, which are specified by
setting `uint64_conversion`.
- `int64_clip`, default option. If an unsigned integer is greater than
`math.MaxInt64`, save it as `int64`; else save `math.MaxInt64`
(9223372036854775807).
- `int64`, force converting an unsigned integer to a`int64`,no mater
what the value it is. This option may lead to exception if the value is
greater than `int64`.
- `text`force converting an unsigned integer to a string, no mater what the
value it is.
### Time Precision
IoTDB supports a variety of time precision. You can specify which precision
you want using the `timestamp_precision` setting. Default is `nanosecond`.
Other options are `second`, `millisecond`, `microsecond`.
### Metadata (tags)
IoTDB uses a tree model for metadata while Telegraf uses a tag model
(see [InfluxDB-Protocol Adapter][InfluxDB-Protocol Adapter]).
There are two available options of converting tags, which are specified by
setting `convert_tags_to`:
- `fields`. Treat Tags as measurements. For each Key:Value in Tag,
convert them into Measurement, Value, DataType, which are supported in IoTDB.
- `device_id`, default option. Treat Tags as part of device id. Tags
constitute a subtree of `Name`.
For example, there is a metric:
```markdown
Name="root.sg.device", Tags={tag1="private", tag2="working"}, Fields={s1=100, s2="hello"}
```
- `fields`, result: `root.sg.device, s1=100, s2="hello", tag1="private", tag2="working"`
- `device_id`, result: `root.sg.device.private.working, s1=100, s2="hello"`
[InfluxDB-Protocol Adapter]: https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
## Secret-store support
This plugin supports secrets from secret-stores for the `username` and
`password` option.
See the [secret-store documentation][SECRETSTORE] for more details on how
to use them.
[SECRETSTORE]: ../../../docs/CONFIGURATION.md#secret-store-secrets
## Configuration
```toml @sample.conf
# Save metrics to an IoTDB Database
[[outputs.iotdb]]
## Configuration of IoTDB server connection
host = "127.0.0.1"
# port = "6667"
## Configuration of authentication
# user = "root"
# password = "root"
## Timeout to open a new session.
## A value of zero means no timeout.
# timeout = "5s"
## Configuration of type conversion for 64-bit unsigned int
## IoTDB currently DOES NOT support unsigned integers (version 13.x).
## 32-bit unsigned integers are safely converted into 64-bit signed integers by the plugin,
## however, this is not true for 64-bit values in general as overflows may occur.
## The following setting allows to specify the handling of 64-bit unsigned integers.
## Available values are:
## - "int64" -- convert to 64-bit signed integers and accept overflows
## - "int64_clip" -- convert to 64-bit signed integers and clip the values on overflow to 9,223,372,036,854,775,807
## - "text" -- convert to the string representation of the value
# uint64_conversion = "int64_clip"
## Configuration of TimeStamp
## TimeStamp is always saved in 64bits int. timestamp_precision specifies the unit of timestamp.
## Available value:
## "second", "millisecond", "microsecond", "nanosecond"(default)
# timestamp_precision = "nanosecond"
## Handling of tags
## Tags are not fully supported by IoTDB.
## A guide with suggestions on how to handle tags can be found here:
## https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html
##
## Available values are:
## - "fields" -- convert tags to fields in the measurement
## - "device_id" -- attach tags to the device ID
##
## For Example, a metric named "root.sg.device" with the tags `tag1: "private"` and `tag2: "working"` and
## fields `s1: 100` and `s2: "hello"` will result in the following representations in IoTDB
## - "fields" -- root.sg.device, s1=100, s2="hello", tag1="private", tag2="working"
## - "device_id" -- root.sg.device.private.working, s1=100, s2="hello"
# convert_tags_to = "device_id"
## Handling of unsupported characters
## Some characters in different versions of IoTDB are not supported in path name
## A guide with suggetions on valid paths can be found here:
## for iotdb 0.13.x -> https://iotdb.apache.org/UserGuide/V0.13.x/Reference/Syntax-Conventions.html#identifiers
## for iotdb 1.x.x and above -> https://iotdb.apache.org/UserGuide/V1.3.x/User-Manual/Syntax-Rule.html#identifier
##
## Available values are:
## - "1.0", "1.1", "1.2", "1.3" -- use backticks to enclose tags with forbidden characters
## such as @$#:[]{}() and space
## - "0.13" -- use backticks to enclose tags with forbidden characters
## such as space
## Keep this section commented if you don't want to sanitize the path
# sanitize_tag = "1.3"
```

View file

@ -0,0 +1,346 @@
//go:generate ../../../tools/readme_config_includer/generator
package iotdb
import (
_ "embed"
"errors"
"fmt"
"math"
"regexp"
"strconv"
"strings"
"time"
"github.com/apache/iotdb-client-go/client"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/internal/choice"
"github.com/influxdata/telegraf/plugins/outputs"
)
//go:embed sample.conf
var sampleConfig string
// matches any word that has a non valid backtick
// `word` <- doesn't match
// “word , `wo`rd` , `word , word` <- match
var forbiddenBacktick = regexp.MustCompile("^[^\x60].*?[\x60]+.*?[^\x60]$|^[\x60].*[\x60]+.*[\x60]$|^[\x60]+.*[^\x60]$|^[^\x60].*[\x60]+$")
var allowedBacktick = regexp.MustCompile("^[\x60].*[\x60]$")
type IoTDB struct {
Host string `toml:"host"`
Port string `toml:"port"`
User config.Secret `toml:"user"`
Password config.Secret `toml:"password"`
Timeout config.Duration `toml:"timeout"`
ConvertUint64To string `toml:"uint64_conversion"`
TimeStampUnit string `toml:"timestamp_precision"`
TreatTagsAs string `toml:"convert_tags_to"`
SanitizeTags string `toml:"sanitize_tag"`
Log telegraf.Logger `toml:"-"`
sanityRegex []*regexp.Regexp
session *client.Session
}
type recordsWithTags struct {
// IoTDB Records basic data struct
DeviceIDList []string
MeasurementsList [][]string
ValuesList [][]interface{}
DataTypesList [][]client.TSDataType
TimestampList []int64
// extra tags
TagsList [][]*telegraf.Tag
}
func (*IoTDB) SampleConfig() string {
return sampleConfig
}
// Init is for setup, and validating config.
func (s *IoTDB) Init() error {
if s.Timeout < 0 {
return errors.New("negative timeout")
}
if !choice.Contains(s.ConvertUint64To, []string{"int64", "int64_clip", "text"}) {
return fmt.Errorf("unknown 'uint64_conversion' method %q", s.ConvertUint64To)
}
if !choice.Contains(s.TimeStampUnit, []string{"second", "millisecond", "microsecond", "nanosecond"}) {
return fmt.Errorf("unknown 'timestamp_precision' method %q", s.TimeStampUnit)
}
if !choice.Contains(s.TreatTagsAs, []string{"fields", "device_id"}) {
return fmt.Errorf("unknown 'convert_tags_to' method %q", s.TreatTagsAs)
}
if s.User.Empty() {
s.User.Destroy()
s.User = config.NewSecret([]byte("root"))
}
if s.Password.Empty() {
s.Password.Destroy()
s.Password = config.NewSecret([]byte("root"))
}
switch s.SanitizeTags {
case "0.13":
matchUnsupportedCharacter := regexp.MustCompile("[^0-9a-zA-Z_:@#${}\x60]")
regex := []*regexp.Regexp{matchUnsupportedCharacter}
s.sanityRegex = append(s.sanityRegex, regex...)
// from version 1.x.x IoTDB changed the allowed keys in nodes
case "1.0", "1.1", "1.2", "1.3":
matchUnsupportedCharacter := regexp.MustCompile("[^0-9a-zA-Z_\x60]")
matchNumericString := regexp.MustCompile(`^\d+$`)
regex := []*regexp.Regexp{matchUnsupportedCharacter, matchNumericString}
s.sanityRegex = append(s.sanityRegex, regex...)
}
s.Log.Info("Initialization completed.")
return nil
}
func (s *IoTDB) Connect() error {
username, err := s.User.Get()
if err != nil {
return fmt.Errorf("getting username failed: %w", err)
}
password, err := s.Password.Get()
if err != nil {
username.Destroy()
return fmt.Errorf("getting password failed: %w", err)
}
defer password.Destroy()
sessionConf := &client.Config{
Host: s.Host,
Port: s.Port,
UserName: username.String(),
Password: password.String(),
}
username.Destroy()
password.Destroy()
var ss = client.NewSession(sessionConf)
s.session = &ss
timeoutInMs := int(time.Duration(s.Timeout).Milliseconds())
if err := s.session.Open(false, timeoutInMs); err != nil {
return fmt.Errorf("connecting to %s:%s failed: %w", s.Host, s.Port, err)
}
return nil
}
func (s *IoTDB) Close() error {
return s.session.Close()
}
// Write should write immediately to the output, and not buffer writes
// (Telegraf manages the buffer for you). Returning an error will fail this
// batch of writes and the entire batch will be retried automatically.
func (s *IoTDB) Write(metrics []telegraf.Metric) error {
// Convert Metrics to Records with Tags
rwt, err := s.convertMetricsToRecordsWithTags(metrics)
if err != nil {
return err
}
// Write to client.
// If first writing fails, the client will automatically retry three times. If all fail, it returns an error.
if err := s.writeRecordsWithTags(rwt); err != nil {
return fmt.Errorf("write failed: %w", err)
}
return nil
}
// Find out data type of the value and return it's id in TSDataType, and convert it if necessary.
func (s *IoTDB) getDataTypeAndValue(value interface{}) (client.TSDataType, interface{}) {
switch v := value.(type) {
case int32:
return client.INT32, v
case int64:
return client.INT64, v
case uint32:
return client.INT64, int64(v)
case uint64:
switch s.ConvertUint64To {
case "int64_clip":
if v <= uint64(math.MaxInt64) {
return client.INT64, int64(v)
}
return client.INT64, int64(math.MaxInt64)
case "int64":
return client.INT64, int64(v)
case "text":
return client.TEXT, strconv.FormatUint(v, 10)
default:
return client.UNKNOWN, int64(0)
}
case float64:
return client.DOUBLE, v
case string:
return client.TEXT, v
case bool:
return client.BOOLEAN, v
default:
return client.UNKNOWN, int64(0)
}
}
// convert Timestamp Unit according to config
func (s *IoTDB) convertTimestampOfMetric(m telegraf.Metric) (int64, error) {
switch s.TimeStampUnit {
case "second":
return m.Time().Unix(), nil
case "millisecond":
return m.Time().UnixMilli(), nil
case "microsecond":
return m.Time().UnixMicro(), nil
case "nanosecond":
return m.Time().UnixNano(), nil
default:
return 0, fmt.Errorf("unknown timestamp_precision %q", s.TimeStampUnit)
}
}
// convert Metrics to Records with tags
func (s *IoTDB) convertMetricsToRecordsWithTags(metrics []telegraf.Metric) (*recordsWithTags, error) {
timestampList := make([]int64, 0, len(metrics))
deviceidList := make([]string, 0, len(metrics))
measurementsList := make([][]string, 0, len(metrics))
valuesList := make([][]interface{}, 0, len(metrics))
dataTypesList := make([][]client.TSDataType, 0, len(metrics))
tagsList := make([][]*telegraf.Tag, 0, len(metrics))
for _, metric := range metrics {
// write `metric` to the output sink here
var tags []*telegraf.Tag
tags = append(tags, metric.TagList()...)
// deal with basic parameter
var keys []string
var values []interface{}
var dataTypes []client.TSDataType
for _, field := range metric.FieldList() {
datatype, value := s.getDataTypeAndValue(field.Value)
if datatype == client.UNKNOWN {
return nil, fmt.Errorf("datatype of %q is unknown, values: %v", field.Key, field.Value)
}
keys = append(keys, field.Key)
values = append(values, value)
dataTypes = append(dataTypes, datatype)
}
// Convert timestamp into specified unit
ts, err := s.convertTimestampOfMetric(metric)
if err != nil {
return nil, err
}
timestampList = append(timestampList, ts)
// append all metric data of this record to lists
deviceidList = append(deviceidList, metric.Name())
measurementsList = append(measurementsList, keys)
valuesList = append(valuesList, values)
dataTypesList = append(dataTypesList, dataTypes)
tagsList = append(tagsList, tags)
}
rwt := &recordsWithTags{
DeviceIDList: deviceidList,
MeasurementsList: measurementsList,
ValuesList: valuesList,
DataTypesList: dataTypesList,
TimestampList: timestampList,
TagsList: tagsList,
}
return rwt, nil
}
// checks is the tag contains any IoTDB invalid character
func (s *IoTDB) validateTag(tag string) (string, error) {
// IoTDB uses "root" as a keyword and can be called only at the start of the path
if tag == "root" {
return "", errors.New("cannot use 'root' as tag")
} else if forbiddenBacktick.MatchString(tag) { // returns an error if the backsticks are used in an inappropriate way
return "", errors.New("cannot use ` in tag names")
} else if allowedBacktick.MatchString(tag) { // if the tag in already enclosed in tags returns the tag
return tag, nil
}
// loops through all the regex patterns and if one
// pattern matches returns the tag between `
for _, regex := range s.sanityRegex {
if regex.MatchString(tag) {
return "`" + tag + "`", nil
}
}
return tag, nil
}
// modify recordsWithTags according to 'TreatTagsAs' Configuration
func (s *IoTDB) modifyRecordsWithTags(rwt *recordsWithTags) error {
switch s.TreatTagsAs {
case "fields":
// method 1: treat Tag(Key:Value) as measurement
for index, tags := range rwt.TagsList { // for each record
for _, tag := range tags { // for each tag of this record, append it's Key:Value to measurements
datatype, value := s.getDataTypeAndValue(tag.Value)
if datatype == client.UNKNOWN {
return fmt.Errorf("datatype of %q is unknown, values: %v", tag.Key, value)
}
rwt.MeasurementsList[index] = append(rwt.MeasurementsList[index], tag.Key)
rwt.ValuesList[index] = append(rwt.ValuesList[index], value)
rwt.DataTypesList[index] = append(rwt.DataTypesList[index], datatype)
}
}
return nil
case "device_id":
// method 2: treat Tag(Key:Value) as subtree of device id
for index, tags := range rwt.TagsList { // for each record
topic := []string{rwt.DeviceIDList[index]}
for _, tag := range tags { // for each tag, append it's Value
tagValue, err := s.validateTag(tag.Value) // validates tag
if err != nil {
return err
}
topic = append(topic, tagValue)
}
rwt.DeviceIDList[index] = strings.Join(topic, ".")
}
return nil
default:
// something go wrong. This configuration should have been checked in func Init().
return fmt.Errorf("unknown 'convert_tags_to' method: %q", s.TreatTagsAs)
}
}
// Write records with tags to IoTDB server
func (s *IoTDB) writeRecordsWithTags(rwt *recordsWithTags) error {
// deal with tags
if err := s.modifyRecordsWithTags(rwt); err != nil {
return err
}
// write to IoTDB server
status, err := s.session.InsertRecords(rwt.DeviceIDList, rwt.MeasurementsList,
rwt.DataTypesList, rwt.ValuesList, rwt.TimestampList)
if status != nil {
if verifyResult := client.VerifySuccess(status); verifyResult != nil {
s.Log.Debug(verifyResult)
}
}
return err
}
func init() {
outputs.Add("iotdb", func() telegraf.Output { return newIoTDB() })
}
// create a new IoTDB struct with default values.
func newIoTDB() *IoTDB {
return &IoTDB{
Host: "localhost",
Port: "6667",
Timeout: config.Duration(time.Second * 5),
ConvertUint64To: "int64_clip",
TimeStampUnit: "nanosecond",
TreatTagsAs: "device_id",
}
}

View file

@ -0,0 +1,641 @@
package iotdb
import (
"math"
"strconv"
"testing"
"time"
"github.com/apache/iotdb-client-go/client"
"github.com/docker/go-connections/nat"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go/wait"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/metric"
"github.com/influxdata/telegraf/testutil"
)
// newMetricWithOrderedFields creates new Metric and makes sure fields are in
// order. This is required to define the expected output where the field order
// needs to be defines.
func newMetricWithOrderedFields(
name string,
tags []telegraf.Tag,
fields []telegraf.Field,
timestamp time.Time,
) telegraf.Metric {
m := metric.New(name, map[string]string{}, map[string]interface{}{}, timestamp)
for _, tag := range tags {
m.AddTag(tag.Key, tag.Value)
}
for _, field := range fields {
m.AddField(field.Key, field.Value)
}
return m
}
func TestInitInvalid(t *testing.T) {
tests := []struct {
name string
plugin *IoTDB
expected string
}{
{
name: "empty tag-conversion",
plugin: func() *IoTDB {
s := newIoTDB()
s.TreatTagsAs = ""
s.Log = &testutil.Logger{}
return s
}(),
expected: `unknown 'convert_tags_to' method ""`,
},
{
name: "empty uint-conversion",
plugin: func() *IoTDB {
s := newIoTDB()
s.ConvertUint64To = ""
s.Log = &testutil.Logger{}
return s
}(),
expected: `unknown 'uint64_conversion' method ""`,
},
{
name: "empty timestamp precision",
plugin: func() *IoTDB {
s := newIoTDB()
s.TimeStampUnit = ""
s.Log = &testutil.Logger{}
return s
}(),
expected: `unknown 'timestamp_precision' method ""`,
},
{
name: "invalid tag-conversion",
plugin: func() *IoTDB {
s := newIoTDB()
s.TreatTagsAs = "garbage"
s.Log = &testutil.Logger{}
return s
}(),
expected: `unknown 'convert_tags_to' method "garbage"`,
},
{
name: "invalid uint-conversion",
plugin: func() *IoTDB {
s := newIoTDB()
s.ConvertUint64To = "garbage"
s.Log = &testutil.Logger{}
return s
}(),
expected: `unknown 'uint64_conversion' method "garbage"`,
},
{
name: "invalid timestamp precision",
plugin: func() *IoTDB {
s := newIoTDB()
s.TimeStampUnit = "garbage"
s.Log = &testutil.Logger{}
return s
}(),
expected: `unknown 'timestamp_precision' method "garbage"`,
},
{
name: "negative timeout",
plugin: func() *IoTDB {
s := newIoTDB()
s.Timeout = config.Duration(time.Second * -5)
s.Log = &testutil.Logger{}
return s
}(),
expected: `negative timeout`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
require.EqualError(t, tt.plugin.Init(), tt.expected)
})
}
}
// Test Metric conversion, which means testing function `convertMetricsToRecordsWithTags`
func TestMetricConversionToRecordsWithTags(t *testing.T) {
var testTimestamp = time.Date(2022, time.July, 20, 12, 25, 33, 44, time.UTC)
tests := []struct {
name string
plugin *IoTDB
expected recordsWithTags
metrics []telegraf.Metric
}{
{
name: "default config",
plugin: func() *IoTDB { s := newIoTDB(); return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.fan", "root.computer.fan", "root.computer.keyboard"},
MeasurementsList: [][]string{
{"temperature", "counter"},
{"counter", "temperature"},
{"temperature", "counter", "unsigned_big", "string", "bool", "int_text"},
},
ValuesList: [][]interface{}{
{float64(42.55), int64(987654321)},
{int64(123456789), float64(56.24)},
{float64(30.33), int64(123456789), int64(math.MaxInt64), "Made in China.", bool(false), "123456789011"},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64},
{client.INT64, client.DOUBLE},
{client.DOUBLE, client.INT64, client.INT64, client.TEXT, client.BOOLEAN, client.TEXT},
},
TimestampList: []int64{testTimestamp.UnixNano(), testTimestamp.UnixNano(), testTimestamp.UnixNano()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.fan",
[]telegraf.Tag{
{Key: "price", Value: "expensive"},
{Key: "owner", Value: "cpu"},
},
[]telegraf.Field{
{Key: "temperature", Value: float64(42.55)},
{Key: "counter", Value: int64(987654321)},
},
testTimestamp,
),
newMetricWithOrderedFields(
"root.computer.fan",
[]telegraf.Tag{ // same keys in different order
{Key: "owner", Value: "gpu"},
{Key: "price", Value: "cheap"},
},
[]telegraf.Field{ // same keys in different order
{Key: "counter", Value: int64(123456789)},
{Key: "temperature", Value: float64(56.24)},
},
testTimestamp,
),
newMetricWithOrderedFields(
"root.computer.keyboard",
nil,
[]telegraf.Field{
{Key: "temperature", Value: float64(30.33)},
{Key: "counter", Value: int64(123456789)},
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
{Key: "string", Value: "Made in China."},
{Key: "bool", Value: bool(false)},
{Key: "int_text", Value: "123456789011"},
},
testTimestamp,
),
},
},
{
name: "unsigned int to text",
plugin: func() *IoTDB { cli002 := newIoTDB(); cli002.ConvertUint64To = "text"; return cli002 }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.uint_to_text"},
MeasurementsList: [][]string{{"unsigned_big"}},
ValuesList: [][]interface{}{{strconv.FormatUint(uint64(math.MaxInt64+1000), 10)}},
DataTypesList: [][]client.TSDataType{{client.TEXT}},
TimestampList: []int64{testTimestamp.UnixNano()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.uint_to_text",
nil,
[]telegraf.Field{
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
},
testTimestamp,
),
},
},
{
name: "unsigned int to int with overflow",
plugin: func() *IoTDB { cli002 := newIoTDB(); cli002.ConvertUint64To = "int64"; return cli002 }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.overflow"},
MeasurementsList: [][]string{{"unsigned_big"}},
ValuesList: [][]interface{}{{int64(-9223372036854774809)}},
DataTypesList: [][]client.TSDataType{{client.INT64}},
TimestampList: []int64{testTimestamp.UnixNano()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.overflow",
nil,
[]telegraf.Field{
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
},
testTimestamp,
),
},
},
{
name: "second timestamp precision",
plugin: func() *IoTDB { s := newIoTDB(); s.TimeStampUnit = "second"; return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.second"},
MeasurementsList: [][]string{{"unsigned_big"}},
ValuesList: [][]interface{}{{int64(math.MaxInt64)}},
DataTypesList: [][]client.TSDataType{{client.INT64}},
TimestampList: []int64{testTimestamp.Unix()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.second",
nil,
[]telegraf.Field{
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
},
testTimestamp,
),
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.plugin.Log = &testutil.Logger{}
require.NoError(t, tt.plugin.Init())
actual, err := tt.plugin.convertMetricsToRecordsWithTags(tt.metrics)
require.NoError(t, err)
// Ignore the tags-list for comparison
actual.TagsList = nil
expected := tt.expected
require.EqualValues(t, &expected, actual)
})
}
}
// Test tag sanitize
func TestTagSanitization(t *testing.T) {
tests := []struct {
name string
plugin *IoTDB
expected []string
input []string
}{
{ // don't sanitize tags containing UnsopportedCharacter on IoTDB V1.3
name: "Don't Sanitize Tags",
plugin: func() *IoTDB { s := newIoTDB(); s.SanitizeTags = "1.3"; return s }(),
expected: []string{"word", "`word`", "word_"},
input: []string{"word", "`word`", "word_"},
},
{ // sanitize tags containing UnsupportedCharacter on IoTDB V1.3 enclosing them in backticks
name: "Sanitize Tags",
plugin: func() *IoTDB { s := newIoTDB(); s.SanitizeTags = "1.3"; return s }(),
expected: []string{"`wo rd`", "`@`", "`$`", "`#`", "`:`", "`{`", "`}`", "`1`", "`1234`"},
input: []string{"wo rd", "@", "$", "#", ":", "{", "}", "1", "1234"},
},
{ // test on forbidden word and forbidden syntax
name: "Errors",
plugin: func() *IoTDB { s := newIoTDB(); s.SanitizeTags = "1.3"; return s }(),
expected: []string{"", ""},
input: []string{"root", "wo`rd"},
},
{
name: "Don't Sanitize Tags",
plugin: func() *IoTDB { s := newIoTDB(); s.SanitizeTags = "0.13"; return s }(),
expected: []string{"word", "`word`", "word_", "@", "$", "#", ":", "{", "}"},
input: []string{"word", "`word`", "word_", "@", "$", "#", ":", "{", "}"},
},
{ // sanitize tags containing UnsupportedCharacter on IoTDB V0.13 enclosing them in backticks
name: "Sanitize Tags",
plugin: func() *IoTDB { s := newIoTDB(); s.SanitizeTags = "0.13"; return s }(),
expected: []string{"`wo rd`", "`\\`"},
input: []string{"wo rd", "\\"},
},
{ // test on forbidden word and forbidden syntax on IoTDB V0.13
name: "Errors",
plugin: func() *IoTDB { s := newIoTDB(); s.SanitizeTags = "0.13"; return s }(),
expected: []string{"", ""},
input: []string{"root", "wo`rd"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.plugin.Log = &testutil.Logger{}
require.NoError(t, tt.plugin.Init())
actuals := make([]string, 0, len(tt.input))
for _, input := range tt.input {
//nolint:errcheck // error cases handled by expected vs actual comparison
actual, _ := tt.plugin.validateTag(input)
actuals = append(actuals, actual)
}
require.EqualValues(t, tt.expected, actuals)
})
}
}
// Test tags handling, which means testing function `modifyRecordsWithTags`
func TestTagsHandling(t *testing.T) {
var testTimestamp = time.Date(2022, time.July, 20, 12, 25, 33, 44, time.UTC)
tests := []struct {
name string
plugin *IoTDB
expected recordsWithTags
input recordsWithTags
}{
{ // treat tags as fields. And input Tags are NOT in order.
name: "treat tags as fields",
plugin: func() *IoTDB { s := newIoTDB(); s.TreatTagsAs = "fields"; return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.fields"},
MeasurementsList: [][]string{{"temperature", "counter", "owner", "price"}},
ValuesList: [][]interface{}{
{float64(42.55), int64(987654321), "cpu", "expensive"},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64, client.TEXT, client.TEXT},
},
TimestampList: []int64{testTimestamp.UnixNano()},
},
input: recordsWithTags{
DeviceIDList: []string{"root.computer.fields"},
MeasurementsList: [][]string{{"temperature", "counter"}},
ValuesList: [][]interface{}{
{float64(42.55), int64(987654321)},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64},
},
TimestampList: []int64{testTimestamp.UnixNano()},
TagsList: [][]*telegraf.Tag{{
{Key: "owner", Value: "cpu"},
{Key: "price", Value: "expensive"},
}},
},
},
{ // treat tags as device IDs. And input Tags are in order.
name: "treat tags as device IDs",
plugin: func() *IoTDB { s := newIoTDB(); s.TreatTagsAs = "device_id"; return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.deviceID.cpu.expensive"},
MeasurementsList: [][]string{{"temperature", "counter"}},
ValuesList: [][]interface{}{
{float64(42.55), int64(987654321)},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64},
},
TimestampList: []int64{testTimestamp.UnixNano()},
},
input: recordsWithTags{
DeviceIDList: []string{"root.computer.deviceID"},
MeasurementsList: [][]string{{"temperature", "counter"}},
ValuesList: [][]interface{}{
{float64(42.55), int64(987654321)},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64},
},
TimestampList: []int64{testTimestamp.UnixNano()},
TagsList: [][]*telegraf.Tag{{
{Key: "owner", Value: "cpu"},
{Key: "price", Value: "expensive"},
}},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
input := tt.input
tt.plugin.Log = &testutil.Logger{}
require.NoError(t, tt.plugin.Init())
require.NoError(t, tt.plugin.modifyRecordsWithTags(&input))
// Ignore the tags-list for comparison
tt.input.TagsList = nil
require.EqualValues(t, tt.expected, tt.input)
})
}
}
// Test entire Metric conversion, from metrics to records which are ready to insert
func TestEntireMetricConversion(t *testing.T) {
var testTimestamp = time.Date(2022, time.July, 20, 12, 25, 33, 44, time.UTC)
tests := []struct {
name string
plugin *IoTDB
expected recordsWithTags
metrics []telegraf.Metric
requireEqual bool
}{
{
name: "default config",
plugin: func() *IoTDB { s := newIoTDB(); return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.screen.high.LED"},
MeasurementsList: [][]string{
{"temperature", "counter", "unsigned_big", "string", "bool", "int_text"},
},
ValuesList: [][]interface{}{
{float64(30.33), int64(123456789), int64(math.MaxInt64), "Made in China.", bool(false), "123456789011"},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64, client.INT64, client.TEXT, client.BOOLEAN, client.TEXT},
},
TimestampList: []int64{testTimestamp.UnixNano()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.screen",
[]telegraf.Tag{
{Key: "brightness", Value: "high"},
{Key: "type", Value: "LED"},
},
[]telegraf.Field{
{Key: "temperature", Value: float64(30.33)},
{Key: "counter", Value: int64(123456789)},
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
{Key: "string", Value: "Made in China."},
{Key: "bool", Value: bool(false)},
{Key: "int_text", Value: "123456789011"},
},
testTimestamp,
),
},
requireEqual: true,
},
{
name: "wrong order of tags",
plugin: func() *IoTDB { s := newIoTDB(); return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.screen.LED.high"},
MeasurementsList: [][]string{
{"temperature", "counter", "unsigned_big", "string", "bool", "int_text"},
},
ValuesList: [][]interface{}{
{float64(30.33), int64(123456789), int64(math.MaxInt64), "Made in China.", bool(false), "123456789011"},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64, client.INT64, client.TEXT, client.BOOLEAN, client.TEXT},
},
TimestampList: []int64{testTimestamp.UnixNano()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.screen",
[]telegraf.Tag{
{Key: "brightness", Value: "high"},
{Key: "type", Value: "LED"},
},
[]telegraf.Field{
{Key: "temperature", Value: float64(30.33)},
{Key: "counter", Value: int64(123456789)},
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
{Key: "string", Value: "Made in China."},
{Key: "bool", Value: bool(false)},
{Key: "int_text", Value: "123456789011"},
},
testTimestamp,
),
},
requireEqual: false,
},
{
name: "wrong order of tags",
plugin: func() *IoTDB { s := newIoTDB(); return s }(),
expected: recordsWithTags{
DeviceIDList: []string{"root.computer.screen.LED.high"},
MeasurementsList: [][]string{
{"temperature", "counter"},
},
ValuesList: [][]interface{}{
{float64(30.33), int64(123456789)},
},
DataTypesList: [][]client.TSDataType{
{client.DOUBLE, client.INT64},
},
TimestampList: []int64{testTimestamp.UnixNano()},
},
metrics: []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.screen",
[]telegraf.Tag{
{Key: "brightness", Value: "high"},
{Key: "type", Value: "LED"},
},
[]telegraf.Field{
{Key: "temperature", Value: float64(30.33)},
{Key: "counter", Value: int64(123456789)},
},
testTimestamp,
),
},
requireEqual: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.plugin.Log = &testutil.Logger{}
require.NoError(t, tt.plugin.Init())
actual, err := tt.plugin.convertMetricsToRecordsWithTags(tt.metrics)
require.NoError(t, err)
require.NoError(t, tt.plugin.modifyRecordsWithTags(actual))
// Ignore the tags-list for comparison
actual.TagsList = nil
expected := tt.expected
if tt.requireEqual {
require.EqualValues(t, &expected, actual)
} else {
require.NotEqualValues(t, &expected, actual)
}
})
}
}
// Start a container and do integration test.
func TestIntegrationInserts(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
const iotdbPort = "6667"
container := testutil.Container{
Image: "apache/iotdb:0.13.0-node",
ExposedPorts: []string{iotdbPort},
WaitingFor: wait.ForAll(
wait.ForListeningPort(nat.Port(iotdbPort)),
wait.ForLog("IoTDB has started."),
),
}
err := container.Start()
require.NoError(t, err, "failed to start IoTDB container")
defer container.Terminate()
t.Logf("Container Address:%q, ExposedPorts:[%v:%v]", container.Address, container.Ports[iotdbPort], iotdbPort)
// create a client and tests two groups of insertion
testClient := &IoTDB{
Host: container.Address,
Port: container.Ports[iotdbPort],
User: config.NewSecret([]byte("root")),
Password: config.NewSecret([]byte("root")),
Timeout: config.Duration(time.Second * 5),
ConvertUint64To: "int64_clip",
TimeStampUnit: "nanosecond",
TreatTagsAs: "device_id",
}
testClient.Log = &testutil.Logger{}
// generate Metrics to input
metrics := []telegraf.Metric{
newMetricWithOrderedFields(
"root.computer.unsigned_big",
nil,
[]telegraf.Field{
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
},
time.Now(),
),
newMetricWithOrderedFields(
"root.computer.fan",
[]telegraf.Tag{
{Key: "price", Value: "expensive"},
{Key: "owner", Value: "cpu"},
},
[]telegraf.Field{
{Key: "temperature", Value: float64(42.55)},
{Key: "counter", Value: int64(987654321)},
},
time.Now(),
),
newMetricWithOrderedFields(
"root.computer.fan",
[]telegraf.Tag{ // same keys in different order
{Key: "owner", Value: "gpu"},
{Key: "price", Value: "cheap"},
},
[]telegraf.Field{ // same keys in different order
{Key: "counter", Value: int64(123456789)},
{Key: "temperature", Value: float64(56.24)},
},
time.Now(),
),
newMetricWithOrderedFields(
"root.computer.keyboard",
nil,
[]telegraf.Field{
{Key: "temperature", Value: float64(30.33)},
{Key: "counter", Value: int64(123456789)},
{Key: "unsigned_big", Value: uint64(math.MaxInt64 + 1000)},
{Key: "string", Value: "Made in China."},
{Key: "bool", Value: bool(false)},
{Key: "int_text", Value: "123456789011"},
},
time.Now(),
),
}
require.NoError(t, testClient.Connect())
require.NoError(t, testClient.Write(metrics))
require.NoError(t, testClient.Close())
}

View file

@ -0,0 +1,59 @@
# Save metrics to an IoTDB Database
[[outputs.iotdb]]
## Configuration of IoTDB server connection
host = "127.0.0.1"
# port = "6667"
## Configuration of authentication
# user = "root"
# password = "root"
## Timeout to open a new session.
## A value of zero means no timeout.
# timeout = "5s"
## Configuration of type conversion for 64-bit unsigned int
## IoTDB currently DOES NOT support unsigned integers (version 13.x).
## 32-bit unsigned integers are safely converted into 64-bit signed integers by the plugin,
## however, this is not true for 64-bit values in general as overflows may occur.
## The following setting allows to specify the handling of 64-bit unsigned integers.
## Available values are:
## - "int64" -- convert to 64-bit signed integers and accept overflows
## - "int64_clip" -- convert to 64-bit signed integers and clip the values on overflow to 9,223,372,036,854,775,807
## - "text" -- convert to the string representation of the value
# uint64_conversion = "int64_clip"
## Configuration of TimeStamp
## TimeStamp is always saved in 64bits int. timestamp_precision specifies the unit of timestamp.
## Available value:
## "second", "millisecond", "microsecond", "nanosecond"(default)
# timestamp_precision = "nanosecond"
## Handling of tags
## Tags are not fully supported by IoTDB.
## A guide with suggestions on how to handle tags can be found here:
## https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html
##
## Available values are:
## - "fields" -- convert tags to fields in the measurement
## - "device_id" -- attach tags to the device ID
##
## For Example, a metric named "root.sg.device" with the tags `tag1: "private"` and `tag2: "working"` and
## fields `s1: 100` and `s2: "hello"` will result in the following representations in IoTDB
## - "fields" -- root.sg.device, s1=100, s2="hello", tag1="private", tag2="working"
## - "device_id" -- root.sg.device.private.working, s1=100, s2="hello"
# convert_tags_to = "device_id"
## Handling of unsupported characters
## Some characters in different versions of IoTDB are not supported in path name
## A guide with suggetions on valid paths can be found here:
## for iotdb 0.13.x -> https://iotdb.apache.org/UserGuide/V0.13.x/Reference/Syntax-Conventions.html#identifiers
## for iotdb 1.x.x and above -> https://iotdb.apache.org/UserGuide/V1.3.x/User-Manual/Syntax-Rule.html#identifier
##
## Available values are:
## - "1.0", "1.1", "1.2", "1.3" -- use backticks to enclose tags with forbidden characters
## such as @$#:[]{}() and space
## - "0.13" -- use backticks to enclose tags with forbidden characters
## such as space
## Keep this section commented if you don't want to sanitize the path
# sanitize_tag = "1.3"