influx/telegraf
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.34.4.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-05-24 07:26:29 +02:00
parent e393c3af3f
commit 4978089aab
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

145
plugins/aggregators/quantile/README.md Normal file
View file

@ -0,0 +1,145 @@
# Quantile Aggregator Plugin
This plugin aggregates each numeric field per metric into the specified
quantiles and emits the quantiles every `period`. Different aggregation
algorithms are supported with varying accuracy and limitations.
⭐ Telegraf v1.18.0
🏷️ statistics
💻 all
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
## Configuration
```toml @sample.conf
# Keep the aggregate quantiles of each metric passing through.
[[aggregators.quantile]]
## General Aggregator Arguments:
## The period on which to flush & clear the aggregator.
# period = "30s"
## If true, the original metric will be dropped by the
## aggregator and will not get sent to the output plugins.
# drop_original = false
## Quantiles to output in the range [0,1]
# quantiles = [0.25, 0.5, 0.75]
## Type of aggregation algorithm
## Supported are:
## "t-digest" -- approximation using centroids, can cope with large number of samples
## "exact R7" -- exact computation also used by Excel or NumPy (Hyndman & Fan 1996 R7)
## "exact R8" -- exact computation (Hyndman & Fan 1996 R8)
## NOTE: Do not use "exact" algorithms with large number of samples
## to not impair performance or memory consumption!
# algorithm = "t-digest"
## Compression for approximation (t-digest). The value needs to be
## greater or equal to 1.0. Smaller values will result in more
## performance but less accuracy.
# compression = 100.0
```
## Algorithm types
### t-digest
Proposed by [Dunning & Ertl (2019)][tdigest_paper] this type uses a
special data-structure to cluster data. These clusters are later used
to approximate the requested quantiles. The bounds of the approximation
can be controlled by the `compression` setting where smaller values
result in higher performance but less accuracy.
Due to its incremental nature, this algorithm can handle large
numbers of samples efficiently. It is recommended for applications
where exact quantile calculation isn't required.
For implementation details see the underlying [golang library][tdigest_lib].
### exact R7 and R8
These algorithms compute quantiles as described in [Hyndman & Fan
(1996)][hyndman_fan]. The R7 variant is used in Excel and NumPy. The R8
variant is recommended by Hyndman & Fan due to its independence of the
underlying sample distribution.
These algorithms save all data for the aggregation `period`. They require a lot
of memory when used with a large number of series or a large number of
samples. They are slower than the `t-digest` algorithm and are recommended only
to be used with a small number of samples and series.
## Benchmark (linux/amd64)
The benchmark was performed by adding 100 metrics with six numeric
(and two non-numeric) fields to the aggregator and the derive the aggregation
result.
| algorithm | # quantiles | avg. runtime |
| :------------ | -------------:| -------------:|
| t-digest | 3 | 376372 ns/op |
| exact R7 | 3 | 9782946 ns/op |
| exact R8 | 3 | 9158205 ns/op |
| t-digest | 100 | 899204 ns/op |
| exact R7 | 100 | 7868816 ns/op |
| exact R8 | 100 | 8099612 ns/op |
## Measurements
Measurement names are passed through this aggregator.
### Fields
For all numeric fields (int32/64, uint32/64 and float32/64) new *quantile*
fields are aggregated in the form `<fieldname>_<quantile*100>`. Other field
types (e.g. boolean, string) are ignored and dropped from the output.
For example passing in the following metric as *input*:
- somemetric
- average_response_ms (float64)
- minimum_response_ms (float64)
- maximum_response_ms (float64)
- status (string)
- ok (boolean)
and the default setting for `quantiles` you get the following *output*
- somemetric
- average_response_ms_025 (float64)
- average_response_ms_050 (float64)
- average_response_ms_075 (float64)
- minimum_response_ms_025 (float64)
- minimum_response_ms_050 (float64)
- minimum_response_ms_075 (float64)
- maximum_response_ms_025 (float64)
- maximum_response_ms_050 (float64)
- maximum_response_ms_075 (float64)
The `status` and `ok` fields are dropped because they are not numeric. Note
that the number of resulting fields scales with the number of `quantiles`
specified.
### Tags
Tags are passed through to the output by this aggregator.
### Example Output
```text
cpu,cpu=cpu-total,host=Hugin usage_user=10.814851731872487,usage_system=2.1679541490155687,usage_irq=1.046598554697342,usage_steal=0,usage_guest_nice=0,usage_idle=85.79616247197244,usage_nice=0,usage_iowait=0,usage_softirq=0.1744330924495688,usage_guest=0 1608288360000000000
cpu,cpu=cpu-total,host=Hugin usage_guest=0,usage_system=2.1601016518428664,usage_iowait=0.02541296060990694,usage_irq=1.0165184243964942,usage_softirq=0.1778907242693666,usage_steal=0,usage_guest_nice=0,usage_user=9.275730622616953,usage_idle=87.34434561626493,usage_nice=0 1608288370000000000
cpu,cpu=cpu-total,host=Hugin usage_idle=85.78199052131747,usage_nice=0,usage_irq=1.0476428036915637,usage_guest=0,usage_guest_nice=0,usage_system=1.995510102269591,usage_iowait=0,usage_softirq=0.1995510102269662,usage_steal=0,usage_user=10.975305562484735 1608288380000000000
cpu,cpu=cpu-total,host=Hugin usage_guest_nice_075=0,usage_user_050=10.814851731872487,usage_guest_075=0,usage_steal_025=0,usage_irq_025=1.031558489546918,usage_irq_075=1.0471206791944527,usage_iowait_025=0,usage_guest_050=0,usage_guest_nice_050=0,usage_nice_075=0,usage_iowait_050=0,usage_system_050=2.1601016518428664,usage_irq_050=1.046598554697342,usage_guest_nice_025=0,usage_idle_050=85.79616247197244,usage_softirq_075=0.1887208672481664,usage_steal_075=0,usage_system_025=2.0778058770562287,usage_system_075=2.1640279004292173,usage_softirq_050=0.1778907242693666,usage_nice_050=0,usage_iowait_075=0.01270648030495347,usage_user_075=10.895078647178611,usage_nice_025=0,usage_steal_050=0,usage_user_025=10.04529117724472,usage_idle_025=85.78907649664495,usage_idle_075=86.57025404411868,usage_softirq_025=0.1761619083594677,usage_guest_025=0 1608288390000000000
```
[tdigest_paper]: https://arxiv.org/abs/1902.04023
[tdigest_lib]: https://github.com/caio/go-tdigest
[hyndman_fan]: http://www.maths.usyd.edu.au/u/UG/SM/STAT3022/r/current/Misc/Sample%20Quantiles%20in%20Statistical%20Packages.pdf

108
plugins/aggregators/quantile/algorithms.go Normal file
View file

@ -0,0 +1,108 @@
package quantile
import (
"math"
"sort"
"github.com/caio/go-tdigest"
)
type algorithm interface {
Add(value float64) error
Quantile(q float64) float64
}
func newTDigest(compression float64) (algorithm, error) {
return tdigest.New(tdigest.Compression(compression))
}
type exactAlgorithmR7 struct {
xs []float64
sorted bool
}
func newExactR7(_ float64) (algorithm, error) {
return &exactAlgorithmR7{xs: make([]float64, 0, 100), sorted: false}, nil
}
func (e *exactAlgorithmR7) Add(value float64) error {
e.xs = append(e.xs, value)
e.sorted = false
return nil
}
func (e *exactAlgorithmR7) Quantile(q float64) float64 {
size := len(e.xs)
// No information
if len(e.xs) == 0 {
return math.NaN()
}
// Sort the array if necessary
if !e.sorted {
sort.Float64s(e.xs)
e.sorted = true
}
// Get the quantile index and the fraction to the neighbor
// Hyndman & Fan; Sample Quantiles in Statistical Packages; The American Statistician vol 50; pp 361-365; 1996 -- R7
// Same as Excel and Numpy.
n := q * (float64(size) - 1)
i, gamma := math.Modf(n)
j := int(i)
if j < 0 {
return e.xs[0]
}
if j >= size {
return e.xs[size-1]
}
// Linear interpolation
return e.xs[j] + gamma*(e.xs[j+1]-e.xs[j])
}
type exactAlgorithmR8 struct {
xs []float64
sorted bool
}
func newExactR8(_ float64) (algorithm, error) {
return &exactAlgorithmR8{xs: make([]float64, 0, 100), sorted: false}, nil
}
func (e *exactAlgorithmR8) Add(value float64) error {
e.xs = append(e.xs, value)
e.sorted = false
return nil
}
func (e *exactAlgorithmR8) Quantile(q float64) float64 {
size := len(e.xs)
// No information
if size == 0 {
return math.NaN()
}
// Sort the array if necessary
if !e.sorted {
sort.Float64s(e.xs)
e.sorted = true
}
// Get the quantile index and the fraction to the neighbor
// Hyndman & Fan; Sample Quantiles in Statistical Packages; The American Statistician vol 50; pp 361-365; 1996 -- R8
n := q*(float64(size)+1.0/3.0) - (2.0 / 3.0) // Indices are zero-base here but one-based in the paper
i, gamma := math.Modf(n)
j := int(i)
if j < 0 {
return e.xs[0]
}
if j >= size {
return e.xs[size-1]
}
// Linear interpolation
return e.xs[j] + gamma*(e.xs[j+1]-e.xs[j])
}

149
plugins/aggregators/quantile/quantile.go Normal file
View file

@ -0,0 +1,149 @@
//go:generate ../../../tools/readme_config_includer/generator
package quantile
import (
_ "embed"
"fmt"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/aggregators"
)
//go:embed sample.conf
var sampleConfig string
type Quantile struct {
Quantiles []float64 `toml:"quantiles"`
Compression float64 `toml:"compression"`
AlgorithmType string `toml:"algorithm"`
newAlgorithm newAlgorithmFunc
cache map[uint64]aggregate
suffixes []string
Log telegraf.Logger `toml:"-"`
}
type aggregate struct {
name string
fields map[string]algorithm
tags map[string]string
}
type newAlgorithmFunc func(compression float64) (algorithm, error)
func (*Quantile) SampleConfig() string {
return sampleConfig
}
func (q *Quantile) Add(in telegraf.Metric) {
id := in.HashID()
if cached, ok := q.cache[id]; ok {
fields := in.Fields()
for k, algo := range cached.fields {
if field, ok := fields[k]; ok {
if v, isconvertible := convert(field); isconvertible {
err := algo.Add(v)
if err != nil {
q.Log.Errorf("adding cached field %s: %v", k, err)
}
}
}
}
return
}
// New metric, setup cache and init algorithm
a := aggregate{
name: in.Name(),
tags: in.Tags(),
fields: make(map[string]algorithm),
}
for k, field := range in.Fields() {
if v, isconvertible := convert(field); isconvertible {
algo, err := q.newAlgorithm(q.Compression)
if err != nil {
q.Log.Errorf("generating algorithm %s: %v", k, err)
}
err = algo.Add(v)
if err != nil {
q.Log.Errorf("adding field %s: %v", k, err)
}
a.fields[k] = algo
}
}
q.cache[id] = a
}
func (q *Quantile) Push(acc telegraf.Accumulator) {
for _, aggregate := range q.cache {
fields := make(map[string]interface{}, len(aggregate.fields)*len(q.Quantiles))
for k, algo := range aggregate.fields {
for i, qtl := range q.Quantiles {
fields[k+q.suffixes[i]] = algo.Quantile(qtl)
}
}
acc.AddFields(aggregate.name, fields, aggregate.tags)
}
}
func (q *Quantile) Reset() {
q.cache = make(map[uint64]aggregate)
}
func convert(in interface{}) (float64, bool) {
switch v := in.(type) {
case float64:
return v, true
case int64:
return float64(v), true
case uint64:
return float64(v), true
default:
return 0, false
}
}
func (q *Quantile) Init() error {
switch q.AlgorithmType {
case "t-digest", "":
q.newAlgorithm = newTDigest
case "exact R7":
q.newAlgorithm = newExactR7
case "exact R8":
q.newAlgorithm = newExactR8
default:
return fmt.Errorf("unknown algorithm type %q", q.AlgorithmType)
}
if _, err := q.newAlgorithm(q.Compression); err != nil {
return fmt.Errorf("cannot create %q algorithm: %w", q.AlgorithmType, err)
}
if len(q.Quantiles) == 0 {
q.Quantiles = []float64{0.25, 0.5, 0.75}
}
duplicates := make(map[float64]bool)
q.suffixes = make([]string, 0, len(q.Quantiles))
for _, qtl := range q.Quantiles {
if qtl < 0.0 || qtl > 1.0 {
return fmt.Errorf("quantile %v out of range", qtl)
}
if _, found := duplicates[qtl]; found {
return fmt.Errorf("duplicate quantile %v", qtl)
}
duplicates[qtl] = true
q.suffixes = append(q.suffixes, fmt.Sprintf("_%03d", int(qtl*100.0)))
}
q.Reset()
return nil
}
func init() {
aggregators.Add("quantile", func() telegraf.Aggregator {
return &Quantile{Compression: 100}
})
}

671
plugins/aggregators/quantile/quantile_test.go Normal file
View file

@ -0,0 +1,671 @@
package quantile
import (
"math/rand"
"testing"
"time"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/stretchr/testify/require"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/testutil"
)
func TestConfigInvalidAlgorithm(t *testing.T) {
q := Quantile{AlgorithmType: "a strange one"}
err := q.Init()
require.Error(t, err)
require.Contains(t, err.Error(), "unknown algorithm type")
}
func TestConfigInvalidCompression(t *testing.T) {
q := Quantile{Compression: 0, AlgorithmType: "t-digest"}
err := q.Init()
require.Error(t, err)
require.Contains(t, err.Error(), "cannot create \"t-digest\" algorithm")
}
func TestConfigInvalidQuantiles(t *testing.T) {
q := Quantile{Compression: 100, Quantiles: []float64{-0.5}}
err := q.Init()
require.Error(t, err)
require.Contains(t, err.Error(), "quantile -0.5 out of range")
q = Quantile{Compression: 100, Quantiles: []float64{1.5}}
err = q.Init()
require.Error(t, err)
require.Contains(t, err.Error(), "quantile 1.5 out of range")
q = Quantile{Compression: 100, Quantiles: []float64{0.1, 0.2, 0.3, 0.1}}
err = q.Init()
require.Error(t, err)
require.Contains(t, err.Error(), "duplicate quantile")
}
func TestSingleMetricTDigest(t *testing.T) {
acc := testutil.Accumulator{}
q := Quantile{
Compression: 100,
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(t, err)
expected := []telegraf.Metric{
testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a_025": 24.75,
"a_050": 49.50,
"a_075": 74.25,
"b_025": 24.75,
"b_050": 49.50,
"b_075": 74.25,
"c_025": 24.75,
"c_050": 49.50,
"c_075": 74.25,
"d_025": 24.75,
"d_050": 49.50,
"d_075": 74.25,
"e_025": 24.75,
"e_050": 49.50,
"e_075": 74.25,
"f_025": 24.75,
"f_050": 49.50,
"f_075": 74.25,
"g_025": 0.2475,
"g_050": 0.4950,
"g_075": 0.7425,
},
time.Now(),
),
}
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": int32(i),
"b": int64(i),
"c": uint32(i),
"d": uint64(i),
"e": float32(i),
"f": float64(i),
"g": float64(i) / 100.0,
"x1": "string",
"x2": true,
},
time.Now(),
))
}
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
epsilon := cmpopts.EquateApprox(0, 1e-3)
testutil.RequireMetricsEqual(t, expected, acc.GetTelegrafMetrics(), testutil.IgnoreTime(), epsilon)
}
func TestMultipleMetricsTDigest(t *testing.T) {
acc := testutil.Accumulator{}
q := Quantile{
Compression: 100,
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(t, err)
expected := []telegraf.Metric{
testutil.MustMetric(
"test",
map[string]string{"series": "foo"},
map[string]interface{}{
"a_025": 24.75, "a_050": 49.50, "a_075": 74.25,
"b_025": 24.75, "b_050": 49.50, "b_075": 74.25,
},
time.Now(),
),
testutil.MustMetric(
"test",
map[string]string{"series": "bar"},
map[string]interface{}{
"a_025": 49.50, "a_050": 99.00, "a_075": 148.50,
"b_025": 49.50, "b_050": 99.00, "b_075": 148.50,
},
time.Now(),
),
}
metricsA := make([]telegraf.Metric, 0, 100)
metricsB := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metricsA = append(metricsA, testutil.MustMetric(
"test",
map[string]string{"series": "foo"},
map[string]interface{}{"a": int64(i), "b": float64(i), "x1": "string", "x2": true},
time.Now(),
))
metricsB = append(metricsB, testutil.MustMetric(
"test",
map[string]string{"series": "bar"},
map[string]interface{}{"a": int64(2 * i), "b": float64(2 * i), "x1": "string", "x2": true},
time.Now(),
))
}
for _, m := range metricsA {
q.Add(m)
}
for _, m := range metricsB {
q.Add(m)
}
q.Push(&acc)
epsilon := cmpopts.EquateApprox(0, 1e-3)
sort := testutil.SortMetrics()
testutil.RequireMetricsEqual(t, expected, acc.GetTelegrafMetrics(), testutil.IgnoreTime(), epsilon, sort)
}
func TestSingleMetricExactR7(t *testing.T) {
acc := testutil.Accumulator{}
q := Quantile{
AlgorithmType: "exact R7",
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(t, err)
expected := []telegraf.Metric{
testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a_025": 24.75,
"a_050": 49.50,
"a_075": 74.25,
"b_025": 24.75,
"b_050": 49.50,
"b_075": 74.25,
"c_025": 24.75,
"c_050": 49.50,
"c_075": 74.25,
"d_025": 24.75,
"d_050": 49.50,
"d_075": 74.25,
"e_025": 24.75,
"e_050": 49.50,
"e_075": 74.25,
"f_025": 24.75,
"f_050": 49.50,
"f_075": 74.25,
"g_025": 0.2475,
"g_050": 0.4950,
"g_075": 0.7425,
},
time.Now(),
),
}
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": int32(i),
"b": int64(i),
"c": uint32(i),
"d": uint64(i),
"e": float32(i),
"f": float64(i),
"g": float64(i) / 100.0,
"x1": "string",
"x2": true,
},
time.Now(),
))
}
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
epsilon := cmpopts.EquateApprox(0, 1e-3)
testutil.RequireMetricsEqual(t, expected, acc.GetTelegrafMetrics(), testutil.IgnoreTime(), epsilon)
}
func TestMultipleMetricsExactR7(t *testing.T) {
acc := testutil.Accumulator{}
q := Quantile{
AlgorithmType: "exact R7",
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(t, err)
expected := []telegraf.Metric{
testutil.MustMetric(
"test",
map[string]string{"series": "foo"},
map[string]interface{}{
"a_025": 24.75, "a_050": 49.50, "a_075": 74.25,
"b_025": 24.75, "b_050": 49.50, "b_075": 74.25,
},
time.Now(),
),
testutil.MustMetric(
"test",
map[string]string{"series": "bar"},
map[string]interface{}{
"a_025": 49.50, "a_050": 99.00, "a_075": 148.50,
"b_025": 49.50, "b_050": 99.00, "b_075": 148.50,
},
time.Now(),
),
}
metricsA := make([]telegraf.Metric, 0, 100)
metricsB := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metricsA = append(metricsA, testutil.MustMetric(
"test",
map[string]string{"series": "foo"},
map[string]interface{}{"a": int64(i), "b": float64(i), "x1": "string", "x2": true},
time.Now(),
))
metricsB = append(metricsB, testutil.MustMetric(
"test",
map[string]string{"series": "bar"},
map[string]interface{}{"a": int64(2 * i), "b": float64(2 * i), "x1": "string", "x2": true},
time.Now(),
))
}
for _, m := range metricsA {
q.Add(m)
}
for _, m := range metricsB {
q.Add(m)
}
q.Push(&acc)
epsilon := cmpopts.EquateApprox(0, 1e-3)
sort := testutil.SortMetrics()
testutil.RequireMetricsEqual(t, expected, acc.GetTelegrafMetrics(), testutil.IgnoreTime(), epsilon, sort)
}
func TestSingleMetricExactR8(t *testing.T) {
acc := testutil.Accumulator{}
q := Quantile{
AlgorithmType: "exact R8",
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(t, err)
expected := []telegraf.Metric{
testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a_025": 24.417,
"a_050": 49.500,
"a_075": 74.583,
"b_025": 24.417,
"b_050": 49.500,
"b_075": 74.583,
"c_025": 24.417,
"c_050": 49.500,
"c_075": 74.583,
"d_025": 24.417,
"d_050": 49.500,
"d_075": 74.583,
"e_025": 24.417,
"e_050": 49.500,
"e_075": 74.583,
"f_025": 24.417,
"f_050": 49.500,
"f_075": 74.583,
"g_025": 0.24417,
"g_050": 0.49500,
"g_075": 0.74583,
},
time.Now(),
),
}
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": int32(i),
"b": int64(i),
"c": uint32(i),
"d": uint64(i),
"e": float32(i),
"f": float64(i),
"g": float64(i) / 100.0,
"x1": "string",
"x2": true,
},
time.Now(),
))
}
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
epsilon := cmpopts.EquateApprox(0, 1e-3)
testutil.RequireMetricsEqual(t, expected, acc.GetTelegrafMetrics(), testutil.IgnoreTime(), epsilon)
}
func TestMultipleMetricsExactR8(t *testing.T) {
acc := testutil.Accumulator{}
q := Quantile{
AlgorithmType: "exact R8",
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(t, err)
expected := []telegraf.Metric{
testutil.MustMetric(
"test",
map[string]string{"series": "foo"},
map[string]interface{}{
"a_025": 24.417, "a_050": 49.500, "a_075": 74.583,
"b_025": 24.417, "b_050": 49.500, "b_075": 74.583,
},
time.Now(),
),
testutil.MustMetric(
"test",
map[string]string{"series": "bar"},
map[string]interface{}{
"a_025": 48.833, "a_050": 99.000, "a_075": 149.167,
"b_025": 48.833, "b_050": 99.000, "b_075": 149.167,
},
time.Now(),
),
}
metricsA := make([]telegraf.Metric, 0, 100)
metricsB := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metricsA = append(metricsA, testutil.MustMetric(
"test",
map[string]string{"series": "foo"},
map[string]interface{}{"a": int64(i), "b": float64(i), "x1": "string", "x2": true},
time.Now(),
))
metricsB = append(metricsB, testutil.MustMetric(
"test",
map[string]string{"series": "bar"},
map[string]interface{}{"a": int64(2 * i), "b": float64(2 * i), "x1": "string", "x2": true},
time.Now(),
))
}
for _, m := range metricsA {
q.Add(m)
}
for _, m := range metricsB {
q.Add(m)
}
q.Push(&acc)
epsilon := cmpopts.EquateApprox(0, 1e-3)
sort := testutil.SortMetrics()
testutil.RequireMetricsEqual(t, expected, acc.GetTelegrafMetrics(), testutil.IgnoreTime(), epsilon, sort)
}
func BenchmarkDefaultTDigest(b *testing.B) {
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": rand.Int31(),
"b": rand.Int63(),
"c": rand.Uint32(),
"d": rand.Uint64(),
"e": rand.Float32(),
"f": rand.Float64(),
"x1": "string",
"x2": true,
},
time.Now(),
))
}
q := Quantile{
Compression: 100,
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(b, err)
acc := testutil.Accumulator{}
for n := 0; n < b.N; n++ {
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
}
}
func BenchmarkDefaultTDigest100Q(b *testing.B) {
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": rand.Int31(),
"b": rand.Int63(),
"c": rand.Uint32(),
"d": rand.Uint64(),
"e": rand.Float32(),
"f": rand.Float64(),
"x1": "string",
"x2": true,
},
time.Now(),
))
}
quantiles := make([]float64, 0, 100)
for i := 0; i < 100; i++ {
quantiles = append(quantiles, 0.01*float64(i))
}
q := Quantile{
Compression: 100,
Quantiles: quantiles,
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(b, err)
acc := testutil.Accumulator{}
for n := 0; n < b.N; n++ {
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
}
}
func BenchmarkDefaultExactR7(b *testing.B) {
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": rand.Int31(),
"b": rand.Int63(),
"c": rand.Uint32(),
"d": rand.Uint64(),
"e": rand.Float32(),
"f": rand.Float64(),
"x1": "string",
"x2": true,
},
time.Now(),
))
}
q := Quantile{
AlgorithmType: "exact R7",
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(b, err)
acc := testutil.Accumulator{}
for n := 0; n < b.N; n++ {
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
}
}
func BenchmarkDefaultExactR7100Q(b *testing.B) {
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": rand.Int31(),
"b": rand.Int63(),
"c": rand.Uint32(),
"d": rand.Uint64(),
"e": rand.Float32(),
"f": rand.Float64(),
"x1": "string",
"x2": true,
},
time.Now(),
))
}
quantiles := make([]float64, 0, 100)
for i := 0; i < 100; i++ {
quantiles = append(quantiles, 0.01*float64(i))
}
q := Quantile{
AlgorithmType: "exact R7",
Quantiles: quantiles,
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(b, err)
acc := testutil.Accumulator{}
for n := 0; n < b.N; n++ {
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
}
}
func BenchmarkDefaultExactR8(b *testing.B) {
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": rand.Int31(),
"b": rand.Int63(),
"c": rand.Uint32(),
"d": rand.Uint64(),
"e": rand.Float32(),
"f": rand.Float64(),
"x1": "string",
"x2": true,
},
time.Now(),
))
}
q := Quantile{
AlgorithmType: "exact R8",
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(b, err)
acc := testutil.Accumulator{}
for n := 0; n < b.N; n++ {
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
}
}
func BenchmarkDefaultExactR8100Q(b *testing.B) {
metrics := make([]telegraf.Metric, 0, 100)
for i := 0; i < 100; i++ {
metrics = append(metrics, testutil.MustMetric(
"test",
map[string]string{"foo": "bar"},
map[string]interface{}{
"a": rand.Int31(),
"b": rand.Int63(),
"c": rand.Uint32(),
"d": rand.Uint64(),
"e": rand.Float32(),
"f": rand.Float64(),
"x1": "string",
"x2": true,
},
time.Now(),
))
}
quantiles := make([]float64, 0, 100)
for i := 0; i < 100; i++ {
quantiles = append(quantiles, 0.01*float64(i))
}
q := Quantile{
AlgorithmType: "exact R8",
Quantiles: quantiles,
Log: testutil.Logger{},
}
err := q.Init()
require.NoError(b, err)
acc := testutil.Accumulator{}
for n := 0; n < b.N; n++ {
for _, m := range metrics {
q.Add(m)
}
q.Push(&acc)
}
}

26
plugins/aggregators/quantile/sample.conf Normal file
View file

@ -0,0 +1,26 @@
# Keep the aggregate quantiles of each metric passing through.
[[aggregators.quantile]]
## General Aggregator Arguments:
## The period on which to flush & clear the aggregator.
# period = "30s"
## If true, the original metric will be dropped by the
## aggregator and will not get sent to the output plugins.
# drop_original = false
## Quantiles to output in the range [0,1]
# quantiles = [0.25, 0.5, 0.75]
## Type of aggregation algorithm
## Supported are:
## "t-digest" -- approximation using centroids, can cope with large number of samples
## "exact R7" -- exact computation also used by Excel or NumPy (Hyndman & Fan 1996 R7)
## "exact R8" -- exact computation (Hyndman & Fan 1996 R8)
## NOTE: Do not use "exact" algorithms with large number of samples
## to not impair performance or memory consumption!
# algorithm = "t-digest"
## Compression for approximation (t-digest). The value needs to be
## greater or equal to 1.0. Smaller values will result in more
## performance but less accuracy.
# compression = 100.0