influx/telegraf
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.34.4.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-05-24 07:26:29 +02:00
parent e393c3af3f
commit 4978089aab
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
4963 changed files with 677545 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

180
plugins/inputs/postgresql/README.md Normal file
View file

@ -0,0 +1,180 @@
# PostgreSQL Input Plugin
The `postgresql` plugin provides metrics for your PostgreSQL Server instance.
Recorded metrics are lightweight and use Dynamic Management Views supplied
by PostgreSQL.
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
## Secret-store support
This plugin supports secrets from secret-stores for the `address` option.
See the [secret-store documentation][SECRETSTORE] for more details on how
to use them.
[SECRETSTORE]: ../../../docs/CONFIGURATION.md#secret-store-secrets
## Configuration
```toml @sample.conf
# Read metrics from one or many postgresql servers
[[inputs.postgresql]]
## Specify address via a url matching:
## postgres://[pqgotest[:password]]@localhost[/dbname]?sslmode=[disable|verify-ca|verify-full]&statement_timeout=...
## or a simple string:
## host=localhost user=pqgotest password=... sslmode=... dbname=app_production
## Users can pass the path to the socket as the host value to use a socket
## connection (e.g. `/var/run/postgresql`).
##
## All connection parameters are optional.
##
## Without the dbname parameter, the driver will default to a database
## with the same name as the user. This dbname is just for instantiating a
## connection with the server and doesn't restrict the databases we are trying
## to grab metrics for.
##
address = "host=localhost user=postgres sslmode=disable"
## A custom name for the database that will be used as the "server" tag in the
## measurement output. If not specified, a default one generated from
## the connection address is used.
# outputaddress = "db01"
## connection configuration.
## maxlifetime - specify the maximum lifetime of a connection.
## default is forever (0s)
##
## Note that this does not interrupt queries, the lifetime will not be enforced
## whilst a query is running
# max_lifetime = "0s"
## A list of databases to explicitly ignore. If not specified, metrics for all
## databases are gathered. Do NOT use with the 'databases' option.
# ignored_databases = ["postgres", "template0", "template1"]
## A list of databases to pull metrics about. If not specified, metrics for all
## databases are gathered. Do NOT use with the 'ignored_databases' option.
# databases = ["app_production", "testing"]
## Whether to use prepared statements when connecting to the database.
## This should be set to false when connecting through a PgBouncer instance
## with pool_mode set to transaction.
prepared_statements = true
```
Specify address via a postgresql connection string:
```text
host=localhost port=5432 user=telegraf database=telegraf
```
Or via an url matching:
```text
postgres://[pqgotest[:password]]@host:port[/dbname]?sslmode=[disable|verify-ca|verify-full]
```
Users can pass the path to the socket as the host value to use a socket
connection (e.g. `/var/run/postgresql`).
It is also possible to specify a query timeout maximum execution time (in ms)
for any individual statement passed over the connection
```text
postgres://[pqgotest[:password]]@host:port[/dbname]?sslmode=[disable|verify-ca|verify-full]&statement_timeout=10000
```
All connection parameters are optional. Without the dbname parameter, the driver
will default to a database with the same name as the user. This dbname is just
for instantiating a connection with the server and doesn't restrict the
databases we are trying to grab metrics for.
A list of databases to explicitly ignore. If not specified, metrics for all
databases are gathered. Do NOT use with the 'databases' option.
```text
ignored_databases = ["postgres", "template0", "template1"]`
```
A list of databases to pull metrics about. If not specified, metrics for all
databases are gathered. Do NOT use with the 'ignored_databases' option.
```text
databases = ["app_production", "testing"]`
```
### Permissions
The plugins gathers metrics from the `pg_stat_database` and `pg_stat_bgwriter`
views. To grant a user access to the views run:
```sql
GRANT pg_read_all_stats TO user;
```
See the [PostgreSQL docs][] for more information on the predefined roles.
[PostgreSQL docs]: https://www.postgresql.org/docs/current/predefined-roles.html
### TLS Configuration
Add the `sslkey`, `sslcert` and `sslrootcert` options to your DSN:
```shell
host=localhost user=pgotest dbname=app_production sslmode=require sslkey=/etc/telegraf/key.pem sslcert=/etc/telegraf/cert.pem sslrootcert=/etc/telegraf/ca.pem
```
## Metrics
This postgresql plugin provides metrics for your postgres database. It currently
works with postgres versions 8.1+. It uses data from the built in
_pg_stat_database_ and pg_stat_bgwriter views. The metrics recorded depend on
your version of postgres. See table:
```sh
pg version 9.2+ 9.1 8.3-9.0 8.1-8.2 7.4-8.0(unsupported)
--- --- --- ------- ------- -------
datid x x x x
datname x x x x
numbackends x x x x x
xact_commit x x x x x
xact_rollback x x x x x
blks_read x x x x x
blks_hit x x x x x
tup_returned x x x
tup_fetched x x x
tup_inserted x x x
tup_updated x x x
tup_deleted x x x
conflicts x x
temp_files x
temp_bytes x
deadlocks x
blk_read_time x
blk_write_time x
stats_reset* x x
```
_* value ignored and therefore not recorded._
More information about the meaning of these metrics can be found in the
[PostgreSQL Documentation][1].
[1]: http://www.postgresql.org/docs/9.2/static/monitoring-stats.html#PG-STAT-DATABASE-VIEW
## Example Output
```text
postgresql,db=postgres_global,server=dbname\=postgres\ host\=localhost\ port\=5432\ statement_timeout\=10000\ user\=postgres tup_fetched=1271i,tup_updated=5i,session_time=1451414320768.855,xact_rollback=2i,conflicts=0i,blk_write_time=0,temp_bytes=0i,datid=0i,sessions_fatal=0i,tup_returned=1339i,sessions_abandoned=0i,blk_read_time=0,blks_read=88i,idle_in_transaction_time=0,sessions=0i,active_time=0,tup_inserted=24i,tup_deleted=0i,temp_files=0i,numbackends=0i,xact_commit=4i,sessions_killed=0i,blks_hit=5616i,deadlocks=0i 1672399790000000000
postgresql,db=postgres,host=oss_cluster_host,server=dbname\=postgres\ host\=localhost\ port\=5432\ statement_timeout\=10000\ user\=postgres conflicts=0i,sessions_abandoned=2i,active_time=460340.823,tup_returned=119382i,tup_deleted=0i,blk_write_time=0,xact_commit=305i,blks_hit=16358i,deadlocks=0i,sessions=12i,numbackends=1i,temp_files=0i,xact_rollback=5i,sessions_fatal=0i,datname="postgres",blk_read_time=0,idle_in_transaction_time=0,temp_bytes=0i,tup_inserted=3i,tup_updated=0i,blks_read=299i,datid=5i,session_time=469056.613,sessions_killed=0i,tup_fetched=5550i 1672399790000000000
postgresql,db=template1,host=oss_cluster_host,server=dbname\=postgres\ host\=localhost\ port\=5432\ statement_timeout\=10000\ user\=postgres active_time=0,idle_in_transaction_time=0,blks_read=1352i,sessions_abandoned=0i,tup_fetched=28544i,session_time=0,sessions_killed=0i,temp_bytes=0i,tup_returned=188541i,xact_commit=1168i,blk_read_time=0,sessions_fatal=0i,datid=1i,datname="template1",conflicts=0i,xact_rollback=0i,numbackends=0i,deadlocks=0i,sessions=0i,tup_inserted=17520i,temp_files=0i,tup_updated=743i,blk_write_time=0,blks_hit=99487i,tup_deleted=34i 1672399790000000000
postgresql,db=template0,host=oss_cluster_host,server=dbname\=postgres\ host\=localhost\ port\=5432\ statement_timeout\=10000\ user\=postgres sessions=0i,datid=4i,tup_updated=0i,sessions_abandoned=0i,blk_write_time=0,numbackends=0i,blks_read=0i,blks_hit=0i,sessions_fatal=0i,temp_files=0i,deadlocks=0i,conflicts=0i,xact_commit=0i,xact_rollback=0i,session_time=0,datname="template0",tup_returned=0i,tup_inserted=0i,idle_in_transaction_time=0,tup_fetched=0i,active_time=0,temp_bytes=0i,tup_deleted=0i,blk_read_time=0,sessions_killed=0i 1672399790000000000
postgresql,db=postgres,host=oss_cluster_host,server=dbname\=postgres\ host\=localhost\ port\=5432\ statement_timeout\=10000\ user\=postgres buffers_clean=0i,buffers_alloc=426i,checkpoints_req=1i,buffers_checkpoint=50i,buffers_backend_fsync=0i,checkpoint_write_time=5053,checkpoints_timed=26i,checkpoint_sync_time=26,maxwritten_clean=0i,buffers_backend=9i 1672399790000000000
```

166
plugins/inputs/postgresql/postgresql.go Normal file
View file

@ -0,0 +1,166 @@
//go:generate ../../../tools/readme_config_includer/generator
package postgresql
import (
"bytes"
"database/sql"
_ "embed"
"fmt"
"strings"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/common/postgresql"
"github.com/influxdata/telegraf/plugins/inputs"
)
//go:embed sample.conf
var sampleConfig string
var ignoredColumns = map[string]bool{"stats_reset": true}
type Postgresql struct {
Databases []string `toml:"databases"`
IgnoredDatabases []string `toml:"ignored_databases"`
PreparedStatements bool `toml:"prepared_statements"`
postgresql.Config
service *postgresql.Service
}
func (*Postgresql) SampleConfig() string {
return sampleConfig
}
func (p *Postgresql) Init() error {
p.IsPgBouncer = !p.PreparedStatements
service, err := p.Config.CreateService()
if err != nil {
return err
}
p.service = service
return nil
}
func (p *Postgresql) Start(_ telegraf.Accumulator) error {
return p.service.Start()
}
func (p *Postgresql) Gather(acc telegraf.Accumulator) error {
var query string
if len(p.Databases) == 0 && len(p.IgnoredDatabases) == 0 {
query = `SELECT * FROM pg_stat_database`
} else if len(p.IgnoredDatabases) != 0 {
query = fmt.Sprintf(`SELECT * FROM pg_stat_database WHERE datname NOT IN ('%s')`,
strings.Join(p.IgnoredDatabases, "','"))
} else {
query = fmt.Sprintf(`SELECT * FROM pg_stat_database WHERE datname IN ('%s')`,
strings.Join(p.Databases, "','"))
}
rows, err := p.service.DB.Query(query)
if err != nil {
return err
}
defer rows.Close()
// grab the column information from the result
columns, err := rows.Columns()
if err != nil {
return err
}
for rows.Next() {
err = p.accRow(rows, acc, columns)
if err != nil {
return err
}
}
query = `SELECT * FROM pg_stat_bgwriter`
bgWriterRow, err := p.service.DB.Query(query)
if err != nil {
return err
}
defer bgWriterRow.Close()
// grab the column information from the result
if columns, err = bgWriterRow.Columns(); err != nil {
return err
}
for bgWriterRow.Next() {
if err := p.accRow(bgWriterRow, acc, columns); err != nil {
return err
}
}
return bgWriterRow.Err()
}
func (p *Postgresql) Stop() {
p.service.Stop()
}
func (p *Postgresql) accRow(row *sql.Rows, acc telegraf.Accumulator, columns []string) error {
var dbname bytes.Buffer
// this is where we'll store the column name with its *interface{}
columnMap := make(map[string]*interface{})
for _, column := range columns {
columnMap[column] = new(interface{})
}
columnVars := make([]interface{}, 0, len(columnMap))
// populate the array of interface{} with the pointers in the right order
for i := 0; i < len(columnMap); i++ {
columnVars = append(columnVars, columnMap[columns[i]])
}
// deconstruct array of variables and send to Scan
if err := row.Scan(columnVars...); err != nil {
return err
}
if columnMap["datname"] != nil {
// extract the database name from the column map
if dbNameStr, ok := (*columnMap["datname"]).(string); ok {
dbname.WriteString(dbNameStr)
} else {
// PG 12 adds tracking of global objects to pg_stat_database
dbname.WriteString("postgres_global")
}
} else {
dbname.WriteString(p.service.ConnectionDatabase)
}
tagAddress := p.service.SanitizedAddress
tags := map[string]string{"server": tagAddress, "db": dbname.String()}
fields := make(map[string]interface{})
for col, val := range columnMap {
_, ignore := ignoredColumns[col]
if !ignore {
fields[col] = *val
}
}
acc.AddFields("postgresql", fields, tags)
return nil
}
func init() {
inputs.Add("postgresql", func() telegraf.Input {
return &Postgresql{
Config: postgresql.Config{
MaxIdle: 1,
MaxOpen: 1,
},
PreparedStatements: true,
}
})
}

379
plugins/inputs/postgresql/postgresql_test.go Normal file
View file

@ -0,0 +1,379 @@
package postgresql
import (
"fmt"
"testing"
"github.com/docker/go-connections/nat"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go/wait"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/plugins/common/postgresql"
"github.com/influxdata/telegraf/testutil"
)
const servicePort = "5432"
func launchTestContainer(t *testing.T) *testutil.Container {
container := testutil.Container{
Image: "postgres:alpine",
ExposedPorts: []string{servicePort},
Env: map[string]string{
"POSTGRES_HOST_AUTH_METHOD": "trust",
},
WaitingFor: wait.ForAll(
// the database comes up twice, once right away, then again a second
// time after the docker entrypoint starts configuration
wait.ForLog("database system is ready to accept connections").WithOccurrence(2),
wait.ForListeningPort(nat.Port(servicePort)),
),
}
err := container.Start()
require.NoError(t, err, "failed to start container")
return &container
}
func TestPostgresqlGeneratesMetricsIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
container := launchTestContainer(t)
defer container.Terminate()
addr := fmt.Sprintf(
"host=%s port=%s user=postgres sslmode=disable",
container.Address,
container.Ports[servicePort],
)
p := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
IsPgBouncer: false,
},
Databases: []string{"postgres"},
}
require.NoError(t, p.Init())
var acc testutil.Accumulator
require.NoError(t, p.Start(&acc))
defer p.Stop()
require.NoError(t, p.Gather(&acc))
intMetrics := []string{
"xact_commit",
"xact_rollback",
"blks_read",
"blks_hit",
"tup_returned",
"tup_fetched",
"tup_inserted",
"tup_updated",
"tup_deleted",
"conflicts",
"temp_files",
"temp_bytes",
"deadlocks",
"buffers_alloc",
"buffers_clean",
"maxwritten_clean",
"datid",
"numbackends",
"sessions",
"sessions_killed",
"sessions_fatal",
"sessions_abandoned",
}
var int32Metrics []string
floatMetrics := []string{
"blk_read_time",
"blk_write_time",
"active_time",
"idle_in_transaction_time",
"session_time",
}
stringMetrics := []string{
"datname",
}
metricsCounted := 0
for _, metric := range intMetrics {
require.True(t, acc.HasInt64Field("postgresql", metric), "%q not found in int metrics", metric)
metricsCounted++
}
for _, metric := range int32Metrics {
require.True(t, acc.HasInt32Field("postgresql", metric), "%q not found in int32 metrics", metric)
metricsCounted++
}
for _, metric := range floatMetrics {
require.True(t, acc.HasFloatField("postgresql", metric), "%q not found in float metrics", metric)
metricsCounted++
}
for _, metric := range stringMetrics {
require.True(t, acc.HasStringField("postgresql", metric), "%q not found in string metrics", metric)
metricsCounted++
}
require.Positive(t, metricsCounted)
require.Equal(t, len(floatMetrics)+len(intMetrics)+len(int32Metrics)+len(stringMetrics), metricsCounted)
}
func TestPostgresqlTagsMetricsWithDatabaseNameIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
container := launchTestContainer(t)
defer container.Terminate()
addr := fmt.Sprintf(
"host=%s port=%s user=postgres sslmode=disable",
container.Address,
container.Ports[servicePort],
)
p := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
},
Databases: []string{"postgres"},
}
require.NoError(t, p.Init())
var acc testutil.Accumulator
require.NoError(t, p.Start(&acc))
defer p.Stop()
require.NoError(t, p.Gather(&acc))
point, ok := acc.Get("postgresql")
require.True(t, ok)
require.Equal(t, "postgres", point.Tags["db"])
}
func TestPostgresqlDefaultsToAllDatabasesIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
container := launchTestContainer(t)
defer container.Terminate()
addr := fmt.Sprintf(
"host=%s port=%s user=postgres sslmode=disable",
container.Address,
container.Ports[servicePort],
)
p := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
},
}
require.NoError(t, p.Init())
var acc testutil.Accumulator
require.NoError(t, p.Start(&acc))
defer p.Stop()
require.NoError(t, p.Gather(&acc))
var found bool
for _, pnt := range acc.Metrics {
if pnt.Measurement == "postgresql" {
if pnt.Tags["db"] == "postgres" {
found = true
break
}
}
}
require.True(t, found)
}
func TestPostgresqlIgnoresUnwantedColumnsIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
container := launchTestContainer(t)
defer container.Terminate()
addr := fmt.Sprintf(
"host=%s port=%s user=postgres sslmode=disable",
container.Address,
container.Ports[servicePort],
)
p := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
},
}
require.NoError(t, p.Init())
var acc testutil.Accumulator
require.NoError(t, p.Start(&acc))
defer p.Stop()
require.NoError(t, p.Gather(&acc))
for col := range ignoredColumns {
require.False(t, acc.HasMeasurement(col))
}
}
func TestPostgresqlDatabaseWhitelistTestIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
container := launchTestContainer(t)
defer container.Terminate()
addr := fmt.Sprintf(
"host=%s port=%s user=postgres sslmode=disable",
container.Address,
container.Ports[servicePort],
)
p := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
},
Databases: []string{"template0"},
}
require.NoError(t, p.Init())
var acc testutil.Accumulator
require.NoError(t, p.Start(&acc))
defer p.Stop()
require.NoError(t, p.Gather(&acc))
var foundTemplate0 = false
var foundTemplate1 = false
for _, pnt := range acc.Metrics {
if pnt.Measurement == "postgresql" {
if pnt.Tags["db"] == "template0" {
foundTemplate0 = true
}
}
if pnt.Measurement == "postgresql" {
if pnt.Tags["db"] == "template1" {
foundTemplate1 = true
}
}
}
require.True(t, foundTemplate0)
require.False(t, foundTemplate1)
}
func TestPostgresqlDatabaseBlacklistTestIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
container := launchTestContainer(t)
defer container.Terminate()
addr := fmt.Sprintf(
"host=%s port=%s user=postgres sslmode=disable",
container.Address,
container.Ports[servicePort],
)
p := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
},
IgnoredDatabases: []string{"template0"},
}
require.NoError(t, p.Init())
var acc testutil.Accumulator
require.NoError(t, p.Start(&acc))
defer p.Stop()
require.NoError(t, p.Gather(&acc))
var foundTemplate0 = false
var foundTemplate1 = false
for _, pnt := range acc.Metrics {
if pnt.Measurement == "postgresql" {
if pnt.Tags["db"] == "template0" {
foundTemplate0 = true
}
}
if pnt.Measurement == "postgresql" {
if pnt.Tags["db"] == "template1" {
foundTemplate1 = true
}
}
}
require.False(t, foundTemplate0)
require.True(t, foundTemplate1)
}
func TestInitialConnectivityIssueIntegration(t *testing.T) {
// Test case for https://github.com/influxdata/telegraf/issues/8586
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
// Startup the container
container := testutil.Container{
Image: "postgres:alpine",
ExposedPorts: []string{servicePort},
Env: map[string]string{
"POSTGRES_HOST_AUTH_METHOD": "trust",
},
WaitingFor: wait.ForAll(
// the database comes up twice, once right away, then again a second
// time after the docker entrypoint starts configuration
wait.ForLog("database system is ready to accept connections").WithOccurrence(2),
wait.ForListeningPort(nat.Port(servicePort)),
),
}
require.NoError(t, container.Start(), "failed to start container")
defer container.Terminate()
// Pause the container to simulate connectivity issues
require.NoError(t, container.Pause())
// Setup and start the plugin. This should work as the SQL framework will
// not connect immediately but on the first query/access to the server
addr := fmt.Sprintf("host=%s port=%s user=postgres sslmode=disable connect_timeout=1", container.Address, container.Ports[servicePort])
plugin := &Postgresql{
Config: postgresql.Config{
Address: config.NewSecret([]byte(addr)),
},
IgnoredDatabases: []string{"template0"},
}
require.NoError(t, plugin.Init())
// Startup the plugin
var acc testutil.Accumulator
require.NoError(t, plugin.Start(&acc))
defer plugin.Stop()
// This should fail because we cannot connect
require.ErrorContains(t, acc.GatherError(plugin.Gather), "failed to connect")
// Unpause the container, now gather should succeed
require.NoError(t, container.Resume())
require.NoError(t, acc.GatherError(plugin.Gather))
require.NotEmpty(t, acc.GetTelegrafMetrics())
}

43
plugins/inputs/postgresql/sample.conf Normal file
View file

@ -0,0 +1,43 @@
# Read metrics from one or many postgresql servers
[[inputs.postgresql]]
## Specify address via a url matching:
## postgres://[pqgotest[:password]]@localhost[/dbname]?sslmode=[disable|verify-ca|verify-full]&statement_timeout=...
## or a simple string:
## host=localhost user=pqgotest password=... sslmode=... dbname=app_production
## Users can pass the path to the socket as the host value to use a socket
## connection (e.g. `/var/run/postgresql`).
##
## All connection parameters are optional.
##
## Without the dbname parameter, the driver will default to a database
## with the same name as the user. This dbname is just for instantiating a
## connection with the server and doesn't restrict the databases we are trying
## to grab metrics for.
##
address = "host=localhost user=postgres sslmode=disable"
## A custom name for the database that will be used as the "server" tag in the
## measurement output. If not specified, a default one generated from
## the connection address is used.
# outputaddress = "db01"
## connection configuration.
## maxlifetime - specify the maximum lifetime of a connection.
## default is forever (0s)
##
## Note that this does not interrupt queries, the lifetime will not be enforced
## whilst a query is running
# max_lifetime = "0s"
## A list of databases to explicitly ignore. If not specified, metrics for all
## databases are gathered. Do NOT use with the 'databases' option.
# ignored_databases = ["postgres", "template0", "template1"]
## A list of databases to pull metrics about. If not specified, metrics for all
## databases are gathered. Do NOT use with the 'ignored_databases' option.
# databases = ["app_production", "testing"]
## Whether to use prepared statements when connecting to the database.
## This should be set to false when connecting through a PgBouncer instance
## with pool_mode set to transaction.
prepared_statements = true