Adding upstream version 1.34.4.
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
parent
e393c3af3f
commit
4978089aab
4963 changed files with 677545 additions and 0 deletions
241
plugins/inputs/dcos/README.md
Normal file
241
plugins/inputs/dcos/README.md
Normal file
|
@ -0,0 +1,241 @@
|
|||
# Mesosphere Distributed Cloud OS Input Plugin
|
||||
|
||||
This input plugin gathers metrics from a [Distributed Cloud OS][dcos] cluster's
|
||||
[metrics component][metrics].
|
||||
|
||||
> [!WARNING]
|
||||
> Depending on the workload of your DC/OS cluster, this plugin can quickly
|
||||
> create a high number of series which, when unchecked, can cause high load on
|
||||
> your database!
|
||||
|
||||
⭐ Telegraf v1.5.0
|
||||
🏷️ containers
|
||||
💻 all
|
||||
|
||||
[dcos]: https://dcos.io/
|
||||
[metrics]: https://docs.mesosphere.com/1.10/metrics/
|
||||
|
||||
## Global configuration options <!-- @/docs/includes/plugin_config.md -->
|
||||
|
||||
In addition to the plugin-specific configuration settings, plugins support
|
||||
additional global and plugin configuration settings. These settings are used to
|
||||
modify metrics, tags, and field or create aliases and configure ordering, etc.
|
||||
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
|
||||
|
||||
[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
|
||||
|
||||
## Configuration
|
||||
|
||||
```toml @sample.conf
|
||||
# Input plugin for DC/OS metrics
|
||||
[[inputs.dcos]]
|
||||
## The DC/OS cluster URL.
|
||||
cluster_url = "https://dcos-master-1"
|
||||
|
||||
## The ID of the service account.
|
||||
service_account_id = "telegraf"
|
||||
## The private key file for the service account.
|
||||
service_account_private_key = "/etc/telegraf/telegraf-sa-key.pem"
|
||||
|
||||
## Path containing login token. If set, will read on every gather.
|
||||
# token_file = "/home/dcos/.dcos/token"
|
||||
|
||||
## In all filter options if both include and exclude are empty all items
|
||||
## will be collected. Arrays may contain glob patterns.
|
||||
##
|
||||
## Node IDs to collect metrics from. If a node is excluded, no metrics will
|
||||
## be collected for its containers or apps.
|
||||
# node_include = []
|
||||
# node_exclude = []
|
||||
## Container IDs to collect container metrics from.
|
||||
# container_include = []
|
||||
# container_exclude = []
|
||||
## Container IDs to collect app metrics from.
|
||||
# app_include = []
|
||||
# app_exclude = []
|
||||
|
||||
## Maximum concurrent connections to the cluster.
|
||||
# max_connections = 10
|
||||
## Maximum time to receive a response from cluster.
|
||||
# response_timeout = "20s"
|
||||
|
||||
## Optional TLS Config
|
||||
# tls_ca = "/etc/telegraf/ca.pem"
|
||||
# tls_cert = "/etc/telegraf/cert.pem"
|
||||
# tls_key = "/etc/telegraf/key.pem"
|
||||
## If false, skip chain & host verification
|
||||
# insecure_skip_verify = true
|
||||
|
||||
## Recommended filtering to reduce series cardinality.
|
||||
# [inputs.dcos.tagdrop]
|
||||
# path = ["/var/lib/mesos/slave/slaves/*"]
|
||||
```
|
||||
|
||||
### Enterprise Authentication
|
||||
|
||||
When using Enterprise DC/OS, it is recommended to use a service account to
|
||||
authenticate with the cluster.
|
||||
|
||||
The plugin requires the following permissions:
|
||||
|
||||
```text
|
||||
dcos:adminrouter:ops:system-metrics full
|
||||
dcos:adminrouter:ops:mesos full
|
||||
```
|
||||
|
||||
Follow the directions to [create a service account and assign permissions][1].
|
||||
|
||||
Quick configuration using the Enterprise CLI:
|
||||
|
||||
```text
|
||||
dcos security org service-accounts keypair telegraf-sa-key.pem telegraf-sa-cert.pem
|
||||
dcos security org service-accounts create -p telegraf-sa-cert.pem -d "Telegraf DC/OS input plugin" telegraf
|
||||
dcos security org users grant telegraf dcos:adminrouter:ops:system-metrics full
|
||||
dcos security org users grant telegraf dcos:adminrouter:ops:mesos full
|
||||
```
|
||||
|
||||
[1]: https://docs.mesosphere.com/1.10/security/service-auth/custom-service-auth/
|
||||
|
||||
### Open Source Authentication
|
||||
|
||||
The Open Source DC/OS does not provide service accounts. Instead you can use
|
||||
of the following options:
|
||||
|
||||
1. [Disable authentication](https://dcos.io/docs/1.10/security/managing-authentication/#authentication-opt-out)
|
||||
2. Use the `token_file` parameter to read a authentication token from a file.
|
||||
|
||||
Then `token_file` can be set by using the [dcos cli] to login periodically.
|
||||
The cli can login for at most XXX days, you will need to ensure the cli
|
||||
performs a new login before this time expires.
|
||||
|
||||
```shell
|
||||
dcos auth login --username foo --password bar
|
||||
dcos config show core.dcos_acs_token > ~/.dcos/token
|
||||
```
|
||||
|
||||
Another option to create a `token_file` is to generate a token using the
|
||||
cluster secret. This will allow you to set the expiration date manually or
|
||||
even create a never expiring token. However, if the cluster secret or the
|
||||
token is compromised it cannot be revoked and may require a full reinstall of
|
||||
the cluster. For more information on this technique reference
|
||||
[this blog post][2].
|
||||
|
||||
[2]: https://medium.com/@richardgirges/authenticating-open-source-dc-os-with-third-party-services-125fa33a5add
|
||||
|
||||
### Series Cardinality Mitigation
|
||||
|
||||
- Use [measurement filtering](/docs/CONFIGURATION.md#metric-filtering)to exclude
|
||||
unnecessary tags.
|
||||
- Write to a database with an appropriate
|
||||
[retention policy](https://docs.influxdata.com/influxdb/latest/guides/downsampling_and_retention/).
|
||||
- Consider using the
|
||||
[Time Series Index](https://docs.influxdata.com/influxdb/latest/concepts/time-series-index/).
|
||||
- Monitor your databases'
|
||||
[series cardinality](https://docs.influxdata.com/influxdb/latest/query_language/spec/#show-cardinality).
|
||||
|
||||
## Metrics
|
||||
|
||||
Please consult the [Metrics Reference][3] for details about field
|
||||
interpretation.
|
||||
|
||||
- dcos_node
|
||||
- tags:
|
||||
- cluster
|
||||
- hostname
|
||||
- path (filesystem fields only)
|
||||
- interface (network fields only)
|
||||
- fields:
|
||||
- system_uptime (float)
|
||||
- cpu_cores (float)
|
||||
- cpu_total (float)
|
||||
- cpu_user (float)
|
||||
- cpu_system (float)
|
||||
- cpu_idle (float)
|
||||
- cpu_wait (float)
|
||||
- load_1min (float)
|
||||
- load_5min (float)
|
||||
- load_15min (float)
|
||||
- filesystem_capacity_total_bytes (int)
|
||||
- filesystem_capacity_used_bytes (int)
|
||||
- filesystem_capacity_free_bytes (int)
|
||||
- filesystem_inode_total (float)
|
||||
- filesystem_inode_used (float)
|
||||
- filesystem_inode_free (float)
|
||||
- memory_total_bytes (int)
|
||||
- memory_free_bytes (int)
|
||||
- memory_buffers_bytes (int)
|
||||
- memory_cached_bytes (int)
|
||||
- swap_total_bytes (int)
|
||||
- swap_free_bytes (int)
|
||||
- swap_used_bytes (int)
|
||||
- network_in_bytes (int)
|
||||
- network_out_bytes (int)
|
||||
- network_in_packets (float)
|
||||
- network_out_packets (float)
|
||||
- network_in_dropped (float)
|
||||
- network_out_dropped (float)
|
||||
- network_in_errors (float)
|
||||
- network_out_errors (float)
|
||||
- process_count (float)
|
||||
|
||||
- dcos_container
|
||||
- tags:
|
||||
- cluster
|
||||
- hostname
|
||||
- container_id
|
||||
- task_name
|
||||
- fields:
|
||||
- cpus_limit (float)
|
||||
- cpus_system_time (float)
|
||||
- cpus_throttled_time (float)
|
||||
- cpus_user_time (float)
|
||||
- disk_limit_bytes (int)
|
||||
- disk_used_bytes (int)
|
||||
- mem_limit_bytes (int)
|
||||
- mem_total_bytes (int)
|
||||
- net_rx_bytes (int)
|
||||
- net_rx_dropped (float)
|
||||
- net_rx_errors (float)
|
||||
- net_rx_packets (float)
|
||||
- net_tx_bytes (int)
|
||||
- net_tx_dropped (float)
|
||||
- net_tx_errors (float)
|
||||
- net_tx_packets (float)
|
||||
|
||||
- dcos_app
|
||||
- tags:
|
||||
- cluster
|
||||
- hostname
|
||||
- container_id
|
||||
- task_name
|
||||
- fields:
|
||||
- fields are application specific
|
||||
|
||||
[3]: https://docs.mesosphere.com/1.10/metrics/reference/
|
||||
|
||||
## Example Output
|
||||
|
||||
```text
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,path=/boot filesystem_capacity_free_bytes=918188032i,filesystem_capacity_total_bytes=1063256064i,filesystem_capacity_used_bytes=145068032i,filesystem_inode_free=523958,filesystem_inode_total=524288,filesystem_inode_used=330 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=dummy0 network_in_bytes=0i,network_in_dropped=0,network_in_errors=0,network_in_packets=0,network_out_bytes=0i,network_out_dropped=0,network_out_errors=0,network_out_packets=0 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=docker0 network_in_bytes=0i,network_in_dropped=0,network_in_errors=0,network_in_packets=0,network_out_bytes=0i,network_out_dropped=0,network_out_errors=0,network_out_packets=0 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18 cpu_cores=2,cpu_idle=81.62,cpu_system=4.19,cpu_total=13.670000000000002,cpu_user=9.48,cpu_wait=0,load_15min=0.7,load_1min=0.22,load_5min=0.6,memory_buffers_bytes=970752i,memory_cached_bytes=1830473728i,memory_free_bytes=1178636288i,memory_total_bytes=3975073792i,process_count=198,swap_free_bytes=859828224i,swap_total_bytes=859828224i,swap_used_bytes=0i,system_uptime=18874 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=lo network_in_bytes=1090992450i,network_in_dropped=0,network_in_errors=0,network_in_packets=1546938,network_out_bytes=1090992450i,network_out_dropped=0,network_out_errors=0,network_out_packets=1546938 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,path=/ filesystem_capacity_free_bytes=1668378624i,filesystem_capacity_total_bytes=6641680384i,filesystem_capacity_used_bytes=4973301760i,filesystem_inode_free=3107856,filesystem_inode_total=3248128,filesystem_inode_used=140272 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=minuteman network_in_bytes=0i,network_in_dropped=0,network_in_errors=0,network_in_packets=0,network_out_bytes=210i,network_out_dropped=0,network_out_errors=0,network_out_packets=3 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=eth0 network_in_bytes=539886216i,network_in_dropped=1,network_in_errors=0,network_in_packets=979808,network_out_bytes=112395836i,network_out_dropped=0,network_out_errors=0,network_out_packets=891239 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=spartan network_in_bytes=0i,network_in_dropped=0,network_in_errors=0,network_in_packets=0,network_out_bytes=210i,network_out_dropped=0,network_out_errors=0,network_out_packets=3 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,path=/var/lib/docker/overlay filesystem_capacity_free_bytes=1668378624i,filesystem_capacity_total_bytes=6641680384i,filesystem_capacity_used_bytes=4973301760i,filesystem_inode_free=3107856,filesystem_inode_total=3248128,filesystem_inode_used=140272 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=vtep1024 network_in_bytes=0i,network_in_dropped=0,network_in_errors=0,network_in_packets=0,network_out_bytes=0i,network_out_dropped=0,network_out_errors=0,network_out_packets=0 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,path=/var/lib/docker/plugins filesystem_capacity_free_bytes=1668378624i,filesystem_capacity_total_bytes=6641680384i,filesystem_capacity_used_bytes=4973301760i,filesystem_inode_free=3107856,filesystem_inode_total=3248128,filesystem_inode_used=140272 1511859222000000000
|
||||
dcos_node,cluster=enterprise,hostname=192.168.122.18,interface=d-dcos network_in_bytes=0i,network_in_dropped=0,network_in_errors=0,network_in_packets=0,network_out_bytes=0i,network_out_dropped=0,network_out_errors=0,network_out_packets=0 1511859222000000000
|
||||
dcos_app,cluster=enterprise,container_id=9a78d34a-3bbf-467e-81cf-a57737f154ee,hostname=192.168.122.18 container_received_bytes_per_sec=0,container_throttled_bytes_per_sec=0 1511859222000000000
|
||||
dcos_container,cluster=enterprise,container_id=cbf19b77-3b8d-4bcf-b81f-824b67279629,hostname=192.168.122.18 cpus_limit=0.3,cpus_system_time=307.31,cpus_throttled_time=102.029930607,cpus_user_time=268.57,disk_limit_bytes=268435456i,disk_used_bytes=30953472i,mem_limit_bytes=570425344i,mem_total_bytes=13316096i,net_rx_bytes=0i,net_rx_dropped=0,net_rx_errors=0,net_rx_packets=0,net_tx_bytes=0i,net_tx_dropped=0,net_tx_errors=0,net_tx_packets=0 1511859222000000000
|
||||
dcos_app,cluster=enterprise,container_id=cbf19b77-3b8d-4bcf-b81f-824b67279629,hostname=192.168.122.18 container_received_bytes_per_sec=0,container_throttled_bytes_per_sec=0 1511859222000000000
|
||||
dcos_container,cluster=enterprise,container_id=5725e219-f66e-40a8-b3ab-519d85f4c4dc,hostname=192.168.122.18,task_name=hello-world cpus_limit=0.6,cpus_system_time=25.6,cpus_throttled_time=327.977109217,cpus_user_time=566.54,disk_limit_bytes=0i,disk_used_bytes=0i,mem_limit_bytes=1107296256i,mem_total_bytes=335941632i,net_rx_bytes=0i,net_rx_dropped=0,net_rx_errors=0,net_rx_packets=0,net_tx_bytes=0i,net_tx_dropped=0,net_tx_errors=0,net_tx_packets=0 1511859222000000000
|
||||
dcos_app,cluster=enterprise,container_id=5725e219-f66e-40a8-b3ab-519d85f4c4dc,hostname=192.168.122.18 container_received_bytes_per_sec=0,container_throttled_bytes_per_sec=0 1511859222000000000
|
||||
dcos_app,cluster=enterprise,container_id=c76e1488-4fb7-4010-a4cf-25725f8173f9,hostname=192.168.122.18 container_received_bytes_per_sec=0,container_throttled_bytes_per_sec=0 1511859222000000000
|
||||
dcos_container,cluster=enterprise,container_id=cbe0b2f9-061f-44ac-8f15-4844229e8231,hostname=192.168.122.18,task_name=telegraf cpus_limit=0.2,cpus_system_time=8.109999999,cpus_throttled_time=93.183916045,cpus_user_time=17.97,disk_limit_bytes=0i,disk_used_bytes=0i,mem_limit_bytes=167772160i,mem_total_bytes=0i,net_rx_bytes=0i,net_rx_dropped=0,net_rx_errors=0,net_rx_packets=0,net_tx_bytes=0i,net_tx_dropped=0,net_tx_errors=0,net_tx_packets=0 1511859222000000000
|
||||
dcos_container,cluster=enterprise,container_id=b64115de-3d2a-431d-a805-76e7c46453f1,hostname=192.168.122.18 cpus_limit=0.2,cpus_system_time=2.69,cpus_throttled_time=20.064861214,cpus_user_time=6.56,disk_limit_bytes=268435456i,disk_used_bytes=29360128i,mem_limit_bytes=297795584i,mem_total_bytes=13733888i,net_rx_bytes=0i,net_rx_dropped=0,net_rx_errors=0,net_rx_packets=0,net_tx_bytes=0i,net_tx_dropped=0,net_tx_errors=0,net_tx_packets=0 1511859222000000000
|
||||
dcos_app,cluster=enterprise,container_id=b64115de-3d2a-431d-a805-76e7c46453f1,hostname=192.168.122.18 container_received_bytes_per_sec=0,container_throttled_bytes_per_sec=0 1511859222000000000
|
||||
```
|
328
plugins/inputs/dcos/client.go
Normal file
328
plugins/inputs/dcos/client.go
Normal file
|
@ -0,0 +1,328 @@
|
|||
package dcos
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/golang-jwt/jwt/v5"
|
||||
)
|
||||
|
||||
const (
|
||||
// How long to stayed logged in for
|
||||
loginDuration = 65 * time.Minute
|
||||
)
|
||||
|
||||
// client is an interface for communicating with the DC/OS API.
|
||||
type client interface {
|
||||
setToken(token string)
|
||||
|
||||
login(ctx context.Context, sa *serviceAccount) (*authToken, error)
|
||||
getSummary(ctx context.Context) (*summary, error)
|
||||
getContainers(ctx context.Context, node string) ([]container, error)
|
||||
getNodeMetrics(ctx context.Context, node string) (*metrics, error)
|
||||
getContainerMetrics(ctx context.Context, node, container string) (*metrics, error)
|
||||
getAppMetrics(ctx context.Context, node, container string) (*metrics, error)
|
||||
}
|
||||
|
||||
type apiError struct {
|
||||
url string
|
||||
statusCode int
|
||||
title string
|
||||
description string
|
||||
}
|
||||
|
||||
// login is request data for logging in.
|
||||
type login struct {
|
||||
UID string `json:"uid"`
|
||||
Exp int64 `json:"exp"`
|
||||
Token string `json:"token"`
|
||||
}
|
||||
|
||||
// loginError is the response when login fails.
|
||||
type loginError struct {
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
}
|
||||
|
||||
// loginAuth is the response to a successful login.
|
||||
type loginAuth struct {
|
||||
Token string `json:"token"`
|
||||
}
|
||||
|
||||
// slave is a node in the cluster.
|
||||
type slave struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
|
||||
// summary provides high level cluster wide information.
|
||||
type summary struct {
|
||||
Cluster string
|
||||
Slaves []slave
|
||||
}
|
||||
|
||||
// container is a container on a node.
|
||||
type container struct {
|
||||
ID string
|
||||
}
|
||||
|
||||
type dataPoint struct {
|
||||
Name string `json:"name"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
Unit string `json:"unit"`
|
||||
Value float64 `json:"value"`
|
||||
}
|
||||
|
||||
// metrics are the DCOS metrics
|
||||
type metrics struct {
|
||||
Datapoints []dataPoint `json:"datapoints"`
|
||||
Dimensions map[string]interface{} `json:"dimensions"`
|
||||
}
|
||||
|
||||
// authToken is the authentication token.
|
||||
type authToken struct {
|
||||
Text string
|
||||
Expire time.Time
|
||||
}
|
||||
|
||||
// clusterClient is a client that uses the cluster URL.
|
||||
type clusterClient struct {
|
||||
clusterURL *url.URL
|
||||
httpClient *http.Client
|
||||
token string
|
||||
semaphore chan struct{}
|
||||
}
|
||||
|
||||
type claims struct {
|
||||
UID string `json:"uid"`
|
||||
jwt.RegisteredClaims
|
||||
}
|
||||
|
||||
func (e apiError) Error() string {
|
||||
if e.description != "" {
|
||||
return fmt.Sprintf("[%s] %s: %s", e.url, e.title, e.description)
|
||||
}
|
||||
return fmt.Sprintf("[%s] %s", e.url, e.title)
|
||||
}
|
||||
|
||||
func newClusterClient(clusterURL *url.URL, timeout time.Duration, maxConns int, tlsConfig *tls.Config) *clusterClient {
|
||||
httpClient := &http.Client{
|
||||
Transport: &http.Transport{
|
||||
MaxIdleConns: maxConns,
|
||||
TLSClientConfig: tlsConfig,
|
||||
},
|
||||
Timeout: timeout,
|
||||
}
|
||||
semaphore := make(chan struct{}, maxConns)
|
||||
|
||||
c := &clusterClient{
|
||||
clusterURL: clusterURL,
|
||||
httpClient: httpClient,
|
||||
semaphore: semaphore,
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (c *clusterClient) setToken(token string) {
|
||||
c.token = token
|
||||
}
|
||||
|
||||
func (c *clusterClient) login(ctx context.Context, sa *serviceAccount) (*authToken, error) {
|
||||
token, err := createLoginToken(sa)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
exp := time.Now().Add(loginDuration)
|
||||
|
||||
body := &login{
|
||||
UID: sa.accountID,
|
||||
Exp: exp.Unix(),
|
||||
Token: token,
|
||||
}
|
||||
|
||||
octets, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
loc := c.toURL("/acs/api/v1/auth/login")
|
||||
req, err := http.NewRequest("POST", loc, bytes.NewBuffer(octets))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Add("Content-Type", "application/json")
|
||||
|
||||
req = req.WithContext(ctx)
|
||||
resp, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
auth := &loginAuth{}
|
||||
dec := json.NewDecoder(resp.Body)
|
||||
err = dec.Decode(auth)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
token := &authToken{
|
||||
Text: auth.Token,
|
||||
Expire: exp,
|
||||
}
|
||||
return token, nil
|
||||
}
|
||||
|
||||
loginError := &loginError{}
|
||||
dec := json.NewDecoder(resp.Body)
|
||||
err = dec.Decode(loginError)
|
||||
if err != nil {
|
||||
err := &apiError{
|
||||
url: loc,
|
||||
statusCode: resp.StatusCode,
|
||||
title: resp.Status,
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
|
||||
err = &apiError{
|
||||
url: loc,
|
||||
statusCode: resp.StatusCode,
|
||||
title: loginError.Title,
|
||||
description: loginError.Description,
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
|
||||
func (c *clusterClient) getSummary(ctx context.Context) (*summary, error) {
|
||||
summary := &summary{}
|
||||
err := c.doGet(ctx, c.toURL("/mesos/master/state-summary"), summary)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return summary, nil
|
||||
}
|
||||
|
||||
func (c *clusterClient) getContainers(ctx context.Context, node string) ([]container, error) {
|
||||
list := make([]string, 0)
|
||||
err := c.doGet(ctx, c.toURL(fmt.Sprintf("/system/v1/agent/%s/metrics/v0/containers", node)), &list)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
containers := make([]container, 0, len(list))
|
||||
for _, c := range list {
|
||||
containers = append(containers, container{ID: c})
|
||||
}
|
||||
|
||||
return containers, nil
|
||||
}
|
||||
|
||||
func (c *clusterClient) getMetrics(ctx context.Context, address string) (*metrics, error) {
|
||||
metrics := &metrics{}
|
||||
|
||||
err := c.doGet(ctx, address, metrics)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return metrics, nil
|
||||
}
|
||||
|
||||
func (c *clusterClient) getNodeMetrics(ctx context.Context, node string) (*metrics, error) {
|
||||
path := fmt.Sprintf("/system/v1/agent/%s/metrics/v0/node", node)
|
||||
return c.getMetrics(ctx, c.toURL(path))
|
||||
}
|
||||
|
||||
func (c *clusterClient) getContainerMetrics(ctx context.Context, node, container string) (*metrics, error) {
|
||||
path := fmt.Sprintf("/system/v1/agent/%s/metrics/v0/containers/%s", node, container)
|
||||
return c.getMetrics(ctx, c.toURL(path))
|
||||
}
|
||||
|
||||
func (c *clusterClient) getAppMetrics(ctx context.Context, node, container string) (*metrics, error) {
|
||||
path := fmt.Sprintf("/system/v1/agent/%s/metrics/v0/containers/%s/app", node, container)
|
||||
return c.getMetrics(ctx, c.toURL(path))
|
||||
}
|
||||
|
||||
func createGetRequest(address, token string) (*http.Request, error) {
|
||||
req, err := http.NewRequest("GET", address, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if token != "" {
|
||||
req.Header.Add("Authorization", "token="+token)
|
||||
}
|
||||
req.Header.Add("Accept", "application/json")
|
||||
|
||||
return req, nil
|
||||
}
|
||||
|
||||
func (c *clusterClient) doGet(ctx context.Context, address string, v interface{}) error {
|
||||
req, err := createGetRequest(address, c.token)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
select {
|
||||
case c.semaphore <- struct{}{}:
|
||||
break
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
resp, err := c.httpClient.Do(req.WithContext(ctx))
|
||||
if err != nil {
|
||||
<-c.semaphore
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
resp.Body.Close()
|
||||
<-c.semaphore
|
||||
}()
|
||||
|
||||
// Clear invalid token if unauthorized
|
||||
if resp.StatusCode == http.StatusUnauthorized {
|
||||
c.token = ""
|
||||
}
|
||||
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return &apiError{
|
||||
url: address,
|
||||
statusCode: resp.StatusCode,
|
||||
title: resp.Status,
|
||||
}
|
||||
}
|
||||
|
||||
if resp.StatusCode == http.StatusNoContent {
|
||||
return nil
|
||||
}
|
||||
|
||||
err = json.NewDecoder(resp.Body).Decode(v)
|
||||
return err
|
||||
}
|
||||
|
||||
func (c *clusterClient) toURL(path string) string {
|
||||
clusterURL := *c.clusterURL
|
||||
clusterURL.Path = path
|
||||
return clusterURL.String()
|
||||
}
|
||||
|
||||
func createLoginToken(sa *serviceAccount) (string, error) {
|
||||
token := jwt.NewWithClaims(jwt.SigningMethodRS256, claims{
|
||||
UID: sa.accountID,
|
||||
RegisteredClaims: jwt.RegisteredClaims{
|
||||
// How long we have to login with this token
|
||||
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Minute * 5)),
|
||||
},
|
||||
})
|
||||
return token.SignedString(sa.privateKey)
|
||||
}
|
223
plugins/inputs/dcos/client_test.go
Normal file
223
plugins/inputs/dcos/client_test.go
Normal file
|
@ -0,0 +1,223 @@
|
|||
package dcos
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
|
||||
"github.com/golang-jwt/jwt/v5"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/influxdata/telegraf/testutil"
|
||||
)
|
||||
|
||||
var privateKey = testutil.NewPKI("../../../testutil/pki").ReadServerKey()
|
||||
|
||||
func TestLogin(t *testing.T) {
|
||||
ts := httptest.NewServer(http.NotFoundHandler())
|
||||
defer ts.Close()
|
||||
|
||||
var tests = []struct {
|
||||
name string
|
||||
responseCode int
|
||||
responseBody string
|
||||
expectedError error
|
||||
expectedToken string
|
||||
}{
|
||||
{
|
||||
name: "Login successful",
|
||||
responseCode: http.StatusOK,
|
||||
responseBody: `{"token": "XXX.YYY.ZZZ"}`,
|
||||
expectedError: nil,
|
||||
expectedToken: "XXX.YYY.ZZZ",
|
||||
},
|
||||
{
|
||||
name: "Unauthorized Error",
|
||||
responseCode: http.StatusUnauthorized,
|
||||
responseBody: `{"title": "x", "description": "y"}`,
|
||||
expectedError: &apiError{
|
||||
url: ts.URL + "/acs/api/v1/auth/login",
|
||||
statusCode: http.StatusUnauthorized,
|
||||
title: "x",
|
||||
description: "y",
|
||||
},
|
||||
expectedToken: "",
|
||||
},
|
||||
}
|
||||
|
||||
key, err := jwt.ParseRSAPrivateKeyFromPEM([]byte(privateKey))
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
ts.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(tt.responseCode)
|
||||
fmt.Fprintln(w, tt.responseBody)
|
||||
})
|
||||
|
||||
u, err := url.Parse(ts.URL)
|
||||
require.NoError(t, err)
|
||||
|
||||
sa := &serviceAccount{
|
||||
accountID: "telegraf",
|
||||
privateKey: key,
|
||||
}
|
||||
client := newClusterClient(u, defaultResponseTimeout, 1, nil)
|
||||
auth, err := client.login(t.Context(), sa)
|
||||
|
||||
require.Equal(t, tt.expectedError, err)
|
||||
|
||||
if tt.expectedToken != "" {
|
||||
require.Equal(t, tt.expectedToken, auth.Text)
|
||||
} else {
|
||||
require.Nil(t, auth)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetSummary(t *testing.T) {
|
||||
ts := httptest.NewServer(http.NotFoundHandler())
|
||||
defer ts.Close()
|
||||
|
||||
var tests = []struct {
|
||||
name string
|
||||
responseCode int
|
||||
responseBody string
|
||||
expectedValue *summary
|
||||
expectedError error
|
||||
}{
|
||||
{
|
||||
name: "No nodes",
|
||||
responseCode: http.StatusOK,
|
||||
responseBody: `{"cluster": "a", "slaves": []}`,
|
||||
expectedValue: &summary{Cluster: "a", Slaves: make([]slave, 0)},
|
||||
expectedError: nil,
|
||||
},
|
||||
{
|
||||
name: "Unauthorized Error",
|
||||
responseCode: http.StatusUnauthorized,
|
||||
responseBody: `<html></html>`,
|
||||
expectedValue: nil,
|
||||
expectedError: &apiError{
|
||||
url: ts.URL + "/mesos/master/state-summary",
|
||||
statusCode: http.StatusUnauthorized,
|
||||
title: "401 Unauthorized",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "Has nodes",
|
||||
responseCode: http.StatusOK,
|
||||
responseBody: `{"cluster": "a", "slaves": [{"id": "a"}, {"id": "b"}]}`,
|
||||
expectedValue: &summary{
|
||||
Cluster: "a",
|
||||
Slaves: []slave{
|
||||
{ID: "a"},
|
||||
{ID: "b"},
|
||||
},
|
||||
},
|
||||
expectedError: nil,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
ts.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
// check the path
|
||||
w.WriteHeader(tt.responseCode)
|
||||
fmt.Fprintln(w, tt.responseBody)
|
||||
})
|
||||
|
||||
u, err := url.Parse(ts.URL)
|
||||
require.NoError(t, err)
|
||||
|
||||
client := newClusterClient(u, defaultResponseTimeout, 1, nil)
|
||||
summary, err := client.getSummary(t.Context())
|
||||
|
||||
require.Equal(t, tt.expectedError, err)
|
||||
require.Equal(t, tt.expectedValue, summary)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetNodeMetrics(t *testing.T) {
|
||||
ts := httptest.NewServer(http.NotFoundHandler())
|
||||
defer ts.Close()
|
||||
|
||||
var tests = []struct {
|
||||
name string
|
||||
responseCode int
|
||||
responseBody string
|
||||
expectedValue *metrics
|
||||
expectedError error
|
||||
}{
|
||||
{
|
||||
name: "Empty Body",
|
||||
responseCode: http.StatusOK,
|
||||
responseBody: `{}`,
|
||||
expectedValue: &metrics{},
|
||||
expectedError: nil,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
ts.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
// check the path
|
||||
w.WriteHeader(tt.responseCode)
|
||||
fmt.Fprintln(w, tt.responseBody)
|
||||
})
|
||||
|
||||
u, err := url.Parse(ts.URL)
|
||||
require.NoError(t, err)
|
||||
|
||||
client := newClusterClient(u, defaultResponseTimeout, 1, nil)
|
||||
m, err := client.getNodeMetrics(t.Context(), "foo")
|
||||
|
||||
require.Equal(t, tt.expectedError, err)
|
||||
require.Equal(t, tt.expectedValue, m)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetContainerMetrics(t *testing.T) {
|
||||
ts := httptest.NewServer(http.NotFoundHandler())
|
||||
defer ts.Close()
|
||||
|
||||
var tests = []struct {
|
||||
name string
|
||||
responseCode int
|
||||
responseBody string
|
||||
expectedValue *metrics
|
||||
expectedError error
|
||||
}{
|
||||
{
|
||||
name: "204 No Content",
|
||||
responseCode: http.StatusNoContent,
|
||||
responseBody: ``,
|
||||
expectedValue: &metrics{},
|
||||
expectedError: nil,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
ts.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
// check the path
|
||||
w.WriteHeader(tt.responseCode)
|
||||
fmt.Fprintln(w, tt.responseBody)
|
||||
})
|
||||
|
||||
u, err := url.Parse(ts.URL)
|
||||
require.NoError(t, err)
|
||||
|
||||
client := newClusterClient(u, defaultResponseTimeout, 1, nil)
|
||||
m, err := client.getContainerMetrics(t.Context(), "foo", "bar")
|
||||
|
||||
require.Equal(t, tt.expectedError, err)
|
||||
require.Equal(t, tt.expectedValue, m)
|
||||
})
|
||||
}
|
||||
}
|
72
plugins/inputs/dcos/creds.go
Normal file
72
plugins/inputs/dcos/creds.go
Normal file
|
@ -0,0 +1,72 @@
|
|||
package dcos
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rsa"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
const (
|
||||
// How long before expiration to renew token
|
||||
relogDuration = 5 * time.Minute
|
||||
)
|
||||
|
||||
type credentials interface {
|
||||
token(ctx context.Context, client client) (string, error)
|
||||
isExpired() bool
|
||||
}
|
||||
|
||||
type serviceAccount struct {
|
||||
accountID string
|
||||
privateKey *rsa.PrivateKey
|
||||
|
||||
auth *authToken
|
||||
}
|
||||
|
||||
type tokenCreds struct {
|
||||
Path string
|
||||
}
|
||||
|
||||
type nullCreds struct {
|
||||
}
|
||||
|
||||
func (c *serviceAccount) token(ctx context.Context, client client) (string, error) {
|
||||
auth, err := client.login(ctx, c)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
c.auth = auth
|
||||
return auth.Text, nil
|
||||
}
|
||||
|
||||
func (c *serviceAccount) isExpired() bool {
|
||||
return c.auth.Text != "" || c.auth.Expire.Add(relogDuration).After(time.Now())
|
||||
}
|
||||
|
||||
func (c *tokenCreds) token(_ context.Context, _ client) (string, error) {
|
||||
octets, err := os.ReadFile(c.Path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("error reading token file %q: %w", c.Path, err)
|
||||
}
|
||||
if !utf8.Valid(octets) {
|
||||
return "", fmt.Errorf("token file does not contain utf-8 encoded text: %s", c.Path)
|
||||
}
|
||||
token := strings.TrimSpace(string(octets))
|
||||
return token, nil
|
||||
}
|
||||
|
||||
func (*tokenCreds) isExpired() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
func (*nullCreds) token(context.Context, client) (string, error) {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (*nullCreds) isExpired() bool {
|
||||
return true
|
||||
}
|
387
plugins/inputs/dcos/dcos.go
Normal file
387
plugins/inputs/dcos/dcos.go
Normal file
|
@ -0,0 +1,387 @@
|
|||
//go:generate ../../../tools/readme_config_includer/generator
|
||||
package dcos
|
||||
|
||||
import (
|
||||
"context"
|
||||
_ "embed"
|
||||
"errors"
|
||||
"net/url"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/golang-jwt/jwt/v5"
|
||||
|
||||
"github.com/influxdata/telegraf"
|
||||
"github.com/influxdata/telegraf/config"
|
||||
"github.com/influxdata/telegraf/filter"
|
||||
"github.com/influxdata/telegraf/plugins/common/tls"
|
||||
"github.com/influxdata/telegraf/plugins/inputs"
|
||||
)
|
||||
|
||||
//go:embed sample.conf
|
||||
var sampleConfig string
|
||||
|
||||
var (
|
||||
nodeDimensions = []string{
|
||||
"hostname",
|
||||
"path",
|
||||
"interface",
|
||||
}
|
||||
containerDimensions = []string{
|
||||
"hostname",
|
||||
"container_id",
|
||||
"task_name",
|
||||
}
|
||||
appDimensions = []string{
|
||||
"hostname",
|
||||
"container_id",
|
||||
"task_name",
|
||||
}
|
||||
)
|
||||
|
||||
const (
|
||||
defaultMaxConnections = 10
|
||||
defaultResponseTimeout = 20 * time.Second
|
||||
)
|
||||
|
||||
type DCOS struct {
|
||||
ClusterURL string `toml:"cluster_url"`
|
||||
|
||||
ServiceAccountID string `toml:"service_account_id"`
|
||||
ServiceAccountPrivateKey string `toml:"service_account_private_key"`
|
||||
|
||||
TokenFile string `toml:"token_file"`
|
||||
|
||||
NodeInclude []string `toml:"node_include"`
|
||||
NodeExclude []string `toml:"node_exclude"`
|
||||
ContainerInclude []string `toml:"container_include"`
|
||||
ContainerExclude []string `toml:"container_exclude"`
|
||||
AppInclude []string `toml:"app_include"`
|
||||
AppExclude []string `toml:"app_exclude"`
|
||||
|
||||
MaxConnections int `toml:"max_connections"`
|
||||
ResponseTimeout config.Duration `toml:"response_timeout"`
|
||||
tls.ClientConfig
|
||||
|
||||
client client
|
||||
creds credentials
|
||||
|
||||
initialized bool
|
||||
nodeFilter filter.Filter
|
||||
containerFilter filter.Filter
|
||||
appFilter filter.Filter
|
||||
}
|
||||
|
||||
type point struct {
|
||||
tags map[string]string
|
||||
labels map[string]string
|
||||
fields map[string]interface{}
|
||||
}
|
||||
|
||||
func (*DCOS) SampleConfig() string {
|
||||
return sampleConfig
|
||||
}
|
||||
|
||||
func (d *DCOS) Gather(acc telegraf.Accumulator) error {
|
||||
err := d.initialize()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
token, err := d.creds.token(ctx, d.client)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d.client.setToken(token)
|
||||
|
||||
summary, err := d.client.getSummary(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for _, node := range summary.Slaves {
|
||||
wg.Add(1)
|
||||
go func(node string) {
|
||||
defer wg.Done()
|
||||
d.gatherNode(ctx, acc, summary.Cluster, node)
|
||||
}(node.ID)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *DCOS) gatherNode(ctx context.Context, acc telegraf.Accumulator, cluster, node string) {
|
||||
if !d.nodeFilter.Match(node) {
|
||||
return
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
m, err := d.client.getNodeMetrics(ctx, node)
|
||||
if err != nil {
|
||||
acc.AddError(err)
|
||||
return
|
||||
}
|
||||
addNodeMetrics(acc, cluster, m)
|
||||
}()
|
||||
|
||||
d.gatherContainers(ctx, acc, cluster, node)
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func (d *DCOS) gatherContainers(ctx context.Context, acc telegraf.Accumulator, cluster, node string) {
|
||||
containers, err := d.client.getContainers(ctx, node)
|
||||
if err != nil {
|
||||
acc.AddError(err)
|
||||
return
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for _, container := range containers {
|
||||
if d.containerFilter.Match(container.ID) {
|
||||
wg.Add(1)
|
||||
go func(container string) {
|
||||
defer wg.Done()
|
||||
m, err := d.client.getContainerMetrics(ctx, node, container)
|
||||
if err != nil {
|
||||
var apiErr apiError
|
||||
if errors.As(err, &apiErr) && apiErr.statusCode == 404 {
|
||||
return
|
||||
}
|
||||
acc.AddError(err)
|
||||
return
|
||||
}
|
||||
addContainerMetrics(acc, cluster, m)
|
||||
}(container.ID)
|
||||
}
|
||||
|
||||
if d.appFilter.Match(container.ID) {
|
||||
wg.Add(1)
|
||||
go func(container string) {
|
||||
defer wg.Done()
|
||||
m, err := d.client.getAppMetrics(ctx, node, container)
|
||||
if err != nil {
|
||||
var apiErr apiError
|
||||
if errors.As(err, &apiErr) && apiErr.statusCode == 404 {
|
||||
return
|
||||
}
|
||||
acc.AddError(err)
|
||||
return
|
||||
}
|
||||
addAppMetrics(acc, cluster, m)
|
||||
}(container.ID)
|
||||
}
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func createPoints(m *metrics) []*point {
|
||||
points := make(map[string]*point)
|
||||
for _, dp := range m.Datapoints {
|
||||
fieldKey := strings.ReplaceAll(dp.Name, ".", "_")
|
||||
|
||||
tags := dp.Tags
|
||||
if tags == nil {
|
||||
tags = make(map[string]string)
|
||||
}
|
||||
|
||||
if dp.Unit == "bytes" && !strings.HasSuffix(fieldKey, "_bytes") {
|
||||
fieldKey = fieldKey + "_bytes"
|
||||
}
|
||||
|
||||
fieldKey = strings.TrimPrefix(fieldKey, "dcos_metrics_module_")
|
||||
|
||||
tagset := make([]string, 0, len(tags))
|
||||
for k, v := range tags {
|
||||
tagset = append(tagset, k+"="+v)
|
||||
}
|
||||
sort.Strings(tagset)
|
||||
seriesParts := make([]string, 0, len(tagset))
|
||||
seriesParts = append(seriesParts, tagset...)
|
||||
seriesKey := strings.Join(seriesParts, ",")
|
||||
|
||||
p, ok := points[seriesKey]
|
||||
if !ok {
|
||||
p = &point{}
|
||||
p.tags = tags
|
||||
p.labels = make(map[string]string)
|
||||
p.fields = make(map[string]interface{})
|
||||
points[seriesKey] = p
|
||||
}
|
||||
|
||||
if dp.Unit == "bytes" {
|
||||
p.fields[fieldKey] = int64(dp.Value)
|
||||
} else {
|
||||
p.fields[fieldKey] = dp.Value
|
||||
}
|
||||
}
|
||||
|
||||
results := make([]*point, 0, len(points))
|
||||
for _, p := range points {
|
||||
for k, v := range m.Dimensions {
|
||||
switch v := v.(type) {
|
||||
case string:
|
||||
p.tags[k] = v
|
||||
case map[string]string:
|
||||
if k == "labels" {
|
||||
for k, v := range v {
|
||||
p.labels[k] = v
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
results = append(results, p)
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
func addMetrics(acc telegraf.Accumulator, cluster, mname string, m *metrics, tagDimensions []string) {
|
||||
tm := time.Now()
|
||||
|
||||
points := createPoints(m)
|
||||
|
||||
for _, p := range points {
|
||||
tags := make(map[string]string)
|
||||
tags["cluster"] = cluster
|
||||
for _, tagkey := range tagDimensions {
|
||||
v, ok := p.tags[tagkey]
|
||||
if ok {
|
||||
tags[tagkey] = v
|
||||
}
|
||||
}
|
||||
for k, v := range p.labels {
|
||||
tags[k] = v
|
||||
}
|
||||
|
||||
acc.AddFields(mname, p.fields, tags, tm)
|
||||
}
|
||||
}
|
||||
|
||||
func addNodeMetrics(acc telegraf.Accumulator, cluster string, m *metrics) {
|
||||
addMetrics(acc, cluster, "dcos_node", m, nodeDimensions)
|
||||
}
|
||||
|
||||
func addContainerMetrics(acc telegraf.Accumulator, cluster string, m *metrics) {
|
||||
addMetrics(acc, cluster, "dcos_container", m, containerDimensions)
|
||||
}
|
||||
|
||||
func addAppMetrics(acc telegraf.Accumulator, cluster string, m *metrics) {
|
||||
addMetrics(acc, cluster, "dcos_app", m, appDimensions)
|
||||
}
|
||||
|
||||
func (d *DCOS) initialize() error {
|
||||
if !d.initialized {
|
||||
err := d.createFilters()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if d.client == nil {
|
||||
client, err := d.createClient()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d.client = client
|
||||
}
|
||||
|
||||
if d.creds == nil {
|
||||
creds, err := d.createCredentials()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d.creds = creds
|
||||
}
|
||||
|
||||
d.initialized = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *DCOS) createClient() (client, error) {
|
||||
tlsCfg, err := d.ClientConfig.TLSConfig()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
address, err := url.Parse(d.ClusterURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
client := newClusterClient(
|
||||
address,
|
||||
time.Duration(d.ResponseTimeout),
|
||||
d.MaxConnections,
|
||||
tlsCfg,
|
||||
)
|
||||
|
||||
return client, nil
|
||||
}
|
||||
|
||||
func (d *DCOS) createCredentials() (credentials, error) {
|
||||
if d.ServiceAccountID != "" && d.ServiceAccountPrivateKey != "" {
|
||||
bs, err := os.ReadFile(d.ServiceAccountPrivateKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
privateKey, err := jwt.ParseRSAPrivateKeyFromPEM(bs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
creds := &serviceAccount{
|
||||
accountID: d.ServiceAccountID,
|
||||
privateKey: privateKey,
|
||||
}
|
||||
return creds, nil
|
||||
} else if d.TokenFile != "" {
|
||||
creds := &tokenCreds{
|
||||
Path: d.TokenFile,
|
||||
}
|
||||
return creds, nil
|
||||
}
|
||||
|
||||
return &nullCreds{}, nil
|
||||
}
|
||||
|
||||
func (d *DCOS) createFilters() error {
|
||||
var err error
|
||||
d.nodeFilter, err = filter.NewIncludeExcludeFilter(
|
||||
d.NodeInclude, d.NodeExclude)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
d.containerFilter, err = filter.NewIncludeExcludeFilter(
|
||||
d.ContainerInclude, d.ContainerExclude)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
d.appFilter, err = filter.NewIncludeExcludeFilter(
|
||||
d.AppInclude, d.AppExclude)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func init() {
|
||||
inputs.Add("dcos", func() telegraf.Input {
|
||||
return &DCOS{
|
||||
MaxConnections: defaultMaxConnections,
|
||||
ResponseTimeout: config.Duration(defaultResponseTimeout),
|
||||
}
|
||||
})
|
||||
}
|
436
plugins/inputs/dcos/dcos_test.go
Normal file
436
plugins/inputs/dcos/dcos_test.go
Normal file
|
@ -0,0 +1,436 @@
|
|||
package dcos
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/influxdata/telegraf/testutil"
|
||||
)
|
||||
|
||||
type mockClient struct {
|
||||
SetTokenF func()
|
||||
LoginF func(ctx context.Context, sa *serviceAccount) (*authToken, error)
|
||||
GetSummaryF func() (*summary, error)
|
||||
GetContainersF func() ([]container, error)
|
||||
GetNodeMetricsF func() (*metrics, error)
|
||||
GetContainerMetricsF func(ctx context.Context, node, container string) (*metrics, error)
|
||||
GetAppMetricsF func(ctx context.Context, node, container string) (*metrics, error)
|
||||
}
|
||||
|
||||
func (c *mockClient) setToken(string) {
|
||||
c.SetTokenF()
|
||||
}
|
||||
|
||||
func (c *mockClient) login(ctx context.Context, sa *serviceAccount) (*authToken, error) {
|
||||
return c.LoginF(ctx, sa)
|
||||
}
|
||||
|
||||
func (c *mockClient) getSummary(context.Context) (*summary, error) {
|
||||
return c.GetSummaryF()
|
||||
}
|
||||
|
||||
func (c *mockClient) getContainers(context.Context, string) ([]container, error) {
|
||||
return c.GetContainersF()
|
||||
}
|
||||
|
||||
func (c *mockClient) getNodeMetrics(context.Context, string) (*metrics, error) {
|
||||
return c.GetNodeMetricsF()
|
||||
}
|
||||
|
||||
func (c *mockClient) getContainerMetrics(ctx context.Context, node, container string) (*metrics, error) {
|
||||
return c.GetContainerMetricsF(ctx, node, container)
|
||||
}
|
||||
|
||||
func (c *mockClient) getAppMetrics(ctx context.Context, node, container string) (*metrics, error) {
|
||||
return c.GetAppMetricsF(ctx, node, container)
|
||||
}
|
||||
|
||||
func TestAddNodeMetrics(t *testing.T) {
|
||||
var tests = []struct {
|
||||
name string
|
||||
metrics *metrics
|
||||
check func(*testutil.Accumulator) []bool
|
||||
}{
|
||||
{
|
||||
name: "basic datapoint conversion",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "process.count",
|
||||
Unit: "count",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
},
|
||||
"process_count", 42.0,
|
||||
)}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "path added as tag",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "filesystem.inode.free",
|
||||
Tags: map[string]string{
|
||||
"path": "/var/lib",
|
||||
},
|
||||
Unit: "count",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"path": "/var/lib",
|
||||
},
|
||||
"filesystem_inode_free", 42.0,
|
||||
)}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "interface added as tag",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "network.out.dropped",
|
||||
Tags: map[string]string{
|
||||
"interface": "eth0",
|
||||
},
|
||||
Unit: "count",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"interface": "eth0",
|
||||
},
|
||||
"network_out_dropped", 42.0,
|
||||
)}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "bytes unit appended to fieldkey",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "network.in",
|
||||
Tags: map[string]string{
|
||||
"interface": "eth0",
|
||||
},
|
||||
Unit: "bytes",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"interface": "eth0",
|
||||
},
|
||||
"network_in_bytes", int64(42),
|
||||
)}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "dimensions added as tags",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "process.count",
|
||||
Tags: map[string]string{},
|
||||
Unit: "count",
|
||||
Value: 42.0,
|
||||
},
|
||||
{
|
||||
Name: "memory.total",
|
||||
Tags: map[string]string{},
|
||||
Unit: "bytes",
|
||||
Value: 42,
|
||||
},
|
||||
},
|
||||
Dimensions: map[string]interface{}{
|
||||
"cluster_id": "c0760bbd-9e9d-434b-bd4a-39c7cdef8a63",
|
||||
"hostname": "192.168.122.18",
|
||||
"mesos_id": "2dfbbd28-29d2-411d-92c4-e2f84c38688e-S1",
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{
|
||||
acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"hostname": "192.168.122.18",
|
||||
},
|
||||
"process_count", 42.0),
|
||||
acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"hostname": "192.168.122.18",
|
||||
},
|
||||
"memory_total_bytes", int64(42)),
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var acc testutil.Accumulator
|
||||
addNodeMetrics(&acc, "a", tt.metrics)
|
||||
for i, ok := range tt.check(&acc) {
|
||||
require.Truef(t, ok, "Index was not true: %d", i)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddContainerMetrics(t *testing.T) {
|
||||
var tests = []struct {
|
||||
name string
|
||||
metrics *metrics
|
||||
check func(*testutil.Accumulator) []bool
|
||||
}{
|
||||
{
|
||||
name: "container",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "net.rx.errors",
|
||||
Tags: map[string]string{
|
||||
"container_id": "f25c457b-fceb-44f0-8f5b-38be34cbb6fb",
|
||||
"executor_id": "telegraf.192fb45f-cc0c-11e7-af48-ea183c0b541a",
|
||||
"executor_name": "Command Executor (Task: telegraf.192fb45f-cc0c-11e7-af48-ea183c0b541a) (Command: NO EXECUTABLE)",
|
||||
"framework_id": "ab2f3a8b-06db-4e8c-95b6-fb1940874a30-0001",
|
||||
"source": "telegraf.192fb45f-cc0c-11e7-af48-ea183c0b541a",
|
||||
},
|
||||
Unit: "count",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
Dimensions: map[string]interface{}{
|
||||
"cluster_id": "c0760bbd-9e9d-434b-bd4a-39c7cdef8a63",
|
||||
"container_id": "f25c457b-fceb-44f0-8f5b-38be34cbb6fb",
|
||||
"executor_id": "telegraf.192fb45f-cc0c-11e7-af48-ea183c0b541a",
|
||||
"framework_id": "ab2f3a8b-06db-4e8c-95b6-fb1940874a30-0001",
|
||||
"framework_name": "marathon",
|
||||
"framework_principal": "dcos_marathon",
|
||||
"framework_role": "slave_public",
|
||||
"hostname": "192.168.122.18",
|
||||
"labels": map[string]string{
|
||||
"DCOS_SPACE": "/telegraf",
|
||||
},
|
||||
"mesos_id": "2dfbbd28-29d2-411d-92c4-e2f84c38688e-S1",
|
||||
"task_id": "telegraf.192fb45f-cc0c-11e7-af48-ea183c0b541a",
|
||||
"task_name": "telegraf",
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{
|
||||
acc.HasPoint(
|
||||
"dcos_container",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"container_id": "f25c457b-fceb-44f0-8f5b-38be34cbb6fb",
|
||||
"hostname": "192.168.122.18",
|
||||
"task_name": "telegraf",
|
||||
"DCOS_SPACE": "/telegraf",
|
||||
},
|
||||
"net_rx_errors",
|
||||
42.0,
|
||||
),
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var acc testutil.Accumulator
|
||||
addContainerMetrics(&acc, "a", tt.metrics)
|
||||
for i, ok := range tt.check(&acc) {
|
||||
require.Truef(t, ok, "Index was not true: %d", i)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddAppMetrics(t *testing.T) {
|
||||
var tests = []struct {
|
||||
name string
|
||||
metrics *metrics
|
||||
check func(*testutil.Accumulator) []bool
|
||||
}{
|
||||
{
|
||||
name: "tags are optional",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "dcos.metrics.module.container_throttled_bytes_per_sec",
|
||||
Unit: "",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{
|
||||
acc.HasPoint(
|
||||
"dcos_app",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
},
|
||||
"container_throttled_bytes_per_sec", 42.0,
|
||||
),
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "dimensions are tagged",
|
||||
metrics: &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "dcos.metrics.module.container_throttled_bytes_per_sec",
|
||||
Unit: "",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
Dimensions: map[string]interface{}{
|
||||
"cluster_id": "c0760bbd-9e9d-434b-bd4a-39c7cdef8a63",
|
||||
"container_id": "02d31175-1c01-4459-8520-ef8b1339bc52",
|
||||
"hostname": "192.168.122.18",
|
||||
"mesos_id": "2dfbbd28-29d2-411d-92c4-e2f84c38688e-S1",
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{
|
||||
acc.HasPoint(
|
||||
"dcos_app",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"container_id": "02d31175-1c01-4459-8520-ef8b1339bc52",
|
||||
"hostname": "192.168.122.18",
|
||||
},
|
||||
"container_throttled_bytes_per_sec", 42.0,
|
||||
),
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var acc testutil.Accumulator
|
||||
addAppMetrics(&acc, "a", tt.metrics)
|
||||
for i, ok := range tt.check(&acc) {
|
||||
require.Truef(t, ok, "Index was not true: %d", i)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGatherFilterNode(t *testing.T) {
|
||||
var tests = []struct {
|
||||
name string
|
||||
nodeInclude []string
|
||||
nodeExclude []string
|
||||
client client
|
||||
check func(*testutil.Accumulator) []bool
|
||||
}{
|
||||
{
|
||||
name: "cluster without nodes has no metrics",
|
||||
client: &mockClient{
|
||||
SetTokenF: func() {},
|
||||
GetSummaryF: func() (*summary, error) {
|
||||
return &summary{
|
||||
Cluster: "a",
|
||||
}, nil
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{
|
||||
acc.NMetrics() == 0,
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "node include",
|
||||
nodeInclude: []string{"x"},
|
||||
client: &mockClient{
|
||||
SetTokenF: func() {},
|
||||
GetSummaryF: func() (*summary, error) {
|
||||
return &summary{
|
||||
Cluster: "a",
|
||||
Slaves: []slave{
|
||||
{ID: "x"},
|
||||
{ID: "y"},
|
||||
},
|
||||
}, nil
|
||||
},
|
||||
GetContainersF: func() ([]container, error) {
|
||||
return nil, nil
|
||||
},
|
||||
GetNodeMetricsF: func() (*metrics, error) {
|
||||
return &metrics{
|
||||
Datapoints: []dataPoint{
|
||||
{
|
||||
Name: "value",
|
||||
Value: 42.0,
|
||||
},
|
||||
},
|
||||
Dimensions: map[string]interface{}{
|
||||
"hostname": "x",
|
||||
},
|
||||
}, nil
|
||||
},
|
||||
},
|
||||
check: func(acc *testutil.Accumulator) []bool {
|
||||
return []bool{
|
||||
acc.HasPoint(
|
||||
"dcos_node",
|
||||
map[string]string{
|
||||
"cluster": "a",
|
||||
"hostname": "x",
|
||||
},
|
||||
"value", 42.0,
|
||||
),
|
||||
acc.NMetrics() == 1,
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var acc testutil.Accumulator
|
||||
dcos := &DCOS{
|
||||
NodeInclude: tt.nodeInclude,
|
||||
NodeExclude: tt.nodeExclude,
|
||||
client: tt.client,
|
||||
}
|
||||
err := dcos.Gather(&acc)
|
||||
require.NoError(t, err)
|
||||
for i, ok := range tt.check(&acc) {
|
||||
require.Truef(t, ok, "Index was not true: %d", i)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
42
plugins/inputs/dcos/sample.conf
Normal file
42
plugins/inputs/dcos/sample.conf
Normal file
|
@ -0,0 +1,42 @@
|
|||
# Input plugin for DC/OS metrics
|
||||
[[inputs.dcos]]
|
||||
## The DC/OS cluster URL.
|
||||
cluster_url = "https://dcos-master-1"
|
||||
|
||||
## The ID of the service account.
|
||||
service_account_id = "telegraf"
|
||||
## The private key file for the service account.
|
||||
service_account_private_key = "/etc/telegraf/telegraf-sa-key.pem"
|
||||
|
||||
## Path containing login token. If set, will read on every gather.
|
||||
# token_file = "/home/dcos/.dcos/token"
|
||||
|
||||
## In all filter options if both include and exclude are empty all items
|
||||
## will be collected. Arrays may contain glob patterns.
|
||||
##
|
||||
## Node IDs to collect metrics from. If a node is excluded, no metrics will
|
||||
## be collected for its containers or apps.
|
||||
# node_include = []
|
||||
# node_exclude = []
|
||||
## Container IDs to collect container metrics from.
|
||||
# container_include = []
|
||||
# container_exclude = []
|
||||
## Container IDs to collect app metrics from.
|
||||
# app_include = []
|
||||
# app_exclude = []
|
||||
|
||||
## Maximum concurrent connections to the cluster.
|
||||
# max_connections = 10
|
||||
## Maximum time to receive a response from cluster.
|
||||
# response_timeout = "20s"
|
||||
|
||||
## Optional TLS Config
|
||||
# tls_ca = "/etc/telegraf/ca.pem"
|
||||
# tls_cert = "/etc/telegraf/cert.pem"
|
||||
# tls_key = "/etc/telegraf/key.pem"
|
||||
## If false, skip chain & host verification
|
||||
# insecure_skip_verify = true
|
||||
|
||||
## Recommended filtering to reduce series cardinality.
|
||||
# [inputs.dcos.tagdrop]
|
||||
# path = ["/var/lib/mesos/slave/slaves/*"]
|
Loading…
Add table
Add a link
Reference in a new issue