dns-oarc/oarc-dsc-datatool
1
0
Fork 0
Code Issues Activity

Adding upstream version 1.4.2.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-04-22 13:23:23 +02:00
parent e344d0b8ae
commit 1ea3e103a7
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
77 changed files with 5760 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

188
man/man1/dsc-datatool.1 Normal file
View file

@ -0,0 +1,188 @@
.TH "dsc-datatool" "1"
.SH NAME
dsc-datatool \- Tool for converting, exporting, merging and transforming DSC data.
.SH SYNOPSIS
.SY dsc-datatool
.OP \-h
.OP \-c CONF
.OP \-s SERVER
.OP \-n NODE
.OP \-x XML
.OP \-d DAT
.OP \-\-dataset DATASET
.OP \-o OUTPUT
.OP \-t TRANSFORM]
.OP \-g GENERATOR
.OP \-\-list
.OP \-\-skipped\-key SKIPPED_KEY
.OP \-\-skipped\-sum\-key SKIPPED_SUM_KEY
.OP \-v
.OP \-V
.YS
.SH DESCRIPTION
Tool for converting, exporting, merging and transforming DSC data.
Please have a look at the wiki article on how to set this up using
Influx DB and Grafana.
https://github.com/DNS-OARC/dsc-datatool/wiki/Setting-up-a-test-Grafana
.SH OPTIONS
.TP
.B -h, --help
show this help message and exit
.TP
.BI "-c " CONF ", --conf " CONF
Not implemented
.TP
.BI "-s " SERVER ", --server " SERVER
Specify the server for where the data comes from. (required)
.TP
.BI "-n " NODE ", --node " NODE
Specify the node for where the data comes from. (required)
.TP
.BI "-x " XML ", --xml " XML
Read DSC data from the given file or directory, can be specified multiple
times.
If a directory is given then all files ending with .xml will be read.
.TP
.BI "-d " DAT ", --dat " DAT
Read DSC data from the given directory, can be specified multiple times.
Note that the DAT format is depended on the filename to know what type of
data it is.
.TP
.BI "--dataset " DATASET
Specify that only the list of datasets will be processed, the list is
comma separated and the option can be given multiple times.
.TP
.BI "-o " OUTPUT ", --output " OUTPUT
.I OUTPUT
has the following format that uses
.I output
to specify the output module and
.I sep
as an options separator.
.EX
<sep><output>[<sep>option=value...]>
.EE
Can be specified multiple times to output to more then one.
Use
.B dsc-datatool --list
to see a list of modules and the man-page of each output for information
about options.
.TP
.BI "-t " TRANSFORM ", --transform " TRANSFORM
.I TRANSFORM
has the following format that uses
.I name
to specify the transformer module and
.I sep
as an options separator.
The
.I datasets
field can specify which dataset to run the transformer on, or use
.I *
to specify all datasets.
.EX
<sep><name><sep><datasets>[<sep>option=value...]>
.EE
Can be specific multiple times to chain transformation, the chain will be
executed in the order on command line with one exception.
All transformations specified for dataset
.I *
will be executed before named dataset transformations.
Use
.B dsc-datatool --list
to see a list of modules and the man-page of each transformer for
information about options.
For a list of datasets see the DSC configuration that creates the XML files
or the documentation for the Presenter that creates the DAT files.
.TP
.BI "-g " GENERATOR ", --generator " GENERATOR
.I GENERATOR
has two formats, one to specify a comma separated list of generators
and one that uses
.I name
to specify the generator module and
.I sep
as an options separator.
.EX
<name>[,<name>,...]
<sep><name>[<sep>option=value...]>
.EE
This option can be given multiple times.
Use
.B dsc-datatool --list
to see a list of modules and the man-page of each generator for
information about options.
.TP
.B --list
List the available generators, transformers and outputs then exit.
.TP
.BI "--skipped-key " SKIPPED_KEY
Set the special DSC skipped key. (default to "-:SKIPPED:-")
.TP
.BI "--skipped-sum-key " SKIPPED_SUM_KEY
Set the special DSC skipped sum key. (default to "-:SKIPPED_SUM:-")
.TP
.BI "--encoding " ENCODING
Set the encoding to use when reading and writing files, default to utf-8.
.TP
.B -v, --verbose
Increase the verbose level, can be given multiple times.
.TP
.B -V, --version
Display version and exit.
.LP
.SH EXAMPLE
.EX
dsc-datatool \\
--server "$SERVER" \\
--node "$NODE" \\
--output ";InfluxDB;file=influx.txt;dml=1;database=dsc" \\
--transform ";Labler;*;yaml=$HOME/labler.yaml" \\
--transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \\
--transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \\
--transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \\
--transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \\
--transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \\
--transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=8" \\
--generator client_subnet_country \\
--generator ";client_subnet_authority;fetch=yes" \\
--xml "$XML"
.EE
.SH "SEE ALSO"
.BR dsc-datatool.conf (5),
.BI dsc-datatool- [ generator | transformer | output ]
.BR <name> (7)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

29
man/man5/dsc-datatool.conf.5 Normal file
View file

@ -0,0 +1,29 @@
.TH "dsc-datatool.conf" "5"
.SH NAME
dsc-datatool.conf \- Configuration file for dsc-datatool.
.SH DESCRIPTION
Not implemented.
.SH "SEE ALSO"
.BR dsc-datatool (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

66
man/man7/dsc-datatool-generator-client_subnet_authority.7 Normal file
View file

@ -0,0 +1,66 @@
.TH "dsc-datatool-generator client_subnet_authority" "7"
.SH NAME
client_subnet_authority \- Generates network authority information (RIR) based on client subnet.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-generator
.I """;client_subnet_authority;<options...>"""
.YS
.SH DESCRIPTION
This generator will lookup the network authority (RIR) based network
information in the
.I client_subnet
dataset and generate a new dataset
.IR client_subnet_authority .
The network authority lookup will be done from imports of CSV files or by
fetching the required data from IANA.
.SH OPTIONS
.TP
.BR csv =<files>
Read the network authority information from the given
.IR file(s) .
This option can be a single file, a list of comma separated files or given
multiple times.
.TP
.BR fetch =yes
Fetch the network authority information from IANA IP address space registry
for IPv4 and IPv6.
.TP
.BR urlv4 =url
URL for IANA IPv4 address space registry, default to
.IR https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv .
.TP
.BR urlv6 =url
URL for IANA IPv6 address space registry, default to
.IR https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv .
.TP
.BR nonstrict =<value>
If set (to any value) then a non-strict mode is enabled which will disregard
any bad data in the dataset, skipping it completely.
Default is strict mode.
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

67
man/man7/dsc-datatool-generator-client_subnet_country.7 Normal file
View file

@ -0,0 +1,67 @@
.TH "dsc-datatool-generator client_subnet_country" "7"
.SH NAME
client_subnet_country \- Generates country information based on client subnet using Maxmind database.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-generator
.I """;client_subnet_country;<options...>"""
.YS
.SH DESCRIPTION
This generator looks up country code in a Maxmind database for the subnets
in the
.I client_subnet
dataset and the dataset
.I client_subnet_country
from that.
.SH OPTIONS
.TP
.BR path =<directory>
Search for the Maxmind database in the specified
.IR directory .
This option can be given multiple times to search in multiple directories.
Default search paths are:
.IR /var/lib/GeoIP ,
.IR /usr/share/GeoIP ,
.IR /usr/local/share/GeoIP .
.TP
.BR filename =filename
The database filename to search for, default to
.IR GeoLite2-Country.mmdb .
.TP
.BR db =/path/to/database.mmdb
Full path to Maxmind database, this option overrides
.I path
and
.IR filename .
.TP
.BR nonstrict =<value>
If set (to any value) then a non-strict mode is enabled which will disregard
any bad data in the dataset, skipping it completely.
Default is strict mode.
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1),
.BR geoipupdate (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

64
man/man7/dsc-datatool-output-influxdb.7 Normal file
View file

@ -0,0 +1,64 @@
.TH "dsc-datatool-output influxdb" "7"
.SH NAME
InfluxDB \- InfluxDB output.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-output
.I """;InfluxDB;<options...>"""
.YS
.SH DESCRIPTION
This output generates InfluxDB importable output to stdout or to a specified
file.
.SH OPTIONS
.TP
.BR timestamp =[start|stop]
Choose which timestamp from the dataset to use in the InfluxDB output to
specify when the metrics took place.
Default to
.I start
timestamp.
.TP
.BR file =<filename>
Specify a file to output to instead of stdout.
.TP
.BR append
If given, the output will be appended to the file specified rather then
overwritten.
.TP
.BR dml
Add a
.I "# DML"
header to the output.
.TP
.BR database =name
If
.I dml
is used, this can be used to specify which database the output should be
imported into.
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1),
.BR influx (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

78
man/man7/dsc-datatool-output-prometheus.7 Normal file
View file

@ -0,0 +1,78 @@
.TH "dsc-datatool-output prometheus" "7"
.SH NAME
Prometheus \- Prometheus output.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-output
.I """;Prometheus;<options...>"""
.YS
.SH DESCRIPTION
This output generates Prometheus importable output to stdout or to a specified
file.
.SS Prometheus' node_exporter
This output can be used together with Prometheus'
.IR node_exporter 's
Textfile Collector to automate statistics gathering but some specific
setup and requirements must be meet.
You must hide the timestamp with option
.B timestamp=hide
because timestamps are not supported by the Textfile Collector.
You must make sure only one XML file from a server+node combination is
processed at a time.
Because otherwise you will get multiple data point for the same metric
and this will generate errors from the Textfile Collector, since it does
not support timestamps and cannot separate the measurements.
You must make sure that only one file (per server+node combo) is generated
for the Textfile Collector to read, and it should be the same between runs.
See Textfile Collectors documentation how to setup that atomically.
.SH OPTIONS
.TP
.BR timestamp =[hide|start|stop]
Choose which timestamp from the dataset to use in the Prometheus output to
specify when the metrics took place.
Default to
.I start
timestamp.
.TP
.BR file =<filename>
Specify a file to output to instead of stdout.
.TP
.BR append
If given, the output will be appended to the file specified rather then
overwritten.
.TP
.BR prefix =<string>
Use the given string as prefix on all metric names.
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1)
.I https://prometheus.io/docs/guides/node-exporter/
.I https://github.com/prometheus/node_exporter#textfile-collector
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

62
man/man7/dsc-datatool-transformer-labler.7 Normal file
View file

@ -0,0 +1,62 @@
.TH "dsc-datatool-transformer labler" "7"
.SH NAME
Labler \- Rewrite numeric labels to textual labels using the provided YAML data as lookup tables.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-transform
.I """;Labler;<dataset>;yaml=<file>"""
.YS
.SH DESCRIPTION
This transformer rewrites labels in datasets based on information provided
in a YAML file.
The YAML structure is as follows:
.EX
---
dataset_name:
DimentionName:
DimentionValue: RewriteTo
.EE
The
.I dataset_name
is the name of the dataset,
.I DimentionName
is the name of the dimension to rewrite and
.I DimensionValue
is the value to change to
.IR RewriteTo .
.SH OPTIONS
.TP
.B dataset
See
.IR dsc-datatool (1)
on how to specify which dataset(s) to run the transformer on.
.TP
.BR yaml =<file>
The YAML file to load rewrite data from (required).
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

60
man/man7/dsc-datatool-transformer-netremap.7 Normal file
View file

@ -0,0 +1,60 @@
.TH "dsc-datatool-transformer netremap" "7"
.SH NAME
NetRemap \- Remap network addresses to other ranges/subnets.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-transform
.I """;NetRemap;<dataset>;<options...>"""
.YS
.SH DESCRIPTION
This transformer can remap network ranges on IP addresses and subnets.
.SH OPTIONS
.TP
.B dataset
See
.IR dsc-datatool (1)
on how to specify which dataset(s) to run the transformer on.
.TP
.BR net =<network>
The network to remap both IPv4 and IPv6 addresses and networks, specify
only the number in a short-form network (e.g. "/<network>").
.TP
.BR v4net =<network>
The IPv4 network to use for IPv4 addresses and networks, default to
.I net
if that is specified.
.TP
.BR v6net =<network>
The IPv6 network to use for IPv6 addresses and networks, default to
.I net
if that is specified.
.TP
.BR nonstrict =<value>
If set (to any value) then a non-strict mode is enabled which will disregard
any bad data in the dataset, skipping it completely.
Default is strict mode.
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP

75
man/man7/dsc-datatool-transformer-reranger.7 Normal file
View file

@ -0,0 +1,75 @@
.TH "dsc-datatool-transformer reranger" "7"
.SH NAME
ReRanger \- Rewrite ranged or numerical statistics into other ranges.
.SH SYNOPSIS
.SY dsc-datatool
.B \-\-transform
.I """;ReRanger;<dataset>;<options...>"""
.YS
.SH DESCRIPTION
This transformer can re-range values in datasets to new rangers.
It supports both values that are a single numerical value or one that is
already a range itself
.RI ( num\-num ).
.SH OPTIONS
.TP
.B dataset
See
.IR dsc-datatool (1)
on how to specify which dataset(s) to run the transformer on.
.TP
.BR key =[low|mid|high]
Specify what value to use when re-ranging an already ranged value
.RI (low\-high).
Default to
.IR mid ,
which is high minus low then divided by 2.
.TP
.BR func =<func>
The function to use when aggregating multiple values into a new ranger,
default to
.IR sum .
Currently only one function exists,
.IR sum ,
which adds the sum of all values into the new range.
.TP
.B allow_invalid_keys
If given the "invalid" keys/values that is not numerical or a range will be
passed through unmodified.
.TP
.BR range =</num>
The new range given as
.I /number
(required).
For example
.I range=/64
will re-range all values into buckets of 64, 0-63, 64-127 and so on.
.LP
.SH "SEE ALSO"
.BR dsc-datatool (1)
.SH AUTHORS
Jerry Lundström, DNS-OARC
.LP
Maintained by DNS-OARC
.LP
.RS
.I https://www.dns-oarc.net/tools/dsc
.RE
.LP
.SH BUGS
For issues and feature requests please use:
.LP
.RS
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
.RE
.LP
For question and help please use:
.LP
.RS
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
.RE
.LP