oarc-dsc/CHANGES

2024-08-30 Jerry Lundström

    Release 2.15.2

    This releases fixes 3 issues detected by code analysis tools:
    - File not closed and memory not freed during error while loading known
      TLD file
    - https://github.com/DNS-OARC/dsc/security/code-scanning/22
      label buffer should be static
    - https://github.com/DNS-OARC/dsc/security/code-scanning/20
      unsigned difference expression

    855f037 CodeQL
    b00cb11 Stack
    c4d3634 Sonar
    26c3b9e Badges, fixes
    b4a9171 Workflow

2024-04-23 Jerry Lundström

    Release 2.15.1

    This release fixes client subnet indexer which overwrote the mask
    options during initialization so the conf option `client_v4_mask` and
    `client_v6_mask` was never used.

    Other changes:
    - Update documentation
    - Update builtin known TLDs based on PSL
    - Update copyright year

    d577a97 Copyright
    f71edff Known TLDs
    dedafdd Client mask
    8ef947c Doc

2023-08-09 Jerry Lundström

    Release 2.15.0

    This release fixes DNS parsing w.r.t. EDNS, implements better loop
    detection during name decompression and adds a lot of EDNS indexers
    and filters.

    Previously the DNS parser expected the additional records to come
    straight after the question section. Meaning that if the DNS packet
    had any answer or authority records, they would be parsed as additional
    records for the OPT record and EDNS information.

    Following new indexers has been added:
    - edns_cookie
    - edns_cookie_len
    - edns_cookie_client
    - edns_cookie_server
    - edns_ecs
    - edns_ecs_family
    - edns_ecs_source_prefix
    - edns_ecs_scope_prefix
    - edns_ecs_address
    - edns_ecs_subnet
    - edns_ede
    - edns_ede_code
    - edns_ede_textlen
    - edns_ede_text
    - edns_nsid
    - edns_nsid_len
    - edns_nsid_data
    - edns_nsid_text

    Following new filters has been added:
    - edns0-only
    - edns0-cookie-only
    - edns0-nsid-only
    - edns0-ede-only
    - edns0-ecs-only

    See man-page dsc.conf(5) for more information.

    Other fixes/additions:
    - Only parse entire DNS message if EDNS indexers are used
    - `dns_protocol`: Implement proper loop detection during decompression
    - `xmalloc`: Check return of `amalloc()` before using `memset()`/`memcpy()` because it's undefined behavior on null pointers

    8259f30 EDNS filters
    41f3b9a strtohex, nsid text
    a666c04 EDNS(0) Client Subnet
    b5164fe EDNS
    7cabfd9 EDNS0 parsing fixes and additional EDNS0 indexers.
    46b1797 memcpy/memset fixes
    8fd7b7a EDNS parsing
    cee2bf7 EDNS0 parsing, multi RR test
    a2c00c9 DNS compression loop detection
    9875a3e RR parsing

2023-06-15 Jerry Lundström

    Release 2.14.1

    Fixed a bug in TLD handling when using `tld_list`, it did not reset
    where it was in the QNAME when nothing was found and could therefor
    wrongly indicate something as a TLD.

    Also fixed a typo in the `dsc.conf` man-page.

    976589d GCOV
    c3afee4 TLD list, doc typo

2023-04-03 Jerry Lundström

    Release 2.14.0

    This release adds new configure option to control the file access to
    the output files, support for newer DNSTAP, improved DNSTAP message
    handling and updated Public Suffix List.

    - Fix #279: Add new conf options to control output file access:
      - `output_user`: set output file user ownership
      - `output_group`: set output file group ownership
      - `output_mod`: set output file mode bits
    - `dnstap`: Move DNSTAP essential attributes checks inside each type and customize them for that specific type
    - Update dnswire dependencies to v0.4.0
    - `encryption_index`: Add support for new DNSTAP DNS-over-QUIC socket protocol
    - Update builtin Public Suffix List (PSL)

    abfe245 DNSTAP
    da06317 Output file access
    af01a48 DOQ transport, PSL update

2023-02-10 Jerry Lundström

    Release 2.13.2

    Updated pcap-thread to v4.0.1:

    Fixed issue with `pcap_dispatch()` during non-threaded timed runs by
    checking packet timestamp and use `pcap_breakloop()` if the run
    should end.
    Based on reports, it looks like `pcap_dispatch()` won't stop
    processing if load is high enough even if documentation says "only
    one bufferful of packets is read at a time".

    Many thanks to Klaus Darilion @klaus3000 (NIC.AT) for the report
    and helping to track down the issue and test fixes!

    e7d92fe Fix COPR
    7ecf217 pcap-thread

2022-04-21 Jerry Lundström

    Release 2.13.1

    This patch release is mainly for build and packages where MaxMind DB
    library is preferred over the legacy GeoIP library.
    MaxMind has announced that the databases for GeoIP will be EOL May 2022
    and recommends switching to GeoIP2 databases.

    Also updated DSC's description, removing references to the now
    discontinued Presenter and pointing to dsc-datatool instead.

    d891e2c Package, description
    c23406c Optional GeoIP
    26dd506 GeoIP

2022-01-28 Jerry Lundström

    Release 2.13.0

    This release fixes a huge performance issue with hashing IPv6
    addresses, adds support for new DNSTAP messages types and protocols,
    and adds two new indexers.

    Thanks to a patch sent in by Ken Renard (@kdrenard) a rather huge
    performance issue related to hashing IPv6 addresses has been solved.
    Old code used a very incorrect assumption about addresses in general
    and while same way was used for IPv4, it didn't hit as hard as it did
    for IPv6.
    New code uses hashing functions on both address types and to quote
    the GitHub issue (by Ken):
    -"This performs about 5% better than what I did (51 sec versus 54 sec)
      for 5GB pcap file with nearly 50/50 split of IPv4 and IPv6 (3.7M/3.5M
      v4/v6 queries).
      Old inXaddr_hash() has been running for 75 minutes and is about 20%
      done. I say this is a winner!"

    Many thanks to Ken for pointing this out and supplying a patch!

    DSC now depends on dnswire v0.3.0 which includes new DNSTAP messages
    types and protocols that was recently added to DNSTAP's Protobuf
    definition.
    The new `UPDATE_QUERY` and `UPDATE_RESPONSE` messages types are
    now supported and are interpret as `AUTH_QUERY` and `AUTH_RESPONSE`.
    The new socket protocols for DOT, DOH and DNSCrypt are also supported
    and are interpret as TCP for indexers such as `ip_proto` and
    `transport`. To get stats on the encryption itself you can use the
    new indexer `encryption`.

    Two new indexers have been added:
    - `label_count`: Number of labels in the QNAME
    - `encryption`: Indicates whether the DNS message was carried over an
      encrypted connection or not, and if so over which. For example
      "unencrypted", "dot" (DNS-over-TLS), "doh" (DNS-over-HTTPS).

    Other changes:
    - `inX_addr`: Rework structure, separate IPv4 and IPv6 addresses
    - Fix some DNSTAP tests
    - `transport_index`: Fix typo in code documentation

    37df703 DNSTAP update, encryption indexer
    d27171f Label count indexer
    6932247 Adding labellen indexer which counts the number of labels in a DNS message
    68cc9c7 New IP hashing

2022-01-13 Jerry Lundström

    Release 2.12.0

    This release adds a new conf option `tld_list` to control what DSC
    considers are TLDs, and a script to convert the Public Suffix List to
    this format (see `man dsc-psl-convert` for more information).

    For example, using this option will allow DSC to gather statistics on
    domains like `co.uk` and `net.au` that would otherwise be counted as
    `uk` and `au`.

    The release also updates the man-pages, clarifying how to use multiple
    `interface` and other similar options. And removes the deprecated cron
    upload scripts.

    e779a87 Remove upload scripts
    2880f93 PSL TLD list
    ea04022 Update Copyright and known TLDs
    5cbc7a4 Output format
    b7e6c35 Doc
    e66dae4 dh_auto_test
    6a3e817 debhelper
    89d033f Bye Travis
    fa1c179 Mattermost

2020-10-20 Jerry Lundström

    Release 2.11.2

    This release fixes a bug in `asn_indexer` that didn't enabled the usage
    of MaxMindDB after successful initiation. Other changes include a typo
    fix in `configure` and a lot of coverage tests.

    395b11a Travis, configure
    ffea9ed Tests
    8b0bebd Tests
    09f8174 Config tests
    d1514d4 Coverage
    66b018c Coverage, ASN indexer

2020-08-18 Jerry Lundström

    Release 2.11.1

    This release fixes a 17-year old code cut&paste mistake in the
    classification indexer, until now it's been classifying funny query
    types based on the query class. This fix was sent in by Jim Hague
    (Sinodun), thanks Jim!

    Other changes are based on code analysis reports and setup for code
    coverage.

    8d4763c Correct funny-qtype classification.
    a1dd55e getline
    29bd143 Coverage
    685e504 SonarCloud
    f759515 Badges

2020-06-01 Jerry Lundström

    Release 2.11.0

    This release updates the built in known TLDs table and adds the optional
    configuration option `knowntlds_file` to, instead of using the built in
    table, load the data from a file.

    If compiled with only MaxMindDB support then ASN and Country indexer
    would complain (and exit) that no database has been specified.
    This release changes the behavior to match that of GeoIP support,
    making it possible to run without specifying a database.

    Other changes:
    - Fix compile warnings
    - COPR packaging fixes
    - `country_indexer`: Fixed typos in log messages (was copied from ASN)
    - Fix issues and false-positives reported by newer version of scan-build

    Commits:
    e937d1 COPR
    1382370 country, asn
    423a813 scanbuild
    2571b97 Compile warnings
    4f69447 Known TLDs

2020-05-07 Jerry Lundström

    Release 2.10.0

    This release adds new configuration options to `dnstap_unixsock` to
    control ownership and permissions for the DNSTAP socket file.

    Other fixes:
    - Unlink the DNSTAP socket file if an error during initialization occur
    - Do hard exit in forks to not run `atexit()` (which will unlink the
      DNSTAP socket file)

    Commits:
    9d1d49a fork
    733b286 DNSTAP socket

2020-04-02 Jerry Lundström

    Release 2.9.1

    This release fixes a few bugs, removes a lot of the debug messages
    about DNSTAP and removes GeoIP from openSUSE/SLE packages as it has
    been deprecated on those platforms.

    Changes:
    - `daemon`: Fix bug with listening for SIGINT when in foreground mode
    - `dnstap`:
      - Fix #217: Unlink UNIX socket on exit if successfully initiated
      - Fix startup bug, `exit()` if unable to initialize
      - Fix #220:
        - Remove/hide a lot of debug messages and the printing of the DNSTAP message
        - Clarify a lot of the info and error messages
        - Prefix all DNSTAP related messages with `DNSTAP: `
    - Fix compile warnings and include headers when GeoIP is missing
    - `asn_indexer`: Fix bug, said unknown IPv4 when it was IPv6

    Commits:
    08bad5b DNSTAP debug
    1232264 LGTM
    589ea7a GeoIP, asn indexer
    4fea0d2 sigint, DNSTAP UNIX socket, DNSTAP init

2020-03-20 Jerry Lundström

    Release 2.9.0

    This release adds support for receiving DNS messages over DNSTAP along
    with documentation updates and eliminated compiler warnings.

    To enable DNSTAP support, install dependencies (check `README.md`) and
    run configure with `--enable-dnstap`.

    New configuration options:
    - `dnstap_file`: specify input from DNSTAP file
    - `dnstap_unixsock`: specify DNSTAP input from UNIX socket
    - `dnstap_tcp`: specify DNSTAP input from TCP connections (dsc listens)
    - `dnstap_udp`: specify DNSTAP input from UDP connections (dsc listens)
    - `dnstap_network`: specify network information in place of missing DNSTAP attributes

    Other changes:
    - Add documentation about extra configure options that might be needed for FreeBSD/OpenBSD
    - Fix compile warnings on FreeBSD 11.2
    - Fix compile warning `snprintf()` truncation
    - Packaging updates

    Commits:
    60e6950 DNSTAP
    af0417b README
    1f1b489 COPR, spec
    435e136 Package
    3f24feb FreeBSD 11 compatibility
    563b986 Funding

2019-04-23 Jerry Lundström

    Release 2.8.1

    Added all missing config options for the response time indexer:
    - `response_time_mode`
    - `response_time_bucket_size`
    - `response_time_max_queries`
    - `response_time_full_mode`
    - `response_time_max_seconds`
    - `response_time_max_sec_mode`

    Commits:
    36f0280 Response time config

2019-02-11 Jerry Lundström

    Release 2.8.0

    This release brings an new indexer `response_time` (funded by NIC.AT!),
    support for MaxMind DB (GeoIP2) and an option to set the DNS port.

    The new indexer `response_time` can track queries and report the time
    it took to receive the response in buckets of microseconds or in
    logarithmic scales (see `response_time_mode`). It will also report
    timeouts, missing queries (received a response but have never seen the
    query), dropped queries (due to memory limitations) and internal errors.

    Here is an example output of log10 mode:

    <array name="response_time" dimensions="2" start_time="1478727151"
        stop_time="1478727180">
      <dimension number="1" type="All"/>
      <dimension number="2" type="ResponseTime"/>
      <data>
        <All val="ALL">
          <ResponseTime val="100000-1000000" count="77"/>
          <ResponseTime val="10000-100000" count="42"/>
          <ResponseTime val="1000-10000" count="3"/>
          <ResponseTime val="missing_queries" count="1"/>
        </All>
      </data>
    </array>

    New configuration options:
    - `asn_indexer_backend`: Control what backend to use for the ASN indexer
    - `country_indexer_backend`: Control what backend to use for the
      country indexer
    - `maxminddb_asn`: Specify database for ASN lookups using MaxMind DB
    - `maxminddb_country`: Specify database for country lookups using
      MaxMind DB
    - `dns_port`: Control the DNS port
    - `response_time_mode`: Set the output mode of the response time indexer
    - `response_time_bucket_size`: The size of bucket (microseconds)
    - Following options exists to control internal aspects of `response_time`
      indexer, see man-page for more information:
      - `response_time_max_queries`
      - `response_time_full_mode`
      - `response_time_max_seconds`
      - `response_time_max_sec_mode`

    Fixes:
    - Add LGTM and fix alerts
    - Update `pcap_layers` with fixes for `scan-build` warnings
    - Fix port in debug output of DNS message, was showing server port
      on responses

    Commits:
    f38a655 License
    48cd44e Man-page, interface any, response time
    8b9345f LGTM Alert
    e57a013 DNS port
    38aa018 Response time statistics
    7a60d53 Cleanup
    5c45ce2 Copyright
    0dc8a3c MaxMind DB (GeoIP2)
    473387b LGTM, README, packages, scan-build

2018-08-14 Jerry Lundström

    Release 2.7.0

    Add support for Linux "cooked" capture encapsulation (`DLT_LINUX_SLL`).

    Fixes:
    - `grok_question()`: Remove usage of `strcpy()`
    - `pcap_tcp_handler()`: Use `snprintf()`
    - `printable_dnsname()`: Use `snprintf()`
    - Fix CID 104450, 186871

    Commits:
    41d59ac man-page HTML
    476d6ed pcap_layers, CID
    747131b Configure options
    43c9ad0 DLT_LINUX_SLL
    8a48667 Support the linux cooked sll frame
    bd4a94f Fix CID 104450

2017-08-21 Jerry Lundström

    Release 2.6.1

    Compatibility fixes for FreeBSD 11.1+ which is now packing `struct ip`.

    Commits:
    c0cd375 Handle compile warnings and FreeBSD's packing of structs
    c528ccb Code formatting and moved external code to own directory

2017-07-11 Jerry Lundström

    Release 2.6.0

    Two new DNS filters and configuration for client subnet netmask has been
    added thanks to pull request submission from Manabu Sonoda (@mimuret), see
    `man 5 dsc.conf` for more details.

    New DNS filters:
    - `servfail-only`: Count only SERVFAIL responses
    - `authentic-data-only`: Count only DNS messages with the AD bit is set

    New configuration:
    - `client_v4_mask`: Set the IPv4 MASK for client_subnet INDEXERS
    - `client_v6_mask`: Set the IPv6 MASK for client_subnet INDEXERS

    Fixes:
    - Set `_DEFAULT_SOURCE`, was giving compile warnings on some platforms
    - Update `pcap-thread` to v2.1.3 for compatibility fixes
    - Fix bug where extra `"` would be OK in configuration
    - Eat all white-space between tokens in configuration
    - Minor documentation corrections

    Commits:
    8a20421 Config parse quote/whitespace bug
    4eb91d8 PR review and corrections
    1dcdbc1 add supports statistics for DNSSEC validation resolver - SERVFAIL
            DNS message filter - AD bit DNS message filter - set custom mask
            for ClientSubnet
    7c4ce7e Update pcap-thread to v2.1.3
    f5d152c Corrected date
    04f137d Prepare SPEC for OSB/COPR
    402c242 Config header is generated by autotools

2017-03-29 Jerry Lundström

    Release 2.5.1

    Various compatibility issues and a possible runtime bug, related to
    pcap-thread, fixed.

    Commits:

    5ed03e3 Compat for OS X
    8605759 Fix compiler warnings
    5fbad26 Update pcap-thread to v2.1.2
    47ed110 Update pcap-thread to v2.1.1

2017-03-02 Jerry Lundström

    Release 2.5.0

    Resolved memory leaks within the IP fragment reassembly code that was
    reported by Klaus Darilion (NIC.AT) and added config option to control
    some parts of the fragment handling.

    Fixes:
    - Add `pcap_layers_clear_fragments()` to remove old fragments after
      `MAX_FRAG_IDLE` (60 seconds)
    - Use correct alloc/free functions for dataset hash
    - Fix spacing in dsc.conf(5) man-page

    New config option:
    - `drop_ip_fragments` will disable IP fragmentation reassembling and
      drop any IP packet that is a fragment (even the first)

    Commits:

    eaee6c0 Drop IP fragments
    3ebb687 Issue #146: Fix leak in fragment handling
    9a5e377 Use correct alloc/free
    35f663c Fix #107: add const

2017-01-27 Jerry Lundström

    Release 2.4.0

    Since there have been a few major issues with the threaded capturing code
    it is now default disabled and have to be enabled with a configure option
    to use: `./configure --enable-threads ...`

    A lot of work has been done to ensure stability and correct capturing,
    as of now `dsc` is continuously running on the testing platforms with
    simulated traffic and tests are performance every 5-15 minutes:

      https://dev.dns-oarc.net/jenkins/view/dsctest/

    With the rewrite of the config parser to C it was missed that Hapy allowed
    CR/LF within the values of the options.  Changing the C parser to allow
    it is a bit of work and having CR/LF within the value may lead to other
    issues so it is now documented that CR/LF are not allowed in config option
    values.

    Fixes:
    - The `-T` flag was just controlling pcap-thread usage of threads, it now
      controls all usage of threads including how signals are caught.
    - Fix program name, was incorrectly set so it would be reported as `/dsc`.
    - Use thread safe functions (_r).
    - Handle very long config lines by not having a static buffer, instead
      let `getline()` allocate as needed.
    - Use new activation in pcap-thread to activate the capturing of pcaps
      after the initial interval sync have been done during start-up.
    - Use factions of second for start-up interval sync and interval wait.
    - Fix memory leaks if config options was specified more then once.
    - Use new absolute timed run in pcap-thread to more exactly end capturing
      at the interval.
    - Fix config parsing, was checking for tab when should look for line feed.
    - Exit correctly during pcap-thread run to honor `dump_reports_on_exit`.
    - Use 100ms as default pcap-thread timeout, was 1s before but the old code
      used 250ms.
    - Various enhancements to logging of errors.

    New config options/features:
    - `pcap_buffer_size` can be used to increase the capture buffer within
      pcap-thread/libpcap, this can help mitigate dropped packets by the
      kernel during interval breaks.
    - `no_wait_interval` will skip the interval sync that happens during
      start-up and start capturing directly, the end of the interval will
      still be the modulus of the interval.
    - `pcap_thread_timeout` can be used to change the internal timeout use
      in pcap-thread to wait for packets (default 100ms).
    - Log non-fatal errors from pcap-thread w.r.t. setting the filter which
      can indicate that the filter is running in userland because lack of
      support or that it is too large for the kernel.

    Special thanks to:
    - Anand Buddhdev, RIPE NCC
    - Klaus Darilion, NIC.AT
    - Vincent Charrade, Nameshield

    Commits:

    ee59572 Fix #111, fix #116: Update pcap-thread to v2.0.0, remove debug
            code
    64befef Update copyright year
    40a1fb4 Fix #139: Use 100ms as default pcap-thread timeout
    2a07185 Fix #137: Graceful exit on signal during run
    f1b3ec3 Issue #116: Try and make select issue more clear
    950ea96 Fix #133: Return from `Pcap_run()` on signal/errors
    667cc91 Issue #116: Add config option pcap_thread_timeout
    3c9e073 Notice if non-fatal errors was detected during activation
    4ea8f54 Fix #108: Document that CR/LF are not allowed within configuration
            line
    9fda332 Check for LF and not tab
    15a1dc0 Use pcap-thread timed run to interface
    1e98f8b Fix potential memory leaks if config options specified more then
            once
    a9b38e9 Add missing LF and indicate what config option was wrong if
            possible
    f8a2821 Use fractions of seconds for both start up interval sync and
            timed run, always adjust for inter-run processing delay
    f47069a Fix #121: Update to pcap-thread latest develop
    fc13d73 Issue #116: Feature for not waiting on the interval sync
    c832337 Fix #122: Update pcap-thread to v1.2.3 for fix in timed run
    4739111 Add `pcap_buffer_size` config option
    7d9bf90 Update pcap-thread to v1.2.2
    ef43335 Make threads optional and default disabled
    c2399cf getline() returns error on eof, don't report error if we are
    5c671e6 Clarify config error message and report `getline()` error
    8bd6a67 Fix #114: Handle very long lines
    47b1e1a Use _r thread safe functions when possible
    0f5d883 Update daemon.c
    f18e3ea Update doc, -T now disables all usage of threads
    57aacbe Honor the -T flag when installing signal handlers

2016-12-22 Jerry Lundström

    Release 2.3.0

    Rare lockup has been fixed that could happen if a signal was received
    in the wrong thread at the wrong time due to `pcap_thread_stop()`
    canceling and waiting on threads to join again. The handling of signals
    have been improved for threaded and non-threaded operations.

    A couple of bugfixes, one to fix loading of GeoIP ASN database and
    another to use the lowest 32 bits of an IP address (being v4 or v6)
    in the IP hash making it a bit more efficient for v6 addresses.

    New functionality for the configure option `local_address`, you can now
    specify a network mask (see `man 5 dsc.conf` for syntax).

    Commits:

    e286298 Fix CID 158968 Bad bit shift operation
    c15db43 Update to pcap-thread v1.2.1
    1ac06ac Move stopping process to not require a packet
    597dd34 Handle signals better with and without pthreads
    bcf99e8 Add RPM spec and ACLOCAL_AMFLAGS to build on CentOS 6
    667fe69 fixed load geoIP ASN database from config-file
    e1304d4 Fix #97: Add optional mask to `local_address` so you can
            specify networks
    5dae7dd Fix #96: Hash the lowest 32 bits of IP addresses

2016-10-15 Jerry Lundström

    Release 2.2.1

    Two bugfixes with one being critical (caused segfault, sorry for that)
    - `pcap-thread` had an issue that threads where not closed on exit
      of `pcap_thread_run()`, this only created many threads in my testing
      environment but reports came in that it segfaulted.
    - When started `dsc` (not in debug mode) it will wait to align with
      the time, that did not get update with the configurable interval
      change.

    Commits:

    6e3654b Fix #90: Update pcap-thread to v1.1.2 to fix a segfault issue
    c9350a3 Fix #92: Honor configured interval when aligning
    acaf617 minor typo

2016-10-10 Jerry Lundström

    Release 2.2.0

    Some big changes in this release are the removal of the C++ configure
    parser library Hapy and the addition of pcap-thread to (hopefully)
    handle capturing packet in a correct and efficiant way.

    With that addition of pcap-thread comes new runtime options:
    - `-m` sets monitor mode on interfaces
    - `-i` sets immediate mode on interfaces
    - `-T` disable the usage of threads in pcap thread

    Bugfixes:
    - d95190a fixes a small memory leak in `Pcap_init()` and a possible
      bug where `-p` might never been used because of not being declared
      external.
    - 55e1056 added check for `netinet/ip_compat.h` for use in
      `src/pcap_layers/pcap_layers.c`.

    Commits:

    0a1ce91 Fix coverity issues
    5a1d410 Delete useless line (related to mayasd#84)
    443db3e Check if the file was previously linked but not yet unlinked
            (Tim CLERC.IM)
    02a7621 Fix #82: Oops, `pcap_thread_set_filter()` had changed during
            development and missed this
    2a8aa29 Move definition of token struct inside and changed name to not
            conflict on FreeBSD
    43da964 Fix #9: Implement conf parser in C and remove dependency of
            Hapy and C++
    9f46f0d Update pcap thread to version 1.1.1
    d95190a Use pcap thread, new options `-miT` and possible bugfix for
            `-p` and a small memory leak fix
    55e1056 Fix #77: Check for netinet/ip_compat
    4e120f9 Fix travis script, only expand dir.

2016-06-28 Jerry Lundström

    Release 2.1.1

    Bugfixes:
    - 22688c1 Fix pcap/select last_ts
      In some cases `select` will return the fd set as if there are packets
      to read but there aren't. That would case the last timestamp to not
      advance and the `Pcap_run` loop to never finish.
      This fix add a check on packets captured and sets last timestamp if
      none where caught.

    Commits:

    d115b3f Correct configuration, missing `;`.
    22688c1 Fix pcap/select last_ts
    b6d3dd8 Fix package dependencies.
    c8979c4 Add debian/ubuntu package files

2016-06-09 Jerry Lundström

    Release 2.1.0

    This release brings a couple of new features, the ability to change
    the interval for which DSC writes out the statistics files and a new
    indexer for AS numbers.
    There was also an issue detected in Ubuntu 16.04 when reading pcap files
    would result in doubling the statistics if running in daemon mode.
    Looking at the strace it might be a issue with the kernel, libc, pcap or
    a combination that results in the open file handle to reset and essentially
    read all the data twice.

    Credits:
    - Klaus Darilion NIC.AT for interval changes and AS number indexer.

    Commits:

    8ab8632 Rewrote ASN indexer to remove the need for malloc/free. Removed
            the need to malloc ipstr. Use same unknown tags as country
            indexer.
    89d4984 add ASN index: maps src-IP to AS number using GeoIP ASNum DB
    8d5c6bf Fix #14: Remove ncap
    c0f00e7 Add test for statistics_interval
    cf9ede2 Add checks and documentation for statistics_interval
    93eeecd new config option 'statistics_interval' (defaults to 60s)
    a28f5d4 Fix #62: Do not go into daemon mode if reading offline files
    2bc1abb Add information about puppet module

2016-06-01 Jerry Lundström

    Release 2.0.0

    This release brings a major update to the DSC software with the separation
    of the Collector and the Presenter, this repository will only include
    the Collector from now on.

    Please read UPGRADE.md for information on upgrading from previous version.

    Major changes / additions are:
    - Use of Automake and rework of the Makefiles
    - Conform to FHS 3.0
    - Man-pages (man dsc, man dsc.conf)
    - Continuous Integration testing using Travis-CI
    - Compatibility testing on Debian, CentOS, FreeBSD and OpenBSD
    - Use of Coverity Scan to find defects
    - JSON output format, see output_format in dsc.conf(5).
    - IPv6 support in country indexer and libgeoip is now runtime
      configurable, see geoip_v4_dat / geoip_v6_dat in dsc.conf(5).
    - Signal handling and optional write reports on exit, see
      dump_reports_on_exit in dsc.conf(5).
    - Upload scripts are deprecated

    Credits:
    - Klaus Darilion NIC.AT for GeoIP IPv6 patch.
    - Michael Braunoeder NIC.AT for NXDOMAIN filter patch.
    - L-root for overflow bugfix IP fragments.
    - McStork for JSON output patch.

    Changes since release candidate:

    1be5148 Fix #57: Flush the pid file to write it out and add test for
            pid file
    0f79aa0 Use Semantic Versioning 2.0.0 semver.org

2016-05-10 Jerry Lundström

    Release 2.0.0-rc.1

    69ef9b4 Add -v to display version
    7e5b403 Fix defects
    0f64128 Add badges
    f795ed3 Old automake needs AM_PROG_CC_C_O
    27ae870 Fix #4: Remove old indexers, update indexers and filters
            documentation, update authors.
    d873411 Fix #38: Use locking to ensure we do not overwrite PID file
    800fe83 Fix #35: GeoIP configurable. Fix debug and syslog in country
            indexer. Add documentation and config example.
    a1dcdf1 Avoid the checks for newer .dat files, this may have performance
            impacts.
    bb9a059 Add IPv6 support to country indexer and add some failure handling
    03e16a8 Include config.h as this defines HAVE_LIBGEOIP. Thus, the geoip
            code was not used yet.
    7808d2e New dsyslog/dsyslogf/dfprint macro. Change fprintf to dfprintf.
    fe47288 Fix #34: Create man-pages.
    7a9b3c3 Fix #33: Handle most signals, new config option to dump reports
            on exit.
    4753eda Add a filter to track NXDOMAIN responses
    1565952 Add info about DSP
    05ef699 Fix make test in dist.
    68bdc9b No default interface in conf example
    2be98be Correct libexec directory. Install etc files in subdirectory.
    319ac4e Only build dist. Build in a build directory.
    96e0e73 Fix #16: conform to FHS. Automake tweaks.
    bbcca74 Fix #23: Add missing changes for master branch
    197ad52 No need to install autoconf/automake in Travis
    eb95ee1 Add license to cron scripts
    4f62420 Fix/add make dist and try it in Travis
    ac4c634 Reconstruct repository to move out presenter. Update licenses.
            Use pcap_layers as a git submodule. Use automake/autoconf.
    89c7f4c Import patched pcap_layers.c code with buffer overflow bugfix
    2787db2 Only wait a certain number of times for the files to appear
    b5d911d Add the first test, simple run and compare gold files
    32fd807 Fix #13: Document ability to read packets from pcap files
    eee217e Rework some of the Makefiles based on some of the patches found in
            the Debian packages
    7a2a67e Fix #12: Add hash for dataset names and check for duplicates
    ed1eba9 Add base64 for certain non-printable characters in JSON output and
            remove extra new-line
    580d543 Add output_format and JSON structure description to documentation
    d2d1ed2 Fix #3: Reworked JSON output format implementation
    2e2f90f Give option for additional output JSON/Extended JSON
    a23b6af Ignore generated files
    f4214f3 Add Travis CI. Remove old TODAY/tar commands. Add ifndef/def to
            all .h files. Fix issue with arpa/nameser_compat.h on OpenBSD by
            checking for the header file and only including it if it exists.
            Rename configure.scan to configure.ac and change contact info and
            version. Update configure using autoconf 2.69 on Ubuntu 14.04 LTS.
            Set CC/CPP/CXX/CXXFLAGS in Makefile found by configure.

2016-01-11 Duane Wessels

    added dfprintf() macro to improve code readability and avoid
    multi-line if (debug_lvl) fprintf(...) statements.

2016-01-11 Duane Wessels

    Commenting out ancount and nscount to silence compiler warnings
    about unused variables.

2016-01-11 Duane Wessels

    Patch from John Heidemann relating to TCP reassembly
    - fix for multiple DNS messages per TCP connection that span
      multiple segments
    - some editorial code changes (== to >=)
    - additional debugging statements

2016-01-11 Duane Wessels

    Minor fix: debugging statement wasn't protected with debug_flag
    check. (thanks John H).

2015-12-23 Duane Wessels

    add prototype for dns_message_handle()

2015-12-23 Duane Wessels

    Experimental feature to drop "received responses" and "sent queries"
    since DSC is generally used to monitor authoritative servers that,
    under normal operation, never receive responses nor send queries.
    Currently protected with #ifdefs.

2015-12-15 Duane Wessels

    Oops, forgot to set dns_message->server_ip_addr for new server_addr
    indexer

2015-11-23 Duane Wessels

    removing one level of the "dsc/dsc" top directory

2015-11-11 aqadeer

    In pcap.c pcap_setnonblock doesn't accept captured files anymore and
    must need a device from where it could do live capture.  For offline
    files to work, a simple check is added to by-pass this porblem.

2015-11-05 Duane Wessels

    Added a 'server' indexer.  This records the server (query destination)
    IP address.

2015-11-05 Duane Wessels

    Renamed "client_ipv4" to "client_ip" because it supports v4 and v6

2015-11-04 Duane Wessels

    Update copy of pcap_layers.c from https://github.com/wessels/pcap_layers

2015-11-04 Duane Wessels

    avoid "void *" pointer arithmetic (Yoshitaka Aharen)

2015-10-09 Duane Wessels

    bugfix: handle receiving TCP dns length prefix out of order.

    A user reported that when a TCP segment containing only the DNS message
    length is received *after* the message it references (i.e., out-of-order),
    then DSC goes into a 100% CPU loop.  Confirmed that the code doesn't work
    correctly when dnslen comes out-of-order, but I wasn't able to easily
    reproduce the 100% CPU bug.

    The fix is to add a "dnslen_bytes_seen_mask" variable that tracks which
    of the two dnslen bytes we've seen.  Once both bytes have been seen,
    then we can proceed to reassembling the message buffer.

2015-10-09 Duane Wessels

    cosmetic: rename ipv4 to ip4 to be consistent with ip6

2015-10-09 Duane Wessels

    Regarding recent change to add pcap_layers library, forgot to add it
    to Makefile.in (vs Makefile) before committing.

2015-10-09 Duane Wessels

    Integrated https://github.com/wessels/pcap_layers with DSC collector.
    The DSC code now includes a copy of the pcap_layers library, which does
    a better job of extracting the layers of a pcap packet.  In particular,
    it does IP fragmentation reassembly, which is important for DSC and
    RSSAC-002. At this time the DSC code still does TCP reassembly, however.

    Removed USE_IPV6 ifdefs.  IPV6 support is now always compiled.

2015-10-08 Duane Wessels

    Removing "DMC *dns_message_callback" because there is only one that
    would ever be used. The callback layer of indirection makes the code
    a little confusing, and also this change is in preparation for bringing
    in a third-party pcap layer handling library which will work slightly
    differently.

2015-10-08 Duane Wessels

    The "ip_message" code has not been in use since a commit back in 2012.
    But the code lingered, until now.

2015-02-25 Duane Wessels

    User reported a concern with the way dsc-xml-extractor.pl called
    the operating system 'mv' command.  It has been replaced with Perl's
    File::Copy::mv().

2013-03-19 Duane Wessels

    Fixed a bug in TCP reassembly when the DNS length field was
    split between two segments.

2012-08-27 Sebastian Castro

    Added 'tc_bit' indexer and dataset to track the frequency
    of responses having the TC bit set.  Useful with DNSSEC
    signed zones.

2012-02-29 Duane Wessels

    Added 'qr_aa_bits' dataset and graph.  It shows the distribution
    of QR/AA values in received messages and may be helpful in
    detecting reflector attacks targeting your name server.

2010-11-29 Duane Wessels

    The dnssec_qtype and dns_ip_version plots were not working due
    the presence of 'dataset' attributes in their definitions.  Also
    fixed the chaos plot.

2010-11-04 Duane Wessels

    Added NSEC3 record type to the extractor.

2010-09-14 Duane Wessels

    A fix to put pcap interfaces into non-blocking mode, which is
    important when reading from multiple interfaces.

2010-08-13 Alexander Mayrhofer

    Added "country_index" to the collector which is an index based
    on country as returned by the GeoIP library.  configure attempts
    to learn if you have GeoIP installed.

2010-08-13 Peter Koch

    Some minor fixes to the collector that allow it to read from
    pcap files on disk, rather than live packets from an interface.

2010-06-02 Henrik Kramsh?j

    Improvements for OpenBSD compatibility

2010-05-01 Duane Wessels

    Added NSEC3 to DNSSEC query types

2010-02-22 Duane Wessels

     Collector bug fixed: USE_IPV6=1 was not passed to .c
     files, which resulted in different-sized data structurs
     and crashes on CentOS.

2009-12-10 Duane Wessels

    Added "priming_queries" and "priming_responses" datasets in
    preparation for root zone signing.

2009-11-11 Duane Wessels

    On the presenter some of the "accum" graphs stopped working
    due to recent rewrites.  These should now be working again.

2009-10-12 Duane Wessels

    The presenter debugging is now configurable via dsc-grapher.cfg.

2009-08-14 Duane Wessels

    In the presenter, there have been some significant changes to
    the perl modules so that they can be used in command line
    (ie non-CGI) utilities, both for creating graphs and for
    reading data.  Most of these changes are purely internal.
    However, you may need to update your dsc-grapher.pl CGI
    program to be like presenter/grapher/dsc-grapher.pl.sample
    in the source distribution.

2009-04-15 Duane Wessels

    In the presenter's refile-and-grok.sh script, note that in
    some environtments it is useful to skip NODEs that don't
    have an incoming directory because they might have been
    "grokked" elsewhere and then rsync'd to you.

2009-02-27 Duane Wessels

    In the collector it is now necessary to include <stdlib.h>
    and <stdio.h>.   Not sure why it wasn't necessary before.
    Perhaps due to 2009-01-26 Hapy library upgrade.

2009-02-19 Duane Wessels

    In the collector, fixed some 64-bit free disk space calculations
    with casting.

2009-01-26 Duane Wessels

    Upgraded to more recent version of Hapy parsing library.

2008-12-31 Duane Wessels

    In the collector, changed the value of pcap_open_live() because
    we always try to read from all interfaces after select() returns.
    If we have multiple interfaces and one of them is quiet, and
    the timeout is too large, we'll drop packets on the other
    interface while waiting for the first to time out.

    Also added a pseudo-dataset that reports on pcap packets captured
    and dropped.


2008-12-02 Duane Wessels

    Grapher now preserves the order of the server list given in
    dsc-grapher.cfg.

2008-11-22 Duane Wessels

    Added some collector indexers and datasets for 2nd and 3rd level
    domains.

2008-10-02 Duane Wessels

    Made the 'no extractor for $dataset' error message non-fatal.
    Now, the unknown dataset will just be skipped and the remaining
    ones will be processed.

2008-09-30 Duane Wessels

    In the presenter, account for the possibility that the collector's
    clock is not in sync.

2008-09-18 Elmar Knipp

    Found bug in upload script where a $PROG.out prevents removal
    of a $YYYYMMDD directory.

2008-09-16 Duane Wessels

    New presenter feature: Node Merging.  Now the DSC grapher can
    automatically "merge" multiple nodes so that they appear as
    one.  For example instead of this dsc-grapher.cfg line:

        server S N1a N1b

    you can merge nodes "a" and "b" with this line:

	server S N1=N1a,N1b

    This feature is also useful for just renaming a node if you
    want it to be displayed as a different name than the directory
    where its files are.

2008-09-15 Duane Wessels

    On the collector, changed the sample config and upload scripts
    to use /usr/local/dsc/run as the default run_dir (was formerly
    /usr/local/dsc/var/run/).

2008-08-21 Duane Wessels

    Added an EDNS bufsiz indexer so we can collect buffer sizes
    advertised by clients.

2008-08-12 Duane Wessels

    Attempt to improve portability (on Solaris) by checking for
    libresolv, libnsl, and libsocket.  Also check for statvfs() vs
    statvs() in configure.

2008-07-29 Duane Wessels

    Updates to collector/cron/upload-rsync.sh so that it behaves
    better with new date-based directories on both collector and
    presenter.

2008-06-27 Duane Wessels

    Added a source port range dataset (for about-to-be-announced
    vulnerability that can be mititgated by improving source port
    randomness).

2008-04-22 Duane Wessels

    Added ./configure script for presenter/dsc code.

2008-04-22 Duane Wessels

    Added support for capturing NCAP data streams.

2008-01-09 Duane Wessels

    Changed the location of the presenter's ".dat" files.  Previously
    there were stored in a directory such as 20080109/qtype/qtype.dat
    and now they will be moved one directory up and stored as
    20080109/qtype.dat.

    The source distribution includes a script found at
    presenter/grapher/update-dat-file-locations.sh that will traverse
    the /usr/local/dsc/data directory and move all the .dat files
    one level up.

    When upgrading to this version of DSC you should first stop the
    refile-and-grok.sh cron job, install the DSC software, and then
    run the update-dat-file-locations.sh script (after reading it).

2008-01-09 Duane Wessels

    Data passed from collector to presenter is now contained in a
    single XML file, rather than one XML file per dataset.  This
    should significantly reduce filesystem pressure (i.e., 20x fewer
    files to be stored and sent) on both sides.

2008-01-07 Duane Wessels

    The collector now stores to-be-uploaded XML files in
    date-named subdirectories.  This reduces filesystem pressure
    in the event that the collector cannot communicate with the
    presenter for an extended period of time.

    To accomodate this change, the upload-prep.sh script has been
    replaced with a Perl version, named upload-prep.pl.  After
    installing DSC you should change your cron job to use the new
    script.

2008-01-04 Ken Keys

    Previously, a child was forked for each collection interval.
    Memory allocated for collected data was freed by simply exiting
    the child.  But this left no way to preserve any data across
    intervals.

    Now, interval collectors do not fork.  To avoid the tedious,
    slow, and error-prone nature of normal malloc-and-free memory
    management, we instead store non-persistent data in memory
    allocated from an "arena", which can be freed quickly and safely.
    We don't need to free individual allocations, but we do need
    to reset some pointers to allocated memory.

    TCP resets are now properly handled and TCP state is freed
    if a connection has been idle for too long.

2008-01-04 Duane Wessels

    The collector configuration (dsc.conf) has a new 'minfree_bytes'
    directive.  If the amount of free disk space on dsc's partition
    falls below this limit, dsc will not write any XML files -- that
    data will be lost.

    The default value is 5 MB.

2007-12-14 Alexander Gall

    Added a 'dns_ip_version' indexer so that we can track DNS messages
    sent over v4 and v6.

2007-12-04 Duane Wessels

    Fixed a collector bug when listening on multiple interfaces of
    different datalink types (e.g., eth0 and lo).

2007-11-26 Duane Wessels

    Added a 'hide_nodes' feature to dsc-grapher.cfg.  If 'hide_nodes'
    is given, then the navigation menu will not display the nodes
    underneath a server.  Note, however, that knowledgeable users
    could still view individual node data by manually if they know
    the names by specifying it in the URL query terms.

2007-10-09 Duane Wessels

    Changed the presenter to not use "data URIs" by default.  Now
    Internet Explorer users should be happier.

2007/06/14 Duane Wessels

    Added -f command line option to keep 'dsc' collector as a foreground
    process.

2007-06-11 Duane Wessels

    dsc-xml-extractor.pl now looks for incoming XML files in
    "incoming/YYYY-MM-DD" directories.

2007-06-07 Duane Wessels

    New TCP code did not compile cleanly on Linux.  Added some
    #ifdefs and macros for Linux compatibility on TCP headers.

2007-05-15 Ken Keys, Duane Wessels

    TCP support
    New 'transport' indexer for DNS messages.  Indicates whether message
    arrived over UDP or TCP.

2007-04-04 Duane Wessels

    IPv6 support
    New 'ip_version' indexer.  Indicates whether DNS message arrived
    via IPv4 or IPv6.