We are happy to announce VAST v1.0!
This release brings a new approach to software versioning for Tenzir. We laid out the semantics in detail in a new VERSIONING document.
Query events based on their import time
#import_time extractor allows for exporting
events based on the time they arrived at VAST. Most of the time, this timestamp
is not far away from the timestamp of when the event occurred, but in certain
cases the two may deviate substantially, e.g., when ingesting historical events
from several years ago.
For example, to export all Suricata alerts that arrived at VAST on New Years Eve as JSON, run this command:
vast export json '#type == "suricata.alert" && #import_time >= 2021-12-31 && #import_time < 2022-01-01'
This differs from the
:timestamp type extractor that
queries all events that contain a type
timestamp, which is an alias for the
time type. By convention, the
timestamp type represents the event time
embedded in the data itself. However, the import time is not part of the event
data itself, but rather part of metadata of every batch of events that VAST
null fields in the JSON export
VAST renders all fields defined in the schema when exporting events as JSON. A
common option for many tools that handle JSON is to skip rendering
fields, and the new
--omit-nulls option to the JSON export does exactly that.
To use it on a case-by-case basis, add this flag to any JSON export.
vast export json --omit-nulls '<query>'
# This also works when attaching to a matcher.
vast matcher attach json --omit-nulls <matcher>
To always enable it, add this to your
vast.yaml configuration file:
Selection and Projection Transform Steps
Reshaping data during import and export is a common use case that VAST now supports. The two new built-in transform steps allow for filtering columns and rows. Filtering columns (projection) takes a list of column names as input, and filtering rows (selection) works with an arbitrary query expression.
Here’s a usage example that sanitizes data leaving VAST during a query. If any
string field in an event contains the value
will not include the event in the result set. The example below applies this
sanitization only to the events
suricata.http, as defined
in the section
# Specify and name our transforms, each of which are a list of configured
# transform steps. Transform steps are plugins, enabling users to write more
# complex transformations in native code using C++ and Apache Arrow.
# Prevent events with certain strings to be exported, e.g., "tenzir" or
expression: ':string !in ["tenzir", "secret-username"]'
# Specify whether to trigger each transform at server- or client-side, on
# import or export, and restrict them to a list of event types.
# Apply the remove-events-with-secrets transformation server-side on
# export to the suricata.dns and suricata.http event types.
- transform: remove-events-with-secrets
Threat Bus 2022.01.27
Thanks to a contribution from Sascha Steinbiss (@satta), Threat Bus only reports failure when transforming a sighting context if the return code of the transforming program indicates failure.
A small peek behind the curtain: We’re building the next generation of Threat Bus as part of VAST. We will continue to develop and maintain Threat Bus and its apps for the time being.
Threat Bus 2022.01.27 is available 👉 here.