Skip to content

Releases: cisagov/Malcolm

Malcolm v25.03.1

28 Mar 14:51
b18b8d3
Compare
Choose a tag to compare

This has been a busy month for Malcolm! We pushed hard to get v25.03.0 out earlier this month, as it contained pretty much just the Keycloak integration one of our partners (and major funding sources) was waiting for. Rather than wait until April for the other stuff that would have gone into the regular end-of-the-month release, I decided to pull those items into this smaller release just a week and a half after the last one.

Malcolm v25.03.1 contains a few enhancements, bug fixes, and several component version updates, including one that addresses a CVE that may affect Hedgehog Linux Kiosk mode and Malcolm's API container.

v25.03.0...v25.03.1

NOTE: If you have not already upgraded to v25.03.0, read the notes for v25.02.0 and v25.03.0 and follow the Read Before Upgrading instructions on those releases.

  • ✨ Features and enhancements
    • Incorporate new S7comm device identification log, s7comm_known_devices.log (#622)
    • Display current PCAP, Zeek, and Suricata capture results in Hedgehog Linux Kiosk mode (#566)
    • Keycloak authentication: configurable group or role membership restrictions for login (#633) (see Requiring user groups and realm roles)
    • Mark newly-discovered and uninventoried devices in logs during NetBox enrichment (#573)
    • Added "Apply recommended system tweaks automatically without asking for confirmation?" question to install.py to allow the user to accept changes to sysctl.conf, grub kernel parameters, etc., without having to answer "yes" to each one.
  • ✅ Component version updates
  • 🐛 Bug fixes
    • Fix install.py error when answering yes to "Pull Malcolm images?" with podman (#604)
    • Order of user-provided tags from PCAP upload interface not preserved (#624)
  • 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • 🧹 Code and project maintenance
    • Ensure Malcolm's NetBox configuration Python scripts are baked into the image in addition to bind-mounting them in docker-compose.yml at runtime.

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.03.0

18 Mar 22:04
68cdc94
Compare
Choose a tag to compare

Malcolm v25.03.0 adds authentication via Keycloak and includes a few component version updates.

v25.02.0...v25.03.0

Read Before Upgrading

  • As described below, a number of changes were made to environment variables in this release. The Malcolm control script should automatically migrate environment variables between Malcolm versions (e.g., moving environment variables from one .env file to another, removing deprecated/unused environment variables from .env files, etc.) as these actions are specified in config/env-var-actions.yml. However, these actions should be taking when migrating from a previous version of Malcolm to v25.03.0:
    • Before upgrading, while Malcolm is up, execute ./scripts/netbox-backup to backup the NetBox database and save the resulting .gz file(s) in case something goes wrong with the migration of the location of the PostgreSQL database or the environment variables associated with it. Should this happen, ./scripts/netbox-restore could be executed afterwards to restore the contents of the NetBox database.
    • If you have not already upgraded to v25.02.0, read the notes for that release and manually update the redis-related environment variables as described there.
    • Once updating to v25.03.0, but before starting Malcolm, run ./scripts/status to automatically migrate the other environment variables as described above.

Release Notes

  • ✨ Features and enhancements
    • Support authentication via Keycloak (#459)
      • In addition to local account management and LDAP authentication, Malcolm can now utilize Keycloak, an identity and access management (IAM) tool, to provide a more robust authentication and authorization experience, including single sign-on (SSO) functionality.
      • Malcolm can connect to an existing Keycloak server or it can use its own embedded Keycloak instance.
      • While this feature has been developed and tested with Keycloak in mind, the lua-resty-openidc library used to implement the OpenID connection functionality may work with other OpenID providers as well. If you find this does work, let us know on the discussions board; if not, please log an issue with details.
      • This feature will pave the way for fine-grained access controls to be implemented in a future Malcolm version.
      • To support this feature, the postgres container has been decoupled from NetBox and now runs independent of that service. This is similar to what was done with the redis container in v25.02.0.
      • To support this feature, the vanilla NGINX web server used internally has been replaced with OpenResty, a version of NGINX extended with Lua.
      • New functionality was added to the authentication setup tool.
      • Refer to the new documentation on this feature for details, including a known limitation when using this authentication method with Hedgehog Linux.
    • Change to ./wipe command behavior
      • Prior to this release, running ./wipe also cleared the contents of the directory of the PostgreSQL database containing the NetBox inventory. PostgreSQL is now used to store both the NetBox inventory and the embedded Keycloak instance data. For this reason, and because it was probably not users' intention to blow away their network inventory with ./wipe, that script no longer deletes this data.
  • ✅ Component version updates
    • OpenSearch and OpenSearch Dashboards to v2.19.1
    • Jinja2 to v3.1.6 to fix "Jinja2 vulnerable to sandbox breakout through attr filter selecting format method" vulnerability (CVE-2025-27516)
    • Fluent Bit to v3.2.8
    • Capa to v9.1.0
  • 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
    • the following are all to support authentication via Keycloak (#459)
      • renamed NGINX_BASIC_AUTH with NGINX_AUTH_MODE in auth-commmon.env; the new code handling this variable should be backwards-compatible with the previously-accepted values
      • added keycloak.env
      • renamed nginx-postgres to postgres.env and completely overhauled the variables in that file
      • added several new environment variables to nginx.env (see the comments in that file for details)
      • removed NETBOX_POSTGRES_DISABLED from in netbox-common.env

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.02.0

27 Feb 21:57
bc6edb4
Compare
Choose a tag to compare

Malcolm v25.02.0 contains some major performance improvements, a few smaller new features and enhancements, several component version updates, bug fixes, and documentation updates.

v25.01.0...v25.02.0

NOTE: As a result of some of the changes to environment variables made for decoupling Redis from NetBox (#580), environment variables from previous version may cause NetBox to fail to connect to Redis which prevents successful startup. To fix this, you should perform the following steps once prior to starting Malcolm:

  1. Stop Malcolm (./scripts/stop)
  2. Change the values for REDIS_CACHE_HOST and REDIS_HOST, removing the netbox- prefix from the values, so that they look like REDIS_HOST=redis and REDIS_CACHE_HOST=redis-cache, respectively.
    • These values were found in netbox.env in previous versions, but are found in redis.env in this release.
    • Alternatively, you may remove the lines for REDIS_HOST and REDIS_CACHE_HOST completely and they will be restored with correct values the next time the control script is run.
  3. Run ./scripts/status which will check the .env files and restore the missing values if you removed them.
  4. Start Malcolm (./scripts/start)
  • ✨ Features and enhancements
    • performance improvements for NetBox enrichment (#547)
      • NetBox enrichment and autopopulation is now approximately 4x faster than it was before (depending on resources)
    • performance improvements for Suricata's processing of uploaded PCAP files (#457)
      • Suricata's processing of large sets of uploaded PCAP files is now approximately 18x faster faster than it was before (depending on resources)
    • add validate_local_site_policy.sh script for validating Zeek local site policy (#598)
    • include corelight/zeek-long-connections plugin to log long connections (#585)
      • new zeek.conn.long field is available to indicate long connections
      • Connections dashboard updated to include this new field
      • see notes below on environment variable additions for configuring this plugin
    • standardize container health checks into scripts for all containers (#491)
      • added container health checks for containers that did not previously have them (live capture containers)
    • significant work-in-progress towards support for Sigma rules via OpenSearch Security Analytics (still incomplete due to some blocking issues upstream, see #475 for details)
      • changed normalization of Windows event log records (evtx) to more closely match Winlogbeat fields which are closer to what the Sigma rules for Windows events use, and updated corresponding Windows Event Logs dashboard
    • dnp3_control.log now includes clear_bit field to indicate if control code clear bit is set or unset
    • improved shared-object-creation.sh's cURL commands so that import failures for OpenSearch/Elasticsearch shared objects are printed to the debug logs rather than being redirected to /dev/null
  • ✅ Component version updates
  • 🐛 Bug fixes
    • ANSI color codes from croc displayed in ssl-client-transmit (#559)
    • clear screen after auth_setup when using Dialog mode (#574)
    • warn and prompt user before changing NetBox database passwords out from underneath existing database (#565)
    • UFW software firewall for Malcolm ISO should automatically open ports for syslog (#560)
      • removed default port allowances (e.g., 5044/tcp, 9200/tcp, etc.) so that they could be set dynamically as part of configuration
  • 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • 🧹 Code and project maintenance
    • use arm-hosted runners for GitHub build actions for arm64 images (#557)
    • decouple redis from netbox (#580)
    • document standards for supply chain and code provenance checking (#555)
    • document incorporating new Suricata rules (and removing old ones) without restarting the Suricata containers (#589)
    • updates to documentation for Docker-based installation exampl...
Read more

Malcolm v25.01.0

17 Jan 21:56
d186745
Compare
Choose a tag to compare

Malcolm v25.01.0 contains quite a few UI/UX improvements; new parsers; a bevy of component version updates including to Arkime, Zeek, NetBox; and several bug fixes.

v24.12.0...v25.01.0

  • ✨ Features and enhancements
    • integrate Omron FINS parser and added corresponding dashboard (#554)
    • integrate PostgreSQL parser (added in Zeek v7.1.0) and added corresponding dashboard (#553)
    • normalize Winlogbeat with Fluent Bit's winlog/winevtlog event and evtx event schemas (#356)
      • Winlogbeat seems to parse more fields from Windows events than Fluent Bit's winevtlog or winlog do, so users forwarding Windows event logs to Malcolm using Fluent Bit may want to evaluate Winlogbeat as an alternative.
    • support syslog ingestion over UDP and/or TCP (#354)
    • clicking field values in Dashboards tables will now pivot to Arkime or NetBox (#551)
    • add navigation pane to all non-network dashboards (#543)
  • ✅ Component version updates
  • 🐛 Bug fixes
    • Extracted File Downloads interface not working with some filenames (#524)
    • user-defined custom field formats for index patterns are overwritten (#542)
    • port numbers should not be shown with commas in Dashboards (#540)
    • pivoting between Arkime and Dashboards doesn't work when Malcolm is behind a reverse proxy (e.g., traefik) (#552)
    • opensearch.keystore not created when running in Hedgehog run profile (#533)
    • ensure all conn.log entries are tagged ics for OT protocols (#541)
  • 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
    • The following variables in ./config/filebeat.env configure Malcolm's ability to accept syslog messages:
      • FILEBEAT_SYSLOG_TCP_LISTEN and FILEBEAT_SYSLOG_UDP_LISTEN - if set to true, Malcolm will accept syslog messages over TCP and/or UDP, respectively
      • FILEBEAT_SYSLOG_TCP_PORT and FILEBEAT_SYSLOG_UDP_PORT - the port on which Malcolm will accept syslog messages over TCP and/or UDP, respectively
      • FILEBEAT_SYSLOG_TCP_FORMAT and FILEBEAT_SYSLOG_UDP_FORMAT - one of auto, rfc3164, or rfc5424, to specify the allowed format for syslog messages over TCP and/or UDP, respectively (default auto)
      • FILEBEAT_SYSLOG_TCP_MAX_MESSAGE_SIZE and FILEBEAT_SYSLOG_UDP_MAX_MESSAGE_SIZE - defines the maximum message size of the message received over TCP and/or UDP, respectively (default: 10KiB for UDP, 20MiB for TCP)
      • FILEBEAT_SYSLOG_TCP_MAX_CONNECTIONS - specifies the maximum current number of TCP connections for syslog messages
      • FILEBEAT_SYSLOG_TCP_SSL - if set to true, syslog messages over TCP will require the use of TLS. When ./scripts/auth_setup is run, self-signed certificates are generated which may be used by remote log forwarders. Located in Malcolm's ./filebeat/certs/ directory, the certificate authority and client certificate and key files should be copied to the host on which the forwarder is running and used when defining its settings for connecting to Malcolm.
    • The following variables in ./config/zeek.env for Malcolm and control_vars.conf for Hedgehog Linux pertain to the new Omron FINS protocol parser:
      • ZEEK_DISABLE_ICS_OMRON_FINS - if set to true, the Omron FINS parser will be disabled
      • ZEEK_OMRON_FINS_DETAILED - if set to true, a verbose Omron FINS details log (omron_fins_detail.log) will be created
  • 🧹 Code and project maintenance
    • Changed ⓒ year to 2025

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v24.12.0

19 Dec 15:28
e1afaec
Compare
Choose a tag to compare

Malcolm v24.12.0 contains several improvements to the Malcolm configuration script, the Malcolm user interface, and the Malcolm API, as well as component version updates and bug fixes. This release also corresponds with the release of malcolm-test (#486), a Malcolm system testing framework.

v24.11.0...v24.12.0

  • Features and enhancements
    • Creation of a Malcolm systems testing framework (#486)
    • Added a number of Zeek packages to detect various CVEs
    • Improvements to the Indices, Ready, and Document Ingest Statistics APIs
    • Use new arkime tag-hiding feature to hide netbox tag from UI (#495)
    • Provide configuration script options for pulling from threat intel feeds (#532)
    • Prompt during configuration whether to enable capture statistics (#504)
    • Add additional EVTX fields to index template (#525) and minor improvements to normalization
    • Add simple readiness indicator to upload page (#528)
    • Add option to upload page to disable NetBox enrichment for the currently-uploaded batch of PCAPs
    • Expose more of the Logstash API passthrough to the Malcolm API
  • Component version updates
  • Bug fixes
    • Zeek DNS records don't open correctly in Arkime sessions (#509)
    • Mandiant threat intel source doesn't get split correctly when using JSON zeek log format (#494)
    • Set indices.query.bool.max_clause_count to 8192 to reflect maximum number of fields
    • Increase Java stack size (-Xss) for Logstash from 1536k to 2048k
    • Minor fixes for parsing Zeek intel.log (some fields not named correctly with Zeek JSON-formatted logs)
    • Fixes to some Zeek dns.log parsing conflicts between ECS's DNS fields and what the Arkime schema is expecting
    • Fixed setting the Signature event severity tags
  • Code and project maintenance
    • Replaced hard-coded Malcolm version number in documentation markdown files with variable-based replacer populated during generation
    • Documentation and screenshot updates

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.11.0

18 Nov 17:55
891cb15
Compare
Choose a tag to compare

Malcolm v24.11.0 contains a new threat intelligence feed integration, a few new API calls, other minor improvements, bug fixes, and component version updates.

v24.10.1...v24.11.0

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.10.1

24 Oct 16:05
cb0d34e
Compare
Choose a tag to compare

Malcolm v24.10.1 contains some minor improvements, a few component version updates, a fix for a regression bug, and a fair amount of code cleanup.

v24.10.0...v24.10.1

  • Features and enhancements
  • Component version updates
  • Bug fixes
    • Fixed OpenSearch anomaly detection default detectors not being created (regression, idaholab#596)
  • Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
    • Malcolm
    • Hedgehog Linux
      • ZEEK_JA4SSH_PACKET_COUNT has been added to control_vars.conf for the same purpose as described above
  • Code and project maintenance
    • Examine distro hardening, fix and update documentation as needed for Malcolm and Hedgehog Linux ISO-installed environments (idaholab#328)
    • Refactoring and code cleanup in the Logstash Zeek pipeline (idaholab#592)
    • Logstash container initialization code now automatically ensures that the Zeek TSV log parsing filters (dissect and split filters) in these files are looking for TAB characters (i.e., automatically replace spaces with tabs in these filter files in case the author forgot to do so) (idaholab#592)
    • Did some code cleanup in the ./shared/bin directory, mostly moving things that were specific to either the Malcolm or Hedgehog Installer ISO environments out of shared and into their respective locations for the ISO installer build.
    • When doing the aquasecurity/trivy-action action, use TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db to try to fall back to an alternative official location for the vulnerability database if the first one fails. Also, pin this action to the v0.28.0 release rather than setting it to master.
    • As it's used pretty ubiquitously in shared scripts by many of the Malcolm containers, the jq utility is now installed across the board during the container image build.
    • Added a script to gather GitHub API metrics for Malcolm downloads (idaholab#594)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.10.0

09 Oct 15:34
556ac85
Compare
Choose a tag to compare

Malcolm v24.10.0 contains fixes for a few regression bugs, minor improvements, and a few component updates.

v24.09.0...v24.10.0

  • Features and enhancements
    • Enable Zeek's parsing of HTTP server and client header names as zeek.http.client_header_names and zeek.http.server_header_names
    • Bumped maximum field limit in OpenSearch templates from 5000 to 6000
    • Some documentation improvements
    • Build improvement: fall back to alternative Zeek .deb download URL (idaholab#585)
    • Build improvement: limit threads for spicy build processes during Zeek package installation (idaholab#571)
  • Component version updates
  • Bug fixes
    • Fix broken dashboards regression from v24.09.0 (idaholab#588)
    • Fix Zeek-extracted files not getting saved to correct location for live Zeek capture (idaholab#590)
    • Fix for building Hedgehog Linux for Raspberry Pi 4 on an M2 MacBook

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.09.0

19 Sep 19:57
150674d
Compare
Choose a tag to compare

Malcolm v24.09.0 contains new features and enhancements, component version updates, and bug fixes.

v24.08.0...v24.09.0

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Mal.Con24 Presentations

19 Sep 18:58
9384e22
Compare
Choose a tag to compare

Here are the slide decks for the presentations that were given at Mal.Con24. Enjoy!