This Week's/Trending Posts

Hand-Picked/Curated Posts

Most Popular/Amcache

Hand-Picked/Weekly News

The Most/Recent Articles

Daily Blog #783: Automating rpm checks

 


Hello Reader,

I'm recreating my 24 year old perl scipt in bash to allow someone to validate all of the installed rpms on a system against both the local rpm DB and the repository it came from.  This should allow a certain since of comfort on if any core system packages have been manipulated.

 

#!/bin/bash

# Files to store results
VERIFIED="verified"
FAILURES="failures"
DEBUG="debug"

# Clean previous results
> "$VERIFIED"
> "$FAILURES"
> "$DEBUG"

# Iterate over installed RPM packages
for package in $(rpm -qa); do
  echo "Processing package: $package"

  # Find repository URL
  repo_url=$(dnf repoquery -q --location "$package" 2>/dev/null | head -n 1)

  if [[ -z "$repo_url" ]]; then
    echo "Repository URL not found for package: $package" | tee -a "$FAILURES"
    echo "$repo_url $package" | tee -a "$DEBUG"
    continue
  fi

  # Get local file hashes from RPM database
  rpm -ql --dump "$package" | while read -r line; do
    file_path=$(echo "$line" | awk '{print $1}')
    rpm_hash=$(echo "$line" | awk '{print $4}')

    # Skip directories and non-executable files
    if [[ ! -x "$file_path" ]]; then
       continue
    fi
    
    if [[ ! -f "$file_path" ]]; then
       continue
    fi

    if [[ -h "$file_path" ]]; then
       continue
    fi
    # Calculate local disk hash
    disk_hash=$(sha256sum "$file_path" 2>/dev/null | awk '{print $1}')

    if [[ "$disk_hash" != "$rpm_hash" ]]; then
      echo "Hash mismatch (Local RPM DB) - Package: $package, File: $file_path" | tee -a "$FAILURES"
      echo "$dish_hash $rpm_hash $package $file_path" | tee -a "$DEBUG"
      continue
    fi

    # Get repository RPM hash
    repo_hash=$(rpm -qp --dump "$repo_url" 2>/dev/null | grep " $file_path " | awk '{print $4}')

    if [[ -z "$repo_hash" ]]; then
      echo "File not found in repository RPM - Package: $package, File: $file_path" | tee -a "$FAILURES"
      echo "$repo_hash $repo_url $file_path" | tee -a "$DEBUG"
      continue
    fi

    if [[ "$disk_hash" == "$repo_hash" ]]; then
      echo "Verified - Package: $package, File: $file_path" >> "$VERIFIED"
    else
      echo "Hash mismatch (Repository) - Package: $package, File: $file_path" | tee -a "$FAILURES"
      echo "$disk_hash $repo_hash $package $file_path" | tee -a "$DEBUG"
    fi
  done
done

echo "Verification complete. Results are stored in '$VERIFIED' and '$FAILURES'."


 

Daily Blog #782: Validating linux packages other than rpms

 

Hello Reader,

      We've talked about validating rpms in several posts now but there are other package managers besides rpm. Let's talk about how we can do the same validation with other package managers.

 

1. Debian/Ubuntu (dpkg & debsums)

Install debsums if you haven't already:

sudo apt install debsums

Verify file hashes for a specific package:

sudo debsums -s <package-name>

Verify a specific file:

sudo debsums -s <package-name> | grep /path/to/file

Verify all installed packages:

sudo debsums -cs

2. Arch Linux (pacman)

Check integrity of a specific package:

pacman -Qkk <package-name>

Verify a single file:

pacman -Qkk <package-name> | grep /path/to/file

Verify all installed packages:

pacman -Qkk

3. openSUSE (rpm & zypper)

openSUSE uses RPM, so you can use standard RPM verification commands:

Check integrity of a file against the RPM database:

rpm -Vf /path/to/file

Verify all installed packages:

rpm -Va

4. Alpine Linux (apk)

Newer Alpine Linux versions (3.15+) include the apk audit command:

Verify integrity of a package:

apk audit <package-name>

Verify all installed packages:

apk audit

rpm

Daily Blog #781: Validating local linux hashes to their distros

 


Hello Reader,

In my previous blog post, I explained how to use the rpm tool to validate a file on disk against the local RPM metadata. But what if you suspect that an entire package—not just a single file—has been tampered with?

This is where we can leverage a powerful feature I mentioned earlier: extracting metadata directly from the Linux distribution’s repository. Since this remote repository should be unaffected by any local security incidents, it allows you to verify that packages like the core system utilities in this example remain unaltered by a potential threat actor.


Step 1: Identify the Repository URL

To fetch the official package version, you first need to determine the correct repository URL. You can do this using the dnf repoquery command:

dnf repoquery --location coreutils (you can specify any package name)

This will return a URL similar to:

https://ftp.redhat.com/pub/redhat/linux/enterprise/9/en/os/x86_64/Packages/coreutils-8.32-31.el9.x86_64.rpm

Step 2: Extract the Official Package Hash

Now that you have the package URL, you can use rpm to retrieve its metadata, including file hashes, without downloading the full package:

rpm -q --dump -p https://ftp.redhat.com/pub/redhat/linux/enterprise/9/en/os/x86_64/Packages/coreutils-8.32-31.el9.x86_64.rpm

Step 3: Compare Local vs. Repository Hashes

To ensure your package is untouched, compare:

  1. The hash of the local file (e.g., /bin/ls)
  2. The hash stored in the local RPM database
  3. The hash from the official repository package

If all three hashes match, you can be highly confident that your package has not been altered.

Of course, this assumes there isn’t a worst-case scenario where the original distribution’s repository has been compromised—but let’s hope it never comes to that!

By following these steps, you can verify system integrity efficiently using native Linux tools

Daily Blog #780: Self validating linux executables

 


Hello Reader,

When I first started doing Digital Forensics and Incident Response (DFIR) back in 2000, one challenge I faced was verifying the integrity of Linux executables. Often, systems didn't record file hashes during installation, and sometimes we didn't even have a reliable set of installation media to compare against.

To solve this problem, I initially wrote a lengthy Perl script. The script went through each installed package on the local system, comparing the file hashes on disk against the original package hashes, and then validating these against the official distribution hashes.

Today, this task is much simpler. On Linux systems using RPM (like RedHat, Fedora, or CentOS), you can quickly verify a file's integrity with a single command:

rpm -Vf /path/to/file

What does this command do?

  • rpm -V (or rpm --verify) checks the integrity of installed packages.
  • -f identifies the installed package that owns the specified file and then verifies the file against the original installed version.

Example Output:

Running the command might give you output like this:

S.5....T.  c /etc/httpd/conf/httpd.conf

Here's what those verification flags mean:

Flag Meaning
S File size differs
M Mode (permissions) differs
5 MD5 checksum differs
D Device number mismatch
L Symlink path differs
U User ownership differs
G Group ownership differs
T Modification time differs
P Capabilities differ
. Test passed (no changes)

If the command produces no output, the file exactly matches what's included in the installed RPM package.

Quick Example:

rpm -Vf /usr/bin/bash

This command verifies the integrity of the bash executable against the installed bash RPM package.



Daily Blog #779: Sunday Funday 3/16/25

 


Hello Reader, 

We've been bouncing around topics a lot and I realized I haven't had a Linux challenge in quite some time.  This week let's see your work as you document all of the logs and artifacts left behind not just from SSH'ing into a linux system but also create a tunnel between the two systems.

The Prize:

$100 Amazon Giftcard


The Rules:

  1. You must post your answer before Friday 3/21/25 7PM CST (GMT -6)
  2. The most complete answer wins
  3. You are allowed to edit your answer after posting
  4. If two answers are too similar for one to win, the one with the earlier posting time wins
  5. Be specific and be thoughtful
  6. Anonymous entries are allowed, please email them to dlcowen@gmail.com. Please state in your email if you would like to be anonymous or not if you win.
  7. In order for an anonymous winner to receive a prize they must give their name to me, but i will not release it in a blog post
  8. AI assistance is welcomed but if a post is deemed to be entirely AI written it will not qualify for a prize. 


The Challenge:

What are all of the artifacts left behind on a Linux system (both server and client) when someone authenticates via SSH and creates a SSH Tunnel.

Daily Blog #778: Solution Saturday 3/15/25

 


Hello Reader,

I guess 'Vibe Coding' isn't a thing for all of you! No winners this week. I'll get tomorrow's challenge back to the blog's regular focus and look forward to seeing your contributions.

 

The Challenge:

 Pick an unsupported DFIR project of your choice and bring it back to life! Add new features and make it work on modern systems. While you are not required to 'vibe code' (AI coding) in this instance it's fully encouraged! Send me links to writeups or github repo's when your done!

 

The Winning Answer:

None


Daily Blog #777: Forensic Lunch Test Kitchen 3/14/25

 


Hello Reader,

Tonight Evan and I tried to fix the workflow and fix some bugs in our CloudTrail log explorer. We had some successes and some failures and ended with the idea that we need a better set of prompts to redefine the problem. We rolled back some changes but made some progress in the end. 

 

Daily Blog #776: Forensic Lunch Test Kitchen 3/13/25

 


Hello Reader,

Tonight on the 'vibe stream' Evan and I added AWS SSO credential support to our AWS CloudTrail explorer script. It lead us into some interesting rabbit holes and ended with us realizing we should get Github commits ready before trying to tweak our workflow. 

 

You can watch it here: