Daily Blog #786: Sunday Funday 3/23/25

 

Hello Reader, 

Last week we focused on SSH logins and tunnels between two linux systems. At the conclusion of this Chris Eng, of the ogmini blog , proposed an interesting question. What would a windows SSH server leave behind? So this week let's find out!

The Prize:

$100 Amazon Giftcard

The Rules:

1. You must post your answer before Friday 3/28/25 7PM CST (GMT -6)
2. The most complete answer wins
3. You are allowed to edit your answer after posting
4. If two answers are too similar for one to win, the one with the earlier posting time wins
5. Be specific and be thoughtful
6. Anonymous entries are allowed, please email them to dlcowen@gmail.com. Please state in your email if you would like to be anonymous or not if you win.
7. In order for an anonymous winner to receive a prize they must give their name to me, but i will not release it in a blog post
8. AI assistance is welcomed but if a post is deemed to be entirely AI written it will not qualify for a prize. 

The Challenge:

 Test what artifacts are left behind from SSHing into a Windows 11 or 10 system using the native SSH server. Bonus points for tunnels. 

Also Read: Daily Blog #785: Solution Saturday 3/22/25

Read More

Daily Blog #785: Solution Saturday 3/22/25

 

Hello Reader,

This week's SSH challenge had several contenders. It's always interesting to see what does and does not get your attention and time! I think this should help many people looking for where to look and also opens the door for some more advanced scenarios that we can explore!

 

The Challenge:

What are all of the artifacts left behind on a Linux system (both server and client) when someone authenticates via SSH and creates a SSH Tunnel.

 

The Winning Answer:

 Chris Eng with the OG Mini blog:

https://ogmini.github.io/2025/03/21/David-Cowen-Sunday-Funday-SSH.html

 

Also Read: Validating linux systems with Yum

Read More

Daily Blog #784: Validating linux systems with Yum

Hello Reader,

In the prior posts I've been posting about using rpm to validate packages, but there are other package managers out there. I've decided to look into each package manager individually and then maybe we can make a conditional script to handle all of them. Here is the yum version:

 

#!/bin/bash

# Files to store results
VERIFIED="verified"
FAILURES="failures"
DEBUG="debug"

# Clean previous results
> "$VERIFIED"
> "$FAILURES"
> "$DEBUG"

# Iterate over installed packages managed by yum
for package in $(yum list installed | awk 'NR>1 {print $1}' | cut -d. -f1); do
  echo "Processing package: $package"

  # Find repository URL
  repo_url=$(yumdownloader --urls "$package" 2>/dev/null | head -n 1)

  if [[ -z "$repo_url" ]]; then
    echo "Repository URL not found for package: $package" | tee -a "$FAILURES"
    echo "$repo_url $package" | tee -a "$DEBUG"
    continue
  fi

  # Download RPM package temporarily
  tmp_rpm="/tmp/${package}.rpm"
  curl -s -L "$repo_url" -o "$tmp_rpm"

  if [[ ! -f "$tmp_rpm" ]]; then
    echo "Failed to download RPM - Package: $package" | tee -a "$FAILURES"
    echo "$repo_url $package" | tee -a "$DEBUG"
    continue
  fi

  # Get repository file hashes from the downloaded RPM
  repoquery_hashes=$(rpm2cpio "$tmp_rpm" | cpio -idmv --no-absolute-filenames 2>/dev/null; find . -type f -exec sha256sum {} \;)

  # Verify files
  echo "$repoquery_hashes" | while read -r repo_hash repo_file; do
    local_file="/$repo_file"

    # Check file existence and type
    if [[ ! -x "$local_file" ]] || [[ ! -f "$local_file" ]] || [[ -h "$local_file" ]]; then
      continue
    fi

    # Calculate local disk hash
    disk_hash=$(sha256sum "$local_file" 2>/dev/null | awk '{print $1}')

    if [[ "$disk_hash" == "$repo_hash" ]]; then
      echo "Verified - Package: $package, File: $local_file" >> "$VERIFIED"
    else
      echo "Hash mismatch (Repository) - Package: $package, File: $local_file" | tee -a "$FAILURES"
      echo "$disk_hash $repo_hash $package $local_file" | tee -a "$DEBUG"
    fi
  done

  # Cleanup extracted files and downloaded RPM
  rm -rf ./* "$tmp_rpm"
done

echo "Verification complete. Results are stored in '$VERIFIED' and '$FAILURES'."

Also Read: Automating RPM checks
 

Read More

Daily Blog #783: Automating rpm checks

 

Hello Reader,

I'm recreating my 24 year old perl scipt in bash to allow someone to validate all of the installed rpms on a system against both the local rpm DB and the repository it came from.  This should allow a certain since of comfort on if any core system packages have been manipulated.

 

#!/bin/bash

# Files to store results
VERIFIED="verified"
FAILURES="failures"
DEBUG="debug"

# Clean previous results
> "$VERIFIED"
> "$FAILURES"
> "$DEBUG"

# Iterate over installed RPM packages
for package in $(rpm -qa); do
  echo "Processing package: $package"

  # Find repository URL
  repo_url=$(dnf repoquery -q --location "$package" 2>/dev/null | head -n 1)

  if [[ -z "$repo_url" ]]; then
    echo "Repository URL not found for package: $package" | tee -a "$FAILURES"
    echo "$repo_url $package" | tee -a "$DEBUG"
    continue
  fi

  # Get local file hashes from RPM database
  rpm -ql --dump "$package" | while read -r line; do
    file_path=$(echo "$line" | awk '{print $1}')
    rpm_hash=$(echo "$line" | awk '{print $4}')

    # Skip directories and non-executable files
    if [[ ! -x "$file_path" ]]; then
       continue
    fi
    
    if [[ ! -f "$file_path" ]]; then
       continue
    fi

    if [[ -h "$file_path" ]]; then
       continue
    fi
    # Calculate local disk hash
    disk_hash=$(sha256sum "$file_path" 2>/dev/null | awk '{print $1}')

    if [[ "$disk_hash" != "$rpm_hash" ]]; then
      echo "Hash mismatch (Local RPM DB) - Package: $package, File: $file_path" | tee -a "$FAILURES"
      echo "$dish_hash $rpm_hash $package $file_path" | tee -a "$DEBUG"
      continue
    fi

    # Get repository RPM hash
    repo_hash=$(rpm -qp --dump "$repo_url" 2>/dev/null | grep " $file_path " | awk '{print $4}')

    if [[ -z "$repo_hash" ]]; then
      echo "File not found in repository RPM - Package: $package, File: $file_path" | tee -a "$FAILURES"
      echo "$repo_hash $repo_url $file_path" | tee -a "$DEBUG"
      continue
    fi

    if [[ "$disk_hash" == "$repo_hash" ]]; then
      echo "Verified - Package: $package, File: $file_path" >> "$VERIFIED"
    else
      echo "Hash mismatch (Repository) - Package: $package, File: $file_path" | tee -a "$FAILURES"
      echo "$disk_hash $repo_hash $package $file_path" | tee -a "$DEBUG"
    fi
  done
done

echo "Verification complete. Results are stored in '$VERIFIED' and '$FAILURES'."

Also Read: Validating linux packages other than rpms

 

Read More

Daily Blog #782: Validating linux packages other than rpms

 

Hello Reader,

      We've talked about validating rpms in several posts now but there are other package managers besides rpm. Let's talk about how we can do the same validation with other package managers.

 

1. Debian/Ubuntu (dpkg & debsums)

Install debsums if you haven't already:

sudo apt install debsums

Verify file hashes for a specific package:

sudo debsums -s <package-name>

Verify a specific file:

sudo debsums -s <package-name> | grep /path/to/file

Verify all installed packages:

sudo debsums -cs

---

2. Arch Linux (pacman)

Check integrity of a specific package:

pacman -Qkk <package-name>

Verify a single file:

pacman -Qkk <package-name> | grep /path/to/file

Verify all installed packages:

pacman -Qkk

---

3. openSUSE (rpm & zypper)

openSUSE uses RPM, so you can use standard RPM verification commands:

Check integrity of a file against the RPM database:

rpm -Vf /path/to/file

Verify all installed packages:

rpm -Va

---

4. Alpine Linux (apk)

Newer Alpine Linux versions (3.15+) include the apk audit command:

Verify integrity of a package:

apk audit <package-name>

Verify all installed packages:

apk audit

Also Read: Self validating linux executables

Read More

Daily Blog #781: Validating local linux hashes to their distros

 

Hello Reader,

In my previous blog post, I explained how to use the rpm tool to validate a file on disk against the local RPM metadata. But what if you suspect that an entire package—not just a single file—has been tampered with?

This is where we can leverage a powerful feature I mentioned earlier: extracting metadata directly from the Linux distribution’s repository. Since this remote repository should be unaffected by any local security incidents, it allows you to verify that packages like the core system utilities in this example remain unaltered by a potential threat actor.

---

Step 1: Identify the Repository URL

To fetch the official package version, you first need to determine the correct repository URL. You can do this using the dnf repoquery command:

dnf repoquery --location coreutils (you can specify any package name)

This will return a URL similar to:

https://ftp.redhat.com/pub/redhat/linux/enterprise/9/en/os/x86_64/Packages/coreutils-8.32-31.el9.x86_64.rpm

---

Step 2: Extract the Official Package Hash

Now that you have the package URL, you can use rpm to retrieve its metadata, including file hashes, without downloading the full package:

rpm -q --dump -p https://ftp.redhat.com/pub/redhat/linux/enterprise/9/en/os/x86_64/Packages/coreutils-8.32-31.el9.x86_64.rpm

---

Step 3: Compare Local vs. Repository Hashes

To ensure your package is untouched, compare:

1. The hash of the local file (e.g., /bin/ls)
2. The hash stored in the local RPM database
3. The hash from the official repository package

If all three hashes match, you can be highly confident that your package has not been altered.

Of course, this assumes there isn’t a worst-case scenario where the original distribution’s repository has been compromised—but let’s hope it never comes to that!

By following these steps, you can verify system integrity efficiently using native Linux tools. 

Also Read: Self Validating Linux Executables

Read More

Daily Blog #780: Self validating linux executables

 

Hello Reader,

When I first started doing Digital Forensics and Incident Response (DFIR) back in 2000, one challenge I faced was verifying the integrity of Linux executables. Often, systems didn't record file hashes during installation, and sometimes we didn't even have a reliable set of installation media to compare against.

To solve this problem, I initially wrote a lengthy Perl script. The script went through each installed package on the local system, comparing the file hashes on disk against the original package hashes, and then validating these against the official distribution hashes.

Today, this task is much simpler. On Linux systems using RPM (like RedHat, Fedora, or CentOS), you can quickly verify a file's integrity with a single command:

rpm -Vf /path/to/file

What does this command do?

• rpm -V (or rpm --verify) checks the integrity of installed packages.
• -f identifies the installed package that owns the specified file and then verifies the file against the original installed version.

Example Output:

Running the command might give you output like this:

S.5....T.  c /etc/httpd/conf/httpd.conf

Here's what those verification flags mean:

Flag	Meaning
S	File size differs
M	Mode (permissions) differs
5	MD5 checksum differs
D	Device number mismatch
L	Symlink path differs
U	User ownership differs
G	Group ownership differs
T	Modification time differs
P	Capabilities differ
.	Test passed (no changes)

If the command produces no output, the file exactly matches what's included in the installed RPM package.

Quick Example:

rpm -Vf /usr/bin/bash

This command verifies the integrity of the bash executable against the installed bash RPM package.

Also Read: Daily Blog #779: Sunday Funday 3/16/25

Read More

Daily Blog #779: Sunday Funday 3/16/25

Hello Reader, 

We've been bouncing around topics a lot and I realized I haven't had a Linux challenge in quite some time.  This week let's see your work as you document all of the logs and artifacts left behind not just from SSH'ing into a linux system but also create a tunnel between the two systems.

The Prize:

$100 Amazon Giftcard

The Rules:

1. You must post your answer before Friday 3/21/25 7PM CST (GMT -6)
2. The most complete answer wins
3. You are allowed to edit your answer after posting
4. If two answers are too similar for one to win, the one with the earlier posting time wins
5. Be specific and be thoughtful
6. Anonymous entries are allowed, please email them to dlcowen@gmail.com. Please state in your email if you would like to be anonymous or not if you win.
7. In order for an anonymous winner to receive a prize they must give their name to me, but i will not release it in a blog post
8. AI assistance is welcomed but if a post is deemed to be entirely AI written it will not qualify for a prize. 

The Challenge:

What are all of the artifacts left behind on a Linux system (both server and client) when someone authenticates via SSH and creates a SSH Tunnel.

Also Read: Daily Blog #778: Solution Saturday 3/15/25

Read More