Malicious activity detection in Oracle Cloud using Falco — Part 1

7 min readOct 9, 2023

I’ve been pontificating about provisioning, monitoring Kubernetes clusters (OKE) on Oracle cloud and deploying applications and scaling applications across regions. But with the exception of a couple of articles, I have barely touched the subject of cloud native security. That’s mainly because when it comes to security, the impostor syndrome is suddenly strong with this one.

In this article, we’ll talk about malicious activity detection on Oracle Cloud using Falco.

What is Falco?

From the horse’s mouth:

Falco is a cloud native security tool designed for Linux systems. It employs custom rules on kernel events, which are enriched with container and Kubernetes metadata to provide real-time alerts. Falco helps you gain visibility into abnormal behavior, potential security threats and compliance violations contributing to comprehensive runtime security.

There are lots of things to like about Falco and below are some of them:

It runs on hosts, containers, Kubernetes and with cloud services: as workloads evolve and migrate to the cloud, it is useful to be able to monitor and detect security problems across the breadth of differing workloads
A Rules Maturity Framework that indicates the readiness level of an existing library of rules
A rules library: an out-of-the-box set of rules that have been battle-tested and help folks get started. These rules are aligned with the MITRE Attack framework which allows you to demonstrate compliance and extend and contribute back.
Integration with many systems for storage, analysis and notification.
Ability to leverage BPF

Let’s get started.

Simple use case: OCI Compute

We’ll start with a simple example using an OCI Compute instance running Oracle Linux. Provision an instance, ssh to it and let’s install Falco. The idea is to feel our way around.

As there’s no pre-built module for Oracle Linux yet, we’ll build one ourselves and in order to do so, we’ll install some package dependencies and ensure the right GCC version is used to build the kernel module for Falco:

sudo dnf config-manager --enable ol8_developer_EPEL,ol8_codeready_builder,ol8_distro_builder
sudo dnf repolist
sudo dnf install -y dkms make
sudo dnf install -y kernel-uek-devel-`uname -r` clang llvm dialog
sudo dnf install gcc-toolset-11

Next, we’ll configure the yum repos for Falco and install it:

sudo rpm --import https://falco.org/repo/falcosecurity-packages.asc
sudo curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo

sudo dnf install -y falco

When you’re asked to pick your preferred driver, select Kmod:

Note that this installation will fail. That’s because of a condition in the falco-driver-loader script which obtain as part of the installation and that we’ll edit now:

sudo vi /usr/bin/falco-driver-loader

Locate this condition and comment it:

        #if [[ ${DRIVER_KERNEL_RELEASE} == *uek* ]]; then
        #        >&2 echo "Skipping because the dkms install always fail (on UEK hosts)"
        #        return
        #fi

Then, run it again with the correct gcc version:

sudo scl enable gcc-toolset-11 -- falco-driver-loader

Verify that the Systemd service files have been created:

sudo systemctl list-unit-files "falco*"

UNIT FILE                        STATE
falco-bpf.service                disabled
falco-custom.service             disabled
falco-kmod-inject.service        static
falco-kmod.service               disabled
falco-modern-bpf.service         disabled
falcoctl-artifact-follow.service disabled

6 unit files listed.

Enable the falco-kmod service:

sudo systemctl enable falco-kmod.service
Created symlink /etc/systemd/system/falco.service → /usr/lib/systemd/system/falco-kmod.service.
Created symlink /etc/systemd/system/multi-user.target.wants/falco-kmod.service → /usr/lib/systemd/system/falco-kmod.service.

sudo systemctl list-unit-files "falco*"
UNIT FILE                        STATE
falco-bpf.service                disabled
falco-custom.service             disabled
falco-kmod-inject.service        static
falco-kmod.service               enabled
falco-modern-bpf.service         disabled
falco.service                    enabled
falcoctl-artifact-follow.service disabled

7 unit files listed.

And start the services:

sudo systemctl start falco-kmod-inject
sudo systemctl start falco

Finally, you can verify their statuses:

sudo systemctl start falco
[opc@o-tnfrla ~]$ systemctl status "falco*"
● falco-kmod-inject.service - Falco: Container Native Runtime Security with kmod, inject.
   Loaded: loaded (/usr/lib/systemd/system/falco-kmod-inject.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2023-10-06 17:39:51 AEDT; 3min 0s ago
     Docs: https://falco.org/docs/
  Process: 50240 ExecStart=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 50240 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 22531)
   Memory: 0B
   CGroup: /system.slice/falco-kmod-inject.service

Oct 06 17:39:51 o-tnfrla systemd[1]: Starting Falco: Container Native Runtime Security with kmod, inject....
Oct 06 17:39:51 o-tnfrla systemd[1]: Started Falco: Container Native Runtime Security with kmod, inject..

● falcoctl-artifact-follow.service - Falcoctl Artifact Follow: automatic artifacts update service
   Loaded: loaded (/usr/lib/systemd/system/falcoctl-artifact-follow.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-10-06 17:40:09 AEDT; 2min 42s ago
     Docs: https://falco.org/docs/
 Main PID: 50255 (falcoctl)
    Tasks: 9 (limit: 22531)
   Memory: 19.0M
   CGroup: /system.slice/falcoctl-artifact-follow.service
           └─50255 /usr/bin/falcoctl artifact follow --allowed-types=rulesfile

Oct 06 17:40:09 o-tnfrla systemd[1]: Started Falcoctl Artifact Follow: automatic artifacts update service.
Oct 06 17:40:10 o-tnfrla falcoctl[50255]: INFO: Creating follower for "falco-rules:0", with check every 6h0m0s
Oct 06 17:40:10 o-tnfrla falcoctl[50255]:  INFO  Starting follower for "ghcr.io/falcosecurity/rules/falco-rules:0"
Oct 06 17:40:11 o-tnfrla falcoctl[50255]:  INFO   (ghcr.io/falcosecurity/rules/falco-rules:0) found new version under tag "0"
Oct 06 17:40:18 o-tnfrla falcoctl[50255]:  ERRO   (ghcr.io/falcosecurity/rules/falco-rules:0) an error occurred while pulling artifact from remote repository: could not verify signature f>

● falco-kmod.service - Falco: Container Native Runtime Security with kmod
   Loaded: loaded (/usr/lib/systemd/system/falco-kmod.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-10-06 17:40:09 AEDT; 2min 42s ago
     Docs: https://falco.org/docs/
 Main PID: 50254 (falco)
    Tasks: 7 (limit: 22531)
   Memory: 11.8M
   CGroup: /system.slice/falco-kmod.service
           └─50254 /usr/bin/falco

Oct 06 17:40:09 o-tnfrla systemd[1]: Started Falco: Container Native Runtime Security with kmod.
Oct 06 17:40:09 o-tnfrla falco[50254]: Falco version: 0.36.0 (x86_64)
Oct 06 17:40:09 o-tnfrla falco[50254]: Falco initialized with configuration file: /etc/falco/falco.yaml
Oct 06 17:40:09 o-tnfrla falco[50254]: Loading rules from file /etc/falco/falco_rules.yaml

Simulating a malicious activity

Since we now have Falco running, let’s simulate a potentially malicious activity and verify that Falco catches it:

sudo cat /etc/shadow > /dev/null

Let’s examine Falco output using journalctl:

sudo journalctl _COMM=falco -p warning
-- Logs begin at Fri 2023-10-06 17:22:01 AEDT, end at Fri 2023-10-06 17:49:42 AEDT. --
Oct 06 17:49:29 falco[50254]: 17:49:29.011960027: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=bash ggparent=sshd gggparent=sshd evt>

We can examine syslog too:

sudo grep Sensitive /var/log/messages
Oct  6 06:49:29 falco[50254]: 17:49:29.011960027: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=bash ggparent=sshd gggparent=sshd evt_type=openat user=root user_uid=0 user_loginuid=1000 process=cat proc_exepath=/usr/bin/cat parent=sudo command=cat /etc/shadow terminal=34816 exe_flags=O_RDONLY container_id=host container_name=host)

A little explanation on the installation procedure

According to my good mate Avi Miller who has an encyclopedic knowledge of Oracle Linux things and went on a sleuthing expedition, the issue stems from the fact that Oracle Linux compiles UEK 7 using GCC 11 which is not the default GCC version on Oracle Linux 8. In order to compile Kernel modules for UEK 7, we must first install GCC 11 before installing Falco.

Because of this issue, the Falco team has also placed a condition inside the install script that bypasses the use of dkms when UEK is detected. We suspect this condition was placed because they ran into the same problem, hence the need to now edit the script and re-enable dkms for UEK.

We assume this is also the reason why the Falco team was unable to successfully build a kernel module for UEK 7. And when you try to build one by yourself, this fails too.

Adding Observability with OCI Logging

So far, we’ve more or less merely followed the Falco guide and ensured it works on Oracle Linux and in this case, we happened to be using OCI as an infrastructure provider. But running on OCI also provides you with a number of advantages, particularly for observability and there are a number of services on OCI that can assist, especially:

OCI Logging and OCI Logging Analytics
OCI OpenSearch Service

Which one you choose is up to you. As the saying goes, beauty is in the eyes of the beholder. Let’s start by shipping the Falco outputs to OCI Logging.

Log into the OCI Console, create a dynamic group and the instance to it.

Next, create a policy to allow the dynamic group to ship logs to OCI Logging:

Next, create a custom log group and a custom log.

On the agent configuration, ensure you set:

group type to Dynamic Group and you select the one you just created above
input type is log path and the file paths is /var/log/messages

Eventually, the Falco message will make it to OCI Logging:

Summary

In this article, we’ve shown you how to install and run Falco on Oracle Linux 8 and work around some of the installation issues. Once Falco is successfully started, we are able to capture events and log them to syslog.

We then use the OCI monitoring agent to ship the Falco log events to OCI Logging where can now do more interesting stuff.

In future articles, we’ll look at the following:

Ship Falco messages to OCI OpenSearch service
Combine with OCI Observability and other OCI services as part of a security observability solution.

Special thanks again to my colleague Avi Miller for his contribution to this article.