Building a Scalable Centralized Logging Architecture with Rsyslog

In a high-traffic, scalable environment—such as one managing multiple Load Balancers (LB) and Web Workers for platforms like Erlangga Exam—efficient log aggregation is the backbone of operational stability. While modern stacks like Graylog or Grafana Loki are popular, rsyslog remains a powerhouse for high-performance transport and initial processing.

Why Choose Rsyslog Over Graylog or Loki?

When building a scalable logging foundation, rsyslog offers specific architectural advantages:

• Low Resource Overhead: Rsyslog is a lightweight C binary that consumes significantly less CPU and RAM than Java-based Graylog or Go-based Loki. This is critical on Web Workers (e.g., unbk-web101) where resources should be reserved for PHP-FPM or Nginx.
• High Throughput & Concurrency: Rsyslog can handle hundreds of thousands of messages per second with minimal latency, making it ideal for high-concurrency exam environments.
• Reliability (Disk Queuing): If the central server is down, rsyslog can “spool” logs to the local disk of the web worker and transmit them once the connection is restored.
• Protocol Flexibility: It speaks standard Syslog (RFC5424), making it compatible with almost every networking device and OS without requiring custom agents.

Phase 1: The Central Rsyslog Receiver (The Hub)

On your dedicated logging server (e.g., 192.168.100.52), you must enable network listening and dynamic file routing.

1. Enable Network Reception Edit /etc/rsyslog.conf to allow TCP/UDP traffic:

module(load="imtcp")
input(type="imtcp" port="514")

1. Configure Dynamic Separation To prevent logs from Loadbalancer and Web Workers from mixing, use a template in /etc/rsyslog.d/00-remote.conf:

$template DynamicRemote,"/data/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
if $fromhost-ip != '127.0.0.1' then {
    action(type="omfile" dynaFile="DynamicRemote")
    stop
}

###Phase 2: Scaling the Web Workers (The Senders) On each web server, rsyslog should “tail” specific service logs (Nginx, ProxySQL, PHP-FPM) and forward them.

1. Standardize Log Tagging Create /etc/rsyslog.d/60-nginx.conf on your workers:

module(load="imfile") # Load only once!

input(type="imfile"
      File="/var/log/nginx/*.access.log"
      Tag="nginx-access"
      Severity="info")

# Forward to the central Hub
*.* action(type="omfwd" 
           Target="192.168.100.52" 
           Port="514" 
           Protocol="tcp")

###Phase 3: Critical Scaling Considerations To maintain this environment at scale, you must address the following:

1. Permission Management Ensure the syslog user owns the /data/log/remote directory.
2. AppArmor/SELinux On Ubuntu 24.04, update AppArmor profiles to allow writing to custom paths like /data/
3. Log Rotation Use a wildcard logrotate policy (/var/log/remote//.log) to ensure disk space is managed across all host subdirectories.

What’s Next?

While rsyslog excels at transport and storage, it lacks a visual interface for complex querying. In our next session, we will cover:

• GoAccess Integration: Generating real-time HTML dashboards from these rsyslog files.
• Graylog as a Backend: Forwarding from the rsyslog hub into Graylog for advanced search and alerting once your port conflicts are resolved.