Archived
Logging Aggregation #8
No Assignees
nini.b-612
nini.b-612
No due date set.
Dependencies
No dependencies set.
Reference: Committee/Secure-OCF-Web-Hosting#8
Reference in New Issue
Block a user
Use branch
loggingIn terms of ease of setup and teaching, alerting and visualization, easy integration with scripts, and overall fit for the objectives of this project, Grafana Loki, coupled with Fluent Bit, are our top contenders. (And as a plus: in terms of signature-based detections, both YARA and ClamAV are compatible options.)
There is a demo coming up for how to get started with logging using Grafana Loki on October 22nd (Info Here). We can register via this link.
The demo is supposed to teach about what it is and what it does; act as an overview of how Loki works, what makes it different, and what it's good at; and show a demo of new & updated Loki 3.0 features so that users can learn how to create metrics from logs & alert on their logs with powerful Prometheus-style alerting rules.
There's also a video source provided on the link that I provided above: Scaling & securing your logs for reliable petabyte-scale logging.
Lastly, here is Grafana Loki on GitHub.
If we use Grafana Loki and Promtail as our logging aggregation tools, there is the benefit of both YARA and ClamAV scanners (which it was requested that I look into) having the ability to be connected and log to stdout or a local file.
Example Command:
echo "[ALERT] $(date): $(basename "$file") matched rule: $rule_name" >> /var/log/scanner.log/var/log/scanner.logto Loki.Note that this benefit is also noted in the signature-based detections issue since it applies to both.
It looks like Promtail is being deprecated but they made something new called Alloy. Promtail Deprecation
Meeting 3 Notes:
Pros of using Alloy (to Replace Promtail)
Future-proofing: Since Promtail is deprecated, Alloy ensures we're not stuck with unsupported software.
Unified collector: Alloy can handle logs and metrics/traces if we ever expand our observability.
Active development: New features, better integration, etc. Alloy is where new work is going.
Migration path available: Since there’s tooling and docs, switching isn’t blind.
Cons / things to watch
Complexity and learning curve: Users report that Alloy’s docs and configuration are less mature than Promtail’s simpler model. For example:
Resource usage: Some users say Alloy uses more CPU/memory than Promtail under similar workloads.
Maturity of examples: Many “getting started” guides and community content are still Promtail-centric, so finding Alloy-specific patterns might require more effort.
We will need to plan and validate the migration: Since our system will involve custom log scripts, uploads, etc., we'll want to ensure that Alloy supports exactly our use-cases (tailing uploads, container logs, file system watchers) and that we understand the config.
Fit for Our Project (Apache + user uploads + student orgs)
Given Our Objectives:
Aggregating messages from scripts (upload scanners, user-data events)
Making them readable and alertable for non-expert student org members
Running inside Docker/Apache + user file-volume context
Wanting ease of teaching + alert setup
Here's How the Above Tradeoffs Might Affect Us:
The fact that Promtail is deprecated means using it now is fine for short term, but if we expect this system to run long term (and we do), going with Alloy makes sense.
If we choose Alloy, we'll invest a little more up front (config, learning) but gain more longevity.
Because part of our goal is ease of teaching, we need something that everyone can pick up quickly. Promtail has more community tutorials currently—but getting familiar with Alloy early means fewer changes later.
Resource constraints: if our volume is modest (script logs + web uploads) then the extra resource cost might not matter. But we should check that Alloy configuration is tuned so we don’t over-run our environment.
Given we'll likely ship logs from containers + file-system + custom scanners, Alloy’s flexibility is a plus.
ChatGPT's Recommendations:
Start immediately with a minimal Alloy setup (instead of Promtail) for our Docker environment.
Use Alloy to tail the log directories (scanner logs, Apache logs, upload logs) and ship to our log aggregator (e.g., Grafana Loki) + our dashboarding system.
Make sure we test (scripts that emit messages, uploads, etc) and confirm Alloy captures them as expected.
Given the learning phase, prepare a simple “user guide” for everyone: how to open Grafana, how to filter logs, how to set simple alerts—so even if the internal config is more complex, from the user’s point of view it’s simple.
Document the migration path (though if we are starting fresh it’s less of a “migration” than “first choice”).
Keep an eye on resource usage and complexity; tune Alloy’s pipeline to be as light as we need.