143 lines
3.3 KiB
Markdown
143 lines
3.3 KiB
Markdown
# webfortilog
|
|
|
|
Flask-based web application that converts WAF log files into aligned text reports or CSV exports.
|
|
|
|
## Features
|
|
|
|
- Upload a UTF-8 log file where each line is a single record
|
|
- Parse shell-style `key=value` and `key="value with spaces"` tokens
|
|
- Support `vendor` mode with fixed columns and `full` mode with dynamic columns
|
|
- Filter by policy and severity with case-sensitive or case-insensitive partial matching
|
|
- Sort by combined datetime or severity ranking
|
|
- Preview results in the browser and download the generated file
|
|
- Run locally with Flask or in Docker with Gunicorn
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
app/
|
|
services/
|
|
templates/
|
|
tests/
|
|
Dockerfile
|
|
pyproject.toml
|
|
wsgi.py
|
|
```
|
|
|
|
## Local usage
|
|
|
|
### Requirements
|
|
|
|
- Python 3.12
|
|
|
|
### Install
|
|
|
|
```bash
|
|
python3.12 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -e ".[dev]"
|
|
```
|
|
|
|
### Run Container
|
|
|
|
```bash
|
|
export FLASK_APP=wsgi.py
|
|
export MAX_UPLOAD_SIZE_MB=100
|
|
flask run --debug
|
|
```
|
|
|
|
Open `http://127.0.0.1:5000`.
|
|
|
|
### Example input file
|
|
|
|
If you have a local WAF export such as `attack_download.log`, you can use it as a real example upload.
|
|
|
|
- Example file: `attack_download.log`
|
|
- Approximate size in the current workspace: `98.5 MiB`
|
|
- The default `MAX_UPLOAD_SIZE_MB=100` setting is sized to accept a file of that size
|
|
|
|
### Test
|
|
|
|
```bash
|
|
pytest
|
|
```
|
|
|
|
## Docker usage
|
|
|
|
### Build
|
|
|
|
```bash
|
|
docker build -t webfortilog .
|
|
```
|
|
|
|
### Run
|
|
|
|
```bash
|
|
docker run --rm -p 8000:8000 -e MAX_UPLOAD_SIZE_MB=100 webfortilog
|
|
```
|
|
|
|
Open `http://127.0.0.1:8000`.
|
|
|
|
## Docker Compose usage
|
|
|
|
### Start the web app
|
|
|
|
```bash
|
|
docker compose up --build web
|
|
```
|
|
|
|
Compose settings are stored in `env`. Update that file to change values such as:
|
|
|
|
- `SECRET_KEY`
|
|
- `MAX_UPLOAD_SIZE_MB`
|
|
- `OUTPUT_DIRECTORY`
|
|
- `OUTPUT_RETENTION_HOURS`
|
|
- `CLEANUP_ON_STARTUP`
|
|
- `CLEANUP_AFTER_DOWNLOAD`
|
|
|
|
### Run the test suite in a container
|
|
|
|
```bash
|
|
docker compose run --rm test
|
|
```
|
|
|
|
## Example usage
|
|
|
|
### Browser upload
|
|
|
|
1. Start the app with `flask run --debug` or `docker compose up --build web`
|
|
2. Open the web UI
|
|
3. Upload `attack_download.log`
|
|
4. Try `vendor` mode with `text` output for a readable preview
|
|
5. Try `full` mode with `csv` output for complete export coverage
|
|
|
|
### Command-line upload example
|
|
|
|
```bash
|
|
curl -X POST http://127.0.0.1:5000/convert \
|
|
-F "log_file=@attack_download.log" \
|
|
-F "mode=vendor" \
|
|
-F "output_format=text" \
|
|
-F "sort_by=datetime" \
|
|
-F "order=asc" \
|
|
-F "policy_cs=" \
|
|
-F "policy_ci=" \
|
|
-F "severity_cs=" \
|
|
-F "severity_ci="
|
|
```
|
|
|
|
## Notes
|
|
|
|
- Temporary output files are written to `instance/outputs`
|
|
- Generated files are cleaned up according to the configured output retention policy
|
|
- The application does not require a database
|
|
- Gunicorn is used as the production WSGI server
|
|
- Parsing and export writing are streamed to reduce memory usage on large uploads
|
|
- Sorting still materializes the filtered record set because global ordering by datetime or severity requires the full filtered input
|
|
- Default upload limit is 100 MiB
|
|
- Set `MAX_UPLOAD_SIZE_MB` to configure the upload limit in megabytes
|
|
- `MAX_CONTENT_LENGTH` is also supported as a lower-level byte-based override
|
|
- `OUTPUT_RETENTION_HOURS` controls how long generated output files are kept
|
|
- `CLEANUP_ON_STARTUP=true` removes expired generated files when the app starts
|
|
- `CLEANUP_AFTER_DOWNLOAD=true` deletes a result only after the response finishes sending
|