545 lines
12 KiB
Markdown
545 lines
12 KiB
Markdown
# 部署与运维设计文档
|
||
|
||
## 一、部署方式
|
||
|
||
### 1.1 Docker Compose（推荐）
|
||
|
||
一键部署，适合中小团队。
|
||
|
||
```
|
||
docker compose up -d
|
||
```
|
||
|
||
包含的服务：
|
||
|
||
| 服务 | 镜像 | 端口 | 说明 |
|
||
|------|------|------|------|
|
||
| light-sentry-api | 自建 (Node.js) | 9000 | API 服务 + 管理后台 |
|
||
| loki | grafana/loki | 3100 | 日志存储 |
|
||
| promtail | grafana/promtail | - | 日志收集（容器日志） |
|
||
| grafana | grafana/grafana | 3000 | 数据可视化 |
|
||
| dozzle | amir20/dozzle | 8080 | 容器日志查看 |
|
||
| nginx | （系统已有） | 80/443 | 反向代理 + SSL |
|
||
|
||
### 1.2 二进制部署
|
||
|
||
适合对 Docker 不熟悉的场景：
|
||
|
||
1. 安装 Node.js 18+
|
||
2. 安装 Loki（单二进制）
|
||
3. 安装 Grafana
|
||
4. 配置 Nginx
|
||
5. 启动 API 服务（pm2 守护）
|
||
|
||
### 1.3 Kubernetes 部署
|
||
|
||
适合大规模、多租户场景：
|
||
|
||
- Helm Chart 部署
|
||
- HPA 自动扩缩容
|
||
- Loki 分布式模式
|
||
- MySQL / PostgreSQL 集群
|
||
|
||
---
|
||
|
||
## 二、资源需求
|
||
|
||
### 2.1 最小配置（测试/个人用）
|
||
|
||
| 资源 | 配置 | 说明 |
|
||
|------|------|------|
|
||
| CPU | 1 核 | 低流量下足够 |
|
||
| 内存 | 1GB | 紧张，可能需要调小 Loki 缓存 |
|
||
| 磁盘 | 20GB | 存 7 天日志（每天 ~2GB） |
|
||
| 网络 | 1Mbps | 上报流量不大 |
|
||
|
||
**承载能力**：
|
||
- 10 个项目以内
|
||
- 每天 1 万事件以内
|
||
|
||
### 2.2 推荐配置（小团队）
|
||
|
||
| 资源 | 配置 | 说明 |
|
||
|------|------|------|
|
||
| CPU | 2 核 | 应对突发流量 |
|
||
| 内存 | 4GB | Loki + API + Grafana 都够 |
|
||
| 磁盘 | 100GB SSD | 存 30 天日志 |
|
||
| 网络 | 5Mbps | 足够 |
|
||
|
||
**承载能力**：
|
||
- 50 个项目以内
|
||
- 每天 100 万事件以内
|
||
|
||
### 2.3 生产配置（中大型团队）
|
||
|
||
| 资源 | 配置 | 说明 |
|
||
|------|------|------|
|
||
| CPU | 4 核以上 | 高并发上报 |
|
||
| 内存 | 8GB 以上 | Loki 内存索引 |
|
||
| 磁盘 | 500GB+ SSD | 大量日志存储 |
|
||
| 网络 | 10Mbps+ | 大流量上报 |
|
||
|
||
**承载能力**：
|
||
- 200 个项目以内
|
||
- 每天 1000 万事件以内
|
||
|
||
---
|
||
|
||
## 三、Docker Compose 配置详解
|
||
|
||
### 3.1 完整配置说明
|
||
|
||
```yaml
|
||
version: '3.8'
|
||
|
||
services:
|
||
# API 服务 + 管理后台
|
||
light-sentry-api:
|
||
build: .
|
||
container_name: light-sentry-api
|
||
network_mode: host
|
||
environment:
|
||
- NODE_ENV=production
|
||
- PORT=9000
|
||
- LOKI_URL=http://127.0.0.1:3100
|
||
- DATA_DIR=/app/data
|
||
volumes:
|
||
- ./data:/app/data
|
||
restart: unless-stopped
|
||
depends_on:
|
||
- loki
|
||
|
||
# Loki - 日志存储
|
||
loki:
|
||
image: grafana/loki:2.9.4
|
||
container_name: light-sentry-loki
|
||
network_mode: host
|
||
command: -config.file=/etc/loki/local-config.yaml
|
||
volumes:
|
||
- ./loki/config:/etc/loki
|
||
- loki-data:/loki
|
||
restart: unless-stopped
|
||
|
||
# Promtail - 收集容器日志
|
||
promtail:
|
||
image: grafana/promtail:2.9.4
|
||
container_name: light-sentry-promtail
|
||
network_mode: host
|
||
volumes:
|
||
- ./promtail/config:/etc/promtail
|
||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||
restart: unless-stopped
|
||
depends_on:
|
||
- loki
|
||
|
||
# Grafana - 可视化
|
||
grafana:
|
||
image: grafana/grafana:10.2.0
|
||
container_name: light-sentry-grafana
|
||
network_mode: host
|
||
environment:
|
||
- GF_SECURITY_ADMIN_PASSWORD=admin123
|
||
volumes:
|
||
- grafana-data:/var/lib/grafana
|
||
restart: unless-stopped
|
||
|
||
# Dozzle - 容器日志查看
|
||
dozzle:
|
||
image: amir20/dozzle:latest
|
||
container_name: light-sentry-dozzle
|
||
network_mode: host
|
||
volumes:
|
||
- /var/run/docker.sock:/var/run/docker.sock
|
||
restart: unless-stopped
|
||
|
||
volumes:
|
||
loki-data:
|
||
grafana-data:
|
||
```
|
||
|
||
### 3.2 Loki 配置
|
||
|
||
`loki/config/local-config.yaml`
|
||
|
||
```yaml
|
||
auth_enabled: false
|
||
|
||
server:
|
||
http_listen_port: 3100
|
||
|
||
ingester:
|
||
lifecycler:
|
||
ring:
|
||
kvstore:
|
||
store: inmemory
|
||
replication_factor: 1
|
||
chunk_idle_period: 30m
|
||
chunk_retain_period: 1m
|
||
max_transfer_retries: 0
|
||
|
||
schema_config:
|
||
configs:
|
||
- from: 2024-01-01
|
||
store: boltdb-shipper
|
||
object_store: filesystem
|
||
schema: v12
|
||
index:
|
||
prefix: index_
|
||
period: 24h
|
||
|
||
storage_config:
|
||
boltdb_shipper:
|
||
active_index_directory: /loki/index
|
||
cache_location: /loki/index_cache
|
||
shared_store: filesystem
|
||
filesystem:
|
||
directory: /loki/chunks
|
||
|
||
limits_config:
|
||
retention_period: 168h # 7 天
|
||
per_stream_rate_limit: 10MB
|
||
ingestion_rate_mb: 50
|
||
ingestion_burst_size_mb: 100
|
||
max_streams_per_user: 10000
|
||
max_chunks_per_query: 2000000
|
||
max_query_series: 5000
|
||
|
||
table_manager:
|
||
retention_deletes_enabled: true
|
||
retention_period: 168h # 7 天
|
||
```
|
||
|
||
---
|
||
|
||
## 四、Nginx 配置
|
||
|
||
### 4.1 反向代理配置
|
||
|
||
```nginx
|
||
server {
|
||
listen 443 ssl http2;
|
||
server_name log.example.com;
|
||
|
||
# SSL 配置
|
||
ssl_certificate /path/to/fullchain.pem;
|
||
ssl_certificate_key /path/to/privkey.pem;
|
||
|
||
# API 服务（Sentry 上报 + 管理后台 API）
|
||
location /api/ {
|
||
proxy_pass http://127.0.0.1:9000/api/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
|
||
# 上报接口超时
|
||
proxy_read_timeout 30s;
|
||
proxy_connect_timeout 10s;
|
||
proxy_send_timeout 30s;
|
||
|
||
# CORS（SDK 上报需要）
|
||
add_header Access-Control-Allow-Origin * always;
|
||
add_header Access-Control-Allow-Methods GET,POST,OPTIONS always;
|
||
add_header Access-Control-Allow-Headers Content-Type,X-Sentry-Auth always;
|
||
if ($request_method = OPTIONS) {
|
||
return 204;
|
||
}
|
||
}
|
||
|
||
# 管理后台
|
||
location /manage/ {
|
||
proxy_pass http://127.0.0.1:9000/manage/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
}
|
||
|
||
# Grafana
|
||
location /grafana/ {
|
||
proxy_pass http://127.0.0.1:3000/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
}
|
||
|
||
# Dozzle
|
||
location /dozzle/ {
|
||
proxy_pass http://127.0.0.1:8080/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
|
||
# WebSocket 支持
|
||
proxy_set_header Upgrade $http_upgrade;
|
||
proxy_set_header Connection "upgrade";
|
||
}
|
||
|
||
# Loki（可选，直接暴露给 Grafana 用，一般不需要外部访问）
|
||
location /loki/ {
|
||
proxy_pass http://127.0.0.1:3100/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
deny all; # 禁止外部访问，只允许内网
|
||
}
|
||
|
||
# 根路径重定向到管理后台
|
||
location = / {
|
||
return 301 /manage/;
|
||
}
|
||
}
|
||
```
|
||
|
||
### 4.2 限流配置
|
||
|
||
```nginx
|
||
# 定义限流 zone
|
||
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=100r/s;
|
||
|
||
location /api/ {
|
||
limit_req zone=api_limit burst=200 nodelay;
|
||
# ... 其他配置
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 五、CI/CD 部署
|
||
|
||
### 5.1 Gitea Actions 示例
|
||
|
||
```yaml
|
||
name: Deploy Light-Sentry
|
||
|
||
on:
|
||
push:
|
||
branches: [ main ]
|
||
workflow_dispatch:
|
||
|
||
jobs:
|
||
deploy:
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
|
||
- name: Deploy via SSH
|
||
uses: appleboy/ssh-action@v1.0.0
|
||
with:
|
||
host: ${{ secrets.SERVER_HOST }}
|
||
username: ${{ secrets.SERVER_USER }}
|
||
key: ${{ secrets.SSH_PRIVATE_KEY }}
|
||
script: |
|
||
cd ~/light-sentry
|
||
git pull
|
||
docker compose build light-sentry-api
|
||
docker compose up -d
|
||
echo "Deploy completed"
|
||
```
|
||
|
||
### 5.2 智能重启（只重启变更的服务）
|
||
|
||
检测变更的文件，只重启相关服务，减少停机时间：
|
||
|
||
- `src/**` → 重启 API 服务
|
||
- `loki/**` → 重启 Loki
|
||
- `grafana/**` → 重启 Grafana
|
||
- `docker-compose.yml` → 全部重启
|
||
- `nginx/**` → reload Nginx
|
||
|
||
---
|
||
|
||
## 六、监控与告警
|
||
|
||
### 6.1 自身监控
|
||
|
||
Light-Sentry 自己也要监控自己：
|
||
|
||
| 监控项 | 方式 | 告警阈值 |
|
||
|--------|------|----------|
|
||
| API 服务存活 | 健康检查接口 + Prometheus | 502 持续 1 分钟 |
|
||
| Loki 服务存活 | Grafana 内置监控 | 连接失败 |
|
||
| 磁盘使用率 | Node Exporter | > 80% 告警 |
|
||
| 内存使用率 | Node Exporter | > 85% 告警 |
|
||
| CPU 使用率 | Node Exporter | > 90% 告警 |
|
||
| 事件上报量 | Loki 查询 | 突增 200% |
|
||
| 错误率 | Loki 查询 | > 5% |
|
||
|
||
### 6.2 健康检查
|
||
|
||
```
|
||
GET /health
|
||
|
||
{
|
||
"status": "ok",
|
||
"timestamp": "2024-01-01T00:00:00Z",
|
||
"uptime": 86400,
|
||
"queue_size": 123,
|
||
"loki": "connected",
|
||
"mysql": "connected"
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 七、备份与恢复
|
||
|
||
### 7.1 需要备份的数据
|
||
|
||
| 数据 | 位置 | 备份频率 | 保留时间 |
|
||
|------|------|----------|----------|
|
||
| 项目配置 | data/projects.json | 每天 | 30 天 |
|
||
| Loki 数据 | loki-data 卷 | 每周 | 4 周 |
|
||
| Grafana 配置 | grafana-data 卷 | 每天 | 30 天 |
|
||
| 告警规则 | MySQL | 每天 | 永久 |
|
||
| 错误聚合 | MySQL | 每天 | 永久 |
|
||
|
||
### 7.2 备份脚本
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
# backup.sh - Light-Sentry 备份脚本
|
||
|
||
BACKUP_DIR="/backup/light-sentry"
|
||
DATE=$(date +%Y%m%d_%H%M%S)
|
||
|
||
# 创建备份目录
|
||
mkdir -p $BACKUP_DIR
|
||
|
||
# 备份配置文件
|
||
tar czf $BACKUP_DIR/config_$DATE.tar.gz /opt/light-sentry/data/
|
||
|
||
# 备份 Loki 数据（可选，量大）
|
||
# docker run --rm -v loki-data:/data -v $BACKUP_DIR:/backup \
|
||
# alpine tar czf /backup/loki_$DATE.tar.gz -C /data .
|
||
|
||
# 备份 Grafana
|
||
docker run --rm -v grafana-data:/data -v $BACKUP_DIR:/backup \
|
||
alpine tar czf /backup/grafana_$DATE.tar.gz -C /data .
|
||
|
||
# 清理 30 天前的备份
|
||
find $BACKUP_DIR -name "*.tar.gz" -mtime +30 -delete
|
||
|
||
echo "Backup completed: $DATE"
|
||
```
|
||
|
||
### 7.3 恢复流程
|
||
|
||
```bash
|
||
# 1. 停止服务
|
||
docker compose down
|
||
|
||
# 2. 恢复配置
|
||
tar xzf backup/config_20240101_000000.tar.gz -C /opt/light-sentry/
|
||
|
||
# 3. 恢复 Grafana
|
||
docker run --rm -v grafana-data:/data -v /backup:/backup \
|
||
alpine tar xzf /backup/grafana_20240101_000000.tar.gz -C /data
|
||
|
||
# 4. 启动服务
|
||
docker compose up -d
|
||
```
|
||
|
||
---
|
||
|
||
## 八、性能优化
|
||
|
||
### 8.1 API 服务优化
|
||
|
||
- **Node.js 集群模式**：利用多核 CPU
|
||
- **内存队列调优**：根据流量调整队列大小和刷盘频率
|
||
- **连接池**：数据库连接池、HTTP 连接池
|
||
- **缓存**：项目配置缓存，不用每次查存储
|
||
|
||
### 8.2 Loki 优化
|
||
|
||
- **调整保留时间**：从 7 天调整到实际需要的时间
|
||
- **降低索引基数**：减少 label 数量
|
||
- **批量写入**：增加 chunk 大小，减少写入次数
|
||
- **内存调优**：根据实际数据量调整内存
|
||
|
||
### 8.3 Nginx 优化
|
||
|
||
- **启用 gzip 压缩**：减少传输体积
|
||
- **启用缓存**：静态文件缓存
|
||
- **TCP 参数调优**：增加连接数限制
|
||
|
||
---
|
||
|
||
## 九、安全加固
|
||
|
||
### 9.1 网络安全
|
||
|
||
- 管理后台加访问密码（HTTP Basic Auth 或登录）
|
||
- Loki 不对外暴露（只允许 Grafana 访问）
|
||
- Dozzle 加访问密码
|
||
- 只开放必要端口（80/443）
|
||
|
||
### 9.2 数据安全
|
||
|
||
- SDK 上报接口 CORS 限制域名（可选）
|
||
- 项目 DSN 泄露后可重置 publicKey
|
||
- 敏感数据自动脱敏
|
||
- 定期备份，异地存储
|
||
|
||
### 9.3 依赖安全
|
||
|
||
- 定期更新依赖
|
||
- 镜像扫描漏洞
|
||
- 最小权限运行
|
||
|
||
---
|
||
|
||
## 十、常见问题排查
|
||
|
||
### 10.1 事件上报失败
|
||
|
||
1. 检查网络是否通：`curl -I https://log.example.com/api/1001/envelope/`
|
||
2. 检查 DSN 是否正确：publicKey 和 projectId 对应
|
||
3. 查看 API 日志：`docker logs light-sentry-api`
|
||
4. 查看 Nginx 日志：`tail -f /var/log/nginx/access.log`
|
||
|
||
### 10.2 Grafana 查不到数据
|
||
|
||
1. 检查 Loki 数据源是否配置正确
|
||
2. 检查时间范围是否正确
|
||
3. 检查 label 拼写是否正确
|
||
4. 查看 Loki 日志：`docker logs light-sentry-loki`
|
||
|
||
### 10.3 服务无法启动
|
||
|
||
1. 检查端口是否被占用：`netstat -tlnp | grep 9000`
|
||
2. 查看容器日志：`docker logs light-sentry-api`
|
||
3. 检查配置文件格式：JSON / YAML 语法
|
||
4. 检查磁盘空间：`df -h`
|
||
|
||
---
|
||
|
||
## 十一、升级策略
|
||
|
||
### 11.1 版本升级
|
||
|
||
```bash
|
||
# 拉取最新代码
|
||
git pull
|
||
|
||
# 重新构建并启动
|
||
docker compose build light-sentry-api
|
||
docker compose up -d
|
||
|
||
# 验证
|
||
curl https://log.example.com/health
|
||
```
|
||
|
||
### 11.2 数据迁移
|
||
|
||
- 配置向后兼容，新版本能读旧版本数据
|
||
- 启动时自动检测并执行数据迁移
|
||
- 迁移前自动备份
|