03-NGINX日志
https://bloodzer0.github.io/ossa/log_analysis/nginx_log/
Nginx配置日志格式
初始化配置
shell
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';配置为JSON格式并添加字段
shell
log_format main escape=json '{ "@timestamp": "$time_local", '
'"remote_addr": "$remote_addr",'
'"remote_port": "$remote_port",'
'"scheme": "$scheme",'
'"request_uri": "$request_uri",'
'"request_method": "$request_method",'
'"request_time": "$request_time",'
'"request_length": "$request_length",'
'"response_status": "$status",'
'"body_bytes_sent": "$body_bytes_sent",'
'"http_referer": "$http_referer",'
'"http_user_agent": "$http_user_agent",'
'"http_x_forwarded_for": "$http_x_forwarded_for",'
'"upstream_addr": "$upstream_addr",'
'"upstream_response_time": "$upstream_response_time"}';Nginx日志字段含义
| 字段 | 含义 | 示例 |
|---|---|---|
| body_bytes_sent | 响应body字节数 | 3650 |
| bytes_sent | 响应总字节数 | 175 |
| host | IP或域名(不包括端口) | 10.10.10.14 |
| http_host | IP或域名(包括端口) | 10.10.10.14:81 |
| http_referer | referer信息 | https://blog.share888.top/ 引导用户到当前请求的页面的来源URL |
| http_user_agent | UA信息 | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36 |
| http_x_forwarded_for | XFF信息 | 192.168.1.1 通过代理服务器的原始客户端 IP 地址。 |
| remote_addr | 客户端地址 | 10.10.10.1 |
| remote_user | 客户端认证用户名 | admin |
| request | 请求URI和协议 | GET /favicon.ico HTTP/1.1 |
| request_body | 请求的body | |
| request_length | 请求长度 | 571 |
| request_method | 请求方法 | GET |
| request_time | 请求处理时间 | 10 请求处理的总时间（秒） |
| response_body | 返回的body | |
| response_header_data | 响应头数据 | |
| schema | 协议 | http |
| server_name | 虚拟主机名称 | |
| server_port | 服务器端口 | |
| server_protocol | 服务器协议 | |
| ssl_cipher | 交换数据中的算法 | |
| ssl_protocol | SSL协议版本 | |
| status | 返回状态码 | 404 |
| time_local | 时间戳 | 16/Jun/2019:23:29:50 -0400 |
| upstream_addr | 后端提供服务地址 | |
| upstream_connect_time | 与服务器连接所花费的时间 | |
| upstream_response_time | 后端处理时间 | 后端服务器响应时间（适用于反向代理） |
| upstream_status | upstream状态 | 200 |
打印request_body
shell
# web_server配置
# The variable’s value is made available in locations processed by the proxy_pass, fastcgi_pass, uwsgi_pass, and scgi_pass directives when the request body was read to a memory buffer.
# 意味着：只有location中用到proxy_pass、fastcgi_pass、scgi_pass命令时，request_body才有值。
log_format main $request_body # 我这里是简写
location ~ \.php$ {
root /usr/share/nginx/html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}打印response_body
shell
# web_server配置
log_format main $response_body # 我这里是简写
server {
listen 80;
server_name localhost;
lua_need_request_body on;
set $response_body "";
body_filter_by_lua '
local response_body = string.sub(ngx.arg[1],1,1000)
ngx.ctx.buffered = (ngx.ctx.buffered or "") .. response_body
if ngx.arg[2] then
ngx.var.response_body = ngx.ctx.buffered
end
';
location ~ \.php$ {
root /usr/local/nginx/html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}完整的Nginx日志配置
shell
log_format main escape=json '{ "@timestamp": "$time_local", '
'"remote_addr": "$remote_addr",'
'"remote_port": "$remote_port",'
'"scheme": "$scheme",'
'"request_uri": "$request_uri",'
'"request_method": "$request_method",'
'"request_time": "$request_time",'
'"request_length": "$request_length",'
'"response_status": "$status",'
'"body_bytes_sent": "$body_bytes_sent",'
'"http_referer": "$http_referer",'
'"http_user_agent": "$http_user_agent",'
'"http_x_forwarded_for": "$http_x_forwarded_for",'
'"upstream_addr": "$upstream_addr",'
'"upstream_response_time": "$upstream_response_time",'
'"request_body": "$request_body", '
'"response_body": "$response_body" }';注意事项
- request_body与response_body如果在上传文件或者下载文件时，body内容会很大，需要考虑对性能和存储的影响。
- 对于timestamp参数，可以不使用time_local而使用time_iso8601。
- 如果运维对于网络的性能有要求，可以考虑使用$tcpinfo_rtt, $tcpinfo_rttvar, $tcpinfo_snd_cwnd, $tcpinfo_rcv_space。
Nginx配置access.log日志格式以及日志按天分割
在Nginx的使用过程中，如果不对Nginx的日志做自定义的配置的话，那么默认的access.log日志默认就只有这么一个日志文件，随着系统使用时间越来越就，日志文件就会越来越大，而且默认的日志记录的格式也不方便分析。所以我们在实际使用Nginx的过程中需要对其日志做一些配置。 在http配置块中进行配置，也可以在每个server块中配置不同的access_log用以区分
shell
http{
# 定义一个变量来存储年月日
map $time_iso8601 $date {
default "date-not-found";
'~^(?<ymd>\d{4}-\d{2}-\d{2})' $ymd;
}
# 定义一个变量来存储时分秒
map $time_iso8601 $time {
default "time-not-found";
'~T(?<hms>\d{2}:\d{2}:\d{2})' $hms;
}
# 定义一个变量来存储毫秒 "msec": "1731929153.581"
map $msec $sss {
default "sss-not-found";
'~\.(?<s3>\d{3})$' $s3;
}
# 定义日志格式为json格式
log_format json_format '{
"timestamp": "$date $time.$sss",
"remote_addr": "$remote_addr",
"remote_user": "$remote_user",
"request": "$request",
"status": "$status",
"body_bytes_sent": "$body_bytes_sent",
"http_referer": "$http_referer",
"http_user_agent": "$http_user_agent",
"http_x_forwarded_for": "$http_x_forwarded_for"
},';
# 定义一个变量来存储年月,可用来作为access.log的日志文件名，按月自动分割日志
map $time_iso8601 $logmonth {
'~^(?<ym>\d{4}-\d{2})' $ym;
default 'date-not-found';
}
# 定义一个变量来存储年月日，可用来作为access.log的日志文件名，按天自动分割日志
map $time_iso8601 $logdate {
'~^(?<ymd>\d{4}-\d{2}-\d{2})' $ymd;
default 'date-not-found';
}
# 开启access.log日志，可设置日志名变量(这里选择上面的按天)以便分割日志；可设置日志格式，以便更直观分析日志
access_log logs/access-$logdate.log json_format;
}配置完需重启Nginx服务，如果发现没对应日志文件生成，看是否是 logs目录没有写入的权限，可以分配权限后再看一下。
分配权限：chmod 777 logs