| 知乎专栏 |
通过命令行改变日志的输出级别
java -jar app.jar --debug 在application.properties中配置 debug=true application.yml debug=true 相同的方式使能TRACE级别的日志 java -jar app.jar --trace application.properties trace=true application.yml trace=true
一般的日志需求可以通过配置 application.properties实现。
Spring Boot中 日志默认是输出到控制台的,这样是为了方便开发人员,但是在生产环境中应该输出到日志文件中。
配置如下
![]() | 提示 |
|---|---|
注意:这两个属性不能同时配置,只需要配置一个即可。 |
![]() | 提示 |
|---|---|
旧版本 logging.file,设置文件,可以是绝对路径,也可以是相对路径。如:logging.file=my.log logging.path,设置目录,如果 logging.file 没有设置,会在该目录下创建spring.log文件作为默认日志文件。 logging.file=target/spring.log #logging.path= 如果仍不能满足可以使用 logback.xml 配置日志。 logging.path=/tmp logging.config=classpath:logback.xml |
几种常见的日志级别由低到高分为:TRACE < DEBUG < INFO < WARN < ERROR < FATAL
显示所有DEBUG信息
logging.level.root=DEBUG
仅仅显示 springframework 调试信息
logging.level.org.springframework.web=DEBUG
仅仅显示 cn.netkiller.web.TestController 调试信息
private static final Logger log = LoggerFactory.getLogger(TestController.class); log.debug(message); logging.level.cn.netkiller.web.TestController=DEBUG
YAML 配置文件写法
logging:
level:
root: info
cn.netkiller.test: debug
cn.netkiller.sharding.MonthShardingAlgorithm: DEBUG
禁止控制台日志输出,同时将日志写入日志文件.
src/main/resources/application.properties
logging.file.path=/tmp logging.file.name=/tmp/spring.log logging.level.root=INFO logging.level.org.springframework.web=DEBUG logging.level.org.hibernate=ERROR
src/main/resources/logback.xml
$ cat src/main/resources/logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
</root>
</configuration>
使用 java -jar project-version-xxx.jar 启动后控制不会再输出日志
定制日志格式有两个配置
分别是控制台的输出格式和文件中的日志输出格式
举例
logging.pattern.console=%d{yyyy/MM/dd-HH:mm:ss} [%thread] %-5level %logger- %msg%n
logging.pattern.file=%d{yyyy/MM/dd-HH:mm} [%thread] %-5level %logger- %msg%n
格式说明
%d{HH:mm:ss.SSS} 日志输出时间
%thread 输出日志的进程名字,这在Web应用以及异步任务处理中很有用
%-5level 日志级别,并且使用5个字符靠左对齐
%logger 日志输出者的名字
%msg 日志消息
%n 平台的换行符
logging.logback.rollingpolicy.file-name-pattern=${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz
#logging.logback.rollingpolicy.max-file-size=1KB
#logging.logback.rollingpolicy.max-file-size=10MB
#logging.logback.rollingpolicy.max-history=7
#logging.logback.rollingpolicy.total-size-cap=100MB
参数说明
配置项 默认值 描述
logging.logback.rollingpolicy.file-name-pattern ${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz 日志文档的文件名格式
logging.logback.rollingpolicy.clean-history-on-start false 应用启动时是否清除以前的文档
logging.logback.rollingpolicy.max-file-size 10MB 存档前每个日志文件的最大大小
logging.logback.rollingpolicy.total-size-cap 0B 日志文件被删除之前,可以容纳最大大小。设置1GB则磁盘存储超过1GB日志后就会删除旧日志文件
logging.logback.rollingpolicy.max-history 7 日志文件的最大保存天数
logging.logback.rollingpolicy.file-name-pattern=${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz
logging.logback.rollingpolicy.max-file-size=10MB
logging.logback.rollingpolicy.clean-history-on-start=false
logging.logback.rollingpolicy.max-history=7
logging.logback.rollingpolicy.total-size-cap=0B
启动 Springboot 增加参数 -Dspring.output.ansi.enabled=ALWAYS
或者写入配置文件
spring.output.ansi.enabled=ALWAYS
日志的用法,首先开发中我们根据实际的需要打印不同级别的日志。
package cn.netkiller.web;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
@Controller
public class TestController {
private static final Logger log = LoggerFactory.getLogger(TestController.class);
@RequestMapping("/test/log")
@ResponseBody
public String log() {
String message = "Test";
log.debug(message);
log.info(message);
log.warn(message);
log.error(message);
log.trace(message);
return message;
}
}
然后通过application.properties配置那些需要显示,那些不需要,以及显示的级别是什么。
配置文件名默认是:logback-spring.xml,使用其他文件名通过下面配置项指定即可。
logging.config=classpath:logback.xml
基本配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yy-MMMM-dd HH:mm:ss:SSS} %5p %t %c{2}:%L - %m%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="stdout"/>
</root>
</configuration>
<logger name="cn.netkiller.controller"/>
<logger name="cn.netkiller.controller.HomeController" level="WARN" additivity="false">
<appender-ref ref="console"/>
</logger>
scan:当此属性设置为true时,配置文件如果发生改变,将会被重新加载,默认值为true。 scanPeriod:设置监测配置文件是否有修改的时间间隔,如果没有给出时间单位,默认单位是毫秒。当scan为true时,此属性生效。默认的时间间隔为1分钟。 debug:当此属性设置为true时,将打印出logback内部日志信息,实时查看logback运行状态。默认值为false。
每个logger都关联到logger上下文,默认上下文名称为“default”。但可以使用设置成其他名字,用于区分不同应用程序的记录。设置后可以通过%contextName来打印日志上下文名称。 <contextName>logback</contextName>
用来定义变量值的标签, 有两个属性,name和value;其中name的值是变量的名称,value的值时变量定义的值。通过定义的值会被插入到logger上下文中。定义变量后,可以使“${}”来使用变量。
<property name="log.path" value="/tmp" />
<encoder>表示对日志进行编码:
%d{HH: mm:ss.SSS}——日志输出时间
%thread——输出日志的进程名字,这在Web应用以及异步任务处理中很有用
%-5level——日志级别,并且使用5个字符靠左对齐
%logger{36}——日志输出者的名字
%msg——日志消息
%n——平台的换行符
上例中<fileNamePattern>${log.path}/logback.%d{yyyy-MM-dd}.log</fileNamePattern>定义了日志的切分方式——把每一天的日志归档到一个文件中,同理,可以使用%d{yyyy-MM-dd_HH-mm}来定义精确到分的日志切分方式。
<maxHistory>30</maxHistory>表示只保留最近30天的日志,以防止日志填满整个磁盘空间。
<totalSizeCap>1GB</totalSizeCap>用来指定日志文件的上限大小,例如设置为1GB的话,那么到了这个值,就会删除旧的日志。
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds" debug="false">
<contextName>logback</contextName>
<property name="log.path" value="target" />
<!--输出到控制台-->
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} %contextName [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<!--输出到文件-->
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.path}/spring.%d{yyyy-MM-dd}.log</fileNamePattern>
</rollingPolicy>
<encoder>
<pattern>%d{HH:mm:ss.SSS} %contextName [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="console" />
<appender-ref ref="file" />
</root>
</configuration>
按日期分割文件
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<appender name="dailyRollingFileAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>logs/spring.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<FileNamePattern>spring.%d{yyyy-MM-dd}.log</FileNamePattern>
<!-- keep 30 days' worth of history -->
<maxHistory>60</maxHistory>
</rollingPolicy>
<encoder>
<Pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg %n</Pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="dailyRollingFileAppender" />
</root>
</configuration>
通过级别分割日志将 info, error, debug 分割到指定文件中。
<configuration scan="true" scanPeriod="10 seconds">
<!-- 控制台日志输出-->
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d %p (%file:%line\)- %m%n</pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
<!-- info日志输出-->
<appender name="INFO_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<pattern>%d %p (%file:%line\)- %m%n</pattern>
<charset>UTF-8</charset>
</encoder>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<File>${LOG_PATH}/www.netkiller.cn.info.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/www.netkiller.cn.info-%d{yyyyMMdd}.log.%i
</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<maxHistory>30</maxHistory>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%msg%n
</Pattern>
</layout>
</appender>
<!-- debug 日志输出-->
<appender name="DEBUG_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<pattern>%d %p (%file:%line\)- %m%n</pattern>
<charset>UTF-8</charset>
</encoder>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>DEBUG</level>
</filter>
<File>${LOG_PATH}/www.netkiller.cn.debug.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/www.netkiller.cn.debug-%d{yyyyMMdd}.log.%i
</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<maxHistory>30</maxHistory>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%msg%n
</Pattern>
</layout>
</appender>
<!--error 日志输出配置 -->
<appender name="ERROR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<pattern>%d %p (%file:%line\)- %m%n</pattern>
<charset>UTF-8</charset>
</encoder>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>
<File>${LOG_PATH}/www.netkiller.cn.error.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/www.netkiller.cn.error-%d{yyyyMMdd}.log.%i</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<maxHistory>30</maxHistory>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%msg%n</Pattern>
</layout>
</appender>
<root level="DEBUG">
<!--
<appender-ref ref="STDOUT" />
<appender-ref ref="INFO_FILE" />
<appender-ref ref="ERROR_FILE" />
<appender-ref ref="DEBUG_FILE" />
-->
<appender-ref ref="ERROR_FILE" />
<appender-ref ref="INFO_FILE" />
<appender-ref ref="DEBUG_FILE" />
</root>
</configuration>
将特定日志输出保存到指定位置
package cn.netkiller;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;
public class Application {
public static void main(String[] args) {
final Logger logger = LoggerFactory.getLogger(Application.class);
Marker notifyAdmin = MarkerFactory.getMarker("netkiller");
logger.info("AAAAAAAAA");
logger.info(notifyAdmin, "BBBBBBBBB");
logger.error(notifyAdmin, "This is a serious an error requiring the admin's attention", new Exception("Just testing"));
}
}
匹配到 marker 的日志才输出,通过 RollingFileAppender 可以保存到指定文件。
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
<marker>netkiller</marker>
</evaluator>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss} %-4relative [%thread] %-5level %logger{35} : %msg %n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss} %-4relative [%thread] %-5level %logger{35} : %msg %n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
每个 userId 生成一个日志文件
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<property name="log.pattern" value="%d{yyyy-MM-dd HH:mm:ss} - [%25.25(%thread)] - [%-5level] - %-30.30(%logger{30}) : %msg%n" />
<appender name="siftingAppender" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<key>userId</key>
<defaultValue>unknown</defaultValue>
</discriminator>
<sift>
<appender name="${userId}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.path}/${userId}.%d{yyyy-MM-dd}.log</fileNamePattern>
</rollingPolicy>
<encoder>
<pattern>${log.pattern}</pattern>
</encoder>
</appender>
</sift>
</appender>
<root level="INFO">
<appender-ref ref="siftingAppender" />
</root>
</configuration>
package cn.netkiller.log;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
public class LogTest {
private static final Logger logger = LoggerFactory.getLogger(LogTest.class);
public LogTest() {
// TODO Auto-generated constructor stub
}
public static void main(String[] args) {
MDC.put("userId","0001");
logger.info("0001用户");
MDC.clear();
MDC.put("userId","0002");
logger.info("0002用户");
MDC.clear();
}
}
https://github.com/logfellow/logstash-logback-encoder/
配置 logstash 增加 tcp 接收输入端。
input {
tcp {
mode => "server"
host => "127.0.0.1"
port => 4567
codec => json_lines
}
}
Maven 配置文件 pom.xml 中添加
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>7.2</version>
</dependency>
然后再resources添加logback.xml文件
<?xml version="1.0" encoding="UTF-8"?> <configuration scan="true" scanPeriod="60 seconds" debug="true"> <include resource="org/springframework/boot/logging/logback/defaults.xml" /> <include resource="org/springframework/boot/logging/logback/console-appender.xml" /> <include resource="org/springframework/boot/logging/logback/file-appender.xml" /> <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>127.0.0.1:4567</destination> <encoder class="net.logstash.logback.encoder.LogstashEncoder"> <providers> <timestamp /> <logLevel /> <threadName /> <loggerName /> <message /> </providers> </encoder> </appender> <root level="info"> <appender-ref ref="CONSOLE" /> <appender-ref ref="logstash" /> </root> </configuration>
[root@netkiller ~]# cat /etc/logstash/conf.d/file.conf
input {
tcp {
port => 4567
codec => json_lines
}
}
filter {
ruby {
code => "event.set('datetime', event.get('@timestamp').time.localtime.strftime('%Y-%m-%d %H:%M:%S'))"
}
}
output {
if "finance" in [tags] {
file {
path => "/opt/log/%{app}.finance.%{+yyyy}-%{+MM}-%{+dd}.log"
codec => line { format => "[%{datetime}] %{level} %{message} %{tags}"}
}
} else if "market" in [tags] {
file {
path => "/opt/log/%{app}.market.%{+yyyy}-%{+MM}-%{+dd}.log"
codec => line { format => "[%{datetime}] %{level} %{message} %{tags}"}
}
} else {
file {
path => "/opt/log/%{app}.unknow.%{+yyyy}-%{+MM}-%{+dd}.log"
codec => line { format => "[%{datetime}] %{level} %{message} %{tags}"}
}
}
}
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="false" scanPeriod="60 seconds" debug="false">
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<logger name="org.springframework.web" level="INFO" />
<logger name="org.springboot.sample" level="TRACE" />
<property name="log.pattern" value="%date{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{35}.%method: %msg%n" />
<springProperty scope="context" name="app" source="spring.application.name" defaultValue="spring-boot-fusion" />
<property name="log.path" value="/tmp" />
<appender name="siftingAppender" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<key>userId</key>
<defaultValue>unknown</defaultValue>
</discriminator>
<sift>
<appender name="${userId}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.path}/${userId}.%d{yyyy-MM-dd}.log</fileNamePattern>
</rollingPolicy>
<encoder>
<pattern>${log.pattern}</pattern>
</encoder>
</appender>
</sift>
</appender>
<springProfile name="prod">
<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>172.18.200.10:4567</destination>
<keepAliveDuration>5 minutes</keepAliveDuration>
<reconnectionDelay>3 second</reconnectionDelay>
<writeBufferSize>8192</writeBufferSize>
<includeCallerData>true</includeCallerData>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<shortenedLoggerNameLength>36</shortenedLoggerNameLength>
<timestampPattern>yyyy-MM-dd HH:mm:ss.Asia/Shanghai</timestampPattern>
<timeZone>Asia/Shanghai</timeZone>
<fieldNames>
<timestamp>@timestamp</timestamp>
<version>@version</version>
<message>message</message>
<logger>logger_name</logger>
<!-- <thread>thread_name</thread> -->
<level>level</level>
<thread>[ignore]</thread>
<levelValue>[ignore]</levelValue>
</fieldNames>
</encoder>
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
<marker>finance</marker>
<marker>market</marker>
<marker>customer</marker>
</evaluator>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
</appender>
</springProfile>
<root level="info">
<springProfile name="dev">
<appender-ref ref="CONSOLE" />
</springProfile>
<springProfile name="test">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</springProfile>
<springProfile name="prod">
<appender-ref ref="CONSOLE" />
<appender-ref ref="logstash" />
</springProfile>
</root>
</configuration>
package cn.netkiller.controller;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import cn.netkiller.LogMarker;
import lombok.extern.slf4j.Slf4j;
@RestController
@Slf4j
public class HomeController {
private static final Logger logger = LoggerFactory.getLogger(HomeController.class);
public HomeController() {
// TODO Auto-generated constructor stub
}
@GetMapping("/")
public String index() {
Marker finance = MarkerFactory.getMarker(LogMarker.finance.toString());
Marker customer = MarkerFactory.getMarker(LogMarker.customer.toString());
Marker market = MarkerFactory.getMarker(LogMarker.market.toString());
logger.info("AAAAAAAAA");
logger.info(finance, "test");
logger.info(finance, "finance");
logger.info(customer, "customer");
logger.info(market, "market");
MDC.put("userId", "0001");
logger.info("0001用户");
MDC.clear();
MDC.put("userId", "0002");
logger.info("0002用户");
MDC.clear();
return "Hello world!!!";
}
}
<dependency> <groupId>org.fluentd</groupId> <artifactId>fluent-logger</artifactId> <version>0.3.4</version> </dependency> <dependency> <groupId>com.sndyuk</groupId> <artifactId>logback-more-appenders</artifactId> <version>1.8.7</version> </dependency>
dnf install -y fluent-bit
启动 fluent-bit
[root@netkiller ~]# fluent-bit -i forward -o stdout Fluent Bit v1.9.7 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2022/09/24 23:25:25] [ info] [fluent bit] version=1.9.7, commit=, pid=1191240 [2022/09/24 23:25:25] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2022/09/24 23:25:25] [ info] [cmetrics] version=0.3.5 [2022/09/24 23:25:25] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224 [2022/09/24 23:25:25] [ info] [sp] stream processor started [2022/09/24 23:25:25] [ info] [output:stdout:stdout.0] worker #0 started
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date - %level - [%thread] - %logger - [%file:%line] - %msg%n</pattern>
</encoder>
</appender>
<appender name="FLUENT" class="ch.qos.logback.more.appenders.DataFluentAppender">
<tag>development</tag>
<label>normal</label>
<remoteHost>localhost</remoteHost>
<port>24224</port>
<maxQueueSize>20</maxQueueSize>
</appender>
<logger name="cn.netkiller.log" level="DEBUG"/>
<root level="DEBUG">
<appender-ref ref="STDOUT" />
<appender-ref ref="FLUENT" />
</root>
</configuration>
[0] development.normal: [1664033186.000000000, {"level"=>"INFO", "logger"=>"cn.netkiller.Application", "thread"=>"main", "message"=>"Starting Application using Java 18 on MacBook-Pro-Neo.local with PID 85696 (/Users/neo/workspace/bottleneck/target/classes started by neo in /Users/neo/workspace/bottleneck)"}]
[1] development.normal: [1664033186.000000000, {"level"=>"INFO", "logger"=>"cn.netkiller.Application", "thread"=>"main", "message"=>"The following 1 profile is active: "prod""}]
[0] development.normal: [1664033187.000000000, {"level"=>"INFO", "logger"=>"org.springframework.data.repository.config.RepositoryConfigurationDelegate", "thread"=>"main", "message"=>"Multiple Spring Data modules found, entering strict repository configuration mode"}]
[1] development.normal: [1664033187.000000000, {"level"=>"INFO", "logger"=>"org.springframework.data.repository.config.RepositoryConfigurationDelegate", "thread"=>"main", "message"=>"Bootstrapping Spring Data Redis repositories in DEFAULT mode."}]
[2] development.normal: [1664033187.000000000, {"level"=>"INFO", "logger"=>"org.springframework.data.repository.config.RepositoryConfigurationDelegate", "thread"=>"main", "message"=>"Finished Spring Data repository scanning in 6 ms. Found 0 Redis repository interfaces."}]
[0] development.normal: [1664033188.000000000, {"level"=>"INFO", "logger"=>"org.springframework.boot.web.embedded.tomcat.TomcatWebServer", "thread"=>"main", "message"=>"Tomcat initialized with port(s): 8080 (http)"}]
[1] development.normal: [1664033188.000000000, {"level"=>"INFO", "logger"=>"org.apache.catalina.core.StandardService", "thread"=>"main", "message"=>"Starting service [Tomcat]"}]
[2] development.normal: [1664033188.000000000, {"level"=>"INFO", "logger"=>"org.apache.catalina.core.StandardEngine", "thread"=>"main", "message"=>"Starting Servlet engine: [Apache Tomcat/9.0.65]"}]
[3] development.normal: [1664033188.000000000, {"level"=>"INFO", "logger"=>"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]", "thread"=>"main", "message"=>"Initializing Spring embedded WebApplicationContext"}]
[4] development.normal: [1664033188.000000000, {"level"=>"INFO", "logger"=>"org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext", "thread"=>"main", "message"=>"Root WebApplicationContext: initialization completed in 2133 ms"}]
[0] development.normal: [1664033189.000000000, {"level"=>"INFO", "logger"=>"org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver", "thread"=>"main", "message"=>"Exposing 14 endpoint(s) beneath base path '/actuator'"}]
[0] development.normal: [1664033189.000000000, {"level"=>"INFO", "logger"=>"org.springframework.boot.web.embedded.tomcat.TomcatWebServer", "thread"=>"main", "message"=>"Tomcat started on port(s): 8080 (http) with context path ''"}]
[1] development.normal: [1664033189.000000000, {"level"=>"INFO", "logger"=>"cn.netkiller.Application", "thread"=>"main", "message"=>"Started Application in 4.224 seconds (JVM running for 4.918)"}]
https://loki4j.github.io/loki-logback-appender/
<dependency>
<groupId>com.github.loki4j</groupId>
<artifactId>loki-logback-appender</artifactId>
<version>1.3.2</version>
</dependency>
<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
<http>
<url>http://localhost:3100/loki/api/v1/push</url>
</http>
<format>
<label>
<pattern>app=my-app,host=${HOSTNAME},level=%level</pattern>
</label>
<message>
<pattern>l=%level h=${HOSTNAME} c=%logger{20} t=%thread | %msg %ex</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<root level="DEBUG">
<appender-ref ref="LOKI" />
</root>
https://logging.paluch.biz/examples/log4j-2.x.html
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <!-- Exclude the Tomcat dependency --> <exclusions> <exclusion> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-tomcat</artifactId> </exclusion> <!-- 禁用 logback --> <exclusion> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-logging</artifactId> </exclusion> </exclusions> </dependency> <!-- 添加Log4j2 依赖 --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-log4j2</artifactId> </dependency> <dependency> <groupId>biz.paluch.logging</groupId> <artifactId>logstash-gelf</artifactId> <version>1.15.0</version> </dependency>
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<properties>
<property name="log.pattern">[%d{yyyy-MM-dd HH:mm:ss}] [${hostName}] [%p] [%t] %l - %m%n</property>
<property name="log.dir">/tmp/logs</property>
<property name="log.level">info</property>
</properties>
<Appenders>
<!-- 控制台 -->
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${log.pattern}" />
</Console>
<!-- INFO级别日志 -->
<RollingFile name="RollingFileInfo" fileName="${log.dir}/info.log" filePattern="${log.dir}/info.%d{yyyy-MM-dd}.log">
<Filters>
<ThresholdFilter level="INFO" />
<ThresholdFilter level="WARN" onMatch="DENY" onMismatch="NEUTRAL" />
</Filters>
<PatternLayout pattern="${log.pattern}" />
<Policies>
<TimeBasedTriggeringPolicy interval="1" modulate="false" />
</Policies>
</RollingFile>
<!-- WARN级别日志 -->
<RollingFile name="RollingFileWarn" fileName="${log.dir}/warn.log" filePattern="${log.dir}/warn.%d{yyyy-MM-dd}.log">
<Filters>
<ThresholdFilter level="WARN" />
<ThresholdFilter level="ERROR" onMatch="DENY" onMismatch="NEUTRAL" />
</Filters>
<PatternLayout pattern="${log.pattern}" />
<Policies>
<TimeBasedTriggeringPolicy interval="1" modulate="false" />
</Policies>
</RollingFile>
<!-- ERROR级别日志 -->
<RollingFile name="RollingFileError" fileName="${log.dir}/error.log" filePattern="${log.dir}/error.%d{yyyy-MM-dd}.log">
<Filters>
<ThresholdFilter level="ERROR" />
</Filters>
<PatternLayout pattern="${log.pattern}" />
<Policies>
<TimeBasedTriggeringPolicy interval="1" modulate="false" />
</Policies>
</RollingFile>
<Gelf name="Gelf" host="udp:172.18.200.10" port="12201" version="1.1" extractStackTrace="true" filterStackTrace="true" mdcProfiling="true" includeFullMdc="true" maximumMessageSize="8192" originHost="%host{fqdn}">
<Field name="timestamp" pattern="%d{yyyy-MM-dd HH:mm:ss.SSS}" />
<Field name="logger" pattern="%logger" />
<Field name="level" pattern="%level" />
<Field name="class" pattern="%C{1}" />
<Field name="method" pattern="%M" />
<Field name="line" pattern="%L" />
<Field name="marker" pattern="%marker" />
<Filters>
<MarkerFilter marker="finance" onMatch="ACCEPT" onMismatch="NEUTRAL" />
<MarkerFilter marker="market" onMatch="ACCEPT" onMismatch="DENY" />
</Filters>
</Gelf>
</Appenders>
<Loggers>
<Root level="${sys:log.level}">
<AppenderRef ref="Console" />
<AppenderRef ref="Gelf" />
<!-- <AppenderRef ref="RollingFileInfo" /> <AppenderRef ref="RollingFileWarn" /> <AppenderRef ref="RollingFileError" /> -->
</Root>
</Loggers>
</Configuration>
@GetMapping("/log")
public String log() {
Marker finance = MarkerFactory.getMarker(LogMarker.finance.toString());
Marker customer = MarkerFactory.getMarker(LogMarker.customer.toString());
Marker market = MarkerFactory.getMarker(LogMarker.market.toString());
logger.info("常规日志");
logger.info(finance, "test");
logger.info(finance, "finance");
logger.info(customer, "customer");
logger.info(market, "market");
return "OK!!!\r\n";
}
@GetMapping("/log/marker")
public String marker(@RequestParam("marker") String marker, @RequestParam("msg") String msg) {
logger.info(MarkerFactory.getMarker(marker), msg);
msg += "\r\n";
return msg;
}
[root@netkiller log]# cat /etc/logstash/conf.d/file.conf
input {
tcp {
port => 4567
codec => json_lines
}
gelf {
port => 12201
use_udp => true
#use_tcp => true
}
}
filter {
ruby {
code => "event.set('datetime', event.get('@timestamp').time.localtime.strftime('%Y-%m-%d %H:%M:%S'))"
}
}
output {
file {
path => "/opt/log/%{marker}.%{+yyyy}-%{+MM}-%{+dd}.log"
codec => line { format => "[%{datetime}] %{level} %{message}"}
}
file {
path => "/opt/log/origin.%{+yyyy}-%{+MM}-%{+dd}.log.gz"
codec => json_lines
gzip => true
}
}
[root@netkiller log]# ls
finance.2022-11-16.log market.2022-11-16.log origin.2022-11-16.log.gz
[root@netkiller log]# cat finance.2022-11-16.log
[2022-11-16 15:02:36] INFO test
[2022-11-16 15:02:36] INFO finance
[2022-11-16 15:21:34] INFO test
[2022-11-16 15:21:34] INFO finance
[root@netkiller log]# cat market.2022-11-16.log
[2022-11-16 15:02:36] INFO market
[2022-11-16 15:21:34] INFO market
[root@netkiller log]# zcat origin.2022-11-16.log.gz |jq
{
"datetime": "2022-11-16 15:21:34",
"timestamp": "2022-11-16 15:21:34.185",
"message": "market",
"host": "macbook-pro-neo.local",
"level": "INFO",
"line": 53,
"@version": "1",
"@timestamp": "2022-11-16T07:21:34.185Z",
"marker": "market",
"logger": "cn.netkiller.controller.HomeController",
"version": "1.1",
"method": "log",
"class": "HomeController",
"source_host": "172.18.5.142",
"facility": "logstash-gelf"
}
{
"datetime": "2022-11-16 15:21:34",
"timestamp": "2022-11-16 15:21:34.143",
"message": "test",
"host": "macbook-pro-neo.local",
"level": "INFO",
"line": 49,
"@version": "1",
"@timestamp": "2022-11-16T07:21:34.143Z",
"marker": "finance",
"logger": "cn.netkiller.controller.HomeController",
"version": "1.1",
"method": "log",
"class": "HomeController",
"source_host": "172.18.5.142",
"facility": "logstash-gelf"
}
{
"datetime": "2022-11-16 15:21:34",
"timestamp": "2022-11-16 15:21:34.184",
"message": "finance",
"host": "macbook-pro-neo.local",
"level": "INFO",
"line": 51,
"@version": "1",
"@timestamp": "2022-11-16T07:21:34.184Z",
"marker": "finance",
"logger": "cn.netkiller.controller.HomeController",
"version": "1.1",
"method": "log",
"class": "HomeController",
"source_host": "172.18.5.142",
"facility": "logstash-gelf"
}
方案一
logging:
config: classpath:log4j2-${spring.profiles.active}.xml
方案二
@SpringBootApplication
public class Application implements CommandLineRunner {
@Autowired
private Environment env;
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
@Override
public void run(String... param) {
if (Arrays.asList(env.getActiveProfiles()).contains("dev")) {
Configurator.initialize(null, "/path/to/log4j2-dev.xml");
} else {
Configurator.initialize(null, "/path/to/log4j2.xml");
}
}
}
#日志配置 无特殊需求无需更改
logging:
config: classpath:log4j2.xml
level:
root: INFO
javax.activation: info
org.apache.catalina: INFO
org.apache.commons.beanutils.converters: INFO
org.apache.coyote.http11.Http11Processor: INFO
org.apache.http: INFO
org.apache.tomcat: INFO
org.springframework: INFO
com.chinamobile.cmss.bdpaas.resource.monitor: DEBUG
<Loggers>
<Logger name="com.ensd.service.sharding.MonthShardingAlgorithm" level="ERROR" />
<Root level="${sys:log.level}">
<AppenderRef ref="Console"/>
<AppenderRef ref="File"/>
<AppenderRef ref="Logstash"/>
</Root>
</Loggers>
引入依赖
<dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-spring-boot</artifactId> </dependency>
例子
<Configuration name="ConfigTest" status="ERROR" monitorInterval="5">
<properties>
<property name="applicationName">${spring:spring.application.name}</property>
</properties>
<Appenders>
<SpringProfile name="dev | staging">
<Console name="Out">
<PatternLayout pattern="%m%n"/>
</Console>
</SpringProfile>
<SpringProfile name="prod">
<List name="Out">
</List>
</SpringProfile>
</Appenders>
<Loggers>
<Logger name="org.apache.test" level="trace" additivity="false">
<AppenderRef ref="Out"/>
</Logger>
<Root level="error">
<AppenderRef ref="Out"/>
</Root>
</Loggers>
</Configuration>
读取方法 ${spring:spring.profiles.active}
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<properties>
<property name="log.pattern">[%d{yyyy-MM-dd HH:mm:ss}] [${hostName}] [%p] [%t] %l - %m%n</property>
<property name="log.home">/tmp/logs</property>
<property name="log.level">info</property>
</properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${log.pattern}" />
</Console>
<Gelf name="dev" host="udp:172.18.200.10" port="12201" version="1.1" extractStackTrace="true" filterStackTrace="true" mdcProfiling="true" includeFullMdc="true" maximumMessageSize="8192" originHost="%host{fqdn}">
<Field name="timestamp" pattern="%d{yyyy-MM-dd HH:mm:ss.SSS}" />
<Field name="logger" pattern="%logger" />
<Field name="level" pattern="%level" />
<Field name="class" pattern="%C{1}" />
<Field name="method" pattern="%M" />
<Field name="line" pattern="%L" />
<Field name="marker" pattern="%marker" />
<Filters>
<MarkerFilter marker="finance" onMatch="ACCEPT" onMismatch="NEUTRAL" />
<MarkerFilter marker="market" onMatch="ACCEPT" onMismatch="DENY" />
</Filters>
</Gelf>
</Appenders>
<Loggers>
<Root level="${sys:log.level}">
<AppenderRef ref="Console" />
<AppenderRef ref="${spring:spring.profiles.active:-dev}" />
</Root>
</Loggers>
</Configuration>
Spring 2.1.4 无法获取配置,解决方法使用 sys,同时启动的时候增加系统配置项 java -Dspring.application.name=netkiller -Dspring.profiles.active=dev -jar netkiller.jar
<property name="service">${sys:spring.application.name}</property>
<property name="environment">${sys:spring.profiles.active}</property>
将 ERROR 和 WARN 级别的日志发送到钉钉群
[root@netkiller ~]# cat /etc/logstash/conf.d/file.conf
input {
tcp {
port => 4567
codec => json_lines
}
gelf {
port => 12201
use_udp => true
#use_tcp => true
}
}
filter {
ruby {
code => "event.set('datetime', event.get('@timestamp').time.localtime.strftime('%Y-%m-%d %H:%M:%S'))"
}
}
output {
file {
path => "/opt/log/%{marker}.%{+yyyy}-%{+MM}-%{+dd}.log"
codec => line { format => "[%{datetime}] %{level} %{message}"}
}
file {
path => "/opt/log/origin.%{+yyyy}-%{+MM}-%{+dd}.log.gz"
codec => json_lines
gzip => true
}
if "ERROR" in [level] or "WARN" in [level] {
http {
url => "https://oapi.dingtalk.com/robot/send?access_token=56c27cb734a56cf549f6977ecc2761c4a16473db02d9d2881d008f9a239ba3e0"
http_method => "post"
content_type => "application/json; charset=utf-8"
format => "message"
message => '{"msgtype":"text","text":{"content":"Monitor: %{host} - %{message}"}}'
}
}
}
neo@MacBook-Pro-M2 ~ % cat workspace/bottleneck/src/main/java/cn/netkiller/Application.java
package cn.netkiller;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MarkerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
@EnableDiscoveryClient
@SpringBootApplication
public class Application {
private static final Logger logger = LoggerFactory.getLogger(Application.class);
@PostConstruct
public void init() {
System.out.printf("==================== init ====================");
logger.warn(MarkerFactory.getMarker("finance"), "XXX 系统启动");
}
@PreDestroy
public void destroy() {
System.out.printf("==================== destroy ====================");
logger.error(MarkerFactory.getMarker("finance"), "XXX 系统销毁");
}
public static void main(String[] args) {
System.out.println("Netkiller bottleneck tool!");
SpringApplication.run(Application.class, args);
}
}
@PostConstruct 可以监控 启动情况
@PreDestroy 可以监控 退出情况
将 Spring boot 日志写入 ELK 有多种实现方式,这里仅提供三种方案:
Spring boot -> logback -> Tcp/IP -> logstash -> elasticsearch
这种方式实现非常方便不需要而外包或者软件Spring boot -> logback -> Redis -> logstash -> elasticsearch
利用 Redis 提供的发布订阅功能将日志投递到 elasticsearchSpring boot -> logback -> Kafka -> logstash -> elasticsearch
Kafka 方法适合大数据的情况。logstash 配置
input {
tcp {
host => "172.16.1.16"
port => 9250
mode => "server"
tags => ["tags"]
codec => json_lines //可能需要更新logstash插件
}
}
output {
stdout{codec =>rubydebug}
elasticsearch {
hosts => ["localhost:9200"] //这块配置需要带端口号
flush_size => 1000
}
}
Spring boot logback.xml 配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property resource="properties/logback-variables.properties" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder charset="UTF-8">
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n
</pattern>
</encoder>
</appender>
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>172.16.1.16:9250</destination>
<encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<!--<appender name="async" class="ch.qos.logback.classic.AsyncAppender">-->
<!--<appender-ref ref="stash" />-->
<!--</appender>-->
<root level="info"> <!-- 设置日志级别 -->
<appender-ref ref="STDOUT" />
<appender-ref ref="LOGSTASH" />
</root>
</configuration>
https://github.com/kmtong/logback-redis-appender
Maven pom.xml 增加 Logback Redis 依赖
<!-- https://mvnrepository.com/artifact/com.cwbase/logback-redis-appender -->
<dependency>
<groupId>com.cwbase</groupId>
<artifactId>logback-redis-appender</artifactId>
<version>1.1.5</version>
</dependency>
Spring boot logback.xml 配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<property name="type.name" value="test" />
<appender name="LOGSTASH" class="com.cwbase.logback.RedisAppender">
<source>spring-application</source>
<type>${type.name}</type>
<host>localhost</host>
<key>logstash:redis</key>
<tags>test-2</tags>
<mdc>true</mdc>
<location>true</location>
<callerStackIndex>0</callerStackIndex>
<!--additionalField添加附加字段 用于head插件显示 -->
<additionalField>
<key>MyKey</key>
<value>MyValue</value>
</additionalField>
<additionalField>
<key>MySecondKey</key>
<value>MyOtherValue</value>
</additionalField>
</appender>
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="LOGSTASH" />
</root>
</configuration>
logstash 配置
input {
redis {
host => 'localhost'
data_type => 'list'
port => "6379"
key => 'logstash:redis' #自定义
type => 'redis-input' #自定义
}
}
output {
elasticsearch {
host => "localhost"
codec => "json"
protocol => "http"
}
}