使用filebeat监听异常日志发送redis

Joe's blog / 2023-08-26 / 原文

使用场景： 自定义监听日志关键字符，第一时间发现生产问题，实测从服务打出异常日志到redis监听到日志延迟在5s左右

适用于：服务机器数有限的情况，目前全部采用手动部署的方式，

一、在目标机器上部署filebeat

官网下载filebeat https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.9.1-linux-x86_64.tar.gz
上传至服务器 /usr/local/filebeat
tarf -zx 解压

编辑/usr/local/filebeat/ 下filebeat.yml 配置文件

  # =============日志输入配置=================
  filebeat.inputs:

  - type: log
    # Unique ID among all inputs, an ID is required.
    id: my-filestream-id
    # Change to true to enable this input configuration.
    enabled: true
    harvester_limit: 1
    # 配置输入过滤条件
    # include_lines: ["/mq/start"]
    paths:
      - /root/java/logs/all.log
  # =============日志输出配置=================

  output.redis:
    hosts: ["10.6.133.65:6379"]
    password: pass1w22ord12543!#
    db: 3
    # 发送至redis list  也可选用发布订阅模式
    datatype: list
    # 默认redis key 必须配置，否则会默认为 filebeat
    key: appLog
    # 配置条件key
    keys:
      - key: "mqStart_list" 
        when.contains:
          message: "/mq/start"
      - key: "delete_list"  
        when.contains:
          message: "数据删除消息"

启动filebeat
```
./filebeat $
```

二、配置redis日志监听

springboot中代码如下

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestTemplate;

import javax.annotation.PostConstruct;
import java.util.concurrent.TimeUnit;

@Component
@Slf4j
public class RedisListListener {

    @Autowired
    private StringRedisTemplate stringRedisTemplate;

    @Autowired
    RestTemplate restTemplate;

    @PostConstruct
    public void startListening() {
        Thread thread = new Thread(() -> {
            while (true) {
                String mqStart_list = null;
                try {
                    mqStart_list = stringRedisTemplate.opsForList().rightPop("mqStart_list", 30, TimeUnit.SECONDS);
                    if (mqStart_list != null) {
                        log.info("监听到redisList新增：{}", mqStart_list);
                        // todo  发送告警

                    }
                } catch (Exception ignored) {
                }
            }
        });
        thread.start();
    }
}