Skip to content

Instantly share code, notes, and snippets.

@icyleaf
Last active November 29, 2024 16:21
Show Gist options
  • Save icyleaf/e98093f673b4b2850226db582447175a to your computer and use it in GitHub Desktop.
Save icyleaf/e98093f673b4b2850226db582447175a to your computer and use it in GitHub Desktop.
mosdns v4.5.3+ config file

mosdns config

个人配置不断摸索和更新

Dashboard

  1. 随着 4.2 版本发布改用 prometheus 库,river_leaves 撸了一个 Grafana 的面板
  2. 我在 5.x INFO log 的基础上通过 vector 二次解析并接入了 loki 后整合到 Grafana 面板 (下图)
  3. 全套配置教程 https://icyleaf.com/2023/08/using-vector-transform-mosdns-logging-to-grafana-via-loki/

iShot_2023-08-07_11 07 38

声明

  1. v4 版本基于 https://jasper1024.com/jasper/20211223034622/ 更新为 4.0 配置格式优化
  2. v5 版本直接采用 sbwmlluci-app-mosdns 内置
  3. 官方文档:https://irine-sistiana.gitbook.io/mosdns-wiki/
  4. geosite.dat, geoip.dat 数据源: https://github.com/Loyalsoldier/v2ray-rules-dat

其他参考

# 日志设置
log:
level: info
file: "/var/log/mosdns.log"
# 数据源设置
data_providers:
- tag: geosite
file: "/usr/share/v2ray/geosite.dat"
auto_reload: true
- tag: geoip
file: "/usr/share/v2ray/geoip.dat"
auto_reload: true
plugins:
################ 匹配器 #################
# 查询 - CN 域名
- tag: query_cn
type: query_matcher
args:
domain:
- "provider:geosite:cn"
# 查询 - GFW
- tag: query_gfw
type: query_matcher
args:
domain:
- "provider:geosite:gfw"
# 查询 - 非 CN 域名
- tag: query_notcn
type: query_matcher
args:
domain:
- "provider:geosite:geolocation-!cn"
# 查询 - ADs
- tag: query_ad
type: query_matcher
args:
domain:
- "provider:geosite:category-ads-all"
# 返回 - CN IP
- tag: response_cnip
type: response_matcher
args:
ip:
- "provider:geoip:cn"
################ DNS #################
- tag: forward_local
type: fast_forward
args:
upstream:
- addr: 202.106.195.68
- addr: 202.106.46.151
- tag: forward_remote
type: fast_forward
args:
upstream:
- addr: tls://8.8.4.4
- addr: tls://8.8.8.8
# 阿里 DNS
- tag: ali
type: fast_forward
args:
upstream:
- addr: "https://dns.alidns.com/dns-query"
dial_addr: "223.5.5.5"
enable_pipeline: true
idle_timeout: 30
trusted: true
- addr: "tls://dns.alidns.com"
dial_addr: "223.5.5.5"
trusted: true
enable_pipeline: true
# DNSPOD DNS
- tag: dnspod
type: fast_forward
args:
upstream:
- addr: "https://doh.pub/dns-query"
dial_addr: "1.12.12.12"
idle_timeout: 30
trusted: true
- addr: "https://doh.pub/dns-query"
dial_addr: "120.53.53.53"
idle_timeout: 30
trusted: true
# 本地 DNS 备用
- tag: localdns
type: forward
args:
upstream:
- addr: "127.0.0.1:53"
# Google DNS
- tag: google
type: fast_forward
args:
upstream:
- addr: "udp://8.8.8.8"
- addr: "https://dns.google/dns-query"
dial_addr: "8.8.8.8:443"
idle_timeout: 30
trusted: true
# Cloudflare DNS
- tag: cloudflare
type: fast_forward
args:
upstream:
- addr: "https://cloudflare-dns.com/dns-query"
dial_addr: "1.1.1.1:443"
idle_timeout: 30
trusted: true
- tag: google_h3
type: "fast_forward"
args:
# 上游服务器。至少要配置一个。可配置多个,会并发请求全部服务器。
upstream:
- addr: "https://dns.google/dns-query"
dial_addr: "2001:4860:4860::8844"
trusted: true
enable_http3: true
- addr: "https://dns.google/dns-query"
dial_addr: "8.8.4.4"
trusted: true
enable_http3: true
- addr: "https://dns.google/dns-query"
dial_addr: "2001:4860:4860::8888"
trusted: true
enable_http3: true
- addr: "https://dns.google/dns-query"
dial_addr: "8.8.8.8"
trusted: true
enable_http3: true
- tag: cloudflare_h3
type: "fast_forward"
args:
upstream:
- addr: "https://cloudflare-dns.com/dns-query"
dial_addr: "2606:4700:4700::1001"
trusted: true
enable_http3: true
- addr: "https://cloudflare-dns.com/dns-query"
dial_addr: "1.0.0.1"
trusted: true
enable_http3: true
- addr: "https://cloudflare-dns.com/dns-query"
dial_addr: "2606:4700:4700::1111"
trusted: true
enable_http3: true
- addr: "https://cloudflare-dns.com/dns-query"
dial_addr: "1.1.1.1"
trusted: true
enable_http3: true
############## 附属功能 ###############
# 内存缓存
- tag: mem_cache
type: cache
args:
size: 1024
lazy_cache_ttl: 86400
lazy_cache_reply_ttl: 30
cache_everything: true
# 修改应答 ttl
- tag: modify_ttl
type: ttl
args:
minimal_ttl: 300
maximum_ttl: 3600
- tag: set_edns0_bufsize
type: bufsize
args:
size: 1232
- tag: set_edns0_client_subnet
type: ecs
args:
auto: true
mask4: 24
mask6: 56
################ 序列 #################
# local 序列
- tag: local
type: sequence
args:
exec:
- parallel:
- - ali
- - dnspod
# - primary:
# - ali
# secondary:
# - forward_local
# fast_fallback: 200 # 这里建议设置成 primary 服务器正常延时的 2~5 倍 单位: 毫秒。
# always_standby: true
# remote 序列
- tag: remote
type: sequence
args:
exec:
- parallel:
- - google
- - cloudflare
# - - google_h3
# - - cloudflare_h3
# - primary:
# - cloudflare
# secondary:
# - forward_remote
# fast_fallback: 200 # 这里建议设置成 primary 服务器正常延时的 2~5 倍 单位: 毫秒。
# always_standby: true
# 核心序列
- tag: core
type: sequence
args:
exec:
- if: query_ad # 屏蔽广告域名
exec:
- _new_nxdomain_response
- _return
- if: query_cn # CN 域名
exec:
- _pad_query
- local
- if: "response_cnip" # 结果是 cnip
exec:
- _return # 结束
- if: query_notcn # 已知的非 cn 域名
exec:
- _prefer_ipv4
- _pad_query
- remote
- if: "!response_cnip" # 结果是 非cnip
exec:
- _return # 结束
- primary: #其他所有情况
- _prefer_ipv4
- _pad_query
- remote
secondary:
- _prefer_ipv4
- _pad_query
- local
fast_fallback: 400 # 这里建议设置成 local 服务器正常延时的 2~5 倍 单位: 毫秒
always_standby: true
- tag: collector
type: metrics_collector
# 执行序列
- tag: main_sequence
type: sequence
args:
exec:
- _query_summary
- collector
- mem_cache
- core
- modify_ttl
# 服务器设置
servers:
- exec: main_sequence
listeners:
- protocol: udp
addr: :5335
- protocol: tcp
addr: :5335
# API 入口设置
api:
http: :8338
log:
level: info
file: "/var/log/mosdns.log"
api:
http: "0.0.0.0:8338"
include: []
plugins:
- tag: geosite_cn
type: domain_set
args:
files:
- "/var/mosdns/geosite_cn.txt"
- tag: geoip_cn
type: ip_set
args:
files:
- "/var/mosdns/geoip_cn.txt"
- tag: geosite_no_cn
type: domain_set
args:
files:
- "/var/mosdns/geosite_geolocation-!cn.txt"
- tag: whitelist
type: domain_set
args:
files:
- "/etc/mosdns/rule/whitelist.txt"
- tag: blocklist
type: domain_set
args:
files:
- "/etc/mosdns/rule/blocklist.txt"
- tag: greylist
type: domain_set
args:
files:
- "/etc/mosdns/rule/greylist.txt"
- tag: ddnslist
type: domain_set
args:
files:
- "/etc/mosdns/rule/ddnslist.txt"
- tag: hosts
type: hosts
args:
files:
- "/etc/mosdns/rule/hosts.txt"
- tag: redirect
type: redirect
args:
files:
- "/etc/mosdns/rule/redirect.txt"
- tag: adlist
type: domain_set
args:
files:
- "/etc/mosdns/rule/adlist.txt"
- tag: local_ptr
type: domain_set
args:
files:
- "/etc/mosdns/rule/local-ptr.txt"
- tag: lazy_cache
type: cache
args:
size: 20000
lazy_cache_ttl: 86400
- tag: reject_3
type: sequence
args:
- exec: reject 3
- tag: reject_blocklist
type: sequence
args:
- exec: query_summary reject_blocklist
- exec: $reject_3
- tag: reject_adlist
type: sequence
args:
- exec: query_summary reject_adlist
- exec: $reject_3
- tag: reject_ptrlist
type: sequence
args:
- exec: query_summary reject_ptrlist
- exec: $reject_3
- tag: reject_qtype65
type: sequence
args:
- exec: query_summary reject_qtype65
- exec: $reject_3
- tag: forward_local
type: forward
args:
concurrent: 1
upstreams:
- addr: 119.29.29.29
bootstrap: 119.29.29.29
enable_pipeline: false
max_conns: 2
insecure_skip_verify: false
idle_timeout: 30
- addr: 223.5.5.5
bootstrap: 119.29.29.29
enable_pipeline: false
max_conns: 2
insecure_skip_verify: false
idle_timeout: 30
- tag: forward_remote
type: forward
args:
concurrent: 1
upstreams:
- addr: tls://8.8.8.8
bootstrap: 119.29.29.29
enable_pipeline: false
max_conns: 2
insecure_skip_verify: false
idle_timeout: 30
enable_http3: false
- addr: tls://8.8.4.4
bootstrap: 119.29.29.29
enable_pipeline: false
max_conns: 2
insecure_skip_verify: false
idle_timeout: 30
enable_http3: false
- addr: tls://1.1.1.1
bootstrap: 119.29.29.29
enable_pipeline: false
max_conns: 2
insecure_skip_verify: false
idle_timeout: 30
enable_http3: false
- addr: tls://208.67.222.222
bootstrap: 119.29.29.29
enable_pipeline: false
max_conns: 2
insecure_skip_verify: false
idle_timeout: 30
enable_http3: false
- tag: modify_ttl
type: sequence
args:
- exec: ttl 0-0
- tag: modify_ddns_ttl
type: sequence
args:
- exec: ttl 5-5
- tag: local_sequence
type: sequence
args:
- exec: query_summary forward_local
- exec: $forward_local
- tag: remote_sequence
type: sequence
args:
- exec: query_summary forward_remote
- exec: prefer_ipv4
- exec: $forward_remote
- tag: has_resp_sequence
type: sequence
args:
- matches: qname $ddnslist
exec: $modify_ddns_ttl
- matches: "!qname $ddnslist"
exec: $modify_ttl
- matches: has_resp
exec: accept
- tag: query_is_local_ip
type: sequence
args:
- exec: $local_sequence
- matches: "!resp_ip $geoip_cn"
exec: drop_resp
- tag: query_is_remote
type: sequence
args:
- exec: $remote_sequence
- tag: fallback
type: fallback
args:
primary: query_is_local_ip
secondary: query_is_remote
threshold: 500
always_standby: true
- tag: query_is_ddns_domain
type: sequence
args:
- matches: qname $ddnslist
exec: $local_sequence
- tag: query_is_local_domain
type: sequence
args:
- matches: qname $geosite_cn
exec: $local_sequence
- tag: query_is_no_local_domain
type: sequence
args:
- matches: qname $geosite_no_cn
exec: $remote_sequence
- tag: query_is_whitelist_domain
type: sequence
args:
- matches: qname $whitelist
exec: $local_sequence
- tag: query_is_greylist_domain
type: sequence
args:
- matches: qname $greylist
exec: $remote_sequence
- tag: query_is_reject_domain
type: sequence
args:
- matches: qname $blocklist
exec: $reject_blocklist
- matches: qname $adlist
exec: $reject_adlist
- matches:
- qtype 12
- qname $local_ptr
exec: $reject_ptrlist
- matches: qtype 65
exec: $reject_qtype65
- tag: fallback_sequence
type: sequence
args:
- exec: query_summary fallback
- exec: $fallback
- tag: main_sequence
type: sequence
args:
- exec: metrics_collector metrics
- exec: $hosts
- exec: jump has_resp_sequence
- matches:
- "!qname $ddnslist"
- "!qname $blocklist"
- "!qname $adlist"
- "!qname $local_ptr"
exec: $lazy_cache
- exec: $redirect
- exec: jump has_resp_sequence
- exec: $query_is_ddns_domain
- exec: jump has_resp_sequence
- exec: $query_is_whitelist_domain
- exec: jump has_resp_sequence
- exec: $query_is_reject_domain
- exec: jump has_resp_sequence
- exec: $query_is_greylist_domain
- exec: jump has_resp_sequence
- exec: $query_is_local_domain
- exec: jump has_resp_sequence
- exec: $query_is_no_local_domain
- exec: jump has_resp_sequence
- exec: $fallback_sequence
- tag: udp_server
type: udp_server
args:
entry: main_sequence
listen: ":5335"
- tag: tcp_server
type: tcp_server
args:
entry: main_sequence
listen: ":5335"
log:
level: debug
file: "/var/log/mosdns.log"
api:
http: "0.0.0.0:8338"
include: []
plugins:
################ 匹配器 #################
# 国内域名
- tag: geosite_cn
type: domain_set
args:
files:
- "/etc/mosdns/rule/whitelist.txt"
- "/var/mosdns/geosite_cn.txt"
- "/var/mosdns/geosite_apple-cn.txt"
- "/var/mosdns/geosite_category-games@cn.txt"
# 国内 IP
- tag: geoip_cn
type: ip_set
args:
files:
- "/var/mosdns/geoip_cn.txt"
# 国外域名
- tag: geosite_no_cn
type: domain_set
args:
files:
- "/etc/mosdns/rule/greylist.txt"
- "/var/mosdns/geosite_geolocation-!cn.txt"
- tag: blocklist
type: domain_set
args:
files:
- "/etc/mosdns/rule/blocklist.txt"
- tag: local_ptr
type: domain_set
args:
files:
- "/etc/mosdns/rule/local-ptr.txt"
- "/var/mosdns/geosite_private.txt"
############## 附属功能 ###############
# 本地缓存
- tag: enable_cache
type: cache
args:
size: 20000
lazy_cache_ttl: 86400
- tag: lazy_cache
type: sequence
args:
- matches:
- "!qname $blocklist"
- "!qname $local_ptr"
exec: $enable_cache
- tag: reject_blocklist
type: sequence
args:
- exec: query_summary reject_blocklist
- exec: reject 3
- tag: reject_ptrlist
type: sequence
args:
- exec: query_summary reject_ptrlist
- exec: reject 3
- tag: reject_qtype65
type: sequence
args:
- exec: query_summary reject_qtype65
- exec: reject 3
################ DNS #################
- tag: local_upstream
type: forward
args:
concurrent: 2
upstreams:
- addr: https://doh.pub/dns-query
bootstrap: 119.29.29.29
enable_pipeline: false
enable_http3: false
- addr: https://dns.alidns.com/dns-query
dial_addr: 223.5.5.5
enable_pipeline: false
enable_http3: false
# default is udp protocol
- addr: 119.29.29.29
- addr: 223.5.5.5
- tag: remote_upstream
type: forward
args:
upstreams:
# clash
- addr: 127.0.0.1:7874
################ 核心序列 #################
- tag: local_sequence
type: sequence
args:
- exec: query_summary local_sequence
- exec: prefer_ipv4
- exec: $local_upstream
- matches:
- has_resp
- resp_ip 10.10.10.0/24 # LAN IPCIDR
exec: ttl 1800-0
- tag: remote_sequence
type: sequence
args:
- exec: query_summary remote_sequence
- exec: $remote_upstream
- exec: ttl 1800-0
# ################ 条件判断序列 #################
- tag: query_is_reject_domain
type: sequence
args:
- matches: qname $blocklist
exec: $reject_blocklist
- matches: qtype 65
exec: $reject_qtype65
- matches:
- qtype 28
exec: reject 0
- matches:
- qtype 12
- qname $local_ptr
exec: $reject_ptrlist
- tag: query_is_local_domain
type: sequence
args:
- matches: qname $geosite_cn
exec: $local_sequence
- matches: "!resp_ip $geoip_cn"
exec: drop_resp
- tag: query_is_no_local_domain
type: sequence
args:
- matches: qname $geosite_no_cn
exec: $remote_sequence
- tag: has_resp_sequence
type: sequence
args:
- matches: has_resp
exec: accept
############### 执行序列 #################
# 兜底序列
- tag: fallback_local
type: sequence
args:
- exec: query_summary fallback_local
- exec: $local_sequence
- matches: "!resp_ip $geoip_cn"
exec: accept
- exec: drop_resp
- tag: fallback_remote
type: sequence
args:
- exec: query_summary fallback_remote
- exec: $remote_sequence
- tag: fallback_sequence
type: fallback
args:
primary: fallback_remote
secondary: fallback_remote
threshold: 500
always_standby: true
# 主序列
- tag: main_sequence
type: sequence
args:
- exec: metrics_collector metrics
- exec: $lazy_cache
- exec: prefer_ipv4
- exec: $query_is_reject_domain
- exec: jump has_resp_sequence
- exec: $query_is_local_domain
- exec: jump has_resp_sequence
- exec: $query_is_no_local_domain
- exec: jump has_resp_sequence
- exec: $fallback_sequence
################ 服务器 #################
- tag: udp_server
type: udp_server
args:
entry: main_sequence
listen: ":5335"
- tag: tcp_server
type: tcp_server
args:
entry: main_sequence
listen: ":5335"
@LinusJun
Copy link

大佬啥时候能分享下 通过 vector 二次解析并接入了 loki 后整合到 [Grafana 面板]的具体步骤 十分感谢

@icyleaf
Copy link
Author

icyleaf commented Aug 23, 2023

@zxjack
Copy link

zxjack commented Oct 31, 2023

按照方案实施了之后发现面板上没有数据,查看vector运行会报错,代码如下:
023-10-31T09:37:05.731911Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap component_name=mosdns-data}: vector::internal_events::remap: Internal log [Mapping failed with event.] has been suppressed 1 times.
2023-10-31T09:37:05.731924Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap component_name=mosdns-data}: vector::internal_events::remap: Mapping failed with event. error="function call error for "parse_timestamp" at (128:184): Invalid timestamp "2023-10-31T17:36:57.313+0800": input contains invalid characters" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
2023-10-31T09:37:05.731959Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap component_name=mosdns-data}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 1 times.
2023-10-31T09:37:05.731968Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap component_name=mosdns-data}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Mapping failed with event." internal_log_rate_limit=true
2023-10-31T09:37:05.732015Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap component_name=mosdns-data}: vector::internal_events::remap: Internal log [Mapping failed with event.] is being suppressed to avoid flooding.
2023-10-31T09:37:05.732031Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=rem
可能是哪里没有搞对吗?

@icyleaf
Copy link
Author

icyleaf commented Oct 31, 2023

@zxjack 博客评论已回复

@llity
Copy link

llity commented Mar 23, 2024

forward clash是不是应该写clash的fake ip哦
- tag: remote_upstream type: forward args: upstreams: # clash - addr: 127.0.0.1:7874

@chenbin3625
Copy link

image image image

使用docker-compose搭建的四合一服务 似乎只有四个面板有数据 不太清楚问题出在哪里 看日志vector一直在打印示例数据 不是实际的log 进入容器查看日志是正确映射的

@icyleaf
Copy link
Author

icyleaf commented Sep 26, 2024

@chenbin3625 请完整使用我的配置文件,不要在默认配置上面编辑。

@chenbin3625
Copy link

感谢回复🙏
确实是完整引入的
docker-compose.yaml

version: '3'

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "4000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana:/var/lib/grafana

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  vector:
    image: timberio/vector:latest-alpine
    ports:
      - "8686:8686"
    volumes:
      - ./vector.toml:/etc/vector/vector.toml
      - /root/dns/mosdns/mosdns.log:/var/log/mosdns.log

vector.toml

data_dir: /tmp/vector

sources:
  mosdns-log-file:
    type: file
    include:
      - /var/log/mosdns.log
    read_from: beginning

transforms:
  mosdns-input:
    type: filter
    inputs:
      - mosdns-log-file
    condition: |
      .file == "/var/log/mosdns.log"      

  mosdns-data:
    type: remap
    inputs:
      - mosdns-input
    drop_on_error: true
    source: |
      .type = "mosdns"
      .app = "mosdns"
      del(.host)
      del(.file)
      del(.source_type)

      message_parts = split!(.message, r'\t')

      .timestamp = parse_timestamp!(message_parts[0], format: "%FT%T%.9fZ"
      .level = message_parts[1]

      if (length(message_parts) == 6) {
        .plugin = message_parts[2]
        .processor = message_parts[3]
        .message = message_parts[4]

        if (exists(message_parts[5])) {
          .metadata = parse_json!(message_parts[5])
          . = merge!(., .metadata)
          del(.metadata)
        }
      } else {
        .processor = message_parts[2]
        .message = message_parts[3]

        if (exists(message_parts[4])) {
          .metadata = parse_json!(message_parts[4])
          . = merge!(., .metadata)
          del(.metadata)
        }
      }

      if (exists(.query)) {
        query_parts = split!(.query, r'\s')
        .domain = query_parts[0]
        .record = query_parts[2]
        .address = query_parts[5]
      }      

sinks:
  # 同步到 loki,根据实际情况修改 endpoint 的值
  loki:
    type: loki
    inputs:
      - mosdns-data
    endpoint: 'http://10.10.11.250:3100'
    encoding:
      codec: json
    labels:
      app: '{{ app }}'
      type: '{{ type }}'
    healthcheck:
      enabled: true

prometheus.yml

global:
  scrape_interval:     1m
  evaluation_interval: 1m

scrape_configs:
  - job_name: mosdns
    scrape_interval: 5s
    # scrape_timeout: 10s

    # metrics_path: /metrics
    static_configs:
      - targets:
        - 10.10.11.250:88

mosdns

log:
  level: info
  file: "/etc/mosdns/mosdns.log"

api:
  http: "0.0.0.0:88"

include: []

@icyleaf
Copy link
Author

icyleaf commented Sep 26, 2024

  debug_mosdns:
    type: console
    inputs:
      - mosdns-data
    encoding:
      codec: json

调试的话这个没加 vector 肯定没输出啊,我怀疑是两个问题:

  1. vector 是否挂载了 mosdns 日志文件
  2. loki url 是否可以让 vector 访问到

@chenbin3625
Copy link

vector容器内探测loki正常
image

vector容器内查看mosdns日志正常
image

加入debug_mosdns也是一直输出示例的数据 我才删掉的 现在加回去还是一样

@icyleaf
Copy link
Author

icyleaf commented Sep 26, 2024

那会不会是你 grafana 没有配 loki source,配好一用 explorer 浏览看看有没有数据

@darkrain88
Copy link

2024-10-19T03:29:00.668193Z INFO vector::topology::builder: Healthcheck passed.

2024-10-19T03:37:32.995231Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for "parse_timestamp" at (128:184): Invalid timestamp "2024-10-19 11:37:29": input contains invalid characters" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

2024-10-19T03:37:32.995332Z ERROR transform{component_kind="transform" component_id=mosdns-data component_type=remap}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Mapping failed with event." internal_log_rate_limit=true

是不是5.3.3 的日期格式改了

@icyleaf
Copy link
Author

icyleaf commented Oct 19, 2024

试试

- .timestamp = parse_timestamp!(message_parts[0], format: "%FT%T%.9fZ")
+ .timestamp = parse_timestamp!(message_parts[0], format: "%F %T")

@darkrain88
Copy link

2024-10-20T01:39:20.243949Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

INFO cache dumped | app=mosdns entries=1019 processor=lazy_cache timestamp=2024-10-20T11:39:16Z type=mosdns

2024-10-20T03:39:22.308749Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=3}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Server responded with an error: 400 Bad Request internal_log_rate_limit=true

2024-10-20T03:39:22.308823Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=3}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(ServerError { code: 400 }) request_id=3 error_type="request_failed" stage="sending" internal_log_rate_limit=true

2024-10-20T03:39:22.308877Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=3}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

好像都连上了, 还有错误

@icyleaf
Copy link
Author

icyleaf commented Oct 20, 2024

难道不同系统的时间戳不一样吗,你的是 %FT%T%.6fZ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment