Loki安装配置与故障排除

安装

Loki、promtail、Grafana

二进制包安装

配置文件下载
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml
二进制包下载
wget https://github.com/grafana/loki/releases/download/v1.6.0/loki-linux-amd64.zip
wget https://github.com/grafana/loki/releases/download/v1.6.0/promtail-linux-amd64.zip
wget https://mirrors.huaweicloud.com/grafana/7.1.5/grafana-7.1.5.linux-amd64.tar.gz

Loki配置文件

更多参数请看官网https://grafana.com/docs/loki/latest/configuration/

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 39095 #grpc监听端口，默认为9095
  grpc_server_max_recv_msg_size: 15728640  #grpc最大接收消息值，默认4m
  grpc_server_max_send_msg_size: 15728640  #grpc最大发送消息值，默认4m

ingester:
  lifecycler:
    address: 192.168.8.253 #IP地址
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0
  max_chunk_age: 20m  #一个timeseries块在内存中的最大持续时间。如果timeseries运行的时间超过此时间，则当前块将刷新到存储并创建一个新块

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /opt/loki/loki_data/index

  filesystem:
    directory: /opt/loki/loki_data/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 30  #修改每用户摄入速率限制，即每秒样本量，默认值为4M
  ingestion_burst_size_mb: 15  #修改每用户摄入速率限制，即每秒样本量，默认值为6M

chunk_store_config:
  max_look_back_period: 168h   #回看日志行的最大时间，只适用于即时日志

table_manager:
  retention_deletes_enabled: true #日志保留周期开关，默认为false
  retention_period: 168h  #日志保留周期

promtail配置文件

更多参数请看官网https://grafana.com/docs/loki/latest/clients/promtail/configuration/

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  grpc_server_max_recv_msg_size: 15728640
  grpc_server_max_send_msg_size: 15728640

positions:
  filename: /opt/loki/promtail/positions.yaml

clients:
  - url: http://192.168.8.253:3100/loki/api/v1/push

scrape_configs:
- job_name: app_log
  static_configs:
  - targets:
      - localhost
    labels:
     #env:dev 打标签
      __path__: /ceph_root/{dev,test}/apps_data/*/log/*log

静态文件配置语法 https://github.com/bmatcuk/doublestar

解压并启动

loki

unzip loki-linux-amd64.zip
nohup ./loki-linux-amd64 -config.file=loki-local-config.yaml &

promtail

promtail-linux-amd64.zip
nohup ./promtail-linux-amd64 -config.file=promtail-local-config.yaml &

grafana

tar -zxvf grafana-7.1.5.linux-amd64.tar.gz 
cd grafana-7.1.5/bin
nohup ./grafana-server &

故障排除

a)

[level=warn ts=2020-07-06T06:54:12.754273854Z caller=client.go:242 component=client host=192.179.11.1:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded (limit: 6291456 bytes/sec) while attempting to ingest '221' lines totaling '101946' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"

当你搭建完成promtail，并且启动发送日志到loki的时候很有可能会碰到这个错误，因为你要收集的日志太多了，超过了loki的限制，所以会报429，如果你要增加限制可以修改loki的配置文件,在limits_config中添加

ingestion_rate_mb: 8

b)

error="open /opt/loki/promtail/positions.yaml: too many open files

这个报错是因为promtail打开的文件句柄过多，超过了服务器的限制，可用ulimit -a查看当前服务器限制信息。以及lsof -p pid | wc -l查看该进程打开文件数。解决方式为修改ulimit，增加open files 增加文件句柄数

c)

level=error ts=2020-07-06T03:58:02.217480067Z caller=client.go:247 component=client host=192.179.11.1:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry for stream '{app=\"app_error\", filename=\"/error.log\", host=\"192.179.11.12\"}' has timestamp too new: 2020-07-06 03:58:01.175699907 +0000 UTC"

这个是两台机器的时间相差太大了，我promtail这台机器的时间没有和ntp服务器同步时间，所以就报了这个错误，只要把时间都同步了就好了