Skip to content

log_parser job from logsearch/18

Github source: adff8e1b or master branch

Properties

elasticsearch

cluster_name

The name of the elastic search cluster

flush_size

Redis queue flush size

Default
100

host

IP / DNS of elasticsearch http endpoint

log_level

The default logging level (e.g. WARN, DEBUG, INFO)

Default
INFO

marvel

agent
enabled

Whether to enable the marvel agent

Default
false

node

tags

A hash of additional tags for the node

port

Port of elasticsearch http endpoint

Default
9200

logstash

metadata_level

Whether to include additional metadata throughout the event lifecycle. NONE = disabled, DEBUG = fully enabled

Default
NONE

logstash_parser

debug

Debug level logging

Default
false

filters

The configuration to embed into the logstash filters section

Default
""

idle_flush_time

How frequently to flush events if the output queue is not full.

message_max_size

Maximum log message length. Anything larger is truncated (TODO: move this to ingestor?)

Default
1.048576e+06

outputs

The configuration to embed into the logstash outputs section

use_local_elasticsearch

Run a local elasticsearch client node

Default
true

workers

The number of worker threads that logstash should use (default: auto = one per CPU)

Default
auto

redis

host

Redis host of queue

key

Name of queue to pull messages from

Default
logstash

port

Redis port of queue

Default
6379

Templates

Templates are rendered and placed onto corresponding instances during the deployment process. This job's templates will be placed into /var/vcap/jobs/log_parser/ directory (learn more).

  • bin/elasticsearch_ctl (from bin/elasticsearch_ctl)
  • bin/log_parser_ctl (from bin/log_parser_ctl)
  • bin/monit_debugger (from bin/monit_debugger)
  • config/elasticsearch/default.json (from config/elasticsearch/default.json.erb)
  • config/elasticsearch/logging.yml (from config/elasticsearch/logging.yml.erb)
  • config/filters_default.conf (from config/filters_default.conf)
  • config/filters_override.conf (from config/filters_override.conf.erb)
  • config/filters_post.conf (from config/filters_post.conf.erb)
  • config/filters_pre.conf (from config/filters_pre.conf.erb)
  • config/input_redis_and_output_elasticsearch.conf (from config/input_redis_and_output_elasticsearch.conf.erb)
  • data/properties.sh (from data/properties.sh.erb)
  • helpers/ctl_setup.sh (from helpers/ctl_setup.sh)
  • helpers/ctl_utils.sh (from helpers/ctl_utils.sh)
  • logsearch/logs.yml (from logsearch/logs.yml)

Packages

Packages are compiled and placed onto corresponding instances during the deployment process. Packages will be placed into /var/vcap/packages/ directory.