Skip to content

parser job from logsearch/200.0.0

Github source: c42a9f04 or master branch

Properties

elasticsearch

cluster_name

The name of the elastic search cluster

config_options

Additional options to append to elasticsearch’s config.yml (YAML format).

data_hosts

The list of elasticsearch data node IPs

discovery

minimum_master_nodes

The minimum number of master eligible nodes a node should “see” in order to operate within the cluster. Recommended to set it to a higher value than 1 when running more than 2 nodes in the cluster.

Default
1

flush_size

Controls how many logs will be buffered and sent to Elasticsearch for bulk indexing

Default
500

log_level

The default logging level (e.g. WARN, DEBUG, INFO)

Default
INFO

logging_options

Additional options to append to elasticsearch’s logging.yml (YAML format).

master_hosts

The list of elasticsearch master node IPs

node

tags

A hash of additional tags for the node

logstash

metadata_level

Whether to include additional metadata throughout the event lifecycle. NONE = disabled, DEBUG = fully enabled

Default
NONE

logstash_parser

debug

Debug level logging

Default
false

elasticsearch_document_id

Use a specific, dynamic ID rather than an auto-generated identifier.

elasticsearch_index

The specific, dynamic index name to write events to.

Default
logstash-%{+YYYY.MM.dd}

elasticsearch_index_type

The specific, dynamic index type name to write events to.

Default
'%{@type}'

filters

The configuration to embed into the logstash filters section

Default
""

idle_flush_time

How frequently to flush events if the output queue is not full.

inputs

A list of input plugins, with a hash of options for each of them. Please refer to example below.

Default
  - options: {}
    plugin: redis
Example
inputs:
- options:
    host: 192.168.1.1
    password: c1oudbunny
    user: logsearch
  plugin: rabbitmq

message_max_size

Maximum log message length. Anything larger is truncated (TODO: move this to ingestor?)

Default
1.048576e+06

outputs

The configuration to embed into the logstash outputs section

plugins

Plugins to run logstash with (array[] = { plugin-name: install-source }; e.g. [ { logstash-filter-cityindex-acctlookup: ‘https://s3.amazonaws.com/.../logstash-filter-cityindex-acctlookup-1.2.9.gem' } ])

Default
[]

timecop

reject_greater_than_hours

Logs with timestamps greater than this many hours in the future won’t be parsed and will get tagged with fail/timecop

Default
1
reject_less_than_hours

Logs with timestamps less than this many hours in the past won’t be parsed and will get tagged with fail/timecop

Default
24

use_local_elasticsearch

Run a local elasticsearch client node

Default
true

workers

The number of worker threads that logstash should use (default: auto = one per CPU)

Default
auto

redis

host

Redis host of queue

key

Name of queue to pull messages from

Default
logstash

port

Redis port of queue

Default
6379

Templates

Templates are rendered and placed onto corresponding instances during the deployment process. This job's templates will be placed into /var/vcap/jobs/parser/ directory (learn more).

  • bin/elasticsearch_ctl (from bin/elasticsearch_ctl)
  • bin/monit_debugger (from bin/monit_debugger)
  • bin/parser_ctl (from bin/parser_ctl)
  • config/elasticsearch/elasticsearch.yml (from config/elasticsearch/config.yml.erb)
  • config/elasticsearch/logging.yml (from config/elasticsearch/logging.yml.erb)
  • config/filters_override.conf (from config/filters_override.conf.erb)
  • config/filters_post.conf (from config/filters_post.conf.erb)
  • config/filters_pre.conf (from config/filters_pre.conf.erb)
  • config/if_it_looks_like_json.conf (from config/if_it_looks_like_json.conf)
  • config/input_and_output.conf (from config/input_and_output.conf.erb)
  • config/timecop.conf (from config/timecop.conf)
  • data/properties.sh (from data/properties.sh.erb)
  • helpers/ctl_setup.sh (from helpers/ctl_setup.sh)
  • helpers/ctl_utils.sh (from helpers/ctl_utils.sh)
  • logsearch/logs.yml (from logsearch/logs.yml)

Packages

Packages are compiled and placed onto corresponding instances during the deployment process. Packages will be placed into /var/vcap/packages/ directory.