From many years of managing Elastic and Open search at scale, Here are some of my most useful DevOps commands ??

In the spirit of the holidays, and in an effort to help my fellow DevOps, Elastic and Open search aficionados, i have gathered a list of useful commands here that i use regularly for maintenance and analysis.

DISCLAIMER: These are pillaged from my obsidian notes, so if there are some un needed sudo's, specific paths or similar, dont hunt me down. Just fix it and reuse :-)

• I attempt to write my commands in "blocks" so they are copy pastable directly to the terminal.
• I also attempt to use as much standard tooling as i can, where i can, as i often work in air gapped environments.
• If i write <IP_ADDRESS> for example, it means you need to replace that value.

Elastic specific

Color coded logstash queue information:

If you, like me, cant get the new fancy logstash cli, this is the second best thing.

See your logstash queues, with color coded errors. Please come with improvements! I also have a version that allows to use watch, and saves the trend in the queue over time, in a separate file. Its a bit bigger though, so ping me if you want a copy.

docker exec logstash curl -s https://localhost:9600/_node/stats/pipelines | jq -r '.pipelines | to_entries[] |
  . as $pipeline |
  [
    $pipeline.key,
    ($pipeline.value.events.in // 0),
    ($pipeline.value.events.out // 0),
    ($pipeline.value.queue.queue_size_in_bytes // 0),
    ($pipeline.value.queue.max_queue_size_in_bytes // 0),
    ($pipeline.value.queue.events_count // 0),
    ($pipeline.value.queue.capacity.max_unread_events // 0)
  ] | @tsv' | awk '
  BEGIN {
    FS=OFS="\t";
    print "Pipeline", "Events_In", "Events_Out", "Queue_Size_Bytes", "Max_Queue_Size_Bytes", "Events_Count", "Max_Unread_Events";
  }
  {
    if ($5 > 0 && $4/$5 >= 0.95) { $4 = "\033[31m" $4 "\033[0m" }
    if ($7 > 0) { $7 = "\033[31m" $7 "\033[0m" }
    print $0
  }' | column -t        

Make list of unique ip addresses in the Logstash log

I've used this often, when a major change is deployed, and the log is flooded. Its a good way to get a list of addresses to figure out where the problem is coming from. In some cases its "all windows servers" because an Active Directory change was made, or a certificate was revoked.

Here it makes a lot of sense to group and look at all the hosts in one context.

docker logs logstash --tail=10000 | grep -oP '(local|remote):\s[\d\.]+:\d+' | sort -u        

Testing network connections and sending test logs

Listen UDP port:

nc -l -k -u -p 9001        

Listen TCP Port:

nc -l -k -p 9001        

Send TCP test message:

echo -n "TEST Message TCP" | nc -q 0 <IP_ADDRESS> 514        

Send UDP test message:

echo -n "TEST Message UDP" | nc -u -q 0 <IP_ADDRESS> 514        

Send test syslog message formatted with today's date and time -1 hour:

This is useful for testing timing, and ingestion pipelines.

Sample Cisco catalyst link up event.

echo -n "<189>123321: SOME-HOST-NAME: 321123: $(date +'%b') $(date +'%d') $(date --date="-1 hour" +'%H:%M:%S') UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet2/4, changed state to up" | nc <IP_ADDRESS> 514 | true        

Log analysis:

Count keywords in log file:

Useful for frankly a lot of things, but if you have a hunch something is trending, or if you have a lot of devices like me, and you want to figure out if a particular device is appearing more than others, this is great.

sudo awk -v keywords="logstash elastic" 'BEGIN {split(keywords, keys); total=0} {total++; for (i in keys) if (index($0, keys[i])) count[keys[i]]++} END {print "Count:"; for (k in count) printf "%-10s %d (%.2f%%)\n", k, count[k], count[k]/total*100}' /var/log/syslog        

Get all log lines since a specific term / string

This is not that common, but sometimes im testing something and i just want to see how much of a mess i made since it applied.

docker logs logstash --tail=1000 2>&1 | awk '/SIGTERM/{flag=1} flag'        

Delete all lines containing a specific word with regex (like vscode, ZED or other search and replace!)

Useful when sifting through a log dump for a particular problem and you have a hunch, but firewall logs, windows event logs or some other noisy component get in the way.

I highly recommend using a log formatter in vscode as well, its very efficient.

Especially if structured json logs are analyzed with json crack after (Thanks Dennis Perto for the hint!)

# All lines NOT containing the hostname: "NOISY_FIREWALL"
^(?!.*NOISY_FIREWALL).*$\n

# All lines containing word:"NOISY_FIREWALL"
^.*NOISY_FIREWALL.*$\n        

Docker

Monitor directory inside container

-n sets the frequency to once per second

watch -n 1 -x docker exec elasticsearch bash $(ls -lahn /usr/share/elasticsearch)        

Color coded docker logs output:

You can of course modify this as you please - its an easy way to get some color into your life, on that bland white and black terminal abyss you stare at all day :-)

docker logs logstash --tail=100 -f | sed -E 's/("message":")(.*)(")/\2/g' | grep --color=always -E '"message":"|([0-9]{1,3}\.){3}[0-9]{1,3}|ERROR|WARN' | sed -E 's/(ERROR)/\x1b[31m\1\x1b[0m/g; s/(WARN)/\x1b[33m\1\x1b[0m/g; s/(([0-9]{1,3}\.){3}[0-9]{1,3})/\x1b[34m\1\x1b[0m/g'        

Finding hardware related issues in docker logs

Sometimes a container restarts before it even starts the services inside. In these cases, you may need to look for disk, cpu or memory problems. Docker events allow you to do that.

This is how you filter it:

docker events --filter 'event=start' --filter 'event=die' --filter 'event=oom' --filter 'event=kill' --format 'table {{.Time}}  {{.Type}}  {{.Action}}  {{.Actor.Attributes.name}}  {{.Actor.Attributes.exitCode}}'        

Get memory usage of a single container

docker stats --no-stream --format?"{{.MemUsage}}"?logstash?| awk -F/?'{print $1}'?| sed?'s/[^0-9\.]*//g'        

Formatted docker ps

Because i dont always need to see all those ports, i just want some simple overview.

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.ID}}\t{{.RunningFor}}\t{{.Image}}"        

Get quick log status of all containers:

I use this to grab log lines from all containers and pipe it elsewhere. This is formatted for printing, but you can edit it as you need. Requires jq, but you can live without the nice formatting.

for container in $(docker ps -a --format '{{.Names}}'); do printf "\n Last 25 log lines from: $container \n" && docker logs $container --tail=25 && printf "\n Additional information from: $container \n" && docker container ls -s -f name=$container --format '{{ json . }}' | jq && printf "\n"; done        

Get all error level logs from container:

Quick example of how you can format and search docker logs.

docker logs --since=168h --details logstash | grep -w "\[ERROR\]"         

Misc, Files etc.

Intro to xargs (it's not hard, and it's really usefull)

# Copies "config.json" to "/backup/config.json.bak"
echo "config.json" | xargs -I {} cp {} /backup/{}.bak

# Moves "src.txt" to "dest.txt" (reads pairs of arguments)
echo "src.txt dest.txt" | xargs -n 2 sh -c 'mv "$0" "$1"'         

- -I {}: Replaces {} with the input in the command.

- -n <N>: Works well with multiple arguments by splitting input into chunks.

Recursively find all file paths to files by extension / name

find ./ -name *.tar        

Recursively find all files containing specific string in current directory:

grep -rnw './' -e 'somestring'        

Update all git repos in dir:

This is not perfect, but it gets the job done if you have a bunch of repos to manage.

Bash:

for dir in */; do (cd "$dir" && git pull || echo "Failed to pull in $dir") || continue; done        

Powershell: (Some of us are occasionally forced to open a window(s))

Get-ChildItem -Directory | ForEach-Object { try { Set-Location $_.FullName; git pull } catch { Write-Host "Failed to pull in $($_.FullName): $_" } finally { Set-Location -Path .. } }        

Dos2Unix with Perl or SED:

We've all been there. If you haven't, you will soon enough.

One of these should work in your environment.

perl -pi -e 's/\r\n/\n/g' file.txt

sed 's/$'"/`echo \\\r`/" input.txt > output.txt

find . -type f -print0 | xargs -0 dos2unix

find . -type f -exec grep -qIP '\r\n' {} ';' -exec perl -pi -e 's/\r\n/\n/g' {} '+'        

Formatted folder size:

This is a bit of oddity but i use it often. If you have a directory of folders, like your elastic search node folder containing the actual indices in text files, it is hard to know what they take up on disk.

This "oneliner" (everything is a oneliner if you're brave enough) attempts to gather all the relevant data about sizing, permissions, paths etc.

{
  echo -e "Size\tPermissions\tUser\tGroup\tMod Time\t\t\tAvailable\tTotal\tPath"
  sudo du -sh ./* | while read size path; do
    disk_info=$(sudo df -h "$path" | tail -1 | awk '{print $4, $2}')
    user=$(sudo stat -c "%U" "$path")
    group=$(sudo stat -c "%G" "$path")
    permissions=$(sudo stat -c "%A" "$path")
    mod_time=$(sudo stat -c "%y" "$path")
    printf "%-8s\t%-11s\t%-8s\t%-8s\t%-20s\t%-10s\t%-10s\t%s\n" "$size" "$permissions" "$user" "$group" "$mod_time" "${disk_info%% *}" "${disk_info##* }" "$path"
  done | sort -h
}        

Log rotate - Kibana config example

# Create file, and add config:
# nano log-rotate.<service-name>.conf

# Chown to root for permissions
# sudo chown root log-rotate.<service-name>.conf

# Apply changes with:
# sudo logrotate -f log-rotate.<service-name>.conf

# View log rotate events with:
# cat /mnt/disk1/logs/rotation.log

# Sample config for kibana.
## Logs are kept for a total of 3Gb * 5, compressed on rotation and the main file is always kept intact.
/mnt/disk1/logs/kibana/kibana.log {
    size 3G
    rotate 5
    compress
    copytruncate
    firstaction
        echo "start rotation $(date)" >> /var/log/kibana-rotation.log
    endscript
    lastaction
        echo "complete rotation $(date)" >> /var/log/kibana-rotation.log
    endscript
}        

I hope it helps you survive the holidays, a bit easier this year ??