lugh.ch

lugh.ch

as nerdy as needed.

Useful Linux, Solaris and macOS commands

Table of contents:

Useful generic commands

Nice and ionice together

nice -n 19 ionice -c2 -n7 <command>

You can also apply this to a running process:

renice -n 19 <pid>
ionice -c2 -n7 -p <pid>

dd, write ISO to disk

cat centos64.img | ssh -C root@x.x.x.xxx "dd of=/dev/sda bs=1M"

Count log entries per second

Helps to calculate the amount of log entries written to a log file.

tail -f /var/log/varnish/access.log | pv -l -i2 -r >/dev/null

Send tar file via SSH

tar cfzp - /dir/to/backup | ssh root@192.168.1.2 "cat > /tmp/destination.tar.gz"

Highlight changing network information (rx,tx etc.)

watch -n 2 -d '/sbin/ifconfig eth0'

Get IP address from ifconfig output

# Linux
ifconfig eth0 | awk -F ' *|:' '/inet addr/{print $4}'
# OS X
ifconfig en1 | awk -F ' *|:' '/inet /{print $2}'

Disk usage

Ordered by size, descending

du -hax /var | sort -rh

Query SRV DNS records

Jabber/XMPP example:

dig SRV _xmpp-client._tcp.example.com
dig SRV _xmpp-server._tcp.example.com

Numeric permission list of all directories/files

Can be useful to restore in case of a fatal chmod -R 777 /.

find / | xargs stat -c 'chmod %a "'%n'"'

Act on files without creating temporary files

E.g. do a diff on unsorted files:

diff -u <(sort file1) <(sort file2)

Convert UNIX timestamp to an human-readable format

date -d @1305547782

Shell: sum columns

<command-that-generates-integers-columns> | paste -sd+ - | bc

Create a big temporary file fast

fallocate -l 10G /var/tmp/tmpfile

Vi resp. Vim

Get rid of all control characters

:%!tr -cd '[:print:]\n'

Fix indentation of the whole file

gg=G

wget

Download website, convert to static HTML

wget -mkxKE -e robots=off http://example.org

curl

URL shortening on command line

curl https://www.googleapis.com/urlshortener/v1/url -H 'Content-Type: application/json' -d '{"longUrl": "http://example.org"}'

Check if your webserver supports gzip compression with curl

curl -I -H "Accept-Encoding: gzip,deflate" http://example.org

curl: verify certificates

Using curl -k is a no-brainer, but really defeats the purpose of encryption. If you want to permanently store certificates as trusted, here's how to do it on Debian. This also works when using libcurl with PHP.

openssl s_client -connect example.org:443 # Get remote certificate
# Save the certificate in /usr/local/share/ca-certificates/example.crt
update-ca-certificates

netcat

Simple webserver

Useful to test connectivity.

while true; do echo "ALL UR  BASE" | nc -l 80; done

Varnish

Filter varnishlog to show all relevant content of XID

varnishlog -d -m TxHeader:<XID number> | awk '$1 !~ /0/ { print $0 }'

Varnish 4: filter by header

varnishlog -b -q "ReqHeader eq 'Host: example.org'"

Varnish 3: filter by URL

Filter for a frontend request (client => Varnish)

varnishlog -c -m RxURL:"/api/rest/products"

Filter for a backend request (Varnish => backend)

varnishlog -b -m TxURL:"/api/rest/products"

Check config file for errors

varnishd -C -f /etc/varnish/default.vcl

Show URLs hitting the backend the most

varnishtop -i txurl

Apache

Show compiled and shared modules loaded in Apache

/usr/bin/httpd -t -D DUMP_MODULES

Start xen guest and attach to virtual console

To access grub and see bootup messages

xm create -c vmname

Package managers

Which package contains that file?

# Red Hat
rpm -qf /bin/mount
# Debian
dpkg -S /bin/mount

Which packages were installed lately?

# Red Hat
rpm -qa --last | tac
# Debian
grep -E '(UPGRADE|INSTALL)' /var/log/aptitude

Which not installed package provides the specified file?

# Red Hat
yum whatprovides /bin/traceroute

Debian: hold a package to prevent updating

echo linux-image-2.6-686-bigmem hold | dpkg --set-selections

Text utilities (sed, grep, awk etc.)

Replace in-file (or in-place edit) with Perl

perl -pi -e 's/oldtext/newtext/' file.txt

Delete a block of text with sed

sed "/startOfBlock/,/endOfBlock/d" file.txt

Pad single-digit fields in MAC addresses

Thanks to Yannick Denzer

echo "a:0:1:0:a:43" | sed -E 's/[^:]+/0&/g;s/[^:]([^:][^:])/\1/g'

Get User-Agent from access log

awk -F\" '($2 ~ "^.* /"){print $6}' /var/log/apache2/access.log

Indent an unformatted XML file

xmlstarlet fo --indent-tab --omit-decl foo.xml

LDAP

ldapsearch with TLS

ldapsearch -x -Z <ldapserver>

ldapsearch SSL against Active Directory (AD)

Requires TLS_REQCERT never in /etc/openldap/ldap.conf.

ldapsearch -x -LLL -D "binduser" -w "bindpw" -b "dc=corp,dc=example,dc=org" -H ldaps://xxx.xx.xx.xx -v

Capturing traffic with tcpdump

tcpdump -i <iface> tcp port <port> and src <ip> -w /root/tcpdump.txt

Open reverse SSH tunnel

ssh -L localport:destination:port user@gateway

MySQL

Show extended information like collation, engine (e.g. InnoDB) etc.

For tables:

USE databasename;
SHOW TABLE STATUS;

For databases:

USE databasename;
SHOW VARIABLES LIKE "character_set_database";
SHOW VARIABLES LIKE "collation_database";

Maintenance

mysqlcheck -u root --auto-repair --check --optimize --all-databases

Show database engine used

SHOW TABLE STATUS FROM `db_name`;

Calculate size of all databases

SELECT table_schema AS "Database name",
SUM(data_length + index_length) / 1024 / 1024 AS "Size (MB)"
FROM information_schema.TABLES
GROUP BY table_schema;

Calculate size of all tables in a database

SELECT table_name AS "Table",
ROUND(((data_length + index_length) / 1024 / 1024), 2) AS size
FROM information_schema.TABLES
WHERE table_schema = "yourdatabase"
ORDER BY size DESC;

Visualize import progress

Requires the "pv" utility.

pv -i 1 -p -t -e /tmp/dump.sql | mysql -u foo -pXXXXX database

Import/export directly from/to compressed (gzip) file

# Export:
mysqldump -u user -p database | gzip > database.sql.gz

# Import:
gunzip < database.sql.gz | mysql -u user -p database

Kill idle (Sleep) processes

mysqladmin proc | grep "<user>.*<db>.*Sleep" | sort -r -n -k6 | awk {'print $2;'} | tr -s '\n' ',' | xargs mysqladmin kill

PostgreSQL

Dump DB to remote file via SSH

pg_dump -U pg-user database | ssh user@192.168.1.1 "cat - > /var/dump.sql"

Get database size

Specific database

SELECT pg_size_pretty(pg_database_size('databasename')) as fulldbsize;

All databases

SELECT
  t1.datname AS db_name,
  pg_size_pretty(pg_database_size(t1.datname)) AS db_size
FROM pg_database t1
ORDER BY pg_database_size(t1.datname) DESC;

Drop all tables in a database/schema

Selects all tables from a schema and creates DROP TABLE statements for each

SELECT
  'DROP TABLE IF EXISTS "' || tablename || '" CASCADE;'
from
  pg_tables WHERE schemaname = 'public';

XSCF: Setting a route

setroute -c add -n 0.0.0.0 -g <gateway-ip> xscf#0-lan#1

Read hardware sensors from Sun server

ipmitool -v -U username -H 192.168.1.1 sdr list all

Solaris

zone commands

zoneadm list -iv # on global zone, list all zones

Determine which package a file belongs to

pkgchk -l -p /path/to/file
grep filename /var/sadm/install/contents

SGD connection fails with "Failed to install SGD Client" on Debian 6 (64bit)

I was missing some 32-bit libraries. Install the ia32-libs package, then the problem should besolved.

user@host:~$ ldd .tarantella/tcc/4.50.937/ttatcc
linux-gate.so.1 =>  (0xf77d5000)
libX11.so.6 => /usr/lib32/libX11.so.6 (0xf769e000)
libXmu.so.6 => /usr/lib32/libXmu.so.6 (0xf7688000)
libXt.so.6 => /usr/lib32/libXt.so.6 (0xf7635000)
libXext.so.6 => /usr/lib32/libXext.so.6 (0xf7626000)
libSM.so.6 => /usr/lib32/libSM.so.6 (0xf761e000)
libICE.so.6 => /usr/lib32/libICE.so.6 (0xf7607000)
libpthread.so.0 => /lib32/libpthread.so.0 (0xf75ee000)
libdl.so.2 => /lib32/libdl.so.2 (0xf75ea000)
libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf75cb000)
libc.so.6 => /lib32/libc.so.6 (0xf7484000)
/lib/ld-linux.so.2 (0xf77d6000)
libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf746b000)
libuuid.so.1 => /lib32/libuuid.so.1 (0xf7467000)
libXau.so.6 => /usr/lib32/libXau.so.6 (0xf7464000)
libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf745e000)

macOS

Create GIF from screen recording

Without installing any application:

Record screen by hitting Shift ⌘ 5. When done, hit the key combination again and stop recording. Run ffmpeg (install with brew) to generate a palette:

ffmpeg -y -i source.mov -vf fps=20,palettegen palette.png

Now generate an animated GIF using the palette generated before:

ffmpeg -i source.mov -i palette.png -filter_complex "fps=20,paletteuse" out.gif

mpd: play next song matching ‘infernal war’

while true; do mpc next | grep -qi 'infernal war' && break; done

Put init script to standard runlevels

# Debian
update-rc.d nagios defaults
# RHEL
chkconfig on ntpd

Sort output by column (separator = $IFS)

ps aux | sort -nk 6

Convert .ts (MPEG-TS) files

mencoder sourcefile.ts -oac mp3lame -ovc lavc -lavcopts aspect=16/9 -o out.avi

OpenSSL

Display fingerprint of a certificate. "type" is md5, sha1, sha256 etc.

openssl x509 -fingerprint -noout -in newcert.pem -<type>

Get certificate for standard SSL-only connections

openssl s_client -connect foo.example.org:443

Get certificate for STARTTLS services

openssl s_client -connect foo.example.org:25 -starttls smtp -CApath /etc/ssl/certs
openssl s_client -connect mail.example.org:143 -starttls imap -CApath /etc/ssl/certs

Display local certificate details

openssl x509 -in /path/to/cert -text

Convert PKCS#7 certificate to PEM format

openssl pkcs7 -in pkcs7.file -text -out cert.pem -print_certs

Convert PKCS#12 (.pfx) to PEM

Host certificate:

openssl pkcs12 -in host.domain.p12 -clcerts -nokeys -out host.domain.cert.pem
openssl pkcs12 -in host.domain.p12 -nocerts -nodes -out host.domain.key.pem

User certificate:

openssl pkcs12 -in export.p12 -clcerts -nokeys -out cert.pem
openssl pkcs12 -in export.p12 -nocerts -out key.pem

Extract all:

openssl pkcs12 -in file.p12 -out cert.pem -nodes

The order in the output file is:

  1. Private key
  2. Identity certificate
  3. Root certificate
  4. Intermediate certificate

256-color terminal stuff

tput colors                 # get colors
export TERM=xterm-256colors # if installed
xrdb -load $HOME/.Xdefaults # to activate color changes in X terminals

Horde Webmail with IMAP proxy, thread sort error

If you encounter errors when opening folders sorted in a special way (by threads for example), just run (Horde Groupware 4+):

horde-clear-cache

Scan SCSI bus for new harddisks

If fdisk doesn't see new disks even after a partprobe, issue:

echo "- - -" > /sys/class/scsi_host/host#/scan

Show I/O operations of all drives

Raise number 5 at the beginning for more accurate results.

iostat 1 5 -xdnN | egrep "[a-zA-Z].*[0-9]\.[0-9][0-9][[:space:]]" | awk {'if ($1 ~ /:\//) print $9,$10,$1; else print $4,$5,$1'} | tail -n +2

Nagios/Icinga: convert timestamps in nagios.log

perl -pe 's/(\d+)/localtime($1)/e' /usr/local/nagios/var/nagios.log | tail -20

Kill and logout Linux shell user

The "skill" command is in the package "procps":

skill -KILL -u username

Linux ACL

Backup and restore Linux ACLs recursively

getfacl -R /dir/with/acls > /tmp/bkp.acl
setfacl --restore=/tmp/bkp.acl --test # omit --test if all is OK

Copy ACL from file/directory A to file/directory B

getfacl file1 | setfacl --set-file=- file2

SMTP: Test mail throughput with Postfix tools

To test the theoretical capable mail volume, use smtp-sink on the destination, which acts as an SMTP blackhole that accepts mail and throws it away. On the sender side, use smtp-source as a bulk mailer with various config options like parallel sessions, mail size etc.

# Destination:
smtp-sink -c -u postfix -M 10000 0.0.0.0:25 1024

# Source:
time smtp-source -s 20 -l 30000 -m 10000 -c -f sender@example.org -t receiver@example.org smtp-sink-address:25

Speed up Software RAID (md) resync

If one of your RAID devices has failed, you might be able to speed up the recovery (values are in Kb/s):

Check the current resync speed:

grep speed /proc/mdstat
[===>.................]  recovery = 19.6% (191587648/975193600) finish=459.7min speed=28404K/sec

Check the current minimal/max bandwidth defined:

cat /proc/sys/dev/raid/speed_limit_max
20000
cat /proc/sys/dev/raid/speed_limit_min
5000

Raise min/max

echo 80000 > /proc/sys/dev/raid/speed_limit_max
echo 40000 > /proc/sys/dev/raid/speed_limit_min

El-cheapo way to measure the runtime of a program

while [ $(ps -p <PID> -o etime= | grep -c ".*") -gt 0 ]; do echo "$(ps -p <PID> -o etime=)" >> /tmp/runtime.log; sleep 30; done

Git

Delete a local and remote tag

Local delete:

git tag -d <tag_name>

Remote delete:

git push --delete origin tagname

Rename a tag

Source: stackoverflow.com/questions/1028649/how-do-you-rename-a-git-tag

git tag new old
git tag -d old
git push origin :refs/tags/old
git push --tags

Add and push tag

git tag -a 0.0.8 <commitid> -m 'message'
git push --tags

nmap

Ping hosts in a subnet

nmap -sP 192.168.1.1-254

Get TLS ciphers from remote server

nmap --script ssl-enum-ciphers -p 443 www.example.com

Bacula

Reports (query.sql)

:Monthly report (current month)
SELECT
  JobId,
  StartTime,
  EndTime,
  Name,
  Level,
  JobErrors AS Errors
FROM
  Job
  LEFT JOIN Status on Job.JobStatus = Status.JobStatus
WHERE
  StartTime > DATE_FORMAT(
    now() - INTERVAL 1 MONTH, '%Y-%m-%d'
  )
ORDER BY
  StartTime ASC;

:Backup size (Full)
SELECT
  Client.Name,
  ROUND(
    AVG(Job.JobBytes)/ 1024 / 1024 / 1024,
    3
  ) AS "Average GB",
  ROUND(
    STDDEV(Job.JobBytes)/ 1024 / 1024 / 1024,
    3
  ) AS "Standard Deviation GB"
FROM
  Job
  INNER JOIN Client ON Job.ClientId = Client.ClientId
WHERE
  Level = 'F'
GROUP BY
  Client.Name
ORDER BY
  "Average GB" DESC;

Migrating VM containers

VMDK to RAW format

qemu-img convert -f vmdk -O raw centos64.vmdk centos64.img

RAW to VMDK format

qemu-img convert -f raw -O qcow2 centos64.dsk centos64.qcow2

VDI to RAW format

VBoxManage clonehd ~/VirtualBox\ VMs/fedora18.vdi fedora18.img --format raw

Image conversion

Append images resp. merge multiple images

convert is from ImageMagick. It will also convert to the selected destination format.

# horizontal merge
convert +append source1.png source2.jpg destination.tif
# vertical merge
convert -append source1.png source2.jpg destination.tif

Ansible

Loops for lists, dictionaries

Loop a list

iplist:
  - 192.168.5.0
  - 192.168.6.0
  - 192.168.7.0

{% for ip in iplist %}
  allow from {{ ip }}
{% endfor %}

Python/Django

Django: Create A-Z index in template

Source: http://stackoverflow.com/questions/3617041/a-z-index-django

{% ifchanged food.name.0 %} <h1>{{food.name.0}}</h1>{% endifchanged %}

Python: update all packages in virtualenv

pip install --upgrade $(pip list --outdated --format=json | jq -j '.[] | .name," "')

Similar posts