Category: UNIX

How to concatenate strings efficiently in bash

The naive implementation for concatenating strings in bash goes something like this:

#!/bin/bash

statement="1234567890"

result=""

for i in $(seq 1 100000)
do
  result=${result}${statement}
done

wc <<< ${result}

This takes around two minutes on a modern machine. This is slow. Very, very slow.

Instead you can build an intermediate array and use the star operator to expand it into a string. Make sure to change the input field seperator to empty, so that you don’t get spaces in between the individual entries:

#!/bin/bash

statement="1234567890"

for i in $(seq 1 100000)
do
statements[${#statements[@]}]=${statement}
done

IFS= eval 'result="${statements[*]}"'

wc <<< ${result}

This will run in a few hundred milliseconds. You get roughly three orders of magnitude speedup.

 

Forwarding emails using fetchmail and msmtp

My goal here was to forward emails from my GMX freemail account to my iCloud account. Up until now, I used GMX’s own forwarding capability, which is a bit hidden in the filters settings. However, iCloud cranked up its spam filtering, and is now using spamhaus blacklists, which very often label the GMX forwarding servers as bad.

Hence I would only get bounce mails instead of the actual mails. Since this is no good, I set up fetchmail and msmtp on my root server to do the forwarding for me. First, fetchmail will get all mail on GMX via POP3 and pass it on to msmtp, which will in turn pass it on to the iCloud mx server.

At first I tried to deliver it via authenticated SMTP, but iCloud refuses mails sent this way, if the header from field does not contain any of your own iCloud aliases. This will most of the time be a problem, since we are trying to forward emails that were sent to you, not sent from you.

So first let’s see the ~/.fetchmailrc (make sure to chmod 0600 it):

poll pop.gmx.net
with proto POP3
user "user@gmx.net"
there with password "secretpassword"
mda "/usr/bin/msmtp -- someuser@icloud.com"
options
no keep
ssl
sslcertck
sslcertpath /etc/ssl/certs
set daemon 300

This will poll GMX every 300 seconds and pass the received mails to msmtp for delivery to someuser@icloud.com.

The corresponding ~/.msmtprc looks like this:

account default
host mx6.mail.icloud.com
port 25
auto_from off
from "user@localdomain"
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log
domain mx.of.localdomain

You can find out the valid mx entries for iCloud by running nslookup -type=mx icloud.com.

The settings above are assuming Debian stable. Other distributions or operating systems may have the SSL certs at different places in the file system.

Getting Emacs for Mac OS X to honour the shell environment

If your are using the excellent Emacs for Mac OS X distribution, you may have noticed that it does not use the environment you may have defined in your ~/.bashrc. This can be a problem if you are using MacPorts, for example to install clojure and Leiningen. You will get an error message like this, when trying to run inferior-lisp in clojure-mode:

apply: Searching for program: No such file or directory, lein

This can be avoided by using the excellent exec-path-from-shell package for Emacs. It is available via (M)ELPA, just install it using list-packages, or download it from github. You can enable it, especially for OS X, by putting this in your init.el:
(when (memq window-system '(mac ns))
    (exec-path-from-shell-initialize))

How to use find and rsync to copy a bunch of files to a local or remote destination

I use rsync a lot. And I use find a lot. You can use them both together, by using the magnificent xargs command. Two problems are there to solve: xargs usually just pastes the arguments from find at the end of the specified command. No way to specify the destination directory. Second, file names with spaces are a problem.

The solutions are:

  • The -print0 option of find uses 0-bytes to separate results, instead of spaces and newlines
  • The -0 option of xargs tells xargs to look for said 0-bytes
  • The -J option of xargs allows us to tell xargs where to put the file names that we feed to it.

So let’s put it together:

pi@raspberrypi ~ $ find . -name ‘your*pattern.jpg’ -print0 | xargs -J % -0 rsync -aP % user@host:some/dir/

Thanks to redeyeblind for the insightful blog post.

How to resize tmux windows using OS X terminal

A while back I pointed out a blog post by someone else, which described how to get C-left and C-right working in tmux and OS X. Now I noticed that I cannot resize windows in tmux. This is because tmux seems to expect xterm keys for C-arrow. The xterm keycodes for the arrow keys are:

  • left key: ^[[D
  • right key: ^[[C
  • up key: ^[[A
  • down key: ^[[B
  • C-left: ^[[1;5D
  • C-right: ^[[1;5C
  • C-up: ^[[1;5A
  • C-down: ^[[1;5B

I can now configure OS X terminal to send those key codes, and tmux works fine. However, other apps running in tmux will break, because they don’t expect to get xterm key codes. I found a workaround in the ArchWiki, which suggests to create your own terminfo entry. I will try that and report back here.

Update: The solution is adding two lines to the ~/.tmux.conf file:

set -g default-terminal “xterm-256color”
setw -g xterm-keys on                   
Update 2: And here is my OS X Terminal.app configuration:

Update 3: And a very last update… It seems that tmux does not support bce (background color erase), which xterm does. This is a problem for progams like htop, vim, or mc. You will see rendering errors, if you do not fix this.

So you need to make your own terminfo file and own terminal type, called xterm-256color-nobce. You can do this on the command line:

pi@raspberrypi ~ $ infocmp xterm-256color | sed ‘s/bce, //’ | sed ‘s/xterm-256color/xterm-256color-bce/g’ > xterm-256color-nobce

This will create a new terminfo source file, which does not advertise the bce feature. You can install this with the following command:

pi@raspberrypi ~ $ sudo tic ./xterm-256color-nobce

This makes the terminal type xterm-256color-nobce available. After this, change your ~/.tmux.conf once again, to use the new terminal type per default:

set -g default-terminal “xterm-256color-nobce”

Update 4: To make the Ctrl+Arrow keys also work in regular OS X Terminal, you need to edit or create ~/.inputrc to contain this:

# xterm keys for skipping a word
“e[1;5C”: forward-word
“e[1;5D”: backward-word
“e[5C”: forward-word
“e[5D”: backward-word

The “1;” variant is otherwise not recognized by GNU readline on OS X.

How to run tmux via ssh instantly

With my Raspberry Pi, what I do very, very often is this:

localhost$ ssh raspberrypi.local    # Here I already type the next command and wait a while
raspberrypi$ tmux attach

This is all well and good, but sometimes the Pi is down, and I will attach to one of my local tmux sessions. Very annoying. Instead you could try to do this:

localhost$ ssh raspberrypi.local tmux attach
not a terminal

Well, that did no good. So a look at the man-page of ssh or a quick search reveals this gem:

localhost$ ssh raspberrypi.local -t tmux attach

This allocates a pseudo terminal, which is needed by tmux to function correctly. This is also done by ssh, if no command is given, but a login shell is spawned.

Some tmux cheats

There’s a lot of stuff you can do with tmux. Here are some nice to know things:

  • C-b C-o: cycle contents of current windows (or swap if there are only two windows)
  • C-b C-SPC: switch between vertical and horizontal split
  • C-b n, p: next or previous screen
  • C-b [: copy mode
    • Use Emacs bindings to copy and paste:
    • C-SPC: begin selection
    • C-w or M-w to copy
  • C-b ] to yank (paste)

    How to copy Mails from iCloud to GMail

    You can use the fabulous imapsync tool to copy mails between IMAP servers. For example you can copy a certain folder from Apple’s iCloud to Google’s gmail: 

    imapsync 
       --noauthmd5 --ssl1 --ssl2 
       --host1 mail.me.com --user1 'your.icloud.name' 
       --host2 imap.gmail.com --user2 'your.gmail.name@googlemail.com' 
       --folder 'your/folder/to/be/copied' --sep1 '/' 
       --prefix1 '' --prefix2 '[Google Mail]' --sep2 '/'

    The important parts here are the user names for the IMAP servers. Note that you need to generate an application specific password, if you are using Google two factor authentication! Also important is the “[Google Mail]” IMAP prefix.

    Edit: It seems gmail has a weird interpretation of all the IMAP folders and stuff. Since they are using labels, the above script might create a weird label for the copied emails, but they will be there nevertheless!