Category: Linux

How to concatenate strings efficiently in bash

The naive implementation for concatenating strings in bash goes something like this:

#!/bin/bash

statement="1234567890"

result=""

for i in $(seq 1 100000)
do
  result=${result}${statement}
done

wc <<< ${result}

This takes around two minutes on a modern machine. This is slow. Very, very slow.

Instead you can build an intermediate array and use the star operator to expand it into a string. Make sure to change the input field seperator to empty, so that you don’t get spaces in between the individual entries:

#!/bin/bash

statement="1234567890"

for i in $(seq 1 100000)
do
statements[${#statements[@]}]=${statement}
done

IFS= eval 'result="${statements[*]}"'

wc <<< ${result}

This will run in a few hundred milliseconds. You get roughly three orders of magnitude speedup.

 

ScummVM fullscreen on a Raspberry Pi TFT touchscreen

I managed to get ScummVM running on the 3.5″ Waveshare touchscreen. This is obviously only nice, if it runs fullscreen. Have a look:

For this to work, you need to adjust your openbox configuration, assuming you are running a default Raspbian with LXDE / openbox. Edit .config/openbox/lxde-pi-rc.xml and add the following to the applications section:

   <application class="scummvm" name="scummvm">  
     <fullscreen>yes</fullscreen>  
   </application>

You can switch between fullscreen and windowed mode by pressing Alt+F11, in the default openbox configuration.

Next, you need to install scummvm and the OpenGL software rasterizer, since the touch screen frame buffer does not support the hardware OpenGL engine that the Pi provides:

$ sudo apt-get install scummvm mesa-utils libgl1-mesa-dri libgl1-mesa-swx11

This will also install dri drivers and some OpenGL utilities, such as glxgears to test performance.

Now you want to tell ScummVM to actually use the OpenGL backend driver, or else it will not scale to, but rather use the default 2x scaler, which is too large for the 480×320 display. Easiest way to do this is to edit ~/.scummvmrc (you have to have run ScummVM at least once for this) and edit the [scummvm] section to have these lines:

[scummvm]
last_fullscreen_mode_width=480
last_fullscreen_mode_height=320
gfx_mode=opengl_nearest

If gfx_mode or the other lines already exist, edit them to your liking.

Installing an ads7846 based Raspberry Pi TFT touchscreen (Waveshare 3.5″ LCD)

I have a Waveshare 3.5″ TFT touch screen for the Raspberry Pi. Those displays come with some software to help set it up. Sadly, this only works for Raspbian based on Debian 7. The new Raspbian Debian 8 (Jessie) does not play well with this setup. So here is what I did to make it work.

IMG_1808First of all, you need to upgrade the kernel on your brand new Jessie installation:

$ sudo rpi-update
$ sudo power off

I had to do the power off, or else the Pi would show only a black screen after reboot. YMMV.

Next up, we need to change the boot parameters, i.e. the /boot/config.txt and /boot/cmdline.txt. Add the following to the end of /boot/cmdline.txt:

fbcon=map:10 fbcon=font:ProFont6x11 logo.nologo

This chooses a smaller font for the framebuffer console and turns off the boot logo.

Second, add the following lines to the end of /boot/config.txt:

dtparam=audio=on
dtparam=spi=on
dtoverlay=ads7846,cs=1,penirq=17,penirq_pull=2,speed=1000000,keep_vref_on=1,swapxy=1,pmax=255,xohms=60,xmin=200,xmax=3900,ymin=200,ymax=3900
dtparam=i2c_arm=on
dtoverlay=w1-gpio-pullup,gpiopin=4,extpullup=1

Next step is setting up all the kernel modules. Make sure /etc/modules contains exactly these line:

spi-bcm2835
snd-bcm2835
i2c-bcm2708
i2c-dev
ads7846
flexfb
fbtft_device

The options for these modules have to be put in a file called /etc/modprobe.d/lcd.conf:

options flexfb  width=320  height=480  regwidth=16 init=-1,0xb0,0x0,-1,0x11,-2,250,-1,0x3A,0x55,-1,0xC2,0x44,-1,0xC5,0x00,0x00,0x00,0x00,-1,0xE0,0x0F,0x1F,0x1C,0x0C,0x0F,0x08,0x48,0x98,0x37,0x0A,0x13,0x04,0x11,0x0D,0x00,-1,0xE1,0x0F,0x32,0x2E,0x0B,0x0D,0x05,0x47,0x75,0x37,0x06,0x10,0x03,0x24,0x20,0x00,-1,0xE2,0x0F,0x32,0x2E,0x0B,0x0D,0x05,0x47,0x75,0x37,0x06,0x10,0x03,0x24,0x20,0x00,-1,0x36,0x28,-1,0x11,-1,0x29,-3
options fbtft_device debug=3 rotate=90 name=flexfb speed=16000000 gpios=reset:25,dc:24
options ads7846_device model=7846 cs=1 gpio_pendown=17  keep_vref_on=1 swap_xy=1 pressure_max=255 x_plate_ohms=60 x_min=200 x_max=3900 y_min=200 y_max=3900

Next up are the X11 configurations. First, add a file /etc/X11/xorg.conf.d/99-calibration.conf with the following content:

Section "InputClass"
    Identifier "calibration"
    MatchProduct "ADS7846 Touchscreen"
    Option "Calibration" "126, 3734, 3892, 199"
    Option "SwapAxes" "1"
EndSection

You also need to edit /usr/share/X11/xorg.conf.d/99-fbturbo.conf. It should already exist, and you need to make it have this content:

Section "Device"
        Identifier      "Allwinner A10/A13 FBDEV"
        Driver          "fbturbo"
        Option          "fbdev" "/dev/fb1"
        Option          "SwapbuffersWait" "true"
EndSection

The change is setting the fbdev to /dev/fb1, which is the touchscreen, instead of /dev/fb0, which is the HDMI output.

Last but not least, you may want to install the input calibration tool:

$ sudo apt-get install xinput-calibrator

After all this, reboot and hopefully the Pi will reboot into a tiny X11 session on the touch screen.

You can calibrate the display by running this, e.g. via an ssh session:

$ DISPLAY=:0 xinput_calibrator 
Calibrating EVDEV driver for "ADS7846 Touchscreen" id=6
        current calibration values (from XInput): min_x=3932, max_x=300 and min_y=294, max_y=3801

Doing dynamic recalibration:
        Swapping X and Y axis...
        Setting calibration data: 2763, 2763, 2942, 2978
        --> Making the calibration permanent 
  copy the snippet below into '/etc/X11/xorg.conf.d/99-calibration.conf' (/usr/share/X11/xorg.conf.d/ in some distro's)
Section "InputClass"
        Identifier      "calibration"
        MatchProduct    "ADS7846 Touchscreen"
        Option  "Calibration"   "2763 2763 2942 2978"
        Option  "SwapAxes"      "1"
EndSection

You simply have to replace the calibration and swap axes lines into your /etc/X11/xorg.conf.d/99-calibration.conf, which we created earlier.

How to setup mDNS lookups on the Raspberry Pi

I’ve got the new Raspberry Pi 2, and was setting it up the other day. The first thing that annoys me with vanilla Debian installations is that they don’t have mDNS/zeroconf/avahi enabled by default. This technology is very useful, since it helps you to advertise services on you local network, lets you resolve host names without the need for setting up a DNS server and much more.

Especially the convenience of not having to remember IP-addresses for your machines is worth the work to set this up. With DHCP you might not even get the same IP for every machine every time.

For this to work, I asked a question over at the Linux & Unix StackExchange. So parts of this blog entry are taken from there.

First, you might want to install avahi on your RasPi;

sudo apt-get install avahi-daemon

This should help with the Pi being resolvable by name from other machines — which also have to support mDNS. For example Macs will come with mDNS-lookup enabled. So you should be able to ping your Pi just by using its name plus the local-domain:

ping raspberrypi.local

Next, you want to install the client side name service support for mDNS:

sudo apt-get install libnss-mdns

Make sure that the /etc/nsswitch.conf contains this line:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

There is probably already a line starting with “hosts:”, which you can simple comment out with the #-sign.

For added convenience, you may want to add the sshd to the advertised services of avahi. Simply add a file /etc/avahi/services/ssh.service containing the following lines:


<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
     <type>_ssh._tcp</type>
     <port>22</port>
  </service>
</service-group>

This should let you use mDNS on you Pi, see the advertised services on you other machines in the local network. If you are using Plex Media Server (see this great post), it will also utilize avahi to advertise its services.

Autocompletion for C and C++ with Emacs 24

I have already done a blog post on auto completion using Emacs. But that was back in Emacs 23 days. Long ago…

Since then a lot has happened. Emacs 24 has been released, package managers like MELPA or ELPA have become standard, and company-mode seems to be winning against auto-complete. Also, clang has made huge strides forward.

So it is time to revisit the task of developing C or C++ using Emacs. I have put online an easy-to-install Emacs init.el that you can use as a start for your own development environment. I am using the OS X version of Emacs, but this should also work on Linux, given that you have clang and git installed.

You can install this init.el by issuing the following commands:

cd
git clone https://github.com/root42/yet-another-emacs-init-el
mkdir ~/.emacs.d
cd ~/.emacs.d
ln -snf ../yet-another-emacs-init-el/init.el .

The file will make sure that upon startup of Emacs all necessary packages are installed, like company, magit but also LaTeX tools like aucTeX or refTeX. You can disable this check (or individual packages) in the init.el.

The code is up on github, so feel free to fork and/or contribute.

When doing simple C++ programming using only standard libraries, you should be ready to go. For more complex projects, you have to tweak the clang parameters, so that the compiler will find the header files. Completion happens automatically after “.“, “->” and “::” but also when pressing M-/. You can rebind this key in the init.el, of course. This is what the completion looks like:

My most commonly used keys that I have re-bound are as follows:
  • f3 – Runs ff-find-other-file, trying to switch between header and implementation for C/C++ programs.
  • f4 – Toggles the last two used buffers.
  • f5, f6 – If tabbar is enabled (tabbar-mode), navigates back/forward through tabs.
  • f7 – Toggle ispell dictionaries (german/english).
  • f8 – Kill current buffer.
  • f9 – Run compile.
  • M-? – Run grep.
  • M-n – Go to next error in compilation buffer.
  • M-S-n – Go to previous error in compilation buffer.
  • M->, M-< – Go to next/previous Emacs frame.
  • M-/ – Run autocompletion using company mode.
  • C-x o, C-x C-o – Go to next/previous Emacs window

Very nice is also the magit-mode, which is a very sane interface to git for Emacs. It looks like this:

You can run it by executing M-x magit-status. Just type ? to get online help. Magit uses simple one-character-commands, like s for stage, c for commit, p for push and so on.

Currently I am evaluating the integration of lldb into Emacs, but haven’t come far enough to say that I have found a powerful and flexible interface, apart from the standard command line. So there’s more to come, hopefully!

How to backup your Raspberry Pi SD-Card

In the following I will explain on how you can backup your Pi’s SD-card, so that whenever it breaks down, you can simply restore the backup image to a fresh SD-Card. This may also help with tinkering, when you totally screwed up your Pi’s Linux-installation. I will explain the first step specifically for Mac OS X, but you can do this similarly on other UNIXes and Linux, by using the mount and umount commands.

So, shut down your Pi using the command sudo shutdown -h now and remove the SD-card after the Pi has done so. Take the card and insert it into your Mac’s SD-card slot.

First of all, we need to find out which disk device has been assigned to the card. We can do this with diskutil list:

user@mymacintosh:~ $ diskutil list                                                                
/dev/disk0                                                                                            
   #:                       TYPE NAME                    SIZE       IDENTIFIER                        
   0:      GUID_partition_scheme                        *512.1 GB   disk0                             
   1:                        EFI EFI                     209.7 MB   disk0s1                           
   2:                  Apple_HFS M4                      511.1 GB   disk0s2                           
   3:                 Apple_Boot Recovery HD             784.2 MB   disk0s3                           
/dev/disk1                                                                                            
   #:                       TYPE NAME                    SIZE       IDENTIFIER                        
   0:     FDisk_partition_scheme                        *15.9 GB    disk1                             
   1:             Windows_FAT_32                         58.7 MB    disk1s1                         
   2:                      Linux                         15.9 GB    disk1s2                           

Then unmount the card using diskutil unmountDisk:

user@mymacintosh:~ $ diskutil umountDisk /dev/disk1                                              
Unmount of all volumes on disk1 was successful                                                     

Next we will use dd to produce an image, which we can compress using bzip2, or pbzip2 for more performance. You can use MacPorts to install pbzip2.

user@mymacintosh:~ $ sudo dd if=/dev/disk1 | pbzip2 > raspberry_pi_$(date “+%Y-%m-%d”).img.bz2 
31116288+0 records in                                                                                                           
31116288+0 records out                                                                                                        
15931539456 bytes transferred in 2347.692010 secs (6786043 bytes/sec)                                                           

Restoring the image is also easy:

user@mymacintosh:~ $ bunzip2 raspberry_pi_2014-03-22.img.bz2 | sudo dd of=/dev/disk1      

How to use find and rsync to copy a bunch of files to a local or remote destination

I use rsync a lot. And I use find a lot. You can use them both together, by using the magnificent xargs command. Two problems are there to solve: xargs usually just pastes the arguments from find at the end of the specified command. No way to specify the destination directory. Second, file names with spaces are a problem.

The solutions are:

  • The -print0 option of find uses 0-bytes to separate results, instead of spaces and newlines
  • The -0 option of xargs tells xargs to look for said 0-bytes
  • The -J option of xargs allows us to tell xargs where to put the file names that we feed to it.

So let’s put it together:

pi@raspberrypi ~ $ find . -name ‘your*pattern.jpg’ -print0 | xargs -J % -0 rsync -aP % user@host:some/dir/

Thanks to redeyeblind for the insightful blog post.

How to setup an OpenWRT router as a WiFi bridge to an Ethernet router

You can use an OpenWRT router or access point to connect WiFi enabled devices to a router, which only has wired Ethernet. For this to work, I am assuming you already have an access point or router running OpenWRT, in this case version 12.09, Attitude Adjustment.
Log into your router, using the LuCi frontend, and go to the Network/Interfaces tab:

There you should see your LAN device. Edit it to have an appropriate IP address from your local subnet. Most often your network will be 192.168.0.0 and your existing router will have the IP 192.168.1.1. But your mileage may vary…

Lets put in a static IP address, so we can find our router in case something goes wrong. Also make sure to set the netmask (in this case 255.255.255.0), gateway and DNS server (both probably should point to your router, 192.168.1.1).

Now go to the Physical Settings tab. Here, it’s important to check “Bridge interfaces” and to select both the ethernet adapter, most likely eth0, and the wireless network. One of the ethernet devices will say “wan”, if your are using a router instead of an access point for this. You don’t want that device.

Hit “Save & Apply” when you are ready. And be sure to have the de-bricking guide ready, if something goes wrong…

How to run tmux via ssh instantly

With my Raspberry Pi, what I do very, very often is this:

localhost$ ssh raspberrypi.local    # Here I already type the next command and wait a while
raspberrypi$ tmux attach

This is all well and good, but sometimes the Pi is down, and I will attach to one of my local tmux sessions. Very annoying. Instead you could try to do this:

localhost$ ssh raspberrypi.local tmux attach
not a terminal

Well, that did no good. So a look at the man-page of ssh or a quick search reveals this gem:

localhost$ ssh raspberrypi.local -t tmux attach

This allocates a pseudo terminal, which is needed by tmux to function correctly. This is also done by ssh, if no command is given, but a login shell is spawned.

How to copy Mails from iCloud to GMail

You can use the fabulous imapsync tool to copy mails between IMAP servers. For example you can copy a certain folder from Apple’s iCloud to Google’s gmail: 

imapsync 
   --noauthmd5 --ssl1 --ssl2 
   --host1 mail.me.com --user1 'your.icloud.name' 
   --host2 imap.gmail.com --user2 'your.gmail.name@googlemail.com' 
   --folder 'your/folder/to/be/copied' --sep1 '/' 
   --prefix1 '' --prefix2 '[Google Mail]' --sep2 '/'

The important parts here are the user names for the IMAP servers. Note that you need to generate an application specific password, if you are using Google two factor authentication! Also important is the “[Google Mail]” IMAP prefix.

Edit: It seems gmail has a weird interpretation of all the IMAP folders and stuff. Since they are using labels, the above script might create a weird label for the copied emails, but they will be there nevertheless!