Archive for category Troubleshooting

Clustercomputing with mpirun and torque

Posted by on Wednesday, 18 July, 2012

Some time ago I wrote about Clustercomputing with Torque, with a focus on discrete event simulations (using the method of independent replications). Turns out that method was not really efficient, so we try something else. Read the rest of this entry »

Finding duplicate files

Posted by on Wednesday, 23 May, 2012

You have a bunch of files (for instance, jpegs). Over time they get moved around, you get a bunch from some family members and before you know it you have a situation where you have the same file multiple times. Of course you can manually sort out these duplicates, but you can also automate duplicate detection.

After a short search, I found this solution on LinuxQuestions.org:

tmp=$(mktemp)
find . -type f |xargs md5sum > $tmp
awk '{ print $1 }' $tmp |sort |uniq -d |while read f; do 
    grep "^$f" $tmp
    echo ""
done

This outputs a list of duplicate files once it has run to completion.

However, it borks when it encounters whitespaces, special characters like the apostrophe etc. A solution:

#!/bin/bash
tmp=$(mktemp)
find . -type f | sed -e "s/'/\\\'/g" |xargs  -I{} md5sum {} > $tmp
awk '{ print $1 }' $tmp |sort |uniq -d | while read f; do 
    grep "^$f" $tmp
    echo ""
done

Use the -I{} and {} to make sure the input to md5sum is not terminated by whitespaces, but only by endlines. Also, the “| sed -e “s/’/\\\’/g”” part replaces every occurence of the apostrophe “‘” with its escaped version “\'” as you would when entering it on the commandeline.

This is able to traverse deep into directory structures, and also accepts any filename I encountered in my dataset. It is however quite CPU intensive, as it calculates the MD5 hash for every file. If you only want to compare based on filename, the whole operation becomes a lot more lightweight.

Duplicate detection with locate/mlocate.db

Actually, it is not necessary to manually index all files, good chance this is already being done by the updatedb cronjob. For instance,

skidder@@spetznas:~$ locate fstab
/etc/fstab

and it also finds some other files containing the string fstab. Unfortunately, mlocate.db is a very simple list of filenames only – a file size and an MD5 would greatly ease the detection of duplicates. So far I have not found a way to do this more efficiently than the shellscript posted above.

Crashed Mac: rescue files with Linux

Posted by on Friday, 11 May, 2012

We have a crashed 2007 Macbook, which simply won’t boot anymore. The guys at the Mac store were not able to do anything; and gave the standard answer: buy a new one! So we did, and luckily we did have Time Machine backups, which we put on the newly bought Macbook Pro. Unfortunately, the last successfully completed backup was from a few months ago – the system had been gradually crapping up more and more, and apparently wasn’t even able to complete its backups for the last few months of its life.

Now the question is, how do we get all the files off of the disk? We use Linux. Read the rest of this entry »

Enigmail password trouble with OpenPGP

Posted by on Thursday, 12 April, 2012

I’m using Thunderbird and Enigmail on Kubuntu, signing all my email and encrypting correspondence whenever possible. Lately (probably due to an update) the password-entry window does not appear. In stead, it complains immediately about:

openpgpg alert - error bad passphrase

I have set the preferences to remember my password for several minutes, so I figured there’s some problem there. I am not using gpg-agent, so EnigMail should fall back to the default.

Checking OpenPGP > Debugging Options > View Console shows what Enigmail is doing. I type my message (signing option checked) and click Send. The console shows:

gpg: gpg-agent is not available in this session
gpg: can't query passphrase in batch mode

Which makes sense, I am not using gpg-agent. Finally, it complains:

gpg; [stdin]: clearsign failed: bad passphrase

And gives up. Googling around I find similar problems, but no solution. Then I noted the following line in the console:

gpg: NOTE: old default options file '~/.gnupg/options' ignored

The ‘options’ file has been obsoleted, gnupg uses gpg.conf now. I checked, indeed the contents of ‘options’ are also present in gpg.conf. Moving ‘options’ away and restarting Thunderbird solved the problem.

Moreover, it also solved my password problem, I now get the password prompt and can happily sign my email.

Bug in Mac80211 in MiXiM post-backoff

Posted by on Friday, 16 December, 2011

Post-backoff was introduced into IEEE 802.11 to make sure that a station could not claim the channel by noting that the channel was idle after its own transmission, and then immediately transmit again. The idea is quite simple; transmission over -> pull a backoff from the contention window and count down (respecting the same rules w.r.t. carrier sense and inter-frame spacing etc.) and finally end up in the idle state. If a new packet was handed down by the network layer during post-backoff, the countdown would continue and result in a transmission. This behaviour is also implemented in MiXiM’s Mac80211 (and prior to that in the Mobility Framework). However, there’s a bug. Read the rest of this entry »