Linux

Yet another goodbye

For those who are about to halt -p, we salute you.

I’m moving out – of my first vserver that is.

I have had the server running for, well

lucretia:~# uptime 
 19:19:59 up 700 days, 22:51,  2 users,  load average: 0.02, 0.06, 0.02

Wow.. Time flies.

I have been migrating the services running in the machine to another, server. Not is the time to do the inevitable.

Sorry old chap, better you than me.

lucretia:~# halt -p

Broadcast message from root@lucretia (pts/1) (Fri Sep 30 19:23:24 2011):

The system is going down for system halt NOW!
lucretia:~# Connection to lucretia.greenpc.dk closed by remote host.
Connection to lucretia.greenpc.dk closed.

Of course, prior to this, I made sure to make a complete copy of the file system like so. Isn’t rsync the best tool?

rsync --progress -poazuHK -e ssh --delete --exclude /proc --exclude sys --exclude dev / home.greenpc.dk:/mnt/primary/backup/lucretia.greenpc.dk

Proftpd and LDAP on Debian Squeeze

This is a short howto (hopefully) providing enough information to install Proftpd and use LDAP as user database.

Background

I have become obsessed with LDAP – at least for the time being. It seem to be the answer to my redundancy and distribution plans.

A production server is in the process of being converted (migrated actually) to have a single SSO LDAP structure.

A virtualization host crash (thank you Linode) forced me to move a couple of sites onto this new fancy LDAP server. Shortly after, a user prompted me about the lack of FTP on the new webhost.

Now the shoe needs to fit.

Installing the required packages

This is the easy part.

# apt-get install proftpd-mod-ldap

The LDAP module will depend on the proftpd server so this is really the only thing you need to install.

Requirements for the LDAP server

The LDAP module for Proftpd is hard coded to lookup only users of objectClass: posixUsers which in my opinion is less intuitive than having a specified schema for proftpd.

An example .ldif is shown below. I have added objectClass: domain, which is unnecessary.

The uidNumber and the gidNumber maps the uid and gid on the system. 115 is proftfd user and 65534 is group nobody. From a ftp client owner will appear as domain.tld or whatever you specify as uid.

version: 1

dn: dc=domain.tld,ou=webhosting,dc=example,dc=com
objectClass: domain
objectClass: top
objectClass: posixAccount
cn: domain.tld
dc: domain.tld
gidNumber: 65534
homeDirectory: /var/www/domain.tld/www
uid: domain.tld
uidNumber: 115
loginShell: /bin/false
userPassword::

Configuring the authentication

First you need to edit /etc/proftpd/ldap.conf to match you LDAP setup. Somthing like this is appropriate.

<IfModule mod_ldap.c>
  LDAPServer ldap://example.com/??sub
  LDAPDNInfo "cn=proftpd,dc=example,dc=com" "password"
  LDAPDoAuth on "ou=webhosting,dc=example,dc=com"
</IfModule>

notice the ??sub after the ldap. This is very important as it specifies the search scope. The configuration parameter LDAPSearchScope is apparently ignored.

Again, a sour comment; the bind should have been done as the user logging in, and not as a dedicated user. Admin is a bad choice – create a dedicated user. Besides, the /etc/proftpd/ldap.conf is world readable!

Next you have to tell proftpd to load the module.
Uncomment the line

LoadModule mod_ldap.c

in /etc/proftpd/modules.conf.

Now you have to uncomment the line.

Include /etc/proftpd/ldap.conf

in /etc/proftpd/proftpd.conf to load the Ldap configuration.

Finally:

While editing proftpd.conf you should also lift the RequireValidShell restriction (or give the user a valid loginShell parameter. If do not do this, you will not be able to log in.

Now is the time to take a look at the standard proftpd configuration and make sure that anonymous login is disabled and ditto /etc/passwd users.

HD time-lapse movies with Motion and Linux

NSLU2 with webcam
The system

Background

I have previously experimented with time lapse videos, but wanted a more dedicated platform which could be set up, and run pretty much anywhere.

This is the first iteration, where the the purpose is to get the system up and running with headless operation.

Components

The original idea was to use a PC Engines alix1d system board in a box1c enclosure, but unfortunately the board I had was running very unstable – so I brought in an old friend of mine:

The Linksys NSLU2 aka. “slug”. The one I had ran Debian 5.0 Lenny, but had to be upgraded in order to get the webcam to work.

I recklessly tried doing a dist-upgrade, but ended up with bricked slug. Guess a fresh installation was the right answer indeed.

Debian Squeeze on a NSLU2

Due to a required proprietary firmware, the official Debian 6.0 installer does not ship with support for the on board Ethernet controller – which is bad because this is the only way of communicating the the device. Well, technically you can use the serial pin header or an USB Ethernet device, but I think I have burned the circuit for the serial port in a previous modding attempt :-\

There is a few guides that  give you directions on how to add the proprietary firmware to the installer image, and after about 5 reflashes I finally had one that worked.

Before starting the installation, I checked around for known installation errors. The installation takes about 5 hours, so you really want to get i right the first time.

I learned that others had experienced out of memory errors during the installation. Though luck.

To the rescue came Martin Michlmayr. He has the answer to all my quarrels; a compiled guide, with a complete Debian 6 userspace and kernel. This saved me a lot of time.

 Install and configure Motion

You can install motion by

apt-get install motion

as root or via sudo.

On Debian (Squeeze in my case), Motion is disabled by default – as many other services. Enable it, as mentioned in the notice:

Not starting motion daemon, disabled via /etc/default/motion ... (warning).

Setting the value start_motion_daemon to yes in /etc/default/motion as such:

start_motion_daemon=yes

 

The trick to disable motion detection in Motion, is to set the threshold to 0 in the config file:

threshold 0

Enabling time-lapse by setting the following in /etc/motion/motion.conf:

# Use ffmpeg to encode a timelapse movie
# Default value 0 = off - else save frame every Nth second
ffmpeg_timelapse 10

In this case, I take a pictures every ten seconds.

You should also adjust the width and height parameters, and the target_dir.

You can also get a copy of my preconfigured motion.conf by running the following set of commands

/etc/init.d/motion stop
mv /etc/motion/motion.conf /etc/motion/motion.conf.orig
wget http://retrospekt.dk/files/motion.conf -O /etc/motion/motion.conf
mkdir /home/motion
chown motion:motion /home/motion
chown root:motion /etc/motion/motion.conf
chmod g+r /etc/motion/motion.conf
/etc/init.d/motion start

An example can be seen here: http://retrospekt.dk/files/timelapse.mpg

MBUS and Ada

I recently got my hand on a OM13006 Power Plug meter development kit from NXP Semiconductors. The entire process of actually getting the kit was a quite cumbersome one, and tale left for another time. Though I would like to thank the people at Future Electronics for their invaluable help.

So, now I have a kit which i have spent €200+ on, and no idea how to program it or use it. The smart metering website at NXP has an SDK zip file which you can download. It contains some diagrams and a lot of C/C# source code respectively. More importantly, it contains the (windows) software also depicted on their website.

The software is pretty functional, but lacks an automated log function (besides raw bus data), so you can grab usages and use them in a software model.

A challenge, then!

As the kit consisted of the a meter to plug between your outlet and device, and a USB dongle, the first step was to figure out how to communicate with the device.

After some spent digging through the supplied source code, I realized that the device was interfacing as a serial device. A quick check with minicom confirmed this.

By then, it was just a matter of parsing the line (ASCII encoded) based protocol.

The sources can be found here, or by going to the projects page.

To build the test application, you need to use gnatmake which is part of GNAT. It is part of any respectable Linux distributions repository. When it is installed, just run

gnatmake -P mbus_logger

To build the sample application

The curse of being a geek

I am currently in Spain working on a EU funded project called Developing Open Source Systems Expertise in Europe – or just DOSSEE (the certificate is invalid).

The first day after I arrived, I saw a bus with the following written on it:
20110318_002.jpg

I think I will leave the punch line to be deducted by the real Linux nerds.

I remembered that I was very please to see a graffiti tag back in Denmark that apparently promoted the use of the Gnu Compiler Collection 😉

A lesson in recovery techniques

I recently got this message from fsck.jfs:

Unrecoverable error writing M to /dev/sdb3. CANNOT CONTINUE.

Okay, so this is an error that can be ignored – right? I can just force mount the partition and extract the data with superblock marked as dirty … right?!

krc@X61s % mount -o ro -f /dev/sdb3 /mnt/rec_mount
krc@X61s % ls /mnt/rec_mount
krc@X61s %

Damn it! This was an 1,4 TB parition with 900GB of data including home videos and .mkv rips of my dvd’s. Most of data could be restored, but a lot work would be lost.

I am running JFS on all my storage drives, as I have found this a good all-round file system especially in smaller devices with limited resources. Unfortunately this a kind of niche file system that does not have a broad variety of recovery tools.
I found jfsrec as the only (non commercial) tool. Unfortunately this tool was unable to read from the partition directly and stopped with an early EOF marker error.

Jfsrec pointed me in the direction of the dd_rhelp tool. This tool turned out to be a life saver. There was just one thing. I needed a disk big enough to hold a complete dump of the partition.

A few days later, armed with a new disk, I was able to continue. I used this guide at debianadmin.com to get started. The command could not be simpler to use:

krc@X61s % dd_rhelp /dev/sdb3 /mnt/rec_target/bad_disk.img

And it started copying data! Yay!
After some time, it settled on a transfer rate of 2500 … KBps! … Wow… This is rather slow…
Quick calculation: (((1400000000)/2500)/3600)/24 = 6.48 days.

One week later:

krc@X61s % ssh atom1
ssh: connect to host atom1 port 22: No route to host

Hmm… I had done this periodically over the last week

krc@X61s % ping atom1
PING atom1 (172.16.0.122) 56(84) bytes of data.
From atom1 (172.16.0.122) icmp_seq=1 Destination Host Unreachable
From atom1 (172.16.0.122) icmp_seq=2 Destination Host Unreachable
From atom1 (172.16.0.122) icmp_seq=3 Destination Host Unreachable
From atom1 (172.16.0.122) icmp_seq=4 Destination Host Unreachable
^C
--- atom1 ping statistics ---
6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5059ms

Hmm… Thats odd. I didn’t remember putting a ; halt -p after the dd_rhelp command.

A few pings and some reflections later I acutally got up and checked the room where the recovery setup is located.

This was what I found:

20110218-154606_redone.jpg

To quote Freddie Frinton;

I’ll kill that cat!

20110218-154628_redone.jpg
Notice the dangling sata power cables in the top of the photo… I have always found Linux a stable operating system, but a system disk physically disappearing is valid excuse for a crash!

Fortunately, dd_rhelp got to finish the disk dump – which was very lucky because after the fall, the damaged disk is now officially dead. It no longer spins up, and is not recognised by bios.

I tried running a fsck.jfs directly on the disk image, and it managed to fix the errors in the partition. Now i could mount the disk image like so:

krc@X61s % sudo mount -o loop /mnt/rec_target/bad_disk.img /mnt/rec_target

And copy the files from /mnt/rec_target.

Whew!

EXIF Manipulation on the Linux commandline

I use jhead for manipulating EXIF data, renaming files and mogrify for
compressing the JPEG’s

the following command will rename all .jpg files in the current folder
to their corresponding “date taken” stamp in the format YYYYMMDD-HHMMSS

jhead -n%Y%m%d-%H%M%S *.jpg

Sometimes you have multiple cameras at a happy occasion, and wish to
collect all the images. But alas, one or more of the cameras have
their clock set to UTC, daylight savings time (winter when it is summer)
or similar. This will create annoyance as the images will not appear
chronologically.
fortunately you kan find the “sinner” and adjust the time with jhead
in a this way:

jhead *.jpg | grep "IXUS 700" -b4 | \
grep "File name" | sed "s/.*: //" | xargs jhead -ta+1

WARNING: This step is optional, just because I cannot tell the difference
between compressed and uncompressed images, maybe you can. A backup is
therefore recommended

The compression on digital cameras are not very efficient, and this
leads to large JPEG image files on your harddisk.
Fortunately you can use “mogrify” to cut JPEG’s down in size with

jhead -cmd "mogrify -quality 80 &i" *.jpg

How to install LabVIEW on a Debian Machine

Labview ships prepackaged to install on rpm based Linux machines (Redhat, Mandrake and so on). But it is quite simple to convert and install it on a Debian machine instead.

The machine I will be using is a Thinkpad T40 with Debian Lenny installed, it should be the same for the current stable (Etch) but this is yet to be confirmed.

First we need to install alien

# aptitude install alien

The conversion process is quite simple, all you have to is to type

# alien -d *.rpm

And after some time (about 15 minutes or so on my laptop) you should be left with a bunch of .deb files.

You might have guessed it.. to install:

# dpkg -i *.deb

Now for the hacking part, when you try to run Labview all you get is this error:

/usr/local/natinst/LabVIEW-8.2/labview: symbol lookup error: /usr/local/natinst/LabVIEW-8.2/linux/libOSMesa.so.4: undefined symbol: _glapi_add_entrypoint

To work around this you need to install libosmesa (of current writing libosmesa6)

# aptitude install libosmesa6

and relink

# (cd /usr/local/natinst/LabVIEW-8.2/linux; rm libOSMesa.so.4; ln -s /usr/lib/libOSMesa.so.6 libOSMesa.so.4)

And it works! If you feel like it, you can also put a little icon on your desktop or in your Applications menu (this part is for gnome), this is the contents of the labview.desktop file with my corrections. It originates from /usr/local/natinst/LabVIEW-8.2/linux/gnome/gnome/apps/Applications/labview82.desktop and there is a similar file for KDE in the /usr/local/natinst/LabVIEW-8.2/linux/kde folder for those who want to make a KDE shortcut.
Contents of labview82.desktop

[Desktop Entry]
Name=LabVIEW 8.2
Comment=LabVIEW Graphical Dataflow Programming Environment
Exec=/usr/local/natinst/LabVIEW-8.2/labview
Icon=/usr/local/natinst/LabVIEW-8.2/linux/icons/labview-3d.xpm
Terminal=false
Type=Application
Categories=Application;Development;X-Red-Hat-Base

This file can be placed either in ~/.local/share/applications/ or in /usr/share/applications