Letting go

It pains me to let my N900 go. We have been together for almost two years now – but I have to move on.

I can still remember the unwrapping of the shiny Nokia box back in that gentle, fresh late-autumn morning back in November ’09.
Moving from a Sony Ericsson K800i, this was a large leap for me, and the N900 would be the phone that introduced me to the world of smartphones.

Suddenly I could do anything! I had a big screen where I could surf the high seas of the treacherous internet. And even with Flash content.
I was able to check anything, anywhere. I was king of information and technology.

As the N900 was build around my favourite Linux distribution; Debian, I felt right at home. A had a busybox and and xterm. This was not just a phone – it was a mini-computer with a built-in phone.

N900 and me hit it off – big time. We went everywhere together, did everything together.

Until…

I was appointed project manager of an Android programming project. I obviously needed a device for the development process (at least that was what I told myself).
For that purpose I bought a HTC Wildfire. This was the first step towards the end.

I could not help being drawn towards the much smoother interface of the HTC phone, despite the lower resolution on the Wildfire compared to the N900.

Suddenly my N900 did not seem so smart anymore. A lot the tasks that was difficult and hacky to do on the N900, was integrated in the to the user interface of the Wildfire.

Last week we broke up definitely. I found my new phone at a cheap Danish phone reseller, at lowered price.
Now I have a HTC Desire Z, because I would the Qwerty keyboard too much.

Still, I is  sad – to let a trusty life companion go.

Is ZFS overly hyped?

When ZFS first appeared, it was well received and praised.
The time was ripe for a modernization of file systems to eliminate the tedious task of having to plan and create volume groups.

My first experience with ZFS was when I was hired to build a NAS for a small business.
Initially, I chose to go the tradtional way and make the raid as a raid5. This took ages (about a day) to complete.
I therefore quite surprised when I saw the difference between creation time of a traditional raid5 and a raidz. The raidz was ready seconds after I had created it.

Also, when my server drowned ZFS helped my to salvage my data very easily .

So, what are the cool things about ZFS:

  • Creating raids take seconds insteads of hours (days)
  • It has filesystem level data integrity
  • Built-in snapshots that uses copy-on-write to preserve disk space. It’s like Apple’s Time machine – only on filesystem level
  • The L2ARC feature is brilliant. I have not yet had a chance to try it out yet though.

I know btrfs is said to be the upcoming zfs killer, but from what I have read, it still lacks most of the features that are fundamental for zfs now – and has been for some time.
The advantage of btrfs is that it has the potential of being a clean room implementation of zfs, addressing some of the design flaws that zfs has.

I’ve only seen the tip of the iceberg, and for all my usage it has proved itself more than worthy. I am still trying to convince a business partner to engage in a more bold and lager scale zfs implementation, but this is still in the idea stage.

The next step for file systems will probably be more from the userspace perspective. Modern computers, like the ipad requires a different filesystem layout – or none at all. The next evolutionary step for file systems will be metadata-driven, and storage will be a large pool distributed over different mediums, like cloud based storage.

The applicability of ZFS is enormous – and in my opinion the only thing holding people back is the lack of trust in the technology.
Even though, its technology as I like it best. Complexity made simple – with rich opportunity to dive into the sea of technical details. So to answer my own question; no, I don’t think

Endnu en ubrugelig valgprognose

Som tidligere omtalt byggede vi på min tidligere uddannelse et stemmesystem med det formål at stille entry- og exitpolls tilgængelig ved et lokalt valgmøde.

Valgmødet foregik i Nykøbing F. og var primært rettet mod unge under 18. Det var altså rent teoretisk.

Vi gjorde os dengang følgende observationer:

Resultat før mødet

Man kan tydeligt se en tendes til en højredrejning i prognosen baseret på stemmer indsamlet ved indgangen.
Til gengæld bliver billedet markant anderledes ved udgangen. Det skal bemærkes at stemmeterminalerne først blev aktiveret ved mødets afslutning, og derfor kun repræsenterer meningsfordelingen hos de “vælgere” der deltog i hele mødet.

Resultat efter mødet

Ud over at en stor del (ca. 50%) af deltagerne har forladt mødet viser tallene også at der nu er en kraftig venstreskæv fordeling – især til Enhedslisten der er gået markant frem.

Både Venstre og Socialdemokratiet er gået tilbage. Dansk Folkeparti er det parti der har lidt størst tab under mødet; ud af de 114 stemmer de fik ved indgangen, er 100 vælgere enten udvandret eller har skiftet mening.

Jeg syntes observationene er interessante, men de har jo intet at gøre med det forestående valg. Måske andre kan hive flere spændende konklusioner ud fra disse tal?

Til de kodeinteresserede kan serverdelen og kommandolinje klienten hentes her:

valgsystem.tar.gz

Politiske partilogoer i svg format

I forbindelse med konstruktionen af et valgsystem tilbage i 2007 havde jeg brug for de enkelte partiers logoer i et vektorbaseret format – da det ikke var alle der havde den slags liggende, gav jeg mig i kast med at kreere dem selv.

Nu da der atter er valg, kan nogen måske få nytte af dem.

De er alle manuelt tracet i inkscape af undertegnede og i svg format.

politiske_logoer_2007.zip

God fornøjelse.

Digitalisering i praksis – *suk*

Public service. Log direkte på uddannelseskort via faelles.uddannelseskort.dk.

I morgen skal jeg starte igen. Men jeg har stadig ikke fået noget uddannelseskort til at rejse med.

Til gengæld har jeg fået en faktura fra DSB. I dag. Med betalingsfrist i dag. Der står også at der kan gå op til 14 dage fra de har min indbetaling til jeg modtager kortet.

En af de ting der under mig, er at på DSBs hjemmeside står der at jeg kan/skal betale med betalingskort.
Søvlpapirshattene ville straks proklamere at det var for at sende nogen penge i retning af PBS Nets (jeg lærer det nok med tiden) – men det tror jeg nu ikke.

Napoleon Bonaparte giver en bedre forklaring.

Never ascribe to malice that which is adequately explained by incompetence.

Jeg betaler fakturaen og tjekker inde på mituddannelseskort.dk, næh det var et typosquatting domæne. uddannelseskort.dk så.
Nej, heller ikke?
Nåå www.uddannelseskort.dk! URL redirection er en også en svær disciplin.

Men hvor kom jeg fra? Nå ja, kortbetalingen! Måske har jeg overset noget.
På siden kan jeg ikke andet end at vælge betalingsform (giro eller betalingskort) og skifte gyldighedsperiode. Altså må det være hos DSB betalingssystemet må være.

Min konklusion bliver her så at DSB ikke har deres online betalingssystem klar endnu – og derfor bliver sorteper sendt videre til bogholderiet der så skal sende fakturaer ud mere eller mindre manuelt.

En ting der dog fanger mit blik inde på www.uddannelseskort.dk er følgende:

Bestillingsstatus:
25-08-2011: Uddannelsesstedet har godkendt din bestilling 
            og sendt den videre til trafikselskabet.

Det har altå taget dem 10 dage (8 hverdage) at finde ud af at jeg er studerende på DTU

Hvorfor i alverden skal mit uddannelsessted nu godkende mig? Hvorfor kan uddannelseskort.dk (undskyld www.uddannelseskort.dk) ikke snakke sammen med minSU som allerede ved at jeg er under uddannelse? Den flaskehals som de 8 dage repræsenterer, er efter min bedste overbevisning, af bureaukratisk karakter. Der skal med andre ord sidde en kontormedhjælper og taste/ringe/brevsende oplysninger rundt og godkende.

Jeg kan ikke i min vildeste fantasi begribe hvorfor det er bedre at have én flaskehals i stedet for flere i form af kortsalgsstederne.

Til dem der ikke ved hvordan systemet var, kommer proceduren her.

  1. Jeg søger om uddannelsesrabat via minSU
  2. Jeg modtager et papir der fortæller hvilke måneder jeg er berettiget til rabat.
  3. Papiret vises ved kortsalgsstedet ved kortkøb og rabatten udløses

Den nye procedure er:

  1. Jeg bestiller uddannelseskort via faelles.uddannelseskort.dk via borger.dk via www.uddannelseskort.dk 14 dage før studiestart – som anbefalet.
  2. Jeg modtager en faktura fra DSB 14 dage efter
  3. DSB sender mit kort (op til) 14 dage efter de har modtaget min betaling
  4. Jeg køber et periodekort ved et kortsalgssted
  5. Jeg refunderer periodekortet når jeg har modtaget mit uddannelseskort

Man skal ikke være være den store logistiker for at erkende at der i det nye system er et enormt ressourcespild i form af mandetimer spredt ud over flere instanser. I øvrigt er proceduren en del længere hvis man går ind via en søgning p Google.

Uddannelseskortet virker fra mit synspunkt mest som digitalisering og sammenlægning for digitaliseringen og sammenlægningens skyld. Slutbrugerens udbytte er i hvert fald ikke eksisterende.

Migrating Dovecot 1.2 Maildir to Dovecot 2.0 dbox

I am in the process of migrating to a new mail server. Therefore I need to, as painlessly as possible, move users. The details about the setup is another story for another day – promise.

This guide is targeted for Debian systems, but the concepts apply for all other systems as well.

Dovecot 2.0 comes with a nice tool called dsync which eases migration by a great deal. Unfortunately, my current mail server runs Dovecot 1.2 and therefore does not have the tool.

What to do, then.

Basically I have thought up three options for migrating.

  1. Using dsync on both sides
  2. Using rsync, then dsync
  3. Using dsync over sshfs

This post will serve as documentation for my experiments with mailbox migration.

If you are in a hurry, you can skip to the conclusion.

Using dsync on both sides

Being that I run Dovecot 1.2 and thus do no have dsync available I will need to pull down the sources and compile them myself. (I do not want to use dpkg’s as they may intervene with the existing installation.)

I got as far as getting the source compiled, but have not investigated further. Some paths were wrong – I cowardly quitted.

Later experiments with the two other approaches have shown that this, most likely, will not prove successful.

Using rsync then dsync

Next solution was to create a two step migration solution. First I used rsync to copy my Maildir mailboxes to the new server.

rsync -poazuHK -e ssh \ 
     root@oldmailserver.tld:/var/spool/postfix/virtual/ \ 
     /var/vmail.migrate/

You can log in as root here, as the -o (preserve ownership) maps the uname to the uid on the target system. Clever :-)

Then, run dsync for each user in order to import the new emails.

dsync -R -u myaddress@mydomain.tld backup \
maildir:/var/vmail.migrate/mydomain.tld/myaddress/Maildir/

Mirroring does not really make sense here as we have a local copy of the mailbox

This approach is by far the fastest and easiest.

Using dsync over sshfs

Notice: This only works with backup and not mirror.

Why? Dovecot2 log format is incompatible with Dovecot1’s that will timeout with a message about an unknown record type (0x8000) after a mirror operation.

# apt-get install sshfs
sshfs -o uid=`id -u vmail` -o allow_other \
vmail@oldmailserver:/var/spool/postfix/virtual/ \
/var/vmail.lucretia/

Remember the -o allow_other or the dsync will fail because the vmail user will not have access to the mount point.

Then, run dsync for each user in order to import the new emails.

dsync -R -u myaddress@mydomain.tld backup \
maildir:/var/vmail.oldhost/mydomain.tld/myaddress/Maildir/

Ownerships is of the essence here. Do not use root as this user will take ownership of dovecot metadata files causing your source mail server to coredump or just stall.
vmail is not the best option either – but I was lazy. You should take advantage of the fact that the vmail folders are (usually) gid vmail. Putting a migration user in this group and chmodding will probably be preferred, security-wise.

This approach works well when refined (eg. usíng the right uid on both sides), but is pretty slow – about 100kb/s sync. This not really acceptable for 1GB+ mailboxes. But as always, your milage may vary.

Your remote Dovecot will keep on running as nothing has happened – if you get the permissions correct. Unfortunately there are problems with the dovecot transaction log resulting in problems with uid of the Mailbox being inconsistent, resulting in something like this:

Error: Corrupted transaction log file /var/vmail/domain.tld/username/dbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log seq 4: indexid changed 1313910265 -> 1313868319 (sync_offset=0)

Conclusion

My previous attempts have lead me to one conclusion: I need to move the mailbox once.

I chose the rsync+dsync approach and then did the following:

  1. Migrated all users to the new server
  2. Updated DNS
  3. rsync’ed first time
  4. Stopped the Dovecot and Postfix service on the old server
  5. rsync’ed second time
  6. dsync’ed the mailboxes
  7. Turned virtual_mailbox_maps and domains into relay_recipient_maps and domains respectively

If you decrease the TTL for you domain up until the move, you can minimize downtime. If you maintain a local DNS – even better.

This is not the fancy minimal down-time approach I had hoped for, but it has been sufficient for my needs. Feel free to contribute feedback.

Troubleshooting

I got a:

dsync(root): Fatal: Mail locations must use the same virtual mailbox
hierarchy separator (specify separator for the default namespace)

Some google-ing revealed that I needed to setup a namespace separator. The technical explanation for this left to the more Dovecot-savy.

In short, add the following to /etc/dovecot/conf.d/10-mail.conf (or uncomment the relevant ones).

namespace {
  separator = /
  inbox = yes
}

An now it works. migration is just a matter of setting up a cron job now, lower the TTL on the domain and move in day or two.

I got some

Error: Can't rename mailbox INBOX to
INBOX_ff3e01082bcf4e4e352c00002b747e8a:
Renaming INBOX isn't supported.

Using rsync->dsync which I haven’t been able to solve yet. Maybe shutting down the Dovecot service on the remote side would help. Race conditions are likely to occur.

Buffalo WLI-U2-KG54L on Debian

The dongle under treatment

I have a Buffalo WLI-U2-KG54L USB wifi dongle. It very convenient as my router and access point is placed in the basement, and I sometimes have computers on the main floor that do not have built-in wifi.

The dongle works out-of-the-box™ on Ubuntu, but as debian are more restrictive, it need a bit more work.

You can find the instructions here: http://wiki.debian.org/zd1211rw

And instructions on how to activate it here: http://wiki.debian.org/WiFi/HowToUse. The part about wpa_supplicant is most relevant, as network manager just works when the device works.

Proftpd and LDAP on Debian Squeeze

This is a short howto (hopefully) providing enough information to install Proftpd and use LDAP as user database.

Background

I have become obsessed with LDAP – at least for the time being. It seem to be the answer to my redundancy and distribution plans.

A production server is in the process of being converted (migrated actually) to have a single SSO LDAP structure.

A virtualization host crash (thank you Linode) forced me to move a couple of sites onto this new fancy LDAP server. Shortly after, a user prompted me about the lack of FTP on the new webhost.

Now the shoe needs to fit.

Installing the required packages

This is the easy part.

# apt-get install proftpd-mod-ldap

The LDAP module will depend on the proftpd server so this is really the only thing you need to install.

Requirements for the LDAP server

The LDAP module for Proftpd is hard coded to lookup only users of objectClass: posixUsers which in my opinion is less intuitive than having a specified schema for proftpd.

An example .ldif is shown below. I have added objectClass: domain, which is unnecessary.

The uidNumber and the gidNumber maps the uid and gid on the system. 115 is proftfd user and 65534 is group nobody. From a ftp client owner will appear as domain.tld or whatever you specify as uid.

version: 1

dn: dc=domain.tld,ou=webhosting,dc=example,dc=com
objectClass: domain
objectClass: top
objectClass: posixAccount
cn: domain.tld
dc: domain.tld
gidNumber: 65534
homeDirectory: /var/www/domain.tld/www
uid: domain.tld
uidNumber: 115
loginShell: /bin/false
userPassword::

Configuring the authentication

First you need to edit /etc/proftpd/ldap.conf to match you LDAP setup. Somthing like this is appropriate.

<IfModule mod_ldap.c>
  LDAPServer ldap://example.com/??sub
  LDAPDNInfo "cn=proftpd,dc=example,dc=com" "password"
  LDAPDoAuth on "ou=webhosting,dc=example,dc=com"
</IfModule>

notice the ??sub after the ldap. This is very important as it specifies the search scope. The configuration parameter LDAPSearchScope is apparently ignored.

Again, a sour comment; the bind should have been done as the user logging in, and not as a dedicated user. Admin is a bad choice – create a dedicated user. Besides, the /etc/proftpd/ldap.conf is world readable!

Next you have to tell proftpd to load the module.
Uncomment the line

LoadModule mod_ldap.c

in /etc/proftpd/modules.conf.

Now you have to uncomment the line.

Include /etc/proftpd/ldap.conf

in /etc/proftpd/proftpd.conf to load the Ldap configuration.

Finally:

While editing proftpd.conf you should also lift the RequireValidShell restriction (or give the user a valid loginShell parameter. If do not do this, you will not be able to log in.

Now is the time to take a look at the standard proftpd configuration and make sure that anonymous login is disabled and ditto /etc/passwd users.

Chenbro ES34169 mini-review

I have had my eye on the Chenbro ES34169 case for some time now. It is relatively inexpensive small case with 4 build-in sata harddisk bays. My plan was to replace the current chassis to make a more compact NAS.

Top-down view
Nice and compact chassis

Unfortunately, the power supply does not work with the Jetway NC9C-550-LF I am using as a NAS mainboard.

I have contacted Chenbro, and got a reply from saying that they are testing the case with Jetway boards.
This was a very bad first impression, but lets not draw hasty conclusions from this.

The case is a designed for use with a mini-ITX board and has room for one 2,5″ drive inside. All cables are included and the power supply ranges from 120W to 180W depending on the product code. The one I tested is a 180W.

Motherboard tray
Motherboard tray for a mini-ITX board

The case has a front door which is dark semi-tranparent. This gives a nice dimmed look when the case is operational. The door is made from plastic and contains a tiny lock which you may be familiar with if you have ever had one of these 3,5″ floppy storage bay back in the days where splitting archived files made sense. Breaking it open will most likely require very little force, hence it is not very secure.

Case door
Case door is semi-transparent

The case includes a CF/SD card reader – althought optional (check before you buy). This really gives the case an edge and opens up for variaty of deployment options.

Behind the door
Card reader an slim cd slot

I’ve taken off the plastic front for no particular reason other than I like to take things apart.

Drive trays without plastic front
Drive trays without plastic front

Hardware installation

Hardware installation was easy. Every drive bay slides out without any force, and mounting the drives is just a matter of screwing in four screws.

All cables are either supplied or already installed, and the mini-ITX board installs without any space problems. Just remember a low profile CPU heatsink.

Getting all the cables fixated, do they don’t apply force to case side can be somewhat challenging – but what can you expect from a case with such limited space.

The case has room for one low profile pci/pci-express card.

Conclusion

The drive bays seem sturdy, but the general impression of the case is: cheap. Maybe this is because the previous NAS case was a full-aluminum case from Lian-Li, but I cannot look away from the fact that the metal bends very easily and seems very soft in general.
Nevetheless, when the case is assembled everything is solid, and there are no wobbely or loose parts.
A good, affordable and very recommendable case.