Migration to Scaleway ARM server

The C1 Server

Scaleway started selling ARM based hosted server in April. I've intended to blog about this for a while, since it was time to upgrade from wheezy to jessie was timely, why not switch provider from an X86 based to ARM one at the same time?

In many ways scaleway node is opposite to what "Enterprise ARM" people are working on. Each server is based on an oldish ARMv7 Quad-Core Marvell Armada XP, instead of a brand new 64-bit ARMv8 cpu. There is no UEFI, ACPI or any other "industry standards" involved, just a smooth web interface and a command line tool to manage your node(s). And the node is yours, it's not shared with others with virtualization. The picture above is a single node, which is stacked with 911 other nodes into a single rack.

This week, the C1 price was dropped to a very reasonable €2.99 per month, or €0.006 per hour.

Software runs on hardware, news at 11

The performance is more than enough for my needs - shell, email and light web serving. dovecot, postfix, irssi and apache2 are just an apt-get away. Anyone who says you need x86 for Linux servers is forgetting that Linux software is open source, and if not already available, can be compiled to any architecture with little effort. Thus the migration pains were only because I chose to modernize configuration of dovecot and friends. Details of the new setup shall be left for another post.

Arm builder updates

Debian has recently received a donation of 8 build machines from Marvell. The new machines come with Quad core MV78460 Armada XP CPU's, DDR3 DIMM slot so we can plug in more memory, and speedy sata ports. They replace the well served Marvell MV78200 based builders - ones that have been building debian armel since 2009. We are planning a more detailed announcement, but I'll provide a quick summary:

The speed increase provided by MV78460 can viewed by comparing build times on selected builds since early april:

Qemu build times.

We can now build Qemu in 2h instead of 16h -8x faster than before! Certainly a substantial improvement, so impressive kit from Marvell! But not all packages gain this amount of speedup:

webkitgtk build times.

This example, webkitgtk, builds barely 3x faster. The explanation is found from debian/rules of webkitgkt:

# Parallel builds are unstable, see #714072 and #722520
# ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
#   NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
#   MAKEARGUMENTS += -j$(NUMJOBS)
# endif

The old builders are single-core[1], so the regardless of parallel building, you can easily max out the cpu. New builders will use only 1 of 4 cores without parallel build support in debian/rules.

During this buildd cpu usage graph, we see most time only one CPU is consumed. So for fast package build times.. make sure your packages supports parallel building.

For developers, abel.debian.org is porter machine with Armada XP. It has schroot's for both armel and armhf. set "DEB_BUILD_OPTIONS=parallel=4" and off you go.

Finally I'd like to thank Thomas Petazzoni, Maen Suleiman, Hector Oron, Steve McIntyre, Adam Conrad and Jon Ward for making the upgrade happen.

Meanwhile, we have unrelated trouble - a bunch of disks have broken within a few days apart. I take the warranty just run out...

[1] only from Linux's point of view. - mv78200 has actually 2 cores, just not SMP or coherent. You could run an RTOS on the other core while you run Linux on the other.

Marvell 78x00 board

What we are having here is a quite non-typical ARM based motherboard. It features a pre-production version of Marvell 78x00 series ARM SoC, clocked at 1GHz. It features multiple on-board sata and gigE connectors, as well as pci-e slots and DDR2 ram slots. Fairly "meh" in x86 world, but on the ARM world most systems are designed mainly for low power consumption, tiny size and low cost. Which, while creating allows creating many exciting devices, makes finding meaty buildd's tricky. When we intruduced armel port, the security people rose the concern that arm buildd's are too slow. Especially from the security Point of view, where getting security fixes out fast is important - every second of waiting for security builds to finish someone could be exploited!

Luckily, some people from Marvell had heard about our need and arranged to donate a developer board for us!

So far, the build speed improvement, compared to our current (Thecus N2100) buildd's is very promising. When looking at build times of some slow-building packages which likely will see security uploads:

Package   | Thecus | MV78100 | HPPA  | MIPS
linux-2.6 | 39h    | 11h     | 6h    | 17h
webkit    | 17h    | 5h      | 3h    | 4h
qt4-x11   | 59h    | 17h     | 7h    | 13h
xulrunner | 14h    | 4h      | 1.5h  | 3h

We see that with a Marvell board we are more in the line with other architectures. And the production hardware is going to be even faster! There is a dual-core version as well, but it's not SMP. Both cores are independent, and would basically both run their own Linux image. If manage to find a easy way to implement and manage that, we could run two theoretically run two independent buildd instances on the same board.

See the Linux Devices article for more details about the MV78x00 in general.