Tuesday, January 25, 2011

How to create .img file

An IMG file contains a raw dump of the content of a disk. We will use losetup to associate loop device with a file and mount it to work with.
  1. Pre-allocate img file (stuff.img) of the size you need (10M):
    dd if=/dev/zero of=/tmp/stuff.img bs=1M count=10
    
  2. Setup a loop device:
    losetup /dev/loop0 /tmp/stuff.img
    
  3. Create ext3 file system on loop device:
    mkfs.ext3 /dev/loop0
    
  4. Mount it:
    mount /dev/loop0 /mnt
    
Once you are done:
  1. Unmount file system:
    umount /mnt
    
  2. Delete loop device:
    losetup -d /dev/loop0
    

Managing LXC container

Just like a virtual machine you can start/stop it:
  1. Start lxc container vm0:
    lxc-start -n vm0 -d
    
  2. Login into console:
    lxc1:~# lxc-console -n vm0
    
    Type <Ctrl+a q> to exit the console
    
    Debian GNU/Linux 6.0 vm0 tty1
    
    vm0 login:
    
  3. Since ssh server is already installed, you should be fine to login (assuing you have dhcp server running in the network and dynamic dns is configured accordingly):
    ssh vm0
    
  4. Shutdown lxc container vm0:
    ssh vm0 halt && lxc-wait -n vm0 -s STOPPED
    
  5. Stop lxc container vm0 (this simply kills all processes related to container):
    lxc-stop -n vm0
    

How to create a LXC container in Debian

Here we are going create an lxc debian container for it's squeeze/testing release (see also Setup LXC container):

Choose Packages

We will setup debian base minimal configuration, however you can customize which packages you would like to have installed (file /etc/lxc/packages):
# Extra packages
packages=\
ifupdown,\
locales,\
libui-dialog-perl,\
dialog,\
isc-dhcp-client,\
netbase,\
net-tools,\
iproute,\
openssh-server,\
vim,\
apt-utils

LXC Prepare Script

The script below does the following:
  1. Downloads installation packages and stores them in a single file located at /var/cache/lxc. The file name format is debian-$release.tar. If the file exists it doesn't create it again, thus saves your bandwidth.
    lxc1:~# ls -lh /var/cache/lxc/
    total 66M
    -rw-r--r-- 1 root root 66M debian-testing.tar
    
  2. The installation goes to /var/lib/lxc. A subdirectory is created for each container.
  3. The installed container comes with configured:
    • SELinux is disabled.
    • User ssh public key is added to container authorized keys (this allows password less ssh login).
    • Initab is setup with 2 virtual consoles only (tty1, tty2).
    • Network interface is configured for DHCP.
    • Host name is set to container name.
    • DHCP client is set to publish client name in DNS.
    • The hosts file resolves container name to 127.0.0.1.
    • Disable IPv6 in the kernel.
    • Disable IPv6 for sshd.
    • The default local is set to en_US.UTF-8 UTF-8.
    • The following services removed: umountfs, hwclock.sh, hwclockfirst.sh.
    • The system is updated/upgrade/cleaned.
    • Password for user root is set to root.
  4. LXC container network adapter settings are set to random MAC address (starting from 00:1E).
  5. The LXC container is registered with in /etc/lxc in order to be able use it with lxc service.
Copy the following into file /usr/local/sbin/lxc-prepare:
#!/bin/bash

hostname=vm0
release=testing
arch=i386

cache=/var/cache/lxc
lib=/var/lib/lxc

. /etc/lxc/packages

check_args() 
{
    if [ ! -z $1 ]; then hostname=$1; fi
    if [ ! -z $2 ]; then release=$2; fi
}

download() 
{
    if [ ! -e "$cache/debian-$release-$arch.tar" ]; then
        debootstrap --verbose --arch=$arch \
          --make-tarball="$cache/debian-$release-$arch.tar" \
          --include $packages --variant=minbase \
          $release "$cache/$release-$arch"
        if [ $? -ne 0 ]; then
           exit 1
        fi
    else
        echo "There is '$cache/debian-$release-$arch.tar'... skipping download."
    fi
}

install() 
{
    if [ ! -d "$lib/$hostname/rootfs" ]; then
        debootstrap --verbose --arch=$arch \
          --unpack-tarball="$cache/debian-$release-$arch.tar" \
          --include $packages --variant=minbase \
          $release "$lib/$hostname/rootfs"
        if [ $? -ne 0 ]; then
           exit 1
        fi
    else
        echo "There is '$lib/$hostname'... skipping install."
    fi
}

configure()
{
    rootfs=$lib/$hostname/rootfs

    mkdir -p $rootfs/selinux
    echo 0 > $rootfs/selinux/enforce

    if [ -f "$HOME/.ssh/id_rsa.pub" ]; then
        mkdir -p $rootfs/root/.ssh
        cp ~/.ssh/id_rsa.pub "$rootfs/root/.ssh/authorized_keys"
    fi

    cat <<EOF > $rootfs/etc/inittab
id:3:initdefault:
si::sysinit:/etc/init.d/rcS
l0:0:wait:/etc/init.d/rc 0
l1:1:wait:/etc/init.d/rc 1
l2:2:wait:/etc/init.d/rc 2
l3:3:wait:/etc/init.d/rc 3
l4:4:wait:/etc/init.d/rc 4
l5:5:wait:/etc/init.d/rc 5
l6:6:wait:/etc/init.d/rc 6
# Normally not reached, but fallthrough in case of emergency.
z6:6:respawn:/sbin/sulogin
1:2345:respawn:/sbin/getty 38400 console
c1:12345:respawn:/sbin/getty 38400 tty1 linux
c2:12345:respawn:/sbin/getty 38400 tty2 linux
EOF

    cat <<EOF > $rootfs/etc/apt/sources.list
 
deb http://ftp.debian.org/debian/ $release main contrib non-free
deb-src http://ftp.debian.org/debian/ $release main contrib non-free

deb http://security.debian.org/ $release/updates main
deb-src http://security.debian.org/ $release/updates main
EOF

    cat <<EOF > $rootfs/etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
EOF

    domain=`hostname -d`
    if [ ! -z $domain ]; then
        domain=.$domain
    fi
    cat <<EOF > $rootfs/etc/hostname
$hostname
EOF

    cat <<EOF >> $rootfs/etc/dhcp/dhclient.conf
send host-name "$hostname";
EOF

    cat <<EOF > $rootfs/etc/hosts
127.0.0.1   localhost
EOF

    cat <<EOF > $rootfs/etc/locale.gen
en_US.UTF-8 UTF-8
EOF

    echo net.ipv6.conf.all.disable_ipv6=1 >\
$rootfs/etc/sysctl.d/disableipv6.conf

    echo "AddressFamily inet" >>\
$rootfs/etc/ssh/sshd_config

    cp /etc/timezone $rootfs/etc/timezone
    cp /etc/localtime $rootfs/etc/localtime
    area=$(debconf-show tzdata | grep Areas | cut -d ' ' -f 3)
    zone=$(debconf-show tzdata | grep Zones/${area} | cut -d ' ' -f 3)

    cat <<EOF | chroot $rootfs
mknod -m 666 /dev/tty1 c 4 1
mknod -m 666 /dev/tty2 c 4 2
locale-gen en_US.UTF-8
update-locale LANG=en_US.UTF-8
echo "tzdata tzdata/Areas select $area" | debconf-set-selections
echo "tzdata tzdata/Zones/$area select $zone" | debconf-set-selections
dpkg-reconfigure -u tzdata
/usr/sbin/update-rc.d -f umountfs remove
/usr/sbin/update-rc.d -f hwclock.sh remove
/usr/sbin/update-rc.d -f hwclockfirst.sh remove
apt-get update
apt-get -y upgrade
apt-get autoremove
apt-get clean
apt-get autoclean
echo "root:root" | chpasswd
EOF

    cat <<EOF > $lib/$hostname/config
lxc.utsname = $hostname
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = $rootfs
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm

# mounts point
lxc.mount.entry=proc $rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=devpts $rootfs/dev/pts devpts defaults 0 0
lxc.mount.entry=sysfs $rootfs/sys sysfs defaults  0 0

# network
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = 00:1E:$(hex):$(hex):$(hex):$(hex)
EOF
}

hex() 
{
    echo "`tr -dc A-F0-9 < /dev/urandom | head -c 2 | xargs`"
}

register()
{
    if [ -h /etc/lxc/$hostname.conf ]; then
        rm /etc/lxc/$hostname.conf
    fi

    ln -s $lib/$hostname/config /etc/lxc/$hostname.conf
}

check_args $1 $2
download
install
configure
register
Make the file executable:
chmod +x /usr/local/sbin/lxc-prepare

Create LXC Linux Container

We are going create a linux container vm0 based on debian testing.
lxc1:~# lxc-prepare vm0 testing
...
lxc1:~# du -hs /var/lib/lxc/vm0
235M /var/lib/lxc/vm0/

Auto start LXC Linux Container

Ensure the following in /etc/default/lxc, this one autostart vm0 and vm1:
# Comment out to run the lxc init script
RUN=yes

# Directory containing the container configurations
CONF_DIR=/etc/lxc

# Start /etc/lxc/example.conf, /etc/lxc/autostart.conf, etc.
#CONTAINERS="example autostart container"
CONTAINERS="vm0 vm1"
Since version 0.7.5 a way lxc autostart has been changed. Instead of listing in CONTAINERS variable make a symbolic link to configuration at /etc/lxc/auto directory.
ln -s /etc/lxc/vm0.conf /etc/lxc/auto/vm0.conf
Read more here.

Monday, January 24, 2011

Debian Squeeze/Testing LXC container patch

Prepare Debian Squeeze/Testing container

  1. Copy the following into file lxc-debian-testing. This is a modified version of /usr/lib/lxc/templates/lxc-debian to get it working with squeeze/testing release of dedian:
    #!/bin/bash
    
    configure_debian()
    {
        rootfs=$1
        hostname=$2
    
        # configure the inittab
        cat <<EOF > $rootfs/etc/inittab
    id:3:initdefault:
    si::sysinit:/etc/init.d/rcS
    l0:0:wait:/etc/init.d/rc 0
    l1:1:wait:/etc/init.d/rc 1
    l2:2:wait:/etc/init.d/rc 2
    l3:3:wait:/etc/init.d/rc 3
    l4:4:wait:/etc/init.d/rc 4
    l5:5:wait:/etc/init.d/rc 5
    l6:6:wait:/etc/init.d/rc 6
    # Normally not reached, but fallthrough in case of emergency.
    z6:6:respawn:/sbin/sulogin
    1:2345:respawn:/sbin/getty 38400 console
    c1:12345:respawn:/sbin/getty 38400 tty1 linux
    c2:12345:respawn:/sbin/getty 38400 tty2 linux
    EOF
     
        # create tty devices
        chroot $rootfs mknod -m 666 /dev/tty1 c 4 1
        chroot $rootfs mknod -m 666 /dev/tty2 c 4 2
    
        # disable selinux in debian
        mkdir -p $rootfs/selinux
        echo 0 > $rootfs/selinux/enforce
    
        # configure the network using the dhcp
        cat <<EOF > $rootfs/etc/network/interfaces
    auto lo
    iface lo inet loopback
    
    auto eth0
    iface eth0 inet dhcp
    EOF
    
        # set the hostname
        cat <<EOF > $rootfs/etc/hostname
    $hostname
    EOF
    
        # set dhcp hostname
        cat <<EOF >> $rootfs/etc/dhcp/dhclient.conf
    send host-name "$hostname";
    EOF
    
        # set default locale
        cat <<EOF > $rootfs/etc/locale.gen
    en_US.UTF-8 UTF-8
    EOF
        
        # reconfigure some services
        if [ -z "$LANG" ]; then
        chroot $rootfs locale-gen en_US.UTF-8
        chroot $rootfs update-locale LANG=en_US.UTF-8
        else
        chroot $rootfs locale-gen $LANG
        chroot $rootfs update-locale LANG=$LANG
        fi
    
        # remove pointless services in a container
        chroot $rootfs /usr/sbin/update-rc.d -f umountfs remove
        chroot $rootfs /usr/sbin/update-rc.d -f hwclock.sh remove
        chroot $rootfs /usr/sbin/update-rc.d -f hwclockfirst.sh remove
    
        echo "root:root" | chroot $rootfs chpasswd
        echo "Root password is 'root', please change !"
    
        return 0
    }
    
    download_debian()
    {
        packages=\
    ifupdown,\
    locales,\
    libui-dialog-perl,\
    dialog,\
    isc-dhcp-client,\
    netbase,\
    net-tools,\
    iproute,\
    vim,\
    openssh-server
    
        cache=$1
        arch=$2
    
        # check the mini debian was not already downloaded
        mkdir -p "$cache/partial-$arch"
        if [ $? -ne 0 ]; then
            echo "Failed to create '$cache/partial-$arch' directory"
         return 1
        fi
    
        # download a mini debian into a cache
        echo "Downloading debian minimal ..."
        debootstrap --verbose --variant=minbase --arch=$arch \
        --include $packages \
        testing $cache/partial-$arch http://ftp.us.debian.org/debian
        if [ $? -ne 0 ]; then
        echo "Failed to download the rootfs, aborting."
        return 1
        fi
    
        mv "$1/partial-$arch" "$1/rootfs-$arch"
        echo "Download complete."
    
        return 0
    }
    
    copy_debian()
    {
        cache=$1
        arch=$2
        rootfs=$3
    
        # make a local copy of the minidebian
        echo -n "Copying rootfs to $rootfs..."
        cp -a $cache/rootfs-$arch $rootfs || return 1
        return 0
    }
    
    install_debian()
    {
        cache="/var/cache/lxc/debian-testing"
        rootfs=$1
        mkdir -p /var/lock/subsys/
        (
            flock -n -x 200
            if [ $? -ne 0 ]; then
                echo "Cache repository is busy."
                return 1
            fi
    
            arch=$(arch)
            if [ "$arch" == "x86_64" ]; then
                arch=amd64
            fi
    
            if [ "$arch" == "i686" ]; then
                arch=i386
            fi
    
            echo "Checking cache download in $cache/rootfs-$arch ... "
            if [ ! -e "$cache/rootfs-$arch" ]; then
                download_debian $cache $arch
                if [ $? -ne 0 ]; then
                    echo "Failed to download 'debian base'"
                    return 1
                fi
            fi
    
            copy_debian $cache $arch $rootfs
            if [ $? -ne 0 ]; then
                echo "Failed to copy rootfs"
                return 1
            fi
    
            return 0
    
        ) 200>/var/lock/subsys/lxc
    
        return $?
    }
    
    copy_configuration()
    {
        path=$1
        rootfs=$2
        name=$3
    
        cat <<EOF >> $path/config
    lxc.tty = 4
    lxc.pts = 1024
    lxc.rootfs = $rootfs
    lxc.cgroup.devices.deny = a
    # /dev/null and zero
    lxc.cgroup.devices.allow = c 1:3 rwm
    lxc.cgroup.devices.allow = c 1:5 rwm
    # consoles
    lxc.cgroup.devices.allow = c 5:1 rwm
    lxc.cgroup.devices.allow = c 5:0 rwm
    lxc.cgroup.devices.allow = c 4:0 rwm
    lxc.cgroup.devices.allow = c 4:1 rwm
    # /dev/{,u}random
    lxc.cgroup.devices.allow = c 1:9 rwm
    lxc.cgroup.devices.allow = c 1:8 rwm
    lxc.cgroup.devices.allow = c 136:* rwm
    lxc.cgroup.devices.allow = c 5:2 rwm
    # rtc
    lxc.cgroup.devices.allow = c 254:0 rwm
    
    # mounts point
    lxc.mount.entry=proc $rootfs/proc proc nodev,noexec,nosuid 0 0
    lxc.mount.entry=devpts $rootfs/dev/pts devpts defaults 0 0
    lxc.mount.entry=sysfs $rootfs/sys sysfs defaults  0 0
    
    lxc.utsname = $name
    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br0
    # It is fine to be commented out
    #lxc.network.ipv4 = 192.168.10.21/24
    # Change this
    # lxc.network.hwaddr = 00:11:22:33:44:00
    EOF
    
        if [ $? -ne 0 ]; then
            echo "Failed to add configuration"
            return 1
        fi
    
        return 0
    }
    
    clean()
    {
        cache="/var/cache/lxc/debian-testing"
    
        if [ ! -e $cache ]; then
            exit 0
        fi
    
        # lock, so we won't purge while someone is creating a repository
        (
            flock -n -x 200
            if [ $? != 0 ]; then
                echo "Cache repository is busy."
                exit 1
            fi
    
            echo -n "Purging the download cache..."
            rm --preserve-root --one-file-system -rf $cache && echo "Done." || exit 1
            exit 0
    
        ) 200>/var/lock/subsys/lxc
    }
    
    usage()
    {
        cat <<EOF
    $1 -h|--help -p|--path=<path> --clean
    EOF
        return 0
    }
    
    options=$(getopt -o hp:n:c -l help,path:,name:,clean -- "$@")
    if [ $? -ne 0 ]; then
        usage $(basename $0)
        exit 1
    fi
    eval set -- "$options"
    
    while true
    do
        case "$1" in
            -h|--help)      usage $0 && exit 0;;
            -p|--path)      path=$2; shift 2;;
            -n|--name)      name=$2; shift 2;;
            -c|--clean)     clean=$2; shift 2;;
            --)             shift 1; break ;;
            *)              break ;;
        esac
    done
    
    if [ ! -z "$clean" -a -z "$path" ]; then
        clean || exit 1
        exit 0
    fi
    
    type debootstrap
    if [ $? -ne 0 ]; then
        echo "'debootstrap' command is missing"
        exit 1
    fi
    
    if [ -z "$path" ]; then
        echo "'path' parameter is required"
        exit 1
    fi
    
    if [ "$(id -u)" != "0" ]; then
        echo "This script should be run as 'root'"
        exit 1
    fi
    
    rootfs=$path/rootfs
    
    install_debian $rootfs
    if [ $? -ne 0 ]; then
        echo "failed to install debian"
        exit 1
    fi
    
    configure_debian $rootfs $name
    if [ $? -ne 0 ]; then
        echo "failed to configure debian for a container"
        exit 1
    fi
    
    copy_configuration $path $rootfs $name
    if [ $? -ne 0 ]; then
        echo "failed write configuration file"
        exit 1
    fi
    
    if [ ! -z $clean ]; then
        clean || exit 1
        exit 0
    fi
    
  2. Make the file executable:
    chmod +x lxc-debian-testing
    

Create LXC Linux Container

We are going create a linux container based on debian testing.
  1. Let give the machine name vm0.
    lxc1:~# mkdir -p /var/lib/lxc/vm0
    lxc1:~# ./lxc-debian-testing -p /var/lib/lxc/vm0 -n vm0
    debootstrap is /usr/sbin/debootstrap
    Checking cache download in /var/cache/lxc/debian-testing/rootfs-i386 ... 
    Downloading debian minimal ...
    ...
    Download complete.
    Copying rootfs to /var/lib/lxc/vm0/rootfs...Generating locales (this might take a while)...
      en_US.UTF-8... done
    Generation complete.
    ...
    Root password is 'root', please change !
    
  2. Set network adapter mac address (must be unique):
    lxc1:~# cd /var/lib/lxc/vm0
    lxc1:/var/lib/lxc/vm0# echo "lxc.network.hwaddr = 00:AF:22:33:44:00" >> config
    
  3. Make some clean up and backup:
    
    lxc1:/var/lib/lxc/vm0# chroot rootfs/
    root@lxc1:/# apt-get update && apt-get -y upgrade && apt-get clean && apt-get autoclean
    ...
    lxc1:/# exit
    lxc1:/var/lib/lxc/vm0# du -hs rootfs
    235M vm0
    lxc1:/var/lib/lxc/vm0# tar czf ../vm0.tgz config rootfs/
    lxc1:/var/lib/lxc/vm0# cd .. && ls -lh | grep tgz
    -rw-r--r-- 1 root root  80M vm0.tgz
    

How to setup LXC containers in Debian

LXC Linux Containers provides the resource management through the control groups aka process containers and resource isolation through the namespaces.

Installation

  1. The basic installation of lxc containers requires three package:
    apt-get install bridge-utils debootstrap lxc
    
  2. We need to mount the virtual control group file system on /cgroup:
    mkdir /cgroup
    echo "cgroup /cgroup cgroup defaults 0 0" >> /etc/fstab
    mount -a
    

Network Bridge

  1. Add network bridge to primary network interface (file /etc/network/interfaces):
    # The primary network interface
    #allow-hotplug eth0
    #iface eth0 inet dhcp
    auto br0
    iface br0 inet dhcp
          bridge_ports eth0
          bridge_fd 0
          bridge_maxwait 0
          bridge_stp off
    
  2. Restart networking:
    /etc/init.d/networking restart
    
What is amazing about lxc is how it is savy to resources. A newly started container takes not much but just extra 1M of memory.