Posts Tagged dynamically allocated storage

Dynamically Allocated Storage Issues with Ubuntu’s Cloud Images

Imagine you’ve provisioned dozens of nodes on your network using Ubuntu’s Cloud Images, expecting them to grow dynamically…

Background

According to Canonical, ‘Ubuntu Cloud Images are pre-installed disk images that have been customized by Ubuntu engineering to run on cloud-platforms such as Amazon EC2, Openstack, Windows and LXC’. Ubuntu also disk images, or ‘boxes’, built specifically for Vagrant and VirtualBox. Boxes, according to Vagrant, ‘are the skeleton from which Vagrant machines are constructed. They are portable files which can be used by others on any platform that runs Vagrant to bring up a working environment‘. Ubuntu’s images are very popular with Vagrant users to build their VMs.

Assuming you have VirtualBox and Vagrant installed on your Windows, Mac OS X, or Linux system, with a few simple commands, ‘vagrant add box…’, ‘vagrant init…’, and ‘vagrant up’, you can provision a VM from one of these boxes.

Dynamically Allocated Storage

The Ubuntu Cloud Images (boxes), are Virtual Machine Disk (VMDK) format files. These VMDK files are configured for dynamically allocated storage, with a virtual size of 40 GB. That means the VMDK format file should grow to an actual size of 40 GB, as files are added. According to VirtualBox, the VM ‘will initially be very small and not occupy any space for unused virtual disk sectors, but will grow every time a disk sector is written to for the first time, until the drive reaches the maximum capacity chosen when the drive was created’.

To illustrate dynamically allocated storage, below are three freshly provisioned VirtualBox virtual machines (VM), on three different hosts, all with different operating systems. One VM is hosted on Windows 7 Enterprise, another on Ubuntu 13.10 Desktop Edition, and the last on Mac OS X 10.6.8. The VMs were all created with Vagrant from the official Ubuntu Server 13.10 (Saucy Salamander)  cloud images. The Windows and Ubuntu hosts used the 64-bit version. The Mac OS X host used the 32-bit version. According to VirtualBox Manager, on all three host platforms, the virtual size of the VMs is 40 GB and the actual size is about 1 GB.

VirtualBox Storage Settings on Windows Host

VirtualBox Storage Settings on Windows Host

VirtualBox Storage Settings on Ubuntu Host

VirtualBox Storage Settings on Ubuntu Host

VirtualBox Storage Settings on Mac OS X Host

VirtualBox Storage Settings on Mac OS X Host

So What’s the Problem?

After a significant amount of troubleshooting Chef recipe problems on two different Ubuntu-hosted VMs, the issue with the cloud images became painfully clear. Other than a single (seemingly charmed) Windows host, none of the VMs I tested on Windows-, Ubuntu-, and Mac OS X-hosts would expand beyond 4 GB. Below is the file system disk space usage report from four host’s VMs. All four were created with the most current version of Vagrant (1.4.1), and managed with the most current version of VirtualBox (4.3.6.x).

Windows-hosted 64-bit Cloud Image VM #1:

vagrant@vagrant-ubuntu-saucy-64:/tmp$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      ext4       40G  1.1G   37G   3% /
none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
udev           devtmpfs  241M   12K  241M   1% /dev
tmpfs          tmpfs      50M  336K   49M   1% /run
none           tmpfs     5.0M     0  5.0M   0% /run/lock
none           tmpfs     246M     0  246M   0% /run/shm
none           tmpfs     100M     0  100M   0% /run/user
/vagrant       vboxsf    233G  196G   38G  85% /vagrant

Windows-hosted 32-bit Cloud Image VM #2:

vagrant@vagrant-ubuntu-saucy-32:~$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      ext4      4.0G 1012M  2.8G  27% /
none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
udev           devtmpfs  245M  8.0K  245M   1% /dev
tmpfs          tmpfs      50M  336K   50M   1% /run
none           tmpfs     5.0M  4.0K  5.0M   1% /run/lock
none           tmpfs     248M     0  248M   0% /run/shm
none           tmpfs     100M     0  100M   0% /run/user
/vagrant       vboxsf    932G  209G  724G  23% /vagrant

Ubuntu-hosted 64-bit Cloud Image VM:

vagrant@vagrant-ubuntu-saucy-64:~$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      ext4      4.0G  1.1G  2.7G  28% /
none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
udev           devtmpfs  241M  8.0K  241M   1% /dev
tmpfs          tmpfs      50M  336K   49M   1% /run
none           tmpfs     5.0M     0  5.0M   0% /run/lock
none           tmpfs     246M     0  246M   0% /run/shm
none           tmpfs     100M     0  100M   0% /run/user
/vagrant       vboxsf     74G   65G  9.1G  88% /vagrant

Mac OS X-hosted 32-bit Cloud Image VM:

vagrant@vagrant-ubuntu-saucy-32:~$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      ext4      4.0G 1012M  2.8G  27% /
none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
udev           devtmpfs  245M   12K  245M   1% /dev
tmpfs          tmpfs      50M  336K   50M   1% /run
none           tmpfs     5.0M     0  5.0M   0% /run/lock
none           tmpfs     248M     0  248M   0% /run/shm
none           tmpfs     100M     0  100M   0% /run/user
/vagrant       vboxsf    149G   71G   79G  48% /vagrant

On the first Windows-hosted VM (the only host that actually worked), the virtual SCSI disk device (sda1), formatted ‘ext4‘, had a capacity of 40 GB. But, on the other three hosts, the same virtual device only had a capacity of 4 GB. I tested the various 32- and 64-bit Ubuntu Server 12.10 (Quantal Quetzal), 13.04 (Raring Ringtail), and 13.10 (Saucy Salamander) cloud images. They all exhibited the same issue. However, the Ubuntu 12.04.3 LTS (Precise Pangolin) worked fine on all three host OS systems.

To prove the issue was specifically with Ubuntu’s cloud images, I also tested boxes from Vagrant’s own repository, as well as other third-party providers. They all worked as expected, with no storage discrepancies. This was suggested in the only post I found on this issue, from StackExchange.

To confirm the Ubuntu-hosted VM will not expand beyond 4 GB, I also created a few multi-gigabyte files on each VM, totally 4 GB. The VMs virtual drive would not expand beyond 4 GB limit to accommodate the new files, as demonstrated below on a Ubuntu-hosted VM:

vagrant@vagrant-ubuntu-saucy-64:~$ dd if=/dev/zero of=/tmp/big_file2.bin bs=1M count=2000
dd: writing '/tmp/big_file2.bin': No space left on device
742+0 records in
741+0 records out
777560064 bytes (778 MB) copied, 1.81098 s, 429 MB/s

vagrant@vagrant-ubuntu-saucy-64:~$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      ext4      4.0G  3.7G  196K 100% /

The exact cause eludes me, but I tend to think the cloud images are the issue. I know they are capable of working, since the Ubuntu 12.04.3 cloud images expand to 40 GB, but the three most recent releases are limited to 4 GB. Whatever the cause, it’s a significant problem. Imagine you’ve provisioned a 100 or a 1,000 server nodes on your network from any of these cloud images, expecting them to grow to 40 GB, but really only having 10% of that potential. Worse, they have live production data on them, and suddenly run out of space.

Test Results

Below is the complete shell sessions from three hosts.

Windows-hosted 64-bit Cloud Image VM #1:

gstafford@windows-host: ~/Documents/GitHub
$ vagrant --version
Vagrant 1.4.1
gstafford@windows-host: /c/Program Files/Oracle/VirtualBox
$ VBoxManage.exe --version
4.3.6r91406
gstafford@windows-host: ~/Documents/GitHub
$ mkdir cloudimage-test-win
gstafford@windows-host: ~/Documents/GitHub
$ cd cloudimage-test-win
gstafford@windows-host: ~/Documents/GitHub/cloudimage-test-win
$ vagrant init
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
gstafford@windows-host: ~/Documents/GitHub/cloudimage-test-win
$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'vagrant-vm-saucy-server'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[default] Fixed port collision for 22 => 2222. Now on port 2200.
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2200 (adapter 1)
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
DL is deprecated, please use Fiddle
[default] Machine booted and ready!
[default] The guest additions on this VM do not match the installed version of
VirtualBox! In most cases this is fine, but in rare cases it can
cause things such as shared folders to not work properly. If you see
shared folder errors, please make sure the guest additions within the
virtual machine match the version of VirtualBox you have installed on
your host and reload your VM.
Guest Additions Version: 4.2.16
VirtualBox Version: 4.3
[default] Mounting shared folders...
[default] -- /vagrant
gstafford@windows-host: ~/Documents/GitHub/cloudimage-test-win
$ vagrant ssh
Welcome to Ubuntu 13.10 (GNU/Linux 3.11.0-14-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information disabled due to load higher than 1.0
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
vagrant@vagrant-ubuntu-saucy-64:/tmp$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 40G 1.1G 37G 3% /
none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup
udev devtmpfs 241M 12K 241M 1% /dev
tmpfs tmpfs 50M 336K 49M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 246M 0 246M 0% /run/shm
none tmpfs 100M 0 100M 0% /run/user
/vagrant vboxsf 233G 196G 38G 85% /vagrant
vagrant@vagrant-ubuntu-saucy-64:/tmp$ dd if=/dev/zero of=/tmp/big_file1.bin bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 5.11716 s, 410 MB/s
vagrant@vagrant-ubuntu-saucy-64:/tmp$ dd if=/dev/zero of=/tmp/big_file2.bin bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 7.78449 s, 269 MB/s
vagrant@vagrant-ubuntu-saucy-64:/tmp$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 40G 5.0G 33G 14% /
none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup
udev devtmpfs 241M 12K 241M 1% /dev
tmpfs tmpfs 50M 336K 49M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 246M 0 246M 0% /run/shm
none tmpfs 100M 0 100M 0% /run/user
/vagrant vboxsf 233G 196G 38G 85% /vagrant
vagrant@vagrant-ubuntu-saucy-64:/tmp$

Ubuntu-hosted 64-bit Cloud Image VM:

gstafford@ubuntu-host:~/GitHub/cloudimage-test-ubt$ vagrant --version
Vagrant 1.4.1
gstafford@ubuntu-host:/usr/bin$ vboxmanage --version
4.3.6r91406
gstafford@ubuntu-host:~/GitHub$ mkdir cloudimage-test-ubt
gstafford@ubuntu-host:~/GitHub$ cd cloudimage-test-ubt/
gstafford@ubuntu-host:~/GitHub/cloudimage-test-ubt$ vagrant init
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
gstafford@ubuntu-host:~/GitHub/cloudimage-test-ubt$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Box 'vagrant-vm-saucy-server' was not found. Fetching box from specified URL for
the provider 'virtualbox'. Note that if the URL does not have
a box for this provider, you should interrupt Vagrant now and add
the box yourself. Otherwise Vagrant will attempt to download the
full box prior to discovering this error.
Downloading box from URL: file:/home/gstafford/GitHub/cloudimage-test-ubt/saucy-server-cloudimg-amd64-vagrant-disk1.box
Extracting box...te: 23.8M/s, Estimated time remaining: 0:00:01))
Successfully added box 'vagrant-vm-saucy-server' with provider 'virtualbox'!
[default] Importing base box 'vagrant-vm-saucy-server'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[default] Fixed port collision for 22 => 2222. Now on port 2200.
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2200 (adapter 1)
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
[default] The guest additions on this VM do not match the installed version of
VirtualBox! In most cases this is fine, but in rare cases it can
cause things such as shared folders to not work properly. If you see
shared folder errors, please make sure the guest additions within the
virtual machine match the version of VirtualBox you have installed on
your host and reload your VM.
Guest Additions Version: 4.2.16
VirtualBox Version: 4.3
[default] Mounting shared folders...
[default] -- /vagrant
gstafford@ubuntu-host:~/GitHub/cloudimage-test-ubt$ vagrant ssh
Welcome to Ubuntu 13.10 (GNU/Linux 3.11.0-15-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information disabled due to load higher than 1.0
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
_____________________________________________________________________
WARNING! Your environment specifies an invalid locale.
This can affect your user experience significantly, including the
ability to manage packages. You may install the locales by running:
sudo apt-get install language-pack-en
or
sudo locale-gen en_US.UTF-8
To see all available language packs, run:
apt-cache search "^language-pack-[a-z][a-z]$"
To disable this message for all users, run:
sudo touch /var/lib/cloud/instance/locale-check.skip
_____________________________________________________________________
vagrant@vagrant-ubuntu-saucy-64:~$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 4.0G 1.1G 2.7G 28% /
none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup
udev devtmpfs 241M 8.0K 241M 1% /dev
tmpfs tmpfs 50M 336K 49M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 246M 0 246M 0% /run/shm
none tmpfs 100M 0 100M 0% /run/user
/vagrant vboxsf 74G 65G 9.1G 88% /vagrant
vagrant@vagrant-ubuntu-saucy-64:~$ dd if=/dev/zero of=/tmp/big_file1.bin bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 4.72951 s, 443 MB/s
vagrant@vagrant-ubuntu-saucy-64:~$ dd if=/dev/zero of=/tmp/big_file2.bin bs=1M count=2000
dd: writing '/tmp/big_file2.bin': No space left on device
742+0 records in
741+0 records out
777560064 bytes (778 MB) copied, 1.81098 s, 429 MB/s
vagrant@vagrant-ubuntu-saucy-64:~$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 4.0G 3.7G 196K 100% /
none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup
udev devtmpfs 241M 8.0K 241M 1% /dev
tmpfs tmpfs 50M 336K 49M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 246M 0 246M 0% /run/shm
none tmpfs 100M 0 100M 0% /run/user
/vagrant vboxsf 74G 65G 9.1G 88% /vagrant
vagrant@vagrant-ubuntu-saucy-64:~$

Mac OS X-hosted 32-bit Cloud Image VM:

gstafford@mac-development:cloudimage-test-osx $ vagrant box add saucycloud32 http://cloud-images.ubuntu.com/vagrant/saucy/current/saucy-server-cloudimg-i386-vagrant-disk1.box
Downloading box from URL: http://cloud-images.ubuntu.com/vagrant/saucy/current/saucy-server-cloudimg-i386-vagrant-disk1.box
Extracting box...te: 1383k/s, Estimated time remaining: 0:00:01)
Successfully added box 'saucycloud32' with provider 'virtualbox'!
gstafford@mac-development:cloudimage-test-osx $ vagrant init saucycloud32
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
gstafford@mac-development:cloudimage-test-osx $ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'saucycloud32'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
[default] The guest additions on this VM do not match the installed version of
VirtualBox! In most cases this is fine, but in rare cases it can
prevent things such as shared folders from working properly. If you see
shared folder errors, please make sure the guest additions within the
virtual machine match the version of VirtualBox you have installed on
your host and reload your VM.
Guest Additions Version: 4.2.16
VirtualBox Version: 4.3
[default] Mounting shared folders...
[default] -- /vagrant
gstafford@mac-development:cloudimage-test-osx $ vagrant ssh
Welcome to Ubuntu 13.10 (GNU/Linux 3.11.0-15-generic i686)
* Documentation: https://help.ubuntu.com/
System information disabled due to load higher than 1.0
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
vagrant@vagrant-ubuntu-saucy-32:~$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 4.0G 1012M 2.8G 27% /
none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup
udev devtmpfs 245M 12K 245M 1% /dev
tmpfs tmpfs 50M 336K 50M 1% /run
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 248M 0 248M 0% /run/shm
none tmpfs 100M 0 100M 0% /run/user
/vagrant vboxsf 149G 71G 79G 48% /vagrant
vagrant@vagrant-ubuntu-saucy-32:~$ exit
gstafford@mac-development:MacOS$ VBoxManage list vms --long hdds
UUID: ee72161f-25c5-4714-ab28-6ee9929500e8
Parent UUID: base
State: locked write
Type: normal (base)
Location: /Users/gstafford/VirtualBox VMs/cloudimage-test-osx_default_1389280075070_65829/box-disk1.vmdk
Storage format: VMDK
Format variant: dynamic default
Capacity: 40960 MBytes
Size on disk: 1031 MBytes
In use by VMs: cloudimage-test-osx_default_1389280075070_65829 (UUID: 51df4527-99af-48be-92cd-ad73110be88c)

Resources

Ubuntu Cloud Images for Vagrant

Fourth Extended Filesystem (ext4)

Similar Issue on StackOverflow

VBoxManage Command-Line Interface

Ubuntu Releases

, , , , , , , , ,

3 Comments