At the highest level, the main thing to take note of is that the Walrus stores its objects in the following directory (on the local operating system): “/var/lib/eucalyptus/bukkits”. If you are asking yourself “why bukkits?”, I will go no further than to tell you to Google the ‘walrus bukkit meme’. Now that we have discussed the directory that backs the Walrus object store, we have two potential paths to go down in terms of Walrus storage architecture. The first path will be with the Walrus in non-HA mode and the second will be with the Walrus in HA mode.
The non-HA mode is the simplest to discuss, so we will start there. We will also use this section to build upon for the discussion of the HA mode Walrus. Having the Walrus in non-HA mode means, at its simplest, that there is only one Walrus node. There is no second node to fail over to. This means that we do not need to worry about replication to another node, and it makes the architecture very simple. Since we are only dealing with a directory on the local filesystem, you can literally back it with anything you could normally back a directory with. This includes local disk, iSCSI backed LVM, an NFS mount or even a Fibre Channel LUN. Each option has its own set of performance constraints, but the performance profile will typically be (from fastest to slowest) Fibre Channel, local disk, iSCSI backed LVM and NFS. These can switch around depending on the individual performance characteristics of each, but assuming you have the most performant of each type, this should be a fair order.
When in HA mode, the Walrus architecture changes. The most significant of these changes is the dependency on DRBD for replication between the two Walrus nodes. DRBD is a block-level replication software, so it is inserted between the physical storage devices and the operating system file system. because DRBD requires a block device as a backing disk, NFS is out of the question for HA Walrus. In this architecture, you will use the physical storage device as a backing disk for each Walrus DRBD resource. This storage device can be local disk, Fibre Channel LUN or iSCSI backed LVM. In the iSCSI backed LVM instance, the LVM volume would be the backing device for DRBD, not the iSCSI device itself. You can also use an iSCSI volume directly as a DRBD backing disk, but it will be much harder to ‘grow’ your Walrus storage over time with this configuration. Using LVM (in general) atop iSCSI, Fiber Channel or local disk is recommended. One thing to note when using LVM on top of iSCSI is that the ‘storage stack’ has to come up in the right order, or there will be serious issues. The proper order, in this case, would be iSCSI LUNs > LVM > DRBD > Walrus service. If the LVM subsystem attempts to come up before the iSCSI LUNs have been mounted, it will fail and the entire system will be down. You can search on Google for options to use with iSCSI and LVM to prevent this. This will most likely change over time, so I left it out of this article intentionally. This will avoid confusion when checks are put into Linux distributions in the near future to prevent out of order starting of storage services.
Understanding the storage considerations for Walrus, before implementing your cloud, can prevent many issues related to performance and growth over time. Deciding on the mode (HA or non-HA) first will guide the next steps in the architecture design process. In a later post, I will discuss storage considerations for the Eucalyptus Storage Controller. This will help frame the overall discussion should you decide to back the Walrus with LUNs exported from the same SAN that will support the Storage Controller.