Eucalyptus and NFS - A match made in network file system heaven.

One of the questions that I am often asked by customers looking at Eucalyptus for a Private Cloud is "how can I use NFS with Eucalyptus?" The scenario is usually one where the customer has deployed a NetApp SAN and prefers to use NFS over iSCSI or has deployed another highly-scalable distributed storage platform, like GlusterFS, with NFS exports.

In a normal HA deployment, Eucalyptus uses local disk for the Walrus storage and iSCSI SAN storage from a supported SAN (NetApp, EMC VNX or Dell EqualLogic). While the performance is much better with these types of storage back-ends, sometimes it is more practical to use something universal, like NFS, for your cloud storage back end.

The solution here is very simple, actually, with one caveat: you will not be able to use HA. In a non-HA implementation, Eucalyptus can in fact use NFS to back the Walrus and Storage Controller, and the configuration is very simple. Set up your cloud following the instructions in the Eucalyptus Installation Guide (for a non-HA setup) and simply mount the NFS partitions under /var/lib/eucalyptus/bukkits (for the Walrus) and /var/lib/eucalyptus/volumes (for the Storage Controller). From this point on, Eucalyptus will use your NFS exports as back-end storage for it's services.

So what do you do about failure in a non-HA setup? There are a few 'standard' Linux faculties to give you redundancy in this scenario. I will cover those in a future blog post, but essentially, you could use standard backups of the front-end components that could be restored in the event of a failure or you could get a little more creative and mirror the front-end components to what is essentially a 'warm spare' that can then be brought online if there is a failure. There are other options as well, and we will discuss those more detail in the future. For now, you can start building your cloud on top of NFS and start testing functionality and performance to see if it meets your use case(s).

EUCA3 Item Bank Open Sourced and Ready for Review

A few months ago we announced our intention to open source the Eucalyptus Certified Professional on Eucalyptus 3 (EUCA3) exam item bank. It is my distinct pleasure to formally announce the immediate availability of this document on GitHub. You can find the entire item bank here. The EUCA3 exam itself will become available in a week or two, and will be announced here when it is ready.

What This Means

If you are a prospective EUCA3 candidate:

You can download the item bank today and use it as a study guide for your exam. The exam itself will still be delivered online (more details to follow) and you will have a copy of the Eucalyptus 3: Design, Build, and Manage course book available in digital form while you take the test. No written notes will be allowed, but you will be allowed to have blank paper and a calculator to assist with any needed calculations during the exam.

If you are a cloud certification vendor:

You may use any of this content that you see fit as part of a cloud certification program or study guide, per the creative commons license specified for all Eucalyptus documentation. In fact, you are welcome to launch your own competitive certification using our content if you like. (CompTIA, I'm looking at you. Eucalyptus+ anyone? We'd love to see it!)

How You Can Contribute

You are welcome to contribute new items to the exam, as well as suggest corrections to existing items. In particular, we know there are some edited items where the item itself is fine, but the notes explaining the item need to be updated. If you have other suggestions for how to improve the exam or process, let me know. I can be reached at jason.eden@eucalyptus.com. Thanks!

Why private cloud? A perfect usage scenario.

One of the questions that is often asked of me as a Cloud Solutions Architect and Consultant is: Why do I need a private cloud? This is a valid question on several levels, but rather than give the usual spiel on 'security' and 'control', I want to delve into a tangible usage scenario where a private cloud makes sense. While this is just one scenario, think about the ways that this can apply to so many more scenarios that are similar or have similar goals and end-states.

There is no question that with AWS, Amazon has stepped forward to provide the de facto public cloud offering, and by proxy, the de facto public cloud API. While many want to be competitors in that space, and generate a lot of hype in trying, there is no valid competition to Amazon AWS today. The infrastructure and API are not of much value to the average person, however. This is where 3rd party developers have stepped in to leverage the infrastructure and API and provide valuable services to everyone. Two great examples of this are Dropbox and Crashplan, who use AWS S3 (Simple Storage Service) to provide cloud storage and backup to end users. Both companies have become wild successes, and each utilizes the power of the AWS API as well as the individual services like EC2 and S3.

Storage has traditionally been the realm of hard drive manufacturers and large storage (SAN) companies, and has almost always been 'on premise.' With the advent of AWS and S3, much more is being done with respect to storage in the cloud. More enterprising storage companies are looking for ways to bridge that gap as customers are eager to consume storage both on premise and in the cloud. One only needs to look to the home storage market to see manufacturers of small home NAS devices offering cloud based storage options, some even offering direct backup to S3. While it is fairly inexpensive to develop technology that will backup small amounts of data to S3, cost becomes prohibitive when the data sets reach into the terabytes or petabytes of data, and require massive amounts of bandwidth to move for testing purposes. This is where a private cloud can help drastically.

In the world of enterprise IT, it is no secret that data is king. Data is the core of any enterprise strategy and both traditional storage companies and Amazon AWS know this. Data is the 'hook' that they have to ensure that leaving their SAN offering or S3 storage repository is both painful and costly. If you are a company looking to develop a technology that integrates with AWS from on premise or sits on AWS entirely, the costs to move large amounts of data or spin up many instances in the cloud for development and testing can get expensive. Furthermore, if you want to test the speed and reliability of your platform in moving data into and out of S3, the bandwidth costs (not to mention restrictions) can be problematic. A private cloud solves these issues by putting a 'public cloud-like' environment on your local LAN. This means that you can develop and test your apps and technology stack at 'line speed' without the added cost for bandwidth and data storage. It may not be practical to test the movement and fidelity of 100TB of data or a petabyte of data into S3 over your ISP connection, but it is perfectly practical to do this on your local LAN.

In a hybrid-cloud scenario, this also gives you the option of encrypting data locally, in your private cloud environment, before it is shipped off to the public cloud. This is huge in terms of offering a secure product or service to end-users who are leery of the public cloud. If your primary focus for your product or service is on premise with only backup (or limited integration with) to the public cloud, it makes even more sense to develop and test on a private cloud.

In summary, if you need to develop any sort of product or service that integrates with the public cloud, and you want to be able to develop and test at local LAN speeds, a private cloud is an excellent choice. Furthermore, if you have large data storage and transport needs, the private cloud becomes the only reasonable solution which gives you the best performance and no additional bandwidth cost. Look to the private cloud for your development and testing needs.

Eucalyptus SAN Adapter: SAN Considerations

As we have discussed in previous blog posts, if you want to have High Availability (HA) for the Eucalyptus Storage Controller (SC), you must use the Eucalyptus SAN Adapter and a supported SAN. NOTE: To use Eucalyptus with supported SAN storage, you must decide whether administrative access can be provided to Eucalyptus to control the SAN. If this is possible in your environment, Eucalyptus can automatically and dynamically manage SAN storage.

As of Eucalyptus version 3.2.1, Eucalyptus supports the following SANs: Dell Equallogic series of SANs (PS 4000 and PS 6000), NetApp Filer FAS 2000 and FAS 6000 series and EMC VNX. For Dell Equallogic, Eucalyptus requires SSH access to enable automatic provisioning. Eucalyptus will manage NetApp SANs via ONTAPI (version 7.3.3 and above). For EMC, Eucalyptus expects that the EMC NaviSecCLI software will be installed on the Storage Controller host.

Use of the Eucalyptus SAN Adapter gives you the following benefits:

• Integrate Eucalyptus block storage functionality (dynamic block volumes, snapshots, creating volumes from snapshots, etc.) with existing SAN devices
• Link VMs in the Eucalyptus cloud directly to SAN devices, thereby removing I/O communication bottlenecks of the physical hardware host
• Incorporate enterprise-level SAN features (high-speed, large-capacity, reliability) to deliver a production-ready EBS (block storage) solution for the enterprise
• Attach SAN devices to Eucalyptus deployments on Xen, KVM, and VMware hypervisors

The Prerequisites for using the SAN Adapter are as follows:

• Dell EqualLogic, PS4000 series and PS6000 series (For more information about Dell EqualLogic SANs, go to http://www.dell.com)
• NetApp, FAS2000 series and FAS6000 series (For more information about NetApp SANs, go to http://www.netapp.com
• EMC VNX Series (For more information about EMC VNX, go to VNX Family http://www.emc.com/storage/vnx/vnx-family.htm)

With all the above said, the Eucalyptus SAN Adapter functions similarly among the three supported SANs, so the real differentiators, or selection criteria, are going to be the typical ones with respect to selecting a storage platform. Namely, things like iSCSI performance, total storage cost, current employee skill set, current deployed storage platform(s) and the existence of fibre channel infrastructure or SAN will play a key role in deciding which platform to go with. 

First off, speaking from a CIO/CTO perspective, if you are satisfied with your current storage provider (assuming it is NetApp, EMC or Dell), and you have invested in training your storage administrators and engineers on that platform, it only makes sense to stick with that platform. This is probably the most important facet to this decision. If you do not currently use a SAN or use a different storage platform, but are looking to purchase one of the three supported SANs for your Eucalyptus Cloud deployment, then you will want to look at other factors like cost and performance. Cost is a very nebulous topic (pun intended) so it is best to get a quote for a similar storage size and feature set across all three and compare. Performance is also very nebulous because there are many factors that can impact it. I will delve more into this in the next section (technical).

In terms of technical differentiation, there are some key things to look at and some historical points to make. First, lets start with what Eucalyptus will be 'doing' with these storage platforms and what that translates to in terms of feature set and technology. Strictly from a SAN Adapter (Storage Controller) perspective, Eucalyptus is using the SAN to create, mount, snapshot and otherwise 'use' iSCSI block devices. The keyword in that last sentence is 'block'. This is where we get into the historical aspect of the storage platforms supported. 

NetApp has historically been a NAS device and an excellent one at that. Over time, they have moved toward offering block level storage instead of just file level storage. This has been done by creating virtual block devices that sit atop the WAFL layer, thus they are virtual constructs that do not actually have direct access to the disk blocks underneath. I have seen several technical blog posts over the years from various NetApp employees stating that WAFL is a filesystem and that WAFL is not a filesystem. Regardless, I do believe that the virtual construct atop WAFL is still the case. If any NetApp employees read this and know this not to be the case, please comment below (preferably with a few diagrams to back up your claims/data) and I can change this section. Regardless, because much of the value add for NetApp has to do with NAS (or filesystem level) services, it would seem counterintuitive to use a NetApp ONLY for block device exports without using any of the file-level services that NetApp is so great at.

Dell EqualLogic and EMC VNX, by contrast, are devices that were designed to be SANs (not NASes) and export block devices with extreme speed and stability. Both of these are essentially iSCSI implementations of what SANs have done for so long with fibre channel. Being that they were designed to export block devices, they do so extremely well, with excellent performance and via the iSCSI protocol that Eucalyptus requires for Storage Controller use. If you are ONLY using the SAN for Eucalyptus Storage Controller/SAN Adapter, it would make more sense to pick a device that is simplified to do just this. One thing to note, with respect to scalability, between the EMC VNX and the EqualLogic is that to scale an EMC, you simply add more shelves to the unit (up to the capacity), whereas with the EqualLogic, you have to add entire EqualLogic units in order to scale. You can not simply add more shelves as with a traditional SAN. This drives up the cost significantly and it may become a wash (in terms of pricing) should you need to scale beyond a single Equal Logic enclosure. Again, if any of this has changed recently, I invite both Dell and EMC employees to provide information justifying a change and I will be glad to make it.

So, now that we have covered many of the non-technical and technical considerations for choosing the right SAN to use with Eucalyptus, which should you chose? Which would I recommend? I will give you a three point list to help you select which is best for your deployment.

• If you are using the SAN for more than just your Eucalyptus deployment and using file-level services such as NFS exports and planning to use more advanced SAN software features such as cloning, snapshotting, mirroring, etc., then go with a NetApp.
• If you are using the SAN ONLY for your Eucalyptus Cloud and ONLY using block-level services/devices, and you do not need to scale beyond the maximum single enclosure size, then go with a Dell EqualLogic.
• If you are using the SAN ONLY for block-level services/devices and you need to scale beyond the single enclosure size of an EqualLogic, then go with an EMC VNX.

Keep in mind that these are VERY simplified reason points to select each. You may have other constraints or decision points to take into consideration. If you are still stuck in trying to select which SAN to go with, Eucalyptus Professional Services would be happy to discuss your use case, needs and considerations to help recommend which SAN to go with. As with all blog posts, if you have any questions or need further clarification on anything contained in this post, please leave a comment below and I will be happy to reply in a comment, or perhaps another blog post should the discussion warrant it. 

EucaStart 2.0 Stage 4 - Customer Production Environment Pre-checks

Over the past couple of weeks Eucalyptus Professional Services has been writing a series of blog posts that provide detailed information on each stage of the EucaStart 2.0 process, see "EucaStart 2.0 - The story so far" for an update.

This post covers Stage 4, the customer production environment pre-checks and here we will investigate a number of pro-active checks that can be carried out to ensure your servers and network are ready for the Eucalyptus IaaS installation.

 

Network

• Network cabling correctly wired
• Are your servers connected to the correct switches? Try out the networking tests in our User Acceptance Testing plan.
• Network switches configured
• Ensure your Network Administrator has updated the switch configuration & firmware.
• Ensure multicast traffic is allowed on the switches. After package installation you can run through the multicast test in step 13.
• VLAN range provided or VLAN clean network
• Are you intending to use MANAGED mode? Run through the Prepare VLAN checks from our installation guide.
• If you cannot allocate the whole VLAN range to Eucalyptus you can configure a specific range using the euca-modify-property command.
• IP addresses and DNS names allocated
• Each server should have a static IP address.
• Each server should have a forward (A) and reverse (PTR) DNS record in your public name server or each system configured with an /etc/hosts entry.
• Pool of public IP addresses allocated
• A range of addresses used to assign to each virtual instance.
• Range of private IP addresses allocated
• An unused class B network is preferred (e.g. 10.10.0.0/255.255.0.0).

 

Storage

• If you intend to use a SAN:
• A supported SAN device; Dell Equallogic, Netapp (FAS2000 or FAS6000 series) or EMC VNX
• Username/Password for Eucalyptus access
• Storage pools configured
• Disk Space on each system
• At least 200GB free disk space on the Storage Controller (/var/lib/eucalyptus/volumes)
• At least 300GB free on the Walrus server (/var/lib/eucalyptus/bukkits)
• At least 200GB free on the Node Controller (/var/lib/eucalyptus)
• See our Reference Architectures for more information on storage requirements.
• See our Storage Controller and Walrus troubleshooting pages for more info.

 

OS Configuration

• RHEL or CentOS 6 x86_64 installed
• SELinux in disabled or permissive mode
• (optional) Network bonding configured
• Bridge interfaces configured on each Node Controller system
• NTP installed & configured
• Direct Internet access from each system
• Alternatively access to a mirror system or proxy with access to:
• Eucalyptus repository (*.eucalyptus.com)
• RHEL or CentOS archive (*.centos.org)
• EPEL (download.fedoraproject.org) 

After this stage, you should be ready to go! You can then move onto Stage 5, which covers installation, configuration and populating your cloud with images.

Can you think of any other checks to carry out before cloud installation?
Over on the Eucalyptus Wiki and Eucalyptus-users mailing list we've been discussing the various checklists and tools available, come and join the discussion!

EucaStart 2.0 - The story so far

As part of the EucaStart 2.0 engagement process, Eucalyptus Professional Services is providing detailed information on each stage of a IaaS cloud deployment in an open and transparent way for all to modify and implement. We hope that this will help our users, customers and partners to setup successful IaaS deployments.

We're really just getting started with EucaStart 2.0, so we warmly welcome any feedback and support if you would like to contribute.

Details on each Stage

Over the past few weeks we've covered a lot of content so I thought it would be worth recapping it and where it fits into the EucaStart story.
 
Our first post by Jason announced, that we were moving our Professional services into the open and since then we've produced detailed posts on many of the stages:

Stage 1 - DBM Course Delivery: Open Source course materials, Contributing to courseware and Renting the Eucalyptus Education Cloud

Stage 2 - Cloud Architecture & Operations Review: Networking, Application Stack, Storage, Storage Controller, Walrus server, Hardware, Compute & Misc considerations.

Stage 3 - Architecture Diagrams & Bill of Materials (BOM): Reference Architecture & Custom Deliverables

Stage 4 - Customer Production Environment Pre-check: Watch this space!

Stage 5 - Install & Configure: Software Installation & Configuration and Cloud Images

Stage 6 - Verifications & Operations Check: Eucalyptus Support and User Acceptance Testing

In addition there has been posts on the Repeatable Deployment Plan and how we plan to publish and use the EucaStart content.


There is more to come, so watch this space! We hope you've enjoyed the content so far. Any comments or suggestions? Please leave us a comment!

Eucalyptus Storage Controller: Storage Considerations

In our last blog post, I discussed the storage considerations for the Eucalyptus Walrus component. Toward the end of the post, I talked about pondering some additional implications of using a SAN to back Walrus if you are going to use a Eucalyptus supported SAN with the SAN adapter for the Storage Controller (SC). In this post, we will delve into the various ways you can deploy the Storage Controller (SC) component, building from non-HA to HA with the Eucalyptus SAN adapter. Keep in mind that using the SAN adapter is a requirement for deploying the SC in HA, so if you do not have a Eucalyptus supported SAN (at the time of this publication this includes EMC VNX, NetApp and Dell Equallogic) you can only deploy in non-HA mode.

How the Storage Controller Functions

The SC functions in one of two ways in a Eucalyptus cloud depending on whether you deploy in HA or non-HA mode. They are quite different, so carefully consider each approach when designing the SC component of your cloud.

Eucalyptus SC Logical Diagram
for non-HA and HA modes.

In non-HA mode, the SC itself becomes a ‘virtual SAN’, utilizing local (or locally mounted) disk space to create virtual block devices that will be exported from itself to the Node Controllers (NCs) and mounted to present block devices to instances (or to the NC itself in the case of Boot from EBS instances.) The key takeaway for non-HA mode is that the SC is ‘in the data path’ meaning that all iSCSI exports are from the SC itself and you are bound by the disk and network performance of the SC.

In HA mode, the SC can be thought of as a proxy that takes EBS requests and orchestrates the provisioning, exporting and mounting of iSCSI LUNs from a supported SAN via the SAN adapter. In this scenario, the SC is NOT in the data path. Node Controllers mount the iSCSI LUNs directly from the SAN, not the SC. Not only is this more performant, in the case of an SC failure, you do not lose data or connectivity. The second SC simply takes over and proxies all of the SC related requests to the SAN and NCs. With all that said, lets take a look at HA and non-HA mode storage considerations.

Non-HA Mode

Eucalyptus non-HA SC data path

Deploying the SC in non-HA mode means, at its simplest, that there is only one node (as opposed to two in HA mode). This means that if the SC fails for some reason, you will lose access to all of your EBS volumes. There are five ‘backend providers’ for the Eucalyptus SC, of which two are valid for non-HA: Overlay and DAS. The other three we will discuss in the HA mode section below.

Overlay

Overlay simply means that Eucalyptus will use the local filesystem and “overlay” storage of your EBS volumes onto that directory. By default, Eucalyptus uses the ‘/var/lib/eucalyptus/volumes’ directory for SC storage in Overlay mode.

Direct Attached Storage (DAS)

When using DAS, you are able to specify a device to use for the SC’s storage space. This can either be a raw device (like /dev/sdb) or a LVM volume group.

The main reason for using DAS instead of Overlay is performance and control, especially if you are using network storage of some type, such as an iSCSI or Fibre Channel LUN. Overlay has the benefit of being much simpler to setup and configure (for small test clouds or POCs) and also allows you to use something like NFS to back the ‘/var/lib/eucalyptus/volumes’ directory instead of a block device. Both have their uses, but DAS is a step up in terms of performance and flexibility which was introduced in 3.2, and is highly recommended if you are able to dedicate a raw device for it’s usage.

HA Mode

Eucalyptus HA SC data path

HA mode is actually the ‘simplest’ in terms of usage because you only have one option: use the SAN adapter. Eucalyptus currently supports three different SANs for use with the SC: NetApp, EMC VNX and Dell EqualLogic. You can find out more about the SAN specific requirements, like software versions, in our documentation here: Eucalyptus SAN Specifications. In HA mode, the SC is simply an orchestrator that communicates with the SAN directly to perform volume operations such as volume creation, deletion, mounting and snapshots. One thing to note is that the SC is also used as a proxy for volume snapshots in HA mode. Eucalyptus will take a SAN-level snapshot via the SANs own software, then copy that to the SC as a staging point to upload it to the Walrus. The great thing about using a SAN with the Eucalyptus SC is that the robustness, speed and data integrity of your volumes is backed by a piece of hardware designed to be highly-available. Also, since the SC is not in the normal data path for volume usage, the NCs are able to connect directly to the SAN, which typically has more performant connectivity than a simple server. It is common knowledge that data is the most important piece of the IT puzzle and by using the Eucalyptus SAN adapter with a supported SAN, you ensure that your data resides on a rock-solid platform. It is generally recommended to use a SAN with Eucalyptus for your SC configuration.