Understanding Storage Services in Azure

Introducing the Azure Storage Services

Azure storage provides the following four services: Blob storage, Table storage, Queue storage, and File storage.

• Blob Storage stores unstructured object data. A blob can be any type of text or binary data, such as a document, media file, or application installer. Blob storage is also referred to as Object storage.
• Table Storage stores structured datasets. Table storage is a NoSQL key-attribute data store, which allows for rapid development and fast access to large quantities of data.
• Queue Storage provides reliable messaging for workflow processing and for communication between components of cloud services.
• File Storage offers shared storage for legacy applications using the standard SMB protocol. Azure virtual machines and cloud services can share file data across application components via mounted shares, and on-premises applications can access file data in a share via the File service REST API.

An Azure storage account is a secure account that gives you access to services in Azure Storage. Your storage account provides the unique namespace for your storage resources. The image below shows the relationships between the Azure storage resources in a storage account:

There are two types of storage accounts:

General-purpose Storage Accounts

A general-purpose storage account gives you access to Azure Storage services such as Tables, Queues, Files, Blobs and Azure virtual machine disks under a single account. This type of storage account has two performance tiers:

• A standard storage performance tier which allows you to store Tables, Queues, Files, Blobs and Azure virtual machine disks.
• A premium storage performance tier which currently only supports Azure virtual machine disks. See Premium Storage: High-Performance Storage for Azure Virtual Machine Workloads for an in-depth overview of Premium storage.

Blob Storage Accounts

A Blob storage account is a specialized storage account for storing your unstructured data as blobs (objects) in Azure Storage. Blob storage accounts are similar to your existing general-purpose storage accounts and share all the great durability, availability, scalability, and performance features that you use today including 100% API consistency for block blobs and append blobs. For applications requiring only block or append blob storage, we recommend using Blob storage accounts.

Note:

Blob storage accounts support only block and append blobs, and not page blobs.

Blob storage accounts expose the Access Tier attribute which can be specified during account creation and modified later as needed. There are two types of access tiers that can be specified based on your data access pattern:

• A Hot access tier which indicates that the objects in the storage account will be more frequently accessed. This allows you to store data at a lower access cost.
• A Cool access tier which indicates that the objects in the storage account will be less frequently accessed. This allows you to store data at a lower data storage cost.

If there is a change in the usage pattern of your data, you can also switch between these access tiers at any time. Changing the access tier may result in additional charges. Please seePricing and billing for Blob storage accounts for more details.

For more details on Blob storage accounts, see Azure Blob Storage: Cool and Hot tiers.

Before you can create a storage account, you must have an Azure subscription, which is a plan that gives you access to a variety of Azure services. You can get started with Azure with a free account. Once you decide to purchase a subscription plan, you can choose from a variety ofpurchase options. If you’re an MSDN subscriber, you get free monthly credits that you can use with Azure services, including Azure Storage. See Azure Storage Pricing for information on volume pricing.

To learn how to create a storage account, see Create a storage account for more details. You can create up to 100 uniquely named storage accounts with a single subscription. See Azure Storage Scalability and Performance Targets for details about storage account limits.

Storage Service Versions

The Azure Storage services are regularly updated with support for new features. The Azure Storage services REST API reference describes each supported version and its features. We recommend that you use the latest version whenever possible. For information on the latest version of the Azure Storage services, as well as information on previous versions, seeVersioning for the Azure Storage Services.

Blob Storage

For users with large amounts of unstructured object data to store in the cloud, Blob storage offers a cost-effective and scalable solution. You can use Blob storage to store content such as:

• Documents
• Social data such as photos, videos, music, and blogs
• Backups of files, computers, databases, and devices
• Images and text for web applications
• Configuration data for cloud applications
• Big data, such as logs and other large datasets

Every blob is organized into a container. Containers also provide a useful way to assign security policies to groups of objects. A storage account can contain any number of containers, and a container can contain any number of blobs, up to the 500 TB capacity limit of the storage account.

Blob storage offers three types of blobs, block blobs, append blobs, and page blobs (disks).

• Block blobs are optimized for streaming and storing cloud objects, and are a good choice for storing documents, media files, backups etc.
• Append blobs are similar to block blobs, but are optimized for append operations. An append blob can be updated only by adding a new block to the end. Append blobs are a good choice for scenarios such as logging, where new data needs to be written only to the end of the blob.
• Page blobs are optimized for representing IaaS disks and supporting random writes, and may be up to 1 TB in size. An Azure virtual machine network attached IaaS disk is a VHD stored as a page blob.

For very large datasets where network constraints make uploading or downloading data to Blob storage over the wire unrealistic, you can ship a hard drive to Microsoft to import or export data directly from the data center. See Use the Microsoft Azure Import/Export Service to Transfer Data to Blob Storage.

Table storage

Modern applications often demand data stores with greater scalability and flexibility than previous generations of software required. Table storage offers highly available, massively scalable storage, so that your application can automatically scale to meet user demand. Table storage is Microsoft’s NoSQL key/attribute store – it has a schemaless design, making it different from traditional relational databases. With a schemaless data store, it's easy to adapt your data as the needs of your application evolve. Table storage is easy to use, so developers can create applications quickly. Access to data is fast and cost-effective for all kinds of applications. Table storage is typically significantly lower in cost than traditional SQL for similar volumes of data.

Table storage is a key-attribute store, meaning that every value in a table is stored with a typed property name. The property name can be used for filtering and specifying selection criteria. A collection of properties and their values comprise an entity. Since Table storage is schemaless, two entities in the same table can contain different collections of properties, and those properties can be of different types.

You can use Table storage to store flexible datasets, such as user data for web applications, address books, device information, and any other type of metadata that your service requires. You can store any number of entities in a table, and a storage account may contain any number of tables, up to the capacity limit of the storage account.

Like Blobs and Queues, developers can manage and access Table storage using standard REST protocols, however Table storage also supports a subset of the OData protocol, simplifying advanced querying capabilities and enabling both JSON and AtomPub (XML based) formats.

For today's Internet-based applications, NoSQL databases like Table storage offer a popular alternative to traditional relational databases.

Queue Storage

In designing applications for scale, application components are often decoupled, so that they can scale independently. Queue storage provides a reliable messaging solution for asynchronous communication between application components, whether they are running in the cloud, on the desktop, on an on-premises server, or on a mobile device. Queue storage also supports managing asynchronous tasks and building process workflows.

A storage account can contain any number of queues. A queue can contain any number of messages, up to the capacity limit of the storage account. Individual messages may be up to 64 KB in size.

File Storage

Azure File storage offers cloud-based SMB file shares, so that you can migrate legacy applications that rely on file shares to Azure quickly and without costly rewrites. With Azure File storage, applications running in Azure virtual machines or cloud services can mount a file share in the cloud, just as a desktop application mounts a typical SMB share. Any number of application components can then mount and access the File storage share simultaneously.

Since a File storage share is a standard SMB file share, applications running in Azure can access data in the share via file sytem I/O APIs. Developers can therefore leverage their existing code and skills to migrate existing applications. IT Pros can use PowerShell cmdlets to create, mount, and manage File storage shares as part of the administration of Azure applications.

Like the other Azure storage services, File storage exposes a REST API for accessing data in a share. On-premise applications can call the File storage REST API to access data in a file share. This way, an enterprise can choose to migrate some legacy applications to Azure and continue running others from within their own organization. Note that mounting a file share is only possible for applications running in Azure; an on-premises application may only access the file share via the REST API.

Distributed applications can also use File storage to store and share useful application data and development and testing tools. For example, an application may store configuration files and diagnostic data such as logs, metrics, and crash dumps in a File storage share so that they are available to multiple virtual machines or roles. Developers and administrators can store utilities that they need to build or manage an application in a File storage share that is available to all components, rather than installing them on every virtual machine or role instance.

High Avalibility with Storage Replications

Replication for Durability and High Availability

The data in your Microsoft Azure storage account is always replicated to ensure durability and high availability, meeting the SLA for Storage even in the face of transient hardware failures.

See Azure Regions for more information about what services are available in each region.

When you create a storage account, you must select one of the following replication options:

• Locally redundant storage (LRS). Locally redundant storage maintains three copies of your data. LRS is replicated three times within a single facility in a single region. LRS protects your data from normal hardware failures, but not from the failure of a single facility.
  LRS is offered at a discount. For maximum durability, we recommend that you use geo-redundant storage, described below.
• Zone-redundant storage (ZRS). Zone-redundant storage maintains three copies of your data. ZRS is replicated three times across two to three facilities, either within a single region or across two regions, providing higher durability than LRS. ZRS ensures that your data is durable within a single region.
  ZRS provides a higher level of durability than LRS; however, for maximum durability, we recommend that you use geo-redundant storage, described below.

  Note:

  ZRS is currently available only for block blobs, and is only supported for versions 2014-02-14 and later.

  Once you have created your storage account and selected ZRS, you cannot convert it to use to any other type of replication, or vice versa.
• Geo-redundant storage (GRS). GRS maintains six copies of your data. With GRS, your data is replicated three times within the primary region, and is also replicated three times in a secondary region hundreds of miles away from the primary region, providing the highest level of durability. In the event of a failure at the primary region, Azure Storage will failover to the secondary region. GRS ensures that your data is durable in two separate regions.
  For information about primary and secondary pairings by region, see Azure Regions.
• Read access geo-redundant storage (RA-GRS). Read access geo-redundant storage is enabled for your storage account by default when you create it. Read access geo-redundant storage replicates your data to a secondary geographic location, and also provides read access to your data in the secondary location. Read-access geo-redundant storage allows you to access your data from either the primary or the secondary location, in the event that one location becomes unavailable.

  
  

Understanding Storage Accounts in Azure

General-purpose Storage Accounts

A general-purpose storage account gives you access to Azure Storage services such as Tables, Queues, Files, Blobs and Azure virtual machine disks under a single account. This type of storage account has two performance tiers:

• A standard storage performance tier which allows you to store Tables, Queues, Files, Blobs and Azure virtual machine disks.
• A premium storage performance tier which currently only supports Azure virtual machine disks. See Premium Storage: High-Performance Storage for Azure Virtual Machine Workloads for an in-depth overview of Premium storage.

Blob Storage Accounts

A Blob storage account is a specialized storage account for storing your unstructured data as blobs (objects) in Azure Storage. Blob storage accounts are similar to your existing general-purpose storage accounts and share all the great durability, availability, scalability, and performance features that you use today including 100% API consistency for block blobs and append blobs. For applications requiring only block or append blob storage, we recommend using Blob storage accounts.

Note:

Blob storage accounts support only block and append blobs, and not page blobs.

Blob storage accounts expose the Access Tier attribute which can be specified during account creation and modified later as needed. There are two types of access tiers that can be specified based on your data access pattern:

• A Hot access tier which indicates that the objects in the storage account will be more frequently accessed. This allows you to store data at a lower access cost.
• A Cool access tier which indicates that the objects in the storage account will be less frequently accessed. This allows you to store data at a lower data storage cost.

If there is a change in the usage pattern of your data, you can also switch between these access tiers at any time. Changing the access tier may result in additional charges. Please seePricing and billing for Blob storage accounts for more details.

For more details on Blob storage accounts, see Azure Blob Storage: Cool and Hot tiers.

Before you can create a storage account, you must have an Azure subscription, which is a plan that gives you access to a variety of Azure services. You can get started with Azure with a free account. Once you decide to purchase a subscription plan, you can choose from a variety ofpurchase options. If you’re an MSDN subscriber, you get free monthly credits that you can use with Azure services, including Azure Storage. See Azure Storage Pricing for information on volume pricing.

To learn how to create a storage account, see Create a storage account for more details. You can create up to 100 uniquely named storage accounts with a single subscription. See Azure Storage Scalability and Performance Targets for details about storage account limits.

Storage account billing

You are billed for Azure Storage usage based on your storage account. Storage costs are based on the following factors: region/location, account type, storage capacity, replication scheme, storage transactions, and data egress.

• Region refers to the geographical region in which your account is based.
• Account type refers to whether you are using a general-purpose storage account or a Blob storage account. With a Blob storage account, the access tier also determines the billing model for the account.
• Storage capacity refers to how much of your storage account allotment you are using to store data.
• Replication determines how many copies of your data are maintained at one time, and in what locations.
• Transactions refer to all read and write operations to Azure Storage.
• Data egress refers to data transferred out of an Azure region. When the data in your storage account is accessed by an application that is not running in the same region, you are charged for data egress. (For Azure services, you can take steps to group your data and services in the same data centers to reduce or eliminate data egress charges.)

Understanding Caching

Understanding Caching

Caching Approach

Characteristics

Advantages

Disadvantages

Comments

Caching Services

Out of Process

In Memory

Distributed

Accessible by all types of Azure technologies. Scalable.

Extra cost for Service

For building high-performance, scalable cloud systems.

Azure In-Role Caching

Out of Process

In Memory

Distributed

Can run in the memory space of existing Roles

Only available to Web/Worker Roles within Cloud Services.

Suitable for adaptation to existing applications migrated to Azure.

ASP.NET/IIS Cache

In-Process

In Memory

Single-Locale

Runs in the memory space of the existing application.

Requires cache synchronization to work with scalable Azure applications.

Will require adaptations to use distributed caching when used with Azure load balancing.

Azure SQL Database

Out of Process

Out of Memory

Distributed

Data is persistently stored in a database table.

Requires custom coding to serialize data, and for management such as expiration.

Slower than in-memory applications, may be throttled in periods of high request.

Azure Table Storage

Out of Process

Out of Memory

Distributed

Data is persistently stored in Azure table storage.

High performance for very large datasets, cost-effective. Requires custom code.

Slower than in-memory, but faster than SQL. May be throttled in periods of high requests.

Azure Blob Storage

Out of Process

Out of Memory

Distributed

Data is persistently stored in Azure Blob storage.

Can leverage geo-redundant persistent storage. Requires custom code.

Slower than in memory, but easy to make highly distributed.

ASP.NET Azure Session State in Azure

ASP.NET Azure Session State in Azure

In addition to the data caching, we can also use Windows Azure Caching as the storage mechanism for ASP.NET Session State and ASP.NET Page Output Caching. The Session State Provider for Windows Azure Caching and the Output Cache Provider for Windows Azure Caching are the out-of-process storage for ASP.NET applications which will greatly work with Windows Azure load balancers. To use Windows Azure Caching as the storage mechanism for ASP.NET Session State and ASP.NET Page Output Caching, first you need to configure Cache Cluster (using one of the two approaches explained above) and then you need to configure your ASP.NET application to use the Windows Azure Caching. After you have configured the Cache Cluster, you can use Windows Azure Caching for ASP.NET Session State and ASP.NET Page Output Caching by adding necessary configuration setting in the web.config file.

The following configuration setting in the web.config allows to use the Session State Provider for Windows Azure Caching for web applications.

  
              type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache"
          cacheName="default"
          useBlobMode="true"
          dataCacheClientName="default" />
  

The cacheName specifies the name of the Cache used for persisting Session State data.

The following configuration setting in the web.config configures the Session State Provider for Windows Azure Caching for web applications

  
    
                     type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache"
            cacheName="default"
            dataCacheClientName="default" />
    
  

Programming Azure Cache

Programming Azure Cache

Programming with Windows Azure Caching

In order to work with Windows Azure Caching from client apps, you need to add the following namespace in your client app:

using Microsoft.ApplicationServer.Caching;

To programmatically access the Windows Azure Caching, you first need to create an instance of DataCache class.

The following code block will create a cache client by creating an instance of DataCache which will take the settings from configuration file.

//Creating a cache object
DataCacheFactory cacheFactory = new DataCacheFactory();
DataCache dataCache = cacheFactory.GetDefaultCache();

The above code block will create the cache client object by using DataCacheFactory object. The GetDefaultCache method of DataCacheFactory object will return the DataCache object.

Adding and Retrieving Data from the Windows Azure Cache

To add an item to Cache object, we can use either Add or Put method of DataCache object. The Add method adds an object to the Cache object with a specified key and will generate an exception if the item is already exists with the same key. The put method adds the object to the Cache object with a specified key if the key does not exist, and replaces if the key is already exists.

The following code block adds the collection of data objects to Cache:

//Query from DB and adding data to Cache 
var categories = categoryRepository.GetAll();
dataCache.Add("categories", categories);

In the above code block, querying data from Database and adding this data to Cache object with a key name “categories” and we can take the data from Cache for the subsequent requests for the same data which can avoid database round trips and improve the performance of the application.

Using the overloads of the Add and Put methods of DataCache object, we can override the timeout interval, which are specified in the Cache Roles settings.

The following code block adds the data to the Cache with time span that allows to specify when the object should expired from Cache.

dataCache.Add("categories", categories, TimeSpan.FromMinutes(45));

The above code block adds the data to the Cache object with a key categories and specified the timeout as 45 minutes.

The following code block retrieves the data from Cache:

//Getting the Cache item with key "categories"
IEnumerable category = dataCache.Get("categories") as IEnumerable;

The Get method of the DataCache object retrieves the data from Cache where you have to give the key as the parameter value for the Get method. The above code block retrieves the data and casting it for the appropriate data type.

Working with a Cache Provider Class

In the previous steps, we have discussed how to perform data caching with Windows Azure Caching. Let’s create a Cache provider class that will help us better manage Data in the cache. Let’s create a contract type for the Cache provider.

public interface ICacheProvider
{
    object Get(string key);
    void Add(string key, object data);
    void Put(string key, object data);
    void Add(string key, object data, TimeSpan? timeout);
    void Put(string key, object data, TimeSpan? timeout);
    bool IsSet(string key);
    void Remove(string key);
}

The Get method returns the cached item based on the given key and Add and Put methods adds and puts an item into Cache. The IsSet method will return true if an item added on the Cache. The Remove method will remove an item from the Cache.

Let’s create a concrete implementation for the ICacheProvider. The below AzureCacheProvider Class is an implementation of ICacheProvider for Windows Azure Caching.

public class  AzureCacheProvider : ICacheProvider
{
private DataCache dataCache=null;
public AzureCacheProvider()
{
    //Creating a cache object by using DataCacheFactory
    DataCacheFactory cacheFactory = new DataCacheFactory();
     dataCache = cacheFactory.GetDefaultCache();
}
//Get cached item
public object Get(string key)
{
   return dataCache.Get(key);
}
public void Add(string key, object data)
{
    Add(key, data, null);
}       
public void Add(string key, object data, TimeSpan? timeout)
{
   //Add item to Cache
    if (timeout.HasValue)
    {
        dataCache.Add(key, data, timeout.Value);
    }
    else
    {
        dataCache.Add(key, data);
    }
}
public void Put(string key, object data)
{
    Put(key, data, null);
}
public void Put(string key, object data, TimeSpan? timeout)
{
    //Put item to Cache
    if (timeout.HasValue)
    {
        dataCache.Put(key, data, timeout.Value);
    }
    else
    {
        dataCache.Put(key, data);
    }
}
//Returns true if Cache item is exist
public bool IsSet(string key)
{            
    return (dataCache.Get(key) != null);
}
public void Remove(string key)
{
    dataCache.Remove(key);
}
}

In the below code block, we are using the ICacheProvider from an ASP.NET MVC application

private readonly ICacheProvider cache;
public CategoryController(ICategoryRepository categoryRepository,ICacheProvider cache)
{            
    this.categoryRepository = categoryRepository;
    this.cache = cache;
}      
public ActionResult Index()
{
    IEnumerable categories;
    var cachedCategories=cache.Get("categories");

    if (cachedCategories != null)
    {
        categories = cachedCategories as IEnumerable;
    }
    else
    {
        categories = categoryRepository.GetAll();
        cache.Put("categories", categories);
    }            
    return View(categories);
}      

In the Index action method, we are taking the data from Cache, if the cached item exists. If the item does not exist, we retrieve the data from Database and put it on Cache so that we can take the data from Cache for the subsequent requests. In this sample code, we are injecting an instance of AzureCacheProvider class through constructor for the contract type ICacheProvider.

Working with Cache in Azure

Working with Caching in Azure

Windows Azure Caching

Windows Azure is providing a caching service which can be used as a caching layer for your Cloud apps hosted on Windows Azure. Windows Azure Caching service is a highly scalable, distributed, in-memory Cache which allows developers to cache all data types, thus reducing the load on databases and improving the performance of the Windows Azure applications.

Using Windows Azure Caching, you can do the following:

• Cache data on Windows Azure Caching
• Use as storage mechanism for ASP.NET session state and ASP.NET page output caching

The new Windows Azure Caching allows you to deploy Cache on the virtual machines in which you have hosted your Windows Azure Cloud Services. Now you can cache on Web Roles and Worker Roles that can be easily scaled, managed and monitored just like your Cloud Services. You can also use the existing Windows Azure Shared Caching along with the new Caching option on the Cloud Services.

Cache Cluster

Cache Cluster is the Cloud Service role instance that you use for hosting your Cache. You can deploy the cache cluster in two different ways. You can deploy in a dedicated VM as well as you can share Caching with your Role instances.

The following are the two types of deployment options available for configuring cache.

· Co-located Role caching - The Cache shares the Virtual Machine resources with the Cloud Service application. In this option, you can share the resources of Web Role and Worker Role for Cache.

· Dedicated Role caching - The role instances dedicated only for your caching. In this way, you would be deploy the Cache in a dedicated VM by using a Cache Worker Role.

Configuring Co-located Role Cache Cluster

If you want to configure your cache along with your Role instances, you can choose the co-located Role cache cluster as the deployment option.

The following steps will configure the Windows Azure Role for using co-located Role Caching

1. In Visual-Studio, right-click on the Windows Azure Role, and click properties.

2. In the properties window, choose the 'Caching' tab.

3. In the Caching tab, check the 'Enable Caching (Preview Release)' checkbox.

4. Choose 'Co-located Role' for the Cache Cluster settings. By default, this will be 'Co-located Role' when we enable Caching.

5. Choose the percentage of Cache size. By default Cache size will be 30 percentage. This means that 30 percentage of VM memory allocated for Cache. You can change the Cache size whatever you want for your caching solution.

6. Configure the settings of Caches. We can specify the 'Time to Live (min)' for the time interval of Cache for expiry and 'Expiration Type' for specifying the expiration type of Cache object.

By default, value of “The Time to Live” would be 10 minutes and the Expiration Type would be Absolute. The Absolute expiration type is specifying that the timer for the expiration would be start after the data added into Cache object. If the expiration time is 10 minutes, the cache will be expire after 10 minutes when the item is added into Cache. The other expiration types are 'Sliding Window' and 'None'. If the Expiration Type is 'Sliding Window', the timer for the expiration time would be reset after each item is retrieved from the Cache. If expiration type is 'None', items in the cache will not be expired. In this scenario, you have to specify “Time to Live (min)” as 0.

Configuring Dedicated Role Cache Cluster

If you want to host your cache in a dedicated VM, you can choose the Dedicated Role cache cluster as the deployment option.

The following steps will configure the Dedicated Role cache cluster

1. Add Cache Worker Role project - To configure Dedicated Role cache cluster, you need to add a Cache Worker Role to your solution. Add a New Cache Worker Role project onto
your Windows Azure project.

When we are adding a new Cache Worker Role project, it will automatically enable Caching with Dedicated Role as the Cache cluster.

2. Configure the settings for Dedicated Cache Role - Right-click on the Cache Worker Role from the Windows Azure project, and click properties and choose Configuration

By default, the VM instance count would be 1 and Virtual Machine size would be small. You can change these setting based on your hosting plan for Dedicated Role Cache Cluster.

3. Configure the Cache settings for expiration time and expiration type.

Working with Cache Clients

In order to work with Windows Azure caching from Cloud Services applications, we need to install Windows Azure Caching client library onto your Web Role and Worker Role applications. The following steps will add a Windows Azure client library to a Web Role application. We can install the Windows Azure Caching client library by using NuGet.

From the following NuGet packages, Install Microsoft.WindowsAzure.Caching

The Windows Azure Caching NuGet package will install the necessary assmebly reference and will also add the configuartion setting on the configuaration file. You can also manually add the necessary assmeblies from the folder C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\2012-06\ref\CachingPreview if you have installed Windows Azure SDK 1.7 on your system. At the time of this writing, Azure SDK 1.8 is the latest.

The Windows Azure Caching NuGet package will add a configuration element dataCacheClients into the web.config file.

The following configuration setting in the web.config file will configured a cache client for a web role application.

    
   

  
  
    
    
      
      
    
  

The identifier in the config settings specifies the name of the role that hosts the cache cluster