Introducing Node-Azure

Node-Azure is a Node.js package (available via NPM) which allows native javascript access to the Windows Azure storage API.

Node-Azure currently supports most of methods on these storage types:

• Blob storage
• Table storage
• Queues

Getting Started

To include the node-azure dependency, and set up the details of your account, include these lines of code:

// include the node-azure dependency
var azure = require('azure');
	
// every request has an account parameter, which is an object like this:
var account = {
  name : "YOURACCOUNTNAME",
  key : "YOURACCOUNTKEY",
  blob_storage_url : "https://YOURACCOUNTNAME.blob.core.windows.net",
  table_storage_url : "https://YOURACCOUNTNAME.table.core.windows.net",
  queue_storage_url : "https://YOURACCOUNTNAME.queue.core.windows.net"
}

Blob Examples

Upload text as a blob:

azure.blob.put_blob(account, "container", azure.blob.BlockBlob, "blobname",
  "Hello world!", {'Content-Type': "text/plain"}, callback);

Download a blob:

azure.blob.get_blob(account, "container", "blobname", function(x) {
  // x == "Hello world!"
});

Table Examples

Insert an entity:

azure.tables.insert_entity(account, 'tablename',
  { RowKey:'123', PartitionKey: 'xyz', Value: 'foo' }, callback);

Get an entity:

azure.tables.get_entity(account, 'tablename', 'xyz', '123', function(entity){
  // x == { RowKey:'123', PartitionKey: 'xyz', Value: 'foo' }
});

Query a table:

azure.tables.query_entities(account, 'tablename', "Value+eq+'foo'", function(entities){
  // entities is an array of matching items
});

Queue Examples

Put a message on a queue:

azure.queues.put_message(account, q, {Test:"Message"}, callback);

Pop the message off the queue:

azure.queues.get_message(account, q, function(message){
  // our javascript object is returned: message.Data
});

To install

run:

npm install node-azure

Or alternatively, manually copy the repository into a node_modules/node-azure folder.

Advertisement

Lucene.NET

An example of how to create a Lucene.NET index, add a document and search it:

using System;
using System.IO;
using Lucene.Net.Analysis;
using Lucene.Net.Documents;
using Lucene.Net.Index;
using Lucene.Net.QueryParsers;
using Lucene.Net.Search;
using Lucene.Net.Store;

class Program
{
    static void Main(string[] args)
    {
        // create the index
        var directory = FSDirectory.Open(new DirectoryInfo(@"d:\index\"));
        var analyzer = new SimpleAnalyzer();
        var indexWriter = new IndexWriter(directory, analyzer, IndexWriter.MaxFieldLength.UNLIMITED);
        var doc = new Document();

        // read a file into the index (in this case a file containing lorem ipsum)
        using (StreamReader sr = new StreamReader(@"d:\foo.txt"))
        {
            doc.Add(new Field("id", @"D:\foo.txt", Field.Store.YES, Field.Index.NO));
            doc.Add(new Field("content", sr.ReadToEnd(), Field.Store.YES, Field.Index.ANALYZED));
        }
        indexWriter.AddDocument(doc);
        indexWriter.Optimize();
        indexWriter.Commit();
        indexWriter.Close();
        // we have created and saved the index

        // create a query against the index
        var queryParser = new QueryParser(Lucene.Net.Util.Version.LUCENE_29, "content", analyzer);
        var query = queryParser.Parse("Lorem");
        var indexSearcher = new IndexSearcher(directory, true);
        var hits = indexSearcher.Search(query);     // this method is to be deprecated, however, I can't find an easier way of doing it!

        // display the results
        Console.WriteLine("Found {0} results", hits.Length());
        for (int i = 0; i < hits.Length(); i++)
        {
            var doc2 = hits.Doc(i);
            Console.WriteLine("Result num {0}, score {1}", i + 1, hits.Score(i));
            Console.WriteLine("ID: {0}", doc2.Get("id"));
            Console.WriteLine("Text found: {0}" + Environment.NewLine, doc2.Get("content"));
        }

        // close 
        indexSearcher.Close();
        directory.Close();
    }
}

Never have I worked with a library with so many obsolete and deprecated methods. It’s quite hard going!

Windows Azure and Node.js

As Rob Blackwell pointed out; Node.js has a clear part to play in cloud computing. But what does an application powered by Node look like? Can we build something entirely with javascript and run it on Azure now? I decided to explore this by building a simple todo application.

The application lists the items in your todo list, new items are added by typing in the input box, and pressing ‘Add’:

Clicking on an item shows a different page, and asks you if you want to delete it.

Very simple. There is no security, no error trapping, and it looks awful in Internet Explorer – but it proves a concept. Let’s dig under the covers.

Design Tenets

1. The application running in the browser should be a static HTML file. This allows us to consider offline use with HTML5 storage if available.
2. When the application is running, only data (JSON) should be exchanged with the server. We should not be rendering markup on the server. This keeps our bandwidth requirements to a minimum.
3. The only code I am going to write is javascript.

The User Interface

The application is a single static HTML file, and uses a browser-based MVC architecture. The application uses a number of standard javascript libraries. In particular, these libraries are used:

Backbone.js

Backbone provides the client-side routing in the application. All links in the application link to anchors (i.e. they only modify parts of the URL after the  ‘#’ ). These links do not cause a round trip back to the server, instead, backbone picks them up and routes you to a controller. So in this application I have these two routes set up:

var AppRouter = Backbone.Router.extend({
  routes: {
    "/view/:id": "viewItem",
    "*actions": "defaultRoute"
  },
  defaultRoute: function(actions){
    two10.actions.defaultAction();
  },
  viewItem: function(id){
    two10.actions.view(id);
  }
});
var router = new AppRouter;

So when the URL matches the first route (i.e.  ‘…/#/view/1234’) the ‘two10.actions.view’ function is called, passing in ‘1234’. Otherwise the second route is matched, and ‘two10.actions.defaultAction’ is called.

This simple piece of functionality is very powerful, as it means my application can be bookmarked. It also means I don’t have a complicated web of javascript functions dotted around my application, I mostly just use links like this

  <a href="#/view/1234">View</a>

JQuery Templates

My controllers need to write html out for the user to see. To do this I use JQuery Templates. There are dozens of options for templating in javascript, I’m just sticking with what I know. The template to display the todo item, with a delete button looks like this (I have simplified the markup):

<script id="view" type="text/x-jquery-tmpl">
  <div>
    <div>
      <h1>${item.get("name")}</h1>
      <a href="#">Back</a>
    </div>
    <div>
      <p>${item.get("name")}</p>
      <p><a href="javascript:two10.actions._delete('${item.cid}');">Delete</a></p>
    </div>
  </div>
</script>

Note how in the link for the delete button bypasses the routing, and invokes the controller directly. This is because I want to emulate a post, and I don’t want the URL for deleting the item to appear in the browser or the routing table.

To actually render the view, my code needs to pass in the model (i.e. the object which holds the todo item):

var item = two10.data.byId(id);        // retrieve the item out of the in-memory collection
$("#body").html($("view").tmpl(item)); // render the template, passing in the item

JQuery Mobile

I attempted at first to use JQuery Mobile to enhance my markup (and make my application look pretty). However there is also a routing capability with JQuery Mobile, which conflicts with Backbone. Despite efforts to disable it, I decided to just use the CSS from JQuery Mobile, and put the enhanced markup in my templates. Whilst this requires more effort when writing the templates, it probably improves runtime performance by reducing the amount of DOM manipulation required in the browser.

Server Side

Serving the page

The server side code is a node.js script. The first thing it’s required to do is to serve up the HTML page to the browser.

var http = require('http');
var fs = require('fs');
http.createServer(function (request, response) {
  switch (request.url) {
    case '/':
      fs.readFile('./default.htm', function(error, content) {
        if (error) {
          response.writeHead(500);
          response.end();
        }
        else {
          response.writeHead(200, { 'Content-Type': 'text/html' });
          response.end(content, 'utf-8');
        }
      });
      break;
 ...

Connecting to Azure Storage

The todo items should be persisted in Azure Table Storage. To do this, I forked a prototype version of the node-azure library which makes it straight forward to save/retrieve records in table storage using these methods:

function insert_entity(account, tablename, data, callback)
function get_entity(account, tablename, partitionKey, rowKey, callback)

The code running on the browser does a GET as soon as the application starts, node retrieves the data from table storage, and just returns it back to the browser. Every time the user changes the items in the todo list, the application does a POST to node, which just saves the data in table storage.

Deploying to Azure

I tried to use the iisnode technique to get node running on a web role, and failed. Instead, I added node as a background startup task to a worker role. These are the steps to set it up:

1. Create a Worker Role, open up port 80, and register a startup command. Your Service Definition file should look like this:

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="WindowsAzureProject2" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WorkerRole name="WorkerRole1" vmsize="ExtraSmall">
    <Startup>
      <Task commandLine="Startup.cmd" executionContext="elevated" taskType="background"/>
    </Startup>
    <Endpoints>
      <InputEndpoint name="Endpoint1" protocol="tcp" port="80" />
    </Endpoints>
  </WorkerRole>
</ServiceDefinition>

2. Create a startup.cmd file, and add it to your Worker Role. The command should contain the instruction to start node, passing in your script:

node server.js

3. Add node.exe, and all of your javascript/html files to your Worker Role project, ensure all these files are set to copy local (node dependencies can be added to a ‘node_modules’ sub-folder).

4. Deploy, and stand back.

Conclusion

Developing in javascript is fun, and more productive than I had thought.

I would like to see more investment in node on the windows platform, currently the Node Package Manager (NPM) hasn’t been ported, (read how to do this manually) and it would be better to have node running under IIS than a background task in a Worker Role.

The power of  routing and templating in the browser shouldn’t be underestimated. With the addition of some HTML5 features, this technique could seriously compete with Silverlight.

I don’t think this stack is the solution for everything, but if you want multi-device support, good scalability and low bandwidth requirements, you should certainly consider it. Bear in mind though, node is in it’s infancy, this is the bleeding edge.

Source

The entire source can be downloaded here.

Pattern for processing Azure Queue Messages in parallel

By using a function that dequeus messages as an IEnumerable, you can easily plug in the parallel task library to process message simultaneously.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.StorageClient;

class Program
{
    static void Main(string[] args)
    {
        var account = CloudStorageAccount.DevelopmentStorageAccount;
        var client = account.CreateCloudQueueClient();
        var queue = client.GetQueueReference("foo");

        Parallel.ForEach<CloudQueueMessage>(GetMessages(queue), (message) =>
        {
            // process your message here
            Console.WriteLine(message.AsString);
            queue.DeleteMessage(message);
        });
    }

    private static IEnumerable<CloudQueueMessage> GetMessages(CloudQueue queue)
    {
        while (true)
        {
            var message = queue.GetMessage();
            if (message != null)
            {
                yield return message;
            }
            else
            {
                yield break;
            }
        }
    }
}

The added advantage to this approach, is that messages added to the queue once the loop has started will still get processed. You could wrap the whole Parallel.ForEach in a loop, and use Thread.Sleep to pause whilst there is nothing in the queue to process. 

NPM NOW WORKS IN WINDOWS IGNORE THIS POST…

NPM NOW WORKS IN WINDOWS – IGNORE THIS POST

Windows is missing the NPM (Node Package Manager) – so the easiest way I have found to manually install a node package, is to create a ‘node_modules’ folder, in the same location as your .js file, and copy in the package’s javascript file (normally located in the lib folder).

You can then include you package in the normal way:

var package = require('package');

Running Classic ASP on Azure

By default, classic asp is not enabled on a Windows Azure Web Role. You can easily install it with a startup task:

1. Create a Startup.cmd file in your ASP.NET project.

2. By default text files are saved as ‘Unicode (UTF-8 with signature) – Codepage 65001’. Unfortunatly this seems to add an extra unwanted character to the start of the file, which prevents the script form being run. Go to ‘Advanced Save Options…’ in the file menu of Visual Studio, and select ‘US-ASCII – Codepage 20127’.

3. On the properties of this file, set ‘Copy to Output Directory’ to ‘Copy if newer’.

4. Add this line to the file:

start /w pkgmgr /iu:IIS-ASP

5. Modify your ServiceDefinition file, to add the startup task:

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="ClassicASPonAzure" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WebRole name="WebRole1" vmsize="ExtraSmall">
     <Startup>
      <Task commandLine="Startup.cmd" executionContext="elevated" taskType="simple" />
    </Startup>
...

6. Publish your instance, and you should have classic ASP enabled.

AzureSugar

A lightweight .NET library which makes working with the Azure API easier.

https://github.com/richorama/AzureSugar

Table Storage

Define a class which represents your table, and optionally supply a name using the ‘TableName’ attribute.

[TableName("Customers")]  // <- this is optional
public class Customer : TableServiceEntity
{
    public string Firstname { get; set; }
    public string Lastname { get; set; }
}

The AzureSugarTableContext allows you to create new customers easily:

using (var context = new AzureSugarTableContext(CloudStorageAccount.DevelopmentStorageAccount))
{
    var customer = context.Create<Customer>();
    customer.Firstname = "John";
    customer.Lastname = "Smith";
}

Primary keys (GUIDs) are automatially assigned. All commits are performed on disposal of the context.

It’s just as easy to query the table:

using (var context = new AzureSugarTableContext(CloudStorageAccount.DevelopmentStorageAccount))
{
    foreach (var customer in (from c in context.Query<Customer>() where c.Firstname == "John" select c))
    {
        Console.WriteLine(customer.Firstname);
    }
}

Queues

Queues are strongly typed. Let’s say we are working with this class.

public class Foo
{
    public string Bar { get; set; }
    public string Baz { get; set; }
}

To push a message on to the queue, just do this:

var foo = new Foo { Bar = "bar", Baz = "baz" };
var queue = new AzureSugarQueue<Foo>(CloudStorageAccount.DevelopmentStorageAccount);
queue.Push(foo);

To pop a message from a queue, we just need to do this:

var queue = new AzureSugarQueue<Foo>(CloudStorageAccount.DevelopmentStorageAccount);
using (var message = queue.Pop())
{
    Foo foo = message.Content;
    // do something with foo
}

Your object is automatically deleted from the queue on the disposal of the message, however, you can have more control over this:

using (var message = queue.Pop(false))
{
    var foo2 = message.Content;
    message.VoteCommit();
}

In this case, the message will only be deleted if ‘VoteCommit’ is called.

The queue name is automatically derived from the type name ‘Foo’, however you can override this as well.

Sticky Sessions and Windows Azure

One of the problems that often crops up with moving legacy applications to the cloud is reliance of sticky sessions. The Azure load balancer works on a round-robin basis, so if your application has been designed to work on a sticky session basis, you may have some work to do.

Although it is possible to use Sticky Sessions in Azure (see this example) it’s not a great fit for cloud architecture, for these reasons:

1. When you provision new instances, only new sessions will be routed to them. Depending on your load balancing logic, new sessions may also still be provisioned on the old instances. This results in it taking a long time (depending on the average length of your session) for load to be evenly distributed across your instances.
2. Cloud solutions should be designed to fail. One of your instances may be removed at any time for patching, or hardware failure. With a sticky session scenario, the clients with a session on this instance will probably have to log in again, or may see an error page, depending on your load balancing logic.
3. Unless your load balancer shares state in some way, it’s likely that you’re load balancing with a single instance (single point of failure) and not eligible for the Microsoft SLA.

So what options are there?

1. Use the Azure AppFabric Cache to share ASP.NET session state across your instances. This walk through shows you how, by simply changing your Web.config.
2. Ideally move to a stateless architecture, making use of cookies or the database where appropriate. Designing an application in this way will provide you with the best scaling potential, and allow you to get the most from the cloud.

Mount Azure Blob Storage as a Windows Drive

Do you need to access blobs as files from your local computer or Azure? AzureBlobDrive solves this problem by mounting blob storage as drive in Windows. Containers appear as a folders, each blob is displayed as a file:

AzureBlobDrive can run either on an Azure instance, or on your local machine.

Works well for these scenarios:

+ If you have an existing application which you want to migrate to Azure, but it needs to read/write to a persistent drive.

+ If you have an application split across Azure and a local data centre, and they need to share files.

+ If you want convenient access to blobs and containers from your local computer.

Not suitable for these scenarios:

• Reading/writing large files.

• Storing files that change a lot (i.e. database files).

Demonstration

Watch a video of AzureBlobDrive in action.

Run locally

1. Install the Dokan driver, which can be downloaded from here: http://dokan-dev.net/wp-content/uploads/DokanInstall_0.6.0.exe
2. Download the AzureBlobDrive source: https://github.com/richorama/AzureBlobDrive
3. Update the app.config file in the Two10.AzureBlobDrive.Console project, to point to your Azure storage account (alternatively you can use the Azure emulator).
4. Run the Two10.AzureBlobDrive.Console project. An ‘R’ drive will be mounted, you should be able to see your containers.

Run in Azure

1. Download the AzureBlobDrive source: https://github.com/richorama/AzureBlobDrive
2. Update the Two10.AzureBlobDrive.Console.exe.config in the Two10.AzureBlobDrive.WorkerRole project, to point to your Azure storage account.
3. Build the solution.
4. Publish the Two10.AzureBlobDrive.CloudProject project to Azure, the role will install the Dokan driver with an elevated startup command, and the worker role will mount the ‘R’ drive in the ‘Run’ method.

Current limitations

AzureBlobDrive is alpha quality code, and has a number of limitations. Some of these limitations are passed on by the inherent limitations of Azure blob storage.

• Files can only be placed in a folder (container). You cannot have files in the root directory.
• Folders cannot contain folders (blob storage does not support hierarchy).
• Root folder names (containers) must be in lower case, and cannot contain spaces and other special characters.
• Performance is poor, and large files are not recommended.
• Files and folders are cached for one minute, so changes made by other machines may not be instantly viewable.
• Files (blobs) cannot be empty (i.e. have a zero size).

How does it work?

AzureBlobDrive uses the Dokan file system driver (http://dokan-dev.net/en/). Dokan provides you with an interface for calls to the file system, such as ReadFile, WriteFile etc… The Two10.AzureBlobDrive project contains a class which implements these methods, accessing blob storage to retrieve the information. The Azure Worker Role is configured to install the Dokan dependencies (Startup.cmd), and then start the Two10.AzureBlobDrive.Console application as a background task.

Some of these limitations will be improved over time. Feel free to fork the repo!

https://github.com/richorama/AzureBlobDrive