Hosting Go (or any CGI apps) on Windows Azure Websites

Run Process on Website

It’s a little known fact, but you can spawn some server-side processes from your Windows Azure Website.

For example, this Node server will execute any command you pass in on the end of the URL:

var childProcess = require('child_process');
var http = require('http');
http.createServer(function(req,res){
    var cmd = req.url.replace("/","").replace(/%20/g, " ");
    childProcess.exec(cmd, function(err, stdout, stderr){
    if (err)
      res.end(JSON.stringify(err));
    else 
      res.end(stdout);
  });
}).listen(process.env.port);

…so running a command like this:

http://xxx.azurewebsites.net/dir

Will give you something like this:

 Volume in drive C has no label.
 Volume Serial Number is 7C6E-015C

 Directory of C:\DWASFiles\Sites\xxx\VirtualDirectory0\site\wwwroot

08/29/2013  01:10 PM                   .
08/29/2013  01:10 PM                   ..
08/29/2013  12:46 PM                44 .gitignore
08/29/2013  12:46 PM               915 iisnode.yml
08/29/2013  12:46 PM               366 server.js
08/29/2013  01:33 PM               640 web.config
               4 File(s)      1,288,915 bytes
               2 Dir(s)   1,069,420,544 bytes free

In fact, you can bring your own application and run that. Just add the .exe, and any dependencies, and start the process as shown in the first example.

This is exactly how CGI works.

Create a CGI Host

There is a node module which will host CGI applications using Node’s HTTP server: https://github.com/TooTallNate/node-cgi

So we can use Node as a CGI host for any application which compiles to Windows!

To create a server, install the cgi module:

$ npm install cgi

And create a server.js file like this:

var cgi = require('cgi');
var http = require('http');
var path = require('path');

var script = path.resolve("./", 'NAME_OF_YOUR_APP.EXE');

http.createServer( cgi(script) ).listen(process.env.port);

Just replace NAME_OF_YOUR_APP.EXE with the name of your application, this executable should be included in your git repository (assuming you’re using git to push to Website!).

Create a CGI Application

To test this, I created a CGI application in Go (test.go):

package main

import (
  "net/http"
  "net/http/cgi"
)

func myHandler(rw http.ResponseWriter, req *http.Request) {
  rw.Header().Add("Content-Type", "text/plain")
  rw.Write([]byte("Hello World!"))
}

func main() {
  http.HandleFunc("/", myHandler)
  cgi.Serve(nil)
}

To compile, just use:

> go build test.go

I added the test.exe file to my repository, switched the ‘NAME_OF_YOUR_APP.EXE’ to ‘test.exe’, and pushed up the changes:

It works! And shows that we can host CGI apps in Windows Azure Websites using languages outside the subset that have primary support on Azure.

Ok, CGI is not the fastest piece of technology in the world, and web programming has moved on, but it could help people move legacy applications up to the Cloud.

Advertisement

Running LevelDB in Windows Azure Websites

LevelDB is becoming an interesting space in the Node.js community. Some healthy innovation is showing how a modular approach to building your own database has some compelling benefits.

Don’t know what LevelDB is? Find out more here.

Windows Azure Websites offers a fast and convenient way to get Node.js applications running quickly. It’s backed by a persistent disk which is replicated for you, so why not make use of that free space by putting your database on there?

Here are some steps to getting LevelDB running in a Node.js application on Windows Azure Websites.

Prerequisites

Before you start, you must be using Windows (the 8.1 preview is currently free to download), and have the following installed:

• 32 Bit version of Node.js.
• Visual Studio C++ 2010 (the free ‘express’ edition will do).
• Python 2.7.x.
• Node-gyp installed globally.

Why 32 bit? Because node-gyp seems to target the processor architecture of the version of Node you have installed, and the shared (free) version of Windows Azure Websites is 32 bit only.

Adding LevelDB

Use NPM to add LevelDB to your app. The level modules includes levelup and leveldown:

$ npm install level

This will also compile the native leveldown module.

If you have any problems at this stage, I recommend cloning the leveldown repository, running `node-gyp configure`, and opening the .sln file located in the build directory. You’ll need to configure the solution to build Release mode.

Once you’ve installed level, you can start using the database in your code like this:

// create a database 
var db = require('level')('db');

// put a value
db.put('key1', 'value1');

// get a value
db.get('key1', function(err, data){
  console.log(data);
});

See the LevelUp documentation for more examples.

Configuring Git

(I’m assuming you’re using git to deploy this to Azure one way or another!)

Adding the modules to git

Because Leveldown is a native module, you can’t compile it on Azure. Therefore you must add the compiled binary to your git repo. That’s straight forward enough, but you’ll find that you have 60MB of dependencies, which is  quite a bit to upload to the Azure Website repo. It’s probably worth removing some of the unnecessary files before you add anything to git.

Trim down the unnecessary files

You can remove everything from this folder, with the exception of the ‘leveldown.node’ file (which is the DLL we just compiled).

  node_modules\level\node_modules\leveldown\build\Release

You can also do away with this folder:

  node_modules\level\node_modules\leveldown\deps

That should bring you down to about 1.2MB.

I found that large git pushes can take a very long time to Azure Websites. I think some throttling takes place after 4MB or so.

Ignore the database files

You should also add the ‘db’ folder (or whatever you choose to name your database) to your .gitignore file, as you don’t want updates to your application to overwrite your database.

Running in Azure

You should now be able to push your application to Windows Azure Websites, and your data will be persisted in LevelDB. However, there are a couple of rules you must stick to:

1. You must keep the number of Node processes to exactly 1. You can scale up to a ‘shared’ or ‘standard’ model, but you can’t scale out. LevelDB cannot be hosted within multiple processes.
2. You are limited to 1GB of data on the disk

Conclusions

LevelDB offers a fast and convenient way to host a database in an Azure Web Site.

Websites run in a 32bit mode, I even found that the 63bit switch didn’t make any difference, but as long as we have the 32bit version of node installed we can compile the native module for this platform.

It’s worth keeping your git repo small by pruning out some of the build artifacts. I did this by deleting the files, but you could do the same with a .gitignore file.

You should keep your database files out of the repository using .gitignore.

You should be careful about the number of Node processes you have running, given the constraints of LevelDB.

It would be interesting to benchmark this against table storage to see if there any performance gains, perhaps another bog post?

Fanning Out Async Await

I wanted to run an async function across a number of items, and I wanted to do this in parallel. This turned out to be quite a straightforward thing to do, once you know how.

Let’s find out how, by trying out a few different approaches.

First of all, here is a piece of async work that needs to be done 100 times. The total needs to be summed up:

Failed Attempt #1

Now, let’s take a really naive approach to running this:

This takes 11000ms to run.

Although we’re using the funky async/await keywords here, we’re  running the code in series, rather than parallel. The ‘await’ keyword will block the execution of the loop until ‘DoWork’ returns.

Failed Attempt #2

Let’s try to improve on this. Where is the first place to go for parallel execution in .NET, the Task Parallel Library right?

This takes 65 ms to run.

Looking good, we’re doing a Parallel For loop now, and we’re using an async lambda. Cool stuff. It also takes just 65ms to run, wow, that’s shorter than the 100ms delay on the task! Unfortunately the TPL doesn’t work well with async/await. The For loop returns before the async tasks have completed. We never get the answer.

Failed Attempt #3

Ok, let’s have another go, and use the Task.WhenAll function, which returns a task which completes when all the tasks you pass it have completed:

This takes 45ms to run.

Even though we’re using the Task.WhenAll, this doesn’t work either. Task.Start doesn’t seem to be compatible with ‘WhenAll’ when async tasks are used. At least it’s faster than the last approach that didn’t work.

Failed Attempt #4

There is a Task.Run, which should be used in this scenario. Let’s modify the sample and try again:

This takes 180ms to run.

Instead of starting the task, we use Task.Run to create and start the task. This now seems to work, and we’re in business. So what’s wrong? Well it turns out that Task.Run dispatches the code to run on the thread pool rathern than the current thread. This isn’t what I want. There’s also a lot of code here, and it doesn’t look very elegant.

Success

Back to the drawing board, and this is what I came up with:

This takes 160ms to run.

When you see the answer it seems obvious, you don’t use await in the loop, you just add all the tasks (promises) to a list, and await the lot when you’ve finished creating them.

Conclusions

Async/await makes code look clean, but the async/parallel/task story in .NET is not a clear one. The parts don’t work together like you’d expect.

You’re never quite sure where your task is going to run, on the thread pool or on the current thread?

Thorough testing of async code is necessary. When awaits are short the await command will return immediately, which means that in real life the scenarios that didn’t work (i.e. attempts #2 & #3) will work correctly. This leads to some weird bugs and race conditions.

Create an RDP file Download Link for an Azure VM

Just because it’s Friday, here’s a route for express (Node.js web framework) which will give you the RDP file for a VM in Azure:

app.get("/rdp/:address", function(req, res){
  res.setHeader('Content-Type', 'application/x-rdp');
  res.setHeader('Content-Disposition', 'attachment; filename=' + req.params.address + '.rdp');
  res.end("full address:s:" + req.params.address + ":3389\r\nprompt for credentials:i:1");
});

To use, add the above lines to your express application and the link below in your web page:

<a href="/rdp/xyz.cloudapp.net">Remote Desktop</a>

(obviously replacing xyz.cloudapp.net with the name of your hosted service)

 

Managing Azure IaaS from Node.js

A Service Management Service was recently added to the Node SDK for Windows Azure. This allows you to interact with the Azure management API, to create, deleted and manage Virtual Machines in Azure.

Getting Set Up

To get started you need to get your management certificates in order. This is probably most easily achieved by creating new ones.

$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
$ openssl x509 -inform pem -in mycert.pem -outform der -out mycert.cer

The mycert.cer file can be uploaded to the Management Portal. The other files we’ll use in our node app.

Next, let’s install the Azure SDK:

$ [sudo] npm install azure

Creating the ServiceManagementService

Now let’s write some JavaScript. To work with the Service Management API, we’ll need an instance of the ServiceManagementService:

var azure = require('azure');
var auth = {keyfile:'mycert.pem',certfile:'mycert.pem'};
var sms = azure.createServiceManagementService('SUBSCRIPTION_ID', auth);

Using the ServiceManagementService to call Azure

Now we have an instance of the SMS, we can start querying the Azure infrastructure. To get a list of Hosted Services, use this function:

sms.listHostedServices(function(err,data){
  if (err) console.log(err);
  console.log(data.body);
});

We can get more information about a particular Hosted Service:

sms.getDeploymentBySlot('HOSTED_SERVICE','production',function(err,data){
  if (err) console.log(err);
  console.log(data.body);  
});

It’s simple to create a new Hosted Service:

sms.createHostedService("NewHostedService", {Location:"North Europe"}, function(err,data){
  if (err) console.log(err);
  // success!
});

To deploy a new Virtual Machine is a bit more involved:

var vmrole = {RoleName: "NewRole",ConfigurationSets: []};

vmrole.ConfigurationSets[0] = {
  ConfigurationSetType: "WindowsProvisioningConfiguration",
  ComputerName: "MACHINE_NAME",
  AdminPassword: "PASSWORD",
  ResetPasswordOnFirstLogon: false };

vmrole.ConfigurationSets[1] = {
  ConfigurationSetType: "NetworkConfiguration",
  InputEndpoints: [
    {LocalPort: 80,Port: 80,Name: "HTTP",Protocol: "tcp"},
    {LocalPort: 3389,Port: 3389,Name: "RDP",Protocol: "tcp"}]}

vmrole.OSVirtualHardDisk = { SourceImageName: "NAME_OF_YOUR_IMAGE"};

sms.createDeployment("NewHostedService","NewDeployment",vmrole,{DeploymentSlot:"Production"}, function(err,data){
  if (err) console.log(err);
  // success! 	
});

I have managed to deploy VMs, start/stop them, mount/unmount disks and delete everything again. :¬)

Gotchas

It’s a bit ugly in places. For disks you have to specify the Lun (the logical number of the disk, between 0-15) when you mount it. This number is not reported to you when you query the machine, and there’s no way of unmounting the disk without specify the Lun, so it’s something you have to keep track of yourself.

Disks are also reported as being mounted on the deployment of a role, but you have request a mount/unmount on the instance. This is a bit confusing and inconsistent.

Further Reading

To explore more of the functionality, you could take a look at this documentation (the first part is currently wrong, but the api is well documented):

https://github.com/WindowsAzure/azure-content/blob/master/ITPro/Linux/HowTo/howto-servicemgt-api.md

Alternatively you could just read the source code of the module:

https://github.com/WindowsAzure/azure-sdk-for-node/blob/master/lib/services/serviceManagement/servicemanagementservice.js

Conclusions

One of the most important feature of any cloud is it’s programmability. Getting the SDK for this right is very important, and it’s surprising that this area isn’t mature. You can do just about everything through PowerShell and the cross-platform tools, but raw programmability is important too.

This SDK is not finished, it’s not documented, and it’s a bit ugly in places. However, it’s a great start.

It’s interesting to see Microsoft making this functionality available for Node.js before .NET languages. We know that the cross-platform tools are Node, so it’s obvious why, it’s just an interesting position to be in.

Ok, there is a sample in C# (the Microsoft.Samples.WindowsAzure.ServiceManagement namespace) but it’s undocumented, unsupported, and not straight forward to use. Richard Conway’s Fluent Management library is good, but not complete (I’m sure Richard would appreciate help!).

The only other viable option seems to be Python.