Updates from August, 2013 Toggle Comment Threads | Keyboard Shortcuts

  • Richard 4:04 pm on August 29, 2013 Permalink |  

    Hosting Go (or any CGI apps) on Windows Azure Websites 

    Run Process on Website

    It’s a little known fact, but you can spawn some server-side processes from your Windows Azure Website.

    For example, this Node server will execute any command you pass in on the end of the URL:

    var childProcess = require('child_process');
    var http = require('http');
    http.createServer(function(req,res){
        var cmd = req.url.replace("/","").replace(/%20/g, " ");
        childProcess.exec(cmd, function(err, stdout, stderr){
        if (err)
          res.end(JSON.stringify(err));
        else 
          res.end(stdout);
      });
    }).listen(process.env.port);

    …so running a command like this:

    http://xxx.azurewebsites.net/dir

    Will give you something like this:

     Volume in drive C has no label.
     Volume Serial Number is 7C6E-015C
    
     Directory of C:\DWASFiles\Sites\xxx\VirtualDirectory0\site\wwwroot
    
    08/29/2013  01:10 PM                   .
    08/29/2013  01:10 PM                   ..
    08/29/2013  12:46 PM                44 .gitignore
    08/29/2013  12:46 PM               915 iisnode.yml
    08/29/2013  12:46 PM               366 server.js
    08/29/2013  01:33 PM               640 web.config
                   4 File(s)      1,288,915 bytes
                   2 Dir(s)   1,069,420,544 bytes free

    In fact, you can bring your own application and run that. Just add the .exe, and any dependencies, and start the process as shown in the first example.

    This is exactly how CGI works.

    Create a CGI Host

    There is a node module which will host CGI applications using Node’s HTTP server:¬†https://github.com/TooTallNate/node-cgi

    So we can use Node as a CGI host for any application which compiles to Windows!

    To create a server, install the cgi module:

    $ npm install cgi

    And create a server.js file like this:

    var cgi = require('cgi');
    var http = require('http');
    var path = require('path');
    
    var script = path.resolve("./", 'NAME_OF_YOUR_APP.EXE');
    
    http.createServer( cgi(script) ).listen(process.env.port);

    Just replace NAME_OF_YOUR_APP.EXE with the name of your application, this executable should be included in your git repository (assuming you’re using git to push to Website!).

    Create a CGI Application

    To test this, I created a CGI application in Go (test.go):

    package main
    
    import (
      "net/http"
      "net/http/cgi"
    )
    
    func myHandler(rw http.ResponseWriter, req *http.Request) {
      rw.Header().Add("Content-Type", "text/plain")
      rw.Write([]byte("Hello World!"))
    }
    
    func main() {
      http.HandleFunc("/", myHandler)
      cgi.Serve(nil)
    }

    To compile, just use:

    > go build test.go

    I added the test.exe file to my repository, switched the ‘NAME_OF_YOUR_APP.EXE’ to ‘test.exe’, and pushed up the changes:

    Untitled

    It works! And shows that we can host CGI apps in Windows Azure Websites using languages outside the subset that have primary support on Azure.

    Ok, CGI is not the fastest piece of technology in the world, and web programming has moved on, but it could help people move legacy applications up to the Cloud.

     
  • Richard 12:47 pm on August 27, 2013 Permalink |  

    Running LevelDB in Windows Azure Websites 

    LevelDB is becoming an interesting space in the Node.js community. Some healthy innovation is showing how a modular approach to building your own database has some compelling benefits.

    Don’t know what LevelDB is? Find out more here.

    Windows Azure Websites offers a fast and convenient way to get Node.js applications running quickly. It’s backed by a persistent disk which is replicated for you, so why not make use of that free space by putting your database on there?

    Here are some steps to getting LevelDB running in a Node.js application on Windows Azure Websites.

    Prerequisites

    Before you start, you must be using Windows (the 8.1 preview is currently free to download), and have the following installed:

    Why 32 bit? Because node-gyp seems to target the processor architecture of the version of Node you have installed, and the shared (free) version of Windows Azure Websites is 32 bit only.

    Adding LevelDB

    Use NPM to add LevelDB to your app. The level modules includes levelup and leveldown:

    $ npm install level

    This will also compile the native leveldown module.

    If you have any problems at this stage, I recommend cloning the leveldown repository, running `node-gyp configure`, and opening the .sln file located in the build directory. You’ll need to configure the solution to build Release mode.

    Once you’ve installed level, you can start using the database in your code like this:

    // create a database 
    var db = require('level')('db');
    
    // put a value
    db.put('key1', 'value1');
    
    // get a value
    db.get('key1', function(err, data){
      console.log(data);
    });

    See the LevelUp documentation for more examples.

    Configuring Git

    (I’m assuming you’re using git to deploy this to Azure one way or another!)

    Adding the modules to git

    Because Leveldown is a native module, you can’t compile it on Azure. Therefore you must add the compiled binary to your git repo. That’s straight forward enough, but you’ll find that you have 60MB of dependencies, which is ¬†quite a bit to upload to the Azure Website repo. It’s probably worth removing some of the unnecessary files before you add anything to git.

    Trim down the unnecessary files

    You can remove everything from this folder, with the exception of the ‘leveldown.node’ file (which is the DLL we just compiled).

      node_modules\level\node_modules\leveldown\build\Release

    You can also do away with this folder:

      node_modules\level\node_modules\leveldown\deps

    That should bring you down to about 1.2MB.

    I found that large git pushes can take a very long time to Azure Websites. I think some throttling takes place after 4MB or so.

    Ignore the database files

    You should also add the ‘db’ folder (or whatever you choose to name your database) to your .gitignore file, as you don’t want updates to your application to overwrite your database.

    Running in Azure

    You should now be able to push your application to Windows Azure Websites, and your data will be persisted in LevelDB. However, there are a couple of rules you must stick to:

    1. You must keep the number of Node processes to exactly 1. You can scale up to a ‘shared’ or ‘standard’ model, but you can’t scale out. LevelDB cannot be hosted within multiple processes.
    2. You are limited to 1GB of data on the disk

    Conclusions

    LevelDB offers a fast and convenient way to host a database in an Azure Web Site.

    Websites run in a 32bit mode, I even found that the 63bit switch didn’t make any difference, but as long as we have the 32bit version of node installed we can compile the native module for this platform.

    It’s worth keeping your git repo small by pruning out some of the build artifacts. I did this by deleting the files, but you could do the same with a .gitignore file.

    You should keep your database files out of the repository using .gitignore.

    You should be careful about the number of Node processes you have running, given the constraints of LevelDB.

    It would be interesting to benchmark this against table storage to see if there any performance gains, perhaps another bog post?

     
  • Richard 2:33 pm on August 20, 2013 Permalink |  

    Fanning Out Async Await 

    I wanted to run an async function across a number of items, and I wanted to do this in parallel. This turned out to be quite a straightforward thing to do, once you know how.

    Let’s find out how, by trying out a few different approaches.

    First of all, here is a piece of async work that needs to be done 100 times. The total needs to be summed up:

    Untitled

    Failed Attempt #1

    Now, let’s take a really naive approach to running this:

    Untitled

    This takes 11000ms to run.

    Although we’re using the funky async/await keywords here, we’re ¬†running the code in series, rather than parallel. The ‘await’ keyword will block the execution of the loop until ‘DoWork’ returns.

    Failed Attempt #2

    Let’s try to improve on this. Where is the first place to go for parallel execution in .NET, the Task Parallel Library right?

    Untitled

    This takes 65 ms to run.

    Looking good, we’re doing a Parallel For loop now, and we’re using an async lambda. Cool stuff. It also takes just 65ms to run, wow, that’s shorter than the 100ms delay on the task! Unfortunately the TPL doesn’t work well with async/await. The For loop returns before the async tasks have completed. We never get the answer.

    Failed Attempt #3

    Ok, let’s have another go, and use the Task.WhenAll function, which returns a task which completes when all the tasks you pass it have completed:

    Untitled

    This takes 45ms to run.

    Even though we’re using the Task.WhenAll, this doesn’t work either. Task.Start doesn’t seem to be compatible with ‘WhenAll’ when async tasks are used. At least it’s faster than the last approach that didn’t work.

    Failed Attempt #4

    There is a Task.Run, which should be used in this scenario. Let’s modify the sample and try again:

    Untitled

    This takes 180ms to run.

    Instead of starting the task, we use Task.Run to create and start the task. This now seems to work, and we’re in business. So what’s wrong? Well it turns out that Task.Run dispatches the code to run on the thread pool rathern than the current thread. This isn’t what I want. There’s also a lot of code here, and it doesn’t look very elegant.

    Success

    Back to the drawing board, and this is what I came up with:

    Untitled

    This takes 160ms to run.

    When you see the answer it seems obvious, you don’t use await in the loop, you just add all the tasks (promises) to a list, and await the lot when you’ve finished creating them.

    Conclusions

    Async/await makes code look clean, but the async/parallel/task story in .NET is not a clear one. The parts don’t work together like you’d expect.

    You’re never quite sure where your task is going to run, on the thread pool or on the current thread?

    Thorough testing of async code is necessary. When awaits are short the await command will return immediately, which means that in real life the scenarios that didn’t work (i.e. attempts #2 & #3) will work correctly. This leads to some weird bugs and race conditions.

     
  • Richard 3:24 pm on August 9, 2013 Permalink |  

    Create an RDP file Download Link for an Azure VM 

    Just because it’s Friday, here’s a route for express (Node.js web framework) which will give you the RDP file for a VM in Azure:

    app.get("/rdp/:address", function(req, res){
      res.setHeader('Content-Type', 'application/x-rdp');
      res.setHeader('Content-Disposition', 'attachment; filename=' + req.params.address + '.rdp');
      res.end("full address:s:" + req.params.address + ":3389\r\nprompt for credentials:i:1");
    });

    To use, add the above lines to your express application and the link below in your web page:

    <a href="/rdp/xyz.cloudapp.net">Remote Desktop</a>

    (obviously replacing xyz.cloudapp.net with the name of your hosted service)

     

     
  • Richard 2:56 pm on August 9, 2013 Permalink |  

    Managing Azure IaaS from Node.js 

    A Service Management Service was recently added to the Node SDK for Windows Azure. This allows you to interact with the Azure management API, to create, deleted and manage Virtual Machines in Azure.

    Getting Set Up

    To get started you need to get your management certificates in order. This is probably most easily achieved by creating new ones.

    $ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
    $ openssl x509 -inform pem -in mycert.pem -outform der -out mycert.cer

    The mycert.cer file can be uploaded to the Management Portal. The other files we’ll use in our node app.

    Next, let’s install the Azure SDK:

    $ [sudo] npm install azure

    Creating the ServiceManagementService

    Now let’s write some JavaScript. To work with the Service Management API, we’ll need an instance of the ServiceManagementService:

    var azure = require('azure');
    var auth = {keyfile:'mycert.pem',certfile:'mycert.pem'};
    var sms = azure.createServiceManagementService('SUBSCRIPTION_ID', auth);

    Using the ServiceManagementService to call Azure

    Now we have an instance of the SMS, we can start querying the Azure infrastructure. To get a list of Hosted Services, use this function:

    sms.listHostedServices(function(err,data){
      if (err) console.log(err);
      console.log(data.body);
    });

    We can get more information about a particular Hosted Service:

    sms.getDeploymentBySlot('HOSTED_SERVICE','production',function(err,data){
      if (err) console.log(err);
      console.log(data.body);  
    });

    It’s simple to create a new Hosted Service:

    sms.createHostedService("NewHostedService", {Location:"North Europe"}, function(err,data){
      if (err) console.log(err);
      // success!
    });

    To deploy a new Virtual Machine is a bit more involved:

    var vmrole = {RoleName: "NewRole",ConfigurationSets: []};
    
    vmrole.ConfigurationSets[0] = {
      ConfigurationSetType: "WindowsProvisioningConfiguration",
      ComputerName: "MACHINE_NAME",
      AdminPassword: "PASSWORD",
      ResetPasswordOnFirstLogon: false };
    
    vmrole.ConfigurationSets[1] = {
      ConfigurationSetType: "NetworkConfiguration",
      InputEndpoints: [
        {LocalPort: 80,Port: 80,Name: "HTTP",Protocol: "tcp"},
        {LocalPort: 3389,Port: 3389,Name: "RDP",Protocol: "tcp"}]}
    
    vmrole.OSVirtualHardDisk = { SourceImageName: "NAME_OF_YOUR_IMAGE"};
    
    sms.createDeployment("NewHostedService","NewDeployment",vmrole,{DeploymentSlot:"Production"}, function(err,data){
      if (err) console.log(err);
      // success! 	
    });

    I have managed to deploy VMs, start/stop them, mount/unmount disks and delete everything again. :¬)

    Gotchas

    It’s a bit ugly in places. For disks you have to specify the Lun (the logical number of the disk, between 0-15) when you mount it. This number is not reported to you when you query the machine, and there’s no way of unmounting the disk without specify the Lun, so it’s something you have to keep track of yourself.

    Disks are also reported as being mounted on the deployment of a role, but you have request a mount/unmount on the instance. This is a bit confusing and inconsistent.

    Further Reading

    To explore more of the functionality, you could take a look at this documentation (the first part is currently wrong, but the api is well documented):

    https://github.com/WindowsAzure/azure-content/blob/master/ITPro/Linux/HowTo/howto-servicemgt-api.md

    Alternatively you could just read the source code of the module:

    https://github.com/WindowsAzure/azure-sdk-for-node/blob/master/lib/services/serviceManagement/servicemanagementservice.js

    Conclusions

    One of the most important feature of any cloud is it’s programmability. Getting the SDK for this right is very important, and it’s surprising that this area isn’t mature. You can do just about everything through PowerShell and the cross-platform tools, but raw programmability is important too.

    This SDK is not finished, it’s not documented, and it’s a bit ugly in places. However, it’s a great start.

    It’s interesting to see Microsoft making this functionality available for Node.js before .NET languages. We know that the cross-platform tools are Node, so it’s obvious why, it’s just an interesting position to be in.

    Ok, there is a sample in C# (the¬†Microsoft.Samples.WindowsAzure.ServiceManagement namespace) but it’s undocumented, unsupported, and not straight forward to use. Richard Conway’s Fluent Management library is good, but not complete (I’m sure Richard would appreciate help!).

    The only other viable option seems to be Python.

     
    • victorcl2 3:05 pm on February 3, 2015 Permalink | Log in to Reply

      Hi.
      I was wondering if you could manage to do the same with the php sdk ?? I been reading the docs and the source code but I can’t find anything related to virtual machines, I need to have a way to build a site where I can manage my virtual machines , is the nodejs sdk the only way to go besides the restuf api?

c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel