Adding Content to an Azure Package

When using CloudServices in Windows Azure, you frequently want to add extra files to your package, such as startup scripts you want to register, or additional content that’s required.

Traditionally this has always been done by adding the file to the project, and setting the ‘Copy to Output Directory’ property to ‘Copy if Newer’. The file will now be included in the package.

There is another way. You can right click on your role in your Cloud Project, and add content or folders directly. Items added here will automatically be included in the package, with no need to adjust the file properties.

The 1.8 (October 2012) SDK release allows you to add a startup task here. Previous versions seemed to raise an error.

The advantage of adding content here is it separates our the Azure concerns from your project. You can use a single codebase for deploying an application on-premises, or in the cloud, without cluttering your project with script files that aren’t needed on-prem. It also stops you from forgetting to set the ‘Copy to Output Directory’ property!

Advertisement

Automating Cloud Service Deployment in Azure

The build and deployment of Cloud Services can be automated, so your continuous integration server can push updates automatically to your (test) infrastructure.

First of all, build the solution, and use CSPack to package the deployment:

%WINDIR%\Microsoft.NET\Framework\v4.0.30319\MSBuild SOLUTION.sln /p:Configuration=Release /t:Publish

You will need to update the command with the name of your solution.

Then using PowerShell, upload to staging, wait for the instances to start, swap staging and production slots, and delete the staging environment.

$path = (Get-location).Path + "\Azure\bin\Release\app.publish"
$hostedService = "YOUR_HOSTED_SERVICE"

# deploy the package to staging slot
New-AzureDeployment -ServiceName $hostedService -Package "$path\ServiceDefinition.cspkg" -Configuration "$path\ServiceConfiguration.Cloud.cscfg" -Slot Staging

do {
	$ready=0
	$total=0

	# query the status of the running instances
	$list = (Get-AzureRole -ServiceName $hostedService -Slot Staging -InstanceDetails).InstanceStatus 

	# count the number of ready instances
	$list | foreach-object { IF ($_ -eq "ReadyRole") { $ready++ } }

	# count the number in total
	$list | foreach-object { $total++ } 

	"$ready out of $total instances are ready"

	# sleep for 10 seconds
	Start-Sleep -s 10
}
while ($ready -ne $total)

# swap staging and production
Move-AzureDeployment -ServiceName $hostedService

# remove the staging slot
Remove-AzureDeployment -ServiceName $hostedService -Slot Staging -Force

Simple!

Pub/Sub with Redis on Azure

After getting publish/subscribe working with ZeroMQ, I turned to Redis (mainly for a working Node.JS client). Redis is an in-memory key-value store, and includes a pub/sub feature.

Redis is easy to get working on Azure, however, I felt the need to make it even easier with a plugin.

Installing Redis on Azure

To create a new server with Redis installed, create a Cloud Project with a Worker Role, use APM to add the Redis Plugin.

> apm install Redis

Then add Redis as a plugin in your Service Definition file:

<WorkerRole name="WorkerRole1" vmsize="Small">
  <Imports>
    <Import moduleName="Diagnostics" />
    <Import moduleName="Redis" />
  </Imports>
</WorkerRole>

Done!

Setting up the Subscriber (in Node.js)

Install the Redis module using NPM:

> npm install redis

In your node code, you can subscribe to messages like this:

var redis = require("redis");
receiveclient = redis.createClient(6379, "WorkerRole1_IN_0");
receiveclient.on("message", function (channel, message) {
  // do something with the message
});
receiveclient.subscribe("pub");

And you can publish messages like this (using a separate client).

sendclient = redis.createClient(6379, "WorkerRole1_IN_0");
sendclient.publish("pub", message);

Discovery

If you’ve got Redis running on the same machine as the Node process, there is no need to specify the port and server name when creating a client. However, if this isn’t the case, Node needs to know the ip address of the instance hosting Redis. If your process is hosted by IISNode, then you can call the Azure API to discover the topology of your deployment, and find the correct IP address. This isn’t the case for me. I’m using  raw sockets and I can’t use IISNode.

So the question is, how do you discover the IP address of your Redis instance?

I created another plugin (HostsWriter) which continually updates the hosts file in Windows, with the IP address of each instance in the deployment.

This means you can just refer to instances by their instance name, and Windows will connect you to the right IP address.

To install, run this command with APM :

> apm install HostsWriter

And add it to the Service Definition file of the WorkerRole hosting your Node.js script.

<WorkerRole name="WorkerRole2" vmsize="Small">
  <Imports>
    <Import moduleName="Diagnostics" />
    <Import moduleName="HostsWriter" />
  </Imports>
</WorkerRole>

Conclusion

Redis is really easy to set up and use. It also works with Node, and module is 100% JavaScript, which is a big tick for me. Whether it’s as fast as ZeroMQ, I’m not sure, but it’s my current favourite!

Pub/Sub with ZeroMQ on Azure

I wanted to do real-time publish/subscribe, and although the Azure Service Bus supports this with topics and subscriptions, it seems to be geared towards guaranteed delivery rather than real-time. You need to make a request to receive a message, which didn’t quite fit the real-time nature of my application. Enter ZeroMQ.

ZeroMQ (or ØMQ) is a library for working with sockets. There are clients for a variety of languages, including C# – although I can’t get the node module to build :¬(. It’s a fairly cool piece of technology, and takes all the pain out of socket programming.

ZeroMQ needs to be installed on the machine, and some C++ runtime DLLs are also required. To make this super easy I created a plugin in the AzurePluginLibrary to do just this. To install, just run this command using APM.

> apm install ZeroMQ

Setting up the Publisher

Next I created a Cloud Project, and a WorkerRole to act as my publisher. To install the ZeroMQ references, you just need to run this nuget command in Visual Studio:

PM> Install-Package clrzmq -Pre

In the ServiceDefinition file for the Worker, add the ZeroMQ plugin, and two endpoints (the port numbers don’t matter):

<WorkerRole name="WorkerRole1" vmsize="Small">
  <Imports>
    <Import moduleName="Diagnostics" />
    <Import moduleName="ZeroMQ" />
  </Imports>
  <Endpoints>
    <InternalEndpoint name="Endpoint1" protocol="tcp" />
    <InternalEndpoint name="Endpoint2" protocol="tcp" />
  </Endpoints>
</WorkerRole>

Now you just need to write the C# code to receive a message from a client, and publish to all connected clients. I put this code in the ‘Run’ method of my RoleEntryPoint.

public override void Run()
{
  var context = ZmqContext.Create();

  // set up the socket which publishes messages to subscribers
  var endpoint1 = GetEndpoint("Endpoint1");
  var publisher = context.CreateSocket(SocketType.PUB);
  publisher.Bind(string.Format("tcp://*:{0}", endpoint1.IPEndpoint.Port));

  // set up the socket which receives messages from subscribers
  var endpoint2 = GetEndpoint("Endpoint2");
  var receiver = context.CreateSocket(SocketType.PULL);
  receiver.Bind(string.Format("tcp://*:{0}", endpoint2.IPEndpoint.Port));

  // publish every message you receive!
  while (true)
  {
    var message = receiver.Receive(Encoding.UTF8);
    publisher.Send(message, Encoding.UTF8);
  }
}

The ‘GetEndpoint’ function in this case just returns the required RoleInstanceEndpoint of the current instance.

private RoleInstanceEndpoint GetEndpoint(string name)
{
    return RoleEnvironment
        .CurrentRoleInstance
        .InstanceEndpoints
        .Where(x => x.Key == name)
        .Select(x => x.Value).FirstOrDefault();
}

Setting up the Subscriber

I then created a WebRole which acts as the subscriber. This also needs the ServiceDefinition file updating to include the ZeroMQ plugin. I added the code into the Global.asax, but for simplicity I have extracted it out into a single code block for this example.

var context = ZmqContext.Create();

// set up the socket which subscribed to messages from the publisher
var endpoint1 = GetEndpoint("Endpoint1");
var subscriber = this.Context.CreateSocket(SocketType.SUB);
subscriber.Connect(string.Format("tcp://{0}:{1}", endpoint1.IPEndpoint.Address, endpoint1.IPEndpoint.Port));
subscriber.Identity = Guid.NewGuid().ToByteArray();
subscriber.SubscribeAll();

// set up the socket which sends messages to the publisher
var endpoint2 = GetEndpoint("Endpoint2");
var sender = this.Context.CreateSocket(SocketType.PUSH);
sender.Connect(string.Format("tcp://{0}:{1}", endpoint2.IPEndpoint.Address, endpoint2.IPEndpoint.Port));

while (true)
{
    var message = subsriber.Receive(Encoding.UTF8);
    // do something with the messages you receive
}

To send a message, you just call:

sender.Send(message, Encoding.UTF8);

In this code snipper, the GetEndpoint function is slightly different:

private RoleInstanceEndpoint GetEndpoint(string name)
{
    return RoleEnvironment
        .Roles["WorkerRole1"]
        .Instances.First()
        .InstanceEndpoints
        .Where(x => x.Key == name)
        .Select(x => x.Value)
        .FirstOrDefault();
}

Conclusion

This sample shows a simple pub/sub scenario with ZeroMQ in Azure. Whenever the ‘Send’ function is called, the message will almost instantly pop out at each of the subscribers. Network disruption and reconnecting the socket is all handled for you. ZeroMQ is a powerful tool. This is just one way of configuring it, there are a number of different scenarios that it can support.