Recent Updates Toggle Comment Threads | Keyboard Shortcuts

  • Richard 10:14 am on September 5, 2014 Permalink | Log in to leave a Comment  

    Redirecting to HTTPS in Node.js on Azure Websites 

    If you’ve got a node.js website running in Azure Websites, you get SSL by default, or if you bring your own domain name, you can easily set up SSL with your own certificate.

    The default behaviour is that the website serves both HTTP and HTTPS requests, but what if you want to switch users automatically over to HTTPS (i.e. disable HTTP).

    The normal approach when using express.js is to use some middleware which detects the protocol and redirects accordingly. Like this:

    // this doesn't work in Azure Websites
    function requireHTTPS(req, res, next) {
        if (!req.secure) {
            return res.redirect('https://' + req.get('host') + req.url);
        }
        next();
    }
    
    app.use(requireHTTPS);
    

    req.secure is just a shorthand for checking the protocol of the incoming request. However, in Azure websites your server is fronted by a web server running ARR, which does the SSL termination. This means the request you see will always be HTTP. The above code samples creates a redirect loop as the middleware continues to redirect to HTTPS, as it keeps seeing HTTP.

    Fortunately ARR adds some additional headers to the request. These are:

    cookie: ARRAffinity=71111a2979a8796b0cc6860d7f98cb7519aaea861cc96f15896edbb25a259064
    host: XYZ.azurewebsites.net
    max-forwards: 10
    x-liveupgrade: 1
    x-arr-log-id: 7cffa211-21f6-42c2-8b7d-e9eacac8adc8
    disguised-host: XYZ.azurewebsites.net
    x-site-deployment-id: XYZ
    x-original-url: /url
    x-forwarded-for: 212.69.41.154:4219
    x-arr-ssl: 2048|128|DC=com, DC=microsoft, DC=corp, DC=redmond, CN=MSIT Machine Auth CA 2|C=US, S=WA, L=Redmond, O=Microsoft, OU=OrganizationName, CN=*.azurewebsites.net 
    

    The x-arr-ssl header is only added for HTTPS requests, so we can use this header to determine whether it’s a HTTP or HTTPS request.

    function requireHTTPS(req, res, next) {
        if (!req.get('x-arr-ssl')) {
            return res.redirect('https://' + req.get('host') + req.url);
        }
        next();
    }
    
    app.use(requireHTTPS);
    

    You could go one better, and only do the upgrade if you’re running in Azure, which means you can run your site in plain HTTP in your local environment:

    function requireHTTPS(req, res, next) {
        if (req.get('x-site-deployment-id') && !req.get('x-arr-ssl')) {
            return res.redirect('https://' + req.get('host') + req.url);
        }
        next();
    }
    
    app.use(requireHTTPS);
    
     
  • Richard 1:39 pm on September 2, 2014 Permalink | Log in to leave a Comment  

    Using the Azure Search service from JavaScript 

    surface-your-data

    The new Azure Search service is a nice ‘search as a service’ offering from Microsoft. Just add your documents, and then run some queries. It’s exposed as a REST API which talks JSON :¬D

    It has a free tier, limited to 3 indexes and 10,000 documents, but you can of course start paying the money and index more stuff.

    I just wrote a JavaScript client (I couldn’t find a Microsoft one) to connect to the API, let’s explore how we can use it.

    Note this article assumes you are familiar with writing JavaScript for node.js and the browser.

    Creating the Search Service

    First, open up the new Azure Portal and go to NEW -> Search and enter some details in.

    You’ll need to switch to the free tier, otherwise you’ll start clocking up a bill.

    We’ll start off using Node.js, as only a few of the features (quite correctly) can be used from the browser.

    Once it’s created go to the properties and keys sections of the search service blade, and make a note of your url and an admin key.

    Installation

    First install the package:

    $ npm install azure-search
    

    Creating the Client

    Now let’s write write some JavaScript in node to create an index.

    var AzureSearch = require('azure-search');
    var client = AzureSearch({
        url: "https://MY_SEARCH_SERVICE_NAME.search.windows.net",
        key:"MY_SEARCH_SERVICE_KEY"
    });
    

    Creating the Index

    Now we have a client, we can create an index in the Search Service. To do this, we need to create a schema, which will tell the service what kind of data we want to store and search. There’s more information about how to create the schema in the Microsoft documentation, but for a simple example, I’ll have some text, and a key that I’ll use to refer to the text.

    var schema = { 
      name: 'myindex',
      fields:
       [ { name: 'id',
           type: 'Edm.String',
           searchable: false,
           filterable: true,
           retrievable: true,
           sortable: true,
           facetable: true,
           suggestions: false,
           key: true },
         { name: 'description',
           type: 'Edm.String',
           searchable: true,
           filterable: false,
           retrievable: true,
           sortable: false,
           facetable: false,
           suggestions: true,
           key: false } ],
      scoringProfiles: [],
      defaultScoringProfile: null,
      corsOptions:  { allowedOrigins: ["*"]} };
    
    client.createIndex(schema, function(err, schema){
      if (err) console.log(err);
      // schema created
    });
    

    Note that at the bottom of the file there’s a corsOptions setting which sets allowedOrigins to ["*"]. We’ll be using this later to access the index from the browser.

    Populating the Index

    Now we’re ready to start adding documents to the index. In the schema we specified id and description fields. So we just need to supply an object with these fields.

    var document = {
      id: "document1",
      description: "this is a document with a description"
    };
    client.addDocuments('myindex', [document], function(err, confirmation){
      if (err) console.log(err);
      // document added
    });
    

    In fact we can send through a batch of document through at once, by adding more items to the array.

    Querying the Index

    We can query the index from node, but the Search Service supports CORS, which allows us to query directly from the browser without having to stand up any of our own server-side infrastructure. This is where the CORS settings came in when we created the schema.

    One thing to be careful of; don’t put your Search Service admin key in a public web page. Instead, go back to the portal and use a query key (see the manage keys button when you’re looking at the keys).

    Untitled

    Now we can create a web page where we can call the index, and pass in some search queries. To do this we need to add a reference to azure-index.min.js (or use browserify, and just require ‘azure-index’).

    <html>
        <head>
            <script src="azure-search.min.js"></script>
        </head>
        <body>
            <script>
    
            var client = AzureSearch({
              url: "https://MY_SEARCH_SERVICE_NAME.search.windows.net",
              key:"MY_QUERY_KEY"
            });
    
            client.search('myindex', {search:'document'}, function(err, results){
              // results is an array of matching documents
            });
    
            </script>
        </body>
    </html>
    

    Note that from the browser, only search, suggest, lookup and count will work.

    For more information, please consult the Microsoft documentation and the documentation for the module.

    Conclusion

    The search service looks quite powerful. We’ve only scratched the surface of the options here. I’m keen to combine the search service with a trace listener, so you can index your application’s logging.

    It’s great to see Microsoft move away from the awkward API conventions used for the storage system, which included complicated header signing and XML. This JSON approach with a simple API key as a header is nice and simple.

    It’s also great to see CORS support our of the box, which makes it easy to consume this service directly from the browser.

    Personally I think API version number looks out of place on the URL, and would be better as a header, but maybe that’s just me.

    I also would prefer not to have to specify my schema. I’d like to just throw JSON objects at the service, and then be able to query on any of the fields, but I guess that’s DocumentDB!

     
  • Richard 10:37 am on August 29, 2014 Permalink | Log in to leave a Comment  

    Overrides in Orleans Configuration 

    If you look at the standard OrleansConfiguration.xml configuration file you get with Orleans, it looks something like this:

    <?xml version="1.0" encoding="utf-8"?>
    <OrleansConfiguration xmlns="urn:orleans">
      <Globals>
        <StorageProviders>
          <Provider Type="Orleans.Storage.MemoryStorage" Name="MemoryStore" />
        </StorageProviders>
        <SeedNode Address="localhost" Port="11111" />
      </Globals>
      <Defaults>
        <Networking Address="localhost" Port="11111" />
        <ProxyingGateway Address="localhost" Port="30000" />
        <Tracing DefaultTraceLevel="Info" TraceToConsole="true" TraceToFile="{0}-{1}.log">
          <TraceLevelOverride LogPrefix="Application" TraceLevel="Info" />
        </Tracing>
        <Statistics MetricsTableWriteInterval="30s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true"/>
      </Defaults>
      <Override Node="Primary">
        <Networking Address="localhost" Port="11111" />
        <ProxyingGateway Address="localhost" Port="30000" />
      </Override>
    </OrleansConfiguration>
    

    This configuration starts a silo listening on port 30000 (ProxyingGateway), and other silos can talk to it on port 11111 (Networking). It also identifies a SeedNode, which is the primary silo in the cluster. The seed node address and port match those specified for this silo, which makes this the primary.

    You’ll also notice the Override section at the bottom, which specifies the same ProxyingGateway and Networking ports again.

    In this case, the Override section doesn’t do much, but suppose we want to start a secondary silo? We can specify the ports for a secondary node to use here, like this (in this case a silo on the same machine, so on different port numbers):

    <?xml version="1.0" encoding="utf-8"?>
    <OrleansConfiguration xmlns="urn:orleans">
      <Globals>
      ... 
      </Globals>
      <Defaults>
      ...
      </Defaults>
      <Override Node="Primary">
        <Networking Address="localhost" Port="11111" />
        <ProxyingGateway Address="localhost" Port="30000" />
      </Override>
      <Override Node="Secondary">
        <Networking Address="localhost" Port="11112" />
        <ProxyingGateway Address="localhost" Port="30002" />
      </Override>
    </OrleansConfiguration>
    

    This allows us to use the same configuration file for both a primary and a secondary node, as the other settings will be the same (they must all specify the same primary (seed node)).

    To make use of the override, when you start the silo, just specify the name as the first argument.

    $ OrleansHost.exe Primary
    $ OrleansHost.exe Secondary
    

    You can also specify the configuration file you want to use, in case you have several (the default is OrleansConfiguration.xml).

    $ OrleansHost.exe Primary MyConfiguration.xml
    

    Interestingly, when starting a number of silos it only seems necessary to tell a client about one of silos (a primary or secondary) it seems to figure out the cluster members automatically.

     
  • Richard 10:22 am on August 13, 2014 Permalink | Log in to leave a Comment  

    Public IP Addresses for Amazon 

    I previously blogged about the public IP address ranges for Azure.

    The public IP addresses ranges for Amazon has recently been updated. Amazon has over 8 million addresses, which is 8 times the number Azure has.

    Interestingly, these are not so evenly distributed:

    Untitled

    Raw data:

    DC Total public IP addresses
    US East (Northern Virginia) 3694592
    US West (Oregon) 1507328
    EU (Ireland) 1332224
    Asia Pacific (Tokyo) 739328
    US West (Northern California) 655360
    Asia Pacific (Sydney) 327680
    Asia Pacific (Singapore) 313344
    South America (Sao Paulo) 245760
    China (Beijing) 65536
    GovCloud 16384
    TOTAL 8897536

    I have also updated my tool to look up which DC a website is hosted in, to include the Amazon IP ranges.

     
  • Richard 11:12 am on August 7, 2014 Permalink | Log in to leave a Comment  

    Enabling SSL for Self Hosted Nancy 

    One of the things I like about Nancy is the ease of creating self hosted HTTP services.

    Here’s how to enable HTTPS for your service.

    First of all, create your SSL certificate:

    $ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem
    $ openssl pkcs12 -export -in cert.pem -inkey key.pem -out mycert.pfx
    

    Then go to ‘Manage Computer Certificates’ in the start menu (I’m using Windows 8.1).

    Right click on ‘Personal’ and import the freshly created mycert.pfx.

    Untitled

    Now go to your newly installed certificate, and get the thumbprint:

    Untitled2

    Now you need to register the URL with Windows:

    $ netsh http add urlacl url=https://+:4443/ user=YOUR_USERNAME
    $ netsh http add sslcert ipport=0.0.0.0:4443 certhash=‎YOUR_THUMBPRINT_WITHOUT_SPACES appid={06aabebd-3a91-4b80-8a15-adfd3c8a0b14} 
    

    You need to substitute your username, and the certificate thumbprint you copied (without the spaces)

    The appid is just a random guid, feel free to create your own.

    Now start nancy on this Uri https://localhost:4443/.

    using (var host = new NancyHost(new Uri("https://localhost:4443/")))
    {
        host.Start();
        Console.ReadKey();
    }
    
     
  • Richard 9:00 am on August 4, 2014 Permalink | Log in to leave a Comment  

    Public IP Addresses for Microsoft Azure 

    Microsoft publish a list of public IP addresses for the compute services hosted in their Azure Datacenters. When they say ‘compute’ they also include SQL Database, and the storage accounts seem to be included too.

    I thought it would be interesting to write a script to count the total number of IP addresses available for each DC.

    Here’s a pretty graph.

    Untitled

    Whilst the number of public IPs is no way of working out the number of machines provisioned in each DC, it does at least give you an idea of the magnitude of each of the regions.

    Some interesting things to note:

    1. The EU data centers are similar sizes, Dublin being slightly bigger.
    2. The 5 DCs in Asia, Brazil and Japan added together are smaller than US West.
    3. The total number of IPs is just over 1 million.
    4. There seems to be a step change between the largest 6, the next 4, and then the 3 small (and new) DCs in Japan and Brazil.
    5. US West is 28 times larger than Brazil South.
    6. Almost 60% of the IPs are in the USA.

    Here’s my processed data in table:

    DC Total public IP addresses
    uswest 179504
    useast 155808
    europenorth 150400
    europewest 144320
    usnorth 118112
    ussouth 109120
    asiaeast 62384
    uscentral 62336
    asiasoutheast 57440
    useast2 46832
    japanwest 18912
    japaneast 10976
    brazilsouth 6288
    TOTAL 1122432

    I think it will be really interesting to keep an eye on the IP ranges, and see how they change over time (or is it just me?).

    I also published a tool to look up if an IP (or domain name) is hosted in Azure, it also tells you which DC it’s in.

     
  • Richard 10:40 am on July 30, 2014 Permalink | Log in to leave a Comment  

    Improving Orleans Silo Startup Times 

    The startup times for Orleans Silos seems a bit variable. I’ve tried a few things out, and got the startup time down when using the Dev/Test Silo. I don’t know if all of these make a difference, but they might be worth trying if you’re as impatient as me.

    • Remove any Storage Providers you’re not using from the configuration.
    • Turn off logging (unless things aren’t working!) by setting TraceToConsole="false" and TraceToFile="false" in the Tracing element.
    • If you’re not putting any client code in the Dev/Test Silo (perhaps you’re connecting another application such as an ASP.NET) then remove the Orleans.OrleansClient.Initialize("DevTestClientConfiguration.xml"); line from Main.cs.
    • Remove any Grain Collection classes you’re not using from the Orleans SDK\LocalSilo\Applications directory.
    • Delete the Orleans.FSharp.dll from SDK\LocalSilo.

    I’ve got startup time down to under 6 seconds.

    My DevTestServerConfiguration.xml file looks like this:

    <?xml version="1.0" encoding="utf-8"?>
    <OrleansConfiguration xmlns="urn:orleans">
      <Globals>
        <StorageProviders>
          <Provider Type="Orleans.Storage.MemoryStorage" Name="AzureStore" />
        </StorageProviders>
        <SeedNode Address="localhost" Port="11111" />
      </Globals>
      <Defaults>
        <Networking Address="localhost" Port="11111" />
        <ProxyingGateway Address="localhost" Port="30000" />
        <Tracing DefaultTraceLevel="Error" TraceToConsole="false" TraceToFile="false"/>
        <Statistics WriteLogStatisticsToTable="false"/>
      </Defaults>
      <Override Node="Primary">
        <Networking Address="localhost" Port="11111" />
        <ProxyingGateway Address="localhost" Port="30000" />
      </Override>
    </OrleansConfiguration>
    

    My DevTestClientConfiguration.xml file looks like this:

    <?xml version="1.0" encoding="utf-8" ?>
    <ClientConfiguration xmlns="urn:orleans">
      <GatewayProvider ProviderType="Config"/>
      <Gateway Address="localhost" Port="30000"/>
      <Tracing DefaultTraceLevel="Info" TraceToConsole="false" TraceToFile="false"/>
      <Statistics WriteLogStatisticsToTable="false"/>
    </ClientConfiguration>
    
     
  • Richard 2:20 pm on July 23, 2014 Permalink | Log in to leave a Comment  

    Using Observers for Pub Sub in Project Orleans 

    Orleans has a cool feature for pub-sub messaging. This allows you to register for grain originated events without you having to poll a grain. Instead you can asked to be called back when the interesting thing happens.

    To do this you create an ‘observer’ object. You then create a reference to this object, and pass it to the grain. The grain maintains a list of all observers. The grain can then send them all a message with a single call.

    To create an observer, create an interface which inherits from IGrainObserver. You should put this in your Grain Interfaces project.

    public interface IFooGrainObserver : IGrainObserver
    {
        void Foo(string message);
    }
    

    Note that the method signature for Foo returns void. This is a requirement for subscribers.

    Next create an implementation of this class. It doesn’t really matter which project you put this in. It’s probably best placed in the project where you want the code to actually run, so if you’re sending a message back to the client, put it alongside your clide code.

    class FooObserver : IFooGrainObserver
    {
        public void Foo(string message)
        {
            Console.WriteLine(message);
        }
    }
    

    We’ll need a grain which is going to receive the subscriptions, and perform the publication.

    This is the grain interface which has a method for registering a subscriber (Subscribe), and another to call when you wish to publish a message (Publish):

    public interface IPubSubGrain : IGrain
    {
        Task Subscribe(IFooGrainObserver observer);
        Task Publish(string message);
    }
    

    And this is the implementation:

    public class PubSubGrain : GrainBase, IPubSubGrain
    {
    
        ObserverSubscriptionManager<IFooGrainObserver> subscribers = new ObserverSubscriptionManager<IFooGrainObserver>();
    
        public Task Subscribe(IFooGrainObserver observer)
        {
            subscribers.Subscribe(observer);
            return TaskDone.Done;
        }
    
        public Task Publish(string message)
        {
            subscribers.Notify(x => x.Foo(message));
            return TaskDone.Done;
        }
    }
    

    Note that Orleans provides a ObserverGrainManager which helps you manage the subscriptions, and send notifications.

    Now to actually make this work.

    When the Grain Interfaces project compiles a factory is created for our subscriber (FooGrainObserverFactory) – just like the factories are created for the grains.

    To use the factory, we pass in an instance of our IFooGrainObserver interface (which will be a FooObserver). This will give us back an object we can then pass to the Subscribe method of our grain.

    This is the subscribe process complete. Now, just call publish.

    Your client code (perhaps this is in your Dev/Test Silo) will look something like this:

    var grain = PubSubGrainFactory.GetGrain(0);
    
    // register a subscriber.
    var observerRef = await FooGrainObserverFactory.CreateObjectReference(new FooObserver());
    await grain.Subscribe(observerRef);
    
    // send a publish
    await grain.Publish("Hello World");
    
    // Hello World is printed on the console by the instance of FooObserver
    

    Simple!

     
  • Richard 8:44 am on June 25, 2014 Permalink | Log in to leave a Comment  

    Orleans vs a Distributed Cache 

    I’m often asked what the difference is between Orleans and a distributed cache. It’s a good observation because on the face of it they have similar aims, to hold objects in memory for the right period of time to provide a performance benefit.

    However, there are a number of differences.

    1. The grains in Orleans are programmable, whereas caches normally just store data.
    2. With a distributed cache you keep objects in memory, but there’s a good chance that the data doesn’t reside on the same machine that’s serving the request. This means that machine processing the request has to retrieve the data from another node. Orleans will send the request to the machine which holds the data, which in theory should be more efficient.
    3. Orleans queues message to each grain and processes them one by one with a single threaded programming model. This stops concurrency problems and race conditions. With caches you have to build a locking system yourself, as described here for memcached.
    4. A grain can register a timer so it can call itself without requiring an external request. Caches generally just respond to external requests.

    People are certainly using Orleans as a cache, but I think it’s capable of much more.

     
    • Roger Alsing 11:02 am on June 25, 2014 Permalink | Log in to Reply

      On #3) my impression is that during a net split, Orleans can spawn the same virtual actor on each side of the network partition, and thus, break the fundamental single threaded behavior of actors.
      I Think Caitie McCaffrey taked about this during a Jepsen webcast.

      If this is true, you can still get race conditions and corrupted data

    • Graham Hay 3:34 pm on June 30, 2014 Permalink | Log in to Reply

      “stops concurrency problems and race conditions” is a bit optimistic :) It might remove certain categories of problems, if you’re lucky, but any interactions between grains will still be subject to the usual problems with distributed systems.

  • Richard 4:10 pm on June 19, 2014 Permalink | Log in to leave a Comment  

    Codename Orleans – Using Generics with Grains 

    In Orleans you can use generics with grain interfaces:

    public interface IGenericGrain<T> : Orleans.IGrain
    {
        Task SetValue(T value);
    
        Task GetValue();
    }
    

    When you talk to this grain from a client application, you can specify the value of ‘T’ on the factory class:

    var grain1 = GenericGrainFactory<int>.GetGrain(1);
    await grain1.SetValue(123);
    await grain1.GetValue();
    
    var grain2 = GenericGrainFactory<string>.GetGrain(2);
    await grain2.SetValue("foo");
    await grain2.GetValue();
    

    You can even extend ‘T’ to the persistence layer:

    public interface IMyGrainState<T> : IGrainState
    {
        T Value { get; set; }
    }
    
    [StorageProvider(ProviderName = "AzureStore")]
    public class GenericGrain<T> : Orleans.GrainBase<IMyGrainState<T>>, IDbGrain<T>
    {
        public Task SetValue(T value)
        {
            this.State.Value = value;
            return this.State.WriteStateAsync();
        }
    
        public Task GetValue()
        {
            return Task.FromResult(this.State.Value);
        }
    }
    

    This seems to work well for value types such as string, int, double etc…

    One word of caution; make sure that you always use the same value for ‘T’ when dealing with a particular grain ID, otherwise you’ll get cast exceptions.

     
c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel
Follow

Get every new post delivered to your Inbox.

Join 601 other followers