So far the only significant operation that we enabled our RestAgent to perform is NavigateTo().  However, for a RestAgent on a mission, going places is only half the story, the other major purpose of the agent is to gather content.

I was planning to show some real examples of missions using Stack Overflow’s API, but I ended up spending so much time explaining why it currently would not work, that my points were lost.  So forgive me as I continue with a somewhat more hypothetical example.  We are going to use an instance of our RestAgent to do a mashup of Twitter users and Stackoverflow users.

var agentFielding = new RestAgent(new HttpClient()); 

agentFielding.RegisterMediaType(“application/twitterdoc+xml”,
                                    new TwitterMediaHandler());
agentFielding.RegisterMediaType(“application/stackoverflowdoc+xml”,
                                    new StackOverflowMediaHandler());

In the following code, Agent Fielding retrieves my Twitter followers and attempts to find matching accounts on Stack Overflow.  In order to interpret the representations that will be retrieved from these two sites, we are going to pretend that these sites actually return non-generic media types and we register handlers for these media types that will transform the wire representation into strong types.

The first step is to navigate Twitter and retrieve the user profiles of my followers:

agentFielding.NavigateTo(new Uri(“http://api.twitter.com/”));
var userSearch = agentFielding.CurrentLinks(“UserSearch”);
userSearch.SetParameter("darrel_miller”);
agentFielding.NavigateTo(userSearch);

var followersLink = agentFielding.CurrentLinks[“Followers”);
agentFielding.NavigateTo(followersLink);

var twitterUserProfiles = new List<TwitterUserProfile>();
var followerLinks = agentFielding.GetCurrentLinks(l=> l.Relation.Name == ”Follower”];
foreach(Link followerLink in followerLinks) {

  var twitterProfile = agentFielding.GetContent(followerLink).ReadAsTwitterUserProfile();
  twitterUserProfiles.Add(twitterProfile);

}

Now we have a list of TwitterUserProfile objects from which we can retrieve the name and search on Stack Overflow.

agentFielding.NavigateTo(new Uri(“http://api.stackoverflow.com/1.0/”)); 

agentFielding.NavigateTo(agent.CurrentLinks[“Users”]);
var searchLink = agent.CurrentLinks[“Search”];
var foundProfiles = new List<StackOverflowUserProfile>();
foreach(var profile in twitterUserProfiles) {   searchLink.SetParameter(profile.UserName);
  agentFielding.NavigateTo(searchLink); 

  var matchingUserLinks = agentFielding.GetCurrentLinks(l=> l.Relation.Name == ”User”])
  foreach(Link userLink in matchingUserLinks) {
       var content = agentFielding.GetContent(userLink);
       foundProfiles.Add(content.ReadAsStackOverflowUserProfile();
  }
}

Don’t get too hung up on the precise details of the above, there is a bit of smoke and mirrors going on.  I’m really just trying to convey the concept that the same agent class can be used to navigate multiple different services.  This brings the concept of the uniform interface a little further into the client code. 

My hope is that in the future that REST api producers will supply media type handlers for their custom media types and we can use a standardized agent like interface for navigating any REST api.  There will be no need the API producers to create complete client side facades.

There is still plenty of coupling in the above example, when it comes to handling specific representations and knowing about the existence of specific links, there is plenty of service specific code.  This is especially true of these type of scripted or (machine to machine) interactions.  Machines are really dumb, so they need to understand a lot of specifics of the services that they are interacting with.  In the next article I’m going to start digging into how you can use an agent to react to a human driven application, and that will  further reduce the coupling. 

var agent = new RestAgent(new HttpClient());
agent.NavigateTo(new Uri(“http://api.stackoverflow.com/”)); 
var questionsLink = agent.CurrentLinks[“Questions”]
agent.NavigateTo(questionsLink);
var questions = agent.CurrentContent.ReadAsJson(); 

Before I talk about the magic that is going on under the covers here, let me take a little jab at my friend StackOverflow.  The above code would not work, because when the team created the api, they chose not to put a root representation on the API. Making the request to the root of the api actually redirects to an html page with API documentation.  I’ve tried scraping the urls out of that document, but it is proving to be painful.

The Link Class

The CurrentLinks property makes a dictionary of links available based on the links contained in the current content and potentially links that were included in the header.  You will notice that the second call to NavigateTo passes a “Link”, not a URL.  So what’s the difference?  Here is a basic implementation of the Link class:

public class Link {

  public string Name{ get; set;}
  public LinkRelation Relation{ get; set;}
  public Uri Url { get; set;}
  public HttpMethod Method { get; set;}
  public HttpContent Content {get; set;}

  public Dictionary<Parameter> QueryParameters {get; set;}
  public Dictionary<Parameter> PathParameters {get; set;}

}

The Link class holds plenty of additional information that is needed to allow the NavigateTo method to transfer to a new state.  The Link objects are created from information contained in retrieved hypermedia content.  If the HttpMethod is a PUT or a POST then the Content property can be used to hold the entity that will be sent to the origin server. 

The LinkRelation property is critical concept.  The Link Relation is one of the two points of coupling between the client and the server.  The other being the media-type.  From an architectural point of view the Link Relation describes the relationship between two resources.  For example, if you have a representation A that contains a link to resource B with the relation “edit”, this can be interpreted as B allows you to “edit” A.  IANA keeps a registry of well known link relations here http://www.iana.org/assignments/link-relations/link-relations.txt.  The WHATWG group are also maintaining a list here http://wiki.whatwg.org/wiki/RelExtensions.

From an implementation perspective, the Link Relation can be used to tell the client mechanical information about how to perform the HTTP request.  For example, what HTTP method to use, what content needs to be posted back to the server, if any. It also can be used to tell the client what query string parameters are valid.  Arguably you can introduce any kind of arbitrary significance to a link relation as long as you are prepared to put up with the client/server coupling.

I have been experimenting with implementing LinkRelation as an actual class instead of just an identifer.  Many different link relations have similar sets of behaviour and by creating a base class with some standard attributes it is possible to represent specific link relations as derived classes and still have plenty of generic mechanisms that process links without scattering the code base with fragile switch statements.  Hopefully as we go into more detail I will show some examples that take advantage of this LinkRelation base class.

So far I have only used very simple URI templates and so my needs have been satisfied with just a collection of query parameter values and path parameter values.  The current draft of the URI template specification introduces many more variants that I could be supported by the Link class.

The Name property is the black sheep of the Link class.  I’m not convinced that we really need this property.  However, I include it to support the scenario where there are multiple links within a hypermedia document that has the same LinkRelation.  How a client is supposed to differentiate between those different links without introducing out of band coupling is not always clear to me.  If the current representation is to be rendered to the user as a set of links then the user should be capable of deciding which they wish to follow.  However, in a scripted (ie. machine 2 machine) scenario, it becomes highly suspect if the script starts referring to link names.  Depending on how precise the media type, is, the link names could be pre-specified in the media-type.  Or code-download could be used to activate links with particular names.  The final solution that I have seen is to drop the name property completely and use very specific link relation values.  I’m not a big fan of this approach because I think it leads to the same explosion problems as using extremely specific media-types.  Highly specific media-types and link relations destroy the possibility of serendipitous reuse.

Hypermedia Content

So now, you have an idea of how I handle links, let’s look at how those links get created in the first place.  As I mentioned before, the CurrentLinks dictionary is populated from the CurrentContent.  The question of course is how can the generic RestAgent know how to pull links out of content that could be any media type.  The solution is a plug in model that allows handlers to be registered in a media type handler registry. 

Behind the scenes I use MEF to load up the media type handlers, but I stuck an interface in front, mainly because I’m not very experienced with MEF and I wasn’t exactly sure how to setup unit tests using MEF.  With the interface I can easily create a fake registry.

public interface IHandlerRegistry {
        IMediaTypeHandler GetHandler(ContentType contentType);
    }

Media type handlers I will cover in more detail at a later date.  All we care about for this series of articles, is that they can be used to do two things:  1) they can convert the stream of bytes coming from the http response into a class that implements IHypermediaContent and 2) they can create and run a controller with RunController(HttpResponseMessage response) method that will “do” whatever that media type is supposed to do.  I realize that sounds very vague, but vague is good when we don’t want to introduce coupling .

public interface IMediaTypeHandler {
    IHypermediaContent GetContent(HttpContent content);
    IMediaTypeController RunController(HttpResponseMessage message);
}

For this article I’m going to ignore RunController and we will get to human driven clients in an upcoming article.  That leaves us with GetContent that returns an IHypermediaContent interface.  That interface looks like this:

public interface IHypermediaContent {
       IDictionary<string, Link> Links { get; }
   } 

Not exactly complex to implement!  Consider what it would take to write a handler that processes, HTML, XHTML, Atom Feeds, HAL, or even one that attempts to scrape links out of Json or generic XML based on some link convention. 

Magical Links

Pulling this all back together, when you navigate the agent to a particular location on the web, the agent will identify the media type of the returned response, determine a handler for that media type, allow the handler to process the bytes and extra the links and store them in a dictionary available via the CurrentLinks property of the agent.

Here is a completely hypothetical use of the RestAgent to crawl safe links on a site:

var agent = new RestAgent(new HttpClient(), new Uri(“http://mythical.com/hypermediaService”));
ExploreLinks(agent);

public void ExploreLinks(RestAgent agent) {
var links = CopyLinks(agent.CurrentLinks);
foreach(Link link in links) {
    if (Link.Method == HttpMethod.GET && Link.LinkRelation.RequiresParameters == false) {
        agent.NavigateTo(link);
        Explore(agent);
    }
}
}

To keep things simple, I only traverse links whose LinkRelation specifies a GET is the appropriate verb and only links that do not require parameters. 

I realize that this is still not a very useful example, but enough of the pieces are in place so that in the next article I will talk more about how to get the agent to actually do something useful.

I write distributed business applications for a living and to date, it appears that I end up supporting the code I write for at least 10 to 15 years.  When ever you deploy and update distributed systems there is a nasty side effect of having different components to update and trying to deploy them simultaneously is rife with pitfalls.  Deploying an update to a server compared with updating software on 50 client machines is just a completely different set of problems and keeping them synchronized can be challenging. 

I realize that many people would argue that using a web browser as a client platform would solve my distribution problems.  It would solve the distribution problem, but it would also introduce a whole host of other problems for the types of applications I write.  Browser based apps are awesome for many things, some things they are not so awesome for.

The REST constraints allow you to manage the coupling between the distributed components and allow them to evolve more independently. This allows the system as a whole to evolve incrementally over long periods of time.  I need this kind of robustness in my line of work.

In the three years that I have been building RESTful systems I have seen lots written about how to design and implement RESTful APIs but very little has been written about how REST client applications need to be written.  The few people I have seen talk about it are Stu Charlton and Guilherme Silveira.

This is hopefully the first in a series of blog posts where I cover what I am learning about writing client applications that are part of a RESTful distributed system.

REST clients are a different breed

One of the core concepts when designing a REST client is the notion of Hypermedia as the Engine of Application State.  I’m not going to try and explain what I think it means, as many people far more eloquent than I, have tried and yet its meaning is still much debated.  Instead, I would prefer to show some code that I believe embodies the concept.

My REST client applications are built around a class called RestAgent.  I like to imagine my instance of the RestAgent class as little guy travelling across the web making requests and fetching data on my behalf [1].  This agent is responsible for holding all of my application state and for transferring state to and from origin servers.

In its most simple incarnation the class looks something like:

public class RestAgent(HttpClient client) {

  private Uri _CurrentLocation;
  private HttpContent_CurrentContent;

  public void NavigateTo(Uri url) {

    _CurrentContent = client.Get(url).Content;
    _CurrentLocation = url;
  }

}

The RestAgent itself does not know any of the nitty gritty details of how to do HTTP interactions, so I need to provide it with some kind of Http client library.  Obviously, I’ve chosen to use the Microsoft Http client, but in theory any library could work. 

At this point the only thing that the RestAgent is really capable of doing is navigating to the provided url and retreiving the returned content.  The important concept here is that at any one point in time, the RESTAgent can only be in one place.  As we expand on the capabilities of the agent we will allow it to accumulate content during its travels and the aggregate of all of that content will be the “application state”, but if our client application really needs to be in multiple places at once, then we will create multiple agent instances.

For just a small taste of how you might use this RestAgent,

var agent = new RestAgent(new HttpClient());

agent.NavigateTo(new Uri(“http://api.stackoverflow.com/1.0/questions”)); 
var questions = agent.CurrentContent.ReadAsJson();

agent.NavigateTo(“http://search.twitter.com/search.atom?q=@darrel_miller”) 
var mentions = agent.CurrentContent.ReadAsSyndicationFeed();

In the next post, I’ll talk more about hypermedia and how the agent can use links from hypermedia content to travel further and gather more content.

[1]  I originally wanted to call this class AgentFielding to anthropomorphize the class even more, but it seemed a touch presumptuous

A canary in a coal mine.

I have been waiting for a question like this for close to a year now. It was inevitable that this day would come and I am sure we are going to see many more questions like this in the coming months.

The warning signs

You are absolutely correct, it does take longer to build RESTful clients than SOAP clients. The SOAP toolkits take away lots of boilerplate code and make client proxy objects available with almost no effort. With a tool like Visual Studio and a server URL I can be accessing remote objects of arbitrary complexity, locally in under five minutes.

Services that return application/xml and application/json are so annoying for client developers. What are we supposed to do with that blob of data?

Fortunately, lots of sites that provide REST services also provide a bunch of client libraries so that we can use those libraries to get access to a bunch of strongly typed objects. Seems kind of dumb though. If they had used SOAP we could have code-gen’d those proxy classes ourselves.

SOAP overhead, ha. It’s latency that kills. If people are really concerned about the number of excess bytes going across the wire then maybe HTTP is not the right choice. Have you seen how many bytes are used by the user-agent header?

Yeah, have you ever tried using a web browser as debugging tool for anything other than HTML and javascript. Trust me it sucks. You can only use two of the verbs, the caching is constantly getting in the way, the error handling swallows so much information, it’s constantly looking for a goddamn favicon.ico. Just shoot me.

Readable URL. Only nouns, no verbs. Yeah, that’s easy as long as we are only doing CRUD operations and we only need to access a hierarchy of objects in one way. Unfortunately most applications need a wee bit more functionality than that.

The impending disaster

There are a metric boatload of developers currently developing applications that integrate with REST services who are in the process of coming to the same set of conclusions that you have. They were promised simplicity, flexibility, scalability, evolvabilty and the holy grail of serendipitous reuse. The characteristics of the web itself, how can things go wrong.

However, they are finding that versioning is just as much of a problem, but the compiler doesn’t help detect issues. The hand written client code is a pain to maintain as the data structures evolve and URLs get refactored. Designing APIs around just nouns and four verbs can be really hard, especially with RESTful Url zealots telling you when you can and cannot use query strings.

Developers are going to start asking why are we wasting our effort on support both Json formats and Xml formats, why not just focus our efforts on one and do it well?

How did things go so wrong

I’ll tell you what went wrong. We as developers let the marketing departments take advantage of our primary weakness. Our eternal search for the silver bullet blinded us to the reality of what REST really is. On the surface REST seems so easy and simple. Name your resources with Urls and use GET, PUT, POST and DELETE. Hell, us devs already know how to do that, we have been dealing with databases for years that have tables and columns and SQL statements that have SELECT, INSERT, UPDATE and DELETE. It should have been a piece of cake.

There are other parts of REST that some people discuss, such as self-descriptiveness, and the hypermedia constraint, but these constraints are not so simple as resource identification and the uniform interface. The seem to add complexity where the desired goal is simplicity.

This watered down version of REST became validated in developer culture in many ways. Server frameworks were created that encouraged Resource Identification and the uniform interface, but did nothing to support the other constraints. Terms started to float around differentiating the approaches, (HI-REST vs LO-REST, Corporate REST vs Academic REST, REST vs RESTful).

A few people scream out that if you don’t apply all of the constraints it’s not REST. You will not get the benefits. There is no half REST. But those voices were labelled as religious zealots who were upset that their precious term had been stolen from obscurity and made mainstream. Jealous people who try to make REST sound more difficult than it is.

REST, the term, has definitely become mainstream. Almost every major web property that has an API supports "REST". Twitter and Netflix are two very high profile ones. The scary thing is that I can only think of one public API that is self-descriptive and there are a handful that truly implement the hypermedia constraint. Sure some sites like StackOverflow and Gowalla support links in their responses, but there are huge gaping holes in their links. The StackOverflow API has no root page. Imagine how successful the web site would have been if there was no home page for the web site!

You were misled I’m afraid

If you have made it this far, the short answer to your question is those APIs (Netflix and Twitter) do not conform to all of the constraints and therefore you will not get the benefits that REST apis are supposed to bring.

REST clients do take longer to build than SOAP clients but they are not tied to one specific service, so you should be able to re-use them across services. Take the classic example, of a web browser. How many services can a web browser access? What about a Feed Reader? Now how many different services can the average Twitter client access? Yes, just one.

REST clients are not supposed to be built to interface with a single service, they are supposed to be built to handle specific media types that could be served by any service. The obvious question to that is, how can you build a REST client for a service that delivers application/json or application/xml. Well you can’t. That’s because those formats are completely useless to a REST client. You said it yourself,

you have to make "guesses" as to what will come back across the pipe as there is no real schema or reference document

You are absolutely correct for services like Twitter. However, the self-descriptive constraint in REST says that the HTTP content type header should describe exactly the content that is being transmitted across the wire. Delivering application/json and application/xml tells you nothing about the content.

When it comes to considering the performance of REST based systems it is necessary look at the bigger picture. Talking about envelope bytes is like talking about loop unwinding when comparing a quick-sort to a shell-sort. There are scenarios where SOAP can perform better, and there are scenarios where REST can perform better. Context is everything.

REST gains much of its performance advantage by being very flexible about what media types it supports and by having sophisticated support for caching. For caching to work well though nearly all of the constraints must be adhered to.

Your last point about readable urls is by far the most ironic. If you truly commit to the hypermedia constraint, then every URL could be a GUID and the client developer would lose nothing in readability.

The fact that URIs should be opaque to the client is one of the most key things when developing REST systems. Readable URLs are convenient for the server developer and well structured URLs make it easier for the server framework to dispatch requests, but those are implementation details that should have no impact on the developers consuming the API.

The Twitter API is not even close to being RESTful and that is why you are unable to see any benefit to using it over SOAP. The Netflix API is much closer but it’s use of generic media types demonstrates that failing to adhere to even a single constraint can have a profound impact on the benefits derived from the service.

It may not be all their fault

I’ve done a whole lot of dumping on the service providers, but it takes two to dance RESTfully. A service may follow all of the constraints religiously and a client can still easily undo all of the benefits.

If a client hard codes urls to access certain types of resources then it is preventing the server from changing those urls. Any kind URL construction based on implicit knowledge of how the service structures its urls is a violation.

Making assumptions about what type of representation will be returned from a link can lead to problems. Making assumptions about the content of the representation based on knowledge that is not explicitly stated in the HTTP headers is definitely going to create coupling that will cause pain in the future.

Should they have used SOAP?

Personally, I don’t think so. REST done right allows a distributed system to evolve over the long term. If you are building distributed systems that have components that are developed by different people and need to last for many years, then REST is a pretty good option.

Here is a bare bones implementation of an OData service, no magic allowed (Ok, without actually going to the command line compiler). 

What you will need for this experiment is:

1) .Net Framework 3.5 sp1

2) A new Visual Studio 2008 console application

3) These references

• System
• System.Core
• System.ServiceModel          <—This brings in the WCF stack
• System.ServiceModel.Web <—This brings in the Web friendly WCF service host
• System.Data.Services         <—This brings in the OData service host

4) and this code

using System;
using System.Collections.Generic;
using System.Data.Services;
using System.Linq;

namespace ODataConsoleService {
    class Program {
        static void Main(string[] args) {

            string serviceAddress = "http://localhost:998";
            Uri[] uriArray = { new Uri(serviceAddress) };
            Type serviceType = typeof(CustomerDataService);

            using (var host = new DataServiceHost(serviceType, uriArray)) {
                host.Open();
                Console.WriteLine("Press any key to stop service");
                Console.ReadKey();
            }

        }

    }

    // Expose IQueryable properties as read-only Atom collections
    public class CustomerDataService : DataService<CustomerService> {
        public static void InitializeService(IDataServiceConfiguration config) {
            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
        }
    }

    // Expose list of customers Customer as IQueryable
    public class CustomerService {
        private List<Customer> _List = new List<Customer>();

        public CustomerService() {

            _List.Add(new Customer() { ID = 1, Code = "BLAH", Name = "Blah" });
            _List.Add(new Customer() { ID = 2, Code = "Bob", Name = "Bob's Hardware" });
            _List.Add(new Customer() { ID = 3, Code = "Bill", Name = "Bills Fishmongers" });
        }

        public IQueryable<Customer> Customers {
            get {
                return _List.AsQueryable<Customer>();
            }
        }

    }

    // Entity to expose in atom:Entry
    public class Customer {
        public int ID { get; set; }  // ID is required to be a viable OData entity
        public string Code { get; set; }
        public string Name { get; set; }
    }

}

That’s it.  That’s all you need to expose a complete Atom Pub compliant service exposing your objects.

Once you have that, you can Run the program and use a web browser to browse to http://localhost:998 to see the collections.   Don’t forget to look in the magic url http://localhost:998/$metadata  Sorry, some magic I am still trying to figure out how to remove!

Credit goes to Alex James and Phani for getting me most of the way there.

It will download only the videos related to the tag name that you specify and currently it is set to download the low quality wmv files.  Feel free to pick whatever content type you like!

You will need the http client library that is buried in here, and you need to set the tagName to whatever interests you, other than that, it should just work.  Please note, there is only one hardcoded url in this client

If you have questions, you can usually find me on twitter @darrelmiller

var httpClient = new HttpClient("http://api.visitmix.com/OData.svc");

string tagName = "REST";

var serviceDocument = httpClient.Get().Content.ReadAsDataContract<AtomPub10ServiceDocumentFormatter>().Document; ;

// Get Url for Tags collection
var tagsUrl = (from co in serviceDocument.Workspaces[0].Collections
               where co.Title.Text == "Tags"
               select new Uri(serviceDocument.BaseUri, co.Link)).FirstOrDefault();

var tagsDocument = httpClient.Get(tagsUrl).Content.ReadAsDataContract<Atom10FeedFormatter>().Feed;

// Find Sessions Link for tag
var sessionsUri = (from tg in tagsDocument.Items
           from lk in tg.Links
           where tg.Title.Text == tagName &&
                lk.RelationshipType == "http://schemas.microsoft.com/ado/2007/08/dataservices/related/Sessions"
          select new Uri(tagsDocument.BaseUri, lk.Uri)).First();

// Follow the link to get the list of sessions
var sessionsForTagFeed = httpClient.Get(sessionsUri).Content.ReadAsDataContract<Atom10FeedFormatter>().Feed; ;

// Get the files links for each session
var sessionFileLinks = (from st in sessionsForTagFeed.Items
              from lk in st.Links
              where lk.RelationshipType == "http://schemas.microsoft.com/ado/2007/08/dataservices/related/Files"
              select new Uri(sessionsForTagFeed.BaseUri,lk.Uri)
              );

// Loop through each list of files and download
foreach (Uri sessionFilesLink in sessionFileLinks) {

    var filesDocument = httpClient.Get(sessionFilesLink).Content.ReadAsDataContract<Atom10FeedFormatter>().Feed;

    var files = from it in filesDocument.Items
                let uri = it.GetMediaResourceUri()
                where it.Content.Type == "video/x-ms-wmv" && !uri.ToString().Contains("wmv-hq")
                select uri;

    foreach (Uri uri in files) {
        try {
            var response = httpClient.Get(uri);
            var filename = uri.Segments.Last();
            using (var file = new FileStream(filename, FileMode.CreateNew)) {
                response.Content.WriteTo(file);
                response.EnsureStatusIsSuccessful();
            }
        }
        catch (WebException ex) {
            Console.WriteLine("Failed to download " + uri.ToString() + " : " + ex.Message);
            // Continue onto next
        }

    }
}

Just paste this code into a Console app and add references to the following DLL’s

• System.ServiceModel.Web (.Net 3.5)
• System.Runtime.Serialization
• Microsoft.Http  ( found in \Program Files (x86)\Microsoft WCF REST\WCF REST Starter Kit Preview 2\Assemblies)
• Microsoft.Http.Extensions  ( as above)

Assuming you have the following XML,

<Customers>
    <Customer Id="99">
        <Name>Bob</Name>
        <Age>39</Age>
        <Address>
            <Street>10 Idle Lane</Street>
            <City>Yucksville</City>
            <PostalCode>xxxyyy</PostalCode>
        </Address>
    </Customer>
    <Customer Id="101">
        <Name>Bill</Name>
        <Age>39</Age>
        <Address>
            <Street>10 Idle Lane</Street>
            <City>Yucksville</City>
            <PostalCode>xxxyyy</PostalCode>
        </Address>
    </Customer>

</Customers>

Using this XSLT

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="text"/>

    <xsl:template match="/">{
        <xsl:apply-templates select="*"/>}
    </xsl:template>

    <!-- Object or Element Property-->
    <xsl:template match="*">
        "<xsl:value-of select="name()"/>" : <xsl:call-template name="Properties"/>
    </xsl:template>

    <!-- Array Element -->
    <xsl:template match="*" mode="ArrayElement">
        <xsl:call-template name="Properties"/>
    </xsl:template>

    <!-- Object Properties -->
    <xsl:template name="Properties">
        <xsl:variable name="childName" select="name(*[1])"/>
        <xsl:choose>
            <xsl:when test="not(*|@*)">"<xsl:value-of select="."/>"</xsl:when>
            <xsl:when test="count(*[name()=$childName]) > 1">{ "<xsl:value-of select="$childName"/>" :[<xsl:apply-templates select="*" mode="ArrayElement"/>] }</xsl:when>
            <xsl:otherwise>{
                <xsl:apply-templates select="@*"/>
                <xsl:apply-templates select="*"/>
    }</xsl:otherwise>
        </xsl:choose>
        <xsl:if test="following-sibling::*">,</xsl:if>
    </xsl:template>

    <!-- Attribute Property -->
    <xsl:template match="@*">"<xsl:value-of select="name()"/>" : "<xsl:value-of select="."/>",
    </xsl:template>
</xsl:stylesheet>

You can generate this JSON

{
    "Customers" : {
        "Customer" : [
            {
                "Id" : "99",
                "Name" : "Bob",
                "Age" : "39",
                "Address" : {
                    "Street" : "10 Idle Lane",
                    "City" : "Yucksville",
                    "PostalCode" : "xxxyyy"
                }
            },
            {
                "Id" : "101",
                "Name" : "Bill",
                "Age" : "39",
                "Address" : {
                    "Street" : "10 Idle Lane",
                    "City" : "Yucksville",
                    "PostalCode" : "xxxyyy"
                }
            }
        ]
    }
} 

It would be really interesting to try and use this Javascript XSLT processor to do the transform directly in the browser.  That way any web api that generates XML could be flipped into a JSON representation for direct access in Javascript.

The Microsoft.Http library provides an alternative solution, assuming you don’t mind taking a dependency on it.  The HttpClient class supports the notion of a pipeline of stages with one of those stages being a transport stage that actually makes the real http request.  If we introduce a new stage prior to the transport stage we can intercept the request and return a response object with pre-canned values.  

//Arrange
var client = new HttpClient();
client.Stages.Add(new FixedTransport((key) => new HttpResponseMessage() {
   Content = HttpContent.Create("Hello World")
}));

//Act
var  response = client.Get("http://example.org");

//Assert
Assert.AreEqual("Hello World",response.Content.ReadAsString());

This unit test creates a instance of the real HttpClient and adds a new processing stage called FixedTransport.  This class is available in Microsoft.Http.Test and unfortunately you cannot create your own version of this class because the method that needs to be overridden to create a custom response is marked as internal.  The Test dll can do it because it is pointed to by the InternalsVisibleTo attribute.  The only other option is to recompile the core library yourself.

The FixedTransport stage is actually derived from a caching stage that can associate a URI to particular response.  In the sample above, I simply ignore the key and return the same content for all URIs.  However, this mechanism would allow you to preload the cache with a set of URI/response pairs and then you could run a test that performed multiple GET requests.

I’m not exactly sure why but from looking through the source it does appear that if you made a non-GET request the key value would be null.  That would allow you to return a specific response to a POST or PUT request based on that null key value.

I hope it is fairly obvious even after this very simple example that you should be able to easily generate pre-canned responses to standard http requests and still have access to the full capabilities of the HttpClient library without having to fake any classes.

Command Query Responsibility Separation(CQRS) is an interesting architecture that is suitable for highly scalable applications with complex business logic.  Instead of me doing a horrible job of trying to explain it, here are links to people who know what they are talking about.

Greg Young – http://codebetter.com/blogs/gregyoung/archive/2009/07/15/unshackle-your-domain.aspx

UDI Dahan – http://www.udidahan.com/2009/12/09/clarified-cqrs/

The reporting side of CQRS requires a simple mechanism for getting read-only data from a server down to the presentation tier with minimal effort.  I am suggesting the use of XML as a wire format and I am going to make the assumption that we have chosen a relational database as the store for the reporting database. More specifically, we require Microsoft SQL Server because, from what I can find, MS SQL Server is the only database engine that supports this level of flexibility when generating XML from relational tables.  

Ideally we should be able to create any desired structure of XML based on data in our relational database.  We need the flexibility to generate data as attributes or elements, create nested structures, use namespaces or not, at a minimum.

I’m going to start showing some of the basics of the FOR XML PATH syntax and build on it until we are creating relatively complex xml documents in a single SQL query.

This is pretty much the simplest query that you can create:

SELECT Code AS 'Code',
       Name AS 'Name'
FROM tblCustomer
FOR XML PATH('Customer')

It produces XML that looks like this:

<Customer>
    <Code>CUSTOMER-ONE</Code>
    <Name>Hypro Networks Inc.</Name>
    </Customer>
<Customer>
    <Code>CUSTOMER-A</Code>
    <Name>Customer A</Name>
</Customer>

The only problem with this is that it is not a valid XML document as there is more than one root element.  To create a proper XML document, you need to do:

SELECT Code AS 'Code',
        Name AS 'Name'
FROM tblCustomer
FOR XML PATH('Customer'), ROOT('Customers')

which then gives you,

<Customers>
  <Customer>
    <Code>CUSTOMER-ONE</Code>
    <Name>Hypro Networks Inc.</Name>
  </Customer>
  <Customer>
    <Code>CUSTOMER-A</Code>
    <Name>Customer A</Name>
  </Customer>
</Customers>

These examples showed how to create data as XML elements, but sometimes it is desirable to use attributes.  Attributes take quite a bit less space and suitable for most content that is small and does not contain significant whitespace.

SELECT Code AS '@Code',
        Name AS '@Name'
FROM tblCustomer
FOR XML PATH('Customer'), ROOT('Customers')

By adding an @ symbol to the front of the column name, the data will be output as attributes:

<Customers>
  <Customer Code="CUSTOMER-ONE" Name="Hypro Networks Inc." />
  <Customer Code="CUSTOMER-A" Name="Customer A" />
</Customers>

One of the things that I like about using XML is that it is easy to group related pieces of data into other elements.  The following query demonstrates how to do this:

SELECT Code AS '@Code',
        Name AS '@Name',
        Tel1 AS 'Contact/@Tel',
        Fax1 AS 'Contact/@Fax1'
FROM tblCustomer
FOR XML PATH('Customer'), ROOT('Customers')

Adding the slash into the column name directs SQL Server to create a new element to contain other nodes.

<Customers>
  <Customer Code="CUSTOMER-ONE" Name="Hypro Networks Inc.">
    <Contact Tel="1-993-345-4146" Fax1="1-993-345-4140" />
  </Customer>
  <Customer Code="CUSTOMER-A" Name="Customer A">
    <Contact Tel="872-494-2000" Fax1="872-494-3008" />
  </Customer>
</Customers>

Just one small warning, columns must be adjacent in the SQL query to exist under the same child element and all attributes of the parent element must be specified in the SQL query before the columns that will be in a child element.

So far we have only considered data from a single table.  Using a simple SQL join it is easy to pull in data from other tables, but we can also create parent/child documents.

SELECT  Code AS '@Code',
        Invoice_Date AS '@Date',
        (SELECT ii.qty AS 'Quantity',
                LEFT(ii.description,20) AS 'Description'
         FROM tblinvoitem ii WHERE ii.invoice_id = iv.id FOR XML PATH('Item'),type)
FROM tblInvoice iv
FOR XML PATH('Invoice'), ROOT('Invoices')

<Invoices>
  <Invoice Code="21201     " Date="2005-05-26T00:00:00">
    <Item>
      <Quantity>1.0000</Quantity>
      <Description>To supply material a</Description>
    </Item>
  </Invoice>
  <Invoice Code="21200     " Date="2005-05-18T00:00:00">
    <Item>
      <Quantity>25.0000</Quantity>
      <Description>S.S. Control Box Cov</Description>
    </Item>
    <Item>
      <Quantity>0.0000</Quantity>
      <Description>S.S. Main Control Bo</Description>
    </Item>
  </Invoice>
</Invoices>

The extra “type” parameter on the end of the subquery indicates to SQL that we want to embed the contents of the sub query as XML rather than just an escaped string.

One limitation that you will probably run into once you create this type of document is that there is no obvious way to insert attributes and elements into the root node.  This can be overcome with a small trick:

SELECT '1.0' AS '@Version',
        'List of Invoices' AS 'Summary',
( SELECT  Code AS '@Code',
        Invoice_Date AS '@Date',
        (SELECT ii.qty AS 'Quantity',
                LEFT(ii.description,20) AS 'Description'
         FROM tblinvoitem ii WHERE ii.invoice_id = iv.id FOR XML PATH('Item'),type)
FROM tblInvoice iv
FOR XML PATH('Invoice'),type)
FOR XML PATH('Invoices')

Using an outer query you can specify the attributes and elements that you want to appear in the root.  Note that there is no alias for the sub query column. 

<Invoices Version="1.0">
  <Summary>List of Invoices</Summary>
  <Invoice Code="21201     " Date="2005-05-26T00:00:00">
    <Item>
      <Quantity>1.0000</Quantity>
      <Description>To supply material a</Description>
    </Item>
  </Invoice>
  <Invoice Code="21200     " Date="2005-05-18T00:00:00">
    <Item>
      <Quantity>25.0000</Quantity>
      <Description>S.S. Control Box Cov</Description>
    </Item>
    <Item>
      <Quantity>0.0000</Quantity>
      <Description>S.S. Main Control Bo</Description>
    </Item>
  </Invoice>
</Invoices>

And of course, no discussion on XML is complete without discussing namespaces.  Here is how you can add namespaces to the XML result:

WITH XMLNAMESPACES(DEFAULT 'http://example.org/Invoices')
SELECT '1.0' AS '@Version',
        'List of Invoices' AS 'Summary',
( SELECT  Code AS '@Code',
        Invoice_Date AS '@Date',
        (SELECT ii.qty AS 'Quantity',
                LEFT(ii.description,20) AS 'Description'
         FROM tblinvoitem ii WHERE ii.invoice_id = iv.id FOR XML PATH('Item'),type)
FROM tblInvoice iv
FOR XML PATH('Invoice'),type)
FOR XML PATH('Invoices')

I’m not going to show you the output of this, because it is ugly!  Actually, the annoying part is that it puts the namespace on the root and then on each row of the sub query. 

Hopefully these examples have shown some of the capability of FOR XML PATH but I find that sometimes technologies work well while you are working on arbitrary examples, but then you hit the real world and they seem to fall short.  So, I decided to try and create an Atom feed using FOR XML PATH.  Here is what I came up with.

WITH XMLNAMESPACES(DEFAULT 'http://www.w3.org/2005/Atom')
SELECT 'Customers' AS 'title',
        'List of customers' AS 'subtitle',
        'http://tavis.net/Customers' AS 'id',
        REPLACE(CONVERT(varchar,(SELECT MAX(COALESCE(ModifiedDate,CreatedDate)) FROM tblCustomer), 121),' ','T') + 'Z' AS 'updated',
    (SELECT cu.Code AS 'title',
            REPLACE(CONVERT(varchar,COALESCE(ModifiedDate,CreatedDate) , 121),' ','T') + 'Z' AS 'updated',
            'Darrel Miller' AS 'author/name',
            'http://tavis.net/Customer/' + LTRIM(STR(cu.ID)) AS 'id',
            (SELECT *
                FROM (SELECT 'alternate' AS '@rel',
                    'http://tavis.net/Customer/1.html' AS '@href'
              UNION
              SELECT 'edit' AS '@rel',
                    'http://tavis.net/Customer/1.html' AS '@href') l
             FOR XML PATH('link'),type ),
            cu.ModifiedDate AS 'updated',
            cu.Name AS 'summary'
        FROM tblCustomer cu
        FOR XML PATH('entry'),type  )
FOR XML PATH('feed')

Ok, so it is not pretty, but with a few T-SQL functions it could be cleaned up quite a bit.  This query produces an atom feed that validates at validator.w3.org/feed and it does it in 8ms on my cheapo server with 20 entries .

The trickiest part of the above query was creating the two link elements.  When you try and create two child elements with the same name, by default SQL Server will attempt to merge the two elements.  By using the UNION and a sub query I was able to create the two separate elements.

So you really can create real world XML documents using FOR XML PATH. I believe this is a useful tool to have under your belt when you are trying to quickly get data out of a database down to a client tier.  It would also be really nice if other database vendors picked up on this feature and implemented in their engines.  One area that I didn’t cover but that is also very useful is that you can create T-SQL Functions that return XML and then call them recursively.  This allows you to build hierarchical XML documents.   My tests so far have also shown that building trees this way is very quick.

I’m hoping to do a follow up article that shows how you can use the XML from FOR XML PATH as input to XSLT to do all sorts of other interesting things like create audit triggers, data import scripts, create HTML pages, XAML  and JSON documents. 

I have two questions for Dion and a few comments.
1) Have you read (not just skimmed a few times) Roy’s dissertation on REST?
2) Have you written both a REST service and a REST client that is in production today?

With all due respect, without actually creating REST applications, I don’t think anyone is qualified to attempt to define principles relating to REST and based on your Twitter bio:

Internationally recognized business strategist, enterprise architect, keynote speaker, author, blogger, and consultant on Web 2.0, SOA, and next-gen business

you sound like a talker, not a walker.  Forgive me if I am wrong.

Here are my comments on the suggested principles, I’ve tried to be as constructive as I can muster:

Principle #1: I think it is a significant oversimplification to refer to a resource  as data. 

Principle #2: Can you clarify what you mean by "favor granularity and depth in linkage"?

Principle #3: I am pretty sure no-one has ever said that URIs should be self descriptive.  In REST the request message should be self-descriptive, the URI being just one part of the message.   The statement “URI should indicate what data format is being used and indicate nested elements with URL segmentation” is way too strong, maybe if you replace “should” with “can”.

Principle #4:  This idea that all legacy databases should be exposed as data-oriented REST apis is crazy-talk.  REST is an architectural style for building distributed APPLICATIONS.  I.e. Exposing functionality over the web, not just data. Your example of cloud computing API’s is a perfect example.  The Sun Cloud API is one of the best REST examples out there at the moment, but it exposes functionality for manipulating those cloud applications.  Functions like turning the instances on and off.  It’s far more than just exposing data.

Principle #5:  You seem to be describing benefits rather than a principle.

#6:  Ok

#7: I don’t get this concept at all.  Does this mean we should all be using text/plain if we want the widest audience?

#8: Maybe I don’t want my resources to crawled by an SEO.  Even if I do, conforming to the uniform interface and delivering standard format representations should be sufficient to meet any search engine requirements.

#9:  “higher realized value”?  Are you sure this isn’t another one of those spoof manifestos?

#10: I think you are referring to the idea that a resource should only have one canonical URI.  I see this topic debated regularly and I do not yet see any consensus.

Ok, I give up for the moment, I’m a bit too depressed to continue.