So far the only significant operation that we enabled our RestAgent to perform is NavigateTo().  However, for a RestAgent on a mission, going places is only half the story, the other major purpose of the agent is to gather content.

I was planning to show some real examples of missions using Stack Overflow’s API, but I ended up spending so much time explaining why it currently would not work, that my points were lost.  So forgive me as I continue with a somewhat more hypothetical example.  We are going to use an instance of our RestAgent to do a mashup of Twitter users and Stackoverflow users.

var agentFielding = new RestAgent(new HttpClient()); 

agentFielding.RegisterMediaType(“application/twitterdoc+xml”,
                                    new TwitterMediaHandler());
agentFielding.RegisterMediaType(“application/stackoverflowdoc+xml”,
                                    new StackOverflowMediaHandler());

In the following code, Agent Fielding retrieves my Twitter followers and attempts to find matching accounts on Stack Overflow.  In order to interpret the representations that will be retrieved from these two sites, we are going to pretend that these sites actually return non-generic media types and we register handlers for these media types that will transform the wire representation into strong types.

The first step is to navigate Twitter and retrieve the user profiles of my followers:

agentFielding.NavigateTo(new Uri(“http://api.twitter.com/”));
var userSearch = agentFielding.CurrentLinks(“UserSearch”);
userSearch.SetParameter("darrel_miller”);
agentFielding.NavigateTo(userSearch);

var followersLink = agentFielding.CurrentLinks[“Followers”);
agentFielding.NavigateTo(followersLink);

var twitterUserProfiles = new List<TwitterUserProfile>();
var followerLinks = agentFielding.GetCurrentLinks(l=> l.Relation.Name == ”Follower”];
foreach(Link followerLink in followerLinks) {

  var twitterProfile = agentFielding.GetContent(followerLink).ReadAsTwitterUserProfile();
  twitterUserProfiles.Add(twitterProfile);

}

Now we have a list of TwitterUserProfile objects from which we can retrieve the name and search on Stack Overflow.

agentFielding.NavigateTo(new Uri(“http://api.stackoverflow.com/1.0/”)); 

agentFielding.NavigateTo(agent.CurrentLinks[“Users”]);
var searchLink = agent.CurrentLinks[“Search”];
var foundProfiles = new List<StackOverflowUserProfile>();
foreach(var profile in twitterUserProfiles) {   searchLink.SetParameter(profile.UserName);
  agentFielding.NavigateTo(searchLink); 

  var matchingUserLinks = agentFielding.GetCurrentLinks(l=> l.Relation.Name == ”User”])
  foreach(Link userLink in matchingUserLinks) {
       var content = agentFielding.GetContent(userLink);
       foundProfiles.Add(content.ReadAsStackOverflowUserProfile();
  }
}

Don’t get too hung up on the precise details of the above, there is a bit of smoke and mirrors going on.  I’m really just trying to convey the concept that the same agent class can be used to navigate multiple different services.  This brings the concept of the uniform interface a little further into the client code. 

My hope is that in the future that REST api producers will supply media type handlers for their custom media types and we can use a standardized agent like interface for navigating any REST api.  There will be no need the API producers to create complete client side facades.

There is still plenty of coupling in the above example, when it comes to handling specific representations and knowing about the existence of specific links, there is plenty of service specific code.  This is especially true of these type of scripted or (machine to machine) interactions.  Machines are really dumb, so they need to understand a lot of specifics of the services that they are interacting with.  In the next article I’m going to start digging into how you can use an agent to react to a human driven application, and that will  further reduce the coupling. 

var agent = new RestAgent(new HttpClient());
agent.NavigateTo(new Uri(“http://api.stackoverflow.com/”)); 
var questionsLink = agent.CurrentLinks[“Questions”]
agent.NavigateTo(questionsLink);
var questions = agent.CurrentContent.ReadAsJson(); 

Before I talk about the magic that is going on under the covers here, let me take a little jab at my friend StackOverflow.  The above code would not work, because when the team created the api, they chose not to put a root representation on the API. Making the request to the root of the api actually redirects to an html page with API documentation.  I’ve tried scraping the urls out of that document, but it is proving to be painful.

The Link Class

The CurrentLinks property makes a dictionary of links available based on the links contained in the current content and potentially links that were included in the header.  You will notice that the second call to NavigateTo passes a “Link”, not a URL.  So what’s the difference?  Here is a basic implementation of the Link class:

public class Link {

  public string Name{ get; set;}
  public LinkRelation Relation{ get; set;}
  public Uri Url { get; set;}
  public HttpMethod Method { get; set;}
  public HttpContent Content {get; set;}

  public Dictionary<Parameter> QueryParameters {get; set;}
  public Dictionary<Parameter> PathParameters {get; set;}

}

The Link class holds plenty of additional information that is needed to allow the NavigateTo method to transfer to a new state.  The Link objects are created from information contained in retrieved hypermedia content.  If the HttpMethod is a PUT or a POST then the Content property can be used to hold the entity that will be sent to the origin server. 

The LinkRelation property is critical concept.  The Link Relation is one of the two points of coupling between the client and the server.  The other being the media-type.  From an architectural point of view the Link Relation describes the relationship between two resources.  For example, if you have a representation A that contains a link to resource B with the relation “edit”, this can be interpreted as B allows you to “edit” A.  IANA keeps a registry of well known link relations here http://www.iana.org/assignments/link-relations/link-relations.txt.  The WHATWG group are also maintaining a list here http://wiki.whatwg.org/wiki/RelExtensions.

From an implementation perspective, the Link Relation can be used to tell the client mechanical information about how to perform the HTTP request.  For example, what HTTP method to use, what content needs to be posted back to the server, if any. It also can be used to tell the client what query string parameters are valid.  Arguably you can introduce any kind of arbitrary significance to a link relation as long as you are prepared to put up with the client/server coupling.

I have been experimenting with implementing LinkRelation as an actual class instead of just an identifer.  Many different link relations have similar sets of behaviour and by creating a base class with some standard attributes it is possible to represent specific link relations as derived classes and still have plenty of generic mechanisms that process links without scattering the code base with fragile switch statements.  Hopefully as we go into more detail I will show some examples that take advantage of this LinkRelation base class.

So far I have only used very simple URI templates and so my needs have been satisfied with just a collection of query parameter values and path parameter values.  The current draft of the URI template specification introduces many more variants that I could be supported by the Link class.

The Name property is the black sheep of the Link class.  I’m not convinced that we really need this property.  However, I include it to support the scenario where there are multiple links within a hypermedia document that has the same LinkRelation.  How a client is supposed to differentiate between those different links without introducing out of band coupling is not always clear to me.  If the current representation is to be rendered to the user as a set of links then the user should be capable of deciding which they wish to follow.  However, in a scripted (ie. machine 2 machine) scenario, it becomes highly suspect if the script starts referring to link names.  Depending on how precise the media type, is, the link names could be pre-specified in the media-type.  Or code-download could be used to activate links with particular names.  The final solution that I have seen is to drop the name property completely and use very specific link relation values.  I’m not a big fan of this approach because I think it leads to the same explosion problems as using extremely specific media-types.  Highly specific media-types and link relations destroy the possibility of serendipitous reuse.

Hypermedia Content

So now, you have an idea of how I handle links, let’s look at how those links get created in the first place.  As I mentioned before, the CurrentLinks dictionary is populated from the CurrentContent.  The question of course is how can the generic RestAgent know how to pull links out of content that could be any media type.  The solution is a plug in model that allows handlers to be registered in a media type handler registry. 

Behind the scenes I use MEF to load up the media type handlers, but I stuck an interface in front, mainly because I’m not very experienced with MEF and I wasn’t exactly sure how to setup unit tests using MEF.  With the interface I can easily create a fake registry.

public interface IHandlerRegistry {
        IMediaTypeHandler GetHandler(ContentType contentType);
    }

Media type handlers I will cover in more detail at a later date.  All we care about for this series of articles, is that they can be used to do two things:  1) they can convert the stream of bytes coming from the http response into a class that implements IHypermediaContent and 2) they can create and run a controller with RunController(HttpResponseMessage response) method that will “do” whatever that media type is supposed to do.  I realize that sounds very vague, but vague is good when we don’t want to introduce coupling .

public interface IMediaTypeHandler {
    IHypermediaContent GetContent(HttpContent content);
    IMediaTypeController RunController(HttpResponseMessage message);
}

For this article I’m going to ignore RunController and we will get to human driven clients in an upcoming article.  That leaves us with GetContent that returns an IHypermediaContent interface.  That interface looks like this:

public interface IHypermediaContent {
       IDictionary<string, Link> Links { get; }
   } 

Not exactly complex to implement!  Consider what it would take to write a handler that processes, HTML, XHTML, Atom Feeds, HAL, or even one that attempts to scrape links out of Json or generic XML based on some link convention. 

Magical Links

Pulling this all back together, when you navigate the agent to a particular location on the web, the agent will identify the media type of the returned response, determine a handler for that media type, allow the handler to process the bytes and extra the links and store them in a dictionary available via the CurrentLinks property of the agent.

Here is a completely hypothetical use of the RestAgent to crawl safe links on a site:

var agent = new RestAgent(new HttpClient(), new Uri(“http://mythical.com/hypermediaService”));
ExploreLinks(agent);

public void ExploreLinks(RestAgent agent) {
var links = CopyLinks(agent.CurrentLinks);
foreach(Link link in links) {
    if (Link.Method == HttpMethod.GET && Link.LinkRelation.RequiresParameters == false) {
        agent.NavigateTo(link);
        Explore(agent);
    }
}
}

To keep things simple, I only traverse links whose LinkRelation specifies a GET is the appropriate verb and only links that do not require parameters. 

I realize that this is still not a very useful example, but enough of the pieces are in place so that in the next article I will talk more about how to get the agent to actually do something useful.

I write distributed business applications for a living and to date, it appears that I end up supporting the code I write for at least 10 to 15 years.  When ever you deploy and update distributed systems there is a nasty side effect of having different components to update and trying to deploy them simultaneously is rife with pitfalls.  Deploying an update to a server compared with updating software on 50 client machines is just a completely different set of problems and keeping them synchronized can be challenging. 

I realize that many people would argue that using a web browser as a client platform would solve my distribution problems.  It would solve the distribution problem, but it would also introduce a whole host of other problems for the types of applications I write.  Browser based apps are awesome for many things, some things they are not so awesome for.

The REST constraints allow you to manage the coupling between the distributed components and allow them to evolve more independently. This allows the system as a whole to evolve incrementally over long periods of time.  I need this kind of robustness in my line of work.

In the three years that I have been building RESTful systems I have seen lots written about how to design and implement RESTful APIs but very little has been written about how REST client applications need to be written.  The few people I have seen talk about it are Stu Charlton and Guilherme Silveira.

This is hopefully the first in a series of blog posts where I cover what I am learning about writing client applications that are part of a RESTful distributed system.

REST clients are a different breed

One of the core concepts when designing a REST client is the notion of Hypermedia as the Engine of Application State.  I’m not going to try and explain what I think it means, as many people far more eloquent than I, have tried and yet its meaning is still much debated.  Instead, I would prefer to show some code that I believe embodies the concept.

My REST client applications are built around a class called RestAgent.  I like to imagine my instance of the RestAgent class as little guy travelling across the web making requests and fetching data on my behalf [1].  This agent is responsible for holding all of my application state and for transferring state to and from origin servers.

In its most simple incarnation the class looks something like:

public class RestAgent(HttpClient client) {

  private Uri _CurrentLocation;
  private HttpContent_CurrentContent;

  public void NavigateTo(Uri url) {

    _CurrentContent = client.Get(url).Content;
    _CurrentLocation = url;
  }

}

The RestAgent itself does not know any of the nitty gritty details of how to do HTTP interactions, so I need to provide it with some kind of Http client library.  Obviously, I’ve chosen to use the Microsoft Http client, but in theory any library could work. 

At this point the only thing that the RestAgent is really capable of doing is navigating to the provided url and retreiving the returned content.  The important concept here is that at any one point in time, the RESTAgent can only be in one place.  As we expand on the capabilities of the agent we will allow it to accumulate content during its travels and the aggregate of all of that content will be the “application state”, but if our client application really needs to be in multiple places at once, then we will create multiple agent instances.

For just a small taste of how you might use this RestAgent,

var agent = new RestAgent(new HttpClient());

agent.NavigateTo(new Uri(“http://api.stackoverflow.com/1.0/questions”)); 
var questions = agent.CurrentContent.ReadAsJson();

agent.NavigateTo(“http://search.twitter.com/search.atom?q=@darrel_miller”) 
var mentions = agent.CurrentContent.ReadAsSyndicationFeed();

In the next post, I’ll talk more about hypermedia and how the agent can use links from hypermedia content to travel further and gather more content.

[1]  I originally wanted to call this class AgentFielding to anthropomorphize the class even more, but it seemed a touch presumptuous

The Microsoft.Http library provides an alternative solution, assuming you don’t mind taking a dependency on it.  The HttpClient class supports the notion of a pipeline of stages with one of those stages being a transport stage that actually makes the real http request.  If we introduce a new stage prior to the transport stage we can intercept the request and return a response object with pre-canned values.  

//Arrange
var client = new HttpClient();
client.Stages.Add(new FixedTransport((key) => new HttpResponseMessage() {
   Content = HttpContent.Create("Hello World")
}));

//Act
var  response = client.Get("http://example.org");

//Assert
Assert.AreEqual("Hello World",response.Content.ReadAsString());

This unit test creates a instance of the real HttpClient and adds a new processing stage called FixedTransport.  This class is available in Microsoft.Http.Test and unfortunately you cannot create your own version of this class because the method that needs to be overridden to create a custom response is marked as internal.  The Test dll can do it because it is pointed to by the InternalsVisibleTo attribute.  The only other option is to recompile the core library yourself.

The FixedTransport stage is actually derived from a caching stage that can associate a URI to particular response.  In the sample above, I simply ignore the key and return the same content for all URIs.  However, this mechanism would allow you to preload the cache with a set of URI/response pairs and then you could run a test that performed multiple GET requests.

I’m not exactly sure why but from looking through the source it does appear that if you made a non-GET request the key value would be null.  That would allow you to return a specific response to a POST or PUT request based on that null key value.

I hope it is fairly obvious even after this very simple example that you should be able to easily generate pre-canned responses to standard http requests and still have access to the full capabilities of the HttpClient library without having to fake any classes.

Handling returned content

When you do make an http request with this library, the body of the response is wrapped inside an HttpContent object. So, when you do:

var content = HttpClient.Get(“http://www.google.com”);

what you get back is an HttpContent object.  How you convert that into something useful, depends on what type of data the content object contains.  The HttpContent.ContentType property will tell you the Internet-media-type type of the data you received.

If you get back text/plain then you can access it like this,

var mytext = content.ReadAsString();

if you get application and application/octet-stream and you know what to do with the bytes, you can simply do

var mybytearray = content.ReadAsByteArray();

However, these examples are pretty primitive.  If you reference the DLL Microsoft.Http.Extensions you will find a variety of richer data extraction methods that have been implemented as extension methods on the HttpContent class.  So you can then do:

var xmlElement = content.ReadAsXElement()
var xmlReader = content.ReadAsXmlReader()

to access stuff that comes to you as application/xml.

One of the more sophisticated methods is ReadAsSyndicationFeed.  This method leverages the RSS/Atom wrappers that are provided by WCF’s System.ServiceModel.Web DLL.

var client = new HttpClient();
var response = client.Get("http://www.stackoverflow.com/feeds");
var feed = response.Content.ReadAsSyndicationFeed();
foreach (SyndicationItem item in feed.Items) {
    Console.WriteLine(item.Title.Text);
}

This example pulls an Atom feed from the front page of Stackoverflow.com.  This really shows the beauty of a standardized data format like Atom.  Those few lines of code above will work in so many places across the web, and now with Microsoft pushing the new OData standard which is based on the Atom Publishing Protocol, many more of the Microsoft products will expose data in this way. e.g. Sharepoint 2010, Azure.

For those of you who feel the burning desire to pull custom objects across the wire, you don’t have to feel ignored as these following methods will take care of all the deserialization work and return you a nice static type.

var customer = content.ReadAsXmlSerializable<Customer>()
var customer = content.ReadAsDataContract<Customer>()

and finally if you want to pull objects across the wire, but are offended by angle brackets, you can use curly braces too.

var customer = content.ReadAsJsonDataContract<Customer>()

The use of extension methods here is really quite elegant because it makes it really easy to add your own “ReadAs” methods and use them in a completely consistent way.  The fact that many of the provided extension methods are factored out into their own Dll means that you do not need to deploy that library, and you can use just your own extension methods.  This has the additional benefit of allowing you to use this library without taking a dependency on WCF if you don’t want to.

Sending Content

To send content you can use either PUT or POST and the way you prepare the content is identical.  The basic process looks like this:

var client = new HttpClient()
var content = HttpContent.Create(<whatever you want to send>);
client.Post(content); // or client.Put(content);

Suppose you want to send some simple text

var content = HttpContent.Create(“Here is some content”, "text/plain");

or maybe just an array of bytes.

var content = HttpContent.Create(new byte[] { 1, 2, 3 }, "application/octet-stream");

How about sending a file?

var c = HttpContent.Create(new FileInfo(“myfile.zip”), "application/octet-stream");

If you are dealing with an existing API that expects a POST from an html form then you will need to post using the content-type application/x-www-form-urlencoded.  That is as easy as,

var c = HttpContent.Create(new HttpUrlEncodedForm() { { "a", "1" }, { "b", "2" } });

and for a more concrete example, twitter always makes a good test subject.  Here is how you do a status update,

var client = new HttpClient();
client.DefaultHeaders.Authorization = Credential.CreateBasic("username","password");
var form = new HttpUrlEncodedForm();
form.Add("status","Test tweet using Microsoft.Http.HttpClient");
var content = HttpContent.Create(form);
var resp = client.Post("http://www.twitter.com/statuses/update.xml", content);

It is pretty easy to continue this pattern and create your own factory methods for converting your favourite data type into an HttpContent object.  There are are quite a few other nice features about the content object relating to how it manages the underlying stream, when it reads from the stream, if it buffers the stream and other such nitty gritty, but I’ll save those details for another day.  To keep track of the other articles I am attempting to write on the subject, you can see the summary post here.

Retrieve some HTTP content from an URL

var client = new HttpClient();
var response = client.Get("http://example.org");

Post some content to an Url

var client = new HttpClient();
var content = HttpContent.Create("This is some string content");
var response = client.Post("http://example.com", content);

Authenticated get

var client = new HttpClient();
client.DefaultHeaders.Authorization = Credential.CreateBasic("user", "password");
var response = client.Get("http://example.org");

and if you need to provide some kind of custom authentication scheme you can simply create credential like this

client.DefaultHeaders.Authorization = new Credential(“My secure authentication header”);

Interestingly, the methods Get and Post are not actually members of the HttpClient class.  They are extension methods that simply call one of the Send methods on the HttpClient class.  If it is more appropriate you can also call the Send methods directly. As you will see, there are plenty of overloads to suit your needs:

public HttpResponseMessage Send(HttpMethod method)
public HttpResponseMessage Send(HttpMethod method, Uri uri)
public HttpResponseMessage Send(HttpMethod method, Uri uri, RequestHeaders headers)
public HttpResponseMessage Send(HttpMethod method, Uri uri, HttpContent content)
public HttpResponseMessage Send(HttpMethod method, string uri)
public HttpResponseMessage Send(HttpMethod method, string uri, RequestHeaders headers)
public HttpResponseMessage Send(HttpMethod method, string uri, HttpContent content)
public HttpResponseMessage Send(HttpMethod method, string uri, RequestHeaders headers, HttpContent content)
public HttpResponseMessage Send(HttpMethod method, Uri uri, RequestHeaders headers, HttpContent content)

I like this approach because it provides a host of easy to call helper methods that are all routed through one of these Send methods. 

Another major advantage of the HttpClient library over the HttpWebRequest class is related to the RequestHeaders class.  All of the Http header values have been wrapped with some kind of helper class that significantly eases the process of setthing header values.

Here are a few random examples of setting header information that pulled from the unit tests:

var req = new RequestHeaders();
req.Accept.Add(StringWithOptionalQuality.Parse("audio/*; q=0.2"));

req.Allow.Add("GET");
req.Allow.Add("POST"); 

var cc = new CacheControl();
cc.MaxAge = TimeSpan.FromSeconds(1);
cc.MaxStale = true;
cc.MaxStaleLimit = TimeSpan.FromSeconds(2);
cc.MinFresh = TimeSpan.FromSeconds(3);
cc.MustRevalidate = true;
cc.NoCache = true;

req.CacheControl = cc;

In addition to these headers that have strongly typed classes there is also the nifty Parse() function that is on both the request and response header collections.

var h = RequestHeaders.Parse("Pragma: LinkBW=2147483647, AccelBW=1048576, AccelDuration=5000"); 

var resp = ResponseHeaders.Parse(
                @"Set-Cookie: mkt1=norm=US; domain=.live.com; path=/
                    Set-Cookie: AFORM=NOFORM; expires=Mon, 20-Jul-2015 23:59:59 GMT; path=/".Trim());

var h = RequestHeaders.Parse(
               @"Accept-Charset: iso-8859-5, unicode-1-1;q=0.8
           Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0
           Accept-Language: da, en-gb;q=0.8, en;q=0.7");

var h = ResponseHeaders.Parse(@"WWW-Authenticate: Digest realm=""testrealm@host.com"",
                qop=""auth,auth-int"",
                nonce=""dcd98b7102dd2f0e8b11d0f600bfb0c093"",
                opaque=""5ccc069c403ebaf9f0171e9517f40e41""");

Who wants to write and test the code necessary to parse this stuff?  Not me, that’s for sure. Hopefully this has provided a taste for what this library is capable of and over the next few weeks I plan on writing a number of other posts that cover other interesting areas of the Microsoft.Http namespace.  To see where I am up to, see the summary page here

Here are the posts I am planning.  I will create the links as the posts are completed.

• The basics
• HttpContent instead of streams
• But can you test it?
• HttpClient on the server

You can get the library here and the official support forum is here.  From what I have been told, this library will not make it into the .Net 4 release but it is still being actively developed.

However, the new client side library is designed to make it easier to talk to HTTP endpoints and it seems to be quite nice.  It is 3500 lines of code, so there is a fair bit of stuff in there, layered on top of HttpWebRequest and I was curious to see what it could do.

I started by looking at the enclosed sample, unhelpfully named “WpfSample” and I got a little sidetracked from my exploration.  This sample is actually a very neat little http request builder tool.  I know this is a cool tool to have, because I have an almost identical one that we built to help us test REST services.  Testing with real web browsers is really annoying because they do all sorts of magic stuff behind the scenes, they are very picky about what content-types they will render and they keep trying to find a favicon! 

Anyway, I was curious to see how this tool made use of the HttpClient class but I was rather dismayed to see all of the code in the “code behind” of the Window1.xaml file.  This makes it really difficult to distinguish between what code is relevant to the web request and what is just UI gunk.  To add salt to the wound, it is a WPF app that uses no data binding and no Commands, just events manipulating the contents of UI elements.  It was a nice looking, handy WPF app that demonstrated this new spiffy HttpClient library with the architecture of a college student’s VB4 project.

Ok, I know it is just a sample.  However, everything that Microsoft puts out is viewed by people who are learning.  If Microsoft create samples with crappy architecture, why are we surprised when developers create applications with crappy architectures.

So despite the ton of work that I was supposed to be doing today, I refactored it.  I created an MVC type of structure with databinding and routed commands.  If you get the WCF REST Starter Kit and extract the zip file that gets put in the “C:\Program Files (x86)\Microsoft WCF REST” folder you will find a solution named Product.sln.  Just add my project which is in the zip file below and you will have, what I hope is a much easier to read version of the HttpClient sample.

http://www.tavis.ca/files/httpClientGui.zip