HTTP Calls in Ruby

An API doesn't exist on its own. There are always two parties involved: The Client and the Server.

In Rails, our apps are often the ones acting as the server, and we typically know how to troubleshoot the issues that inevitably arise. We can tail the logs to see what the incoming request looks like (the path, headers, params), how we responded, etc... but with backend development going more and more to a microservices architecture, our Rails apps are going to more often have to act as the client in addition to the server.

More likely than not, being the client is going to mean having to make HTTP requests and parse JSON responses. More specifically, this means crafting the correct URL, HTTP headers for authentication, pagination, response format, and finally, the body of the request which is most likely either as form data (application/x-www-form-urlencoded) or as JSON.

Thankfully most of the more popular APIs, such as Twitter, Slack, and Stripe, already have client libraries written for them that encapsulate (hide) the actual lower-level HTTP request building and response parsing.

But when you have to connect to your internal services or lesser known APIs, which may not have a client library already built for them, well, we're on our own. The rest of this article will explore some of the different options available in Ruby for dealing with HTTP and some techniques for organizing the code and troubleshooting when things go wrong or you want a bit more visibility into what is happening.

There are quite a few HTTP libraries available in the Ruby ecosystem for making HTTP requests, so I won't be able to touch on them all. A few of the more popular ones are:

• Net::HTTP: Part of the Ruby standard library. While it can accomplish everything you need it to, there is probably a reason why so many third-party HTTP libraries exist. Personally I don't find it simple to work with and tend to opt for one of the other options in this list.

• Curb: This gem provides bindings for the libcurl program (the same one you use from the command line with the curl command). It's fast, mostly because a lot of the heavy lifting is being done outside of Ruby, and the libcurl program is fast.

• HTTParty: A very popular library (22 million downloads) that wraps the Net::HTTP code, providing an easier API to work with.

• HTTP: While not as popular as the other libraries, this one has been a favourite of mine recently when connecting to APIs. It provides a chainable interface for building HTTP requests and provides support for all of the usual functionality you'd expect from an HTTP library.

• Excon: Another very popular library (25 million downloads) written in pure Ruby. It has a clean API and is easy to use.

I won't show an example here of how to use each one of these libraries; you'll be able to find a simple one on the home page of each of them. In the next section, we'll be working through an example with the HTTP library and how we can encapsulate and organize our code.

It's best to provide some sort of interface for interacting with these API endpoints or services and hiding all of the lower-level HTTP request and response details. Other programming shouldn't need to be exposed to the specifics of the implementation, and that same implementation should be able to be changed/refactored without the need to change the public API.

Let's take a look at how we might do this for an API. We'll use the lcboapi.com API. It provides a nice little interface to access information relating to the beverages and stores of the LCBO (the governmental corporation in Ontario, Canada, responsible for the retail and distribution of alcohol for the province).

We'll be working with four classes:

• Lcbo::Products: The public interface we'll be working with to fetch details about a specific product.

• Lcbo::ProductRequest: Handles the HTTP request for a given Product ID.

• Lcbo::ProductResponse: Handles the HTTP response and knows how to build an Lcbo::Product.

• Lcbo::Product: The actual details of a product we fetch.

Before we dive into the implementation, let's take a look at how to use it:

require_relative 'lib/lcbo'
key = ENV.fetch('LCBO_API_KEY')
product = Lcbo::Products.new(key).fetch(438457)
puts product.name
# Hopsta La Vista
puts product.tags.inspect
# ["hopsta", "la", "vista", "beer", "ale", "canada", "ontario", "longslice", "brewery", "inc", "can"]

This class acts as the public interface to fetch details about one or more LCBO Products. At this point, I've only implemented the fetch method, which will return the details for a single product. This class' job is to build Requests and handle Responses; it controls the flow and knows what order the API calls must be made in.

module Lcbo
  require_relative 'product_request'
  require_relative 'product_response'
  class Products
    attr_accessor :key
    def initialize(key)
      @key = key
    end
    def fetch(product_id)
      connection = HTTP
      product_response = ProductRequest.new(key, product_id, connection).response
      fail LcboError, product_response.error_message unless product_response.success?
      product_response.product
    end
  end
end

Each Request class knows how to make a single request to the API (basically one endpoint). It constructs the HTTP request, filling in the Header details and any other information that needs to be included in that request. This is also potentially where you could introduce caching if it notices you have a local version of the response already on hand. It will respond with a Response object.

module Lcbo
  class ProductRequest
    attr_reader :key, :product_id, :connection
    def initialize(key, product_id, connection)
      @key = key
      @product_id = product_id
      @connection = connection
    end
    def response
      http_response = connection
        .headers('Authorization' => "Token #{key}")
        .get(url)
      ProductResponse.new(http_response)
    end
    def url
      "https://lcboapi.com/products/#{product_id}"
    end
  end
end

The Response object knows how to deal with the response from a single API endpoint. It can return specific details about the response and/or construct other objects from that response.

module Lcbo
  class ProductResponse
    attr_reader :http_response
    DEFAULT_ERROR_MESSAGE = 'There was an error retrieving product details.'.freeze
    def initialize(http_response)
      @http_response = http_response
    end
    def success?
      http_response.status == 200
    end
    def error_message
      data.fetch('message', DEFAULT_ERROR_MESSAGE)
    end
    def product
      Product.new(data.fetch('result'))
    end
    private
    def data
      http_response.parse(:json)
    end
  end
end

The last class we'll take a look at is the Lcbo::Product class. It's constructed by the ProductResponse and represents a single product in the LCBO API.

module Lcbo
  class Product
    attr_accessor :details
    def initialize(details)
      @details = details
    end
    def name
      details['name']
    end
    def tags
      details.fetch('tags', '').split(' ')
    end
  end
end

Programming is easy when everything works first try (hint: never happens). APIs can be tricky because the HTTP requests may be expected to be formatted in a very specific way, with this header or that body. It can be tricky just by looking at the Ruby code to figure out what the final HTTP request (and response) actually looks like.

httplog is a nice gem that monkey patches most of the main HTTP libraries to be able to log incoming and outgoing traffic to the console ($stdout or wherever you want). The information it provides may help you realize you spelled a header wrong or that something was missing.

An example of what it looks like:

D, [2016-09-11T22:05:11.353063 #5345] DEBUG -- : [httplog] Sending: GET https://lcboapi.com/products/438457
D, [2016-09-11T22:05:11.353158 #5345] DEBUG -- : [httplog] Header: Authorization: Token MY_API_KEY
D, [2016-09-11T22:05:11.353184 #5345] DEBUG -- : [httplog] Header: Connection: close
D, [2016-09-11T22:05:11.353202 #5345] DEBUG -- : [httplog] Header: Host: lcboapi.com
D, [2016-09-11T22:05:11.353234 #5345] DEBUG -- : [httplog] Header: User-Agent: http.rb/2.0.3
D, [2016-09-11T22:05:11.353268 #5345] DEBUG -- : [httplog] Data:
D, [2016-09-11T22:05:11.353320 #5345] DEBUG -- : [httplog] Connecting: lcboapi.com:443
D, [2016-09-11T22:05:11.502537 #5345] DEBUG -- : [httplog] Status: 200
D, [2016-09-11T22:05:11.502658 #5345] DEBUG -- : [httplog] Benchmark: 0.1491872170008719 seconds
D, [2016-09-11T22:05:11.502815 #5345] DEBUG -- : [httplog] Header: Server: nginx/1.6.2
etc...

An even more powerful tool is to use a proxy server to spy on your outgoing requests and their incoming responses. mitmproxy is a fantastic tool for gaining insight into the HTTP traffic your app produces (or your phone, or your computer).

To use mitmproxy, you'll first need to install it and then download a certificate pem file. Once you have that, you can start your Rails server or run a Ruby script and set an ENV var to point to your custom SSL certificate.

• Rails app: SSL_CERT_FILE=/Users/leighhalliday/mitmproxy-ca-cert.pem bundle exec rails s -p 3000

• Rails console: SSL_CERT_FILE=/Users/leighhalliday/mitmproxy-ca-cert.pem bundle exec rails c

• Ruby script: SSL_CERT_FILE=/Users/leighhalliday/mitmproxy-ca-cert.pem ruby demo.rb

Ensure mitmproxy is running on localhost port 8080 (default) by starting it with the command mitmproxy.

Once that is done, we can tell our HTTP library to route its traffic through mitmproxy, and then watch the information come in. mitmproxy allows you to filter the requests based on patterns, retry requests, and export them to a file to be shared or viewed later. It reminds me a little bit of the Chrome/Firefox Network inspector tab.

With the HTTP library, you will need to change this line: connect = HTTP to connect = HTTP.via('localhost', 8080)

Coming up with your own abstractions for communicating with APIs or microservices can be very valuable. It enables easier testing, encapsulates the low-level details, and provides reusable pieces of code. Also take some time to give httplogger and mitmproxy a try, in order to gain more insight into what the requests and responses actually look like.

I'd also like to thank @nwjsmith for introducing me to some of the libraries, techniques, and tools discussed in this article.