An API doesn’t exist on its own. There are always two parties involved: The Client and the Server.
In Rails, our apps are often the ones acting as the server, and we typically know how to troubleshoot the issues that inevitably arise. We can tail the logs to see what the incoming request looks like (the path, headers, params), how we responded, etc… but with backend development going more and more to a microservices architecture, our Rails apps are going to more often have to act as the client in addition to the server.
More likely than not, being the client is going to mean having to make HTTP requests and parse JSON responses. More specifically, this means crafting the correct URL, HTTP headers for authentication, pagination, response format, and finally, the body of the request which is most likely either as form data (application/x-www-form-urlencoded) or as JSON.
Thankfully most of the more popular APIs, such as Twitter, Slack, and Stripe, already have client libraries written for them that encapsulate (hide) the actual lower-level HTTP request building and response parsing.
But when you have to connect to your internal services or lesser known APIs, which may not have a client library already built for them, well, we’re on our own. The rest of this article will explore some of the different options available in Ruby for dealing with HTTP and some techniques for organizing the code and troubleshooting when things go wrong or you want a bit more visibility into what is happening.
Which HTTP Clients Are Available in Ruby?
There are quite a few HTTP libraries available in the Ruby ecosystem for making HTTP requests, so I won’t be able to touch on them all. A few of the more popular ones are:
- Net::HTTP: Part of the Ruby standard library. While it can accomplish everything you need it to, there is probably a reason why so many third-party HTTP libraries exist. Personally I don’t find it simple to work with and tend to opt for one of the other options in this list.
- Curb: This gem provides bindings for the
libcurlprogram (the same one you use from the command line with thecurlcommand). It’s fast, mostly because a lot of the heavy lifting is being done outside of Ruby, and thelibcurlprogram is fast. - HTTParty: A very popular library (22 million downloads) that wraps the
Net::HTTPcode, providing an easier API to work with. - HTTP: While not as popular as the other libraries, this one has been a favourite of mine recently when connecting to APIs. It provides a chainable interface for building HTTP requests and provides support for all of the usual functionality you’d expect from an HTTP library.
- Excon: Another very popular library (25 million downloads) written in pure Ruby. It has a clean API and is easy to use.
I won’t show an example here of how to use each one of these libraries; you’ll be able to find a simple one on the home page of each of them. In the next section, we’ll be working through an example with the HTTP library and how we can encapsulate and organize our code.
Wrapping the HTTP client
It’s best to provide some sort of interface for interacting with these API endpoints or services and hiding all of the lower-level HTTP request and response details. Other programming shouldn’t need to be exposed to the specifics of the implementation, and that same implementation should be able to be changed/refactored without the need to change the public API.
Let’s take a look at how we might do this for an API. We’ll use the lcboapi.com API. It provides a nice little interface to access information relating to the beverages and stores of the LCBO (the governmental corporation in Ontario, Canada, responsible for the retail and distribution of alcohol for the province).
We’ll be working with four classes:
Lcbo::Products: The public interface we’ll be working with to fetch details about a specific product.Lcbo::ProductRequest: Handles the HTTP request for a given Product ID.Lcbo::ProductResponse: Handles the HTTP response and knows how to build anLcbo::Product.Lcbo::Product: The actual details of a product we fetch.
Before we dive into the implementation, let’s take a look at how to use it:
require_relative 'lib/lcbo'
key = ENV.fetch('LCBO_API_KEY')
product = Lcbo::Products.new(key).fetch(438457)
puts product.name
# Hopsta La Vista
puts product.tags.inspect
# ["hopsta", "la", "vista", "beer", "ale", "canada", "ontario", "longslice", "brewery", "inc", "can"]The Controller
This class acts as the public interface to fetch details about one or more LCBO Products. At this point, I’ve only implemented the fetch method, which will return the details for a single product. This class’ job is to build Requests and handle Responses; it controls the flow and knows what order the API calls must be made in.
module Lcbo
require_relative 'product_request'
require_relative 'product_response'
class Products
attr_accessor :key
def initialize(key)
@key = key
end
def fetch(product_id)
connection = HTTP
product_response = ProductRequest.new(key, product_id, connection).response
fail LcboError, product_response.error_message unless product_response.success?
product_response.product
end
end
endThe Request
Each Request class knows how to make a single request to the API (basically one endpoint). It constructs the HTTP request, filling in the Header details and any other information that needs to be included in that request. This is also potentially where you could introduce caching if it notices you have a local version of the response already on hand. It will respond with a Response object.
module Lcbo
class ProductRequest
attr_reader :key, :product_id, :connection
def initialize(key, product_id, connection)
@key = key
@product_id = product_id
@connection = connection
end
def response
http_response = connection
.headers('Authorization' => "Token #{key}")
.get(url)
ProductResponse.new(http_response)
end
def url
"https://lcboapi.com/products/#{product_id}"
end
end
endThe Response
The Response object knows how to deal with the response from a single API endpoint. It can return specific details about the response and/or construct other objects from that response.
module Lcbo
class ProductResponse
attr_reader :http_response
DEFAULT_ERROR_MESSAGE = 'There was an error retrieving product details.'.freeze
def initialize(http_response)
@http_response = http_response
end
def success?
http_response.status == 200
end
def error_message
data.fetch('message', DEFAULT_ERROR_MESSAGE)
end
def product
Product.new(data.fetch('result'))
end
private
def data
http_response.parse(:json)
end
end
endThe Product Class
The last class we’ll take a look at is the Lcbo::Product class. It’s constructed by the ProductResponse and represents a single product in the LCBO API.
module Lcbo
class Product
attr_accessor :details
def initialize(details)
@details = details
end
def name
details['name']
end
def tags
details.fetch('tags', '').split(' ')
end
end
endLogging Outgoing Traffic
Programming is easy when everything works first try (hint: never happens). APIs can be tricky because the HTTP requests may be expected to be formatted in a very specific way, with this header or that body. It can be tricky just by looking at the Ruby code to figure out what the final HTTP request (and response) actually looks like.
With httplog
httplog is a nice gem that monkey patches most of the main HTTP libraries to be able to log incoming and outgoing traffic to the console ($stdout or wherever you want). The information it provides may help you realize you spelled a header wrong or that something was missing.
An example of what it looks like:
D, [2016-09-11T22:05:11.353063 #5345] DEBUG -- : [httplog] Sending: GET https://lcboapi.com/products/438457 D, [2016-09-11T22:05:11.353158 #5345] DEBUG -- : [httplog] Header: Authorization: Token MY_API_KEY D, [2016-09-11T22:05:11.353184 #5345] DEBUG -- : [httplog] Header: Connection: close D, [2016-09-11T22:05:11.353202 #5345] DEBUG -- : [httplog] Header: Host: lcboapi.com D, [2016-09-11T22:05:11.353234 #5345] DEBUG -- : [httplog] Header: User-Agent: http.rb/2.0.3 D, [2016-09-11T22:05:11.353268 #5345] DEBUG -- : [httplog] Data: D, [2016-09-11T22:05:11.353320 #5345] DEBUG -- : [httplog] Connecting: lcboapi.com:443 D, [2016-09-11T22:05:11.502537 #5345] DEBUG -- : [httplog] Status: 200 D, [2016-09-11T22:05:11.502658 #5345] DEBUG -- : [httplog] Benchmark: 0.1491872170008719 seconds D, [2016-09-11T22:05:11.502815 #5345] DEBUG -- : [httplog] Header: Server: nginx/1.6.2 etc...
With mitmproxy
An even more powerful tool is to use a proxy server to spy on your outgoing requests and their incoming responses. mitmproxy is a fantastic tool for gaining insight into the HTTP traffic your app produces (or your phone, or your computer).
To use mitmproxy, you’ll first need to install it and then download a certificate pem file. Once you have that, you can start your Rails server or run a Ruby script and set an ENV var to point to your custom SSL certificate.
- Rails app:
SSL_CERT_FILE=/Users/leighhalliday/mitmproxy-ca-cert.pem bundle exec rails s -p 3000 - Rails console:
SSL_CERT_FILE=/Users/leighhalliday/mitmproxy-ca-cert.pem bundle exec rails c - Ruby script:
SSL_CERT_FILE=/Users/leighhalliday/mitmproxy-ca-cert.pem ruby demo.rb
Ensure mitmproxy is running on localhost port 8080 (default) by starting it with the command mitmproxy.
Once that is done, we can tell our HTTP library to route its traffic through mitmproxy, and then watch the information come in. mitmproxy allows you to filter the requests based on patterns, retry requests, and export them to a file to be shared or viewed later. It reminds me a little bit of the Chrome/Firefox Network inspector tab.
With the HTTP library, you will need to change this line: connect = HTTP to connect = HTTP.via('localhost', 8080)
Conclusion
Coming up with your own abstractions for communicating with APIs or microservices can be very valuable. It enables easier testing, encapsulates the low-level details, and provides reusable pieces of code. Also take some time to give httplogger and mitmproxy a try, in order to gain more insight into what the requests and responses actually look like.
I’d also like to thank @nwjsmith for introducing me to some of the libraries, techniques, and tools discussed in this article.
| Reference: | HTTP Calls in Ruby from our WCG partner Florian Motlik at the Codeship Blog blog. |
