How To Build A Web App, part 17 of ?: jsonapi-resources internals
This is the seventeenth in a series of articles taking you through all the actual steps in building a web app. If you’re an aspiring developer, if mucking around with teensy beginner tutorials frustrates you, if you’d love to build a properly substantial app that does fab things, these articles are for you.
Word to the wise: these tutorials don’t depict a polished, pristine workflow. You won’t find “follow directions A, B, C” taking you in a perfectly straight line. Nah. These tutorials depict mess and chaos and grit. They meander. They show what real web-dev is actually like. It’s a standard trait of professionals that you should make your trade look easy. Not here, baby. If I’ve written these articles properly, beginner-to-intermediate devs will read them and think “Oh thank Christ, turns out these self-proclaimed Senior Developers struggle just as much as me! My impostor syndrome is just a syndrome!”
Last time, we continued our merry process of adding timestamps into our API: should you submit one or both start/end timestamps, our Venues Search code will return only venue data with at least one gig after start, or before end. Then, whilst adding a column at to the Gigs table, I discovered a horrid horrid JSON-response bug, and spent 27 years squelching. Then I thought it’d be hilarious to charge off on a tangent, and rewrite our entire API’s response-data structure from the ground up, to conform to a data standard called JSON-API.
Today, I shall continue this tangent, by filling in those gaps by adding jsonapi-resource resource-tests, then resume messing around with our API responses and its data structures.
Simple, right? Right. Let’s get cracking, we gots shit to code.
jsonapi-resource Resources
Remember them? A “Resource”, in this context, is a file that jsonapi-resource uses to interface with ActiveRecord.
At face value, that should be quite simple — though if you peruse that link, you can see its Resource doc is rather a sprawling beastie. It implements all kinds of nonsense, for JSON-API is huge. I can assure you right this second we’re not going to do the whole lot! Just the basics. Then test them.
Which basics specifically, you ask? Let’s see … skim skim skim docs docs docs … there’s a few things I would like to implement: attributes, fetchable fields, namespacing, relationships, and filtering. These shall serve nicely in returning that sample data structure I’d described in article 16.
What are these things, though? I shall take you through them, and use VenuesResource as a nice handy example. I should point out that yes, most or even all of that will overlap with the work we’ve already done on our VenuesController. Which I will probably throw out in favour of this stuff! But such is the web-dev lifecycle. It’ll simplify our code tremendously. Happens all the time. It’s all good.
Hm … now that I examine these docs more closely, I can see my two disparate goals, of (1) writing a few bog-standard resource files plus accompanying tests, and (2) using jsonapi-resource’s functionality to return handy Venue search data, turn out to be more interconnected than I’d thought.
Here’s why. With jsonapi-resources, to implement some rudimentary searching, all you need is this:
# config/routes.rb (hypothetical example only)Rails.application.routes.draw do
...
jsonapi_resources :venues
...
end# app/controllers/venues_controller.rb (hypothetical example only)class VenuesController < JSONAPI::ResourceController
end
And that’s it! Adding jsonapi_resources :venues creates your classic CRUD routes for :venues, and adding a controller extending off JSONAPI::ResourceController inherits its corresponding controller methods for those routes. Then you make requests like GET localhost:3000/venues?filter[attr]=value, and jsonapi-resources will do the rest.
Yes. I’ll be the first to admit all this is a bit smooshed-together. We’re juggling a lot of concepts all at once. Tell you what, I’ll throw together a completed example, for Venues, then take you through it.
Here you go:
# config/routes.rb (actual)Rails.application.routes.draw do
...
namespace :api do
namespace :v1 do
jsonapi_resources :venues
end
end
...
end
# app/controllers/application_controller.rbclass ApplicationController < ActionController::API
include JSONAPI::ActsAsResourceController
end
# app/controllers/api/v1/venues_controller.rbmodule Api
module V1
class VenuesController < ApplicationController
end
end
end
And the meat!
# app/resources/api/v1/venue_resource.rbmodule Api
module V1
class VenueResource < JSONAPI::Resource
attributes :name, :ticketmaster_id
has_many :gigs def fetchable_fields
super - [:ticketmaster_id]
end filter :acts,
verify: -> (values, context) {
values.map(&:to_i)
},
apply: -> (records, value, _options) {
records.includes(gigs: :act).where('acts.id IN (?)', value)
}
end
end
end
The first three files don’t really require much explanation: I’m still using the api/v1 namespacing because it’s convenient and groovy for backwards compatibility. The only proper difference between my first hypothetical example and this actual code is the controller: instead of the VenueController extending a JSONAPI::* controller, I’m still sticking with extending the standard Rails ApplicationController, and simply include-ing the JSONAPI stuff into it. That way, every standard Rails controller will gain access to it. It’s possible we’ll have future Rails controllers that don’t require jsonapi-resource functionality, but that’s far-future content, with logged-in users, and authentication. Way distant.
Now. VenueResource. Lots of things going on! First, you’ll see we’ve namespaced it too, like the others. Nothing fiddly.
jsonapi-resources requires us to manually list a resource’s accessible attributes, so we’ve done that. Any model’s attribute we wish jsonapi-resources to never ever go near, we just simply don’t mention.
Same with relationships. We must list them manually. Done.
Fetchable attributes/fields is a teensy bit different. This is how we list which fields our API should return. And luckily, we already have a existing example! I can’t think of a single reason why our client-app-UI-to-be should ever know about our models’ Ticketmaster IDs. They exist in our database only to uniquely identify which of our records have corresponding records inside Ticketmaster’s database. The only time we ever use them is when we’re hitting Ticketmaster’s API from our server. And never the client. So that means we can safely and happily remove :ticketmaster_id from our models’ fetchable attributes.
You may be wondering what that super word means. It’s an object-oriented programming Thing. Read this for full details, but long story short, whenever you use it inside any method, it’s a call to the overrided method of the same name belonging to that class’s superclass. See how every resource class extends JSONAPI::Resource? That class also extends JSONAPI::BasicResource, which has a fetchable_fields method. Out of morbid curiosity, I tracked down that method, which, yes, simply rattles off every single field you’ve assigned to your resource. The idea of this super usage and overriding is, its default behaviour is to rattle off all fields, but if you change your mind, you can manually specify different fields. Done.
Now, the last bit. What’s this filter business? I’ll repeat it here so’s we can zoom in on it more closely:
filter :acts,
verify: -> (values, context) {
values.map(&:to_i)
},
apply: -> (records, value, _options) {
records.includes(gigs: :act).where('acts.id IN (?)', value)
}What’s that about?
Three bits. First up, filter :acts. The idea is, routes, controllers, and filters work together to let us do this: GET /api/v1/venues?filter[acts]=?, and our web server will squirt back Venue JSON. You can add any filter you like to any attribute: filter :blargh lets you do GET /api/v1/venue?filter[blargh]=value, which then returns all JSON for Venues where their ‘blargh’ attribute equals ‘value’. Right.
Two:verify: is a sanitiser. It performs a bit of polishing-up to ensure that whatever value that our incoming filter[acts] HTTP query parameter supplies is indeed a list of queryable Act IDs. Integers.
And three: apply: performs the actual filtering query code: ActiveRecord joins and where and suchlike.
Thing Nine Zillion: off the top of my head, I don’t believe we’ve encountered Ruby’s -> (foo, bar) { } syntax yet. It’s called a Lambda. Usual story: I won’t charge off on too much of a tangent, read this for more, but a lambda is basically a self-contained chunk of code you can sling around from one Ruby method to another, and run at your leisure.
These two lambdas may look familiar! You can see the verify lambda is calling to_i on our incoming act IDs, like our existing VenuesController#massage_stringly_typed_params! method, and the apply lambda is calling joins and where. Just like our former VenuesController#index method.
Anyhoo. The point I’m making in the here-and-now is, yes I would like to replace #massage_stringly_typed_params! and #index with verify and apply respectively.
Does that mean we’ll be throwing out VenuesController#validate_params!? Honestly, maybe! And does that mean we’ll abandon JsonSchema’s hoity-toity validation majicks there? Maybe also! The whole reason I wanted to use it in the first place was to be really properly specific about exactly what inputs and fiddly-about-y bits our API should receive, and I would like to continue that with jsonapi-resources.
Our next topic, therefore, will be to investigate how jsonapi-resources handles crap inputs. And validate.
And is this article getting just a tad long and meandering? Yup it absolutely is. Welcome to web-dev, baby. Totally normal to go through various and multiple iterations of your code base and your feature set as you learn more and more about the properties and strong points and limitations of your feature set and your libraries. Like, I’ve only just discovered right this second that it might be an idea to totally ditch JSON-API in the uncomfortably near future. Seriously.
That said … after a few more seconds, I’ve just read through that article, and darn and gosh and lawks, I think I miiight genuinely be about to squirt off a hilariously dramatic “your privilege is showing” at the author. Er. Ah. Here’s why. No disrespect to Phil, the author himself, by the way, though now I’ve read his article, it turns out Phil’s beef with JSON-API is that the entire project was supposed to be a reaction against HTTP requests being expensive, so whenever our API customers request great sprawling data-structures, zillions of models, we should therefore wodge all these models into as few requests as possible.
(By the way, if you’re unfamiliar with “expensive”, it’s a catch-all compsci term for any kind of technical-fiddly-about-y operation that gobbles lots of computing processing cycles and takes ages. Not necessarily money. Read this for more.)
So HTTP. Expensive? The gist of Phil’s article is that a single HTTP request gobbles up quite a lot of network-router processing power:
The main point here is that HTTP/1.0 forces you to handle DNS, HTTP handshakes, SSL negotiation, etc., on every single call, and if your homepage wants 10 things from the API then it’s doing all of that junk 10 times… HTTP/1.1 actually solved this with Connection: keep-alive, but not many people seem to use that.
And fair enough.
Then…
HTTP/2 not only maintains a single connection per domain for reuse, but these calls can be handled asynchronously. You could requests all the A’s, then the callback for that async request asks for some B’s as and when the A’s respond. And C and D.
If you can use Server Push then you don’t have to wait for the A’s to respond before you can request the B’s. The server will just start sending you the B’s!
Seriously, HTTP/2 calling is fast.
Hm! I suspect Phil lives in the USA. I, on the other hand, live in New Zealand. It’s one of the most remote countries in the world. An HTTP request (of either 1 or 2) originating in New Zealand usually doesn’t have its destination there (though glorious exceptions abound). Most requests zip across half the globe. And we hit the fundamental speed limits of the universe: the speed of light. No sparkling assemblage of techy-structures can ever reduce the theoretical minimum time for a photon to cross the globe:
20,040km (half the world’s circumference (at the equator (the world’s an oblate spheroid, not a sphere, long story))) / 299,792km/s (speed of light in a vacuum) == ~1/15 second.
A fifteenth of a second is the theoretical minimum traversal time for every HTTP request going halfway round the world. Unless you go straight down and through the Earth’s core with a neutrino gun or something. Bumping the request’s version from 1 to 2 won’t drop that time. Now add the inevitable increases due to DNS server packet-switching, physical zigzagging of undersea cables, all kinds of goodies. In practice, I’ve found global ping can be a good half a second. My peculiar circumstances means my local bottlenecks are IP packet traversal time, and not the power consumption of the Internet’s servers.
Soooo let us grubby Kiwi Styxians therefore embrace HTTP1, and entwine ourselves with JSON-API, and cock a snoot to the absurd tyranny of HTTP2.
Nah, I’m only kidding really. In practice, in general, I think HTTP2 is a fantastic upgrade (read Phil’s article for the in-depth reasons why), and it’s only in Hadean armpits of the world like New Zealand that its assumptions may not totally hold. Why my unconscionable presumption re “your privilege is showing”, then? Eh, no particular reason, it just amused me. I mean sure, it’s technically true, he’s enjoying ping values I can only dream about, and it’s making me green with envy. That technically constitutes “privilege”. Or maybe it’s because violence is drama, drama is entertaining, and entertainment is marketable.
Phil, if you’re reading this, then if you like, you can totally let me have it in one of those vicious tweet-wars I hear today’s cool kids routinely indulge in. Bring it on, baby.
…Besides, actually, just realised, in later articles I’ll take you through deploying this Gigs app to a production server … which will in all likelihood be some kind of virtual server … with its physical substrate probably located in, you guessed it, the USA. Hah. That entire tangent was kind of pointless, then, wasn’t it. But would you expect anything less from me?
Okay, where were we…?
Right. We were verifying that jsonapi-resources can replicate the input validation heavy lifting we’d already built for our search with JsonSchema.
That is a little fiddly. If you take another look at the Act ID bit of validate_params!, you’ll see the goal is to block and flag any submitted values of Act IDs that aren’t positive integers, because we then shove our sanitised Act ID values directly into an ActiveRecord call, and if it turns out those Act ID values aren’t integers, we’ll get a bad query and throw errors. (No errors will happen with negative integers, so strictly speaking the positive-integer filter isn’t necessary, but IDs on any table will always be positive integers, so I thought it’d just be nice to remind the user that negative integers are a bridge to nowhere.)
What blocking/flagging/filtering should and shouldn’t we apply to jsonapi-resources’ filter:?
Let’s sort that out next time.