How To Build A Web App, part 19 of ?: API Act ID filtering working at last
This is the nineteenth in a series of articles taking you through all the actual steps in building a web app. If you’re an aspiring developer, if mucking around with teensy beginner tutorials frustrates you, if you’d love to build a properly substantial app that does fab things, these articles are for you.
Word to the wise: these tutorials don’t depict a polished, pristine workflow. You won’t find “follow directions A, B, C” taking you in a perfectly straight line. Nah. These tutorials depict mess and chaos and grit. They meander. They show what real web-dev is actually like. It’s a standard trait of professionals that you should make your trade look easy. Not here, baby. If I’ve written these articles properly, beginner-to-intermediate devs will read them and think “Oh thank Christ, turns out these self-proclaimed Senior Developers struggle just as much as me! My impostor syndrome is just a syndrome!”
Last time, we decided we’d abandon the absurd tyranny of using include() in our tests instead of straight-up true-blue hand-on-heart eq: simply testing that our response JSON include-ed various items of demo Venue json utterly failed to reveal the fact that our API response returned 13 Venues, not five. 13! Geez. After getting to the bottom of, and fixing, that bug, We decided we’d refactor our tests to detecting the entirety of our API response’s Venue JSON: five Venues, ordered by updated_at, descending.
Then we ran into a few (more) hiccups. Turned out jsonapi-resources was … let’s say, incomplete. It claims it can serialize its resources. It cannot. At least the latest version can’t. I had to bump its Gemfile version down to 0.9.11 to get it working.
Finally, to round off our hilarious fiasco, I discovered I’d forgotten to add t.timestamps to all tables.
But that is now sorted and we can continue.
After much headbutting, I made these modifications to our “Not supplying any act IDs” test:
# spec/requests/search_spec.rb...describe 'Filtering by Act' do
include_examples 'general jsonapi behaviour for', Venue context 'Not supplying any act IDs' do
let(:params) {
{ filter: { acts: '' } }
} it 'returns every single venue, ordered by updated_at desc' do
resources = [venue4, venue2, venue3, venue5, venue1]
.map do |venue|
Api::V1::VenueResource.new(venue, nil)
end serialized_resources = JSONAPI::ResourceSerializer
.new(Api::V1::VenueResource)
.serialize_to_hash(resources) expect(response_json).to eq serialized_resources
end
end ...end
Two things may jump out at you. First, :params. I’d changed
let(:params) { {} }to
let(:params) {
{ filter: { acts: '3,4,5' } }
}Nothing too fancy, it just sets our request’s query params; it ensures our HTTP request’s exact URL is this: GET http://localhost:3000/api/v1/venues?filter[acts]=3,4,5.
The other things is the actual meat of the test. Neat, right? Two bits: first, generate an array of resources from the array of Venue models we’re testing. List them ordered by updated_at, descending:
resources = [venue4, venue2, venue3, venue5, venue1].map do |venue|
Api::V1::VenueResource.new(venue, nil)
endThen serialize, then test:
serialized_resources = JSONAPI::ResourceSerializer
.new(Api::V1::VenueResource)
.serialize_to_hash(resources)expect(response_json).to eq serialized_resources
Awesome. Let’s test it. Testing:
$ rspec spec/requests/search:spec.rb:58
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[58]}}
self_link for Api::V1::GigResource could not be generated
self_link for Api::V1::GigResource.venue(BelongsToOne) could not be generated
self_link for Api::V1::GigResource.act(BelongsToOne) could not be generated
FFailures:
1) Search Filtering by Act Not supplying any act IDs returns every single venue, ordered by updated_at desc
Failure/Error: expect(response_json).to eq serialized_resources expected: {"data"=>[something far too bloody big to fit here]}
got: {"data"=>[something else that is also far too bloody big to fit here]} (compared using ==) Diff:
...
+"links" => {"..."}
Failed? … Ohh, the complete response body has extra fiddly bits: links and so on. I’m not feeling overly strict about testing every last iota of the entire request body: we’re testing only those five Venues, not their extra metadata. Only the JSON-API response’s "data" key.
In that case, change
expect(response_json).to eq serialized_resourcesto
expect(response_json['data']).to eq serialized_resources['data']…and test again:
$ rspec spec/requests/search_spec.rb:58
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[58]}}FFailures: 1) Search Filtering by Act not supplying any act IDs returns eery single venue, ordered by updated_at desc
Failure/Error: expect(response_json['data']).to eq serialized_resources['data'] expected: nil
got: [{"attributes"=>{"created-at"=>[blargh] }]
(compared using ==)
Wait, what? Now what’s wrong? What’s this expected: nil business? Why would it expect nil? Is it talking about serialized_resources? That’s not nil! What gives?
Remember Byebug? I decided it was high time to inspect serialized_resources. I fired it up and jumped in.
$ rspec spec/requests/search_spec.rb:58
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[58]}}[74, 83] in /Users/michaelclarke/Work/gigs/spec/requests/search_spec.rb
74: serialized_resources = JSONAPI::ResourcesSerializer
75: .new(Api::V1::VenueResource)
76: .serialize_to_hash(resources)
77: byebug
78: expect(response_json['data']).to eq serialized_resources['data']
79: end
80: end
81:
82: context "Acts 1, 2, 5" do
(byebug) pp serialized_resources
{:data=>
[{"id"=>"4",
"type"=>"venues",
"links"=>"{"self"=>"/api/v1/venues/4"},
"attributes"->
{"created-at"=>"2020-02-29T21:04:21.406Z",
"updated-at"=>"2020-05-23T08:44:53.052Z",
"name"=>TT Eatery"},
"relationships"=>
{"gigs"=>
{"links"=>
{"self"=>"/api/v1/venues/4/relationships/gigs",
"related"=>"/api/v1/venues/4/gigs",
:data=>[{:type=>"gigs", :id=>"10"}, {:type=>"gigs", :id=>"11"}]}}},
{"id"="2",
...
Aha. There’s the problem. Can you spot it?
That first key. It’s :data. A symbol. It should be "data". A string. And :data repeats again on that second-to-last line! Its array’s keys are also symbols. Every key in the response’s entire JSON structure should be strings. And they’re not.
Why not, you may ask? That … is a really good question. It’s obviously another bug, though this one is far smaller and more benign and doesn’t tempt me to ragequit. If you consult jsonapi-resources’ Serializer docs (and pay close attention to that URL’s version, we’d fallen back to “v0.9”, remember, not “v0.10”), take a look at its example demo JSON structure. Its keys are all strings and no symbols.
Well okay then. We shall have to just simply whack our own patch on too. We’ll manually change serialized_resources' keys to strings.
Modifying large and complicated data structures like this can be a horrendous pain in the ass. But! The Rails devs have foreseen this. They have added all kinds of fascinating doohickeys to the Ruby language, all packaged up in a single library. It’s called ActiveSupport. It’s all kinds of awesome. One of its many methods is #deep_stringify_keys!. It does exactly what it says on the tin: converts any big honkin’ hash’s keys into strings, in-place. Including any child hashes within. And their child hashes. Deep as you please. Hence “deep”.
(Here’s another Thing: you may be wondering why some Ruby method names have an exclamation mark and others don’t. Here’s the deal. It’s a Ruby convention. A method’s exclamation mark’s presence or absence denotes whether a given method is quote-unquote “destructive”, or dangerous in some way. (Ultra-l33t devs like you and me call a ! a “bang”, so we’d pronounce deep_stringify_keys! “deep-stringify-keys-bang”). Exactly what “destructive” means varies ginormously depending on context: here, deep_stringify_keys calculates and returns an entirely new copy of the hash you’ve called it on, whereasdeep_stringify_keys! instead modifies the existing object, in-place: you’re overwriting your existing data! Similarly, ActiveRecord models have the methods save and save! — if a model is invalid, then save returns false, and save! throws an exception.)
But anyway. Behold!
it "returns every single venue, ordered by updated_at desc" do
resources = [venue4, venue2, venue3, venue5, venue1].map do |venue|
Api::V1::VenueResource.new(venue, nil)
end serialized_resources = JSONAPI::ResourceSerializer
.new(Api::V1::VenueResource)
.serialize_to_hash(resources)
.deep_serialize_keys! expect(response_json['data']).to eq serialized_resources['data']
end
And test:
$ rspec spec/requests/search_spec.rb:79
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[79]}}
.Finished in 0.61068 seconds (files took 2.88 seconds to load)
1 example, 0 failures
Is that not a sight to warm the cockles of your heart?
We have our completed test. Let’s commit and diff that.
Resume testing Act ID filtering
Okay, where were we? Testing our Act ID filtering, that’s where. Onward!
Our first search tests were, all along, supposed to test what the API returns when we supply it specific Act IDs: return only Venues and Gigs performed by the Acts we’d supplied. Right.
I had a go at rewriting the “Acts 1, 2, 5” test thusly:
context "Acts 1, 2, 5" do let(:params) {
{ filter: { acts: [act1.id, act2.id, act5.id].join(',') } }
} describe "returning only venues 4, 2, 3: updated_at desc" do subject {
vr_4_2_3 = [venue4, venue2, venue3].map do |venue|
Api::V1::VenueResource.new(venue, nil)
end
JSONAPI::ResourceSerializer.new(Api::V1::VenueResource)
.serialize_to_hash(vr_4_2_3)
.deep_stringify_keys!['data']
} it { is_expected.to eq response_json['data'] }
end ...end
A return to subject { ... } syntax! I rather like it. The it { is_expected_to eq 'blargh' } syntax tickles me just right.
Let us test:
$ rspec spec/requests/search_spec.rb:101
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[101]}}
FFailures:
1) Search Filtering by Act Acts 1, 2, 5 returning only venues 4, 2, 3: updated_at desc should eq {"errors"=>[{"code"=>"500", "detail"=>"Internal Server Error", "meta"=>{"backtrace"=>["/Users/michael...ues\".\"updated_at\" DESC LIMIT $1 OFFSET $2"}, "status"=>"500", "title"=>"Internal Server Error"}]}
Failure/Error: it { is_expected.to eq response_json }[ginormous JSON stack trace]
Whoa. What happened this time? Why return a 500 Internal Server Error? A 500 error isn’t just any regular old test failure — it means there’s some kind of programming error inside the server that made it fall over.
A quick skip and a jump into the [ginormous JSON stack trace] set us straight. The response JSON’s errors.0.meta.exception key equals this:
PG::UndefinedTable: ERROR: missing FROM-clause entry for table \"acts\"\nLINE 1: SELECT \"venues\".* FROM \"venues\" WHERE (acts.id IN (1,2,5)) ...\n ^\n: SELECT \"venues\".* from \"venues\" WHERE (acts.id IN (1,2,5)) ORDER BY \"venues\".\"updated_at\" DESC LIMIT $1 OFFSET $2Iiiiinteresting. Malformed SQL. You may recall our earlier chats about ActiveRecord, back in article number … um, umm … number …
…Huh. Well I’ll be. I’d just assumed that when I’d started banging out my first actual Rails code for your discernment and delectation in article 13, then naturally I’d have explained the core basics of ActiveRecord. But a perusal tells me that I had not!
All right then. Here goes:
ActiveRecord
The whole point of Rails’s ActiveRecord library is to join together databases and programming: it wraps a single database table row with a programming Object. It links together the interactions of these Ruby objects and data with database-friendly SQL: an ActiveRecord query like v = Venue.find(1) becomes SQL’s SELECT * FROM venues WHERE id=1. v.update_attribute 'name', 'blargh' becomes UPDATE venues SET name='blargh' WHERE id=1. That kind of thing.
jsonapi-resources does something vaguely similar. Remember when we defined our per-model resource files, in article 17? The Venue resource file was of particular interest. Its filter method, and its apply argument, is the basis for generating the SQL to query our database for our sweet sweet JSON.
Just as a refresher, here’s that filter method again:
# app/resources/api/v1/venue_resource.rb...filter :acts,
verify: -> (values, context) {
values.map &:to_i
},
apply: -> (records, value, _options) {
q = records.includes(gigs: :act)
q = q.where('acts IN (:acts)', acts: value) if value.length > 0
q
}
It’s apply that does the heavy lifting. That’s what generates our buggy SQL. Here’s that SQL error message again, cleaned up a bit:
SELECT venues.* FROM venues WHERE (acts.id IN (1,2,5)) ORDER BY venues.updated_at DESC LIMIT $1 OFFSET $2;Aha. If you’ve ever written SQL manually, you can probably see what’s wrong: our query’s tables and columns, inside SELECT abc FROM xyz mentions only venues, but our filtering code, inside WHERE, mentions only acts. PostgreSQL has no way of knowing the connection between venues and acts.id.
We have to tell it. We have to use a thing called an Inner Join. If I was writing my SQL manually instead of getting ActiveRecord and jsonapi-resources to do it, I’d write something like this:
SELECT * FROM venues
INNER JOIN gigs ON gigs.venue_id = venues.id
INNER JOIN acts ON acts.id = gigs.act_idGetting déjà vu? We’ve bumped into inner joins before. On that huge bug-fix tangent I charged off on back in article 15….
…Wait. Speaking of tangents …
Quick tangent: technical debt
Time to mention another Thing! I’d not mentioned something called “technical debt”, had I? See, it’s entirely possible you’d been wondering the following:
“Wait wait just a cotton pickin’ minute — all this new jsonapi-resources query/search code —it just looks like we’re re-implementing our SearchController code in
app/controllers/api/v1/search_controller.rbwith this new stuff? What? We’re doing the same job again? WHY?
Yup. Welcome to the messy complexities of real-world dev. We’d written all this API::V1::SearchController code before encountering the glories of jsonapi-resources. Turns out modifying the latter makes for a better program than creating the former from scratch! After we’ve got our Act ID tests into a nice state, I’ll delete the search controller code, and clean up any references to it elsewhere in the app.
“Clean Up” is the appropriate phrase here. Some devs don’t. Some managers don’t. Some companies have a “well the app works, doesn’t it? What’s the problem?” attitude to the health and hygiene of their apps’ code bases. They either don’t know or don’t care that more often than not, their code is a horrid apocalyptic abyss of spaghetti and Franken-code and a zillion other abominations. In future articles I’ll go into more detail about this; but for now, yeah, I’m acknowledging that we’re re-implementing our search controller, and we’ll delete it later.
Tangent over; return to Act ID testing
Sooo what jsonapi-resources filter code should we write, to shove the right INNER JOIN SQL-code into our SQL? Same as last time: we use a Thing called a join. Simply change
# app/resources/api/v1/venue_resource.rb...q = records.includes(gigs: :act)
to
# app/resources/api/v1/venue_resource.rb...q = records.includes(gigs: :act).joins(gigs: :act)
And test again!
$ rspec spec/requests/search_spec.rb:101
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[101]}}
FFailures: 1) Search Filtering by Act Acts 1, 2, 5 returning only venues 4, 2, 3: updated_at desc should eq [{"attributes"=>{"created-at"=>"2020-03-13T17:41:05.251Z", "name"=>"Orange Pizza", "updated-at"=>"202...ated"=>"/api/v1/venues/3/gigs", "self"=>"/api/v1/venues/3/relationships/gigs"}}}, "type"=>"venues"}]
Failure/Error: it { is_expected.to eq response_json['data'] } expected: [{"attributes"=>{"created-at"=>"2020-03-13T17:41:05.251Z", "name"=>"Orange Pizza", "updated-at"=>"202...ated"=>"/api/v1/venues/3/gigs", "self"=>"/api/v1/venues/3/relationships/gigs"}}}, "type"=>"venues"}]
got: [{"attributes"=>{"created-at"=>"2020-03-13T17:41:05.251Z", "name"=>"Orange Pizza", "updated-at"=>"202...ated"=>"/api/v1/venues/3/gigs", "self"=>"/api/v1/venues/3/relationships/gigs"}}}, "type"=>"venues"}](compared using ==)Diff:
@@ -6,7 +6,7 @@
"links"=>{"self"=>"/api/v1/venues/4"},
"relationships"=>
{"gigs"=>
- {"data"=>[{"id"=>"11", "type"=>"gigs"}],
+ {"data"=>[{"id"=>"10", "type"=>"gigs"}, {"id"=>"11", "type"=>"gigs"}],
"links"=>
{"related"=>"/api/v1/venues/4/gigs",
"self"=>"/api/v1/venues/4/relationships/gigs"}}},
@@ -32,7 +32,14 @@
"links"=>{"self"=>"/api/v1/venues/3"},
"relationships"=>
{"gigs"=>
- {"data"=>[{"id"=>"3", "type"=>"gigs"}],
+ {"data"=>
+ [{"id"=>"3", "type"=>"gigs"},
+ {"id"=>"4", "type"=>"gigs"},
+ {"id"=>"5", "type"=>"gigs"},
+ {"id"=>"6", "type"=>"gigs"},
+ {"id"=>"7", "type"=>"gigs"},
+ {"id"=>"8", "type"=>"gigs"},
+ {"id"=>"9", "type"=>"gigs"}],
"links"=>
{"related"=>"/api/v1/venues/3/gigs",
"self"=>"/api/v1/venues/3/relationships/gigs"}}},# ./spec/requests/search_spec.rb:101:in `block (5 levels) in <top (required)>'Finished in 0.7243 seconds (files took 3.31 seconds to load)
1 example, 1 failureFailed examples:rspec ./spec/requests/search_spec.rb:101 # Search Filtering by Act Acts 1, 2, 5 returning only venues 4, 2, 3: updated_at desc should eq [{"attributes"=>{"created-at"=>"2020-03-13T17:41:05.251Z", "name"=>"Orange Pizza", "updated-at"=>"202...ated"=>"/api/v1/venues/3/gigs", "self"=>"/api/v1/venues/3/relationships/gigs"}}}, "type"=>"venues"}]
Oh.
Dear god, now what?
Um … looks like our Venue resource JSON doesn’t have all its bits. The JSON we’re generating within our tests has all its properly attached Gig IDs for each Venue … but our API’s response JSON doesn’t.
Why? Search me. Let’s find out together. Usual story: as I’m typing this, I genuinely have no idea what’s causing this bug. Total mystery.
Isn’t a good night’s sleep wonderful?
Okay. A day passes. I revisit this apparently insoluble conjecture …
… And facepalm.
Duh. Obviously the response JSON is showing a reduced subset of each Venue’s Gigs. That’s the whole point of submitting these Act IDs: we’re asking only for Venues and Gigs performed by Acts with the IDs we’re submitting. Good lord I can be a moron sometimes.
But all right then. No-one’s perfect. Moving on. What’s our next step?
Modify subject to reflect that same Gig-Act filtering, that’s what. After some faffing, I decided this would be just super:
subject {
vr_4_2_3 = [venue4, venue2, venue3].map do |venue|
Api::V1::VenueResource.new(venue, nil)
end JSONAPI::ResourceSerializer.new(Api::V1::VenueResource)
.serialize_to_hash(vr_4_2_3)
.deep_stringify_keys!['data']
.each do |svr| # 'serialised venue resource'
svr['relationships']['gigs']['data'].select! do |sgr|
[1, 2, 5].include? Gig.find(sgr['id']).act.id
end
end
end
}
Got all that?
You can see I’ve whacked a .each loop on the end of .deep_stringify_keys!['data']. It iterates over each “svr”, serialized venue resource; jumps into its gigs data; then filters them, in-place, with Array#select!: retain only those gigs whose act’s ID is one of 1, 2 or 5.
Testing:
$ rspec spec/requests/search_spec.rb:107
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[107]}}
.Finished in 0.5988 seconds (files took 2.99 seconds to load)
1 example, 0 failures
Beautiful ❤
But … one thing bugs me. That subject { ... } syntax isn’t really all that readable. Not obvious what’s going on, is it? Very few labels. We can’t properly understand the code execution flow, or what its intentions are. Let’s do proper justice to it.
How about …
context "Acts 1, 2, 5" do let(:acts) { [act1, act2, act5] } let(:params) { { filter: { acts: acts.map(&:id).join(',') } } } describe "returning only venues 2, 4, 3: updated at desc" do let(:venueresources_4_2_3) {
[venue4, venue2, venue3].map do |v|
Api::V1::VenueResource.new(v, nil)
end
} let(:svrs_4_2_3_data) {
JSONAPI::ResourceSerializer.new(Api::V1::VenueResource)
.serialize_to_hash(venueresources_4_2_3)
.deep_stringify_keys!['data']
} let(:serialized_vr_4_2_3_data_w_gigs_for_acts_1_2_5) {
svrs_4_2_3_data.each do |svr|
svr['relationships']['gigs']['data'].select! do |sgr|
acts.include? Gig.find(sgr['id']).act
end
end
} it "returns only the venues and gigs attended by Acts 1, 2 and 5" do
expect(serialized_vr_4_2_3_data_w_gigs_for_acts_1_2_5)
.to eq response_json['data']
end
...
end
...
end
That, I think, is a little better. Two main differences here: (1) we’re letting a hell of a lot more, space out our computations with named variables, splat a lot more labelling this way and that. Though you can see I’m straining a bit on that last one: serialized_vr_4_2_3_data_w_gigs_for_acts_1_2_5 is getting a bit silly. Almost as bad as Hungarian notation. (WARNING Hungarian Notation is a desperate abyss of complexity and despair and wailing and gnashing of dreadful teeth, click on that link at your own risk.)
Oh yeah and (2), we’ve let :acts be its own separate standalone array of Act objects, so’s we can do this: acts.include? Gig.find(sgr['id']).act. Neat. Terser.
And test, just to be sure …
$ rspec spec/requests/search_spec.rb:107
Run options: include {:locations=>{"./spec/requests/search_spec.rb"=>[107]}}
.Finished in 0.91215 seconds (files took 3.19 seconds to load)
1 example, 0 failures
Beautiful-er ❤ ❤
And that, I think, will do for today! Let us commit and diff that shit. Awesome.
We’ve finally got rudimentary Act ID filtering working. Next time, let us plump out our two tests to be a bit more expansive. I’ll take you through the concept of tests as documentation. Catch you on the flip side.