How To Build A Web App, part 11 of ?: Writing our first test
This is the eleventh in a series of articles taking you through all the actual steps in building a web app. If you’re an aspiring developer, if mucking around with teensy beginner tutorials frustrates you, if you’d love to build a properly substantial app that does fab things, these articles are for you.
Last time, we got our app’s testing environment set up. Today we’ll write actual tests.
Okay! Environment configured. Now, tests. Where were we?
Test-driven development, part deux
Test-driven development is a method of programming that proposes that when you build any new feature at all, you write your tests first and your code second. Not only is it a way of being really specific about just what you’re building, but you’re also defining its success and completion.
Are all your tests passing? Sweet, feature development is complete.
That’s the theory anyway. Now I have to admit that I personally have a love-hate relationship with test-driven development. Don’t get me wrong! Tests are awesome. There’s nothing quite like running your tests in their thousands with a single keystroke, watching them all succeed, and knowing you’ve just verified the stability of your entire code base.
But … in my experience, the physical act of building an app is less like precision engineering, and more like craftswomanship and clay-sculpting (or for you PC-gone-mad types, craftspersonship, ha). When I build a new self-contained feature, I’m massaging all its layers together in one go: the high-level component structure, the line-by-line code syntax, and everything in between. I personally don’t know exactly what lowest-level functions and methods need to exist, and be tested, until I’ve actually written them.
So I shall write my code and my tests roughly in parallel, and I would encourage you to do the same.
Let’s throw together a high-level design for our actual API-call-and-schema-tweaking architecture. Then we’ll know what tests to write.
Components:
(1) The code that hits the Ticketmaster API.
(2) The code that analyses the response pages, decides which bits we should save, and which we shouldn’t.
We’d seen in article 9 that Ticketmaster’s API response returns an absolute ton of stuff. We need to trawl through it all, get each gig’s venue, each gig’s act — and whilst trawling, likely say “wait, that specific bit of per-gig info actually looks kinda awesome, we should totally save that too! Change our spec!”
Do keep in mind that I’m typing this post as I design and build the app myself. Right this second, I genuinely don’t know what precious goodies all that API JSON might conceal. I’m guessing things like gig/venue/act images … but who knows what else? Let’s find out together. It’ll be fun! If we encounter fab things, let’s absolutely widen our scope and add more features.
Okay then. Feature-plan time. You may recall, way back in article 3, that I talked about something called the Request-Response cycle. Web apps function by waiting around until a client, like a web browser, pokes it in some way: the client Requests some resource. Then the web server Responds.
That’s mostly true, and we’ll get to Responses in later articles. Here, though, is an exception.
There’s a Thing called a Rake task. Incredibly briefly, it’s a snippet of Ruby/Rails code you may want to run outside the Request/Response cycle. You don’t want to wait around for a website user to click on a web-browser button to fire it off. We’re going to use a Rake task to manage our API-chatter code.
We’ve already used a Rake task. Recall back in article 8, we constructed our first proof-of-concept database tables: Gigs, Venues, Acts. We wrote a database migration class, db/migrate/20190910020025_create_acts_gigs_venues_tables.rb (the exact timestamp for your migration class will be different), telling Rails the changes we’d like to make to our database.
And then, rake db:migrate actually ran that class, executed those changes.
Rails comes with a great big bunch of existing Rake tasks. If you like, run rake --task to see that list.
So, designing architecture in parallel with tests. What should we architect, and what should we test?
Design and test a Ticketmaster Service
Rails has a Thing called services.
What’s a service? It’s actually a really broad term and can mean just about anything. It’s a way of grouping similar functionality, that does one thing and does it well. Read this if you’d like to learn more. I use services when writing code that hits any third party API.
We’ll need to not only test the methods this service contains, but also test the Ticketmaster API response itself, and continually make sure it does in fact contain the data, the keys, the values that we depend on it containing.
You may think it’s overkill to test and verify the data structures of a third party API, and you’re not actually far wrong. Problem is, third party APIs mess around with their own data all the time. The solid, reputable ones give months and months of notice of even the teensiest change, and I’d be astonished if Ticketmaster wasn’t one of them. But! If Ticketmaster decides, for any reason at all, that they’re going to change their API response’s JSON schema — that’s us shafted.
The whole point of these tests is to catch broken code as early as possible. Take it from me, there’s a world of difference between
Whoa, one of our tests has failed … I see the problem, writing a quick hotfix, deploying to Production, done!
Versus
Oh god, a feature on Production has already been broken for weeks, but our users haven’t told us, they’ve just simply run off to our competitors, we’ve only noticed because our active user base has halved, if only we’d first written tests to ensure our third party APIs returned their data in the structure we’d expected, if only, if only, if ooooonly!
Yeah, I’m being melodramatic. It’s fun! But also it has a point. So let’s write these tests. Here’s how.
First things first. Automated testing is something we want to run over and over again, pretty much indefinitely. Running our entire test suite, just once, might ping Ticketmaster’s API maybe a hundred times. And we might want to run our test suite ten, twenty times a day. We’d chew up our Ticketmaster API quota fast. (5000/day, 5/second).
How do we reconcile these? How do we run our test suite as often as we damn well please, but not chew up our quota?
There’s a gem called VCR. It records outgoing HTTP requests squirted forth by your tests, then caches their responses. Whenever another test makes the same request, VCR intercepts that request. Rather than making a second request, VCR instead returns its previous, stored request-copy. It preserves our API account quota. Also, we’re not waiting for a real HTTP response to scurry across the entire globe and then back, but instead just fetching a local copy. So it’s much, much faster. It’s neat. It’s fab. We shall use it.
Second things second. How do we actually test any given JSON structure, a schema? There’s a gem called JsonSchema, which does that very thing. You can supply its functions a schema-structure and some test-JSON. They test whether the latter conforms to the former. We’ll use this inside our tests to do the actual testing.
Sweet. Rate-limit concepts sorted, schema-test mechanism concepts sorted. Let’s implement them.
What actual data structure are we storing and testing?
I am hitting the Ticketmaster API using this URL: https://app.ticketmaster.com/discovery/v2/events.json?classificationName=Comedian&size=1&apikey=(key)
Which returns this: https://gist.github.com/tyrant/25b2d24c1c283baa16ab77a25ea48335
That right here is a single sample event JSON block. 458 lines, too long to show here, so I’ve moved it to its own Github gist.
Take a look through all that…
…After skimming through that, here’s what I’d like to retain and save:
We’d like to iterate through every single Event object returned by the Ticketmaster API. For each incoming event: compare its id (gist line 7) against all locally saved Gig-object ids (we’ll need to add a ticketmaster_id column to Gigs). If a table row with that ticketmaster_id value doesn’t exist, create a new row. If it already exists, skip over it and move on.
There are all kinds of useful other goodies we could totally add as Gig attributes, too. We’ve already got name (gist line 5). We could also add url (line 9); start/end times (lines 119–134); price (lines 174–181); seat_map (line 190). Any other juicy-looking tidbits? Let me know in the comments.
Can’t have a Gig without a Venue. This Ticketmaster’s Venue JSON kicks off on line 213. Lots of nice handy attributes: name (already got it); ticketmaster_id (line 216); url (218); images (lines 220–227); address (231–244); lat/lng (245–248). Latitude and longitude specifically is something we’ll need later, when displaying these things on our web-UI map. But one thing at a time.
Can’t have a Gig without an Act. Ticketmaster calls them Attractions. Fair enough, each to their own. Its JSON kicks off on line 305. What might we pluck? I like name (already got it); ticketmaster_id (308); url (310); images (324–395).
There’s no end to the juicy fruities we could pluck here. But we can add all this other stuff in later articles. MVP, baby! (That means minimum viable product, by the way, not most valuable player). One thing at a time. All we’ll need to grab, at this particular time, is each Gig/Venue/Act’s ticketmaster_id and name.
Okay then, let’s write some tests for that.
Here’s that same Ticketmaster API JSON again, but this time stripped down to only what we’re testing:
{
"_embedded": {
"events": {
"name": EVENT NAME HERE,
"type": "event",
"id": EVENT TICKETMASTER ID HERE, ... (crud) ... "_embedded": {
"venues": [
"name": VENUE NAME HERE,
"type": "venue",
"id": EVENT TICKETMASTER ID HERE, ... (other crud) ... ],
"attractions": [
"name": ATTRACTION NAME HERE,
"type": "attraction",
"id": ATTRACTION TICKETMASTER ID HERE, ... (unlikely El Dorado but who knows) ... ]
}
}
}, ... (additional crud) ...}
So! Our tests shall verify that the response JSON does indeed possess this structure.
How? I mentioned previously a Ruby gem called JsonSchema, which tests this very thing. How x2? Here’s a basic-usage example from its own Github page:
schema = {
type: 'object',
required: ['a'],
properties: {
a: { type: 'integer' }
}
}JSON::Validator.validate! schema, { a: 5 }
# trueJSON::Validator.validate! schema, { a: 'taco' }
# "The property '#/a' of type String did not match the following type: integer"
We’ll implement this shortly.
Rails Credentials
One more thing. To hit the Ticketmaster API, I used this URL:
https://app.ticketmaster.com/discovery/v2/events.json?classificationName=Comedian&size=1&apikey=kW7Cb9udjrC7APKYA9mtaSe2KWRMGfNote the bit at the end: apikey=kW74Cb9udjrC7APKYA9vmtaSe2KWRMGf. That’s my own personal Ticketmaster API developer key (tweaked a tad). It’s private info! It needs to be stored securely.
Rails has a Thing called Credentials Management. I won’t waste words describing it, follow this. Onward.
Test API Response Schema
Now we can move onto the actual test. Here’s the code:
# spec/services/ticketmaster_spec.rbrequire 'rails_helper'describe "Ticketmaster service" do
describe "The Ticketmaster API response itself" do let!(:schema) {
{
type: 'object',
required: ['_embedded'],
properties: {
_embedded: {
type: 'object',
required: ['events'],
properties: {
events: {
type: 'array',
items: {
type: 'object',
required: ['name', 'type', 'id', '_embedded'],
properties: {
name: { type: 'string' },
type: { type: 'string' },
id: { type: 'string' },
_embedded: {
type: 'object',
required: ['venues', 'attractions'],
properties: {
venues: {
type: 'array',
items: {
type: 'object',
required: ['name', 'type', 'id'],
properties: {
name: { type: 'string' },
type: { type: 'string' },
id: { type: 'string' }
}
}
},
attractions: {
type: 'array',
items: {
type: 'object',
required: ['name', 'type', 'id'],
properties: {
name: { type: 'string' },
type: { type: 'string' },
id: { type: 'string' }
}}}}}}}}}}}}
} it "returns the schema we're expecting" do query = {
classificationName: 'Comedian',
size: 1,
apikey: Gigs::Application.credentials.ticketmaster_key
} path = 'https://app.ticketmaster.com/discovery/v2/events.json'
response = HTTParty.get(path, query: query).parsed_response
errors = JSON::Validator.fully_validate(schema, response)
expect(errors).to eq []
end
end
end
Beautiful, isn’t it?
That great big sprawling :schema structure … even by my standards, I realise that is going to be quite incredibly opaque. You … might have to just trust me on this, and accept that :schema follows the standards laid out in https://jsonapi.org/. All it does is check that the aforementioned bits of data exist in the places we expect, and that they’re strings.
Run rspec spec/services/ticketmaster_spec.rb:53. If all goes well, you should see something like this:
$ rspec spec/services/ticketmaster_spec.rb:53
Run options: include {:locations=>{"./spec/services/ticketmaster_spec.rb"=>[53]}}
.Finished in 0.97086 seconds (files took 4.86 seconds to load)
1 example, 0 failures
Success! You can see this specific commit here.
And that’s it! We’ve set up our test environment, we’ve written our first test.
Next time: we’ll wrap the API-response test we’ve just written with a VCR cassette, then continue architecting our Rails Service.