How To Build A Web App, part 13 of ?: save Ticketmaster API data; cute little rant about automated tests
This is the thirteenth in a series of articles taking you through all the actual steps in building a web app. If you’re an aspiring developer, if mucking around with teensy beginner tutorials frustrates you, if you’d love to build a properly substantial app that does fab things, these articles are for you.
Last time, we wrote tests for TicketmasterService.get_all_gigs. These tests ensure that the raw JSON response we get from Ticketmaster’s API does indeed follow the structure we’re expecting from them.
Today, we’ll write the code that receives the output from .get_all_gigs, iterates over its content, compares it with our current database content, and decides which bits of .get_all_gigs should be saved, and which discarded: TicketmasterService.update_existing_gigs. We’ll also write a few tests for .update_existing_gigs.
FactoryBot
I promised a while back I’d explain a gem called FactoryBot. That time has come!
FactoryBot is a library for automating away the boring bits of creating shitloads of database records for testing purposes. The idea is: imagine, for whatever test-flavoured reason, that you need to make twenty copies of an ActiveRecord mode, which itself has many zillions of attributes. Either you’d manually bang out twenty-times-a-zillion freakin’ attributes…
Or just type this:
list_of_20_models = FactoryBot.create_list :model_with_zillions_of_attributes, 20This works in conjunction with predefined model factory files. Once we’ve created these, we can then belt out database-related tests much much much faster.
So! We’ll create three new files, one per model: spec/factories/act.rb, spec/factories/gig.rb, and spec/factories/venue.rb.
Here’s their content. Each factory is defined inside a FactoryBot.define block, and you define each factory’s attributes with blocks of their own too. Like so:
# spec/factories/act.rbFactoryBot.define do
factory :act do
name { Faker::Artist.name }
end
end
What’s Faker::Artist.name? Faker itself is a ginormous random-data generator. Here’s its main category list. Take a peek through that. Faker::Artist.name spits out a random artist’s name each time you call it. And FactoryBot.build :act generates a new (and unsaved) Act object with a totally new name attribute value each time you call it. Simple.
FactoryBot.build's twin is FactoryBot.create. The former makes a new record; the latter makes a new record, and also saves it to the database.
> FactoryBot.build(:act).new_record?
true
> FactoryBot.build(:act).id
nil
> FactoryBot.create(:act).new_record?
false
> FactoryBot.create(:act).id
12345 Same with the other factories. This is the one for Gigs:
# spec/factories/gig.rbFactoryBot.define do
factory :gig do
act
venue
end
end
FactoryBot handles relationships between objects by requiring you only to denote belongs_to relationships. All you need to do is mention the model.
And for Venues:
# spec/factories/venue.rbFactoryBot.define do
factory :venue do
name { Faker::Restaurant.name }
end
end
Any particular reason why I chose restaurants specifically out of all of Faker’s string methods? Nah, not really. The random string generated for a Venue’s name need only be a string. That’s all. It just makes a tad more semantic sense to use names of restaurants than, say, a random alphanumeric hex-string.
Anyhoo. Let’s commit that.
Ticketmaster ID Migration
Next, we’ll need to do a bit of database-updating. You’ll recall we’ve been making a ton of noise about Ticketmaster IDs. They’re how we distinguish each Act, Gig, Venue from each other, at least from Ticketmaster’s point of view.
But we haven’t actually added those columns to our database yet. Let’s do that, with a nice quick rails g migration AddTicketmasterIdsToTheUniverse:
# db/migrate/20191113063032_add_ticketmaster_ids_to_the_universe.rbclass AddTicketmasterIdsToTheUniverse < ActiveRecord::Migration[5.2]
def change
add_column :acts, :ticketmaster_id, :string
add_column :gigs, :ticketmaster_id, :string
add_column :venues, :ticketmaster_id, :string add_index :acts, :ticketmaster_id
add_index :gigs, :ticketmaster_id
add_index :venues, :ticketmaster_id
end
end
Columns and indexes too.
I also see I’d made a mistake in an earlier migration, on the Gigs table: the foreign keys I’d made turned out to be venues_id and acts_id, not venue_id and act_id. Let’s fix that with rails g migration OopsChangeGigsForeignKeys:
# db/migrate/20191113063352_oops_change_gigs_foreign_keys.rbclass OopsChangeGigsForeignKeys < ActiveRecord::Migration[5.2]
def change
rename_column :gigs, :acts_id, :act_id
rename_column :gigs, :venues_id, :venue_id
end
end
Now run the usual rake db:migrate and rake db:migrate RAILS_ENV=test, and commit.
FactoryBot Support
One common thing we do with FactoryBot is create this:
# spec/support/factory_bot.rbRSpec.configure do |config|
config.include FactoryBot::Syntax::Methods
end
Why’s that, Mikey? I hear you yowl. Here’s for why. It’s nothing overly horrendous or gigantic, but when you’re writing tests, you’ll find yourself creating some 27 million factories. It simply means you’ll type build :foo instead of FactoryBot.build :foo. Easier. Create and save and commit and diff and moving on.
Models
I was about to say it was time to start writing the tests themselves, but of course that’s not quite true, is it? Just realised I’d neglected to create the model files themselves. No point in writing model-factories without the actual models, right?
So. Same again. Act, Gig, Venue:
# app/models/act.rbclass Act < ApplicationRecord
has_many :gigs, inverse_of: :act
end
# app/models/gig.rbclass Gig < ApplicationRecord
belongs_to :venue, inverse_of: :gigs
belongs_to :act, inverse_of: :gigs
end
# app/models/venue.rbclass Venue < ApplicationRecord
has_many :gigs, inverse_of: :venue
end
Tests
Okay. Infrastructure complete. Now it’s time for tests!
After some thought, I wrote this. I shall talk you through it.
# spec/services/ticketmaster_service_spec.rb# Assumptions: we won't ever get an existing gig with
# changed/updated attributes or relationships. Though we were wrong # about Ticketmaster venues never having null names, weren't we.
# It's possible we're wrong about this too! But let's cross that
# bridge should we ever come to it.describe '.update_existing_gigs' do let!(:act1) { create :act, ticketmaster_id: 'aid1' }
let!(:venue1) { create :venue, ticketmaster_id: 'vid1' }
let!(:venue2) { create :venue, ticketmaster_id: 'vid2 '}
let!(:gig1) { create :gig, ticketmaster_id: 'gid1', act: act1, venue: venue1 }
let!(:gig2) { create :gig, ticketmaster_id: 'gid2', act: act1, venue: venue2 } let!(:incoming_gigs) {
[{
name: 'gig1',
ticketmaster_id: 'gid1', # Existing gig
venue: {
name: 'venue1',
ticketmaster_id: 'vid1', # Existing venue
},
act: {
name: 'act1',
ticketmaster_id: 'aid1', # Existing act
},
}, {
name: 'gig2',
ticketmaster_id: 'gid2', # Existing gig
venue: {
name: 'venue2',
ticketmaster_id: 'vid2', # Existing venue
},
act: {
name: 'act1',
ticketmaster_id: 'aid1', # Existing act
},
}, {
name: 'gig3',
ticketmaster_id: 'gid6', # New gig!
venue: {
name: 'venue3',
ticketmaster_id: 'vid1' # Existing venue
},
act: {
name: 'act3',
ticketmaster_id: 'aid1', # Existing act
},
}, {
name: 'gig4',
ticketmaster_id: 'gid7', # New gig!
venue: {
name: 'venue7',
ticketmaster_id: 'vid7' # New venue!
},
act: {
name: 'act32',
ticketmaster_id: 'aid32' # New act!
},
}]
} it "creates two new gigs" do
expect {
TicketmasterService.update_existing_gigs(incoming_gigs)
}.to change {
Gig.count
}.by 2
end it "creates one new venue" do
expect {
TicketmasterService.update_existing_gigs(incoming_gigs)
}.to change {
Venue.count
}.by 1
end it "creates one new act" do
expect {
TicketmasterService.update_existing_gigs(incoming_gigs)
}.to change {
Act.count
}.by 1
end
end
First up, we’ve got the five let! blocks. We’ve seen these before, on the #get_all_gigs test. They’re an RSpec thing, and their nature is a tad beyond the scope of this article. Read this for more.
Inside the blocks are five FactoryBot calls. Our goal here is to create a bunch of objects in our database, partially overlapping :incoming_gigs. Our test will create new database objects for the not-already-created data, and ignore the rest.
They’re taking full advantage of our spec/support/factory_bot.rb support file, meaning we can type create instead of FactoryBot.create. Remember the difference between create and build, by the way: create makes a new object, and saves it too. build makes a new object, but doesn’t save it.
Ranting is fun!
What’s the point of all this build versus create malarkey? Well. Any amount of database work makes tests run slower. Bad! Should you ever find yourself working on a really big app, its code base may well have thousands upon thousands of tests. The longest I personally have ever seen a test suite run was well over an hour and a half. Ridiculous! The whole point of automated tests, you understand, is to detect bugs as early as humanly possible. I personally have worked in companies where the non-techie management have proclaimed that tests are an expensive waste of time, let’s just get features written and make our customers happy … but then, weeks later, these managers cannot figure out why their company’s user base has halved, and the remaining customers make a ton of harrumph-sounds about how this app is a buggy piece of crap.
And do management then take responsibility for their mistakes? No sir they do not. They point the finger at us devs, whinging that we’d pushed buggy code.
Anyhoo. Sorry, male lasses and female lads, I’m sure you can tell I write from bitter, frustrated experience. I genuinely don’t mean to relay that onto your head also. But long story short: WRITE TESTS, baby.
So what was my point? Right. Test suite execution time. It’s a mark of badly written tests that their execution takes ages and ages. If you’ve got ten thousand tests, running them may take hours. Bad. The idea is, you might make a change to the app, major or minor, then run your tests to verify absence of nasty buggy side effects. If your tests run in two minutes, then awesome, you can get things done fast, nice zippy turnaround. If your tests run in two hours … then, boy, aren’t you just pissing your valuable time up against the wall, right?
So. Writing fast tests. This right here is a gigantic topic all by itself. The goal of these posts is to build a web app and skim over each of its components in juuuuust enough detail to give junior/intermediate devs an idea of what they do and why. As much as I’d love to do a deep dive into writing snappy tests, I’ll have to just simply provide you with links to a deeper dive on this topic, should you wish to explore more yourself. If you’d like to learn more about test shortcuts, read this, and if you’d like to learn more about ways to run only tests for the code affected by any change you’ve made, rather than every test in entire suite, read this.
Anyhoo. Moving on. Long story short, it’s a good rule of thumb to avoid saving ActiveRecord objects if you can possibly avoid it. But this particular test involves the objects already being saved and present in the database, so save we must.
Resuming our test-code analysis: next, we’ve got :incoming_gigs. It’s a great honkin’ structure I wrote. It’s an example of what TicketmasterService.get_all_gigs might output. It’s four gigs. Two have identical Ticketmaster IDs to gigs already in the database. Two do not. Of the two new gigs, one new gig has acts and venues already existing. The other new gig has totally new acts and venues.
So if we process this incoming gig data, we’d expect it to create two new Gig objects; one new Act object; and one new Venue object. And that’s exactly what the three tests measure.
If you’ve not seen RSpec test syntax before, it can be utterly bamboozling. I personally find they’re trying a bit too hard to be “human-readable”, as I believe the cool kids phrase it. Here’s their official documentation page on their shenanigans, do read more if that’s what floats your boat.
Actual Service Code
I faffed around for a bit, then settled on this:
# app/services/ticketmaster_service.rb# Iterate over every gig from .get_all_gigs. For each
# not already existing, create it. The newly created
# ones will have acts and venues that may or may not
# already exist locally. Query this. If they don't,
# create them too.
# Param: an array of gig hashes.def self.update_existing_gigs(gigs) gigs.each do |incoming_gig| Gig.find_or_create_by({
ticketmaster_id: incoming_gig[:ticketmaster_id]
}) do |gig| gig.act = Act.find_or_create_by({
ticketmaster_id: incoming_gig[:act][:ticketmaster_id]
}) do |act|
act.name = incoming_gig[:act][:name]
end gig.venue = Venue.find_or_create_by({
ticketmaster_id: incoming_gig[:venue][:ticketmaster_id]
}) do |venue|
venue.name = incoming_gig[:venue][:name]
end
end
end
end
Here’s what’s going on. .update_existing_gigs receives a hash structure of gigs, then iterates over them. For each, it tries to find a Gig in our database with an existing Ticketmaster ID. If it can’t find that, then it creates a new Gig with that Ticketmaster ID value.
This new-or-existing gig then tries to find both an Act and a Venue by their Ticketmaster IDs too. And same again — if they don’t already exist, they get created. And if we’re creating new Acts and/or Venues, we also assign the Ticketmaster-generated name.
Note that we’re using find_or_create_by for all three. Here’s the official docs if you’d like to read further, but if that’s too much, a quick once-over:
record = RecordName.find_or_create_by(attribute: current_value) do |new_object|
new_object.attribute = some_other_value
endThe idea is, it first tries to locate any existing record equalling whatever attribute value you supply. If any such records exist, the record variable gets populated with the first match. If, however, ActiveRecord finds zero records by that value, it creates a new one, runs the block, saves, and populates record with that newly created record instead. We do that three times.
The time has now come to actually run our tests. After about 43 years of freakin’ buildup. Here goes.
$ rspec spec/services/ticketmaster_service_spec.rb:110
Run options: include {:locations=>{"./spec/services/ticketmaster_service_spec.rb"=>[110]}}
...Finished in 0.75993 seconds (files took 4.87 seconds to load)
3 examples, 0 failures
Success!
Now anyway. We could get into the weeds here, testing ever more fiddly little details, but I think that’ll do for now. Let us commit, diff.
Please don’t be put off too much by the gigantic mountains of boilerplate code here! Yes, we’ve been at it for hours and hours. Most of that is down to this being one of the project’s earliest tests, so we have to build a shitload of infrastructure ahead of time. Later tests will be easier, I promise.
That’ll do for today. Next time: we’ll begin writing our own actual API, which we’ll use later to actually query and retrieve the gigs we’re storing now.