How To Build A Web App, part 9 of ?: connect to the Ticketmaster API

This is the ninth in a series of articles taking you through all the actual steps in building a web app. If you’re an aspiring developer, if mucking around with teensy beginner tutorials frustrates you, if you’d love to build a properly substantial app that does fab things, these articles are for you.

Last time we built some neat database tables for our app. Today we’ll connect to a third-party API and start adding rows to them.

What’s an API?

“API” stands for Application Programming Interface. In its most general sense, it’s a way for one program to connect to another. Different APIs serve different data formats, some human-friendly, some program-friendly.

Websites do this all the time. We’ve already gone over that in article 3. Viewing google.com? Your web browser hits up the Google web server’s API, then renders its response. It serves up machine-friendly data (HTML) that your web browser is good at converting into nice beautiful web pages for us humans to consume.

The API we’re interested with, here and now, is owned by a company called Ticketmaster. It hosts a zillion details for a zillion events all over the world. You can nab as much raw data for these events as you please. We’re going to siphon off this raw data and store it in our own local database, then in later articles, write a browser UI that connects to our web server, grabs our stored data, and displays it.

One Stepping Stone Too Many?

You may ask, waaaitasec, why don’t we just connect our browser UI directly to Ticketmaster’s API? Why have we bothered with a web server and database at all?

Couple of reasons. First, rate limits. Check out Ticketmaster’s Getting Started guide. About halfway down is the Rate Limit section, where it points out that standard developer accounts aren’t allowed to query their API more than 5000 times a day, or more than 5 times a second. Otherwise you’ll get weasel-customers writing scripts that just spam and spam and overwhelm their servers forever.

The app we’re eventually going to build will need to update/refresh its gigs data every time the page loads, every time you click and drag and move the map area it’s viewing, and every time you zoom in and out. Even just a single user will gobble up that 5000/day limit fast.

What we’ll do is write an API script that reaches out from our own web server, hits the Ticketmaster API once an hour, then stores that data in our local database, within the local server. Then our web UI can hit that local server as often and as merrily as we please. No rate limits!

Second reason: different APIs, different data schemas. I’d very much like to build a gigs app that hits up not only Ticketmaster’s API, but also most other gigs and events APIs, to aggregate as much sweet sweet comedy data as possible: Eventfinda (a New Zealand event site), Ents24 (a UK event site), and whatever other site or API that tickles my fancy.

Take a second look at Ticketmaster’s Getting Started guide, this time at the Data Model bit. It shows how Ticketmaster is modelling/schema-ing all its gigs data. Keep in mind that not just Ticketmaster, but all these other sites, are repositories not just for standup comedy, but for any event or gig or attraction or act of any type at all. They’re broad, we’re specialised.

I’ve not yet looked the exact data schemas for all these other APIs just yet, but I’d be astonished if all were 100% identical to Ticketmaster’s. For each of them, we’ll need to hook up our app to them, receive their API responses, do a bit of schema-transmogrifying to slot their data into our databases, then do the actual slotting. Each different API and schema will require different schema-transmogrifying, and I’ll take you through each when we get to them.

Sooo … rate limits, and multiple API schemas, are the two reasons why we’ve even bothered with a server/database layer at all. If we’d only ever wanted to use Ticketmaster and nothing else, and had a premium account without rate limits, we could just connect directly to Ticketmaster from our web browser.

So let’s do the actual connecting.

API Consumer Key

What’s that, you ask? It’s how an API authenticates its requests.

If you’re not familiar with “authentication”, it’s basically the process of proving to a computer your identity. You’re proving you are who you claim.

Humans do that with usernames and passwords. Programs totally can too … but it’s more common to do that with a string of data called a key. Lots of reasons why you use keys over passwords, but one is that us humans are often really bad at guessing nice strong hard-to-hack passwords.

An automatically generated key, however, is guaranteed totally random and scrambled and is as unguessable as it’s possible to get.

Ticketmaster uses a system called OAuth, the nitty-gritty of which is way, way beyond these articles’ scopes. But long story short, when you first sign up, it provides you a public key, and whenever your app connects, you provide that public key.

Here’s one similar to mine: kW7Cb9udjrC7APKvqYA9matSe2KWRMGf

See what I mean? Imagine manually guessing that!

I’ll be using my own key. You’ll need your own. Hop onto Ticketmaster’s main API page, https://developer.ticketmaster.com, and you’ll see a button marked “get your API key”. Jump through its hoops and sign up for a developer account. No hurry. I’ll wait.

Return comedy-gigs data from Ticketmaster’s API

Great, sorted. Next, we’ll go through how to prod the API to get it to squirt back at us the data we want. We won’t do anything with that data, not yet. We’ll get to that in the next article. Here we’re just focusing on how to prod Ticketmaster’s API to make it return the right data of the right variety.

Ticketmaster has extensive API docs. Here’s its page re its Discovery API: https://developer.ticketmaster.com/products-and-docs/apis/discovery-api/v2/ It returns all data about any and all of the entities in its entire giant world-spanning data sets. That’s what we’ll use.

After a bit of digging round, I found the right docs and the right endpoint for getting comedy-gig data specifically: https://developer.ticketmaster.com/products-and-docs/apis/discovery-api/v2/#search-events-v2 As docs pages go, I have to acknowledge that Ticketmaster’s are pretty good! They give extensive info on every endpoint and how to use them. You can see that to the right of the “Event Search” title, there’s a dropdown menu named “Javascript”. Let’s test this event-search endpoint.

Select the dropdown, and click on its other entry, “cURL”. cURL is a command line program on any Mac or Linux system, you can run it just like all the other ones we’ve used so far, Git, Rails etc. You’ll see the textarea that’s now appeared below it has a sample command, something like this: curl \ —-include 'https://app.ticketmaster.com/discovery/v2/events.json?size=1&apikey=kW7Cb9udjrC7APKvqYA9mtaSe2KWRMGf'

Copy and paste that into your command line, and run it. If all goes well, it should then show something like this:

$ curl \
> --include 'https://app.ticketmaster.com/discovery/v2/events.json?size=1&apikey=kW7Cb9udjrC7APKvqYA9matSe2KWRMGf'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Accept-Ranges: bytes
Accept-Ranges: bytes
Accept-Ranges: bytes
Access-Control-Allow-Headers: origin, x-requested-with, accept
Access-Control-Allow-Methods: GET, PUT, POST, DELETE
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3628800
Age: 0
Age: 0
Content-Type: application/json;charset=utf-8
Date: Mon, 07 Oct 2019 03:43:10 GMT
Rate-Limit: 5000
Rate-Limit-Available: 4994
Rate-Limit-Over: 0
Rate-Limit-Reset: 1570503621650
Server: nginx
Set-Cookie: TMSO=seed=0bad7b43a051&exp=1570503622&kid=key1&sig=0x8cfc1d4ea6311175e3065825f302e90f6074129de9d1e567454b33bdd673c8cd5df4f841558e575adca518cd7357f2f76887af063f2a813ceb8dd83c97874ea3; Path=/; Expires=Tue, 08 Oct 2019 03:00:22 GMT
Vary: Accept-Encoding,Accept-Encoding
Via: 1.1 varnish
Via: 1.1 varnish
X-Application-Context: application:default,jash1,docker:8080
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Served-By: cache-bwi5149-BWI, cache-sea1022-SEA
X-Timer: S1570417222.654540,VS0,VE267
X-TM-SESSION-BID: bk2ar98rs4ov6qvfgrs52h8bd
X-TM-SESSION-SID: w+sAGn/CRp7JSApCaoAAAHKE
X-TM-UNIQUE-ID: w+oAGn/CRp6P7QpCaoAAAHKE
Content-Length: 10762
Connection: keep-alive
{ a ton of JSON, like, hundreds of lines of it }

Success! We’ve got the API returning a sample response.

Two quick explanations: all these keys and values, from Accept-Ranges: bytes, down to Connection: keep-alive, are Things called HTTP response headers. They’re behind-the-scenes stuff web browsers deal with, and normally we’d not be bothered with them, but cURL displays the beasties by default, so I thought I’d best mention them.

The second, chunkier response component, which as you can see is so chunky I’ve truncated it, is our actual JSON data. “JSON” is a Thing called JavaScript Object Notation, and it’s a data-structure language used pretty much everywhere in programming. This particular JSON is sample data provided by the docs’ sample endpoint, provided just to give us devs a feel for the sort of thing it returns normally.

By the way, I’m sure you’ve noticed that having our data all fired back in one great horrendous lump isn’t exactly human-friendly. I use sites like https://jsonformatter.org/. If you copy and paste the response JSON into that site’s left textarea, and click “Format/Beautify”, it’ll restructure that data into something a bit nicer and easier to read.

So that’s our sample API endpoint. What’s the actual API endpoint? What will provide us with our sweet sweet Gigs spigot?

I had to do a bit of searching through the docs to figure out what search-filters to apply, but got there in the end. Here it is:

curl \
--include 'https://app.ticketmaster.com/discovery/v2/events.json?classificationName=Comedian&apikey=kW7Cb9udjrC7APKvqYA9matSe2KWRMGf'

Note the classificationName=Comedian bit. That does the trick.

Run that. Why yes it’s an even ginormous-er JSON blob. I ran that through the JSON formatter site, and it came out to 8598 lines.

Excellent. We have our spigot. The URL above is what we shall hit in future code to yoink Ticketmaster’s standup comedy data.

And we’re just about done! But! Before we wrap up, I’d like to draw your attention to its last few lines:

  ...  
  "page": {
    "size": 20,
    "totalElements": 1341,
    "totalPages": 68,
    "number": 0
  }
}

I’m sure you’ve realised quite a while back that the data structures used here are bloody complex. There are all kinds of tips and tricks that APIs like Ticketmaster use to keep that complexity organised. One of them is pagination: when you’re dealing with thousands and millions of items of information, you don’t fire back the whole lot all at once unless you really have to. Instead, return them in batches. Pages.

This 8000+-line response describes only the first 20 gigs. This is page one. There are 1341 gigs total. 68 pages.

We’re going to be busy.

Next time: we’ll go through these humongous JSON gigs-data responses, decide which of these data we’d like to save and which to throw away — then save them.