REST API’s are powerful tools for automation and data analysis among other tasks, but they are also one of the most frustrating parts of building a new application. It’s cool I can theoretically combine so much data, but sometimes it’s extremely hard due to limitations, other times it makes no sense.
This transcends knowing about an API or experience. Some API’s suck compared to their applications, others lack in other ways. Sometimes you get documentation that works, other times the documentation is gated or blatantly wrong. You can write to a specification, but it’s useless if the specification is literally wrong. If you’ve been lucky in your career, you haven’t dealt with this, but one bad vendor and you’re stuck writing a trivial integration that can take months.
Working with a good REST API is trivial, but it’s the bad ones that stick out and show you just how far from standard you can get. I’m not going to name and shame as I am under NDA for some of my work (I don’t remember what’s covered with what). There are plenty of good API’s that just make sense and work like they should. The problem is that the inverse is true, and unfortunately really common. Let’s see what goes wrong from documentation, development, data, and the actual implementation and support of a REST API.
Documentation
A lot of vendors have great documentation for their REST API, but even more have terrible documentation. Another concern is whether the documentation is up-to-date or even relevant to what you’re looking to do. Some API’s have amazing documentation, but it’s stuck behind a paywall or another gate that makes it near impossible to thoroughly vet without substantial investment of resources.
Gated Documentation
Gated documentation drives me crazy. You can’t justify purchasing a product until you know what you can do with it, but some vendors don’t let you know what you can do with it until you buy in at some level (especially with a contract). Some might offer trial access, while others require a special developer account to just see the specifications for the API. Gated documentation just means it’s harder to know what solution will work for a given task.
A specific RMM (Remote Management and Monitoring) tool as well as PSA (Professional Services Automation) vendor (prone to being in the news for security breaches, not that that helps narrow it down any this year) requires a special developer account to see anything at all about the API besides how great it is if you sign up. This requires an application and extra red tape (as well as around a week or two of waiting) just to see if you can make it do anything useful. Depending on how long the application process takes, this can set a development task back weeks. This isn’t the biggest deal, but it just adds trash on the dumpster fire that the rest of the API usually is.
API Documentation Doesn’t Match Marketing Materials
This seems to go hand in hand with gated documentation. A lot of gated API providers have a mismatch with their marketing collateral about what exactly is available. You get claims like: “You can pull full lists of clients and all of their data easily!” But, curiously enough, it’s missing the fact that the API is limited to 100 calls an hour and you need 5 calls per individual client for the basics, let alone the actual data you’re trying to pull.
Other times, you find how easy an API is with how it follows some standard to the letter and you get in and learn nothing works right. We had this with a common accounting software provider where a standard C# library for working with REST API’s (RestSharp) was incompatible. It was a completely standard API, supposedly, but the golden standard for C# just didn’t work with it (and only it).
Documentation Is Just Wrong
I hate how many times I’ve encountered this, but it’s pretty common. The documentation doesn’t just suck (and actually, the documentation is usually great minus accuracy): it’s straight wrong. Functions aren’t documented or don’t exist, API calls don’t take the listed parameters, authentication requires something different (the more they talk about “standard OAuth2” and aren’t a major vendor, the more this seems to happen in my experience). Another common one is the mention of query types but the filters just don’t exist (or actually work).
All you need to do is run this function to get what you need (except it doesn’t work). The listed call doesn’t work as expected, or the returns are incorrect (even better is when the types don’t match the spec). Even worse, they imply that all functions have the same unified, underlying behavior and some (undocumented of course) just don’t have it. Don’t forget functions that are undocumented exceptions to all the rules in ways that make absolutely no sense. And, every single time, it’s the specific function you need for why you’re using the API in the first place which is 50 shades of fu…ailed. Too bad there was no way to test before buying in of course.
Development Implications
We aren’t even to actually building an application yet (the Eldritch horror is mainly academic at this point), but there are plenty of implications from limitations that pop-up in the actual development cycle. You’ll find out just how many (or few) API calls you have available as well as how hard it can be to determine how long it takes to build it out. Naturally, the documentation and other tools you have will (more often than not) shed as much light on the subject as a Vantablack mirror.
Call Limits Are Prohibitive
Behind every great (on paper) REST API is a poorly designed data return and a draconian call limit. These limitations seem fine until you get deeper into the API. 500 calls an hour sounds great on paper until you find out the vendor only allows a few pieces of data per return and requires a new token (and a new call) per extra page (if you’re lucky, this limitation is listed in a nice unindexed, gated forum posting). Well, it will only take a full month to pull one specific client’s full data and then you can move on to the next one. Analytics will be a breeze with 12 months of mismatched, floating data!
Prohibitively low hard call limits tend to be coupled with no real error handling mechanism so if you run out of calls, maybe it’s the same as a bad password or maybe it’s the same as the API being down and unable to connect (more often than not, those are even the same). Naturally, this is never really documented right (nor are the actual specifics that complicate everything). Who wants error messages or similar anyway?
Timelines Are Hard to Determine
It’s easy enough to determine how long something will take if you know all of the specifics about a given API or similar, but the problem we’re seeing here is that (more often than not) there’s more to it than the documentation. I end up padding my time estimates and still end up painfully wrong with some API’s. I gave a joke estimate of 6 months (we had multiple projects in parallel) with a connector that should have been trivial and I still ended up being wrong because of how bad it was.
Most development tasks are pretty straightforward, but I’ve had API connectors go from something that should only need a couple hundred lines of novel code to spanning multiple libraries with thousands upon thousands of lines just to whip the data into something useful (this isn’t even including how we store it or further work with it). It all depends on how accurate your source data for assessing the task is and whether you can try it before you have to actually begin giving estimates. While a good team will understand, it’s just another wrench in the gears.
Data Structuring
How data is structured can make the difference between whether it’s easy to work with or painful to deal with. Different API’s have different limitations and a nice console is no guarantee of a sane set of data when you get your JSON returns. Sometimes the data just doesn’t fit what you’d expect, other times it just plain doesn’t match the product itself.
Counterintuitive Data Structuring
There are plenty of REST API’s where the data makes no real sense with how it’s structured. Sometimes, the data is split up in ways that just don’t make sense, other times, you run into arrangements where the data is organized in way that flies counter to the arrangement or organization in the product itself. For instance, with a common PSA (Professional Services Automation) tool, the web application presents data with different divisions than the API does. In one version of the API (we’ll get to the later version’s shortcomings below), the same data is available, but it requires substantial work to reverse engineer (also called data munging) how everything goes together.
There are other API’s where the names or ID’s just don’t match up. It’s an array of phone‘s here but a scalar telephone there, or even worse the names or divisions just don’t make sense. Data hierarchies may also run counterintuitive to what you expect with a tree in a place there isn’t one in the application or similar. It feels like amateur hour at best.
Data Structuring Doesn’t Match the Product
More often than not, you’ll find an application presents far more data in the application itself than you can get out of the REST API. There are all sorts of wonderful things which can be worked into an external application (if the vendor would just cooperate). For instance, the previously mentioned PSA tool has a newer version of their API which just plain does not offer even a fraction of the data in the newer API they’re forcing people to move to.
A certain (previously) open DNS solution provider does similar where their API looks attractive at first glance but turns out to be completely useless for any kind of real automation or analytics. It doesn’t work for billing, it doesn’t work for automation, it solves a problem that no one asked for (and that I still can’t identify). The problem is the data just isn’t there.
Implementation
We got through the documentation, we got through the data structuring, now we’re dealing with the real implementation. Some of this happens early and convolutes the process, other times you’re at the mercy of stupid decisions from the vendor. Don’t worry, the best part of working with some API’s is the job security from the terrible job they do managing their products. No one else is stupid enough to be desperate enough to learn that trash after all.
Tokens and Authorization
When I hear OAuth 2.0 is required for an API, I start to cry. Not because OAuth 2.0 is bad, but because with my luck, whatever vendor I’m stuck building an integration for has no idea what they’re doing with it. It’s standard except for this bug and this bug and this bug. Oh, and those standard options in the RFC? The vendor probably doesn’t even know how to spell RFC, let alone read one.
The other piece to consider is how your authorization is going to work for an API. I tend to spend about a third of my time working with the basic functionality of a new REST API just dealing with authorization (except the mythical uniforms that can actually do things right). Refresh token dates tend to be as meaningless as the vendor’s claims that it’s standard OAuth 2.0 or that it “just works”. Sprinkle in the requirements to have a login from a specified URL with no easy testing opportunity and you get the kind of API that only a company that lobbied the US Government (to the point of preventing them from making tax software) could produce.
No Testing In Dev Mode
It should be no surprise that some REST API’s don’t allow full access in developer mode. I get the security ramifications and the idea behind it, but it makes developing an application substantially harder. Maybe they offer a sandbox, but either the features you needed are missing or else it’s just garbage and completely different.
It makes sense to a degree, but you’d expect at least some kind of actual test mode so you can iron the bugs out before having to publish an application (even to a test environment). Either you just publish straight to the production site like one of the madmen who make the API’s I’m complaining about seem to do (or you have to support an extra stack and secure it while operating in development mode). Thanks for requiring a callback URL that can’t just be an IP address for me to test things (even with limited access).
It doesn’t help that the sample code is often ancient at best and doesn’t actually compile anymore or even work. Combine this with wrong documentation and you get to explore the depths of StackExchange if you’re (somewhat) lucky, or some questionable forum.
There’s nothing like having to legit publish a debug page to test things before committing to a method. Token expired? Just open this app page and log in again. That’s just standard for an automation process. Oh yeah, you gotta reset the token literally every two days. Oh, you read the refresh token expiration date? Yeah, 24-48 hours versus 1 year is pretty equivalent, don’t worry about it, just log in. Why would you want automation to be automated anyway?
Vendors Ruin Everything
Wow, everything popped off without a hitch (minus all the stuff above)! The vendor sees things getting too good and pulls that rug from out underneath your application. Remember that API user type we required you to start moving to? Well, we decided to pull support for it on the old API because we made a new API. No, it doesn’t have any of the features you needed, but hey, it’s just a live application you build your livelihood off of, what’s the worst that could happen?
Vendors love to reinvent the wheel because otherwise their project managers may not feel they’re accomplishing anything otherwise. It’s just the cost of doing business and relying on another business that isn’t beholden to real standards. Of course, that vendor is always the one that is most central to your process.
Conclusion
I could go on for ages about how idiotic some of the API’s I’ve worked with are, but there’s nothing like diving into JSON and working with a new REST API to realize how backwards things could get. For every API I learn that doesn’t suck, I’m saddled with like five that are complete and utter garbage. REST offers some great functionality and possibilities, but the API is only as good as the programmers behind it.
What frustrates you with REST API’s that I missed? This is my list of pet peeves, but what gets your blood boiling? Things get complicated fast with some of the terrible ideas you’ll see in a live API. What did I miss and what do you agree with?
Image by Elf-Moondance from Pixabay