A few weeks back, I reported on the developer outrage that was provoked by Yahoo!’s sudden and unexpected shutdown of a service that, for well-more than a decade, was being consumed as an API by an untold number of apps. All of those applications broke. Never ones to be shy, developers voiced their displeasure.
But the service in question was not an official API. It was a feature of Yahoo! Finance — an everyday HTML button -- that was not intended for developers to consume in their applications. It was really intended for ordinary Web users who might want to download a CSV file containing stock data for subsequent import and analysis with their spreadsheets. But, in a move that was tantamount to screen-scraping, developers jumped on it like it was an API when they discovered more than a decade ago that Yahoo! wasn’t requiring identification or authentication of any sort to use the service.
Without identifying themselves, developers could emulate a click of that button once per day, or once per second. It was the equivalent of a free, near real-time stock quote API; an incredibly valuable service to a great many applications and users. Especially since there are a great many API providers out there charging hard cash for the exact same thing. I’m guessing that Yahoo!’s new owners (Verizon) spotted a giant resource pig in their log files, traced it to a bunch of machines “clicking” the button with absolute wreckless abandon, and thought “WTF!?”
Off went the service. Even the button intended for ordinary Web users broke.
Whether the “APIs” are official or not, there are hundreds if not thousands of APIs out there that, like that download button, are wide open. In other words, no form of authentication or identity is required to use them. For some API providers, the thinking is that API adoption is inversely proportional to any friction in access. But as I pointed out in my second article, if developers can access an API or service without identifying themselves, the service provider has no sure-fire way of contacting those developers to warn them of something like an upcoming service disruption.
Even if Yahoo! wanted to contact the developers whose apps were consuming the CSV download, it had no way of doing so (whether it really wanted to make contact with them — I certainly would have if I was involved — is another question altogether).
The situation got me to wondering whether it was time for Web APIs — and all Web traffic in general — to be capable of bearing warnings along with normal payloads. Or, what I think of as an in-band warning. Sort of like when a bill arrives in the mail with a special indicator on it that the previous month's payment is overdue.
In researching the options, I came across a possibility on Stack Overflow that I also wrote about. The bad news is that I misunderstood the implementation details and so what I wrote — that the approach relies on extending the HTTP status code that comes with every Web response — was technically inaccurate. The good news is that even with the correct technical details, the idea of doing in-band notifications not only remains unchanged, it can apply no matter what the HTTP status code is. In other words, an in-band service warning can essentially be appended to an HTTP response of any type and status.
So, in this article, I will not only offer the correct technical details, I will also offer a tutorial that anyone can try in order to see it in action for themselves.
First, the correct technical details.
Originally, I thought the idea was based on the notion that any 3-digit HTTP status with a 2 in the first digit (eg: 200, 201, 299, etc.) was still viewed as as much of an HTTP success as an actual HTTP status of 200. So, my original explanation suggested that when a Web service (an API, a site, etc.) wanted to issue a warning as a part of an otherwise successful response, that it could flip the status to 299 (instead of 200) and append the status with some text that offered additional warning details.
However, in a tweet, NordicAPIs co-founder Travis Spencer pointed out the error and directed me to the actual technical details of the solution on Mozilla’s website.
Instead of relying on the HTTP status, the actual implementation relies on a special HTTP header known as the Warning header. The origins of this warning Header can be found in IETF RFC 7234, the original intention of which is to define "HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages.” But the Internet is full of RFCs and specs that were originally intended for one thing and then got extended to support other ideas. And rather than propose an entirely new HTTP specification, maybe it’s easiest just to take advantage of what’s already there.
A bit deeper into RFC 7234 is section 5.5.7 which describes the aforementioned optional “Warning: 299” HTTP header. The Mozilla document offers some easy to read implementation details, noting that the syntax is as follows:
Warning: <warn-code> <warn-agent> <warn-text> [<warn-date>]The Stack Overflow post showed some examples of what this warning might look like:
Simple Example:
Warning: 299 - "Deprecated API”Slightly More Complex:
Warning: 299 api.blazingFrog.com "Deprecated API : use betterapi.blazingFrog.com instead. Old API maintained until 2015-06-02”In the first example, a hyphen is substituted for the warn-agent parameter (this is allowed) and the warn-text (always in quotes) is very simple. But the second example includes the warn-agent along with a much more detailed explanation in the warn-text parameter.
My idea is for the API industry to work together to come up with a standard vocabulary for the warn-text that all API and site operators could use in order to package a spectrum of service warnings into HTTP responses. For example, much the same way there’s a fixed set of numerical codes that can be offered as the aforementioned warn-code (110,111,112, 214, 299, etc.), the API industry could agree to a standard vocabulary that could be provided as the warn-text. For example. a machine-consumable comma delimited string that involves a sub-code (150), a brief explanation (Service Retirement Pending), and a URI that the machine can crawl for more information:
Warning: 299 - “150, Service Retirement Pending, http://yoursite.com/details/150.htm”Earlier this week, I not only proposed this idea in a presentation given to the December edition of the monthly Washington, DC API Meetup, I mocked its functionality using my MacBook, Node.js, Express.js, and ngrok. The idea was simply to show what it might look like. Here’s how I did it (and how you can do it as well).
Frst, make sure Node.js is installed on your machine. Node.js is a web application server that’s based on the Javascript language. If it’s not already installed, get the download from https://nodejs.org/.
Then, install Express.js. Express.js is a framework for Node.js that eliminates a lot of the pain that’s associated with coding Web applications or writing code that provisions APIs from scratch.
Continued from page 1.
If Express.js isn’t already installed, run the following command at your console prompt:
$ npm install express body-parser --saveThe last thing you need to install is ngrok. Download it from https://ngrok.com. Once it is installed, run ngrok in your console to listen on port 3000:
$ ngrok http 3000What does ngrok do? It basically gives you a public Web address that will route traffic to an application server running on your computer, no matter where your computer is or what network it is attached to (so long as that network’s firewall isn’t blocking port 3000 or whatever other port you picked). After running ngrok, you will see something like this:
The URI shown in the “Forwarding” line will serve as public URI that we use to test our code.
Next, we need to write a tiny bit of server-side Javascript that will do the following:
1. Load the Express.js framework
2. Listen on port 3000 for HTTP requests
3. Respond with “Hello World”
4. Include an example 299-type Warning Header with that response
5. Notify the service operator (you) that a warning has been issued and show the IP address that it was issued to.
The code I’m offering for this workflow falls on the starkest end of what’s minimally viable. For example, my method for detecting the source IP address of the inbound request isn’t 100 percent reliable.
At your system’s console, create or navigate to a directory (folder) that you’ll save your code to. Then, using your favorite text editor (Sublime, Atom, Notepad, vi, etc.), save the following block of code to a file in that directory called “serviceWarning.js.”
const express = require('express')
const app = express()
app.get('/', function(req, res){ res.setHeader("Warning", '299 - "150, Service Retirement Pending, http://yoursite.com/details/150.htm”';); res.send('Hello World\n'); console.log(‘Warning 150 Issued to '+req.headers['x-forwarded-for']);
});
app.listen(3000, () => console.log('Example app listening on port 3000!’))Once saved, enter the following command at your console:
$ node serviceWarning.jsIt should respond with:
Example app listening on port 3000!Leave that window alone and let’s start a new console window (we’ll call this Console #2) and at the prompt in that window, let’s hit the new endpoint with the following cURL command
$ curl -i http://<your ngrok URL here>If it’s all working correctly, you should see something like the following output in Console #2:
HTTP/1.1 200 OK
X-Powered-By: Express
Warning: 299 - "150, Service Retirement Pending, http://yoursite.com/details/150.htm";
Content-Type: text/html; charset=utf-8
Content-Length: 12
ETag: W/"c-ZIpqb//9qgutsjuLr5C2Fo3Razo"
Date: Thu, 07 Dec 2017 17:31:38 GMT
Hello WorldThen, in Console #1, you should see the following:
Warning 150 Issued to www.xxx.yyy.zzzwhere www.xxx.yyy.zzz is the source IP address of the machine. Ask your friends to run the same cURL command from their computers and you should see their IP addresses show up.
Next, go to Google Chrome as your browser, launch a fresh browsing tab, and open the Developer Tools by picking View > Developer > Developer Tools. Within the Developer Tools pane, click the Network tab and for the active browser tab, visit your ngrok URL. In the left pane of the browser, you will see “Hello World”. In the right pane (the Developer Tools pane), select the first line in the left hand column (the line containing the ngrok URL) and you should see the contents of the response header, including the warning that your code returned (as shown below).
Instead of using the the existing RFC 7234 Warning header as the vehicle for such warnings, another approach (that I mentioned in my original post) would be to create a new one called X-API-Warn as some API providers such as Clearbit have done. This approach might reduce the risk of interfering with some other miscellaneous 299 warning that’s actually for some form of cache control. But on the same token, the likelihood of that seems extremely low and the IETF strongly discourages the invention of new X-prefixed headers.
But let’s say the industry decided to pursue that option instead. It would still need to agree upon a standard array of parameters and vocabulary so machines would know what to do with the contents of the header. Which header to use is really inconsequential to the outcome. It’s all about that standard vocabulary. So, if you’re interested in joining me on the journey to pick an approach and establish some vocabulary, let me know at david.berlind@programmableweb.com.