DEVASC REST API Deep Dive – API Types / Methods / Status Codes / Headers / Pagination / Rate-Limit / Tools / Python Scripted REST API Calls – Easy points on exam day!

This will cover most things REST API at the fundamental level needed for the DEVASC, some items / tools not entirely clear here should be independently reviewed, as this article is long enough with me doing a full screenshot walkthrough of Postman REST API Tool 🙂

With that, lets get this party started!

What an API is, and the different types of API

APIs at the most basic can be thought of as a set of functions / procedures for one application to communicate with another, which can be used more commonly known on Web Applications such as tracking on a Commerce website of the item, to applications or even hardware on the local PC / Database making calls between different Applications.

There are 3 fairly broad categories that APIs can fall into:

• Service API – This type of API exists to perform a function the application cannot perform itself or is not an authority on doing, for example PayPal “Pay Now” buttons embedded in different eCommerce sites or being re-directed to MFA logins when trying to access a work resource, this API essentially offloads the “Authentication” of the user to another resource
• Information API – This type of API requests a snapshot of information from another application or Database, much more in the realm of Controller / Device features such as Telemetry / Statistics / Provisioned Device Configs / etc
• Hardware API – This type of API relies not just on Software communication, but is specific to its correlating hardware functionality, such as Thermostats or GPS when you are watching to see where your Dominos Pizza delivery driver is because your starving!

There are 3 corresponding API Access Types / Levels of Access to these API Types:

• Private – This access type is for a single person or personnel of a company only
• Partner – This is similar to the Meraki API from my previous post where an API Key can be generated in Dashboard, and shared with Partners to give them access to make changes to the network without giving them full Admin Access to your Meraki Dashboard
• Public – This is a publicly available resource, these are a good source of testing / creating API’s for other developers to help develop their API and for beginners to learn with

All of these API Types / Access Level work within the HTTP Request / Response model

Though some of these APIs can request some pretty sensitive or complex information (like payment information or GPS coordinates), it is all sent via HTTP modeling a Request and Response each API Call, where the Software making the API Call is the “Client” and the resource being called by the API is considered the “Server” in this model.

When requesting a Web Page from a Web Server via HTTP, it provides back a HTML response that present a web page, with underlying mechanisms not seen to the page such as Headers and Status codes like the well known 200 OK response or the dreaded 404 Page Not Found.

There are 4 elements to make a successful request to a Web Server:

• URL (Uniform Resource Locator)
• Method (GET / POST / PUT)
• Header(s)
• Body

URL – Uniform Resource Locator or URI (Uniform Resource Identifier)

The URL or URI (Uniform Resource Identifier) in API specific cases use 4 components:

• Protocol
• Server / Host Address
• Resource
• Parameters

I had gone over this same example in a previous article, but shown below is an example of an API on this webpage making a call to the WordPress Database using the search function in the side bar, and provides the Parameters it is looking for – In this case “devnet” posts:

https://loopedback.com/?s=devnet

HTTPS is the Protocol, loopedback.com is the Host Address, “/” itself is the resource as it searches the root of my entire website (Server / Host), with the Parameters of ?s=devnet.

List of Methods (and what they do)

• GET – Get a web resource from Server (read only)
• POST – Create new data on Server / Resource (write only)
• PUT / PATCH – Partial update to resource on Server (write)
• HEAD – Requests the Header a GET request would have gotten from the Server, this is special case use as it contains ‘last modified’ data that can be used to check against the locally cached copy of the Header / Resource
• DELETE – Delete data from the target Server / Resource
• CONNECT – Used to make SSL Connections through Proxy servers by making a request the Proxy to make a connection to another Host for a response back directly
• TRACE – This is a “Diagnostic” request to see what actions a Server takes when this request is received by the contents it sends back to the Client
• OPTIONS – This requests from the Server what Methods it will accept

Then there is of course the RESTful HTTP and CRUD Verbs from last post:

When studying with the Cisco Study Group I participated in, this was often tested in Python with a Library of Books, that contained “Titles” / “Authors” / Etc that you could GET / POST / PUT / DELETE to add / read / update an Authors name / remove a book from that library.

One great resource for this is in the “Start Now” section of the Cisco Developer website, but there are also several modules on the “Learning” part of Cisco developer website:

https://developer.cisco.com/startnow

https://developer.cisco.com/learn

I have a sticky on the top of my blog or by clicking here.

Time for a Deep Dive into two Methods you will use most as a beginner – GET and POST

GET is the first Method to really be comfortable with to practice working with APIs, as being a sort of read-only / fetch data request API Call, it is harmless to test against a resource (though of course don’t use this against production or resources you shouldn’t access).

GET requests are exactly the definition of “Idempotent” as no matter how many times you run it against a resource, it will never change the state of the system its being run against.

A GET request consists of 3 necessary pieces of information and 2 optional:

• GET – Methods must be in all uppercase letters, in this case it is GET
• URI – The Server / Host Address of the resourced to be queried
• HTTP Version – This is either HTTP v1.1 or v1.0, if the Client sends v1.1 and the Server only supports v1.0, it will send that back in response to the request
• Header – This is optional with GET requests, only used if specific data formats / languages / authorization is needed to access the resource
• Body – This is optional and contains the “Query String” which is yet to be reviewed

^^ For exam day, I would know Header / Body is optional, and that if a Client sends a HTTPv1.1 GET request to a Server that support v1.0 it will respond to the Client with that!

POST requests are EXACTLY the same format – Expect they contain a “Content Type” and “Content Length” within the Request as well, which is required in addition to the “POST” / “URI” / “HTTP Version” with the Header and Body (contains Query string) also optional.

HTTP Headers – Request and Responses Reviewed!

Headers help to define the Data represented in the API Call, and are divided into Request and Response Headers, the Request being what is sent by the Client, and the Response is what the Server sends back to the Client with information about the Response.

Pulling from my “ICNDB” API GET Request example, lets first look at Request Headers:

Then when the Response is received, the “Response Header” has more info about its Data:

I could almost highlight everything in this Response Header for good info, it gives the TimeStamp / Data Type used in Application / Access-Control (* for none) / Methods Allowed (GET only) / Cache-Control (None / Must Re-Auth each request) / Encoding Type (br)!

Though Optional API Request / Response Headers carry crucial details for critical items:

• Request and Response body details
• Request Authorization details
• Response “Caching” details (as seen in Cache-Control)
• Request and Response Cookies

The Response Cookie is actually not in the Response Header in Postman for this call, it is seen in its own “Cookies” column next to the Header Contents in the Response Header:

^^ Note this is the same Cookie from the “Request Header” that we got in response, using the “Timestamp” in the Header we can see the Cookie expires in 1 hour, and that it uses HTTP and is NOT secured as shown in the Cookie output itself.

Also one thing to note that I will zoom in on from the “Response” Header field above:

Proxy Servers or Services (like Cloudflare) will often use Key:Value Pairs along with Arrays as seen for “Endpoints” Key:Value Pair, though it is important to know that the “Content Type” of “application\json” refers to the contents of the Body – NOT this Proxy information!

Though most are seen here, I want to go through the full list of Header Information:

• Authorization – Credentials to Authenticate to access the resource
• WWW-Authenticate – Sent by Server in “Response Header” if it needs some kind of credentials to allow the request to proceed, will send respond code 401 Unauthorized
• Accept-Charset – Sent in “Request” to tell Server what character sets are accepted by the Client sending the Request
• Content-Type – Defines the data type sent in the Body of the Request / Response, when sent in the Request Header helps the Server process the Request faster
• Cache-Control – This is set by the Server, defines how many times a Header can be re-used

A high level view of Response Codes before a much more in depth look at each:

• 1xx – Informational – Server received the Request and is processing to Respond
• 2xx – OK – Request understood / accepted / fulfilled
• 3xx – Redirect – Resource has moved, Response will include further details for action required to fulfill the Request
• 4xx – Client Error – Bad Request / Unauthorized / Something wrong on Client Side
• 5xx – Server Error – Server is unable to process the Request

Then to jump right into the specific error codes and what they mean:

• 100 – Continue – Indicates Request received and Server is working to form a Response
• 200 – OK – Indicates the Request successfully fulfilled
• 301 – Moved Permanently – Resource moved permanently to a new location, Server will Response with new Server Address in Header, new Request must be pointed there
• 302 – Found but Moved – Resource is temporarily moved, new location will be in Response Header / Client will need to form a new Request just like a 301 error
• 304 – Not Modified – This is a special response to an “if-modified-since” conditional value in a GET request, if not modified the Server will indicate this with a 304 response
• 400 – Bad Request – Server couldn’t understand the Request, likely Syntax Error
• 401 – Authentication Required – Requests needs credentials to access Resource
• 403 – Forbidden – Client might have Credentials, but not Authorized to access Resource
• 404 – Not Found – The Resource cannot be found on the target Server
• 405 – Method not Allowed – The method being used is not accepted by Server / Resource
• 408 – Request Timeout – The Request was received but took to long to reach the Server
• 414 – Request URI too large – URI Supplied by Request is larger than Server will accept
• 429 – Too many Requests – Rate-Limiting has kicked in to stop a flood of Requests sent
• 500 – Internal Server Error – Server is unable to process the request for unknown reasons
• 501 – Method Not Implemented – Method used in Request is invalid, likely caused by typing “Get” instead of “GET” and Server does not understand the syntax of Method
• 502 – Bad Gateway – Proxy or Gateway received bad response from Upstream Server
• 503 – Server Unavailable – Server may be oversubscribed, indicating to try again later
• 504 – Gateway Timeout – Proxy or Gateway received a timeout from upstream Server

^^^ I would make flash cards for those for exam day, I personally fumble some of the more obscure response codes (and even the simple but similar ones), so get those down cold!

Status 429 “Too many requests” is from Rate-Limit being triggered by a flood of requests, this is related to any resource like “WebEx” if it begins to receive too many requests.

A good website I might plan to use (and possibly share my Google Drive flash cards here):

https://www.cram.com/flashcards/create

The different Structured Data Formats used with RESTful APIs – XML, JSON, YAML

XML

XML encodes its data between descriptor tags and is a variation of HTML, or “superset” of HTML, which of course was originally built to describe communications of webpages with Web Servers via HTTP.

JSON

The Value in Key:Value pair must be one of the following data types:

This begins with Key {:Loopedback_Employees”: with the following values demo’d here:

• Integer / Float (Number)
• String (String Literal)
• Array (Can be embedded in single string, or down script shown here)
• Boolean (True / False)
• Object – An object in JSON is a properly enclosed {Key:Value} Pair
• Null – Empty object {} or array [] Brackets, can have white space

I would practice copying this string from Visual Studio Code, and pasting it into Notepad if you are on a PC, so you can see this logic on a single string shown here in plain text:

{“LoopedBack_Employees”:[ {“EmployeeNumber” : 1,“Name”: {“First”: “Dave”, “LastNames”: [{“Madein Name”: “Anderson”, “Hypenated Name”: “-Loopy!”}]},“Email”: “Dave.Anderson@Loopedback.com”,“FIRED!!!”: false},{“EmployeeNumber”: 2.87,“Name”: {“First”: “Loopy”, “Last”: “Doopy”},“Email”: “Loopy.Doopy@Loopedback.com”,“FIRED!!!”: true},{“EmployeeNumber” : 3,“Name”: {“First”: “Some”, “Last”: “Body”},“Email”: “Some.Body@Loopedback.com”,“FIRED!!!”: {}}]}

I wanted to color in the Key: in Red with the Value in Blue, and the Brackets that make up Objects and Arrays as Black, as for me that helps me visually digest that mess of code 🙂

Some things to note about the formatting of this script on a single line:

• White space does not matter with JSON, scripts or single line
• JSON is one object, comprised of nested Arrays and Objects
• A JSON Object is a Key:Value Pair
• JSON Objects that are part of the same “Value” are comma separated except for the very last object before the } or ] closes the Object / Array!

For the last one I personally think of the Key: <— as keeping that Colon, so then everything to the right side of that is considered the “Value” of that Key:, until closed by a bracket.

I’d play with that formatting and perhaps coding it from a single run on line and pasting it back into VSC to see if you can parse it out correctly so there are no grammatical errors.

YAML

Like XML is a “superset” of HTML, YAML is considered a “superset” of JSON that is considered the most human readable serialization language, which has a very tight data structure:

When it comes to YAML and DEVASC day I’d know the differences in 3 YAML format terms:

• Scalars – The Key:Value pair in the YAML File
• Sequences / Lists – Ordering Data by Indexes
• Dictionary Mappings – Scalars with nested data (May include other Data Types)

The Scalar / Dictionary Mappings can be seen here with Nested Scalars inside a Scalar:

All these languages work essentially the same, if you are comfortable with Serialization / De-Serialization with Python, API formatting is the same with some extra terminology.

WebHooks

This is one of the things I really love about Meraki, I cannot wait to get back to reviewing it here soon, is WebHooks among many other alerting they offer for networks(!):

I used https://webhook.site to general a URL to create a Test WebHooks in “Network-Wide -> Alerting” at the bottom of the page, and the really cool thing is the output on the site:

It clearly shows this was a user-defined, event driven POST message via HTTPS to this web Server, printing out things like the Network Names / ID / other information I’ve blotted out.

WebHooks are known as a Reverse API, as instead of being intentional, it is reactionary in Nature when it makes its POST Call to a Server, and those this WebHook Test Site does not offer any kind of “validation” this does exist as clearly shown in the “Shared Secret” portion.

This Validation Token is a unique Token generated by the Target WebHook POST server, this validates it both on the Server and Client side, as the Token should match on both sides when the triggered Call is sent a Response is sent back to the Client – The tokens should match.

The Target server can also invalidate or revoke the Token at any time, the same way the Meraki API Key can be revoked on Dashboard at any time, immediately cutting off access to any systems using it to perform Automated tasks.

Note for exam day – When using Tokens / WebHook validation, the service will send an initial GET Request for “/validationKey” and the WebHook Server will respond back with the Token, then the event driven POST WebHooks will be received properly!

With all the talk of POSTs with WebHooks, it is important to note that for this Authorization / Validation Token, the initial communication will be a GET for that Token from the Server.

NGROK is also a solution / Tool for testing WebHooks as well as many other services, as it allows you to Tunnel over a Public URL to a locally running service, I’ve played with it while trying to test WebEx Bots in Windows OS (it didn’t go great) but will be trying again and documenting here in my Ubuntu VM for a proper environment to run it in.

Sequence Diagrams for REST APIs

I made an example REST Sequence Diagram awhile back that I will review here:

The Y Axis should be the flow or timeline of events, while the X-Axis explains the events themselves or the flow of them, this is an example of how someone with the API Token from a Meraki Dashboard could send a REST “POST” to configure all networks within an Organization with a standardized SNMPv3 config with a uniform username / password.

This could also apply to any web service, like the search function on this website could be explained through a similar flow chart, with “WordPress” being the Front End where the API GET Request is sent to the WordPress Database, which then responds back to that GET Request with a list of Webpages that meets the search criteria.

Main Takeaway – Flow Charts should show the communication from the Client (your PC) to the Front End service (like Google), and how the Google Front End / Webpage communicates with Back End Databases / Servers and how the Back End systems respond.

The 6 REST Constraints – The things that make an API “RESTful”

RESTful APIs are meant to be universal in how they operate, so the “Constraints” are the rules they must abide by to be considered “RESTful” APIs, there are 6 rules in total:

• Client / Server
• Stateless
• Uniform Interface
• Cache
• Layered System
• Code on Demand

Client / Server

This means the Client and Server must operate completely independent of each other, the Client only needs the URL or URI to initiate a request, and the Client always will initiate the Request and the Server only Responding to each individual request as it receives it.

Stateless

REST Services must be “stateless” to be considered RESTful, meaning Request sent to the Server should contain all necessary information to process and Respond, this allows easier load-balancing of traffic to multiple Servers to process the Requests faster / more efficiently.

Uniform Interface

This Constraint has 4 sub-constraints that make up “Uniform Interfaces” which are:

• Identifiers of Resources – Target URL / URI / Resources should be Static for consistency, even if the Resource changes the Identifier to reach it must remain constant
• Self Descriptive Messages – Each message should contain all information necessary for the Server to easily understand and process the request / response
• Resource Manipulation through Representation – “Representation” refers to the formatting of Resource Manipulation in terms of Data Format accepted by the Server (XML / JSON), which the Client may change and the Server can Reject
• Hypermedia as the Engine of Application State (HATEOS) – Server Responses should embed links into Responses to applicable resources the Client can use

The URI / Resource name should be static and make sense, messages should be descriptive enough to be easily processed and formatted or “represented” properly, and Server Responses should contain HTTP Links to other resources for the Request as needed.

Cache

The Server must state whether the Response is Cacheable or non-Cacheable, and how long the Response Cached copy will remain Valid, allowing the Client to use its Cached Response from the Server thus not needing to make additional Requests / Optimizing performance.

Layered System

This expands on the Client / Server model on the Server side Infrastructure or Architecture, meaning that Proxies / Load-Balancers / Etc are “Layers” of the System, and should be easy to add or remove Layers without disruption of the Service.

Code on Demand

This is an Optional Constraint (good to know for exam day), this states that a Client is able to download Code from the Server in its Request, and the Server will Respond with the code requested generally in the form of a script which is executable by the Client.

REST API Versioning / Version Update strategies

REST Versioning is being able to update the API design without breaking the function it performs while updating the Version, there are 4 Methods of REST API Versioning:

• URI Path Versioning – This Method uses the Version # of the API in the URL
• Query Parameter Versioning – This Method uses Query Parameter (?x=1.1) is an example of the Parameter portion of the URL, rather than just building it into the URL
• Customer Header Versioning – This Method uses custom Headers that contain an added ‘attribute’ within the header, this avoids need to alter the URI / URI Query
• Content Negotiation Versioning – This Method allows the “Resource Representation” to be Versioned rather than changing the URL as well, and gives most granularity

To review quick, the first two methods or strategies utilize the URI Path to point to a specific Version, while the other two use the API Header / Representation to Version a resource.

REST API Pagination

Pagination limits the output allowed to be returned from a Request, similar to how your email inbox limits the emails you see at one time / on one page, Pagination does this for REST API Requests using two different methods of applying this Pagination:

• Off-set Pagination – Uses API Parameters “offset” and “limit” to define the # of results allowed to be returned within a range of #’s
• Key-set Pagination – This is the recommended method of Pagination as it uses a set # of Pagination that does not use arbitrary values, a “Continuation” or “Cursor” token allows for a specific set of Queries to be sent / received

Rate-Limiting / Monetization

This is built into REST APIs by developers for several reasons on the Server side:

• Efficiency – A flood of Requests will eventually bottleneck resources, causing it to under perform at high traffic volume times, and will make the API seem broken
• Business Impact – Software that doesn’t work when its most critical that it does work is bad for business, so API Rate-Limiting is used to keep a consistent working state during slow and busy times, showing stability and therefor more value
• Security – Allowing API Servers / Resources to be hammered with unlimited Requests opens up the door for Malicious attacks while the system is overwhelmed by either an intentional or unintentional DDoS attack from the Service being used

Rate-Limiting is very importantly about Security first a foremost, but its also about making a stable piece of Software that shows value, so the rate of Requests should be factored into decisions surrounding Resources / Money charged for higher rate-limits as they may be required by a customer but that means you have cost in Hard Resources – This means considering Tiered pricing for different TPS or Transaction Per Second.

Once a Server hits the Rate-Limit, it will begin returning an HTTP Status Code of 429 “Too many requests” with a “retryAfter” value in the Response Header from the Server, letting the Client know when it can send its next Request.

Rate-Limiting on the Client side touches on some topics already reviewed:

• Using Cached Responses where possible to limit Requests from Clients
• Avoid excessive polling via WebHooks to trigger updates
• Downloading Data during non-peak hours where possible
• Hone your Query Filters to target more specific resources

Both of these should be deployed from end-to-end ideally, however developers may not control the client side of things, but Rate-Limiting can be done on both sides!

REST Tools

Postman

Postman is a big one for REST API as it can do so many things we love as Developers:

• Learning API Request / Response with a simple GUI as shown above
• Creating “Collections” of commonly used Requests, or Requests that must be run in a sequence, Postman Collections allow you to sequence Requests for execution in order
• Importing / Exporting Collections to the larger Developer community
• Writing Test Scripts within the body of a Request to execute remotely
• Chaining Requests which allows Response Output to trigger follow up Requests
• Generating Code Samples in different Data Formats

^^^ That last one can be seen here, which is actually another “Rest Tool” – “curl”:

My “GET” Request to Chuck Norris jokes can be displayed in all these different languages!

If you don’t have Postman, go download it for free, and Google “rest api test” for tons of different resources to play with the tool, Postman is a powerful one for REST APIs!

“curl”

Though displayed above in Postman, this is generally used in a Bash Prompt that is available on most platforms and systems, in Linux you would “pip install curl” to download it.

curl sends API Calls directly to a URI, and has a few common modifiers to use with it:

• -X – Specifies HTTP Method, default is “GET” Method
• -d – Tells curl to pass Data into remote Server either by embedding data in the curl command or by using a file to pass data
• -D – Dump Header information of Request
• –b – Tells curl to pass along cookie data
• -H – Tells curl to add an HTTP Header to the Request
• -insecure – Ignore HTTPS Certification Validation
• -s – Tells curl to execute attached modifier in “Silent mode”

This will form a line that looks something like some examples below:

curl -sD -X GET https://loopedback.com/get?s=ThisPost

curl -sD -X POST https://loopedback.com/post -H ‘Authorization: Basic’ -H ‘charset: something’ -d ‘Hello World!’

This will execute the API Call just like Postman, and will return the Response in Bash:

This is how it is supposed to look, a GET Request and Response, whereas I made typo in “ranndom” in another call, and got a flood of output in Response:

^^^ Looks like this Server / API could use some Server Side Rate-Limiting for typos 🙂

HTTPie

I have not used this yet, but at https://httpie.io/run you can try it embedded in the page:

Its very much like the “Visual Studio Code” of REST API Calls, it offers quite a few features:

• It is very intuitive in both Syntax and runs JSON Natively
• Very colorful to assist with viewing Syntax / output
• Can utilize HTTPS / Proxies / different Authentication types
• Comes with support and lots of downloads / extensions to customize

I won’t spend more time than that, I am not sure it’d be seen on exam day, but its worth a quick look at the Tool for the sake of completion.

Python!

For Python to work properly with API Calls from Bash, you must install “requests” library:

Using “import requests” then “dir(requests” I can view all the functions in the Library:

However this would be a script run from Bash, not the IDE within Ubuntu, so to VSC I go:

Then from the Bash Command prompt running this “GET” against the Chuck Norris Jokes:

Then I change the “GET” on line 7 to “TRACE” to see what happens, and get an error code:

That Method is NOT allowed by this API Server! And we know that because the status codes are all posted up there^^^ 🙂

Now if I wanted to try to post my own Chuck Norris joke to the Database:

Though oddly it gives a very similar Response with a 200 OK and a Joke back:

That was kind of odd it allowed a POST to the Database, I am curious if I added to the random jokes or if it just defaulted back to a “GET” Response.

Finally one thing I want to point out is embedding Authorization into Python Scripts:

Or because it is JSON formatting it can also be written in a JSON format / comma separated:

This is all I will cover here, there is a lot more depth I will be labbing before exam day in another post, but this is the main concepts I want to add in this REST API Deep Dive post.

All of these Tools / Strategies can be used for REST API Debugging / Troubleshooting!

Using Error Codes, Methods like “TRACE” in the API Call, using Python to send test API Calls and see what response comes back are all good ways to debug / troubleshoot a REST API.

Status Codes will be key in both troubleshooting real world issues, and especially on exam day, I still have to reference them on this page so I will be using online flash cards to have those down absolutely solid before exam day.

With that, I am calling this review of all things REST API finished and moving along!

I have another post that describes different methods of Authentication / Rate-Limiting such as Token Bucket / Leaky Bucket / lots of extra stuff not reviewed here, so I might go take another look at that post on REST APIs to get as much info as possible before exam day.

Until next time!!! 🙂