4 posts tagged with "openapi"

In the stackql project we needed an API to serve configuration file packages (stackql providers) to the stackql application at runtime.

Traditional artifact repositories or package managers were unsuitable as they were mainly designed for container images, JavaScript modules, Python packages etc. The artifacts, in this case, are signed tarball sets of OpenAPI specification documents (text files).

We have recently moved our provider registry (stackql-provider-registry) to use Deno Deploy as the serving layer (the API).

The code​

The code is reasonably straightforward as shown here:

The deployment​

We are using GitHub Actions to push assets and code to Deno Deploy, this was straightforward as well as you can see here:

if you have enjoyed this post, please consider buying me a coffee ☕ to help me keep writing!

This article walks through the process of converting service specific OpenAPI specifications from Google Discovery REST URLs using Python.

Full code for this article can be found at stackql/google-discovery-to-openapi

Google publishes JSON specifications for all of their APIs (including GCP services as well as other APIs associated with other products - like analytics or workspace). These specifications can be accessed without authentication starting with the root document (https://discovery.googleapis.com/discovery/v1/apis) which contains metadata and the URL for each service specific document (for services like compute or storage).

Code Overview​

The program fetches the service document for each service that is included and not explicitly excluded (configured through variables in the program). Non preferred services (beta or alpha versions) can be included by setting the variable get_preferred_only to False.

An OpenAPI spec is constructed for each service based upon the data in the service discovery doc. In many cases this is a straightforward one to one mapping, such as to top level info, title and description values, it gets more complicated with parameters and schemas where some extra logic is required to keep the json pointers ($ref) valid.

Extracting Paths and Verbs​

The real magic is in extracting paths and verbs in a compliant OpenAPI format, as Google nests this data (potentially multiple levels deep) under resources.

The first step is to identify methods nested under a resources object (which can be mapped to operations - with a path and HTTP verb required to populate an OpenAPI spec), this function does this:

Now each method can be processed yielding an operation (combination of path and verb), this is done using this function:

Full source code can be found at stackql/google-discovery-to-openapi.

if you have enjoyed this post, please consider buying me a coffee ☕ to help me keep writing!

I had a scenario where I needed to find values for a key in a complex JavaScript object which could be nested n levels deep.

I found numerous approaches to doing this, most were overly complicated, so I thought I would share the most straightforward, concise process.

the Code​

You can do this in a straightforward function implementing the "tail call recursion" pattern to search for a key (key) from the root of an object (obj), excluding any keys in excludeKeys.

This will return a list of values for the given key, searching all levels in all branches of the object.

function getAllValuesForKey(obj, key, excludeKeys=[], values=[]) {
    for (let k in obj) {
        if (typeof obj[k] === "object") {
            if(!excludeKeys.includes(k)){
                getAllValuesForKey(obj[k], key, excludeKeys, values)
            }
        } else {
            if (k === key){
                values.push(obj[k]);
            }
        }
    }
    return values;
}

Example​

In parsing an OpenAPI or Swagger specification, I am looking for all of the schema refs in a successful response body, for example:

paths:
    '/orgs/{org}/actions/permissions/selected-actions':
        get:
          ...
          responses:
            '200': '...'

however these refs can present in various different ways depending upon the response type, such as:

'200':
    $ref: '#/components/responses/actions_runner_labels'

or

'200':      
    content:
        application/json:
          schema:
            $ref: '#/components/schemas/runner'

or

'200':
  content:
    application/json:
      schema:
        anyOf:
          - $ref: '#/components/schemas/interaction-limit-response'

or

'200':
  content:
    application/json:
      schema:
        type: object
        required:
          - total_count
          - runners
        properties:
          total_count:
            type: integer
          runners:
            type: array
            items:
              $ref: '#/components/schemas/runner'

To find all of the schema refs without knowing the response type or structure I used the above function as follows (excluding refs for examples):

function getRespSchemaName(op){
    for(let respCode in op.responses){
        if(respCode.startsWith('2')){
            return getAllValuesForKey(op.responses[respCode], "$ref", ['examples']);
        }
    }
}

You can find this implementation in openapi-doc-util and @stackql/openapi-doc-util.

simple!

if you have enjoyed this post, please consider buying me a coffee ☕ to help me keep writing!

Open API specifications can get quite large, especially for providers with upwards of 500 routes or operations.

The challenge is to create standalone documents scoped by a service or path within the parent API specification and include only the components (schemas, responses, etc.) that pertain to operations included in the child document.

When I went looking for library or utility to do this, I couldn’t find one... so I have developed one myself.

It's a simple command (nodejs based but can be run in a bash terminal or from the Windows command line) which requires a few options, including:

• the provider name (e.g. github)
• a provider version which is a version you set - allowing you to make minor modifications to the output documents (e.g. v0.1.0)
• a service discriminator which is a JSONPath expression to identify a service name within each route in the parent file, this is used to assign operations to services in separate documents (e.g. '$["x-github"].category')
• an output directory (e.g. ./dev)

and of course, the openapi spec document you are splitting up.

an example is shown here:

openapi-doc-util split \
-n github \
-v v0.1.0 \
-s '$["x-github"].category' \
-o ./dev \
ref/github/api.github.com.yaml

Help for the command is available using openapi-doc-util split.

The net result is 59 self-contained, service scoped documents, containing only the components referenced by routes in the service document.

You can access this utility via NPMJS or via GitHub.

Splitting up a large open API spec document, is the first stage in developing a StackQL provider which we will discuss next time!

if you have enjoyed this post, please consider buying me a coffee ☕ to help me keep writing!