OpenApi Export and Swagger UI Binding

[Nanashi-lab]

/closes #1178
/claim #1178

Docs is still left to do. Golem-Cli needs to be updated to latest.
CI api-and-integration-tests, API Test keeps timing out after 45 mins, I will figure and fix this.

I am ready for a review @afsalthaj I have added the explanation comment on content-type and status. Do tell if I have that right ? and if so is the current way of handling it is robust ?

Link to Videos and Files -

Video One - Link
This covers the Export Endpoint, and the Export CLI Command, and Round-Trip Test

1. Add a bunch of API's
2. Export the APIs in OpenApi yaml format
3. Delete all the original APIs
4. Import the OpenApi yaml APIs
5. Checkout the /export endpoint

Video Two - Link
This covers the Swagger CLI Command, and Swagger UI Binding

1. Add Api for test:llm from golem-llm
2. Deploy the Api which includes Swagger UI binding
3. Access the Swagger UI binding endpoint and run through the invoke tests
4. Showcase swagger command in CLI for both undeployed and deployed API

If you want to check the actual conversion between AnalysedType and OpenApi Spec, you can either

1. Get the wasm files, api files and openapi conversion files from repo link
2. or You can find detailed conversion in file api_oas_convert_tests.rs

[Nanashi-lab]

The Changes Requested

Done

• Improved the de-serialization for GatewayBindingType, we no longer use custom de-serialization, which was previously used for wit-worker
• export_api_definition name has been changed export_openapi_spec
• Removed the pub from name and version in ResolvedGatewayBindingComponent
• We now use CompiledHttpApiDefinition instead of HttpApiDefinitionResponseData
• Removed the macro, now it is a function, for figuring out the analysed type to OpenApi for Integers
• If all {header, status, body} are missing, it tries to match the whole record to AnalysedType, if there is no record, it tries to match it what is available with AnalaysedType
• default_object_schema was a mistake, now if there is no requestbody or no response, it sends None. So swaggerui understand there is no request or no response.
• _ = "Response" , we now use http crate to figure out the Description
• We now use CompiledRoute instead of RouteResponseData, part of the change from moving to CompiledHttpApiDefinition
• request.header are now parsed and part of the OpenApi Schema
• we do now parse header, body and status into separate parts and we use this data to generate the yaml
• We reuse deserialization for names, like "swagger-ui", "default", "file-server" from golem-common
• The Cors response is now directly calculated using from_http_cors which converts into a valid response
• Swagger-UI Binding, now has the openapi_yaml spec, When compiledhttp is created, we compile with openapi_yaml set to none, once it is compiled, we generate the open api spec, update all swaggerui route. The swagger ui handler now uses this directly to generate the swagger-ui. A Cleaner solution
• workerNameInput, is not there are in test anymore since we moved to CompiledHttpApiDefinition
• One top of all the tests before, I added one api with 10 routes, each testing a wide variety of Analysed type.

Minor Unresolved

• route.path is AllPathPatterns, hence the .to_string()
• For query, path and header, I haven't implemented a is_primitive check ? I was not sure what to do in the else part the statement.

Major Unresolved

• Content-Type, json vs text. Because RibOutputMapping only has the type value, and not the literal value, this is currently not possible. headers and status are values which come from runtime. There will be a comment with more detailed explanation below.

[Nanashi-lab]

@afsalthaj this is the explanation. I have addressed almost all comments from old PR, with the major exception of content-type in response.

Content-Type and Status

RibOutputTypeInfo only contains the AnalysedType, and not the actual value of status and header (Content-Type). The exact values are calculated at runtime.

Both are valid, working rib-script. In first we can set status at runtime, and in second we can set content-type during runtime.

let input = request.body.input;
let worker = instance("worker");
let result = worker.echo("Test");
{status: input: u64, body: result}

let input: string = request.body.input;
let worker = instance("worker-static");
let result = worker.echo("Test", 3);
{headers: {Content-Type: input, userid: "foo"}, status: 200: u64, body: result}

For the Status Case -

I determine the status by using method (POST -> 201, GET-> 200, and so on) and for all other cases (Where rib may output multiple types {200, "Succes"} and {400, "Error"}) we use Default Response

  /v0.0.1/test1:
    post:
      responses:
        default:
          description: Created
          content:
            application/json:
              schema:
                type: string
        '201':
          description: Created
          content:
            application/json:
              schema:
                type: string

For the Content-Type Case -

Current Golem-Rib behavior for String and other Primitive types, if you try application/text or application/octet-stream or any valid name. It will try to output it in that form, including image/png. For application/json which is the default, it outputs json.

If the AnalysedType is Complex, for all other content-type it will output a messaging "So and so analaysed-type could not be translated to content-type". for application/json it will output the json

In the first-case currently in api-oas-convert we pass application-json, if swagger ui receives application-text it handles it gracefully.

[afsalthaj]

Thanks @Nanashi-lab for raising the PR again

Content-Type, json vs text. Because RibOutputMapping only has the type value, and not the literal value, this is currently not possible. headers and status are values which come from runtime. There will be a comment with more detailed explanation below.

Could you please explain this in detail? All information should still be available in the rib output type info.

Also I hope this PR addresses the concerns raised I in this PR. #1454

[Nanashi-lab]

More thoughts on RibOutputTypeInfo on figuring exact values of status and content-type
Below is the static rib script, with no calls to workers, and predefined status, body and header values.

{headers: {Content-Type: \"application/text\", userid: \"foo\"}, status: 200: u64, body: \"Success\"}

RibOutputTypeInfo

          "analysedType": {
            "fields": [
              {
                "name": "headers",
                "typ": {
                  "fields": [
                    {
                      "name": "Content-Type",
                      "typ": {
                        "type": "Str"
                      }
                    },
                    {
                      "name": "userid",
                      "typ": {
                        "type": "Str"
                      }
                    }
                  ],
                  "type": "Record"
                }
              },
              {
                "name": "status",
                "typ": {
                  "type": "U64"
                }
              },
              {
                "name": "body",
                "typ": {
                  "type": "Str"
                }
              }
            ],
            "type": "Record"
          }

This is the RibOutputTypeInfo Structure, the actual values for status, body and header are in the rib-expression and the exact value is calculated when the api is called. RibOutputTypeInfo contains NameTypePair and not ValueAndType

We cannot extract content-type, status etc from rib-expression, because it can be set by the user at runtime, also string scanning might lead to false information from comments example // application/text.

[Nanashi-lab]

Another way to handle this would be, since application/text or application/xml both only handle primitive types like string

If the body is primitive, we can add all possible outcome

I am going to move forward with the solution @jdegoes & @afsalthaj if you have any comments on status and content-type issue, and how I am handling it, please tell me. (Look at comment above for more information)

responses:
  '200':
    description: Success
    content:
      application/json:
        type: string 
      application/xml:
        type: string
      text/plain:
        type: string
      application/text:
        type: string 

[jdegoes]

@afsalthaj Perhaps Rib can return "singleton types" when possible.

It's not likely the content type or status code is a variable: it's likely it's a literal, embedded into the Rib script. If Rib had any kind of support for singleton types, e.g. SingletonStr(String), which implies a weak form of subtyping (because the literal type "foo" is a subtype of String, meaning it can be substituted anywhere a String expression is expected), then OpenAPI export would have all the information it needs to precisely specify status code and headers.

[Nanashi-lab]

Content-Type will be mostly literal. Status can have a few possible Literal values, for result and variant.
example for result,
status = match result { err(_) => 400: u32, ok(_} => 200: u32 }, If Riboutputtypeinfo could have a list of all possible values that would great.

In OpenAPI v3, */* is a valid content-type, so it will be possible to do something like this

responses:
  '200':
    description: Success
    content:
      application/json:
        type: string
      '*/*':
        type: string   

This works also for complex types, because for complex type in application/text or others, output string error

[Nanashi-lab]

Most of the code is done. Once the merge between cloud and oss happens, I will rebase, and fix any leftover issues.