New connection plugin for Arista eAPI

[Qalthos]

Proposal: eAPI connection plugin

Author: Nathaniel Case <@Qalthos>

Date: 2018-03-27

• Status: New
• Proposal type: network
• Targeted Release: 2.6
• Estimated time to implement: 4 weeks?

Motivation

Implement eAPI as a first order connection plugin like connection: network_cli & connection: netconf
Allows deprecation of connection: local & provider: for Arista devices

Problems

• As a user, I want to manage Arista devices that can only be accessed via eAPI
• As a user, I want to connect to an Arista device over eAPI without using connection: local and provider so I can use basic Ansible connection options instead, including specifying http(s) port and encrypted or non-encrypted transport type (HTTP/HTTPS)
• As a user, I want existing playbooks using connection: local to continue to function as before without user intervention
• As a user, I want to securely connect to eos devices via a proxy
• As a user, I want to securely connect using using username & password or a certificate
• As a user, I want to have enough information to debug when the connection/authentication fails
• As a user, I want to be able to control my operating mode (such as privilege mode)
• As a user, I want to be able to reset the connection using the Ansible meta handler reset_connection

Solution proposal

This connection plugin works like the existing network_cli and netconf plugins, and should have the following properties

• When the connection is created
  • builds endpoint URI from connection options
  • check that network_os is set and load appropriate cliconf plugin
• When _connect(self) is called, ensure socket_path is valid
• close(self) is a no-op
• put_file, fetch_file, and exec_command all redirect to an internal local connection, to allow non-networking modules to function on the local host, as is done in existing network connection plugins.
• When send(self, command) is called
  • a JSON-RPC request is built from the command
    • This will either be done in the connection or sent to an api request builder plugin based on network_osdepending on applicability with nxapi
  • The request is sent to the URI built earlier
  • The response from the device is unpacked
    • If the response indicates an error occurred, raise AnsibleConnectionFailure with the details
  • Return the response

Testing

• Add eAPI connection case to existing eos integration tests
• Move all connection: local tests to single test covering both transports
• Each user story should be defended
• High code coverage for new code

Documentation

• EOS docs fragments need to be updated to remove mentions of "This option is only required if you are using eAPI" with reference to connection: local
• EOS platform specific page needs to be altered to replace any reference to connection: local with the new connection type
• The network section of the become guide should be updated to include the new connection type

[privateip]

close(self) is a no-op

Just for clarification here, close is a noop on the connection side but still closes the local socket.

[jmcgill298]

I think there needs to be a more generic http connection plugin that could support or be the base of support for several different APIs. @dagwieers worked some on a connection plugin for ACI, and has done some work with different authentication/cookie handling with the current ACI connection code:

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/network/aci/aci.py#L291

ACI supports username/password and certificate authentication, and uses cookies to maintain session after login.

[privateip]

@jmcgill298 Can you provide more details specific to this comment?

I think there needs to be a more generic http connection plugin that could support or be the base of support for several different APIs.

Specifically, It would be good to capture the user stories that you would want to support. Given that 2.6 is a very short turn around release, we would need to have this discussion sooner rather than later or the work will be at risk.

[jmcgill298]

what do you mean by user stories?

[privateip]

Look under the Problems section in the Proposal. Each line there is a user story that basically lays out a generalized requirement. That will form the basis for developing and testing the solution. Each bullet should be reflective of an action (expectation) you have from the connection plugin code to be developed.

[jmcgill298]

I would say that the most important benefit is to the developers and contributors. A single code base is easier to maintain and troubleshoot, and an existing http connection plugin allows new APIs to be added to the support matrix more easily than requiring someone to learn how the connection plugins works before learning how to write modules.

Users have multiple devices that have a mix of connection options, and some which are realistically only http. You could take the above user stories and substitute aci, f5, nxos, citrix, infoblox, aws, etc.

[privateip]

I would say that the most important benefit is to the developers and contributors. A single code base is easier to maintain and troubleshoot, and an existing http connection plugin allows new APIs to be added to the support matrix more easily than requiring someone to learn how the connection plugins works before learning how to write modules.

Sure but we still need to work off of requirements and use cases

Users have multiple devices that have a mix of connection options, and some which are realistically only http. You could take the above user stories and substitute aci, f5, nxos, citrix, infoblox, aws, etc.

Again we need to catalog the user stories in order to design the connection plugin correctly. This particular proposal calls for implementing JSON-RPC using an HTTP/S transport. That is not what ACI uses for instance. If I'm not mistaken, it uses REST. I can't speak intelligently to all of the platform requirements hence why we need to capture the user stories.

[jmcgill298]

I perceive the question to be, how do all the other APIs work, which to me seems unrelated to user stories, the users want functionality, the details are insignificant. I put in my first response that Dag has done more work with ACI on authentication and the new mechanisms they offer in 3.x versions. ACI is a REST API, uses username/password or certification for authentication, returns a cookie for session reuse, and can use either xml/json for input/output.

I know that F5 offers SOAP and REST APIs, but not sure of all the mechanics of them

[privateip]

User stories define the requirements of the connection plugin. You have stated you want a more abstracted connection plugin for HTTP based connections. This particular proposal defines a connection plugin for using JSONRPC over HTTP. If you want this proposal to cover your use cases, then the requirements need to be added as user stories.

Conversely if you want to have a discussion around a more abstracted connection plugin that will support any messaging structure over HTTP (such as REST or SOAP) and you do not want it tied to this connection plugin, then I would recommend writing a proposal and/or bringing it up in IRC, preferably at an IRC meeting so everyone can weigh in on the discussion.

We have committed to delivering a first order connection plugin for implementing eAPI in Ansible 2.6. That can either be a targeted connection plugin such as this proposals indicates or a more generic HTTP connection instance with a pluggable messaging system built on top of it. I'm not inflexible about which approach we take here; however, since this needs to be delivered in Ansible 2.6 and there is a short window for new features, we need to have the discussion in the next couple of weeks otherwise, it becomes a 2.7 deliverable.

[jmcgill298]

Conversely if you want to have a discussion around a more abstracted connection plugin that will support any messaging structure over HTTP (such as REST or SOAP) and you do not want it tied to this connection plugin, then I would recommend writing a proposal and/or bringing it up in IRC, preferably at an IRC meeting so everyone can weigh in on the discussion.

Please read through this weeks meeting, I did exactly this and was told to copy my comments to this proposal, which is what I have done.

I would say that it is tied to this proposal though, as it shouldn't be several http connection plugin recreating the same code over and over again. I do suspect that there will need to be divergence across the APIs, so that there will likely need to be custom plugins for APIs, but could use base functionality from the generic plugin.

[dagwieers]

I am thrilled with this, because what I have been looking at for ACI was a big hack :-/

Personally, I think we should investigate all the different module-sets that we currently have using HTTP and REST-based API's. Look in what ways they differ, and how they can use a common class that each can extend given specific needs. This will ensure we design something from the ground up with at least the common/existing modules in mind.

We already have a large set of modules (AWS, VMware, ACI, ...) that could leverage connection plugins but just need this common framework. And granted, we might as well end up with 2 or more main classes (e.g. JSON-RPC and REST) but the generic interface would look similar.

I am convinced that if we do this right from the start, we can get a lot more contributors joining in making connection plugins for their modules (with a much improved user-experience), at least it will be well-documented (by example) because of this. This is leveraging the existing community (better) before/during implementation.