allow setting Kubernetes node affinity in KFP pipelines

[mikekaram]

Similar to #2823
Is your feature request related to a problem? Please describe.
It is common practice in Kubernetes to select where pods are running by providing a node affinity to pods that match a node label or field, thus making sure that the pod will be scheduled to specific nodes that meet specific requirements (e.g., belong to specific zones or regions, have federated non-shareable data, have custom HW that the vendor hasn't created any device plugins and admission controllers yet or other use cases).

Currently, there is no way to specify a Node affinity in Elyra Kubeflow pipelines, this means that the selection of the node for running Elyra pipelines can be done only with node tolerations, a cumbersome practice in certain use cases.

Describe the solution you'd like
We can resolve this by allowing users to specify a list of "node affinities" in a similar way to the "node taints" (see PR #2848).

We only need to allow the user to set the type of node affinity (requiredDuringSchedulingIgnoredDuringExecution or preferredDuringSchedulingIgnoredDuringExecution), (weight*), key, operator, and value(s).
* Since only the preferredDuringSchedulingIgnoredDuringExecution type needs a weight for its definition, I think it's an implementation decision if the user is allowed to select this parameter as well.

Additional context
See the Assigning Pods to Nodes k8s docs
See kfp.dsl.BaseOp().add_affinity() for more information about adding affinity in KFP
See the Node Affinity v1 core spec for details about how a node affinity is defined.

[lresende]

I understand this is common for Admins, but how do we expect a data scientist working on this to know how to set these properties? My humble opinion is that we should expose nobs (e.g. GPU count) and based on that we internally would have the capabilities to translate this to whatever is needed in the runtime-specific way.

Having said that, I will not oppose adding these if you provide a pull request.