app mesh issues

gistfile1.txt

AppMesh-K8s integration API issues

Clarify concepts:

Below are my understanding of AppMesh’s object’s corresponding Kubernetes constructs.(correct my if i’m wrong)

1. *AppMesh VirtualService* is like a *union* of K8s Service and K8s Ingress.
    1. when a VirtualService is using a VirtualNode as backend, it’s similar to a K8s Service.
    2. when a VirtualService is using a VirtualRouter as backend, it’s similar to a K8s Ingress.
2. *AppMesh VirtualNode* is like a K8s Deployment or ReplicaSet, which groups a set of K8s Pods. 

Current issues in API:

1. VirtualService CR is a namespace scoped, however the namespace is not respected. 
    1. Under different namespaces, two K8s VirtualService can co-exist and representing a single AppMesh VirtualService object. e.g. *namespace-a: my-api *and* namespace-b: my-api* will both referencing the same AppMesh VirtualService *my-api*, thus will cause endlessly reconcile them back and forth.
        1. note: The current solution in various examples is to ask user to follow K8’s name style for services DNS as name for VirtualService. e.g. *my-api.namespace-a* or  *my-api.namespace-a.svc.cluster.local* 
    2. The K8s VirtualService’s name must be a fully qualified DNS name, which will be the corresponding name for AppMesh VirtualService. And we shouldn’t do any automatic name mangling(like append a namespace to name) since an K8s VirtualService is not necessarily model a K8s Service behavior. (*don’t panic, our current API handles this correctly*). 
    3. Question: is it valid use case to create a VirtualService named “www.google.com” (http://www.google.xn--com-9o0a/) to intercept traffic? (would like to check whether VirtualService is designed to be in-cluster DNS names only).
2. VirtualNode CR have ambiguous naming mangling hacks
    1. Under same namespace, two K8s VirtualNode can co-exists and represent a single AppMesh object. e.g. *namespace-a: my-node* and *namespace-a: my-node.namespace-a* will both referencing the same AppMesh VirtualNode *my-node-namespace-a*.
        1. note1: there can be more of this, *namespace-b: my-node.namespace-a* is also referencing the same AppMesh VirtualNode *my-node-namespace-a* as well even it’s in “*namespace-b*”.  The logic is if a name contains a “*.*”, we assume it already contains a namespace. That’s why I called it naming mangling hacks.
        2. note2: actually there are more cases like this(*though customer are unlikely to encounter these cases under normal usage*): *namespace-b: my.node.namespace-a*, *namespace-b: my-node.namespace.a*, ... will all reference the same AppMesh VirtualNode *my-node-namespace-a* as well.
        3. minor issue: when compositing the AppMesh VirtualNode’s name, it would be better to have used “*_*” instead of “*-*” as separator, since “*_*” is a valid AppMesh name component but not invalid K8s name component.
    2. what if we have a mesh span multiple clusters, how should the naming mangling change? should we supports backwards compatible upgrade from single cluster to multiple cluster?
    3. Question: Is virtual node always going to target pods within a same namespace? technically it don’t need to, but if we follow Kubernetes’s convention this condition holds.
3. The reference between VirtualService & VirtualNode is ambiguous.
    1. VirtualService reference VirtualNode via a field called *VirtualNodeName*. Guess what this name is?
        1. It can be the VirtualNode CR’s name. e.g. “*my-node*”.  And the namespace of VirtualNode is implicit to be the same namespace of VirtualService.
        2. It can be the VirtualNode CR’s name & namespace concatenate with “*.*”. e.g. *my-node.namespace-a*
        3. It *cannot be* the VirtualNode name in AppMesh. e.g. “*my-node-namespace-a*”
        4. In summary, the VirtualService reference VirtualNode via the *name of VirtualNode CR* with name naming mangling.
    2. VirtualNode reference VirtualService via a field called *virtualServiceName*. Guess what this name is?
        1. It cannot be the VirtualNode CR’s name. e.g. “*my-api*”. Otherwise, what’s the namespace of that CR? what if I have both *namespace-a: my-api* and *namespace-b: my-api*?
        2. It cannot be the VirtualNode CR’s name & namespace concatenate with “*.*”
        3. It can be the VirtualService name in AppMesh. e.g. “*my-api*”.
        4. In summary, the VirtualNode reference VirtualService via the *name of VirtualService in AppMesh* without naming mangling. 
    3. We can see that, there are multiple ways to reference a VirtualNode, and it relies on naming mangling hack base on whether there is an “.” in name. Also, VirtualService reference VirtualNode *via the name of VirtualNode CR*, while VirtualNode reference VirtualService *via the name of VirtualService in AppMesh*.
4. The logic to determine the set of pods that belongs to an VirtualNode is tricky.
    1. The default logic relies on pods to be managed by a ReplicaSet and then by a Deployment and The Deployment *name-namespace* will be same as *mangled VirtualNode name*.  This is ambiguous since pods don’t have to be managed by Deployment.  
    2. We allow pods to specify an annotation to override the VirtualNode it belongs to, but i think it’s inversion of control, where it should be the VirtualNode’s ability to select pods instead of pods to choose VirtualNode. The difference is customer need to modify their application to accommodate AppMesh if they have to do this override.
5. The VirtualService CR always requires a VirtualRouter. 
    1. From AppMesh API, an VirtualService’s backend can be directly a VirtualNode. which is modeling a L4 service similar to K8s Service.  However, we’ll alway carry the burden of VirtualRouter even if we don’t need it.
    2. This also prevents us to reuse a VirtualRouter from multiple VirtualServices. e.g. “*svc.namespace*” or “*svc.namespace.cluster.local*”. 
6. The AppMesh Injector requires a meshName environment variable set, which serves as default meshName for all pods within cluster(there is a pod-wise annotation to override this behavior). However, we support create multiple mesh in the cluster, there is nothing in the mesh to indicate this default semantic.