IPC20 work on Acme Air / NetflixOSS 2012-07-15

gistfile1.txt

Reviewers?
Allen Wang, Thomasz Bak - Ribbon
Nitesh Kant - Eureka, Karyon
Ben Christensen - RxJava

Acme Air NetflixOSS IPC20 Experimental Branch:
	https://github.com/aspyker/acmeair-netflix/tree/ipc20
	
Approach to "porting"
- I know that Acme Air Web App isn't really ready for RX, but I could get it done in waves
- First wave:  convert auth service micro service to be Karyon 2.0
- Second wave:  convert web app (which calls auth service) to Ribbon 2.0 + Rx (blocking sequential calls)
- Actual second wave thus far:  create acmeair-ribbon-test to create a J2SE Ribbon 2.0 + Rx client
- Third wave:  split more microservices from web app and get them called via true async/parallel Rx and expose new "per screen/device API".
	
Wave 1: Karyon 2.0 Auth Service
===========================

Key code:
- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-auth-service/src/main/java/com/acmeair/services/authService/AuthServiceApp.java
https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-auth-service/src/main/java/com/acmeair/services/authService/AuthServiceResource.java
https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-auth-service/src/main/resources/ACMEAIR_AUTH_SERVICE.properties
https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-auth-service/src/main/resources/ACMEAIR_AUTH_SERVICE-local.properties

Problems:
- Couldn't return JSON from Resource class as I couldn't tell Jersey to automatically map JSON
- confirmed as bug:  https://github.com/Netflix/karyon/issues/79

- Default EurekaModule assumes AmazonDataCenter (vs. MyOwn)
- This is a change in behavior from Karyon 1.0.  Still working issue in github.
- Have worked around it by going to true older behavior:
-- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-auth-service/src/main/java/com/acmeair/services/authService/AuthServiceApp.java#L27
- https://github.com/Netflix/karyon/issues/80

Not yet addressed:
- Configuration of healthcheck URL.  The sample seems to put the healthcheck on /hello/healthcheck which I think means it would be on the same port.  I think in the past Karyon had the management plane aspects on another port?
https://github.com/Netflix/karyon/blob/master/karyon-examples/hello-netflix-oss/src/main/java/com/netflix/hellonoss/server/HelloworldResource.java#L77
- Same questions on karyon-admin-web

How to run:
./gradlew clean :acmeair-auth-service:run
- note that I put an intentional sleep into validateToken to force testing of Hystrix in wave 2
- To run this, you need a eureka server and cassandra server loaded up with data.  It shouldn't be hard to mock out fake impls to disconnect from cassandra.  I use my dockerlocal acmeair to setup and load up cassandra easily.  I also use this dockerlocal acmeair for eureka.

Wave 2 (actual):  Simple test client to do RxJava+Ribbon2.0
===========================

Key code:
- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/java/com/acmeair/tests/ribbon/Test.java
- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/resources/config.properties

How to run:
./gradlew clean :acmeair-ribbon-test:run
- same need for eureka as auth-service if you want to do the Eureka aspects

Progress:
- It took a while to get started as Rx was new to me.  I think I have a pretty straight forward simple two calls in sequence example.
- I found it hard to find a similar example as most examples were nested calls in parallel and didn't do anything more than System.out.println the intermediate results.  This means there is no good example of the case where all the data needs to be joined back into a single result to the client of this RxJava code.  It would be helpful if the examples in ribbon-2.0 for rx actually formed up a single JSON document to be sent back to the client of a call like JSON doc = getMovieScreenForXBoxInterface() which would mean various leaf nodes of RxJava would have to statefully combine data.

- After I got this working, I started into working with Hystix.  I added the fallback command which was pretty well documented and tested if the service wasn't up as well as if the service was slow to respond.
-- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/java/com/acmeair/tests/ribbon/Test.java#L43

- I was able to control Hystrix as a whole on timeouts, but couldn't get Archaius working for specific command groups:
-- works:  https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/resources/config.properties#L3
-- doesn't work:
- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/resources/config.properties#L1
- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/java/com/acmeair/tests/ribbon/Test.java#L44
-- This may be due to the fact that I'm doing very simple J2SE, but given the property for Hystrix in general is picked up, I'm not so sure.  A good example that showed Hystrix being configured via archaius within Ribbon 2.0 would be helpful.  I couldn't find a documented example.

- I wasn't able to get Eureka discovery of the auth service to work.  I did what we did in IPC10 adding the same eureka configuration that worked for the auto service service registration.
-- https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/resources/config.properties#L7
-- When I do this, it tells me that it has an empty server list:

2014-07-15 17:10:08,952 [main] WARN  com.netflix.loadbalancer.RoundRobinRule  - No up servers available from load balancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=acmeair-auth-service-client,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:com.netflix.loadbalancer.ConfigurationBasedServerList@4c17a1bc

-- I was able to get either a) hardcoded or b) Ribbon20 listOfServers to work 
--- a) https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/java/com/acmeair/tests/ribbon/Test.java#L32
--- b) https://github.com/aspyker/acmeair-netflix/blob/ipc20/acmeair-ribbon-test/src/main/resources/config.properties#L5
- I tried to debug the code in ribbon quickly and it seemed like this was what always got called:

com.netflix.loadbalancer.LoadBalancerBuilder<T>.buildDynamicServerListLoadBalancer() line: 80	
com.netflix.client.netty.http.NettyHttpClient<I,O>.<init>(com.netflix.client.config.IClientConfig, com.netflix.client.RetryHandler, io.reactivex.netty.pipeline.PipelineConfigurator<io.reactivex.netty.protocol.http.client.HttpClientResponse<O>,io.reactivex.netty.protocol.http.client.HttpClientRequest<I>>, java.util.concurrent.ScheduledExecutorService) line: 100	
com.netflix.client.netty.RibbonTransport.newHttpClient(com.netflix.client.config.IClientConfig) line: 125	
com.netflix.ribbon.http.HttpResourceGroup.<init>(java.lang.String, com.netflix.ribbon.ClientOptions) line: 38	
com.netflix.ribbon.http.HttpResourceGroup.<init>(java.lang.String) line: 33	
com.netflix.ribbon.Ribbon.createHttpResourceGroup(java.lang.String) line: 27	
com.acmeair.tests.ribbon.Test.<init>(int) line: 29	
com.acmeair.tests.ribbon.Test.main(java.lang.String[]) line: 75	

serverListImpl is null which means it returns whatever is in the configuration vs. from discovery.

Wave 2 (for real):

- At this point I'm wondering if I should go ahead and assume moving to a netty/karyon2.0 hosted web app would solve some of the Eureka/Archaius issues or if I should understand them first in J2SE.

Wave 3:

- Not close to it yet.