We intend to write an article that technically outlines the way Whole Foods Market employs Symfony2 to create a scalable, available and maintainable codebase for our API. The intention is to not endorse any specific vendor, but to outline specific architectural choices we made from an application and infrastructure perspective. Since this is a technical article, we will need to support the ideas under discussion by providing code samples. Code we release will not divulge any trade secrets or compromise the security of our application in any way. All technical teams will be presented this article, in its entirety, and have the opportunity to modify, review and comment on it prior to publishing.
- Why we are using this framework over any other
- Why are we using PHP
- A high level description of internal architectural choices
- Effectively using the service container to manage dependencies and why it matters
- Decoupling application components and creating clear lines of communication
- Using thin controllers and introducing the service layer
- Introducing the data layer
- Employing the Single Responsibility Principle
- Testing considerations and Continuous Integration
- Handling errors and exceptions
In this article we plan to outline our transition from an API written in node to PHP, why we did it and some of the lessons we learned along the way. Node is faster than PHP, but the difference we measured was in the milliseconds, not seconds. A response that is 10 ms slower in PHP was not going to break the bank for us. What matters to us the most is application architecture, inheritance, encapsulation, polymorphism, dependency injection and testing. Node was becoming increasingly difficult to maintain because of the way javascript allows for these concepts to be implemented.
While making this decision, we evaluated several frameworks that we thought would be up for the task. We narrowed down the list to three contenders viz. Slim, Silex and Symfony. We load tested the other two and found that even though they fall under the "micro" category, they are not as fast as Symfony. This may seem counter intuitive at first, but we found Symfony to be faster because of the way classes are compiled and because of the way class caching and the service container works. We also wanted a framework that would scale with us as our codebase scaled, has a service container, an eventing system, can easily manage multiple environments, has flexible routing and an active community. Another big reason is that we are primarily a PHP shop and most developers in the building are PHP developers who are actively working on other PHP projects, some of which use Symfony.
Finally, we love PHP! PHP has gotten a lot of bad press lately, and in some cases, its hard to deny the arguments against the language. Take a look at PHP Sadness for details. The fact is PHP5 is easy to learn, fun to develop in, has a large pool of eligible developers and contractors (we work in downtown Austin), and at the time of this writing, powers more than 80% of all web applications on the internet! We are also very excited about the type safety and performance improvements in PHP7 and cant wait to take it for a test drive!
Whole Foods Market's API is powered by Amazon Web Services. We use Route 53 to point to an Elastic Load Balancer. Behind the ELB we have two EC2 instances, running nginx that act as proxies.
The proxy server's sole task is to route certain endpoints to node and others to PHP. We did this so that we can control the amount of traffic and migrate over specific endpoints, one at a time.
The proxies point to two EC2 instances for the old node app and two instances for the new PHP app. Both the apps share the same MongoDB replica set, so there are no database issues to contend with.
We use Ansible to configure and provision all our instances. We have a local development environment that is provisioned using Ansible and powered by vagrant. Three other environments exist viz. integration, QA and production that are exact copies of each other, and follow the aforementioned architectural diagram. Each environment also has its own mongo replica set. We use a blue-green deployment strategy in which we create an entirely new cluster of instances, deploy the new version of the code to it. Once all the tests pass, we point the load balancer to the new green cluster, using Ansible orchestration, while leaving the old, blue cluster, intact. If anything goes wrong, we can quickly flip back to the blue cluster, and triage the issues we had on the new green cluster.
Our application uses yaml based routing. We decided to go with this approach over annotations because we like to have all our routes consolidated in one location. We use swagger to document our API and since we are leveraging the (NelmioApiDocBundle)[https://github.com/nelmio/NelmioApiDocBundle) we wanted our annotations to be as terse as possible and only contain API documentation for the specific controller method, and nothing else.
stores-tlc-sales:
path: /v2/stores/{tlc}/sales{trailingSlash}
defaults: { _controller: WfmApiBundle:Store:getSalesByStore, trailingSlash : "/" }
requirements: { trailingSlash : "[/]{0,1}" }
methods: [GET]
requirements:
tlc: \w+The service container is one of the most useful and central architectural pieces in our application. We use it to manage all our dependencies. If you're not familiar with it, drop everything you're doing and learn about it now!
Here is an example of how we define a service.
services:
service.store:
class: Wfm\ApiBundle\Service\StoreService
calls:
- [setMongo, ["@client.mongo"]]
- [setAddressUtil, ["@util.address"]]
- [setFilter, ["@filter.store"]]
- [setHydrator, ["@util.hydrator"]]
- [setMongoQueryStrategy, ["@strategy.store_mongo_query_strategy"]]Think of a service as an encapsulated unit of work that coordinates other SRP objects that do something. Our services are the glue between HTTP requests, business logic objects, query generating strategies for various search types and the data layer. Here is an example of how we use a service from the service container.
/**
*
* Get all sales for one store by TLC (Three Letter Code)
*
* @ApiDoc(
* https = false,
* section = "Store",
* output = "json",
* description="Get sales by store",
* parameters={
* {"name"="fields", "dataType"="string", "required"=false, "description"="Specify an optional comma separated list of fields e.g. tlc,name,address"},
* },
* statusCodes={
* 200="Successful response",
* 400="Client submitted a bad request, an error message will be provided with a reason",
* 401="Authentication is required and has failed or has not yet been provided",
* 500="A fatal error occurred and no data will be returned. The API team will automatically be notified of this condition"
* },
* tags={
* "stable"
* }
* )
* @param string $tlc Three letter code for store
* @param Request $request Injected HTTP Request Object
* @return JsonResponse
*/
public function getSalesByStoreAction($tlc, Request $request)
{
/** @var StoreService $storeService All store related calls */
$storeService = $this->get('service.store');
$storeService->setFields($request->get('fields'));
$sales = $storeService->getSales($tlc);
return $this->jsonResponse($sales, $storeService->getHeaders());
}The primary purpose of our controllers is to take request data, fetch a service from the container, set request data on the service,
call the appropriate method on the said service and hand off the returned data to JsonResponseTrait, which converts the data to a JSON object with headers.
There are a couple of good reasons to do it this way:
- Framework Independence - We could switch to a new framework with ease as most of our code is not in the controllers
- Testing Services independently - Since all the code that does work is encapsulated in a service, we can unit test it easily without having to worry about unit testing our controllers
We don't use an ORM for our data layer for a couple of reasons - MongoDB ODM is not ready for prime time. At the time of this writing, it is in beta, and EmbedMany relationships do not appear to be working properly. - We noticed a measurable increase in the amount of time taken to hydrate Entities
Since mongo is a schema-less data store we needed to represent the schema using models in the data layer.
We have a Store object, which contains many StoreSale objects. Each StoreSale object can and does contain many StoreSaleItem objects.
Whenever we query a Store we expect all the fields to contain the appropriate default values populated.
If a field is missing in mongo, we want the field to show up in the JSON response, but we also need to make sure that the data type we advertised is preserved.
We needed a solution that would read the raw JSON from mongo, and throw it up against a set of nested models, and create an appropriate JSON respresentation from the result, with all defaults populated and all data types preserved.
The data types are indicated in the model as annoations like so:
class Store extends AbstractModel
{
/**
* @var string
*/
protected $country = 'USA';
/**
* @var array
*/
protected $geo_location = array(
'type' => null,
'coordinates' => array(
null,
null
)
);
/**
* @var bool
*/
protected $has_alcohol = false;
/**
* Any sale(s) that this store may have
* @var StoreSale[]
*/
protected $sales;
} The $sales property is special, because it is an array of StoreSale objects. Each of these sub-models have their own set of primitive data and arrays containing other nested models.
class StoreSale extends AbstractModel
{
/**
* @var StoreSaleItem[]
*/
protected $items = [];
}In this fashion, we can authoritatively represent the schema, with any defaults we may need. We then created a class that would recursively hydrate the models from mongo data.
As mentioned earlier, we unit test our service layer and all the business logic contained therein. Since the application is an API, we use Codeception and the Symfony2 module for acceptance testing. Basically what that does is it runs through the routing in Symfony, and calls the code as if it were being hit over the network. The only difference is it all happens under the covers and we don't have to contend with setting up a webserver.
Here is an exmple of a simple API test. We created an abstract parent that contains some common methods that all our tests use. In this example, we are seeding a local copy of mongo with some test data. At the end of the test, we are validating the data contained in the response.
<?php
use Symfony\Component\HttpFoundation\Response;
use Wfm\ApiBundle\Exception\MissingDependencyException;
class StoresShowCest extends AbstractCest
{
/**
* Runs before all tests in this class
* @throws MissingDependencyException
*/
public function _before()
{
parent::_before();
$this->helper->seedCollection($collectionName = 'stores', $dataFile = 'StoreLamar.json');
$this->helper->createIndex($collectionName = 'stores', $field = 'geo_location', $type = '2dsphere');
}
public function shouldReturnOneStoreLamarTest(ApiTester $I)
{
$I->wantTo('Test one store only');
$I->sendGet('/v2/stores/LMR');
$I->seeResponseCodeIs(Response::HTTP_OK);
$I->seeResponseIsJson();
$I->haveHttpHeader('Content-Type', 'application/json');
$I->seeResponseContainsJson(
array(
'status' => 'OPEN'
)
);
$I->dontSeeHttpHeader('wfm-query-skip');
$I->dontSeeHttpHeader('wfm-query-sort-field');
$I->dontSeeHttpHeader('wfm-result-count');
$I->seeHttpHeader('wfm-status-code', 'SUCCESS');
$I->seeHttpHeader('wfm-status-message', 'Found matching record.');
$I->seeHttpHeader('wfm-timestamp');
$I->seeHttpHeader('wfm-query-fields', 'sales');
$I->dontSeeHttpHeader('wfm-query-limit');
$response = $I->grabResponse();
$store = json_decode($response);
$this->assertStoreData($store);
}
}We recognized that there are two general categories of errors that could occur in our API: - User errors - Missing input, invalid input, invalid data types etc... - Developer errors - Missing dependencies, database issues, missing polymorphic implementations, fatal errors etc...
In the case of user errors, we didn't need to know about them, but in case of Developer errors, we needed immediate notification and logging it somewhere was not enough.
The solution we came up with is to create two classes that extend Exception
<?php
namespace Wfm\ApiBundle\Exception;
/**
* Class DeveloperException gets thrown for all systemic issues that developers need to be aware of
* @package Wfm\ApiBundle\Exception
*/
class DeveloperException extends \Exception
{
}<?php
namespace Wfm\ApiBundle\Exception;
/**
* Class UserException gets thrown for all user related errors that developers don't need to act upon
* @package Wfm\ApiBundle\Exception
*/
class UserException extends \Exception
{
}We then craft specific child classes that extend either of these depending on the situation.
<?php
namespace Wfm\ApiBundle\Exception;
/**
* Class EmptyResponseException is thrown when an empty result-set is encountered
* @package Wfm\ApiBundle\Exception
*/
class EmptyResponseException extends UserException
{
}<?php
namespace Wfm\ApiBundle\Exception;
/**
* Class MissingDependencyException gets thrown when there is a required missing dependency
* @package Wfm\ApiBundle\Exception
*/
class MissingDependencyException extends DeveloperException
{
}Now that we have a system to indicate which exceptions we care about, how do we deal with them? To solve this problem we created an event.
Basically, we catch all our exceptions in one location and figure out what we need to do with them.
Any exception that has a parent DeveloperException needs to get sent to rollbar, with a full stack trace, the others can get logged, and will eventually end up in papertrail.
<?php
namespace Wfm\ApiBundle\EventListener;
use Wfm\ApiBundle\Traits\JsonResponseTrait;
use Wfm\ApiBundle\Exception\MissingDataException;
use Wfm\ApiBundle\Exception\DataNotFoundException;
use Wfm\ApiBundle\Exception\EmptyResponseException;
use Wfm\ApiBundle\Exception\InvalidSearchParameterException;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Exception\HttpExceptionInterface;
use Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent;
use \Rollbar;
use \Exception;
/**
* Class WfmExceptionListener listens for kernel.exception event and responds
* @package Wfm\ApiBundle\EventListener
*/
class WfmExceptionListener
{
use JsonResponseTrait;
/**
* This list of exceptions will not get sent up to rollbar
* @var array
*/
protected $nonRollbarReportExceptions = array(
'Wfm\ApiBundle\Exception\EmptyResponseException',
'Symfony\Component\HttpKernel\Exception\NotFoundHttpException'
);
/**
* @param GetResponseForExceptionEvent $event The Exception event
* @throws Exception
* @return GetResponseForExceptionEvent
*/
public function onKernelException(GetResponseForExceptionEvent $event)
{
$environment = getenv('ENVIRONMENT');
$exception = $event->getException();
$response = $this->errorJsonResponse($exception);
if ($exception instanceof HttpExceptionInterface) {
$response->setStatusCode($exception->getStatusCode());
$response->headers->replace($exception->getHeaders());
} elseif ($exception instanceof DataNotFoundException) {
$response->setStatusCode(Response::HTTP_NOT_FOUND);
} elseif ($exception instanceof EmptyResponseException) {
$response = $this->itemNotFoundJsonResponse($exception->getMessage());
} elseif ($exception instanceof InvalidSearchParameterException) {
$response = $this->invalidSearchParameterJsonResponse($exception->getMessage());
} elseif ($exception instanceof MissingDataException) {
$response = $this->missingRequiredParameterJsonResponse($exception->getMessage());
} else {
$response->setStatusCode(Response::HTTP_INTERNAL_SERVER_ERROR);
// If we are in unit test land, throw so its shows up in the CLI
if (php_sapi_name() == 'cli') {
throw $exception;
}
}
// Should we report this exception to rollbar
if (($environment == 'production' || $environment == 'qa') && $this->shouldReportExceptionToRollbar($exception)) {
$config = array(
'environment' => $environment,
'root' => '/usr/share/nginx/html',
'handler' => 'agent',
'agent_log_location' => '/usr/share/nginx/rollbar'
);
Rollbar::init($config, $setExceptionHandler = false, $setErrorHandler = false, $reportFatalErrors = true);
Rollbar::report_exception($exception);
}
// Always render error responses as JSON
$response->headers->set('Content-Type', 'application/json');
$event->setResponse($response);
return $event;
}
/**
* Should we report this exception to rollbar or not?
* @param Exception $e The thrown exception
* @return bool
*/
protected function shouldReportExceptionToRollbar(Exception $e)
{
$class = get_class($e);
$parentClass = get_parent_class($class);
/** @var bool $notifyDeveloper Should the developer be notified? */
$notifyDeveloper = (strpos($parentClass, 'DeveloperException') !== false) || (strpos($class, 'DeveloperException') !== false);
return !in_array($class, $this->nonRollbarReportExceptions) && $notifyDeveloper;
}
}In this fashion we report all our exceptions to rollbar, if they are important to us, otherwise we just convert them to JSON, and return back to the client.
Our API powers the iOS and Android app and also powers most of the data queries on wholefoodsmarket.com. Switching to Symfony has been an adventure, and we have yet to migrate over other endpoints. We are quite pleased with the state of PHP and Symfony as it stands, and are looking forward to using PHP7 and Symfony3. Feel free to reach out with any questions or corrections you may have and we will get them answered promptly.