I want to start a complete rewrite of gw2treasures.com. In this post I want to outline the reasons for the rewrite and what requirements/features/… I want for the new version.
At the end of 2012 I wrote a bot for the german Guild Wars2 Wiki with a website to review all edits my bot made. When the official Guild Wars 2 API was released on May 14th 2013 I added a list of items to this website with some basic search tools. This list was used to help me and other wiki editors with creating pages for all items.
I started adding more features to it but quickly decided to write a standalone website with detailed pages for each item.
Development for gw2treasures.com started in November 2013.
gw2treasures.com runs on a single server and is written with Laravel, a PHP framework. The Laravel version currently used, 4.2, was released 6 years ago and updating to newer versions is difficult, because so much has to be rewritten. Laravel is a great framework, but not being able to use a current version and PHP in general makes it a pain to add new features to the site.
Not only the server-side framework is outdated, for the few features that use javascript a 6 year old jQuery version is used. Complex interactive pages are not really possible. The few interactive things I have (like tooltips) are difficult to maintain and extend.
Now I want to rewrite everything using more modern technologies and architecture. There are many new features I want to implement that require the rewrite. Everything will be written in typescript, so that I am able to easily share code between all components.
Instead of running everything on one server I want to be able to scale everything independently, especially the workers. Every service will be a docker image that can be deployed to whatever orchestration I decide on (probably docker swarm or k8s). This is not a micro service architecture, the API contains almost all logic.
All requests from the outside will first go through a reverse proxy, which fulfils a few different purposes: It has to proxy the requests to the correct backend service instance (either the UI or API Server), block everything that doesn't look like a regular request (wp-admin, phpmyadmin, …) and authenticate requests.
For all requests that require authentification the reverse proxy will make a request to the IAM to verify the session token and replace it with a shortlived JWT generated by the IAM. If the session is missing or invalid the reverse proxy will already respond with a 401. The backend services can then check the JWT to see if the request is authorized and get additional data (username, …) from it.
The IAM (Identity and Access Management) is responsible for all user related data and managing sessions. The current gw2treasures.com doesn't provide a login function, but that's something I definitely want to change for the new one. Keeping the user data separate from all the other application data seems like a good idea.
The UI server will just serve static html, css and javascript. Some files served by this might require authentication (for example the js chunk for an admin backend). Everything is generated by react, so this is a SPA (Single Page Application).
There will be SSR (Server Side Rendering), but probably only static at build time and I don't know yet how much. Maybe just some skeleton UIs (for example every /items/* url will be served the item.html entry point, everything else the index.html) to improve the perceived load time. Everything will be heavily chunked and loaded async, also making use of all the new (currently still experimental) concurrent rendering and datafetching with suspense (read more).
There will definitely be another lengthy post just about the UI and many experiments, before I decide all the details.
All logic takes place in the API. All database requests should be done from here. The API can schedule tasks to be run by a worker. This can either be a regular job (check for new items every X minutes) or triggered by a specific action.
Different workers can register themselves to the API. The API manages a queue of scheduled tasks that need to be done. The worker will then work on the highest priority task and return the result to the API. The API handles the result (probably write something to the database). There will be different tasks, most will involve querying the Guild Wars 2 API, but some will also just work with existing data to aggregate results or similar. I split the workers in static and authenticated in the graphic above, mostly just so I can scale or stop them individually.
Ideally all GW2 API requests will only done by workers and not by any other service, so they can each use an internal rate limiting bucket to throttle requests. Otherwise it would need some central management (proxy?) to not get rate limited by the API. This of course requires, that each node in the docker swarm only runs exactly one worker and that every node has a dedicated external IP.
Okay, lets get to the list of features I want for the rewrite. I will write concepts and create mockups for all of them in the future before I start the implementation, this is just a rough list.
Obviously I want to keep all the database functionality there is right now where you can look up details of items, achievements, skills, and so on. One change I really want to make to this are build revisions. This allows you to look at a skill at build version 103632 and also compare it with a diff to a different version.
I want to greatly improve the search. Everything should be searchable, with advanced filters and better auto complete. And not only entries for the database should be indexed, the user should also be able to enter "settings" and get a link to the settings page.
Right now I only show the current TP prices for each item, in the future I want to save the trading post history like many other sites already do. Instead of just buy/sell price (and maybe some percentiles) I also want to have the historic crafting price of the item in the same graph. For this I need more detailed info about each recipe and want to display this in a rich crafting calculator like on gw2efficiency.
Adding a user registration allows many new features. We can personalize settings, provide watch lists and maybe do some other things like comments or uploading images (for armor sets, …).
With users we also have the possibility to add API keys. On every item we can show on which accounts/characters you already have that item, which achievements are already done, create customized crafting guides using owned materials and many more things. This also provides us with statistics over the whole user base (how many people have completed this achievement, …).
Using multiple accounts should be integrated in every functionality and not require the user to switch the active account. Instead the user should flag accounts as main/alt/hidden in their settings, to control which accounts are shown. The achievement status could then only be shown for main accounts, but the crafting calculator also uses materials stored on alt accounts. Sharing accounts with different users should also be a thing (and not just by sharing the API key, but in some way that allows the owner to see and control who else has access to that account).
Character and account pages can be used to share characters, but only if the owner is allowing this. They should include equipment, specializations, inventory, ….
I also want to allow other website to use a "Login with gw2treasures.com" functionality. For this there will be a way for other developers to create applications. These applications can then use OAuth to login and receive a custom subtoken for the GW2 API. Users can manage which applications they currently give access and review which permissions they require. This should also provide some analytics to the developers (number of users, daily requests, …). Maybe this application feature could also be used for gw2treasures.com itself, so I can get the same analytics features for my internal API requests.
Some other features include: Current WvW Matches, Interactive Map, …. There are probably a thousand features I am forgetting, but thats a rough overview.
I don't have much time to work on this. But I really want to get it done. Don't expect anything usable this year. Over the next few months I want to write some concepts and do some technical experiments.
Soon™️
Thanks for reading this far. I'm trying to write down all my thoughts and experiences during the development. Mostly for me to keep everything ordered, but maybe some of you find this interesting.
One day you might read this on the rewritten gw2treasures.com with comments but for now, if you have any feedback or suggestions the best way to discuss will be in the gw2treasures channel on the GW2 Development Community discord.