I used to be a hobby php developer back in the days when I was a student, but for the last 10 years I’ve mostly worked in the embedded software scene. Last year I started working as a contractor web developer and I’ve been learning and working with web technologies ever since.
With this background I’ve been learning and finding out what are the best ways to build web services these days and what tools and processes are effective for getting things done. There is a huge amount of information available in the web, but few articles offer more than bits and pieces of complete solutions and then again, web tech keeps changing all the time making it challenging to keep up with the latest tech and practices. With this series of blog articles I’m trying to gather a more complete alternative solution on which starting developers and small teams could base their initial service architecture and development processes.
The setup described here is quite general and customisable so while it won’t be the end game for successful large scale services, I hope it’s a reasonable starting point for building modern and scalable web services.
Article Series Content
Topic 1 – Introduction & Local Development Environment
- Overall system architecture
- Overview of the implemented authentication service
- Selected services and selected technologies
- Local development environment setup
- Simple service components implementation
Topic 2 – Test Environment Setup
- Overview of the test strategy
- Test environment setup
- Continuous integration service setup
- Version control service setup
Topic 3 – Test Driven Development
- Front end service implementation
- App back end service implementation
- Auth back end service implementation
Topic 4 – Acceptance Test Setup and Testing
- Acceptance test environment overview
- Acceptance test environment setup
- Acceptance tests implementation for the service
Topic 5 – Production Environment Setup
- Production environment overview
- Production environment setup
Topic 6 – Production Environment Finalisation
- Container monitoring
- Error management
- Log management
Due to the already long length of the articles I’ve dropped certain topics out from the first article series. However I’m planning to cover at least some of the following topics afterwards:
- Performance measurements and system scaling
- Database replication
- JWT secret management
- Performance, features and usability comparison with other container orchestration tools and hosting providers
- Consistent API responses & API documentation for the back end services
- Improve certain caveats of the implemented authentication service with an in-memory database solution
Overview of the Implemented Service
The developed service will be a user authentication service bundled with a simple App Service which counts and returns the requests made to it. The service works quite well for these articles since it adds a bit more complexity to the system setup compared to a simple front end + back end case, but the services themselves are simple.
The authentication service will be based on JSON Web Tokens (JWT) and it relies heavily on JWT-related articles written by Auth0 fellows. Personally I’m quite intrigued by the JWT-approach since it makes it possible to write stateless services and the horizontal scaling is less of an issue compared to session based approaches, but then again it also has its drawbacks.
Both refresh and access tokens will be used in the authentication service. The token definitions may vary a bit depending on the source.
- Refresh tokens are long lived tokens, 30 days in this case
- A client gets a refresh token by doing a login against the Auth Service
- Refresh tokens grants access to get access tokens which are used for accessing the App Service
- Refresh tokens can be renewed without a new login if the renewal is done when the refresh token is still valid
- A user logout function in Auth Service will set all already issued refresh tokens as invalid requiring a user to login again to get new refresh tokens
- Clients store the refresh tokens inside secure, http only cookies limited for a certain domain and path
- Short lived tokens, 30 minutes in this service
- A client gets an access token by requesting it from the Auth Service with a valid refresh token
- Access tokens are used (and required) for accessing the App Service
- Clients store the access tokens in local storage or a similarly secure location
All traffic between the clients and the authentication service will be transmitted over a secure https connection.
Pros & Cons
With this kind of authentication scheme the App Service validates the requests from the received access tokens and it doesn’t need to query a database for authentication purposes. This way the design already leans towards a microservices-based system architecture by separating responsibilities and functionalities into different services which can be controlled – and scaled when needed – quite independently.
The downside of this kind of architecture is a force logout for a user can’t be completed in the case that an account’s refresh token or login credentials are compromised; after a password change and logout the existing refresh tokens are rendered unusable, but existing access tokens would still be usable for a maximum of 30 minutes. For regular services, I see this as a very minor compromise, but when dealing with services requiring very strict security measures these kind of details may need to be dealt with. There are ways to handle these cases quite efficiently, but the solutions that I’m aware of make the system a tad more complicated so I’m not going to include those solutions in this basic setup.
The micro service oriented system will be based on service containers.
Reverse Proxy and Load Balancer Service
This component is going to handle the load balancing and proxying the traffic to the right services.
The Auth Service will provide APIs for user login and getting, renewing and expiring refresh tokens and getting access tokens.
The App Service will provide a simple API which counts the requests made to the API and returns the current count.
Auth Database Service
To store user and authentication related data.
App Database Service
To store application data.
Containers and Orchestration
One of my personal goals for the last few months has been to learn more about containers. Based on what I’ve read and learned so far, I’m very excited about the possibilities they provide and so I decided to use containers in this article, Docker containers more specifically.
Many little details must be handled to use containers in practice. For starters, the containers must be built, distributed, and deployed into the desired environments. There are already a bunch of different container orchestration tools available to help with container management and while investigating these options, I found out about Kontena which is an open source container and microservices platform. Kontena’s feature list, which includes things like a private image-registry, built-in virtual networking and aggregated logs and stats, is very impressive. Also their developers have been very helpful in their Slack channel, so I chose to use Kontena’s container and microservices orchestration platform in these articles.
For the moment I lack the experience to be able to compare Kontena with other available orchestration tools, but I might do some kind of comparisons later.
Reverse Proxy and Load Balancer
This component will be based on HAProxy mostly due to the fact that Kontena provides a HAProxy-based load balancer with some nice additional features. To keep environments consistent, HAProxy-based load balancers will be used in all environments.
SSL termination will be configured for the load balancer in this setup, which makes the certification management quite easy since this way the certificates don’t need to be set for upstream servers. One additional benefit is the SSL related work offloading from the front end and back end servers to the load balancer. The traffic between services in Kontena Grid is encrypted by default by Weave’s encrypted sleeve tunnels. So even if the used nodes were physically located in different data centres, the data between service containers – like load balancer and upstream services – is always encrypted.
Auth Service and App Service
Sequelize is an ORM module for Node.js. It provides the basic features for working with databases including simple tools for managing database migrations via The Sequelize Cli. Even if Sequelize doesn’t provide all the possible bells and whistles it’s quick to get started with it due to the low amount of boilerplate code needed.
In general, my experience with SQL-based databases isn’t very broad, but in the latest projects I’ve used PostgreSQL and I’ve been happy with it. There are some small details I’ve liked better in PostgreSQL compared to MySQL/MariaDB, but then again I haven’t encountered any deal breakers with either of them.
However, I’m not sure what the precise situation with the scaling front is. With MariaDB, for example, Galera Cluster should be quite easy to set-up and I’m not aware of such easy and scalable solutions for PostgreSQL. It’s possible to configure PostgreSQL into various high availability master/slave modes, but purely feature wise they can’t quite compete with Galera.
I’m selecting PostgreSQL for this setup – and I’m sure it’s a fine choice in general – but when dealing with services requiring lots and lots of database writes, it would be good to do more research about the subject before committing to PostgreSQL.
Even if it’s fun to tinker with your own servers, version control tools, and continuous integration systems, it takes time and resources to set them up and keep them running. By outsourcing these services, developers should be able to put more of their precious time to the service development, which is very important for solo developers or small teams lacking specialised DevOps resources. All of the selected services for these articles also provide some kind of free tier or trial period to make it possible to try them out before committing to the chargeable service tiers. Project and company related privacy and security policies may also limit the 3rd party service options, but in general I’d recommend to use them when feasible.
Kontena provides plugins and guides for very easy and straightforward setup for various popular hosting platforms. My choice is UpCloud, since their virtual servers are competitively priced and their services performance is excellent! By signing up through their Kontena campaign page they will give you $25 in credits to start with. Instructions are also available for many other platforms like Amazon, Azure and Digital Ocean.
Source Code Hosting
GitLab also has a free tier available for private repositories, and based on the documentation it seemed like they are actively developing their services forward to include many frequently needed CI and CD features. For the moment 3rd party CI service support for GitLab isn’t quite as comprehensive as for GitHub, but it’s becoming more and more common. I selected GitLab for this article and my experience with GitHub is limited, but most likely it’s possible to configure GitHub the same way the GitLab is configured here.
Continuous Integration Service
All CI services don’t support GitLab Git repositories, but some do. The CI service choice was difficult since none of the alternatives seemed to be perfect, but in the end I selected Shippable since it offers quite a lot already in the free tier. However, the documentation and tutorials regarding pipelines were quite bad when I tried them a few months ago, but they might be in a better shape now.
GitLab CI requires a local or cloud based CI runner for running the CI tasks. Local runner is good in certain cases, but since I’m mostly using laptops with limited resources I prefer to use cloud service resources for running the CI. I didn’t want to spend time configuring and maintaining my own runner in the cloud either. I recently learned GitLab also provides free CI runners (with certain limitations) so I’d most likely try out GitLab CI, in case I run into troubles with Shippable.
CodeShip, CircleCI, Travis etc. are capable options as well, but due to lack of GitLab support, limited free tier, or somewhat expensive first chargeable tier I ended up selecting Shippable over these.
Docker Image Registry
The built containers have to be stored and kept available somewhere. I’m using DockerHub to host Docker images in this article, but feel free to use other similar services like Quay.io. Kontena also includes a private image registry incase a private hosting solution is preferred. However, depending on the implemented service the docker images may require quite a lot of space, so if private registry is used, it’s good to prepare to invest some time for the registry maintenance. I’d recommend to start with some external image registry provider.
The article series will contain guidelines for setting up environments for DTAP-based setup: Development, Testing, Acceptance and Production. This is a good start for common cases, but depending on certain factors, there may be the need for additional environments for field testing, performance testing or for development teams to play around with.
Local Development Environment
Local development environment will differ from the rest of the environments. Custom Docker containers for the developed components will be built to enable automatic code reloading, for example. Also the system will be run using Docker Compose instead of Kontena.
- Custom configured HAProxy based load balancer and reverse proxy
- Next.js based front end service
- Node.js based back end services
- PostgreSQL database containers
- Access via xip.io domain names or directly to containers via open ports
- Self signed SSL certificates
- Manual deployment via Docker Compose
Test, Acceptance Test and Production Environments
The test, acceptance test and production environments are identical from the container and architecture point of view. The containers built by the CI system for test environment will be used as they are in the acceptance test and production environments. Kontena will be used for the deployments and all environments are hosted in UpCloud.
- Kontena HAProxy based load balancer and reverse proxy
- Next.js based front end service
- Node.js based back end services
- PostgreSQL database containers
- Access via real domain name
- Let’s encrypt certificates
- Automated deployment
With this introduction part the scope for the article series is now set. In the next part – which will follow soon – the local development environment together with simple versions of the service components will be implemented.