Welcome! You are on the verge of embarking on an adventure of discovery and understanding with this book, I hope you enjoy reading it as much as I enjoyed writing it!
On this journey, you will learn all you need to know to build your own micro-services-based application with Django. But before we get started we need to get a few definitions out of the way! This is what this chapter will be about...
Micro-services Vs monoliths
First off, what is a micro-service and how is it different from more traditional architectures? For most of computing history, programs have been entities running on a single computer; either yours or another, remote computer. Think about Adobe Photoshop as one example of a program running on your own computer. Think about these old mainframe programs, rendered on green monochrome screens as programs running on another computer; but server-side rendered websites are another type of code that runs mostly on another computer (the web server), your own computer or mobile device just being there to render whatever the server has built.
In the very beginning, those programs were held on a stack of punch-cards, computers usually only able to run a single program at a time.
The next evolution after punch cards were files, we still mostly use file to store code, but in the beginning programs were stored in single files without the ability to reference any other file, everything program was its single entity only capable of using the, often limited, standard library of the language they were programmed in. Think of the .bas files on a Commodore 64 for example.
You can see here a picture of Margaret Hamilton standing next the code she and her team wrote to send the Apollo mission to space. Seeing that stack of paper, the term monolith probably starts to make sense.
The next evolution of software engineering was to be able to reference other code files, modules or libraries from the main file, allowing you to split your code, try to use the separation of concerns principle and group code in different files, each file dedicated to a specific concern. We still use this principle in Python today. Whenever we write an import statement to make some code available, whether that code was written by us or by someone else, whether is only being used for this one program or if it was intended to be re-usable.
Jump forward a few years and re-usable code has now become a thing. One of the techniques that made re-usability a success is, in my opinion, Object-Oriented-Programming (or OOP for short). A technique at which Python was one of the first languages to excel at. Having to write class definitions encouraging developers to then use them more than once, having them think about inheritance and grouping common features in parent classes.
But, at this point, programs still exist as a single unit, each of them spread among multiple files, using several libraries but still running as a single unit, designed to accomplish one main goal, every task required to accomplish this goal being performed by that single program or project unit. This is what we reference as a monolith.
The next evolution of software development was to make these libraries and modules that had a specific purpose or concern into services of their own. Those services made accessible to the main program by different means. XML-RPC and COM might come to mind when you think of those early services.
The final evolution as far as we are concerned, made easier by the rise of the cloud, was to make these service smaller and smaller so that they would not be responsible of a whole concern but rather one or only a few tasks. This architecture is what we call micro-services.
Different types of micro-services
Now that we are clear on the concept of micro-service, let's have a look of different types of implementation people commonly refer to as micro-service.
One of the simplistic type of micro-service around is probably Amazon's single-function Lambdas. Those are very isolated pieces of code performing one single task, they are often used to perform compute-heavy tasks as they are also on-demand, giving the user access to a lot of processing power but only having to pay for the actual CPU time they have been using over time. It is very cost-effective for tasks that do not have to run often but that, when they do, need a lot of CPU time.
This architecture can be very scalable as several instances running the same Lambda can be spun up at the same time if needed, therefore easily handling peaks in traffic and having it all managed by the top-level infrastructure; in this case, Amazon but most cloud providers are now either providing their own version of this or looking into it.
Lambda's are not as well suited for thing like low-to-medium traffic services that require an "initialization time", ie: reading a configuration file or connecting to other services as, after being unused for a given amount of time, they will be spun down (shut off) meaning that the next time a request is mad to that function it will have to be "cold booted" again, performing all those initialization tasks ones more. This is a significant overhead, both in time and money.
On the other end of the spectrum, we have services deployed as distinct hosts on one or more bare-metal server(s). Each service having its own separate host and host configuration, it is not hard to move them to a different bare-metal service in case they become too CPU- or IO-intensive. This is probably the easiest way one can get from deploying regular monolith web applications to deploying micro-services.
This, although possible makes scaling non-trivial bringing the need to manage things like internal DNS, load-balancing and other somewhat more complicated issues depending on how comfortable you are with server configuration and management.
In between those two extremes, there is managed container-based deployments using tools like Docker and Kubernetes (or K8s for short), this has the advantage of each container being pre-configured to handle a specific task (like serving as a database host or performing complex computations) with all the required dependencies, making it easy to deploy it to any machine or service that is compatible with the chosen container type.
The top-level system will also take care of concerns mentioned above like load-balancing and DNS management, making it easier to deploy and manage each service.
Benefits of using micro-services
As mentioned earlier, micro-services being an evolution of the use of libraries and modules for the purpose of separation of concerns, they should each have a very limited scope. Of course, this is something people working with monolith and modules often try to achieve as well; but when all your code resides in the same place, it is rather easy to get side-tracked and inter-reference modules with each other, turning the code in a spaghetti-bowl-like structure where it gets harder and harder to make the difference between one module and the other, wondering if a function should be added here or there, having to open a multitude of files to find out that the piece of code you were looking for was finally in the first file you open.
In short, micro-services makes separating concerns and architecturing your code easier.
Since micro-services address a limited scope they are often easier to test.
Having each service now running completely independently also makes deploying a certain piece of code that is more CPU-heavy or IO-intensive on a separate node (or server). You can even mic and match, putting a CPU-heavy service with an IO-intensive one in order to try to make the most of that node.
Micro-services are more flexible when it comes to hardware
Running completely separated from each other, different micro-services may use different versions of the same library with one service for example taking advantage of that brand new feature that was just introduced while another service depending heavily on security is sticking to the LTS version of that same library. Be careful though, this is a double-edged sword!
Micro-services are more flexible when it comes to library usage
Batteries included Vs minimalist framework
Django advertises itself as coming with "batteries included". By that meaning that Django has a lot of features and functionalities out of the box. For that very reason a lot of people don't even consider using Django for micro-services as they see the dichotomy between "small service that only performs a few tasks" and "lots of included functionalities".
The fact Django comes with a lot of functionalities doesn't mean you do have to be using all of those features to take advantage of the framework itself.
On the other hand we have minimalist frameworks that come with very few features other than routing. With this type of framework, if you ever need to perform a common action such as connecting to a database or providing authentication, you will have to either write those functionalities or resort to some third party library. The biggest risk here being for developers to decide on using home made code for solving problems they might have underestimated, one of the few things that come to mind right away are authentication or date management.
Even if it's not perfect, I'll try to go with a car analogy here.
If you were to fix a car, would you rather have access to a fully-equiped garage (at no extra cost) even though you know you won't be using half of the tools available in said garage or would you rather do that in the parking-lot in front of your house, knowing that you have very few tools at your disposal but that there is a DIY store nearby where you will be able to find most tools you'll need and that are not included in the service kit you received with your car?
Some of you might decide to go with the second solution, arguing that if you do that often enough you will start building up your own supply of tools and that it might be beneficial. This is the minimalist framework route.
I would imagine most of you would rather go for the fully equipped garage though. This is the Django route.
Is Django really too bloated?
"Django is too bloated for micro-services" is an argument I have seen or heard several times. Not being able to question the person who said that at the time, I tried to understand what they meant by that.
First, I looked at memory usage. It is true that Django, running with DEBUG=Truetakes at least 200Mb of Ram which can be scary if you want to try to run it on a small node that only has 256Mb available. Once you set DEBUG=False though (ie: in production), the amount of Ram used goes down drastically and is similar, even sometimes smaller than the same functionality running on another framework.
Since the claim is not Ram-related, I then ran test to compare CPU load between Django and other frameworks, thinking that maybe Django would be using more CPU either at low or higher request rates. Once again Django performed similarly to other frameworks.
As a last resort, Django having its own ORM (Object Relational Mapper), I wondered if that claim might have to do with the database layer of Django and went ahead to test that. Once again, the results were similar to other framework.
In summary, if someone wants to make the claim that "Django is too bloated for micro-services", I would like to know what exactly they mean by that and if they have evidence to support that claim. Until that happens, I will claim that "Django is not too bloated for micro-services"
Storing extra batteries safely (configuring Django for micro-services)
As mentioned above, chances are that when writing a single micro-service, you are not gonna use all the features of Django (although you might still use most of them across multiple micro-services for the same project). One of the first things to do is therefore to look at Django's settings.py and remove whatever is not going to be needed.
The first thing we are going to look at is the default INSTALLED_APPS:
'django.contib.admin', is only going to be useful (probably only during development) if you are going to be connecting to a database. A micro-service that does not connect to any Django-supported data backend will not need this application. A micro-service that would provide the time of day for example will not need that.
'django.contrib.auth', will be useful if the micro-service you are writing is in charge or keeps track of authentication. Authentication is mostly useful for micro-services exposed to the public. Some microservices are not, they are only called from other micro-service. Be mindful that the defaults for django.contrib.auth are to use a database backend. In a significant portion of cases, this config line can be omitted as well.
'django.contrib.contenttypes', has 2 main uses; it allows for things like GenericForeignKey as well as creating default admin permissions for different models. If you are not going to be using any of those 2 features (which is very likely with micro-services), you can also remove this application.
'django.contrib.sessions', is used to keep a "session" which is a permanent state between requests. Most micro-services are stateless and you will therefore most likely not need it. You might want to keep it for DEBUG=True though if your micro-service uses Django REST Framework and requires authentication and uses session authentication or if you use the admin. If you don't know what Django REST Framework is, don't worry about it right now, we will cover it extensively later in this book.
'django.contrib.messages', is the application responsible for displaying success/failure messages and other informations on web pages and inside the admin. If you use the admin in DBUG=True, you will have to keep it, otherwise you can remove it.
Finally 'django.contrib.staticfiles', is used by both the admin and Django REST Framework's browsable interface. You will only most likely use it only if DEBUG=True.
A typical INSTALLED_APPS for a micro-service might look like this:
'django.middleware.common.CommonMiddleware', is the only truly required line in this config as it is central to Django's internals.
'django.middleware.security.SecurityMiddleware', and 'django.middleware.clickjacking.XFrameOptionsMiddleware', are strongly recommended if your micro-service is going to be accessible publicly but are not needed if that particular micro-service is going to be only accessible by other micro-services.
'django.contrib.sessions.middleware.SessionMiddleware', is the middleware associated with 'django.contrib.sessions',, so you'll only need it if 'django.contrib.sessions', is part of your INSTALLED_APPS.
'django.middleware.csrf.CsrfViewMiddleware', is useful if your micro-service is going to be accessed by web pages created by that same micro-service. Most micro-services are only serving API's so it is very unlikely. You will need it though if you are using the admin. This middleware is not needed if you use Django REST framework as it have its own implementation of CSRF.
'django.contrib.auth.middleware.AuthenticationMiddleware', is not going to be used unless you chose to have 'django.contrib.auth', in your INSTALLED_APPS.
Finally 'django.contrib.messages.middleware.MessageMiddleware', is the middleware associated with 'django.contrib.sessions', and will not be useful unless 'django.contrib.sessions', is in your INSTALLED_APPS
A common MIDDLEWARE configuration for a micro-service could look like this:
TEMPLATES is only used to render HTML pages which is very unlikely for most micro-services. You might need it though if you decided to use the admin or Django REST Framework's browsable interface when DEBUG=True
TEMPLATES in the default settings.py looks like this:
You can see some context_processors are being loaded. Once again, some of those are linked to applications listed in INSTALLED_APPS. Namely 'django.contrib.auth.context_processors.auth', and 'django.contrib.messages.context_processors.messages', should be removed if you have decided to not list the corresponding applications in INSTALLED_APPS.
Another default that does not appear in this particular config are templates loader. Loaders are used to determine how to look for templates. The default value for this is:
'django.template.loaders.filesystem.Loader', is used to load templates from the filesystem (ie: if you have a project-wide templates folder). This is most likely not the case for most micro-services, even when DEBUG=True, it can therefore be stripped as well.
A common TEMPLATES declaration for micro-services could look like this: