Understand the size
We all agree that serving 100000 requests per second is a bit different from serving 100 requests per minute. The approach to solving the problem is different. This is why it is important to understand what is the “size of the input” or, put another way, how big the problem is.
Otherwise, there are two scenarios. The best case is that you spend 6 months designing and implementing a system that is used at 10%. The worst case is when you spend 1 month designing a system that is used at 180%. If the best case is a waste of time/resources, you don’t want to find yourself in the worst case.
To avoid this situation, we have to ask the right questions.
– How many requests the system should satisfy?
– What is the expected response time?
– How many resources do we have?
– What about deadlines?
The right questions depend on the context, but the target is one and only one: understand the size of the problem.
Stand on the shoulders of giants…
I’ll tell you a secret. The chance someone else already solved your problem is high. Very high. All you have to do is a search in the literature to find out if there is a solution for a problem matching your use case. Avoid home-made solution to well-known problems, they only bring other problems. There are lots of companies having “their Hibernate”, “their Kafka”, etc. because:
– “We have a different use case” (I want to see it)
– “The performance of technology X is not enough for us” (really?)
– “We can do it better” (this is the funniest 😆)
Bottom line: once you know your requirements and the size of the problem, do a search in the literature.
There is no point in reinventing the wheel.
…But remember you are not a giant
It’s ok to build-up on existing solutions, but avoid overshooting. Remember that you are not Google. In the ocean of cool technologies out there, the most famous/innovative/used not always is the best one for you. Deploy a Kafka cluster to process 5 messages a day probably is not a good idea. Choose the technology that will do the job with the minimum complexity. This decision will pay you in the long run.
Implement your solution trying to make it understandable by your grandma. Avoid fancy and super complex implementations. Put them aside in favour of a simple and understandable one. This will make the code more maintainable. Leave optimization to the moment they are necessary.
More formally, your implementation should follow the Rule of Least Power. The original rule refers to the choice of the programming language. In this context, we can read it as:
“Among the available solutions, choose the least powerful one that can solve your problem.”
I learned this rule when I started using functional programming. It allows you to implement solutions with an unpaired elegance. However, such solutions sometimes are too much complex. I prefer an implementation a little less elegant and efficient, but a lot more understandable and maintainable. You will not be the only one reading your code.
We are engineers, our work is to solve problems, in whatever form they appear. We need to apply our engineering skills and analyze the problem pragmatically to deliver the correct solution. We need to remember that it is not the solution for producing the desired result. It is the one that does it requiring the least effort, and with the least complexity. I hope the methodology I described in this story will help you make a step toward such an achievement.