![\"book](\"\/assets\/img\/on-philosophy-of-software-design-0.png\")
<\/p>\n\n
Software Design<\/h1>\n\n
Without any surprise, \u201cA Philosophy of Software Design\u201d is about Software Design, ie. the macro view one should take to architecture or design programs<\/strong>: looking at the problem and decomposing it into independent smaller pieces. For us at BlaBlaCar: what are the user features we have to support, and how should they be architectured.<\/p>\n\n So the book is not about Software Development (ie. programing languages, for loops, objects and classes, inheritance, etc.), nor Software methodologies (ie. Agile, versioning, design patterns, etc.), although they are of course all linked.<\/p>\n\n This focus on Software Design is fundamentally important. As a discipline, it\u2019s not present that much in the day to day work of Software Engineers<\/strong>. \nThat\u2019s very paradoxal given how much leverage investing in good Software Designs can have - we\u2019re talking in order-of-magnitude improvements here. That\u2019s where we should invest more of our time.<\/p>\n\n We won\u2019t try to sum up the book in its entirety here. You can read the book\u2019s table of content for that: it gives a pretty strong overview of the book already, and enough food for thought to start designing software better. Let\u2019s focus on some key points that I\u2019ve found relevant for us instead.<\/p>\n\n Software Design is all about managing Complexity<\/strong>, and this should be the start of all software initiatives. Ousterhout defines complexity in a pragmatic way, \u201canything related to the structure of a software system that makes it hard to understand and modify\u201d, and proposes that we reason with a cost\/benefit ratio.<\/p>\n\n When a codebase is complex, it costs a lot to do even a small change with low benefit. When an architecture is simple, it costs little to implement a significant change that has a high benefit.<\/p>\n\n We have two kinds of Complexities<\/strong> to deal with. The first is inherent to the problem solved. 2FA login for example is an intrinsically complex feature. This should be challenged and managed. The second kind of Complexity does not relates to the problem solved. Add the Redux library in a legacy web codebase and the code is immediately harder to grasp. This Complexity should be removed.<\/p>\n\n The issue with Complexity is that it\u2019s incremental. It compounds over space (the codebase) and time. A bit of Complexity somewhere is not a critical issue but multiple complex parts there and there can make the global structure extremely challenging overnight.<\/p>\n\n Consider Complexity as a factor to be multiplied<\/strong> and not merely added.<\/p>\n\n Our job as Software Engineers, or Software Designers we should say, is to understand a given problem and manage its inherent Complexity<\/strong> the best we can, and in doing so, remove any unnecessary Complexity. How to do that?<\/p>\n\n First, we should not introduce unnecessary Complexity<\/strong>. The book details a lot of good postures to follow that I will rephrase and illustrate like so: be obvious with the code (eg. via the names you choose for variables, methods, etc.), be consistent in the codebase (eg. use an uniform code style, document conventions and patterns, choose a few Boring Technologies<\/a> and use them extensively), use comments (they should be written close to the code and provide high-level clues) and make impossible states impossible<\/a>.<\/p>\n\n Ousterhout also makes a great deal of a fundamental concept: Modules<\/strong>. Think about anything that can be used to group code and design a modular codebase: \u201cpackages\u201d, \u201ccomponents\u201d, \u201cclasses\u201d, \u201clibraries\u201d, \u201cservices\u201d, etc. Modules help manage Complexity that comes from dependencies between different part of code.<\/p>\n\n We distinguish shallow and deep modules<\/strong>. Deep modules are \u201cthose whose interfaces are much simpler than their implementations\u201d and should be preferred: they hide a big chunk of information and the related Complexity behind a simple interface. The module\u2019s users can then reason from an higher design layer, using high-level Abstractions<\/strong> from the module\u2019s interface without having to know its internals.<\/p>\n\n A good deep module is for example an AutoComplete UI component that has an interface very similar to a standard input, but it deals with suggestions fetching and options sorting and filtering by itself. It removes the \u201cautocomplete burden\u201d from the developper using it, appart from providing some high-level options via its interface if needed.<\/p>\n\n Deep modules follow the Unix philosophy<\/a>: \u201cdo one (hard) thing and to it well\u201d. They therefore help to organize the problem space by layers of Abstractions<\/strong>, and pull the Complexity downwards inside their implementations.<\/p>\n\n That\u2019s a quite challenging task to do, specially when the same people are working across layers. As developers, we need to consistently change posture between being the provider of a module, or its user. This can lead to unfortunate information leakage: \u201cthe same knowledge is used in multiple places\u201d, and a trivial change then requires to update multiple different modules.<\/p>\n\n As Rich Hickey would say<\/a>: \u201cDesign is separating things into things that can be composed\u201d. We separate things by hiding them inside modules.<\/p>\n\n In reading this book from the perspective of a very Business-driven company<\/strong>, we can quickly think about arguments that will be made against this vision.<\/p>\n\n With Deep Modules, don\u2019t we risk to over-engineer our code? Not more than with shallow code, the book argues. It\u2019s not because an interface is designed with an high-level, so quite general purpose, Abstraction that its implementation is supporting any potential use case.<\/p>\n\n Is it really \u201cAgile\u201d to take time for the design of the software? Good question. The book argues that with Agile we are tented to ship features with a \u201ctactical mindset\u201d: as soon as possible without organizing them in a robust design. Should we start by designing everything before coding then? Of course not: \u201cit isn\u2019t possible to visualize a complex system well enough at the outset of a project to determine the best design\u201d.<\/p>\n\n Instead, we should move incrementally on the problem<\/strong>, like we classically move incrementally on the implementation of a feature. We need to welcome architectural change in the middle of a project (and even maybe throw our first code), add abstractions one after another, and refactor on the go with your design becoming better.<\/p>\n\n For Ousterhout, an iteration\u2019s output should be Abstractions, not features.<\/p>\n\n To conclude, we can only praise the book for being able to highlight very well the challenges of Software Design. There is no absolute method. Any tactic used to the extreme can do more harm than good.<\/p>\n\n That\u2019s where Software Design is a collective, always ongoing, effort. The Design of a program should be documented by the team responsible for it, discussed and adapted regularly<\/strong>, for example with Defenses of Design documents<\/a>.<\/p>\n\n \u201cWorking Code Isn\u2019t Enough\u201d<\/strong>, Ousterhout argues. We need to invest into decomposing the problem at hand. This requires time to understand and challenge it properly. The code produced should be strategic, not tactical, and that\u2019s specially true for parts of our software that serve long-lasting features.<\/p>\n\n But what is \u201clong-lasting\u201d, we may ask? Actually lot of things are long-lasting-enough to benefit for a proper Software Design. It would be a fault to not take the few person-hours that are needed to produce a robust enough design, even for a part of the codebase that is not part of the very few highly critical pieces of code.<\/p>\n\n That\u2019s the mindset we followed on some projects that were described as temporary set of features by the Business. We nevertheless took the time to understand the problem space, design frontend-oriented endpoints that provided enough high-level Abstractions to iterate with minimal effort on the feature. This has been very valuable in shipping the feature on time, and working on it again after that.<\/p>\n\n The corollary of Software Design is that some projects will lead to surprising big and costly software changes<\/strong>. That\u2019s expected. By adding new features requirements or new constraints, the problem space of a program can fundamentally change. We have to adapt the program structure deeply for the new features it has to support.<\/p>\n\n That\u2019s true when following a Software Design mindset, that\u2019s also true without. It\u2019s better to take the time needed for such a refactoring than to hack around and force the code into doing something it\u2019s not designed for. That\u2019s would be the best way to shoot ourselves in the foot by introducing significant technical debt that will soon impact the team negatively.<\/p>\n\n Let\u2019s also use the book ideas to reason at a bigger scale. Because \u201cSoftware Architecture and Organization are two side of the same coin\u201d<\/a>, lot of the insights on managing Software Complexity can be applied to the way we organize ourselves<\/strong>. There is a bridge to be made between \u201cA Philosophy of Software Design\u201d, \u201cTeam Topologies\u201d<\/a> and others ressources like \u201cThe Effective Engineer\u201d<\/a> that use the Complexity prism also (eg. code complexity, system complexity, product complexity, organization complexity).<\/p>\n\n An effective team communicates much like optimized code: with clarity, modularity, and a focus on simplicity.<\/p>\n\n Addy Osmani<\/p>\n<\/blockquote>\n\n A Philosophy of Software Design from John Ousterhout is ~190 pages long but a quick read, specially when you go over the examples given quickly and focus on the most relevant insights for your current context. It will be quite an unusual read in the Computer Science section of your library and hopefully give you a new perspective on Software Engineering.<\/p>\n\n \u2013<\/p>\n\n This post has originaly been shared internaly at BlaBlaCar. This is a slightly edited version.<\/em><\/p>\n\n","pubDate":"Fri, 01 Mar 2024 00:00:00 +0000","link":"https:\/\/guillaume.wuips.com\/tech\/posts\/2024-03-01-on-philosophy-of-software-design","guid":"https:\/\/guillaume.wuips.com\/tech\/posts\/2024-03-01-on-philosophy-of-software-design"},{"title":"Operating Node.js in Kubernetes at scale at BlaBlaCar","description":" At BlaBlaCar, there is a dedicated team of frontend engineers in charge of the\nuser-facing web application. This frontend team\u2026 is also a backend team.<\/p>\n\n Every BlaBlaCar web user consumes a web application. It\u2019s a Single Page\nApplication (SPA) based on Typescript and React. Alongside this SPA, the\nfrontend team also owns the Node.js service used for Server-Side Rendering\n(SSR).<\/p>\n\n Why do we need SSR? When a user asks for a web page, we want to return this page\nfully rendered for both performance and SEO reasons. We don\u2019t want the user, or\na crawler, to receive an empty SPA skeleton without any content included yet.<\/p>\n\n But operating this Node.js server at scale in a Kubernetes cluster is not as\ntrivial as it might look, especially for a team without previous infrastructure\nbackground. We had to understand deeply how Node.js works under the hood,\nmeasure how our server behaves with traffic and\u2026 learn Kubernetes. Such a big\ntopic when the day to day job is about writing new UI components!<\/p>\n\n Let\u2019s look at the challenges we have overcome in the past few months. We will\nshow how we use CPU and memory metrics and other Kubernetes options to configure\nhow we scale Node.js. This article assumes the reader has basic knowledge about\nKubernetes or similar container orchestration tools.<\/p>\n\n All web traffic targeting our SPA goes through our Node.js server. We need to\nadapt the resources that support the service to match how much traffic we have\nto serve.<\/p>\n\n We\u2019re scaling horizontally to handle more or less traffic. Like the majority of\nmicroservices, we add more Pods when traffic goes up. We remove some Pods when\ntraffic goes down. Each Pod contains its own Container of our Node.js server. It\nserves part of the web traffic directed to the application in the cluster.<\/p>\n\n The point in this setup is to know when we need to scale up and down. It\u2019s not a\nsurprise that we have to monitor the CPU for that as it indicates how much we\nuse the Pod.<\/p>\n\n When the average CPU on all Pods exceeds a certain threshold, we add more Pods.\nWhen the CPU goes down, we remove Pods. This allows us to only use resources\nwhen needed. We don\u2019t want to pay for 20 Pods when only 5 would be enough.<\/p>\n\n The Horizontal Pod Autoscaler (HPA) controls this scaling. Let\u2019s say our HPA\nconfig is this one:<\/p>\n\n And our Deployment config is the following:<\/p>\n\n Kubernetes instructs the workload resource to scale up when we reach\n We chose resources limits and requests values after multiple iterations. The\nsmaller the values, the smaller the Pods are. With small Pods, we need more\nreplicas to handle the load than with bigger ones. We observed a clear overhead\non the memory side because of Node.js loading the service\u2019s code in memory. This\nlimits how small a Pod can be.<\/p>\n\n In our example, we never go below 4 or above 30 Pods. Why 4 and not 1 or 2? This\nallows us to be sure we always have some servers running, on different Nodes.\nThe This basic setup was a good first step to operate our service on the cluster\nwithout too much issue. There were parameters we had to carefully choose here,\nby classic trial and error. The main one is of course the CPU target percentage.\nWe don\u2019t want it to be too low to not provision unused resources, but not either\nso high that scaling happens too late or that we end up in a noisy neighbor\nsituation. 70% works well for us.<\/p>\n\n Our CPU-based HPA worked great until a calm and sunny Saturday afternoon a few\nmonths ago. A random credential-stuffing attack targeting BlaBlaCar started. It\nsoon generated a lot of traffic and overloaded our Pods.<\/p>\n\n Such an attack is usually not a big deal for us. We have the security tools and\nthe scaling in place to mitigate the potential impact for our users. It should\nnot even require any human action. But this time the traffic received by our\nNode.js servers doubled in less than 30 seconds. Pods were overloaded and unable\nto handle the traffic, and our HPA started scaling down our Deployment! Why?<\/p>\n\n CPU is not the only metric to have in mind when operating code. Memory is here\ntoo. We may have learned in engineering school how the two are actually like the\ntwo sides of the same coin.<\/p>\n\n During the incident the CPU usage dropped significantly. This justifies the HPA\nasking the workload to scale down.<\/p>\n\n In the meantime, memory usage for Pods was significantly more important than\nusual.<\/p>\n\n What happened? Pods had to handle twice as much traffic. The Server-Side\nRendering endpoint was consuming much more memory than usual. We were making\nmore calls to our backends to fetch data and we were rendering more pages.\nBackend services were struggling too. This led to more IOs and more data waiting\nin memory. Pods were overloaded, yet the limited metric was not CPU, but memory.<\/p>\n\n A snowball effect took place: some pods crashed with Out Of Memory errors (OOM).\nBecause memory usage reached the config limit level, Kubernetes killed the Pods\nas a sanction. New pods were started but crashed too. The remaining ones were\nnearly in a frozen state: memory bloated but CPU not at all. This led to the HPA\nasking to scale down Pods: the CPU metric was indeed below the target.<\/p>\n\n We were DDOSed and had to manually scale up the Deployment.<\/p>\n\n Important notice: this has been written before Node.js 12 and the addition of\ncontainer-aware memory. Using Node.js 12 or higher, the service would not have\ncrashed because of OOM. Instead, it would have reached the container memory\nlimit and then seen reduced performance because of some memory swapping.<\/em><\/p>\n\n We learned the hard way that the CPU metric is not enough for our HPA. We have\nto read memory usage too.<\/p>\n\n Of course, memory does not reflect the Node.js server activity in a linear way.\nSo it\u2019s not about making the metric the principal indicator we\u2019re using to scale\nup or down. But we do have to make the HPA aware of a certain limit.<\/p>\n\n Turns out it\u2019s very easy to do with Kubernetes. We have to add another metric in\nthe HPA config, like so:<\/p>\n\n There are two ways to understand this addition. We can say we\u2019re asking the HPA\nto scale up if memory usage reaches 85%. Or we can say we\u2019re asking the HPA to\nnot scale down if memory is over 85%. Both are valid. In our situation the\nsecond one suits the rationale quite well.<\/p>\n\n If we had this kind of memory limit in place at the time of the incident, the\nmemory usage would never have reached the problematic level. And Pods would not\nhave scaled down.<\/p>\n\n Writing the post mortem made the team invest more time in the infrastructure\ndimension of the service. We fixed our HPA, then started some background\ninvestigation on how to operate Node.js in Kubernetes better.<\/p>\n\n We first looked at how others were doing. Some are scaling on a custom Node.js\nmetric: the Event Loop Utilization\n(ELU)<\/a>. Can we do the\nsame? We tried but finally rolled back to the CPU\/memory duo. ELU seems\ninteresting when you have important computation off the main thread, or garbage\ncollection impacting the CPU measurement too much. This was not our case.<\/p>\n\n Instead, we invested time in having a more robust and elastic setup. We\ndistributed Pods on nodes with\naffinity<\/a>\nand topology\nspread<\/a>\nconstraints to reduce the risk of seeing the Deployment impacted too much when\nlosing a Pod.<\/p>\n\n We\u2019re also scaling up more rapidly and with more granularity by configuring our Pods to be smaller, cleaning the assets built into our Docker image and using a distroless<\/a> base Node.js image. With more Pods, but smaller ones, we are consuming nearly the same amount of resources with only a small overhead. We\u2019ve gained in granularity as the HPA is scaling up and down.<\/p>\n\n Because it is useless to have a lot of small Pods for granularity if the\ncontainer takes a lot of time to start, we have improved our procedure, going\nfrom a start-up time of 2 minutes in the worst scenario to something under 30\nseconds.<\/p>\n\n We also leveraged HPA\u2019s new behavior config to scale up as fast as possible but\nscale down slowly. Paired with Kubernetes\u2019 scaling window, this prevents\n\u201cflapping\u201d, i.e. seeing the Deployment being scaled up and down continuously.<\/p>\n\n To sum up the key points of our infrastructure setup:<\/p>\n\n Our service now consumes less resources when not needed, for example at night,\nbut is also able to handle short and big traffic increases better. This reduced\nthe amount of false positive alerts for the on-call person to nearly 0. More\nimportantly, we now handle traffic changes better, improving the user experience\noverall.<\/p>\n\n This is backup post, primarily published on BlaBlaCar\u2019s blog<\/a><\/p>\n\n","pubDate":"Fri, 01 Jul 2022 00:00:00 +0000","link":"https:\/\/guillaume.wuips.com\/tech\/posts\/2022-07-01-operating-nodejs-in-kubernetes","guid":"https:\/\/guillaume.wuips.com\/tech\/posts\/2022-07-01-operating-nodejs-in-kubernetes"},{"title":"Where is my store stored?","description":" Some of us have concepts in mind that they keep returning to over an over, as if\nobsessed with them. For me as a frontend software engineer, it\u2019s all about the\nconcepts of State<\/em> and Store<\/em>. I keep looking at technical issues through this\nlens - current and previous colleagues know this all too well (sorry\ncolleagues!). I\u2019ve actually already written about\nit<\/a> in the past.<\/p>\n\n I have changed companies a few month ago to join BlaBlaCar and the very cool\nfrontend team building and maintaining its web app. After this move I find\nmyself in situations that make me change how I think about the \u201cfrontend Store\ndilemma\u201d I kept referring to. There is now a new question I ask myself again and\nagain looking at our frontend codebases:<\/p>\n\n Where is the store stored?<\/p>\n\n Let\u2019s go back to the difference between the concepts of State<\/em> and Store<\/em>\neven though we sometimes use the two terms indistinctively in frontend\ncodebases. In every frontend app, data (the State) is stored somewhere (the\nStore).<\/p>\n\n When we\u2019re thinking about states, we\u2019re in the world of ideas. We ask\nquestions about what those states represent, which transition between states are\nvalid, or how should we shape our states.<\/p>\n\n When we deal with stores, we\u2019re not in the world of ideas anymore. We\u2019re back in\nthe reality of the code that runs somewhere, mainly in a Javascript runtime. We\nask where do we store out data, how do we read it, how do we write it, how can\nwe subscribe to changes?<\/p>\n\n In this post, we\u2019re focusing on the \u201cstore\u201d kind of questions.<\/p>\n\n The reality of the store that lives \u201csomewhere\u201d in the code is the very\nproblem. The store in itself, whether a simple variable or a full Redux store,\nhas be stored somewhere in our browser, NodeJS or other fancy javascript\nengine runtime.<\/p>\n\n Deciding where our stores are stored<\/em> is not trivial and comes with big\nimpacts. Let me draw here the three main potential locations we can use for our\nstores.<\/p>\n\n We can use modules to host our stores. It\u2019s quite convenient and it makes full\nuse of the Javascript ES6 language.<\/p>\n\n Let\u2019s say we have a module storing Cars (our state here).<\/p>\n\nComplexity<\/h1>\n\n
![\"2](\"\/assets\/img\/on-philosophy-of-software-design-1.png\")
<\/p>\n\n![\"complexity](\"\/assets\/img\/on-philosophy-of-software-design-2.png\")
<\/p>\n\nManaging Complexity<\/h1>\n\n
![\"abstract](\"\/assets\/img\/on-philosophy-of-software-design-3.png\")
<\/p>\n\n![\"abstract](\"\/assets\/img\/on-philosophy-of-software-design-4.png\")
<\/p>\n\nBut in real life?<\/h1>\n\n
![\"iteration](\"\/assets\/img\/on-philosophy-of-software-design-5.png\")
<\/p>\n\nThe challenges of Software Design<\/h1>\n\n
![\"other](\"\/assets\/img\/on-philosophy-of-software-design-6.png\")
<\/p>\n\n\n
![\"Abstract](\"\/assets\/img\/bbc-infrastructure.png\")
<\/p>\n\nScaling when\u00a0needed<\/h1>\n\n
Scaling on\u00a0CPU<\/h2>\n\n
\/\/ hpa.yml<\/span>\napiVersion<\/span>:<\/span> autoscaling\/v2beta2<\/span>\nkind<\/span>:<\/span> HorizontalPodAutoscaler<\/span>\nmetadata<\/span>:<\/span>\n name<\/span>:<\/span> spa-service<\/span>\n labels<\/span>:<\/span>\n app<\/span>:<\/span> spa-service<\/span>\nspec<\/span>:<\/span>\n scaleTargetRef<\/span>:<\/span>\n apiVersion<\/span>:<\/span> apps\/v1<\/span>\n kind<\/span>:<\/span> Deployment<\/span>\n name<\/span>:<\/span> spa-service<\/span>\n minReplicas<\/span>:<\/span> 4<\/span>\n maxReplicas<\/span>:<\/span> 30<\/span>\n metrics<\/span>:<\/span>\n -<\/span> type<\/span>:<\/span> Resource<\/span>\n resource<\/span>:<\/span>\n name<\/span>:<\/span> cpu<\/span>\n target<\/span>:<\/span>\n type<\/span>:<\/span> Utilization<\/span>\n averageUtilization<\/span>:<\/span> 70<\/span>\n<\/code><\/pre><\/div><\/div>\n\n\/\/ deployment.yml<\/span>\napiVersion<\/span>:<\/span> apps\/v1<\/span>\nkind<\/span>:<\/span> Deployment<\/span>\nmetadata<\/span>:<\/span>\n name<\/span>:<\/span> spa-service<\/span>\n labels<\/span>:<\/span>\n app<\/span>:<\/span> spa-service<\/span>\nselector<\/span>:<\/span>\n matchLabels<\/span>:<\/span>\n app<\/span>:<\/span> spa-service<\/span>\ntemplate<\/span>:<\/span>\n metadata<\/span>:<\/span>\n labels<\/span>:<\/span>\n app<\/span>:<\/span> spa-service<\/span>\n spec<\/span>:<\/span>\n containers<\/span>:<\/span>\n -<\/span> name<\/span>:<\/span> spa<\/span>\n image<\/span>:<\/span> spa<\/span>\n resources<\/span>:<\/span>\n limits<\/span>:<\/span>\n cpu<\/span>:<\/span> 700m<\/span>\n memory<\/span>:<\/span> 450Mi<\/span>\n requests<\/span>:<\/span>\n cpu<\/span>:<\/span> 600m<\/span>\n memory<\/span>:<\/span> 320Mi<\/span>\n<\/code><\/pre><\/div><\/div>\n\naverageUtilization<\/code>. In our situation it means the Deployment will scale up when\nthe average CPU reaches the target of 70% of the requested CPU value. The\nDeployment scales down when the metric is below the target.<\/p>\n\nmaxReplicas<\/code> helps us staying in control of the scaling. We don\u2019t want to\ndiscover that the service scaled to hundreds of Pods for an unknown reason\nduring the night without being paged first.<\/p>\n\nWhen CPU is not\u00a0enough<\/h1>\n\n
Back to\u00a0school<\/h2>\n\n
![\"Graph](\"\/assets\/img\/operating-drop.png\")
<\/p>\n\n![\"Graph](\"\/assets\/img\/operating-memory-important.png\")
<\/p>\n\n![\"Graph](\"\/assets\/img\/operating-pod-drop.png\")
<\/p>\n\nScaling on\u00a0Memory<\/h1>\n\n
\/\/ hpa.yml<\/span>\napiVersion<\/span>:<\/span> autoscaling\/v2beta2<\/span>\nkind<\/span>:<\/span> HorizontalPodAutoscaler<\/span>\n...<\/span>\nspec<\/span>:<\/span>\n ...<\/span>\n metrics<\/span>:<\/span>\n -<\/span> type<\/span>:<\/span> Resource<\/span>\n resource<\/span>:<\/span>\n name<\/span>:<\/span> cpu<\/span>\n target<\/span>:<\/span> \n type<\/span>:<\/span> Utilization<\/span>\n averageUtilization<\/span>:<\/span> 70<\/span>\n -<\/span> type<\/span>:<\/span> Resource<\/span>\n resource<\/span>:<\/span>\n name<\/span>:<\/span> memory<\/span>\n target<\/span>:<\/span> \n type<\/span>:<\/span> Utilization<\/span>\n averageUtilization<\/span>:<\/span> 85<\/span>\n<\/code><\/pre><\/div><\/div>\n\n![\"Graph](\"\/assets\/img\/operating-pod-fixed.png\")
<\/p>\n\nBetter operating Node.js in Kubernetes<\/h1>\n\n
\n
\n\n![\"Abstract](\"\/assets\/img\/where-is-store-stored-illustration.png\")
<\/p>\n\nState vs Store - one more time<\/h1>\n\n
The multiple places we can store our stores<\/h1>\n\n
Modules<\/h2>\n\n
\/\/ cars\/store.js<\/span>\n\nconst<\/span> _cars<\/span> =<\/span>