ClojureScript. Amplified.

"… we're going to build a web app with storage, authentication and everything …"

But how?

My favorite building blocks 🤩

I have tried out some tools that I find very useful. As you may have guessed, the code is developed with ClojureScript. A language I appreciate more each day. Developing an app, without the Interactive REPL? That would be taking a huge step backwards.

I have found that Material-UI works really well in the development of the app I'm working on. The components library cover many usage scenarios, it is well documented and there's a lot of code examples to learn from.

Recently, I added Storybook and really like the idea of developing user interfaces according to Component Driven Design. It reminds me of how I experiment with code using Test Driven Development. Just like TDD, I think tools like Storybook and Devcards changes the way we develop, to the better. Start small & build quality in.

Let's begin with a building block from my favorites list: AWS Amplify.

A basic setup for AWS Amplify & ClojureScript

The official AWS Amplify documentation is all about JavaScript. In this post, and in this example code repo, you will find a ClojureScript setup that I think will get you up & running quickly. First, set up AWS Amplify by following the first parts of the Official getting started guide. You will be creating an AWS account and installing a CLI to initialize your setup.

If want more details, have a look at Hey Webpack, Hey ClojureScript. It's a post describing a ClojureScript & Amplify specific setup that is used in the example repo.

Start coding, but where? 🤔

A pitfall might be where to place the auto generated client code from Amplify. I usually put files like the aws-exports.js at the same level as the root namespace.

In the example code repo, the ClojureScript source code lives in the src/main folder. That's also where I've added the aws-exports.js file.

Configure Amplify in the init function of your app.

Login & Logout with ClojureScript & the Amplify CLI

Adding authentication can easily be done by using the Amplify CLI.

From the docs:

"... The Amplify CLI supports configuring many different Authentication and Authorization workflows, including simple and advanced configurations of the login options, triggering Lambda functions during different lifecycle events, and administrative actions which you can optionally expose to your applications ..."

Use the Auth features in Amplify UI

There's a very useful Higher-Order React Component (aka HOC) in the AWS Amplify UI Library, that will render login and signup views.

I've chosen this less-lines-of-code approach by using the withAuthenticator HOC. To customize things like signup fields or styling, there is also a fully customizable Authorization component available in the Amplify UI library. Or you can use your own components.

The views entry point.

The passed in component is wrapped in the withAuthenticator HOC.

If you already have browsed the code in my example repo, you may have noticed the Reagent reactify-component and as-element functions. These functions are needed when passing components between ClojureScript and JavaScript.

Read more about React Components, elements and instances and how the interop is done with Reagent.

Create, read, update and delete

So far, we have used built in features that will render UI views and handle all auth communication with a backend service. To store, read and update data, we can use the Amplify API feature. Use the CLI to create a backend:

I have chosen GraphQL in my example code, but there's also a REST API option. When using GraphQL, all you need to do is to define data models. Amplify will create the backend infrastructure according to your models. In a matter of minutes, there are AWS AppSync endpoints & DynamoDB tables created for your app.

GraphQL

Here's a GraphQL model. An authenticated user should be able to add data to the profile, like a summary or an "About me" text. The user will be able to create, delete and update the data in the profile. Other authenticated users can read the data when they visit the user's profile.

The schema files belongs to the backend, and lives in the amplify folder of your repo.

async ClojureScript

The Amplify Client Library handles the communication with the backend services. All calls are asynchronous. In ClojureScript, you can use core.async. I haven't yet dig into the core.async stuff, so I'm using Promises here. I think it's a quite straight forward approach.

The data going through the wire is JSON. To make the developer experience better, I think it might be a good idea to add conversion functions to-and-from Clojure data structures. Like this one, converting user settings JSON to Clojure data:

Read more about the ^js type hints in ClojureScript.

Finally: Ship it! 🚢

Here's an excerpt from an amplify.yml that defines the build steps. If you add a definition file in your repo, AWS Amplify will use it in the CI/CD process.

This is the amplify.yml that I use in my example code repo.

There's more details at the official AWS docs about build settings and about amazon-linux-install.

Add UI building blocks to the amplified app

Now we have our amplified ClojureScript app. Let's add UI building blocks from Material-UI, a library that is very popular in the JavaScript & React ecosystem. Here, I'm using it via a ClojureScript library called reagent-material-ui. The library makes it possible to write components using Clojure data structures. You can read more about Material-UI and ClojureScript in my previous post Material Design in a Functional World.

Totally unnecessary css styling used here, but why not?

Testing the UI building blocks with ✨ Storybook ✨

I find Storybook very useful when experimenting with components and user interfaces in isolation. Being able to focus on one specific thing. There's many different aspects of developing an app, and it is sometimes difficult to handle all of them at the same time.

Storybook is a place for trying out things like css, styling, different viewports and UI events. You won't try out the AWS Amplify code here. The Interactive REPL is a better tool for that.

You'll find more about Storybook in Component Driven ClojureScript with Storybook.

Finally, check out my GitHub repo for ideas on how to create apps with the building blocks AWS Amplify, Material-UI, Storybook and ClojureScript.

That's a combination I'd like to call ✨ClojureScript. Amplified. ✨

^{Top photo by Drew Patrick Miller on Unsplash}

Material Design in a Functional World

"... Material is a design system created by Google to help teams build high-quality digital experiences for Android, iOS, Flutter, and the web ..."
^{from material.io}

ClojureScript & React is A 💕 Story

With Clojure, you can write functional & minimalistic code. With ClojureScript and Reagent, you can write functional & minimalistic React components. By adding re-frame to it, we also have a powerful way of handling state changes and triggering events.

The code you write in ClojureScript is instantly compiled to JavaScript React components that runs in your browser. The entire JavaScript ecosystem is available with this kind of setup.

"... React components for faster and easier web development. ..."
^{from material-ui.com}

Material-UI

Material-UI is a very popular React framework. And we can use it with ClojureScript. But the JavaScript interop can sometimes be bit of a hassle. Fortunately, there is a great ClojureScript library called reagent-material-ui that simplifies this a lot.

One thing to notice is that the Material-UI components use React Hooks heavily to inject styling and theming into components. This is something to think about when using Reagent. Basically, just use the functional component syntax [:f> my-component] and you’re all good.

Have a look at the reagent-material-ui docs about common pitfalls in the React/Reagent interop and a fully working example in this GitHub repo.

Storybook: a playground for components

Here’s me playing around with the Card component, using ClojureScript and Storybook. I describe the Storybook integration in detail in this post.

The source code is available at GitHub.

No copy-paste?

I guess it can be a bit overwhelming for a Clojure developer to look at example code from the Material-UI site, with JavaScript/JSX syntax that is quite verbose (you’ll be even more overwhelmed by the TypeScript examples).

A JavaScript developer would probably copy-paste the example code and make some adjustments. We can’t do that. But it is worth the effort writing the code from scratch when discovering that the result is way more simplistic than the original. The amount of code written is usually half the size of the original JavaScript code. Yeah, that’s Clojure!

ClojureScript to the left, JavaScript to the right. Less is more.

Check out my GitHub repo with examples on how you can implement Material-UI components in ClojureScript, with hooks and styles - while having fun writing functional code.

^{Top photo by Thierry Lemaitre on Unsplash}

Component Driven ClojureScript with Storybook

Component Driven?

Start small. Develop components in isolation. Don’t forget to pause & reflect on the code you’ve written. Can it be split into several small components?

✨ When you’re ready, combine your components to create features.

✨ Add the features to a view.

✨ Add the view to your app.

You can read more about Component Driven User Interfaces here.

Storybook

I’ve just begun to use Storybook and currently learning about it. What I especially like so far, is that the tool encourages you to write UI components that are independent … and functional?

This makes it possible to develop user interfaces in isolation. Just pass data and actions as parameters into your components. Keep it simple. This way of working reminds me a lot of Polylith, an architecture I have used in backend focused Clojure projects and that I enjoy a lot.

ClojureScript & Storybook?

You will find a fully working example in this repo. I have added Storybook to a ClojureScript web app, using shadow-cljs, with additional npm scripts located in package.json.

Update: a new version of shadow-cljs supports ns-regexp for the npm-module target. No need to manually add new stories!

The code in my GitHub repo is a continuation of the work I’ve described in an earlier post, about combining shadow-cljs with Webpack and AWS Amplify.

The Clojure Interactive REPL and Storybook working well together.

No JavaScript quirks?

Well, yeah some. The JavaScript ecosystem is evolving fast, tools are updated with breaking changes and sometimes that causes compatibility issues. In this case it is Storybook and Webpack 5. But I have solved it with a few extra rows of configuration. Don’t worry.

An example of a Storybook story written in ClojureScript

Have a look at the fully working example at GitHub.

Many thanks to Shaolang Ai, that wrote this blog post about shadow-cljs and Storybook that I’ve learned a lot from.

^{Top photo by Etienne Girardet on Unsplash}

Hey Webpack, Hey ClojureScript

✅ ClojureScript is great.

✅ AWS Amplify is great.

🤔 Why are there errors in my browser console?

ClojureScript. Amplified.

Some time ago, I decided to give AWS Amplify a try for a new project. With Amplify, you can quickly create and deploy web apps and AWS backend services by using your command line.

Even though the Amplify code examples and docs are all about JavaScript, I wasn't too keen on abandoning my favorite REPL powered language: Clojure. 🤩

So, I decided to combine two great things: AWS Amplify & ClojureScript!

Amplifed, yes. But with 🔔.

I have learned a lot from this blog post by Robert J Berger about how to setup AWS Amplify with ClojureScript. New versions of the AWS Amplify libraries are released regurarly. The current Major version include breaking changes that probably will cause ClojureScript web apps to raise errors and stop working.

Until recently, I have had to rollback updates and use earlier versions of the Amplify libraries. Nothing wrong with that. But my inner Alarm Bells have been ringing constantly since noticing the word legacy in the official Amplify docs for those library versions (the aws-amplify-react library and the Amplify Auth feature in particular).

The newer Amplify libraries rely on Webpack specific features under the hood. The example setup described in the official Amplify docs include using react-scripts, that (if I'm not mistaking) will package code with Webpack. That kind of setup is not supported by shadow-cljs, a tool that is very common in ClojureScript projects.

Breaking things

In addition to being stuck using legacy code, the app will break each time I add a third party library from npm. I have learned that it is caused by not having a fixed version for amplify libraries defined in my package.json file. A quick fix is to set all AWS Amplify libraries to a fixed version and use that one forever. A version that is becoming more legacy for each day. 🔔🔔🔔

Fix the broken things

The main issue with AWS Amplify libraries, ClojureScript and shadow-cljs is described in this GitHub thread. There is also a suggested solution hinted by Thomas Heller, the creator of shadow-cljs.

A solution: shadow-cljs & Webpack working together

I have created an example app using AWS Amplify and ClojureScript in this repo. Here, shadow-cljs will only create a target JavaScript file containing requires to all of the third party JavaScript dependencies.

Webpack will use that target to create a JavaScript bundle. In a continuous deploy environment, this can easily be automated by executing two scripts instead of one (the shadow-cljs build and webpack).

This solution makes it possible to use the very latest versions of the AWS Amplify libraries, without any css quirks or being forced to use legacy code. Hey Webpack, Hey ClojureScript!

Nowadays, my inner alarm bells are nice & quiet.

^{Photo by Kamil Pietrzak on Unsplash}

Confessions of an Open Source Contributor

I am one of the contributors to an Open Source project called zookeeper. It's an Apache ZooKeeper client made for Node.js. You can find the package on npm and the source code at GitHub.

What’s ZooKeeper?

"Apache ZooKeeper ... providing configuration information, naming, synchronization and group services over large clusters in distributed systems ..."

In short, Apache ZooKeeper is made of two parts: a service, and clients that communicate with it. Included in the packaging from Apache, there is a client made for the Java platform and also a C based client. You will most likely find one for your favorite language out there as a third party Open Source project.

Why joining Open Source?

A couple of years ago, I was part of a team in an organisation that use Apache ZooKeeper heavily. The team maintains a Node.js based app that communicates with a ZooKeeper service.

One day, the app began to raise strange warnings and unexpected errors. We looked into it and found out that the reason was probably caused by the client no longer being compatible with the API of the service.

There wasn't much happening in the client's repository:
unanswered questions from users, issues not taken care of and pull requests not being reviewed.

The project was abandoned.

So, we used a different client that seemed fairly updated and stable. The strange behavior in our app was now gone. Great! It worked on our machines and in production. All good!

🙋"Hey, team. I can't install your app on my Windows machine."

Sometimes other teams need to run our app locally for testing and such things. It turned out that our new client didn't support Windows at all.

Wait a minute, isn't Node.js supported in all major platforms? Well, yes. But this one is building a Native Node.js AddOn using bash scripts. Windows don't have that.

🤦‍♂️

Even though it wasn't my fault only, all I could think of is that it was my fault. I have ruined it for the teams at this workplace.

What else to do than try fix it on my spare time? So, I grabbed the source code to try to understand what's going on behind the scenes.

😭

And that's how my Open Source journey began: triggered by guilt and feeling ashamed.

Coding in the Dark

Late nights and weekends were spent trying to figure out how to solve this huge problem, including C/C++ code, CMake, bash and node-gyp.

One day, I figured it out. I had succeeded in making it work on Mac OS X, Linux and on Windows! The next day - a sunny morning in beautiful Stockholm - I went to the office smiling. Exhausted by the lack of sleep, but happy.

😎

Do I have to fix everything? 😟

While struggling with the source code, I realized there were more things that should be fixed. By then, I also realized that this Open Source project suffered from similar issues as the one we had replaced earlier: there wasn't really that much activity in the repo.

This is probably a common pattern: you work with something on a daily basis and contribute to Open Source projects. The years go by, you drift way from the project, simply because you now work with other things unrelated to the project.

How to maintain an Open Source project

Instead of grabbing the next task to spend late nights and weekends with, I decided to learn more about how to maintain Open Source projects.

I found a very useful guideline at GitHub, and I began to add notes about all kinds of things I thought should be done. Other Open Source projects were using labels in a way that I liked, so I added labels like help-wanted and good-first-issue to most of the tasks.

I think labels can help people getting an overview of the tasks and to understand the degree of difficulty.

Every time a question comes up, or someone is filing a bug report, I have replied to most of them within a day or two. I think this is important. The person reporting will know there is someone that has actually received and read the message.

As a result, really nice things have happened. The project has received Pull Requests from people located all over the world. Solving a broad variation of things from typos in the docs, to rewrites of the source code to modern JavaScript.

What's happening today?

Today, I think the project is stable and working well. I don't work nights or weekends. Even though I have moved on and no longer work with ZooKeeper as before, the feedback from users is my main inspiration to keep on working. Also, the amount of users of the client looks pretty good.

It inspires me to do the best I can with the project.

^{Photo by Justin Lim on Unsplash}

Almost like Clojure

Practicing different programming languages has given me new insights in programming concepts, code styles and tools. My current crush is Clojure. I like the functional and data oriented style of it, the simplicity and minimalism. I think my favorite thing with Clojure, though, is how the REPL is used.

“What’s so cool about that repple thing?” 🧐

Testing at your fingertips 🚦

When using a code editor powered by a REPL, you can instantly evaluate variables, code blocks, functions or the entire file, just by a hitting a key combination. The result pops up right next to the cursor. This means that testing the code you currently write is at your fingertips. You are practicing a Deluxe version of Test Driven Development.

No copy-paste here 👮

It is not necessary to copy-paste code snippets into the actual REPL window or in a shell. In fact, you shouldn’t touch the REPL window at all. Because you don’t want to miss out on all the good stuff like autocomplete, hints and smart navigation that you have in your favorite editor.

For Clojurians only? 💭

I really would like to have a similar workflow in other languages too. It turns out there are some pretty good tools out there, the ones I’ve tried out so far are for JavaScript and ✨Emacs ✨.

Interactive JavaScript development 😍

Here’s me playing around with the node-zookeeper code. The cursor never leaves the editor window, the evaluation result is printed out in the Node.js REPL window to the right. More verbose than I’m used too in Clojure, but I think it works well.

Trying out code and ideas like this is a very nice workflow. I think you will find less need for debugging or console.log statements.

Keep on learning 📚

Learning a new programming language or tool can be really difficult, but also fun when the pieces finally start coming together. The knowledge gathered is something you can bring with you to other areas.

<sup>Have a look at my post about Emacs and my current configuration. The Node.js REPL feature is made possible using the js-comint package.

Photo by Marc-Olivier Jodoin on Unsplash</sup>

Simple (within parentheses)

• - Hey, what's ClojureScript?
• - It's a thing that compiles Clojure to JavaScript.
• - I get it. What's Clojure?
• - It's a Lisp.
• - ???
• - You know, a functional programming language that uses a lot of parentheses.
• - I … get it? 🤯

Don't worry. In this post, I will share some of the basics that I have learned about Clojure and ClojureScript.

Clojure?

With Clojure, you can write simple, functional and minimalistic code. Some of the cool things with Clojure are:

• data is immutable
• the REPL is a powerful tool
• few keystrokes to make things happen

Clojure runs on top of Java, suitable for all kinds of "backend" things. It also runs on top of JavaScript. Yes, JavaScript - making it possible to develop all kinds of apps, tools, products and websites. There's even an option to run in a .NET runtime, but I haven’t tried out that one yet.

What does it look like?

Even though the syntax may look unfamiliar, you can already see in this silly example the less amount of keystrokes needed using Clojure, compared to JavaScript.

Parenthesis?

Everything you write in Clojure is a list, wrapped in parentheses. That may sound weird. I agree, it is weird. But your eyes will get used to it. In short, the format of writing something in Clojure is:

(operator operands) or even (function arguments)

Here is an example:

... and how it would be written in JavaScript:

If all this is new to you, you might have this expression in your face right now: 😬

That is perfectly normal. I have looked like that a lot during 2020. When digging deeper, you may need to go through The 12 Stages of Learning Clojure.

ClojureScript?

With ClojureScript, you develop web apps and write simple React components. You write code in Clojure, compile it to native JavaScript and run it in your favorite browser. The compilation happens all the time and automagically as you type.

Just like with Java or .NET in the backend, ClojureScript has native access to JavaScript. You can use and interact with your favorite npm packages and React components just as you would with JavaScript.

Reagent and React

I'm going to fast-forward a bit by adding a ClojureScript library called Reagent. With Reagent, you write React components but with Hiccup instead of JSX. Wait, what?

View components are built up using Clojure data structures. The example above creates html elements with Clojure vectors and keywords. This style is called Hiccup. It will save you a lot of keystrokes and you'll have a lot less of scrolling in your code editor.

What about state?

I'll fast-forward again by adding a very nice library called re-frame.

From the re-frame docs:
“... It has reactivity, unidirectional data flow, pristinely pure functions, interceptors, coeffects, conveyor belts, algebraic effects, statechart-friendliness and claims an immaculate hammock conception. All while being both simple and easy. ...” 💥🤯💥

With re-frame, data is stored in one place called the "app db". Your components will fire events and subscribe to changes in the app db. The cool thing is that you write code in a sequential way, making it very readable and easy to understand. However, the syntax of the actual events and subscriptions might take a while to understand, at least if you are new to Clojure in general.

Here, we have a Reagent component that adds a button component, only when the app is disconnected. It is subscribing to changes of disconnected in the app db. React will re-render the components when the app state has changed.

Cool, but can I use my favorite editor?

Yes, most likely. I know there are Clojure plugins for VS Code, Atom, Sublime Text and intelliJ. I use Emacs, because ... I just like it.

There is something important and special going on with developers and their text editors. I think you will be able to find your favorite dev setup. I also think that you will learn a lot of new programming concepts and styles when reading more about this cool language and its libraries.

• Clojure
• ClojureScript
• Reagent
• re-frame

^{Photo by Marcos Paulo Prado on Unsplash}

Documentation: The Good Parts

Writing documentation is probably the most boring part of software development. As developers, many of us avoid it. We write docs only when someone else force us, twist our arms or assign us JIRA tickets.

- Was it the project manager, or maybe that senior tech lead guy with the massive beard that made you write the docs?

What I want to describe in this post isn't complex Word documents explaining some software. It isn't a guide on how to write something clever about a public FirstName property of a Person class.

It's about JavaScript and data types. Keeping it simple, and fun.

 

JavaScript is dynamic
JavaScript won't stop you from passing in whatever you want to a function, even when the code is expecting data of a specific type. It will crash during runtime instead, yes. When parsing or iterating something that cannot be parsed or iterated.

- No type checks for you!

This is also what makes the language very flexible. Simplistic syntax, functions as first class citizens and with a little less noise. There is no need for explicit data type declarations, generics or interfaces, making it easy to write code in less than 100 characters width. In fact, you will find yourself writing code in less than 80 chars quite often. In addition, having only one thread to worry about is a good thing (for most of us). And I believe the V8 engine is fast enough too.

- It would be nice if there was something that could let us have a little bit of both.

Did you say TypeScript?
TypeScript is cool. But I'm not sure it is the way to go for back-end code written for Node.js. Node.js itself is probably a bad choice if you are in great need of strongly typed language features on the server. Why not go for C# instead? 

If your code lives in the browser, well, that is something different. You are basically stuck with JavaScript for web development - whether you like it or not. TypeScript, or WASM, can probably help you - after some initial setup, headache & build scripts.

 
Have you tried JSDoc?
During my work on the Open Source project node-zookeeper, I have learned how JSDoc can help us make the JavaScript Developer Experience a lot better - without the headache. Your editor will give you hints and warnings about parameters and data types.

Here's an example:

/** @returns {number} */
function getRandom () {
    ...
}

With parameters:

/**
 * @param {number} minValue
 * @param {number} maxValue
 * @returns {number}
 */
function getRandom (minValue, maxValue) {
    ...
}
 
In the example, documentation is used only for describing the data types of input parameters and the return type. Nothing else. I would recommend you to try hard and refactor the actual code, making additional comments not needed. Is the function name getRandom good enough?

- Maybe I should add some comments to the code here?

(don't do it)

Your editor is smart
Your IDE or editor will figure out the JSDoc syntax, even for custom type definitions. Custom type definitions are for objects - custom data types - that cannot be described as built in strings, booleans, numbers, objects or arrays.

Another example, this one is from the typedefs.js of the node-zookeeper project, describing a ZooKeeper node changed callback:

/**
 * Watch callback
 * @typedef {callback} watchCb
 * @param {number} type
 * @param {number} state
 * @param {string} path
 */ 

The type definition for the watcher can be referenced in the code like this:
/** @param {watchCb} callback */

The node-zookeeper project has a typedefs.js file in the root folder. A smart IDE, like WebStorm, will parse it and use the type definitions to give you and your coder friends a nice developer experience.

When dynamic isn't good enough, use JSDoc. But be careful, use it only when needed. Or else, you will be stuck in the Documentation Swamp.

Experimental Node Mode

The latest Long-Term Support (LTS) version of Node.js has support for writing and consuming modules in ECMAScript style - without any tools or frameworks. That is good news! I have mostly been writing back-end code using the CommonJS standard, but have always preferred the ECMAScript way, commonly used in front-end development with tools like Babel.

Now, with Node.js 12, there is native support for ECMAScript modules. But it has to be manually enabled, by using the '--experimental-modules' flag.

The current version - Node.js 13 - doesn't need a flag to enable ECMAScript modules. Great! It is on by default. But the runtime will log a message on startup, a warning that the feature is not ready for production yet. Maybe we can expect it to be a default setting, without experimental warnings, in the next LTS later this year?

I have tried out ECMAScript modules in Node.js, using the experimental flag and the type: module in package.json.

Import tooling support, not perfect, but good enough.

Cool, but not perfect yet
I ran into issues when writing & running unit tests. My favorite unit testing library is tape, but it doesn't support ECMAScript modules natively:
Must use import to load ES Module

What about the AVA test runner ? No native support yet. Mocha? I couldn't find any info about it, but tests won't run.

Tape is just JavaScript 👍
It is convenient, but tape unit tests doesn't require the built in runner. Both AVA and Mocha need a runner. Using tape, you can run the unit tests just like any other JavaScript file with the Node.js CLI.

Don't try this at ... work?
Besides that using a default unit testing runner usually is a good thing, there's also a huge drawback with the experimental & non-standard approach that I describe in this post. The node command will run a single file only. 👀 

But, hey, we're in Experimental Mode right! A workaround would be to execute a custom runner containing all our unit tests. Just don't forget to add every new file to it.

"scripts": {
    "test": "node --experimental-modules test/run.js | tap-spec"
}

(from the script section of package.json)

The test/run.js file:
import 'path-to-unit-test'
import 'path-to-another-unit-test'


Try this at home, folks. Don't try it at work (yet). 🏡

The Developer who blocked the Event Loop

Recently, our team started working on using Node.js v12 in our v10 app. Luckily, we have unit tests that cover most of the core features. In Node.js 12, one unit test failed. 😱

The test assert that the event loop isn't blocked, even when a large amount of data is processed. Data is processed using async patterns with promises. The unit test add a separate task, triggered with setTimeout, and should be executed during the processing of the large amount of data. But now it is processed after. What? 

Something has changed.

Everything is fine.

Oh no, the code is blocking the event loop.

Here's example code I've written for this post that will behave differently in Node.js 10 and 12.
setTimeout(() => log('first timeout: done!'));

setTimeout(() => {
  processData()
    .then(processData)
    .then(processData)
    .then(processData);
});

// this one should run before all the "processData" calls are done.
setTimeout(() => log('second timeout: done!'));

[full example here]

Breaking changes? 💔
There's a lot of good things in the blog and change log about the Node.js 12 release, but I can't find any breaking API changes. What about Node.js 11? I couldn't find anything about changed event loop behavior there either. 

Promises can block the Event Loop 💤
I found a blog post (linked below) and a GitHub conversation about harmonizing how tasks are prioritized in the browser and in Node.js. Promises and features like setTimeout are since Node.js 11 grouped into micro and macro tasks, with different prioritization in the event loop.

A simple hack 
Here's one way to make the example code behave like it does in Node.js 10. This will make the example code non blocking.

processData()
  .then(() => {
     setTimeout(processData);
});

I hope this post will give you some ideas and help when upgrading the Node.js version. 

Merry Christmas & Happy Coding!
🎅


References
New Changes to the Timers and Microtasks in Node v11.0.0
MacroTask and MicroTask execution order

Updates to the Apache ZooKeeper Node.js client

At last, Node.js is platform independent again!

Eh, what?

Apache Zookeeper?
From the docs: "... ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C. ..."
(the docs)

A Node.js Zookeeper Client
You will find several Node.js implementations when browsing on npmjs.org, and one of them is simply called: zookeeper. Recently, I was added to the list of maintainers of the project, hosted on GitHub.

Background
The project got my attention when we at work were looking for a new ZooKeeper client, because the API of the current one we were using had gone out of date. We were all developing on Mac OS X, and pushed changes to a CI using a Linux docker container. Implementing the new client seemed like a breeze.

Until a developer in a peer team came by our spot at the office:
- Hey, I cloned the app repo and followed your setup guide. But I get errors when installing packages. I think it's the "zookeeper" dependency that fails. Do you know what's wrong?

We quickly learned it was because Windows. We realized that the entire build process - compiling C code to a native Node.js add-on on the fly - used non-Windows compliant features like make, bash and gcc.

Oh sh*t. How can this be solved?
What about bypassing it with Docker? I've learned that Docker on Windows can be painful, because of access right issues between a Windows host and Linux in a container. Windows Subsystem for Linux (WSL) would probably work after some tweaking, but adding WSL as a requirement to run (or develop) an app is ... well ... a bit depressing.

How about fixing the root cause of the problem, the source code? I did choose that path and have spent a lot of time learning about Node.js add-ons, (a little bit of) C and C++, Windows equivalents to bash commands, node-gyp, and how to compile the Apache Zookeeper C Client from source.

Months later, I hit the "Create Pull Request" button. Shortly after, the changes were merged and an updated npm package was published.

Besides supporting Windows, the zookeeper npm package also builds on Node.js 10. Previous versions failed, because the use of deprecated V8 features.

Details of recent changes can be found in the Changelog.

Contribute to the Open Source project
If you are interested in contributing, check out the issues tab at the GitHub repo. Currently there are some issues labeled with "good first issue", go ahead and grab one!

NDC Oslo Video: JavaScript in 2017 - You might (not) need a framework

The sessions from NDC Oslo 2017 are online!


I had a good time in Oslo, got new insights and left the Norwegian capital with inspiration. There were many interesting talks and lots of fellow geeks to chat with. I am also glad that so many people joined my talk. Thank you! I hope you have learned something useful.


Here's the recording of my NDC Oslo 2017 talk:

(and here's the code used in the talk)

Click the Youtube "Like" button if you like it! Also, I would very much appreciate your feedback. Please write your feedback at the comments section of the Youtube page (or this blog post) if you come up with ideas, questions, suggestions or things that should be improved.

Thank you NDC Oslo!

You might (not) need a JavaScript library

JavaScript today is like:
"Yeah great, and now you need Webpack, Babel, npm, yarn, React ...".

Redux? Relax.

Developing a front end web project of today can be a bit intimidating, especially when the home base is back end development. What about all those frameworks and tools, why use them?

I am currently learning about React, and I like it. React have JSX, ES2017 and a nice logo. That is cool, but my favorite thing with React is how code is organized. A user interface is built with components: small and isolated "packages" of html & JavaScript. That is a pattern I like.

Components with vanilla JavaScript?
When learning about React patterns, I started thinking about how this could be done without using React, ES2017, Webpack or any of the other libraries and frameworks out there. Is that even possible?

Okay, but why?
I want to learn and understand the problems that are solved using a library. Also, I want to find out what problems can be solved without an npm install. One way of doing that is to write all code with plain old vanilla JavaScript, html, css and find out what pain is removed by which library. Also, I think it would be a fun challenge!

Example code
You will find all code referenced in this blog post at my GitHub page

No build step required
So, I spent some late nights coding and learning. The code in the main branch of this repo does not require any build steps or npm package downloads. The "listItem component" is made of two parts: JavaScript in a code file and an html template in a separate file. The render function will create a DOM object containing html from the template, with data that is passed in to it, and return it in a callback.

code from listItem.js:

      function render(props, done) {

        // load the template using a helper
        templates.load('/src/listItem/listItem.html', function (el) {

          // 'el' is the html template element
          el.textContent = props.data;
          el.addEventListener('click', props.onClick);

          // pass the element to be added to the DOM
          if (done) {
            done(el);
          }
        });
      }

      return {
        render: render
      };
    

The listItem.html template:

      <li class="listItem" title="the listItem component"></li>
    

This is of course a very simplistic example with a single tag html template, but I think it already highlights issues: where's the data added? To understand where data is added, we have to read & understand the contents of the render function. I think it would be nice if the data to be rendered is visible in the actual template.

Time to grow a Mustache?
A template render engine can solve that. Here's the same component, using a template engine called Mustache.js. You will find the code in a separate branch of the GitHub repo.

      function render(props, done) {
        // pass data to a modified template loader
        templates.load('/src/listItem/listItem.html', props, function (el) {
          el.addEventListener('click', props.onClick);

          if (done) {
            done(el);
          }
        });
      }
    

compare with the vanilla code

The templates helper now use Mustache to render the html from the template and the data. It is now possible to write html templates with placeholders for data like this:

      <li class="listItem" title="the listItem component">{{data}}</li>
    

code from the templates.js file:

      container.innerHTML = Mustache.render(template, data);
    

More issues?
If you look at the source code in the main branch you'll notice the JavaScript files is written with a coding style called IIFE (immediately invoked function expression). It is used to isolate code and makes it possible to write modules without using any framework. Also, every single file is added with a script tag in the html body of the main page (index.html). Some modules depend on others and have to be added in the correct order. That's not great.

      <script src="src/templates.js"></script>

      <script src="src/listItem/listItem.js"></script>
      <script src="src/list/list.js"></script>
      <script src="src/terminal/terminal.js"></script>
      <script src="src/nav/nav.js"></script>
      <script src="src/logView/logView.js"></script>

      <script src="src/app.js"></script>
    

Solution: JavaScript AMD modules
In a separate branch, I have converted all of the immediately invoked function expressions (IIFE) to AMD modules. I use Require.js that takes care of module loading and dependencies. Instead of a very long list of html script tags, there is only an entry point defined.

from the index.html file

      <script data-main="src/app" src="lib/vendor/requirejs.js"></script>
    

Here is the listItem component as an AMD module:

      define(['templates'], function (templates) {
        function render(props, done) {
          templates.load('/src/listItem/listItem.html', props, function (el) {
            el.addEventListener('click', props.onClick);

            if (done) {
              done(el);
            }
          });
        }
        return {
          render: render
        };
      });
    

compare it with the previous branch

But wait. Don't we have native modules in Javascript now?
Oh, I forgot. It is 2017 and ECMAScript 2015 was released almost two years ago. A nice module system was included in it. Finally there is a common standard in the language! I have rewritten the code to ES2017 style - with arrow functions, the const keyword and most importantly, the ES import/export feature. Now, the listItem component looks like this:

      import load from 'templates';

      export function render(props, done) {
        load('/src/listItem/listItem.html', props, (el) => {
          el.addEventListener('click', props.onClick);

          if (done) {
            done(el);
          }
        });
      }
    

compare the ES2017 code with old school JavaScript

I think the code has improved a bit. ES2017 is great, but there are trade offs to be aware of. Many browsers don't have enough support for this version of JavaScript yet. To make it work in all kinds of browsers and devices we need to introduce a build step: the code need to be compiled from ES2017 to vanilla JavaScript with Babel.

The package.json file in the project has quite a few scripts compared to the original framework-and-build-step-free version. In addition to dependencies like Mustache.js and Require.js, there is a compile step and a Babel polyfill dependency added:

      "scripts": {
        "deps:lib": "mkdir -p -v lib/vendor",
        "deps:requirejs": "cp node_modules/requirejs/require.js ...
        "deps:mustache": "cp node_modules/mustache/mustache.min.js ...

        "deps:polyfill": "cp node_modules/babel-polyfill/dist/polyfill...
        "deps": "npm run deps:lib && npm run deps:requirejs && npm run ...
        "transpile": "babel src --out-dir lib --source-maps",

        "lint": "eslint src",
        "build": "npm run lint && npm run transpile && npm run deps",
        "start": "npm run build && live-server"
      }
    

More frameworks, more problems?
When browsing the page there is now a couple of third party libraries loaded to the client, besides our own modules. This might cause a not so great experience for users with a slow connection.

Bundling & minification
While we're at it, why not add another build step that will bundle all JavaScript files to one single file? This will reduce the number of requests from the browser. With minification we also will get rid of a couple of Kilobytes. The entry point is now one bundled and minified JavaScript file.

      <script data-main="lib/bundle/main"
              src="lib/vendor/requirejs.js"></script>
    

The source code in this branch is compiled from ES2017 to browser friendly AMD modules. With Require.js, there is a tool for bundling & minification included (called R.js) and used in this branch. compare the branches

Heard about Webpack?
The scripts section of the package.json file is quite massive now and probably difficult to understand. By using Webpack, most of those build steps are no longer necessary. Webpack does a lot of things, it's like a swiss army knife (that's both good and bad, I guess).

package.json with Webpack:

      "scripts": {
        "lint": "eslint src",
        "build": "npm run lint && webpack",
        "start": "webpack-dev-server"
      }
    

compare the two branches, with bundling vs with Webpack

Where did it all go, how is that even possible? Okay, I forgot to mention Webpack.config. Sorry. Some of the build magic live in that file now.

So, did Webpack make any difference?
One nice thing with Webpack is that there is no longer any need for Require.js. Webpack will resolve ES2017 modules and convert them to plain vanilla JavaScript before the bundling & minification. Also, Webpack has a local dev server feature (with auto reloading on file change) that I like.

Add React to the mix
This is how the listItem component looks like when converted to React. The template files are gone, everything is written in the JavaScript modules using the JSX syntax. There is no longer need for a custom template loader or mustaches. Compared to the source code in the previous branch, this one has less code. I like less code.

      import React from 'react';

      function ListItem(props) {
        return <li className='listItem' title='the listItem component'
        onClick={props.onClick}>{props.data}</li>;
      }

      export default ListItem;

    

React: Before vs After

Conclusion
By experimenting with one library at a time, I have learned about the value added and also some of the trade offs that comes with using a tool or a framework. Sometimes, plain vanilla JavaScript is just enough, and sometimes a framework or library will make life easier. You might (not) need a JavaScript library.