Introduction

Recently I announced the release of CL-Protobufs, a full featured protocol buffer library for Common Lisp. It does not include a gRPC package. In this post, I will make an example hello-world server that will take a protocol buffer message as input and respond with a protocol buffer as output. I will describe some of the limitations of the given code and some ways to improve it. 

I will expect a basic understanding of the ASDF build system; if you don’t have one please see an example such as this.

The reader should find the code in my github repo. It contains three files:

1. hello-world-server.lisp
2. hello-world-server.proto
3. hello-world-server.asd

The .asd file is for integration with the ASDF build system. The proto file is where we define the request and response. 

Code Discussion:

The actual implementation is in the hello-world-server.lisp file. Let’s briefly go over the files.

hello-world-server.asd:

The useful information in this file is:

• defsystem: The Lisp system is called hello-world-server. 
• defsystem-depends-on: To load the system you will need to load cl-protobufs first. This is so we can compile the proto file into a file that Lisp will understand. Call this the Lisp proto schema file.
• depends-on: We will use hunchentoot as our web server.
• module: We have one module src.
  • protobuf-source-file: This is an asdf directive given to us by cl-protobufs It will look for a file hello-world.proto in our current directory and call lisp-protoc on this file.
    • Note: You can specify a different directory to look in with proto-pathname. 
    • Warning: You must have the lisp-protoc plugin compiled and in your $PATH to call this directive successfully. Please go to CL-Protobufs and follow the protoc installation directions.
  • file: A lisp file hello-world-server.lisp

If you have a background using asdf that may have been obvious.The defsystem-depends-on call to cl-protobufs and the protobuf-source-file call may be non-obvious to newer lispers.

hello-world.proto

This is where we place the request and response definitions. This will be compiled into a lisp file by protoc that lisp will be able to interpret with cl-protobufs. Note the package name is hello_world. To read more about protocol buffers please visit the Google developer documentation and to learn more about the API for cl-protobufs please visit the CL-Protobufs readme.

hello-world-server.lisp

This is where the application code lives. The defpackage tells us mostly what we already know, we will use Hunchentoot as our web server library and the cl-protobufs library which will allow us to serialize and deserialize our protocol buffer objects. We will also be using a package: cl-protobufs.hello-world. This package is defined as part of the code generated by the Lisp protoc plugin and contains everything we need to work with the native lisp protocol buffer objects:

• cl-protobufs.hello-world:request
• cl-protobufs.hello-world:response

A good tutorial on Hunchentoot can be found here. We define a handler for our server that will be at /hello and will be available on port 4242. We give a parameter request of type string. 

Inside of the request body is where the real protocol buffer work is done. The handler will take in a request which will be of type string. This string will actually hold a simple array, i.e. #(values) which will be the serialized protocol buffer request message ‘cl-protobufs.hello-world:request. As an example: 

(cl-protobufs:serialize-object-to-bytes
  (cl-protobufs.hello-world:make-request :name "fish"))

Will give us a serialized request object whose string name entry is “fish”. When run in the REPL we get:

#(10 4 102 105 115 104) 

We call read-from-string to get the array as an array and then call make-array to get an octet array. In cl-protobufs:deserialize-object-from-bytes we expect an octet-array so this is required. We deserialize the message with deserialize-object-from-bytes. If we received a message with a set name we return a response with a response string “Hello {name}”, otherwise we just return a response with response string “Hello”. Finally we serialize the created response and return it as a string.

Calling with my “fish” request I get the response:

#(10 10 72 101 108 108 111 32 102 105 115 104)

Then calling 

(cl-protobufs:deserialize-object
  'cl-protobufs.hello-world:response
  (make-array 12 :element-type '(unsigned-byte 8) 
                 :initial-contents 
                 #(10 10 72 101 108 108 111 
                   32 102 105 115 104)))

I get 

#S(CL-PROTOBUFS.HELLO-WORLD:RESPONSE :%RESPONSE "Hello fish" 
                                     :%BYTES NIL 
                                     :%%IS-SET #*1) 

As should be expected.  

Limitations and Extensions

The first limitation is the work we have to do to read the request. We shouldn’t have to call read-from-string at all, and then we shouldn’t have to make a new octet array. What we should do is make our call with octets-to-string and then call string-to-octets on the received message. This can easily be done by importing trivial-utf-8. I was using Postman to make the calls in debugging this simple hello-world-server. This should be easy to fix for the next post.

We shouldn’t even have to do that. We should be able to set the message request in the handler and the octets-to-string then deserialize call should be handled for us. The final serialize and utf-8-string-to-bytes call should also be handled like that. This is easily handled by around methods. Given the ingenuity of Lispers, there’s probably even better answers.

Final Remarks

I hope you found this example interesting. This is only a simple server. Also, I haven’t used Huntchentoot very much so I probably wrote some terrible server code. I 

1. Hope to improve my lisp server code.
2. Hope to get a gRPC server out someday.

Thank you for reading. I hope to see some interesting lisp code using cl-protobufs!

I would like to thank @rongut, @cgay, and @benkuehnert for reviewing this document.

I’m one of the main maintainers for the Google Common Lisp protocol buffer library:

https://github.com/qitab/cl-protobufs

I’ve been meaning to do a write up on it because it’s a heavily used library for my team, developed at Google. Internally we only use the SBCL Compiler (Steel Bank Common Lisp), while other compilers are also available for Common Lisp. In this post I will describe the benefits of getting Common Lisp libraries such as cl-protobufs compiling on multiple Common Lisp compilers.

Recently I’ve been working on getting cl-protobufs to compile with CCL and ABCL. Most of my coworkers have been wondering why I bother. It seems fairly unlikely people outside of Google will use cl-protobufs and we only use SBCL. I hope this is wrong; it would be great to see more adoption of protocol buffers in the Lisp community, and thanks to Ben Kuehnert we have the only fully proto2 and proto3 compatible protocol buffer library in Common Lisp. But I digress.

Why would I bother getting protobufs to compile in both CCL and ABCL?

The answer is fairly simple; we want cl-protobufs to be available to a wider array of Common Lisp users; and SBCL allows certain constructs that other compilers don’t, so having cl-protobufs run in CCL and ABCL we can find more bugs. 

A clear example, and the majority of the bugs I found in cl-protobufs, is the use of make-instance for creating structure-classes. SBCL will allow this — you can’t set the slot values with make-instance, but you can create the instance. In CCL, make-instance expects all of the init-forms of the structure variables to be static, i.e. not functions, so fails. For example, if you have the struct

(defstruct my-struct
    (foo (make-array ‘(3)
                     :element:initial-elements ‘(1 2 3))
                     :type (simple-vector))))

You can call (make-instance ‘foo) in SBCL but not in CCL or ABCL. Calling make-instance on a structure-class object is undefined behaviour, so this is a bug!

CCL, like SBCL, compiles down to machine code, but ABCL compiles down to JVM bytecode. Why would I care that cl-protobuf can run on ABCL? We find lots of interesting bugs by using ABCL.

Here’s a simple example:

(defun positive-p (my-int)
    (declare (optimize (speed 3) (safety 0))
             (type fixnum my-int))
    (> my-int 0))

What should (positive-p nil) return? In SBCL we expect my-int to be a fixnum, and we tell the compiler to believe us. Speed 3 safety 0 means just treat it as a fixnum. The output on my laptop is T. In ABCL nil is not a fixnum, and there’s no way to cast it to a fixnum, so this is a type error.

Note: 

(defun safe-positive-p (my-int)
    (> my-int 0))

(safe-positive-p nil) throws an error in SBCL.

Why is this a problem? Why are you telling the compiler to compile at speed 3 safety 0? When code has to be fast, this makes it fast. I’ll discuss using the debugger in a later blog post.

So why try to get cl-protobufs working on different Lisp variants? It finds lots of bugs and undefined behavior! 

Special thanks to Ron Gut and Carl Gay for looking over this post.

First, I hate the new WordPress wysiwyg editor, so apologies.

My wife is a college lecturer, she usually teaches 3-4 classes per semester. I am a software engineer, I work 40 hours a week, usually more. During this time of Covid it’s difficult to find childcare. Do you even want someone else to take care of your child? With my wife teaching this opens up a problem.

Previously I posted thanking Google for giving me time off to take care of my daughter. This is a great benefit, and I’m very thankful to Google for giving me this benefit. That being said, it has one really big downside. A large part of performance is based on impact, and it’s hard to have next level impact when you’re only working 60% of what everyone else is working.

I’ve been working at Google for almost 4 years. I got to L4 in about a year and a half to two years. It was a fairly easy promotion process, just sure you’re a relative mature programmer who can do some smaller level system design. It’s probably the level most engineers are at, and it’s the level we hire PhD grads at. I was happy to get the promotion, it came with nice benefits and all is great.

Two years later I have a daughter and another one on the way. I want to get a nice house for my family, a yard to play in and a garage to work on my motorcycle (and learn some woodworking). But in the area I live, it’s hard to afford a decent house with the salary an L4 at Google makes. One of my friends went to Mountain View when they got hired and were told it takes an L5 to buy a decent house within an hour commute, now it’s L6…

So I’m left with seemingly two possibilities. I can take the extra time off to spend with my daughter, or I can try to find the extra time and work for that promotion. It’s not a hard choice for me, my daughter will win out 100% of the time every time. That being said, it still makes me wonder. Would the extra work, the promotion, be worth it?

Thankfully the problem is simple, if you can spend more time with your family, you should do it.

Here’s a pointer: pointers suck!

First, no they don’t, but whats a pointer? For those who are still reading this and are not programmers, pointers are variables that contain a memory address. Say you have

void foo() {
int x = 5;
int* y = &x; // *y == x
cout << y; // print address of x
cout << *y; // print 5 (ie x)
cout << x; // print 5 (ie x)
}

Now x is a variable that stores the value 5, and &x says “heres where x is stored”. So saying y = &x tells us that y contains the address of x. In fact, you could read int* as the memory address of an int.

Then *y says, go to the address y is pointing to. What if we did *x? We’d get an error. This being said, &y is still valid, it’s the memory address of the memory address of an int, or int**.  Finally, “cout << x” just says display the value of x to the screen.

If your not a programmer and you don’t understand that’s okay. If you’re a programmer complaining about me eliding certain facts, my apologies.

Now, we if I had a function

void foo(int* y) {
cout << *y;
}
void bar() {
int* z;
foo(z);
}

We would expect all to be okay, we print the int that y is pointer to. But what would that value be? Well,  int* z is saying z will be a pointer to an int, we don’t know where that pointer is, but we will set it soon.

A memory address is an integer from 0 to 2^32 -1 or 2^64 – 1 depending on your processor, and a memory address is stored as an integer. Lets say your processor is 64 bit so 0 to (2^64)-1. So y will be a value from 0 to 2^64 – 1, but its unlikely that this place in memory will contain an integer. Most likely, int* z will set the value z to 0, so “cout << z” should just return zero, but we can’t be sure. Then what will cout << *y; return? It will probably crash whatever program your running.

So why do we use them? They make our code faster. If z was a pointer to 5000 integers:

void foo1(set y) { cout << *y; }
void foo2(set* y) { cout << *y; }
void bar() {
set z = {...}; // z sets 5000 ints
foo1(z);
foo2(&z);
}

Then foo1 has to copy 5000 integers where foo2 just uses those integers!

That being said, this difference has caused a lot of the bugs you’ve heard of, and wasted hours of my life!

Hope you learned something!

Today, I wish to discuss the internal culture of the U.S. and China. This comes from recent talks between my wife, some of my coworkers, and I.

A common discussion my wife an I have is on the separatist bent of Taiwan and Tibet. My wife is continually irked by people saying their from the country Taiwan, as her upbringing tells her that Taiwan is a province of China. Growing up in the United States I grew up to believe that Taiwan should be a country.

The common argument she asks is what if Vermont (or Hawaii) decided to leave the United States. I shrug this off and say that if a majority of Vermonters decided to leave the U.S. then we should let them, though history does not vindicate this answer. There was a separatist movement in the United States in the mid 1800’s, and the federal government wouldn’t let that happen. One would imagine that the U.S. government would have much the same action.

The U.S. is a republic, we vote for who’s in office as a collective. If we allowed California to leave because Trump was elected (or Texas because of Obama) then why bother with this collection of states at all?

To further drive at this point, I was talking to friends at a math conference about evolution being taught in school, and they asked the question: What if the other side gained power and required creationism be taught? It’s a tough question, maybe laws should be decided at the lowest possible level of governance, states should be given most of the power, and we should question when the federal government starts dictating terms that states should otherwise handle.

The argument that this impedes social change is not lost on me, and the world is imperfect. But again, if you agree with Obama having sweeping presidential power, you must then deal with Trump, you don’t get to give power to only your friends.

Note: This runs counter to China, where the federal government tries to retain all power. But different countries have different values, each of which has evolved over thousands of years.

In the U.S. we believe that democracy is important. To a large degree, we believe that personal autonomy and civil rights are key to what and who we are. Cambridge just passed a chopping down tree ban, my personal view is it’s my damn tree and I’ll chop it down if I see fit, who is the town to tell me not to? Now we must temper this with a need to live together, you can’t go around chopping down all of trees as we need to breathe air (and trees help with that). I should be able to send messages to my friends without the state reading them (thanks Signal). As Benjamin Franklin said “Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.” 

Chi Wang in the South China Morning Post had an interesting article stating China had no use for a democracy. This runs counter to my world view, shouldn’t everyone want a democracy. Talking to my wife, she agrees (at least in part) with this article. The internal cultural of a democracy is just not there, it wouldn’t fit the population. This will probably leave a bad taste in the mouth of most American readers (as it does for me) but it’s a worthwhile thing to think about. When discussing national safety she would completely disagree with Benjamin Franklin, it’s the job of the government to keep the population safe, the government knows best and if this requires giving up personal freedoms so be it.

I’m interested to see where on this divide my daughter lands.