Tag: devops

ZooKeeper, ClickHouse Docker MacOS

by firodj, posted in Programming

ZooKeeper

The issue with zookeeper is chmod the mounted volume and got permission denied when running as root. But if run as default user, it got permission denied when trying to read config. Here is a simple solution, create a Dockerfile:

# Use the base image
FROM bitnami/zookeeper:3.7.1

# Switch to root user to change permissions
USER root

# Change ownership or permissions of /opt/bitnami
RUN chown -R 1001:1001 /opt/bitnami && chmod -R 775 /opt/bitnami

# Switch back to the non-root user
USER 1001

And build the image with:

$ docker build -t custom-zookeeper:3.7.1 .

ClickHouse

ClickHouse also perform chmod on its startup and can cause permission denied. The helpful solution could be useful from here: https://clickhouse-experts.com/blog/2021/03/20/clickhouse-container-permissions-and-faster-startup/

Define environment variables:

CLICKHOUSE_DO_NOT_CHOWN=1
CLICKHOUSE_UID=101
CLICKHOUSE_GID=101

These is solution for image clickhouse/clickhouse-server:24.1.2-alpine.

Crash Course Chef Infra

by firodj, posted in Programming

My experience is using Ansible, then my current company using Chef, by default I should learn first to use. They both do same thing to provisioning the VM, install software what I our apps need on the VM.

Confusion at first because too many terminology:

  • Chef,
  • Chef Infra,
  • Chef Solo,
  • Cookbook,
  • Recipe,
  • Chef Zero,
  • Chef Server,
  • Chef Kitchen,
  • Knife,
  • Chef Client,
  • Chef DK,
  • Chef Workstation,
  • Berkshelf,
  • Kitchen,
  • Supermarket, etc.

My goals is to be able to compare current JRuby deployment with Ruby MRI deployment on my team. If the benchmark result good then swap the engine. My problem with JRuby is:

  • Not energy saving,
  • TDD become slow.

The existing cookbook is tightly coupled with JRuby and don’t know why prefer using JRuby instead MRI. Is that because:

  • Java thread pool,
  • Java monitoring tool?

Tbh, I still don’t get it, no data or metric, and how meaningful that feature for our us.

The question, How is possibilty? Where to start? Actually my team is focus on Product, not on Infra or Devops. And Devops was handled by special team. No one has experience here, just reach out Devops team, they may too buzy. May be I could digging on company documentation, NOT FOUND! but wait found one two docs and it is actually here on my team wiki :‑P

  • How creating new services and setup the VM, and I know there is a command like this:
    $ sudo EDITOR=vi knife node edit $(hostname) -c /etc/chef/client.rb then run
    sudo chef-client.
    These commands let me knows my goals is Chef Infra (client and server).
  • Chef tutorial with chef zero, playing with chef on your laptop. But what I need is Chef Infra not Chef Zero, but it’s okay to do this effort.

From that docs there is lot of things I learn:

  • There is knife command to edit/see node config with knife node edit .  First I think this command run on client but then I got it wrong.
  • I know a recipe is a things that VM do to install software, and it resides in a cookbook. You could see run_list consists of list with format [COOKBOOK::RECIPE].
  • Argument -c /etc/chef/client.rb makes me curious and gotcha! the server is here.
  • And chef-client will execute the run_list. It run on client, should be. But it doesn’t need to explicitly client.rb. Why?

The cookbook name gives me idea to search on my company source code repo about the recipe contents. So each cookbook will have its repo. You could inspect the repo metadata.rb and see the name. But you may not go that far since every cookbooks that client need alread cached on /var/chef/cache/cookbooks/ folder. Reading through the source still don’t get me an idea what is the flow.

Where to start understanding the cookbook? Should I buy a book? I prefer gooling and expect someone out there share it or google book coulde give a peek. Found book Learning Chef  (2015) by Mischa Taylor, cs. (Is he a family of  Taylor Otwell :‑/ ).

I need a development kit, ChefDK (Development Kit) was mentioned on that book. But now Chef suggest to use Chef Workstation with ChefDK built in. I checked the compatibility of chef version:

  • my company VM setup is Chef 12.19
  • the book using Chef 11, and
  • Chef Workstation is Chef 15

Wow major version changes, but show must go on. Stick with google when found error message during the journey.

Install Vagrant and VirtualBox. Its a plus to be familiar with the command vagrant and VBoxManage, even though you only need kitchen.

From the Learning Chef (2015) Book:

  • Chapter 1-3, I skipped because I don’t need learn Ruby again. Beside the installation steps is different now. I following on the chef site for installation.
  • Chapter 4-10, Just quick read and DO that exercise, it will tell you more concept about chef.
    • Everything you do to execute the run_list is Test Kitchen. Create/Destroy/Login a VM. Execute your recipe with Converge.
    • A node variable on the recipe is predefined with current client info and state.
    • Ohai is always run each converge to populate your current client/node info.
    • Cookbook structure, and Flow of cookbook, and Attribute default values and how to override.
    • You could call Chef Solo and Chef Zero actually the local/offline version of Chef Infra.
    • Supermarket to store community shared cookbook thay may be useful.
  • Chapter 11-12, What will you actually on each cookbook:
    • Doing development always with Chef Zero.And introduce a cli and predefined function to do search a node by specific attribute to not hard code other dependant services on cookbook.

The rest of books is not used by me. Environment is not stored on the cookbook, and the best practice is to use shared then K/V server such us Consul or Etcd. Alternatively chef server have data bag that could be share among clients.

Back to cookbook that is used when deploy my ruby app, let say it rails-cookbook (1). it depends on ruby-cookbook (2). Again depends on ruby-setup-cookbook (3). The recipe in (1) call same recipe in (2), what is this? This is called cookbook wrapper that is seems common on chef. The wrapper usually created to pre or post-configure such as setup node attributes before doing the same thing. The (2) use resources that is defined in (3). The (3) only contains custom resources and the action. From this point my source of knowledge is search from chef docs, read other cookbooks, and googling.

Ruby is a pain with different versions. Chef has it own version too and in its embedded/bin folder. Your system may also have too. Thanks to rbenv we could change version per project basis and using rbenv-chef-workstation plugin is recommended.

Since we develop our apps using rbenv, the cookbook also rbenv, so I choose the VM also using rbenv. The existing rbenv-cookbook (4) seems outdated and overkill, so I need create my own version. But reading other cookbooks give us insight and learn something such us:

  • libraries: store helper function that could be call directly from recipes/resources. I am not interested to do OOP here.
  • resources: define repetitive or specific action. Since multiple recipe could be only called once per-chef-client running.

Then I create new cookbook ultimate-ruby-cookbook (5) for several reasons:

  • To not bothering with existing (1)(2)(3) cookbooks above.
  • Specific OS for only Ubuntu 16.04 to remove too many if-es on (1)(2)(3). And easy to read cookbook.
  • My own version of (4).
  • Better Test Kitchen experience, lock the chef to be same with company VM with require_chef_omnibus: 12.19.36 on kitchen.yml
  • I’ve also create Mock server with Vagrant on different repo for the kitchen converge to run smoothly. The cookbook depends on package installation from company repo server and reading k/v from consul, etc.
  • Mock Server will serve nginx to proxying aptly publish and consul-ui. Publish the package and import the k/v to consul that will be needed during converge.
  • For example, My Test Kitchen VM will have IP 192.168.33.36, and My Mock Server IP 192.168.33.10,  and my Laptop will be known as IP 192.168.33.1.
  • Create a test-cookbook in test/cookbooks as a wrapper to setup these attributes.

What is need to improvement:

  • The first converge will take too much time since compiling ruby by ruby-build. And the role of thumb is to precompile and store it to aptly repo for fast deployment and for security reason.
  • Not understanding Compile-Time vs Run-Time leads improper updated attributes to be used on recipe. On my case I create a recipe with order: install-ruby (A) then execute-ruby (B). The install-ruby set attributes ruby_bin_path to the actual path of installed ruby bin path. The execute-ruby will read attributes ruby_bin_path and expect to get the updated value which is set by the execute-ruby. When converge the attributes gone wrong, empty or not updated. Why?
    1. Compile time:  (A) set attributes is not executed, (B) read the attribute to set a property .
    2. Run Time: (A) running then set attributes, (B) running with already set attributes (old).
  • The solution is to use function lazy evaluated, or move into resource the action todo in (B).

After multiple try and error finally the cookbook will be used on Company Chef Infra. The magic really happen with running ​​bundle exec berks upload  on CI runner. But still got error try to produce locally to get some insight then found:

  • Gem berkshelf without version will install 7.x that is not compatible with ruby version on CI runner or MAYBE the chef version. After search all versions of berkshelf found 6.3.4 is suitable. The thing  I don’t like here is bundler cannot find suitable version since the gem version is not restricted.
  • Different CI runner have different ruby version, some got 2.3. some got 2.4, some don’t have bundle, some times success, some times failed, with retry Luck! Maybe try to tagging with CI runner.
  • After success run: sudo knife cookbook list -c /etc/chef/client.rb to see if cookbook successfully uploaded.
  • The when run ​chef-client the ohai_plugin borrowed from (4) failed to run even the kitchen chef omnibus version already set with same version with real VM. Need to remove it since Not used.
  • Just stick to node.chef_envrionment to get the environment of VM, not from other attributes since it may have different value.

I think that is a lesson I’ve learned for almost two weeks to be able to use MRI ruby on VM.