Searching the web I found setting up an API key for Nginx Ingress Controller is not well documented. Currently documented authentication methods supported by Kubernetes Nginx Ingress controller include Basic Authentication, Client Certificate, External Basic and External OAUTH. Another common authentication mechanism is the API Key. API keys are used frequently for machine-to-machine communications. The API key could be a unique string in a HTTP header or a query string parameter. Below I will show you how to setup a API key.
Below is my Nginx Ingress controller configuration. The base path I’m trying to protect with the API key is /v1/api. If $apikey_is_k does not equal 1 return 401 response. We’re not done yet we also need to add http-snippet to the Nginx ConfigMap (Fig 2.)
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($apikey_is_ok != 1) {
return 401;
}
In addition, to adding the configuration-snippet to the path we are trying to protect we need to add a http-snippet to the Nginx configmap for the nginx ingress controller. Nginx map function documentation give further explanation. If you have not already created a configmap for your nginx ingress controller you need to make sure the name of the configmap matches the nginx controllers name. For example, in my case the controller is named nginx-ingress-controller.
kubectl get configmap -n nginx-ingress
NAME DATA AGE
nginx-ingress-controller 8 184d
In Fig 2 pay attention to $http_apikey apikey is the name of the header which the client request contains the API key. $apikey_is_ok is the name of the variable which will be evaluated against the map values. Map values BmDzvO71VkG50NirFjXJ or of8tLLi2GXicqCvWuoEE are the API keys which will be validated against the client HTTP request. Fig 2 http-snippet
In summary, setting up API key validation in Nginx Ingress for kubernetes is not very difficult. This option does not appear to be discussed very frequently. I hope this helps anyone else looking for this type of solution.
What is a spot instance? Spot instances are how cloud providers sell their unused capacity, often at a big discounts. Currently I see most VM pricing discounted anywhere from 70% to 90%.
Every major cloud provider currently supports spot instances. AWS first released spot instance in Dec 2019. Microsoft nearly 10 years later released spot instances in May 2020. Google cloud also has spot instance offering which they named Preemptible VM Instances. For this post I will mainly focus on Azure’s spot instances and how one can mitigate these downsides.
Fig 1. Unused Azure datacenter capacity
When you create a Spot instance in Azure, Microsoft provides data on eviction rates and the current reduced rate you can purchase the VM at. For example, here you can purchase the D8s_v3 at a cost of $0.11201 cents per hour versus the standard pay-as-you-go pricing of $0.384 per hour, a 70% reduction in cost.
Fig 2. See all Sizes (Eviction Rate / Discount / Cost)
Create A Azure Spot Instance
When creating a VM one must select Azure Spot Instance. A VM can only be made a Spot Instance when it is created. If your VM is not a spot instance and you would like to make the VM a spot instance there are various scripts available on the internet to do this. You will also want to make sure you select Eviction type capacity. This will reduce the number of evictions that might occur on your Sport Instance. However, the pricing is variable. For example, if current going price for a D8s_v3 is $0.11201 cents per hour, and then there is a spike in demand the price may float up to $0.12480 cents per hour. As the caption says your VM will not be evicted for pricing reasons. Only if Azure needs the capacity for pay-as-you-go workloads. Eviction policy is also another important selection. If you are running static VM images in Azure for your development environment you want to ensure the Eviction Policy is set to Stop/Deallocate.
Fig 3. Create Spot VM
Another interesting selection is the View pricing history and compare prices in nearby regions. You will be able to view pricing in the last few months as well as compare against other regions.
Fig 4. Eviction Rate
How do we solve for VM evictions?
Do we really want to ask our developers to login into Azure Portal and restart VMs after each eviction???
My first try at solving the eviction problem.
Initially when I set out to solve this problem I started looking into Azure Functions and Azure Event Grid. Unfortunately after building a working process with Azure function and Azure Event Grid which worked flawlessly using the simulated eviction API, I ran into a big issue. What happens when an actual eviction event is generated by Azure and a simulated eviction are different. 😦 Thanks Microsoft!!!
A simulated eviction seems to fall under ResourceActionSuccess event type (see example payload below). From what I could gather using requetbin and the event grid webhook to capture the Event Grid events. There doesn’t seem to be a eviction event which gets sent on event grid when an actual eviction event is generated. I suspect an actual eviction doesn’t fall into one of the categories listed above.
As you can see in the simulated eviction the action field clearly states this is an eviction action. This made it very easy to parse and determine if the event coming off the event grid was an eviction.
Still determined, I considered triggering the Azure Function off a deallocation event, however, I was not keen on using deallocation as a trigger event, as it might cause too many false VM starts when someone was actually trying to deallocate their VM. Here is an example of a deallocate event when an actual eviction occurs.
Fig. 5 Azure Function and Event Grid start VM process
My second try at solving the eviction problem. Success!!!
I was determined to make this happen as the cost savings potential for a big environment was too great. I had remembered some earlier work I had done with the Azure Instance Metadata service, I investigated if the metadata service on each instance would expos eviction events. It turns out the VMs do. Each VM on Azure exposes a metadata via Azure Instance Metadata Service. Through this service one can retrieve information about the service like the subscription the VM is running in, resource group, IP address, VM size, tags and etc…
The basic idea behind this process, is we monitor the Azure Instance Metadata Service for an eviction event. Once the eviction event is received the process will call Jenkins via its reset endpoint and start the VM after it has been deallocated/evicted from Azure.
Fig 6. VM restart process initiated from client side
The downside of this architecture is you have to deploy and manage a small client side service running on each VM. However, with tools like Ansible, Chef, Puppet or Cloud-Init this usually isn’t too much trouble. The upside writing this service does not require any knowledge of Azure functions. Theoretically this architecture could be implemented on AWS as well, they provide instance metadata similar to Azure.
Simulating the eviction Process
Microsoft recently released REST api to simulate eviction process on Azure VMs. The ability to simulate an eviction made it much easier for me to develop and a process the eviction monitor service.
# you can log into Azure portal and if using Chrome go to dev tools
# and get your Bearer token from one of the requests
TOKEN=<my auth token>
curl -H "Accept: */*" \
-H "Content-Length: 0" \
-H "Host: management.azure.com" \
-H "Authorization: Bearer ${TOKEN}" \
-X POST \
https://management.azure.com/subscriptions/a6074dfd-ce1d-4687-af3a-6e7c3d2d6852/resourceGroups//providers/Microsoft.Compute/virtualMachines/myvm123/simulateEviction?api-version=2020-12-01
Fig 7. Azure Portal Active Log Simulated Eviction
I went to work and wrote small service in Python which queries the metadata service on the VM. I the installed the service via systemd service file. The python code, Jenkinsfile and Python code are published on github. Below is the output of systemd logs from the eviction_notify app right after the simulated eviction curl POST. Microsoft will give you a minimum of 30 seconds notification before the VM will be shut down. However, I have seen notifications of up to 3 to 4 minutes.
What are some other approaches you can use to reduce the chance your VMs might get evicted?
Another way you can mitigate the chance your VM might get evicted is to run different sizes across different Availability Zones and regions. AWS actually has a really good article on this in reference to its kubernetes offering EKS. Although the post is about Kubernetes one could apply the same principles to any group of VMs. Demand for compute can very across availability zones, VM sizes and regions. Spreading the VMs across all three will greatly reduce the chance that all VMs will get evicted at a single point in time.
VM Sizes and Availability Zones
What has my experience with Spot Instances been?
Although I have only several months of experience with Spot Instances, I have a pretty good understanding of the ins-and-outs of Spot Instances. Recently I was testing deployment of a 12 node Cassandra cluster using Ansible, I deployed this Cassandra cluster with Spot Instances and set Eviction Type to Capacity. After about a week of testing I noticed on average 3 to 4 evictions total across the whole cluster per day. For the small inconvenience, I was saving 75% per day on the cost of compute. And with the process I built to start the VM upon eviction, most times I did not notice a node went down for a few minutes. I am looking at additional use cases for Spot Instances, and will publish the findings here.
Where can I get the source code?
Source code for eviction_notify has been published on github
Summary
In summary spot instance can save a substantial amount on public cloud costs. For example, if your monthly cloud bill is $10,000 for dev/test environments. By shutting down environments for 12 hours a day you can reduce the cost to $5,000. By implementing spot instances you can potentially lower this to $1,500 per month assuming 70% discount. Between shutting down VMs and Spot instances there is a potential for big savings. In addition, spot instances also allow you to use much more powerful instance types for less cost in dev environments. I also showed how to mitigate the down side of using Spot instances, by engineering a solution to reduce the impact of the evictions.
This guide will walk you through the steps to setup Razor a next generation provisioning solution for bare metal and virtual servers. Razor puts an API on top of bare metal and virtual server provisioning. Razor makes it extremely easy to provision one, two or two hundred servers very quickly. The software was co-developed between EMC and Puppet Labs. The entire Razor project was open sourced and freely available to anyone who wants to use the software.
This tutorial was written using CentOS 7
STEP 1. Install Postgres
To setup Razor you will first need the Postgres SQL database installed. Postgres is used to stored facts about the nodes being provisioned through razor.
Install the Postgres database server and initialize the databases
STEP 4. Download and extract the Razor microkernel
$ cd /var/lib/razor/repo-store/
$ wget http://links.puppetlabs.com/razor-microkernel-latest.tar
$ tar -xvf ./razor-microkernel-latest.tar
STEP 5. Install the tftp server
yum install tftp tftp-server xinetd
cd /var/lib/tftpboot/
service xinetd start
service xinetd status
chmod 664 /var/lib/tftpboot/*
vim /etc/xinetd.d/tftp
# configure disable to no
disable = no
Install the ipxe boot
cd /var/lib/tftpboot
wget http://boot.ipxe.org/undionly.kpxe
chmod 644 ./undionly.kpxe
Save the Razor iPXE bootstrap script as bootstrap.ipxe
cd /var/lib/tftpboot
curl http://172.16.1.75:8150/api/microkernel/bootstrap?nic_max=4 > ./bootstrap.ipxe
chmod 644 ./bootstrap.ipxe
STEP 6. Install the CentOS ISO
This is the ISO Which we will install when we PXE boot servers with Razor
Download the CentOS iso
yum install nginx
systemctl start nginx
systemctl enable nginx
cd /usr/share/nginx/html/
wget http://mirrors.centos.webair.com/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1511.iso
chmod 644 ./CentOS-7-x86_64-DVD-1511.iso
# Verify the ISO can be downloaded
wget http://localhost/CentOS-7-x86_64-DVD-1511.iso
STEP 7. Configure the DHCP server on your network so clients can PXE boot.
Getting the PXE boot to work can be a bit tricky. You must define an iPXE class in the DHCP server.
Create a Policy / Class
Configure the DHCP Scope Options for the iPXE class
When you are complete conifguring the iPXE DHCP class your scope options should look like this.
STEP 8. Use the Razor Client to create a CentOS 7 Repo
STEP 9. Use the Razor Client to create a tag. In this example tag I create a very specific tag which specifies if a node boots on the network with this MAC address then install CentOS 7.
Over the last month I have been evaluating container clustering software. I started with Kubernetes, Rancher which uses swarm and Mesos. I am going through these evaluations to determine which container clustering software will fit my employer’s needs best.
ENVIRONMENT CENTOS 7.0 running three Mesos masters and two Mesos slaves
NAME: mesos01.lab.net
IP 172.16.1.80
services: zookeeper, marathon, mesos-master
NAME: mesos02.lab.net
IP 172.16.1.81
services: zookeeper, marathon, mesos-master
NAME: mesos03.lab.net
IP 172.16.1.82
services: zookeeper, marathon, mesos-master
NAME: mesos04.lab.net
IP 172.16.1.83
services: mesos-slave
NAME: mesos05.lab.net
IP 172.16.1.84
services: mesos-slave
$ go get github.com/tools/godep
$ go get github.com/mesosphere/mesos-dns/logging
$ go get github.com/mesosphere/mesos-dns/records
$ go get github.com/mesosphere/mesos-dns/resolver
STEP 2. Clone the mesos-dns repository and build the mesos-dns binary.
$ git clone https://github.com/mesosphere/mesos-dns.git
$ cd ./mesos-dns
$ go build -o mesos-dns
After building mesos-dns you should have a mesos-dns binary file in your
./mesos-dns directory
STEP 3. In the ./mesos-dns directory there is a config.json.sample example file.
Copy this file and edit it for your own environment.
$ cp config.json.sample config.json
This link describes the each of the fields in the config.json file.
STEP 4. Run the mesos-dns with the config.json file to verify it is properly formatted.
$ ./mesos-dns -config=config.json
On the mesos slave create a directory for the config.json file.
I have designated mesos04.lab.net as the mesos-dns server for my
cluster.
$ mkdir /etc/mesos-dns
STEP 5. Copy the mesos-dns binary to the mesos slave which you have designated as the mesos-dns server. In this example I copy the mesos-dns service to mesos slave mesos04.
STEP 6. Configure the constraints for the mesos-dns service. This essentially tells the marathon to constrain the mesos-dns service to host mesos04.lab.net. For example, you may want to designate two nodes in your cluster to run mesos-dns. The constrains directive ensures that mesos-dns does not try to run on other hosts.
Constraints: hostname:CLUSTER:mesos04.lab.net
STEP 7. Update the network-script file with IP address of the host running mesos-dns.
STEP 8. After updating the network-script file restart the network service
systemctl restart network
STEP 9. If you have any applications running in marathon you should be able to look them up using mesos-dns. For example, I had a application named nodehello2. I was able to resolve the application using mesos-dns.
[root@mesos04 mesos-dns]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2f6d8a4f99fd 172.16.1.60:5000/node_hello:2.0 "/bin/sh -c '/node/bi" 35 hours ago Up 35 hours 0.0.0.0:31495->8081/tcp mesos-a78b235a-8427-4743-9bcc-5d6aed338412-S3.3698d9f9-a25a-457a-8602-50d9c26e70a7
38ca56e041f3 172.16.1.60:5000/node_hello:1.0 "/bin/sh -c '/node/bi" 35 hours ago Up 35 hours 0.0.0.0:31884->8081/tcp mesos-a78b235a-8427-4743-9bcc-5d6aed338412-S3.e16a1e3e-a662-40da-b353-318de55178dc
[root@mesos04 mesos-dns]$ curl http://nodehello2.marathon.mesos:31495
Version 2.0
Hello World
[root@mesos04 mesos-dns]$ curl http://nodehello2.marathon.mesos:31884
Version 1.0
Hello World
STEP 11. You can also return the ports of the application. For example, nodehello2 is running on port 31472 on s2.marathon.slave.mesos and port 31495 on s3.marathon.slave.mesos.
I have been spending a lot of time getting familiar with Docker. Mostly working with Docker and Kubernetes. If you deploy a Kubernetes cluster you will most likely want to setup your own Docker Registry to pull down custom Docker images from.
I performed the following configuration on CentOS 7 and Docker version 1.8.
STEP 1. Pull down the Docker Registry image file.
docker run -d -p 5000:5000 --restart=always --name registry registry:2
STEP 2. Verify the image is running
docker ps
STEP 3. Verify the image you are trying to push exists on the local machine
docker images
At this point should have the ability to push docker images to the Registry.
However, while trying to configure the Docker Registry I keep receiving the following errors. It turns out by default docker uses HTTPS. Since this is a lab environment I opted to disable the certificate check. I would not recommend this in a production environment.
[root@docker01 node]> docker push docker01.lab.net:5000/node_hello
The push refers to a repository [docker01.lab.net:5000/node_hello] (len: 1)
unable to ping registry endpoint https://docker01.lab.net:5000/v0/
v2 ping attempt failed with error: Get https://docker01.lab.net:5000/v2/: tls: oversized record received with length 20527
v1 ping attempt failed with error: Get https://docker01.lab.net:5000/v1/_ping: tls: oversized record received with length 20527
To enable a an insecure registry add the line below to your /etc/sysconfig/docker file.
[root@docker01 nfs]> vim /etc/sysconfig/docker
INSECURE_REGISTRY='--insecure-registry docker01.lab.net:5000'
restart the docker engine
systemctl restart docker
Now tag your image in the Repository.
[root@docker01 nfs]> docker tag node_hello docker01.lab.net:5000/node_hello
I wrote those post mainly for my own notes. The cbrestore command is a bit picky on the order in which you pass in the arguments. This is just a quick note for myself as it a task I don’t perform everyday.
Note that you do not have to pass in the exact location of the bucket folder you are trying to restore. Just the root folder which holds the bucket-* folder.
[someuser@host ~]> ls -la /mnt/mount/tmp/
total 16
drwxr-xr-x. 0 root root 4096 Jan 15 10:55 .
drwxr-xr-x. 23 root root 0 Jan 15 10:51 ..
drwxr-xr-x. 0 root root 4096 Jan 15 10:55 bucket-Store
drwxr-xr-x. 0 root root 4096 Jan 15 10:55 bucket-Web
[someuser@host ~]>
/opt/couchbase/bin/cbrestore /mnt/mount/tmp/ --bucket-destination=Web --bucket-source=Web http://cb01:8091 -u Administrator -p devPassword
OpenVPN was surprisingly easy to setup on in my lab environment. My setup included a CentOS 7 server running the latest version of OpenVPN server and a Windows 7 client running the latest OpenVPN client. I also have a Netgear router which I configured with a static route.
Before we begin you will need certificates
1. A computer to run OpenVPN (I used CentOS 7)
2. OpenVPN server will need a certificate
3. OpenVPN client will need a certificate
4. A home router which can be configured with static routes
5. A way to generate certificates for your vpn server and clients
Download the openVPN source code and compile it into an RPM
OpenVPN Server Configuration
Copy the sample server.conf file to /etc/openvpn/server.conf
Here is a list of settings I configured from the defaults server.conf file to get my OpenVPN server working.
# Which TCP/UDP port should OpenVPN listen on?
# If you want to run multiple OpenVPN instances
# on the same machine, use a different port
# number for each one. You will need to
# open up this port on your firewall.
port 1194
# TCP or UDP server?
proto udp
# "dev tun" will create a routed IP tunnel,
# "dev tap" will create an ethernet tunnel.
# Use "dev tap0" if you are ethernet bridging
# and have precreated a tap0 virtual interface
# and bridged it with your ethernet interface.
# If you want to control access policies
# over the VPN, you must create firewall
# rules for the the TUN/TAP interface.
# On non-Windows systems, you can give
# an explicit unit number, such as tun0.
# On Windows, use "dev-node" for this.
# On most systems, the VPN will not function
# unless you partially or fully disable
# the firewall for the TUN/TAP interface.
dev tun
# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key). Each client
# and the server must have their own cert and
# key file. The server and all clients will
# use the same ca file.
#
# See the "easy-rsa" directory for a series
# of scripts for generating RSA certificates
# and private keys. Remember to use
# a unique Common Name for the server
# and each of the client certificates.
#
# Any X509 key management system can be used.
# OpenVPN can also use a PKCS #12 formatted key file
# (see "pkcs12" directive in man page).
ca lab-ca.pem
cert vpnserver.pem
key vpnserver-key-nopass.pem # This file should be kept secret
# Diffie hellman parameters.
# Generate your own with:
# openssl dhparam -out dh1024.pem 1024
# Substitute 2048 for 1024 if you are using
# 2048 bit keys.
dh dh1024.pem
# Configure server mode and supply a VPN subnet
# for OpenVPN to draw client addresses from.
# The server will take 10.8.0.1 for itself,
# the rest will be made available to clients.
# Each client will be able to reach the server
# on 10.8.0.1. Comment this line out if you are
# ethernet bridging. See the man page for more info.
server 10.8.0.0 255.255.255.0
# Push routes to the client to allow it
# to reach other private subnets behind
# the server. Remember that these
# private subnets will also need
# to know to route the OpenVPN client
# address pool (10.8.0.0/255.255.255.0)
# back to the OpenVPN server.
push "route 172.16.1.0 255.255.255.0"
# Certain Windows-specific network settings
# can be pushed to clients, such as DNS
# or WINS server addresses. CAVEAT:
# http://openvpn.net/faq.html#dhcpcaveats
push "dhcp-option DNS 172.16.1.21"
# The keepalive directive causes ping-like
# messages to be sent back and forth over
# the link so that each side knows when
# the other side has gone down.
# Ping every 10 seconds, assume that remote
# peer is down if no ping received during
# a 120 second time period.
keepalive 10 120
# Select a cryptographic cipher.
# This config item must be copied to
# the client config file as well.
cipher AES-128-CBC # AES
OpenVPN client Configuration
Download the OpenVPN client from here.
OpenVPN client configuration is saved here on Windows:
C:\Program Files\OpenVPN\config\client.ovpn
Here is a list of OpenVPN client settings I configured to get my OpenVPN client connected.
# Specify that we are a client and that we
# will be pulling certain config file directives
# from the server.
client
# Use the same setting as you are using on
# the server.
# On most systems, the VPN will not function
# unless you partially or fully disable
# the firewall for the TUN/TAP interface.
;dev tap
dev tun
# Are we connecting to a TCP or
# UDP server? Use the same setting as
# on the server.
;proto tcp
proto udp
# The hostname/IP and port of the server.
# You can have multiple remote entries
# to load balance between the servers.
# put my public IP here
remote 71.XX.XX.XXX 1194
# SSL/TLS parms.
# See the server config file for more
# description. It's best to use
# a separate .crt/.key file pair
# for each client. A single ca
# file can be used for all clients.
ca lab-ca.pem
cert usercert.pem
key key-pass.pem
# Select a cryptographic cipher.
# If the cipher option is used on the server
# then you must also specify it here.
cipher AES-128-CBC
Generate Certificates:
I generated my certificates using a Microsoft 2012 Certificate Authority. I generated one for certificate for the VPN server and another for the VPN client. I exported them from Microsoft CA in PFX format and used this Guide to convert them to PEM format.
My openVPN server certificate properties:
CN=vpn.lab.net
Subject Alternative Name=vpn
Subject Alternative Name=test02.lab.net
Subject Alternative Name=test02
My openVPN user certificate properties:
CN=user OU=WAU OU=US DC=lab DC=net
On the OpenVPN server copy the PEM files to /etc/openvpn/
On the OpenVPN Windows client copy the PEM files to C:\Program Files\OpenVPN\config\
Router Configuration
Configure your home router with a static route to the OpeVPN server on your home network
VPN client subnet: 10.8.0.0/255.255.255.0
OpenVPN Server: 172.16.1.36
Start the OpenVPN service on the OpenVPN server
systemctl start openvpn
Test Client Connection
On Windows 7 I noticied it was required to run the OpenVPN as administrator
If you where successfully connected you should see “client is now connected”
In this blog post I will describe the steps needed to configure rundeck to use SSL. I go through the steps of requesting a certificate from a Microsoft CA then exporting them to a Linux rundeck server. I then go through the steps of importing the certificates into a java keystore. And finally the configuration steps needed to get rundeck working with SSL.
STEP 1. Request a certificate
Open the mmc.exe > add/remove snapin > certificates > local computer
STEP 2. Click Next
STEP 3. Configure the CN (common name) and Subject Alternative names.
STEP 4. Mark private key as exportable
STEP 5. Select Enroll
STEP 6. Export the certificate
STEP 7. Export private key
STEP 8. Export the certificate and private key in PKCS #12 format
STEP 9. Set private key password
STEP 10. Export the the Certificate Authorities certificate.
This certificate will be placed in the the trusted CA Java keystore. Do not export the private key for the CA, export the CA as DER format.
STEP 10. SFTP the certificate to your Linux Rundeck Server
I placed the rundeck.pfx file in /etc/rundeck/ssl
Also place the ca.cer file in /etc/rundeck/ssl
STEP 11. Create a keystore for the rundeck.pfx certificate
Create a Java keystore to hold the new rundeck certificate
STEP 13. Import the Certificate and Private Key into the Java keystore
Use the alias id from the previous command as the source alias and destination alias.
keytool -import -alias ca -file /etc/rundeck/ssl/lab-ca-der.cer -keystore /etc/rundeck/ssl/ca -storepass password
Trust this certificate? [no]: yes
Certificate was added to keystore
STEP 16. Review of previous steps
a. At this point we should have requested and received a certificate from the Microsoft CA
b. Export the CA’s certificate
c. Created a java keystore for our rundeck certificate
d. Created a java keystore for our CA certificate
STEP 17. Configure Rundeck /etc/rundeck/etc/ssl.properties
Configure the path to the certificate keystore and CA keystore you created earlier
This guide was written using the rundeck 2.4.2 RPM installed on CentOS 6.5. I go over the steps needed to setup Active Directory authentication in Rundeck
STEP 1. CREATE Active Directory Group
In Active Directory create a new group named “rundeckusers.” Then add your users to that AD group.
Enter the following configuration settings into your jaas-ldap.conf file. You will need to configure the username/password for the user which will bind to Active Directory. You will also need to configure the userBaseDn. This is the OU which recursive searches for users will be performed on. In addition, configuring the roleBaseDn. The roleBaseDn is the OU where your “rundeck” AD user group is.
You’ll need to configure / modify to two lines. Add the path to the jaas-activedirectory.conf file and the loginmodule name, “activedirectory.” The login module name is the same as the name used in the jaas-activedirectory.conf file.
STEP 4. Create file /etc/rundeck/rundeckusers.aclpolicy
Add the ACL policy below for the admin in Rundeck. The group field should be the Active Directory user group “rundeckusers.” All users in the AD group with have admin access in rundeck.
description: Admin project level access control. Applies to resources within a specific project.
context:
project: '.*' # all projects
for:
resource:
- equals:
kind: job
allow: [create] # allow create jobs
- equals:
kind: node
allow: [read,create,update,refresh] # allow refresh node sources
- equals:
kind: event
allow: [read,create] # allow read/create events
adhoc:
- allow: [read,run,runAs,kill,killAs] # allow running/killing adhoc jobs
job:
- allow: [create,read,update,delete,run,runAs,kill,killAs] # allow create/read/write/delete/run/kill of all jobs
node:
- allow: [read,run] # allow read/run for nodes
by:
group: [rundeckusers]
---
description: Admin Application level access control, applies to creating/deleting projects, admin of user profiles, viewing projects and reading system information.
context:
application: 'rundeck'
for:
resource:
- equals:
kind: project
allow: [create] # allow create of projects
- equals:
kind: system
allow: [read] # allow read of system info
- equals:
kind: user
allow: [admin] # allow modify user profiles
project:
- match:
name: '.*'
allow: [read,import,export,configure,delete] # allow full access of all projects or use 'admin'
storage:
- allow: [read,create,update,delete] # allow access for /ssh-key/* storage content
by:
group: [rundeckusers]
STEP 5. Configure Secure LDAP
Import the CA certificate which was used to setup Secure LDAP on the Active Directory Domain Controller. To secure the LDAP connection between the rundeck server and the AD domain controller it is recommended to import and trust the CA used on the domain controller. Then configure the jaas-ldap.conf file to use ldaps.
The vmware ovftool tool can be used to copy a VM between two ESXi servers which are not connected via shared storage. This comes in handy in a home lab environment. In the example below I am copying the VM “WIN10” to another ESXi host on my home network.
[root@mysql04p ~] ovftool -ds=datastore1 vi://[email protected]/WIN10 vi://[email protected]
Enter login information for source vi://172.16.1.11/
Username: root
Password: ********
Opening VI source: vi://[email protected]:443/WIN10
Enter login information for target vi://172.16.1.12/
Username: root
Password: ********
Opening VI target: vi://[email protected]:443/
Deploying to VI: vi://[email protected]:443/
Transfer Completed
Completed successfully
[root@mysql04p ~]
In my home lab I’ not running a full VMware vsphere cluster. The free version of ESXi does not offer the clone feature. When testing various applications I often run into the requirement to clone VMs on the same ESXi host. This can easily be accomplished with the ovftool. Below I clone the VM “KVM01” to “KVM01v2.”
[root@mysql04p ~] ovftool -ds=datastore1 --name=KVM01v2 --diskMode=thin vi://[email protected]/KVM01 vi://[email protected]/
Enter login information for source vi://172.16.1.11/
Username: root
Password: ********
Opening VI source: vi://[email protected]:443/KVM01
Enter login information for target vi://172.16.1.11/
Username: root
Password: ********
Opening VI target: vi://[email protected]:443/
Deploying to VI: vi://[email protected]:443/
Over the last month I have been evaluating container clustering software. I started with Kubernetes, Rancher which uses swarm and Mesos. I am going through these evaluations to determine which container clustering software will fit my employer’s needs best.
Tater's Tech Blog 