Skip to content

coding-cheatsheet.md

Latest commit

 

History

History
519 lines (490 loc) · 15.2 KB

coding-cheatsheet.md

File metadata and controls

519 lines (490 loc) · 15.2 KB

coding-cheatsheet

This is a cheatsheet to improve the quality of life when coding. I only list common used ones. For a complete git cheatsheet, please check this repository.

Table of Content

Linux operations

Basic Linux operations

Check NVIDIA cuda installabtion

nvcc --version

Count the number of files with a specific name in a folder recursively

# Count the number of all files and directories in the current directory
ls -1 | wc -l

# Count the number of all files in the current directory recursively
find . -type f | wc -l

# Count the number of all files in the DIR_NAME directory recursively
find DIR_NAME -type f | wc -l

# Count number of files with a specific file name "NAME.png" in the current folder recursively
find . -type f -name "NAME.png" | wc -l

# Count number of files with a specific file extension in the current folder recursively
find . -type f -name "*.png" | wc -l

# Count number of files with a specific file extension in the DIR_NAME folder recursively
find DIR_NAME -type f -name "*.png" | wc -l

Give sudo permission to an user

sudo usermod -aG sudo [USERNAME]

Create a SSH key pair under "~/.ssh" and add the key

ssh-keygen -t rsa -b 4096 -C "[email protected]"
ssh-add ~/.ssh/id_rsa

If ssh-add does not work, run the following line before running ssh-add:

ssh-agent bash

Zip a file using tar

tar -cvzf a.tgz [FOLDER_NAME]

Unzip a tar file

tar -xzvf [FILE_NAME]

Zip a file or folder with password

zip -e archivename.zip filetoprotect
zip -er archivename.zip foldertoprotect

Get the total size of a folder with name "video"

du -sh videos/

Show disk usage

df -h

Partition and mount disks

First need to get the drive path (e.g., /dev/sda, /dev/sdb, /dev/sdc).

sudo fdisk -l

Then, partition the drive. Here we use "/dev/sdb" as an example.

sudo parted /dev/sdb

Now run the following inside the parted shell. Command "mklabel gpt" means creating the GUID Partition Table. Command "mkpart primary 0% 100%" means creating a partition that uses the entire drive. You can specify the disk size, for example, by using "mkpart primary 0TB 4TB".

mklabel gpt # the GUID Partition Table 
unit TB
mkpart primary 0% 100%
print
quit

Next run the following to check the exact drive path (e.g., /dev/sdb1, /dev/sdb2)

sudo fdisk -l

Run the following to mount the disk drive partition. Here we use an example to mount "/dev/sdb1" to "/workspace".

sudo mkdir /workspace
sudo mkfs -t ext4 /dev/sdb1
sudo mount /dev/sdb1 /workspace

Now run the following to check if the disk partition is there, and reboot the machine. It is very important to reboot the machine, as we will need the drive's UUID later. Without rebooting, the UUID can be wrong.

df -h
sudo reboot now

After rebooting, run the following to get the partition's UUID (using /dev/sdb1 as the example).

blkid /dev/sdb1

Finally, we need to put the disk partition information in fstab. See here for the fstab documentation.

sudo vim /etc/fstab

# Add the following to the file
UUID=[uuid] /workspace ext4 defaults,noatime 0 0

# Below is an example
# UUID=849hyr87-j83y-89jd-3j7s-89jyh3gtd70v /workspace ext4 defaults,noatime 0 0

To check if it is working, reboot the system and check the disk status (in this example, "/workspace" should exist after rebooting).

sudo reboot now
df -h

Fix improper LVM partitions

Sometimes the LVM partitions may not use the entire available disk space, and we want to fix this. The solution is obtained from this link.

# We need to resize the logical volume to use all the existing and free space of the volume group
$ lvm
lvm> lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
lvm> exit

# And then, we need to resize the file system to use the new available space in the logical volume
$ resize2fs /dev/ubuntu-vg/ubuntu-lv
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/ubuntu-vg/ubuntu-lv is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 58
The filesystem on /dev/ubuntu-vg/ubuntu-lv is now 120784896 (4k) blocks long.

# Finally, you can check that you now have available space:
$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               3.9G     0  3.9G   0% /dev
tmpfs                              786M  1.2M  785M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  454G  3.8G  432G   1% /

Extend disk space

The commands are borrowed from this link. First, we need to use parted to get the extra partition that is available on the disk. Replace the /dev/vdb thing with the disk that you want to extend. You can use df -h to check the disk.

sudo parted /dev/vdb

Now run the following inside the parted shell.

print

You will see a warning as shown below. Type 'Fix' to automatically correct this.

Warning: Not all of the space available to /dev/vdb appears to be used, you can fix the GPT to use all of the space (an extra 4294967296 blocks) or continue with the current
setting? 
Fix/Ignore? Fix

Then, you will see output similar to the following.

Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  2199GB  2199GB  ext4         primary

Next, enlarge the partition with the following command (1 is the number of the partition you want to expand).

resizepart 1

You will then be asked where the partition should end, and here you enter the desired (maximum) value of your disk. In this example, 4398GB (the amount when you use the print command in the parted shell).

End? (2199GB)? 4398GB

Next, quit the parted shell.

quit

Finally, use the resize2fs command to resize the file system properly.

resize2fs /dev/vdb1

Create a ramdisk with 20G

First, create ramdisk.

sudo mkdir -p /mnt/ramdisk1
sudo mount -t tmpfs -o size=20G tmpfs /mnt/ramdisk1

Then, add informtion to the fstab.

sudo vim /etc/fstab

# Add the following to the file
# ramdisk for breathecams
tmpfs  /mnt/ramdisk1  tmpfs  rw,size=20G  0   0

After that, run the following to mount the ramdisk without rebooting the machine.

sudo mount /mnt/ramdisk1

Add a new user

Add new user on a machine, set a default password for the user, force the user to set a new password, create the home directory for the user, and set the default shell for the user to bash.

sudo useradd USERNAME
sudo passwd USERNAME
sudo passwd --expire USERNAME
cd /home/
sudo mkdir USERNAME
sudo chown -R USERNAME USERNAME
sudo chgrp -R USERNAME USERNAME
sudo usermod -s /bin/bash USERNAME

git operations

Basic git operations

Clone a repository with all submodules

git clone --recursive [REPOSITORY_URL]

Pull latest changes and all the submodules

git pull --recurse-submodules

Check status

git status

Add the changes of only one file

git add [YOUR_FILE_NAME]

Add all changes (be careful when using this command since you may add many unrelated files to the repository)

git add .

Undo the step of adding changes

git reset

Undo the step of adding changes of one file

git reset [YOUR_FILE_NAME]

Create a commit. Before commiting changes, always check the list of files that you just add (by using "git status").

git commit -m "Add your commit here."

Push code to the remote repository

git push

Check the differences between the current state and the previous commit

git diff

List branches

git branch

Create a branch and switch to it

git checkout -b [NEW_BRANCH_NAME]

Switch to a specific branch

git checkout [BRANCH_NAME]

Revert the changes of the file to the HEAD

git checkout HEAD [FILE_NAME]

Delete a local branch

git branch -D [BRANCH_NAME]

Pull the commits from the main branch

git pull

Rebase the local branch based on the main branch

git pull --rebase

Stash current changes (so that you can pull or rebase)

git stash

Pop the stashed changes back

git stash pop

Add submodule

git rm --cached -r [FOLDER_NAME]
git submodule add [REPOSITORY_URL] [FOLDER_NAME]

Edit the local repository config

git config -e --local

Hide ^M symbol at the end of line in git diff

git config --global core.whitespace cr-at-eol

Configure Git to ensure line endings in files you checkout are correct for macOS

git config --global core.autocrlf input

Combined git operations

Problem fix

If git push is pending for a long time, try the following solution to increase the postBuffer size and then push again.

git config --global http.postBuffer 5M

Commit flow

Commit the changes of files to a branch in the repository

git status
git diff
git add [FILE_1_TO_ADD]
git add [FILE_2_TO_ADD]
# ... keep adding files
git status
git commit -m "[COMMIT MESSAGE]"
git push

Branch rebase flow

Rebase a feature branch based on the main branch (so that the changes in the main branch is pulled to the feature branch).

git checkout main
git pull --rebase
git checkout [FEATURE_BRANCH]
git rebase main
git pull --rebase
git push

Merge a remote branch into the current branch and preserve commit history, using the periscope-public-engagement-tool remote repository as an example

git remote add -f periscope-public-engagement-tool https://github.com/TUD-KInD/periscope-public-engagement-tool.git

# If we want to preserve history
git merge periscope-public-engagement-tool/main --allow-unrelated-histories

# If we want to squash the commits
git merge periscope-public-engagement-tool/main --allow-unrelated-histories --squash
git commit

git push
git remote rm periscope-public-engagement-tool

Add a remote branch as a subtree in the current repository (squash the commit history), using the periscope-public-engagement-tool remote repository as an example

# Initialize the subtree
git subtree add --prefix periscope-public-engagement-tool https://github.com/TUD-KInD/periscope-public-engagement-tool.git main --squash
git push

# Update the subtree
git subtree pull --prefix periscope-public-engagement-tool https://github.com/TUD-KInD/periscope-public-engagement-tool.git main --squash
git push

You can simplify the workflow by adding the sub-repository as a remote:

# Initialize the subtree
git remote add -f periscope-public-engagement-tool https://github.com/TUD-KInD/periscope-public-engagement-tool.git
git subtree add --prefix periscope-public-engagement-tool periscope-public-engagement-tool main --squash
git push

# Update the subtree
git fetch periscope-public-engagement-tool main
git subtree pull --prefix periscope-public-engagement-tool periscope-public-engagement-tool main --squash
git push

conda operations

Document for managing conda environments can be found here.

Basic conda operations

Activate an environment

conda activate [ENVIRONMENT_NAME]

Deactivate an environment

conda deactivate

List all environments

conda env list

Create an environment

conda create -n [ENVIRONMENT_NAME]

List the installed packages in an environment

conda list

Remove an environment

conda env remove -n [ENVIRONMENT_NAME]

Remove a conda package from an environment

conda remove -n [ENVIRONMENT_NAME] [PACKAGE_NAME]

Export the environment

conda env export > environment.yml

Create an environment from "environment.yml"

conda env create -f environment.yml

To update an environment, edit the contents of "environment.yml" file accordingly and then run the following command

conda env update --prefix ./env --file environment.yml  --prune

pip operations

Install a package

pip install [PACKAGE_NAME]==1.1.2

Update a package

pip install --upgrade [PACKAGE_NAME]

Install a package with version number

pip install [PACKAGE_NAME]==[VERSION_NUMBER]

Update a package to a version number

pip install --upgrade [PACKAGE_NAME]==[VERSION_NUMBER]

Remove a package

pip uninstall [PACKAGE_NAME]

postgresql operations

Access the postgres superuser

# For Ubuntu
sudo -u postgres psql postgres

# For Mac OS
psql postgres

Remove postgresql

# For Ubuntu
dpkg -l | grep postgres
sudo apt-get --purge remove [package_names]

# For Mac OS
brew remove postgresql

Start postgresql

sudo systemctl start postgresql

Stop postgresql

sudo systemctl stop postgresql

Restart postgresql

sudo systemctl restart postgresql

Stop the postgresql

sudo systemctl stop postgresql

ffmpeg operations

Re-encode videos using H.264 format so that the videos can be played on browsers.

ffmpeg -i input.mp4 -c:v libx264 -crf 18 -preset slow -c:a copy output.mp4

A shell script to re-encode all videos with the ".mp4" file extension in all subfolders of the current folder. Only the videos that are not encoded in H.264 will be processed.

#!/bin/bash

# Define the folder where you want to process .mp4 files
folder="."

# Use 'find' to locate all .mp4 files in subfolders and process each one
find "$folder" -type f -name "*.mp4" -exec sh -c '
  for file do
    # Use ffprobe to check the video codec
    codec=$(ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 "$file")

    # Check if the codec is not h264
    if [ "$codec" != "h264" ]; then
      output="${file%.*}_processed.mp4"  # Temporary output filename
      ffmpeg -i "$file" -c:v libx264 -crf 18 -preset slow -c:a copy "$output"
      mv "$output" "$file"  # Rename and replace the original file
    fi
  done
' sh {} +