It is important to note that, starting around 2020, Apple came out with its own chips instead of using Intel chips – these are referred to as Apple silicon chips. This change caused issues with older versions of Python and with some Python libraries. We should not encounter these in this book since we are using newer versions, but it is something to keep in mind if you ever run into issues. This book was developed on an Apple M2 Pro chip, which is an Apple silicon chip.

The code in this book was developed and tested on a computer with the following specifications:

• MacBook Pro
• Chip: Apple M2 Pro
• RAM: 16 GB
• OS: Sequoia 15.1
• Python version: 3.12.2

If you do not have a MacBook, there are several options you could explore using cloud workstations. These options will give you a handy alternative where you can set up the appropriate environment.

If you have an AWS account, you can set up a macOS EC2 instance to use as a cloud workstation. They offer an M2 Mac Pro option matching the computer this book was developed on.

Here are some additional sites you can check out if you want to use a Mac workstation on the cloud:

• Mac In Cloud: https://www.macincloud.com/
• Mac Stadium: https://www.macstadium.com/vdi

In the next section, we’ll walk through setting up your system and making sure you have Python and some of the basic libraries you need (we’ll install more libraries as we go).

Free Benefits with Your Book

Your purchase includes a free PDF copy of this book along with other exclusive benefits. Check the Free Benefits with Your Book section in the Preface to unlock them instantly and maximize your learning experience.

Next, we will begin setting up your required software libraries, including Python itself. If you are already using a different Python distribution, you are strongly encouraged to consider Anaconda, as it has become the de facto standard for data science and bioinformatics. Also, it is the distribution that will allow you to install software from bioconda (https://bioconda.github.io/).

Getting ready

Python can be run on top of different environments. For instance, you can use Python inside the Java Virtual Machine (JVM) (via Jython or with .NET via IronPython). However, here, we are not only concerned with Python but also with the complete software ecology around it. Therefore, we will use the standard (CPython) implementation, since the JVM and .NET versions exist mostly to interact with the native libraries of these platforms.

For our code, we will be using Python 3.12. If you were starting with Python and bioinformatics, any operating system would work. But here, we are mostly concerned with intermediate to advanced usage, and so we will focus on macOS.

If you are on Windows and do not have easy access to macOS or Linux, don’t worry. Modern virtualization software (such as VirtualBox and Docker) will come to your rescue, which will allow you to install a virtual OS on your operating system. Another option is to use Windows Subsystem for Linux (WSL2), which allows you to run Linux on Windows. For documentation on WSL2, look here:

• https://learn.microsoft.com/en-us/windows/wsl/install
• https://www.windowscentral.com/how-install-wsl2-windows-10
• https://docs.docker.com/desktop/features/wsl/

Another option for you will be to use a cloud workstation (see the Technical requirements section).

Bioinformatics and data science are moving at breakneck speed; this is not just hype, it’s a reality. When installing software libraries, choosing a version might be tricky. Depending on the code that you have, it might not work with some old versions or perhaps not even work with a newer version. Hopefully, any code that you use will indicate the correct dependencies – though this is not guaranteed. In this book, we will fix the precise versions of all software packages, (or provide you with a minimal version, or specify one in the associated chapter YAML file as appropriate. Check your chapter’s README.md file or the Updates section of each notebook for more information.) and we will make sure that the code will work with them. It is quite natural that the code might need tweaking with other package versions.

The software developed for this book is available at https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-fourth-edition. To access it, you will need to install Git. First, make sure HomeBrew is installed (https://brew.sh/):

brew install git

You can go to the GitHub page for the book and get the HTTPS link for downloading the source:

git clone https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Fourth-Edition.git

This will download the source code to your computer.

Before you install the Python stack, you will need to install all of the external non-Python software that you will be interoperating with. The list will vary from chapter to chapter, and all chapter-specific packages will be explained in their respective chapters. Most of the software is available via bioconda (https://bioconda.github.io/) (also called conda for short) or is pip installable (https://pypi.org/project/pip/).

Where possible in this book, we will allow you to do everything from your Jupyter notebook, even installing the software. To do this, we will use the ! command, which allows you to run a command that you would normally run from your Terminal from the notebook instead – for example:

! ls

This will run the ls or list directory command as if it had been run from the Terminal.

In some cases, for more involved installations, you will need to go into the Terminal, but we’ll advise you on how to do those steps as we go through the relevant recipes.

You will need to install some development compilers and libraries, all of which are free. On Ubuntu, consider installing the build-essential package (apt-get install build-essential), and on macOS, consider Xcode (https://developer.apple.com/xcode/).

We will mention many amazing Python libraries in this book, but here is a brief overview of some of the most important ones:

Table 1.1 – Major Python packages that are useful in bioinformatics

We will use pandas to process most table data.

How to do it...

To get started, take a look at the following steps:

1. Start by downloading the Anaconda distribution from https://www.anaconda.com/products/individual. We will be using version 2024.06, although you will probably be fine with the most recent one. You can accept all the installation’s default settings, but you might want to make sure that the conda binaries are in your path (do not forget to open a new window so that the path can be updated).
2. As an alternative to downloading from the website, you can use this command:

   curl -O https://repo.anaconda.com/archive/Anaconda3-2024.06-1-MacOSX-x86_64.sh

3. If you have another Python distribution, be careful with PYTHONPATH and existing Python libraries. It’s probably better to unset PYTHONPATH. As much as possible, uninstall all other Python versions and installed Python libraries. These steps will help reduce future confusion about which installation of Python you are pointing to.
4. Let’s go ahead with the libraries. We will now create a new conda environment called bioinformatics_base with biopython=1.84, as shown in the following command (type it in your Terminal):

   conda create -n bioinformatics_base python=3.12

5. Let’s activate the environment, as follows:

   conda activate bioinformatics_base

6. Let’s add the bioconda and conda-forge channels to our source list:

   conda config --add channels bioconda
   conda config --add channels conda-forge

   Note: Conda channels are remote hosting locations that store common packages we may need.

7. Also, install the basic packages:

   ! conda install biopython==1.84 jupyterlab==4.3.0 matplotlib==3.9.2 numpy==2.1.0 pandas==2.2.3 scipy==1.14.1

   As an alternative to the above, you can also set up your conda environment using a file that specifies the packages needed. It is provided as bioinformatics_base.yml. It is a YAML file, which stands for "YAML Ain't Markup Language" (https://yaml.org/ To use the file run this command:

   conda env create –f ~/work/CookBook/Ch01/bioinformatics_base.yml

   This will install the required packages for you.

Tip

We often install the latest version of the package by just typing something like conda install biopython, but in this book, we will often do something called “pinning the version.” This means we write an explicit version to help with the reproducibility of the code. We won’t pin the version in every example throughout the book. In most cases, your code should work fine with the latest version. However, we’ll include version pinning where it’s necessary. If any version-specific issues arise in the future, notes will be added to the README.md file for each chapter and in the Updates section of the corresponding notebook.

8. Now, let’s save our environment so that we can reuse it later to create new environments in other machines or if you need to clean up the base environment:

   conda list –e > bioinformatics_base.txt

Tip

On the left side of your Terminal, you will see what Anaconda environment you are in so you can always tell where you are at. For instance, right now, it should say (bioinformatics_base).

One thing that can be confusing is that using the python -V command in this environment could show an older version. This is because Python 3 is referred to via the python3 command. To fix this, you want to alias the Python command. Typically, it is easiest to put this in your shell file, which is a file that is always run when you open a Terminal window. In Linux, it was .bashrc, but on macOS, you will use the .zshrc file (often pronounced z-shark).

Solution: Open your ~/.zshrc file in a text editor

Add the following line to the end of the file:

alias python=python3

Now save it.

To run it, you can type source ~/.zshrc.

Now, when you run python -V or python --version, you should see that it is 3.12. If you are in a notebook and want to double-check your version, you can run ! python -V in a cell.

There’s more...

If you prefer not to use Anaconda, you will be able to install many of the Python libraries via pip using whatever distribution you choose. You can go through the book and keep installing packages in bioinformatics_base if you want. But you may, at times, find that you want to create an environment specific to a particular chapter to help isolate any complexity in package installations. Let’s look at how to do that real quick:

For example, imagine you want to create an environment for machine learning with scikit-learn. You can do the following:

1. First, we need to deactivate our current environment:

   conda deactivate

2. Create a clone of the original environment with the following:

   conda create -n scikit-learn --clone bioinformatics_base

3. Add scikit-learn:

   conda activate scikit-learn
   conda install scikit-learn

See Also

• For more information on Git, see: https://git-scm.com/docs
• For an excellent course on using Git and GitHub, see: https://www.udemy.com/course/git-and-github-bootcamp/
• For a review of the differences between Python 3 and Python 2, see: https://powerfulpython.com/blog/whats-really-new-in-python-3/
• For more background on Anaconda, see: https://www.edureka.co/blog/python-anaconda-tutorial/
• For a course on Anaconda, see: https://www.udemy.com/course/anaconda-tutorial/

Docker is the most widely used framework for implementing operating system-level virtualization. This technology allows you to have an independent container: a layer that is lighter than a virtual machine but still allows you to compartmentalize software. This mostly isolates all processes, making it feel like each container is a virtual machine. Containers will be discussed in more detail in Chapter 14, Cloud Basics.

Docker works quite well at both extremes of the development spectrum: it’s an expedient way to set up the content of this book for learning purposes and could become your platform of choice for deploying your applications in complex environments.

Conda and Docker are key tools to help maintain software compatibility and reproducibility across different systems and libraries. We’ll discuss reproducibility more in Chapter 15, Workflow Systems.

Note

This recipe is an alternative to the previous recipe. Normally, if you have a Mac and are using it for your Jupyter notebooks, you will not need the Docker container. If you have a Windows machine or cannot get certain code to work in your environment, the Docker container can be useful to provide you with an environment that is set up properly already for you.

Getting ready

The first thing you have to do is install Docker. Go to https://www.docker.com/. Install Docker Desktop for your appropriate operating system (remember to check the Apple versus Intel silicon discussion in the Technical requirements section if you are using macOS). You’ll also need to sign up for a Docker account and record your username and password.

How to do it...

Docker Desktop must be running and you need to be signed in before downloading the Docker file. To get started, follow these steps:

1. Use the following command from your Terminal:

   docker build -t bio https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Fourth-Edition.git#main:docker/main

Tip

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository. Doing so will help you avoid any potential errors related to the copying and pasting of code.

You can find the commands for this section in the chapter’s README.md file.

2. Now you are ready to run the container, as follows:

   docker run -ti -p 9875:9875 -v YOUR_DIRECTORY:/data bio

3. Replace YOUR_DIRECTORY with a directory on your operating system. This will be shared between your host operating system and the Docker container. YOUR_DIRECTORY will be seen in the container in /data and vice versa.

   In this case, -p 9875:9875 will expose the container’s TCP port 9875 on the host computer port, 9875.

   Especially on Windows (and maybe on macOS), make sure that your directory is actually visible inside the Docker shell environment. If not, check the official Docker documentation on how to expose directories. To access the Docker image while it’s running, hover over the Docker Desktop icon. All the files available in the book’s GitHub repository will be mirrored in the Docker image.

4. Now you are ready to use the system. Point your browser to http://localhost:9875 and you should get the Jupyter environment.

If this does not work on Windows, check the official Docker documentation (https://docs.docker.com/manuals/) on how to expose ports.

See also

• Docker is the most widely used containerization software and has seen enormous growth in usage in recent times. You can read more about it at https://www.docker.com/.
• A security-minded alternative to Docker is Red Hat Openshift, which can be found at https://www.redhat.com/en/technologies/cloud-computing/openshift.
• If you are not able to use Docker, for example, if you do not have the necessary permissions, as will be the case for most compute clusters, then take a look at Singularity at https://www.sylabs.io/singularity/.
• For a good course on Docker, see: https://www.udemy.com/course/docker-and-kubernetes-the-complete-guide/.

All of our work will be developed inside Jupyter Notebook. Jupyter has become the de facto standard for writing interactive data analysis scripts. Unfortunately, the default format for Jupyter notebooks is based on JSON. JSON is JavaScript Object Notation (https://www.json.org/json-en.html). This format is difficult to read, difficult to compare, and needs exporting to be fed into a normal Python interpreter. To obviate that problem, we will extend Jupyter with jupytext (https://jupytext.readthedocs.io/), which allows us to save Jupyter notebooks as normal Python programs. We will start with an overview of Jupyter Notebook, and then look into jupytext. Recall that we installed Jupyter Notebook in the first recipe of this chapter, when we installed the jupyterlab package using conda.

How to do it…

1. To run Jupyter, on the Terminal, type the following:

   jupyter notebook

   This will open the Jupyter browser, and you will see a home page that looks something like this:

Figure 1.1 – The Jupyter browser home page

This home page gives you an overview of your files, so you can open, rename, and download them, and so on.

2. Let’s click on one of the files and open it. We will see something like this:

Figure 1.2 – An example of a notebook

Here, we see a menu that allows us to save or download files and perform other actions. Each cell can be executed by clicking the play button. You can also run multiple cells. When you run a cell, you will see its output below.

In some cases, you may need to restart your kernel – use the Kernel | Restart Kernel... method.

Jupyter notebook resources

This would be a good time to pause and take some time to learn more about Jupyter notebooks. There are numerous keyboard shortcuts that are worth learning to speed up your development:

Tutorial: https://www.datacamp.com/tutorial/tutorial-jupyter-notebook

Keyboard Shortcuts: https://towardsdatascience.com/jypyter-notebook-shortcuts-bf0101a98330

Now that we have set up our Jupyter Notebook environment, let’s take a look at a handy tool called Jupytext.

Jupytext

Sometimes, you will want to convert your notebooks into formats other than ipynb – for example, you might want to get them into .py format. For this, we can use jupytext - https://github.com/mwouts/jupytext. This handy plugin will allow us to save Jupyter notebooks in formats other than .ipynb. Remember to get out of the Jupyter browser first. To do this on a Mac, you would close the Jupyter browser window, then go to the Terminal where you started it. Then, click Ctrl + Z to kill the process.

To install jupytext, we will run the following:

pip install jupytext

Now let’s start up the Jupyter browser again:

jupyter notebook

Now we’ve launched the Jupyter browser again, open the Welcome notebook. Go to the File | Jupytext menu. Here is what it looks like:

Figure 1.3 – The Jupytext menu within the Jupyter browser

To save your notebook in another format, you pair it and choose a format. For instance, if you choose to pair it with the light format, you will get a regular Python (.py) formatted file in your current working directory.

Here is what our Welcome.py file looks like in our working directory when paired with the light format:

Figure 1.4 – The Welcome.py notebook in the Light format produced by Jupytext

There are several other popular formats supported by Jupytext. You can read more about them here: https://jupytext.readthedocs.io/en/latest/index.html.

Warning

Remember that the recipes in this book are normally meant to be run inside Jupyter notebooks. This means, typically, we will not always use print to output content. In a notebook, if you simply put the name of a variable and run it, it will print out the result for you. If you are not using notebooks (e.g. you are writing Python scripts and executing them from the terminal), you may want to add print statements to your code. Even within a notebook, you may find it useful at times to add your own print statements to inspect variables and debug code.

In addition to the Jupyter browser, there is a more integrated environment called JupyterLab: https://jupyterlab.readthedocs.io/en/latest/. It allows you to run Terminals and other widgets inside the same environment as your notebook. To get to it, you can click View | Open JupyterLab. You can check it out if you are interested, but it is not necessary to get through the book.

A welcome notebook called Welcome.ipynb has been placed in the GitHub repository for this book in the Ch01 folder. You can use it to test out your notebook environment. This notebook also contains many handy links to help you learn Python and explore bioinformatics!

To recap everything, here are your main options for performing the recipes in this book:

Table 1.2 – System and OS options for use with this book

See also

• Python Introductory Course: https://www.udemy.com/course/python-for-absolute-beginners-u/
• Intermediate Python Course: https://www.udemy.com/course/100-days-of-code/
• For basic instructions on IPython magics, see: https://ipython.readthedocs.io/en/stable/interactive/magics.html
• A list of third-party extensions for IPython, including magic ones can be found at https://github.com/ipython/ipython/wiki/Extensions-Index
• For a list of JupyterLab widgets, see: https://github.com/search?q=topic%3Ajupyterlab-extension&type=Repositories
• For a deep dive into modern Python 3.12, check out Modern Python Cookbook by Steven F. Lott (Packt Publishing): https://www.amazon.com/Modern-Python-Cookbook-updated-techniques/dp/1835466389

  Get This Book's PDF Version and Exclusive Extras Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page.
  Note: Keep your invoice handy. Purchases made directly from Packt don’t require an invoice.