We have already shown on this blog post how to download GOES-16 data from the NOAA’s Amazon Simple Storage Service (S3) GOES Archive using this great web interface developed by Brian Blaylock, from the University of Utah.

Now we’re going to show you how to do the same, using a Shell Script developed and shared by Dr. Marcial Garbanzo from the University of Costa Rica (Thanks Marcial!).

This procedure was tested on Ubuntu 16.04.

1-) Download and install the AWS Command Line Interface using the following command:

sudo apt install awscli

2-) Save the script below at the folder of your preference. In our example it was saved at /home/GOES-16_AWS/downloadGOES16.sh:

#!/bin/bash

#########################################
# LICENSE
#Copyright (C) 2012 Dr. Marcial Garbanzo Salas
#This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
#This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
#########################################

#########################################
# AUTHOR
# This program was created at the University of Costa Rica (UCR)
# It is intended as a tool for meteorology students to obtain data from GOES16
# but it can be used by operational and research meteorology.
#########################################

#########################################
# Warning: This program can download a LARGE amount of information
# and this can cause problems with limited bandwidth networks or
# computers with low storage capabilities.
#########################################

#########################################
# CLEANING FROM PREVIOUS RUNS
#
rm DesiredData.txt
rm FullList.txt
#########################################

echo "GOES16 ABI data downloader"

#########################################
# CONFIGURATION
#
# YEAR OF INTEREST
YEARS='2017'

# DAYS OF THE YEAR
# Can use this link to find out: https://www.esrl.noaa.gov/gmd/grad/neubrew/Calendar.jsp
# Example: 275 for October 2nd, 2017
# NOTE: There is only about 60 days previous to the current date available
DAYS="342 343 344 345 346"

# CHANNELS
# Example: CHANNELS='C01 C02 C03 C04 C05 C06 C07 C08 C09 C10 C11 C12 C13 C14 C15 C16'
CHANNELS='C06'

# ABI PRODUCTS
# For a description look into:
# https://aws.amazon.com/public-datasets/goes/
# and
# http://edc.occ-data.org/goes16/getdata/
# Example: PRODUCTS='L1b-RadC L1b-RadF L1b-RadM L2-CMIPC L2-CMIPF L2-CMIPM L2-MCMIPC L2-MCMIPF L2-MCMIPM'
PRODUCTS='L1b-RadF L2-CMIPF'
#########################################

#########################################
# Get list of remote files available
# PART 1. Obtain full list of files
#
for PRODUCT in $PRODUCTS; do
for YEAR in $YEARS; do
for DAY in $DAYS; do

aws s3 --no-sign-request ls --recursive noaa-goes16/ABI-$PRODUCT/$YEAR/$DAY/ | awk '{print $3";"$4}' >> FullList.txt

done
done
done

#
# PART 2. Select only desired channels
for CHANNEL in $CHANNELS; do
grep $CHANNEL FullList.txt >> DesiredData.txt
done
#########################################

#########################################
# DOWNLOAD
#

for x in $(cat DesiredData.txt);
do
SIZE=$(echo $x | cut -d";" -f1)
FULLNAME=$(echo $x | cut -d";" -f2)
NAME=$(echo $x | cut -d"/" -f5)

echo "Processing file $NAME of size $SIZE"
if [ -f $NAME ]; then
 echo "This file exists locally"
 LOCALSIZE=$(du -sb $NAME | awk '{ print $1 }')
 if [ $LOCALSIZE -ne $SIZE ]; then
 echo "The size of the file is not the same as the remote file. Downloading again..."
 aws s3 --no-sign-request cp s3://noaa-goes16/$FULLNAME ./
 else
 echo "The size of the file matches the remote file. Not downloading it again."
 fi
else
 echo "This file does not exists locally, downloading..."
 aws s3 --no-sign-request cp s3://noaa-goes16/$FULLNAME ./
fi

done
#########################################

echo Program ending.

3-) As the administrator (root), you must change the file permissions in order to execute the script, with the chmod +x * command for example:

chmod +x *

4-) Change the script according to your needs (Year, Julian Day, Channels and Products).

gedit downloadGOES16.sh &

You may change the following:

• Year of interest: Line 38
• Julian day (three digits!): Line 44 (as mentioned on the script, you may check the julian days at this link)
• ABI Channel: Line 48 (C01 to C16)
• ABI Products to Download: Line 56

The ABI Products (Line 56) may be:

• L1b-RadC: Level 1b Radiances (CONUS)
• L1b-RadF: Level 1b Radiances (Full-Disk)
• L1b-RadM: Level 1b Radiances (Mesoscale)
• L2-CMIPC: Level 2 CMI (CONUS)
• L2-CMIPF: Level 2 CMI (Full-Disk)
• L2-CMIPM: Level 2 CMI (Mesoscale)
• L2-MCMIPC: Level 2 CMI (CONUS) – All 16 bands [2 km] in a single NetCDF file.
• L2-MCMIPF: Level 2 CMI (Full-Disk) – All 16 bands [2 km] in a single NetCDF  file.
• L2-MCMIPM: Level 2 CMI (Mesoscale) – All 16 bands [2 km] in a single NetCDF file.

5-) Test it! For our test we choose:

• Year of interest (Line 38): ‘2018’
• Julian day (three digits!) (Line 44): ‘010’ (day of this blog post :))
• ABI Channel (Line 48): ‘C13’
• ABI Products to Download (Line 56): ‘L2-CMIPF’

Execute the script:

./downloadGOES16.sh

If it’s the first time you’re running the script, you should see messages saying that the ‘DesiredData.txt’ and ‘FullList.txt’ log files couldn’t be removed. That’s normal, for they weren’t created yet.

Then, the script will start downloading all the chosen files available for that day. If you already have a given file for that day on this directory, the script will not download it.

After the script execution you’ll see the “Program ending.” message, and all the GOES-16 files you choose to download will be available on the directory you run the script.

We start running the script at 14:15 UTC. The last file download was from 14:11 UTC:

IMPORTANT: As seen on the script, this program can download a LARGE amount of information and this can cause problems with limited bandwidth networks or computers with low storage capabilities.

Note: Using AWS, you may download up to 60 days of historic data from GOES-16. 

All right! Now you may use our Python Tutorials in order to plot the data downloaded from AWS too! The image below shows the plot of the file mentioned above:

You may use this script operationally, putting it on cron and passing the year, julian day, channel and product as parameters to the script! Great!