1 David Beazley @dabeaz http://www.dabeaz.com Presented at PyCon'2013, Santa Clara, CA March 13, 2013
3.3 • Support files: http://www.dabeaz.com/pydata • Also, datasets passed around on USB-key
using the interactive shell >>> print "hello world" hello world >>> 37*42 1554 >>> for i in range(5): ... print i ... 0 1 2 3 4 >>> 12 >>> is the interpreter prompt for starting a new statement ... is the interpreter prompt for continuing a statement (it may be blank in some tools) Enter a blank line to finish typing and to run
in .py files # helloworld.py print "hello world" • Create with your favorite editor (e.g., emacs) • Can also edit programs with IDLE or other Python IDE (too many to list) 13
Python program is a sequence of statements • Each statement is terminated by a newline • Statements are executed one after the other until you reach the end of the file. 17
is just a name for some value • Name consists of letters, digits, and _. • Must start with a letter or _ height = 442 user_name = "Dave" filename1 = 'Data/data.csv' 19
Numbers a = 12345 # Integer b = 123.45 # Floating point • Text Strings name = 'Dave' filename = "Data/stocks.dat" 20 • Nothing (a placeholder) f = None
A few common operations a = 'Hello' b = 'World' >>> len(a) # Length 5 >>> a + b # Concatenation 'HelloWorld' >>> a.upper() # Case convert 'HELLO' >>> a.startswith('Hell') # Prefix Test True >>> a.replace('H', 'M') # Replacement 'Mello >>> 22
values a = int(x) # Convert x to integer b = float(x) # Convert x to float c = str(x) # Convert x to string • Example: >>> xs = '123' >>> xs + 10 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: cannot concatenate 'str' and 'int' o >>> int(xs) + 10 133 >>> 23
if a < b: print "Computer says no" else: print "Computer says yes" • If-elif-else if a < b: print "Computer says not enough" elif a > b: print "Computer says too much" else: print "Computer says just right" 24
Relational operators < > <= >= == != • Boolean expressions (and, or, not) if b >= a and b <= c: print "b is between a and c" if not (b < a or b > c): print "b is still between a and c"
a loop • Executes the indented statements underneath while the condition is true 26 n = 10 while n > 10: print 'T-minus', n n = n - 1 print 'Blastoff!'
over a sequence of data • Processes the items one at a time • Note: variable name doesn't matter 27 names = ['Dave', 'Paula', 'Thomas', 'Lewis'] for name in names: print name for n in names: print n
print statement (Python 2) print x print x, y, z print "Your name is", name print x, # Omits newline • The print function (Python 3) 29 print(x) print(x, y, z) print("Your name is", name) print(x, end=' ') # Omits newline
file f = open("foo.txt","r") # Open for reading f = open("bar.txt","w") # Open for writing • To read data data = f.read() # Read all data • To write text to a file g.write("some text\n") 30
a huge library of functions • Example: math functions import math x = math.sin(2) y = math.cos(2) 33 • Reading from the web import urllib # urllib.request on Py3 u = urllib.urlopen('http://www.python.org) data = u.read()
u = urllib.urlopen('http://ctabustracker.com/ bustime/map/getBusesForRoute.jsp?route=22') >>> data = u.read() >>> f = open('rt22.xml', 'wb') >>> f.write(data) >>> f.close() >>> • Start the Python interpreter and type this • Don't ask questions: you have 5 minutes...
major cities provide a transit API • Example: Chicago Transit Authority (CTA) http://www.transitchicago.com/developers/ • Available data: • Real-time GPS tracking • Stop predictions • Alerts
latitude 41.980262 longitude -87.668452 Travis doesn't know the number of the bus he was riding. Find likely candidates by parsing the data just downloaded and identifying vehicles traveling northbound of Dave's office. Dave's office is located at:
Write a program that periodically monitors the identified buses and reports their current distance from Dave's office. When the bus gets closer than 0.5 miles, have the program issue an alert by popping up a web-page showing the bus location on a map. Travis will meet the bus and get his suitcase.
parse doc = parse('rt22.xml') • Parsing a document into a tree <?xml version="1.0"?> <buses rt="22"> <time>1:14 PM</time> <bus> <id>6801</id> <rt>22</rt> <d>North Bound</d> <dn>N</dn> <lat>41.875033214174465</lat> <lon>-87.62907409667969</lon> <pid>3932</pid> <pd>North Bound</pd> <run>P209</run> <fs>Howard</fs> <op>34058</op> ... </bus> ... root time bus bus bus bus id rt d dn lat lon doc
doc.findall('bus'): ... • Iterating over specific element type root time bus bus bus bus id rt d dn lat lon doc bus Produces a sequence of matching elements
doc.findall('bus'): ... • Iterating over specific element type root time bus bus bus bus id rt d dn lat lon doc bus Produces a sequence of matching elements
doc.findall('bus'): ... • Iterating over specific element type root time bus bus bus bus id rt d dn lat lon doc bus Produces a sequence of matching elements
doc.findall('bus'): ... • Iterating over specific element type root time bus bus bus bus id rt d dn lat lon doc bus Produces a sequence of matching elements
doc.findall('bus'): d = bus.findtext('d') lat = float(bus.findtext('lat')) • Extracting data : elem.findtext() root time bus bus bus bus id rt d dn lat lon doc bus "North Bound" "41.9979871114"
map : Maybe Google Static Maps https://developers.google.com/maps/documentation/ staticmaps/ • To show a page in a browser import webbrowser webbrowser.open('http://...')
Talk to your neighbors • Consult handy cheat-sheet • http://www.dabeaz.com/pydata
more complex data • Example: A place marker Bus 6541 at 41.980262, -87.668452 • An "object" with three parts • Label ("6541") • Latitude (41.980262) • Longitude (-87.668452) 54
ordered (like an array) bus = ('6541', 41.980262, -87.668452) id = bus[0] # '6541' lat = bus[1] # 41.980262 lon = bus[2] # -87.668452 • However, the contents can't be modified >>> bus[0] = '1234' TypeError: object does not support item assignment 56
a tuple bus = ('6541', 41.980262, -87.668452) id, lat, lon = bus # id = '6541' # lat = 41.980262 # lon = -87.668452 • This is extremely common • Example: Unpacking database row into vars 57
your bike on the lakefront path, you seek a new road biking challenge involving large potholes and heavy traffic. Your Task: Find the five most post-apocalyptic pothole-filled 10-block sections of road in Chicago. Bonus: Identify the worst road based on historical data involving actual number of patched potholes.
are publishing datasets online • http://data.cityofchicago.org • https://data.sfgov.org/ • https://explore.data.gov/ • You can download and play with data
need to parse CSV data import csv f = open('potholes.csv') for row in csv.DictReader(f): addr = row['STREET ADDRESS'] num = row['NUMBER OF POTHOLES FILLED ON BLOCK'] 69 • Use the CSV module
to make lookup tables potholes_by_block = {} f = open('potholes.csv') for row in csv.DictReader(f): ... potholes_by_block[block] += num_potholes ... 70 • Use a dict. Map keys to counts.
to manipulate strings >>> addr = '350 N STATE ST' >>> parts = addr.split() >>> parts ['350', 'N', 'STATE', 'ST'] >>> num = parts[0] >>> parts[0] = num[:-2] + 'XX' >>> parts ['3XX', 'N', 'STATE', 'ST'] >>> ' '.join(parts) '3XX N STATE ST' >>> 71 • For example, to rewrite addresses
to account for bad data for row in csv.DictReader(f): try: n = int(row['NUMBER OF POTHOLES FILLED']) except ValueError: n = 0 ... 73 • Use try-except to catch exceptions (if needed)
list by applying an operation to each element of a sequence. >>> a = [1,2,3,4,5] >>> b = [2*x for x in a] >>> b [2, 4, 6, 8, 10] >>> 76 • Shorthand for this: >>> b = [] >>> for x in a: ... b.append(2*x) ... >>>
values of a specific field addrs = [r['STREET ADDRESS'] for r in records] • Performing database-like queries filled = [r for r in records if r['STATUS'] == 'Completed'] 78 • Building new data structures locs = [ (r['LATITUDE'], r['LONGITUDE']) for r in records ]
key-function 80 records.sort(key=lambda p: p['COMPLETION DATE']) records.sort(key=lambda p: p['ZIP']) • lambda: creates a tiny in-line function f = lambda p: p['COMPLETION DATE'] # Same as def f(p): return p['COMPLETION DATE'] • Result of key func determines sort order
groups of sorted data 81 from itertools import groupby groups = groupby(records, key=lambda r: r['ZIP']) for zipcode, group in groups: for r in group: # All records with same zip-code ... • Note: data must already be sorted by field records.sort(key=lambda r: r['ZIP'])
packages • numpy/scipy (array processing) • matplotlib (plotting) • pandas (statistics, data analysis) • requests (interacting with APIs) • ipython (better interactive shell) • Too many others to list 83
all of that biking, but you can never be too careful. Your Task: Analyze Chicago's food inspection data and make a series of tasty pie charts and tables
should not be long • For plotting/ipython consider EPD-Free, Anaconda CE, or other distribution • See samples at http://www.dabeaz.com/pydata
• Python coding • Functions, modules, classes, objects • Data analysis • Numpy/Scipy, pandas, matplotlib • Data sources • Open government, data portals, etc.
dabeaz
tomtwinkle
fruitriin
naramomi7
sunterras
naruyoshimi
yohfee
sembayui
hynek
tigertora7571
konokenj
hiroyukimori
okuramasafumi
lara
pwiseman
akosma
ianozsvald
cromwellryan
tenderlove
bermonpainter
mfonobong
schroneko
aleyda
eileencodes
holman
