pyDAL, a pure Python Database Abstraction Layer

By Vasudev Ram

pyDAL is a pure Python Database Abstraction Layer. So it seems to be something like the lower layer of SQLAlchemy, i.e. SQLAlchemy Core, the library that is used by the upper layer, SQLAlchemy ORM. See the SQLAlchemy (0.8) documentation.

From the pyDAL site:

[ It dynamically generates the SQL in real time using the specified dialect for the database back end, so that you do not have to write SQL code or learn different SQL dialects (the term SQL is used generically), and your code will be portable among different types of databases.

pyDAL comes from the original web2py's DAL, with the aim of being wide-compatible. pyDAL doesn't require web2py and can be used in any Python context. ]

IOW, pyDAL has been separated out into a different project from web2py, a Python web framework, of which it was originally a part.

The use of an ORM (Object Relational Mapper) vs. writing plain SQL code (vs. using an intermediate option like pyDAL or SQLAlchemy Core), can be controversial; there are at least some pros and cons on both (or all 3) sides. I've read some about this, and have got some experience with using some of these options in different projects, but am not an expert on which is the best approach, and also, it can vary depending on your project's needs, so I'm not getting into that topic in this post.

pyDAL seems to support many popular databases, mostly SQL ones, but also a NoSQL one or two, and even IMAP. Here is a list, from the site: SQLite, PostgreSQL, MySQL, Oracle, MSSQL, FireBird, DB2, Informix, Ingres, Cubrid, Sybase, Teradata, SAPDB, MongoDB, IMAP.

For some of those databases, it uses PyMySQL, pyodbc or fbd, which are all Python database libraries that I had blogged about earlier.

I tried out pyDAL a little, with this simple program, adapted from its documentation:

import sys
import time
from pydal import DAL, Field
db = DAL('sqlite://storage.db')
db.define_table('product', Field('name'))
t1 = time.time()
num_rows = int(sys.argv[1])
for product_number in range(num_rows):
    db.product.insert(name='Product-'.format(str(product_number).zfill(4)))
t2 = time.time()
print "time to insert {} rows = {} seconds".format(num_rows, int(t2 - t1))
query = db.product.name
t1 = time.time()
rows = db(query).select()
for idx, row in enumerate(rows):
    #print idx, row.name
    pass
t2 = time.time()
print "time to select {} rows = {} seconds".format(num_rows, int(t2 - t1))

It worked, and gave this output:

$ python test_pydal2.py 100000
No handlers could be found for logger "web2py"
time to insert 100000 rows = 18 seconds
time to select 100000 rows = 7 seconds

Note: I first ran it with this statement uncommented:

#print idx, row.name

to confirm that it did select the records, and then commented it and replaced it with "pass" in order to time the select without the overhead of displaying the records to the screen.

I'll check out pyDAL some more, for other commonly needed database operations, and may write about it here.
There may be a way to disable that message about a logger.

The timing statements in the code and the time output can be ignored for now, since they are not meaningful without doing a comparison against the same operations done without pyDAL (i.e. just using SQL from Python with the DB API). I will do a comparison later on and blog about it if anything interesting is found.

- Vasudev Ram - Dancing Bison Enterprises - Python training and consulting

Signup to hear about new products or services from me.

Contact Page

Share |

pypyodbc, a pure Python ODBC library

By Vasudev Ram

In my recent post, PyODBC for Linux, Windows and Mac OS X, Ben Rousch commented: "I've had better luck with pure-Python PyPyODBC than with PyODBC."

So I downloaded and tried out pypyodbc.

Features of pypyodbc (from its web page):

Design Goal: Small, Maintainable, Cross Platform, Compatible, Portable

Features
One pure Python script, runs on CPython / IronPython / PyPy , Python 3.3 / 2.4 / 2.5 / 2.6 / 2.7 , Win / Linux / Mac , 32 / 64 bit
Very similar usage as pyodbc ( can be seen like a re-implementation of pyodbc in pure Python )
Simple - the whole module is implemented in a single python script with less than 3000 lines
Built-in functions to create and compress Access MDB files on Windows

I first tried the simple test program given on the pypyodbc page, which just creates a new MS Access database using Python. It worked.

Then I added code to that test program to insert three rows to the table and then and then retrieve the rows from the database and print them.

Here is the code:

# test_ppo.py

import pypyodbc 
             
pypyodbc.win_create_mdb('C:\\vr\\pypyodbc\\db2.mdb')
connection_string = 'Driver={Microsoft Access Driver (*.mdb)};DBQ=C:\\vr\\pypyodbc\\db2.mdb'
connection = pypyodbc.connect(connection_string)

SQL = 'CREATE TABLE saleout (id COUNTER PRIMARY KEY,product_name VARCHAR(25));'
connection.cursor().execute(SQL).commit()

SQL = "INSERT INTO saleout values (1, 'apple');"
connection.cursor().execute(SQL).commit()

SQL = "INSERT INTO saleout values (2, 'banana');"
connection.cursor().execute(SQL).commit()

SQL = "INSERT INTO saleout values (3, 'orange');"
connection.cursor().execute(SQL).commit()

SQL = 'SELECT * FROM saleout;'
cursor = connection.cursor().execute(SQL)
for row in cursor:
    print row

And I got the expected output:

C:\vr\pypyodbc>python test_ppo.py
(1, u'apple')
(2, u'banana')
(3, u'orange')

- Vasudev Ram - Dancing Bison Enterprises

Share |

PyODBC for Linux, Windows and Mac OS X

By Vasudev Ram

PyODBC is like the odbc module in the Standard Python library, but claims to work on Linux, Windows and Mac OS X. I've used the odbc module from the stdlib earlier. Using it is straightforward for basic CRUD-type usage.

PyODBC is supposed to "implement the Python Database API Specification v2.0, but additional features have been added to simplify database programming even more." I am trying out PyODBC (have taken an initial look at the examples, seems similar to the stdlib odbc module) and will report on it a little later.

ODBC on Wikipedia.

- Vasudev Ram - Dancing Bison Enterprises

Share |