Python and Cryptography

Using Python to work with Databases

Copyright 2010 by Stephen Vermeulen
Last updated: 2010 Jul 22
2D and 3D Graphics in Python

120px 2d alpha amongst ascii bad benefits blogs brief cellspacing cid clone comes comparing couchdb date dec elementtree entries examines exist extends few groups https interfaces land largely load located love managing mechanism mention middle n objects papers pdf pickle protocolostomy pymotw reactor recording references simpler sqlalchemy step stored techspot title tplus1 tracked trouble ui viewing vs win32com
A large number of support libraries exist to allow one to work with various relational and object databases from a Python program.

  • 2010-Jul-22: The openstack cloud initiative includes a distributed object storage system called Swift that is written in Python. [9321]
  • 2010-Jun-25: Simple Python object persistence with Durus and ZODB. [9243]
  • 2009-Nov-26: Could CloudStore (a clone of the Google File System, GFS) be used from Python? [8804]
  • 2009-Oct-23: myconnpy is an implementation of the MySQL Client/Server protocol in Python. [8670]
  • 2009-Oct-23: tokyocabinet is a Python wrapper for the Tokyo Cabinet database library. [8668]
  • 2009-Oct-23: dobbin is a new (in late 2009) object database for Python. This is taking a different approach to ZODB in that it all data that you access is read only until you check it out. This should have some significant benefits, including easier use with multiple threads and processes and a smaller memory usage footprint. Located here on PyPI. [8578]
  • 2009-Aug-04: tempstorage is a RAM based storage implementation for ZODB. [8362]
  • 2009-Jul-28: The ZODB Browser allows you to inspect persistent objects stored in a ZODB (including viewing historical changes to them). [8344]
  • 2009-Jul-24: transaction is a transaction management package for Python, mainly used for ZODB. [8323]
  • 2009-Jun-18: freebase is a Python client library for access in the database service. [8164]
  • 2009-Jun-05: pyOODB an object-oriented BTree Database for Python. [8088]
  • 2009-Jun-04: KirbyBase is a simple database written in pure Python, it gets used with IronPython here. This database stores the records in simple CSV text files. [8074]
  • 2009-May-19: pod is an object database system for Python built from cPickle and sqlite. [7991]
  • 2009-May-01: Geraldo is a report engine for Python and Django applications. [7923]
  • 2009-Apr-23: schevo is a next-generation database system for Python. It is being re-licensed. [7873]
  • 2009-Feb-24: stockpyle is a library that provides write-through storage for object caching and persistence. [7641]
  • 2009-Feb-04: A simple database-like thing, essentially a dictionary with persistence. [7536]
  • 2009-Jan-30: pymssql is a simple database interface to MS-SQL. [7507]
  • 2009-Jan-28: pymongo is a driver for the Mongo, document-based, database. [7495]
  • 2009-Jan-28: keas.pbstate and keas.pbpersist can be combined with ZODB to allow for an object database that stores its data in Google's "protocol buffers" format, which may prove to be more portable between languages than pickling is. [7491]
  • 2009-Jan-21: esedb is a Python module that uses the Esent database engine (which is used by Windows Update and Microsoft Exchange) to provide another Python database. [7464]
  • 2008-Dec-19: RelStorage is a backend for ZODB that stores pickles in a relational database. This can allow multiple ZODB clients to share the same database without additional configuration. [7370]
  • 2008-Dec-04: Durus is a transactional persistent object system for Python. The PyPI page for it is here [302]
  • 2008-Dec-03: The Zope Object DataBase (ZODB, the package is here) can be used without the rest of the Zope environment to give Python programs a transactional object database. A brief introduction to using it is here (and discusses a bit of how object references are maintained persistently. An IBM DeveloperWorks discussion of it with some simple use example code. The ZODB/ZEO Programming Guide is online here, a stand-alone PDF copy can be obtained here.

    tempstorage provides a RAM based storage implementation for ZODB.

    Introduction to the Zope Object Database provides a good summary of what the ZODB is, how it behaves and how to use it. FileStorageBackup talks about the design of the FileStorage type of storage system for ZODB, as well as the repozo tool for backing it up and some other integrity related tools. This article includes a list of tools that can inspect or analyze ZODB databases.

    How to Love ZODB and Forget RDBMS.

    CouchDB for ZODB Users takes a look at CouchDB and how it compares to ZODB. [6392]

  • 2008-Oct-28: Some more views on using the ZODB, in ZODB-LESS ZOPE the author reports on bad performance with the 2.7 series of ZODB. This article mentions problems with query performance when their database hit 30K objects. [7108]
  • 2008-Sep-15: The PyMOTW takes a look at the anydbm module which provides access to DBM-style databases. [6856] [1] [2]
  • 2008-Jul-29: gerald is a tool for recording and comparing database schemas. It is also available here on SourceForge and here on PyPi. [6605]
  • 2008-Jul-28: pysimpledb is an attempt to make working with databases in Python simpler. [6599]
  • 2008-Jul-23: sqlcmd is an SQL command line tool. [6567]
  • 2008-Jul-23: dbmigrate is a utility set for managing database migrations. [6565]
  • 2008-Jul-04: wikidBASE, a project that crosses a wiki web application with a database system, allowing for a database to be designed and evolved over time without resorting to tricky tools. Sounds like this might be useful as a system design tool as well. [311]
  • 2008-Jun-20: twist talking to the ZODB in Twisted Reactor Calls. [6396]
  • 2008-Jun-11: dabo (home page here) is a 3-tier framework for desktop database applications. This also includes a wrapper layer for wxPython (called dabo.ui) that can be used separately. [6316] [1]
  • 2008-Jun-05: Autumn is a simple Python ORM (object-relational mapper). [6288]
  • 2008-May-23: Using Python to update selected records in a database or using an SQL command to do this. [6215]
  • 2008-May-21: dbtxt is a small, free database written in Python. It gets mentioned on Slashdot here. [6205]
  • 2008-May-17: Using Git as a versioned data store in Python examines using a Git DVCS repository to provide version tracked storage to a Python application. Perhaps a better fit would be to use Mercurial (discussed here) as it is largely written in Python? [6177]
  • 2008-May-13: Some performance tips for users of MySQL. [6154]
  • 2008-May-12: pylo allows you to write Silo visualization files that are used by the Vislt program for large-scale data visualization. [6139] [1]
  • 2008-May-09: The rather inconsistent world of date objects in the land of the database. [6084]
  • 2008-May-06: The SAP DI Commander IronPython starts to manipulate SAP databases. [6057]
  • 2008-Apr-25: Recreate MS Access table in SQLite, this has some limitations but could be used to help migrate data from one database to another. Also includes some use of the win32com API to get at the Access database. [5968] [1] [2]
  • 2008-Apr-25: Multithread support for SQLite access, a work around to an SQLite limitation. [5967] [1]
  • 2008-Apr-14: python-sybase, a module to provide for the Sybase relational database system. [5801]
  • 2008-Apr-08: Using SQLAlchemy with Grok with some mention of Storm. [5743]
  • 2008-Apr-06: shove is a common object storage front end, this allows you to store Python objects to a variety of different database or file-system backends (even things like Amazon S3) using a single API. [5611] [1]
  • 2008-Mar-23: Transactional Objects, a class that allows changes to be committed or rolled back. [5325] [1]
  • 2008-Mar-12: dbcook is a framework to abstract database mappings of objects and relations to isolate the database from the application code. [5262]
  • 2008-Mar-12: DyBase is an embedded object oriented database that can be used with Python. [5261]
  • 2008-Mar-12: Some thoughts on how to use SQLObject. [5259]
  • 2008-Mar-10: bsddb3 is a Python wrapper for the BerkeleyDB. The project home page is here. [5244]
  • 2008-Mar-07: Metakit is a structured database for Python (and other languages). This gets referenced here. [5241]
  • 2008-Mar-01: A class for extraction and manipulation of data from Microsoft Access. [5200] [1]
  • 2008-Feb-28: Yet another Python ORM system is being developed. [5181]
  • 2008-Feb-05: JSONStore, a simple database for JSON documents. The Rest API for this. [5036]
  • 2007-Dec-29: Storm SQLAlchemy and Geniusql are performance compared here and here. [4535]
  • 2007-Oct-26: ceodbc, an ODBC interface for Python. [3519]
  • 2007-Oct-26: pyodbc, an ODBC interface for Python. [3518]
  • 2007-Sep-30: PyDbLite, a small in-memory database engine [2408] [1]
  • 2007-Sep-14: More on ZODB perfomance, this time its compared to Durus. [1896]
  • 2007-Sep-13: A comparison of the performance of the ZODB and SQLite3 where the ZODB does pretty well. [1886]
  • 2007-Aug-24: PEP249 contains the specification for the 2.0 revision of the Python database API. [577]
  • 2007-Aug-24: What is the fastest database interface for Python applications to use? In this simple test of adding 100000 records MySQLdb comes out on top with about 1.3 seconds.  Other interfaces take up to 68 seconds. [310]
  • 2007-Aug-24: pg8000, a PostgreSQL/Python interface written in pure Python [309]
  • 2007-Aug-24: Dee, makes Python relational. Extends the Python language with relational database concepts. [308]
  • 2007-Aug-24: PyDbLite, a small, fast, in-memory database [307]
  • 2007-Aug-24: Using Relational Databases with Python, a paper and slide presentation. [306]
  • 2007-Aug-24: PythonReports is a database report builder module for Python. [305]
  • 2007-Aug-24: Using ElementTree with Durus to make a simple distributed XML object database. [304]
  • 2007-Aug-24: Using BerkeleyDB as a backend storage engine for Durus [303]
  • 2007-Aug-24: memcached, is a distributed memory object caching system aimed at speeding up dynamic web applications by reducing database load. This looks like it might be useful if you have a model that generates data and a number of users that needs to see portions of that data (perhaps with overlapping views). One reference to using this is here (3-Oct-06). [301]
  • 2007-Aug-24: A small tutorial of sqlite usage [300]
  • 2007-Aug-24: Dumping MySQL data in CSV format using SQLAlchemy [299]
  • 2007-Aug-24: Adding Python scripting to SAP [298]
  • 2007-Aug-24: BuzHug, a pure Python database engine [297]
  • 2007-Aug-24: Cerealizer, a Pickle-like module intended to be secure. [296]
  • 2007-Aug-24: Python PalmDB, a pure python library to read/write/modify Palm PDB and PRC format files. [295]
  • 2007-Aug-24: Why the schema should be in the database. [294]
  • 2007-Aug-24: FileSystemStorage, aims to be a Python DBAPI 2.0 compliant driver for treating the file system as a database. This driver will provide a mechanism for creating entities via SQL, querying via SQL, and connection to local or remote hosts (remote is via ssh), among other features. [293]
  • 2007-Aug-24: Storm, SQLAlchemy and Geniusql compared from a performance perspective (Aug'07). [292]
  • 2007-Aug-24: Why SQLAlchemy Impresses Me, a brief look at using SQLAlchemy [291]
  • 2007-Aug-24: sqlsoup, using sqlalchemy [290]
  • 2007-Aug-24: Using SQLAlchemy with SQLite for storage of lists of objects of varing types. Declaritive mapping with SQLAlchemy. A step-by-step SQLAlchemy tutorial. Elixir (home page) is a declarative mapper for SQLAlchemy. [289]
  • 2007-Aug-24: Emmet, is a persistant object system for Python, looks like it largly makes pickles of objects and stores them in a SQLite database [288]
  • 2007-Aug-24: MySQL interface for Python [287]
  • 2007-Aug-24: SAP R/3 RFC language bindings for Python [286]
  • 2007-Aug-24: pickleshare is a module that implements a way of sharing a number of pickle files in a safe way amongst a number of processes. [285]
  • 2007-Aug-24: C#3.0 has a feature called LINQ that allows for better SQL database integration, here are some thoughts and sample code (SQLComp) for doing something similar in Python. [284]
  • 2007-Aug-24: DBClass, is another object-relational mapping system. [283]
  • 2007-Aug-24: Problems with SQLObject and Unicode [282]
  • 2007-Aug-24: The Firebird database might be another candidate for use as a datastore, see the KInterbasBD extension package for Python. [281]
  • 2007-Aug-24: Premolo is an alternative to SQLObject. [280]
  • 2007-Aug-24: Using an SQL database with Python, the pysqlite (also at package provides an interface to SQLite, other interfaces, like SQLObject,  are available and are listed here. Some discussion of the pysqlite package can be found here. SQLitePP is an object wrapper for SQLite, which includes a Python interface. Some problems and limitations of SQLite. [279]

              back to home