Status of the OpenStack port to Python 3

Python 3 has been around for about 5 years, and we have excellent reasons to make sure OpenStack runs well on it. Unfortunately, this is not the case. In this article, we’ll see what works, what doesn’t, and what you can do to help. Note that we are targetting the latest released version, Python 3.3.

Python Clients

OpenStack provides a REST API that allows users to write applications using the language of their choice; it also comes with Python applications. Unfortunately, not all of them work with Python 3. However, they are usually quite small, and should not be too hard to port.
The Keystone client is used by many other clients, so it had to be ported first; we will first see explain how it was ported to Python 3, then we will talk about all the other Python clients.

The Keystone client

The main issue with the Keystone client was HTTPretty, an HTTP client mocking tool for Python that was not working with Python 3. I ported it and released the 0.8.0 version in early February, and it is now used in the Keystone client. Thanks to Gabriel Falc√£o, the author of HTTPretty, for his support!

Other than that, the keystone client suffered from the usual issues that arise
when porting software to Python 3:

  • API changes in the standard library (ex: httplib renamed to http.client);
  • API changes in the third-party libraries used;
  • text string versus bytes issues: in Python 2, ‘foo’ was a byte type, but in Python 3, it’s a unicode text string.

All in all, around 20 patches were needed to make sure that version 0.6.0, released on February 13th, works with Python 3. Go out and test it! Thanks to the Keystone core developers for their help!

Other clients

Our main goal is to enable the tests on Python 3.3 (enable the ‘py33′ gate and make it vote) on as many clients as possible. Congrats to the 8 (out of 16) clients for which this is already the case:

  • Ceilometer
  • Cinder
  • Ironic
  • Keystone
  • Marconi
  • Nova
  • Tuskar
  • Trove

Most of the other clients have the py33 gate enabled, but keep them as ‘non-voting’. The Savanna client even has tests working on Python 3, but as they do not consider them to cover enough of the code, they chose not to make the py33 gate vote. The OpenStack client will only work if it uses the master branch of the Glance client. Other clients still suffer from a lot of issues and are being worked on.

Server-side work

On the server-side, things are more complicated. Most pieces of software depend on eventlet, which only works with Python 2; they also have quite a lot of other dependencies preventing them for moving to Python 3. For instance, here are some blockers for ceilometer, cinder, glance, heat, horizon, keystone, neutron, nova and swift.

  • eventlet is blocking 9 projects (ceilometer, cinder, glance, heat, horizon, keystone, neutron, nova, swift)
  • oslo.config is blocking 7 projects (ceilometer, cinder, glance, heat, keystone, neutron, nova)
  • sqlalchemy-migrate is blocking 6 projects (ceilometer, cinder, glance, heat, keystone, nova)
  • python-swiftclient is blocking 5 projects (ceilometer, cinder, glance, heat, horizon)
  • paste is blocking 5 projects (cinder, glance, keystone, neutron, nova)
  • python-glanceclient is blocking 4 projects (ceilometer, cinder, horizon, nova)
  • python-neutronclient is blocking 4 projects (heat, horizon, neutron, nova)
  • python-cinderclient is blocking 4 projects (glance, heat, horizon, nova)
  • oslo.rootwrap is blocking 3 projects (cinder, neutron, nova)
  • ecdsa is blocking 3 projects (cinder, heat, nova)
  • suds is blocking 3 projects (cinder, glance, nova)
  • oslo.messaging is blocking 3 projects (glance, keystone, nova)
  • python-ceilometerclient is blocking 3 projects (ceilometer, heat, horizon)
  • pycadf is blocking 2 projects (keystone, nova)
  • python-heatclient is blocking 2 projects (heat, horizon)
  • python-troveclient is blocking 2 projects (heat, horizon)
  • boto is blocking 2 projects (glance, nova)
  • rtslib-fb is blocking 1 project (cinder)
  • django-openstack-auth is blocking 1 project (horizon)
  • pam is blocking 1 project (keystone)
  • taskflow is blocking 1 project (cinder)
  • qpid-python is blocking 1 project (heat)
  • dnspython is blocking 1 project (swift)
  • django-compressor is blocking 1 project (horizon)
  • thrift is blocking 1 project (ceilometer)
  • jsonrpclib is blocking 1 project (neutron)
  • netifaces is blocking 1 project (swift)
  • mysql-python is blocking 1 project (ceilometer)
  • websockify is blocking 1 project (nova)

This list was generated using the Python classifiers on PyPI, so some packages might be listed as blocking even though they perfectly work with Python 3. Note that the main blocking point is the use of eventlet: a possible solution would be to use asyncio.

How you can help

Half the clients now work with Python 3, and we now have a better idea of what’s preventing the rest of OpenStack from moving to Python 3. What can you do to help fix what doesn’t work yet ?

The easiest thing to do is to help us identify the real issues. To do this, you can use Brett Cannon’s caniusepython3 module by running “./caniusepython3.py -r requirements.txt” in your favorite OpenStack project. Then please update the wiki page that lists the status of the projects. Want to go further ? See whether the dependencies reported by caniusepython3 are really broken on Python 3; their setup.cfg/setup.py file might just be missing the required classifiers. If this is the case, please send a patch upstream to fix that, and send a second patch to caniusepython3 that adds the project to the whitelist.

You can also do reviews of patches that help with Python 3 support; bonus points if you’re a core developer. You may want to take a look at the list of projects that are currently being ported and need reviews

You could also start porting some code! You may want to add your name and a link to your gerrit dashboard in the wiki so that others know what you’re working on. You will encounter these issues:

  • Syntax issues (easy to fix)
  • Incompatible third-party libraries to fix/replace (not so easy to fix, require working with upstream)
  • Mix of string and bytes

We’d love to see you join us!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

  1. Pingback: OpenStack Community Weekly Newsletter (Feb 14 – 28) » The OpenStack Blog

  2. Pingback: Use the new asyncio module and Trollius in OpenStack