Python-gstreamer, threading and the main loop

by Sander Marechal

This is just a quick heads-up for people who try to use python-gstreamer in a thread. Threading in Python has a few unexpected problems if you're new to it and it took me quite a few hours on Google searching for some scraps of documentation (and even diving into the source) before I got it right.

My jukebox hack that I touched upon in my last post uses threads. The main thread doesn't do much of anything right now except watch two child threads. One thread will hold a simple twisted TCP server to control the jukebox. The other will hold the gstreamer audio player. The trouble started with the gstreamer mainloop. When I started the gobject mainloop below in the player thread, all my other threads froze.

  1. loop = gobject.MainLoop()
  2. loop.run()

I had to reimplement that anyway because the player thread also needs to see occasionally if there are commands from the twisted thread. Here's the new loop:

  1. loop = gobject.MainLoop()
  2. context = loop.get_context()
  3.  
  4. while 1:
  5.         # Handle commands here
  6.         ...
  7.         context.iteration(True)

The result was the same: thread freeze. The True in the above code makes context.iteration() idle until there is work to do. Setting context.iteration() to False makes it return immediately. That stopped the freeze but ramped up my CPU usage to 100%. As it turns out, Python doesn't use Operating System threads but implements it's own threading because of portability, but that threading only works within python. When a thread calls a function that's written in C, such as gobject.context.iteration(), then Python freezes all threads until that function returns (unless that C function is wrapped in Py_BEGIN_ALLOW_THREADS/Py_BEGIN_ALLOW_THREADS statements). This is called Global Interpreter Lock or GIL.

Initially I tried getting rid of the gobject.MainLoop altogether, but in Python you can't have a thread simply idle so I had to have some sort of mainloop. This blog post suggest that any loop will do, even a while-true loop, but everything I came up with ramped my CPU usage to 100%.

I finally found out that Python's gobject module has a function to enable/disable threading when it's calling C functions: gobject.threads_init(). By default this is off so you have to turn it on before iterating the gobject mainloop:

  1. loop = gobject.MainLoop()
  2. gobject.threads_init()
  3. context = loop.get_context()
  4.  
  5. while 1:
  6.         # Handle commands here
  7.         ...
  8.         context.iteration(True)

The above code makes gstreamer play smooth and lets all other threads do what they must. I hope that this saves someone a couple of hours breaking their head over their frozen threads. Happy coding!

Creative Commons Attribution-ShareAlike

Comments

#1 ndr k (http://aendruk.wordpress.com/)

Thank you so much! That was driving me crazy.

#2 Anonymous Coward

Thanks, this was very helpful. Thanks also for writing the steps you took.

#3 Anonymous Coward

Just running "gobject.threads_init()" before "loop.run()" solved my problem.

#4 Anonymous Coward Olivier

absolument genial :-)
I just hope that it is not an unsupported hack that is going to stop working with the next gobject release ..

#5 Sander Marechal (http://www.jejik.com)

It's not an unsupported hack. The article is from January 2007 and there have been quite a few gobject releases in the past two years. You're safe :-)

#6 Pascal Giard (http://fofix.googlecode.com)

Hmmm... while i had a slightly different issue (bus.enable_sync_message_emission() didn't work), using this partially fixes things.

It now does send the sync messages (and my callback gets called) but my python thread gets stuck after ~5 seconds.

I'm working around this very hackishly by NOT using the code you suggest and by looking at the last-message every 41.666 ms :-(

Find the code there: http://tr.im/wvvO
See the vidSetup() and run() methods.

Any comment/suggestion is welcome.

#7 Sander Marechal (http://www.jejik.com)

It's hard to say what is going wrong there Pascal. Fofix is quite complex. If I had to guess then I'd say that the thread that is running the gstreamer video player is using a main loop that is not compatible with the gtk/gobject mainloop. As a result, messages are not handled correctly.

But as I said, that's really a stab in the dark.

#8 splicer

Thanks, this was useful!

#9 Eloff

Thanks, this helped me alot with a dbus loop that used to block the whole application

#10 JoulusnefeSon

Thanks you very much

#11 Not Anonymous Coward

I have been fighiting with threading and gobject problem for 3 days. I wish I found your article earlier.
Immense thanks!

#12 Anonymous Coward

Just for completeness (I found this page while searching for a solution to threading gobject + dbus without blocking) I thought I'd point out this webpage:

http://wiki.python.org/moin/DbusExamples

The first lines show how to enable threading with gobject. Hope that's helpful to anyone else who stumbles across this article.

#13 Anonymous Coward

Your solution looks good, but there are some inaccuracies in this post:

* Python uses OS-level threads. It does not provide its own threading implementation.
* The GIL does not freeze threads in C. It just prevents multiple threads from accessing the Python interpreter simultaneously. A C thread can carry on merrily in the background as long as it doesn't need to touch any Python objects (which would require acquiring the GIL first).

#14 Anonymous Coward

thank u so much man.... if only i saw this earlier i would have saved 42 hours of my time... :)

#15 Flulleycoatty

How I can write PM to other users? Thanx

#16 Sander Marechal (http://www.jejik.com)

This blog does not have PM functionality. If another user has filled in the "website" field then you can click on his name to go to his website and search for contact information there. Otherwise you are out of luck.

Some people leave their e-mail address but I do not disclose e-mail addresses. But if you send me a message (use the contact form) then maybe I can forward the message for you.

#17 Anonymous Coward

Fantastic! thank you so much!!!

#18 Anonymous Coward

This was very helpful, spent an hour diagnosing my app only to find out gobject had killed the other threads. Thanks!

#19 Gabriele Barchiesi

ATTENTION! gobject.threads_init() must be called before gobject.MainLoop(), in particular gstreamer pipelines it can causes Segmentation Fault or Garbage Collector problems. So:

gobject.threads_init()
loop = gobject.MainLoop()
context = loop.get_context()
while True:
context.iteration(True)

#20 Rob van den Bogaard

Thanks a lot, it makes my headless Radiotray actually parse playlists and play a radio stream! (Had to rip out all gtk/dbus stuff and forge a main loop.)

#21 Matt (http://freedomboxfoundation.org)

Dude, thank you so much! I'm doing exactly the same thing (twisted server + D-Bus calls using Glib main loop) and was seeing same problem. I googled and your post was the first thing to come up.
Post a new comment

Registration is not required to post comments, but cookies must be enabled. One of the advantages of registration is that you can edit your comments later on (editing not yet implemented). You can register or login here.




Your e-mail address will not be published, but your website URL will. All links that you post will tagged rel="nofollow" to throw off spammers. You are allowed to use the following XHTML tags in your comment: <em> <strong> <u> <b> <i> <strike> <blockquote> <big> <small> <ul> <ol> <li> <a href=""> <pre> <code> <tt> <br>. Please allow up to 60 second processing time after you post a comment. Our spam filters may take some time.