Lone Wolves

#1 Guido van Steen

Wonderful python code! Thanks very much!

#2 coolluck (http://tech.coolluck.org/)

Thank you for your daemon code
I think it'll be better add checking pidfile existance
because atexit handler may delete pidfile before execute os.remove(self.pidfile) in exception Handler "except OSError, err"

...
except OSError, err:
err = str(err)
if err.find("No such process") > 0:
if os.path.exists(self.pidfile):
os.remove(self.pidfile)
...

#3 Sander Marechal (http://www.jejik.com)

Thank you for your suggestion. I have updated the daemon code accordingly.

#4 narthollis

Thanks very much - highly useful bit of code!

And far more elegant than my previous solution.

Many Thanks :)

#5 Anonymous Coward

What is the license on this code? Can I have it under GPL?

#6 Sander Marechal (http://www.jejik.com)

@Anonymous Coward

This is really basic stuff and not very original. It's Public Domain, so do with it as you please.

#7 Jacob Singh (http://pajamadesign.com)

Thanks for the code,

I've got it working, but I have one problem. I'm trying to do some stuff before exiting. So, in my code:

signal.signal(signal.SIGTEM,myhandler)
def myhandler(signum,frame):
#do something
sys.exit()

I assume that my sys.exit is calling itself by raising a SIGTERM somehow which is really annoying. How can I stop this?

Thanks!
Jacob

#8 Sander Marechal (http://www.jejik.com)

@Jacob: Use the atexit function instead. If you look at line 59 you see that I am registering the self.delpid() function to be run when the daemon shuts down. That function will be run by automatically by Python when the program stops.

So, add the stuff you want to do before shutdown to the self.delpid() function.

#9 Anonymous Coward

excellent, thx. i'm running on os x and only got things to work by commenting out this section:
# redirect standard file descriptors
#sys.stdout.flush()
#sys.stderr.flush()
#si = file(self.stdin, 'r')
#so = file(self.stdout, 'a+')
#se = file(self.stderr, 'a+', 0)
#os.dup2(si.fileno(), sys.stdin.fileno())
#os.dup2(so.fileno(), sys.stdout.fileno())
#os.dup2(se.fileno(), sys.stderr.fileno())

any idea?
thx

#10 Sander Marechal (http://www.jejik.com)

Nope, none. The section you commented out should work on MacOSX as well as Linux because both are basically Unix-like POSIX systems. I'm not nearly familiar enough with MacOSX to be able to tell why it doesn't work. Sorry.

#11 Anonymous Coward

k. seems that /dev/null isn't working, at least for my 2.52, os 10.5x config. however, py's os.devnull works like a charm.

#12 Sander Marechal (http://www.jejik.com)

There's no /dev/null on a Mac machine? MacOSX 10.5 claims to be POSIX compliant, and I believe that having /dev/null is required by POSIX.

Can you tell me what os.path.devnull is set to in Python on OSX?

#13 Preston Hunt (http://prestonhunt.com)

A beautiful piece of code, thank you for sharing!

#14 Guilherme Gall

Nice code, very useful. I think it would be useful the creation of a method to show the status of the process/pidfile to allow the identification of a manual kill.

Something like that:

def status(self):
try:
pf = file(self.pidfile, 'r')
pid = int(pf.read().strip())
pf.close()
except IOError:
pid = None

try:
procfile = file("/proc/%d/status" % pid, 'r')
procfile.close()
except IOError:
sys.stdout.write("there is not a process with the PID specified in %s\n" % self.pidfile)
sys.exit(0)
except TypeError:
sys.stdout.write("pidfile %s does not exist\n" % self.pidfile)
sys.exit(0)

sys.stdout.write("the process with the PID %d is running\n" % pid)


#15 Jonathan Manning

For the /dev/null issue, on OS X or Windows, try:

if (hasattr(os, "devnull")):
DEVNULL = os.devnull
else:
DEVNULL = "/dev/null"

Then use DEVNULL instead of the string "/dev/null" in the default arguments.
~J

#16 HosipLan (http://kdyby.org)

Thank you i'm going to use this for my little robot and it helped a lot

#17 Raj (http://rajorshi.net/blog)

Very useful, thanks a lot :)

#18 Anonymous Coward

Hi. Great code, however i have a problem writing to a log file in my run function :

def run(self):
while True:
print "test1\n"
time.sleep(1)

This does not log anything in the log file specified in the constructor of Daemon. If I remove the sleep(1) call, it works. I was thinking of a buffering issue but removing the buffering to stdout in daemon.py did not change anything.

Any idea ?

#19 Sander Marechal (http://www.jejik.com)

You may want to flush stdout before you sleep() using self.stdout.flush().

A far better alternative however is to use the "logging" module instead.

#20 Anonymous Coward

Thanks - sys.stdout.flush() works. self.stdout.flush() does not.
I tried replacing the print statement with logging.info("Test"). This has the same buffer issue as print statement. Can you explain why you redirect in your code stdout if it cannot be used reliably for output ? For example, when the daemon is stopped, there is no flush or close in the stop function and information that was sent to the log file is lost.

#21 Sander Marechal (http://www.jejik.com)

When you use the logging module you should log to a file, not to stdout. A true daemon has no environment. No parent process, no working directory and no stdin, stdout and stderr. That's why I redirect everything to /dev/null and that's why you need to fork() twice.

Using stdout and flush() can sorta work, but you don't have a real daemon anymore. Consider this. Open a terminal, start your program and then close the terminal. What happens to stdout now? It's gone. That's why it needs to be redirected to /dev/null. And that's why all daemons use logfiles or log to the syslog.

#22 Vik

Excellent stuff. Thanks. I have just started learning python & this has helped me a lot.
I have a question, I need to start some scripts and stop them when I want and would also like to know their status, I have attached to this daemon, but then when I am starting the daemon, in ps -aux only the daemon is shown, not that those scripts have started. Or maybe I am going in the wrong way... Can someone help me?

#23 Sander Marechal (http://www.jejik.com)

I m not quite sure what you mean Vik. Is your source code online somewhere? If not, you can post it here (use the pre tags) or post it to a service like pastebin.com.

#24 Roel

I'd like to be able to ask the status of my daemon by issuing 'daemon status'. So I implemented the status method in my subclass of Daemon and added the necessary lines to the starter script. However, I do not get the desired result as the starter script creates a new instance and then call the .status() from that instance instead of querying the status from the running instance. How would you solve this issue? Thanks for your help and the code!

#25 Sander Marechal (http://www.jejik.com)

What status do you want to show? Just running or not running? Just try to read the pidfile and see if that process still exists. For anything above and beyond that you need a way to communicate between the base script and the running instance. So I suggest you take a look at the python library under the inter-process communication section.

#26 Andr

Thank you it works perfect when the class is in the module I use it from...

When I extract this class to a submodule it does not work. It seems that the wrong process is terminated after the first fork...

Here is my modulcode... Help would really be appriciated...

#!/usr/bin/env python

try:
    import os
    import sys
    import time
    import atexit
    from signal import SIGTERM
except StandardError, e:
    import sys
    print "Error while loading libraries: "
    print e
    sys.exit()

class HcpDaemon:
    """
    A generic daemon class.

    Usage: subclass the Daemon class and override the run() method
    """

    def __init__(self, pidfile, stdin='/dev/null', stdout='/dev/null', stderr='/dev/null'):
        self.stdin = stdin
        self.stdout = stdout
        self.stderr = stderr
        self.pidfile = pidfile

    def daemonize(self):
        try:
            pid = os.fork()
            if pid > 0:
                # exit first parent
                sys.exit(0)
        except OSError, e:
            print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
            sys.exit(1)

        # decouple from parent environment
        os.chdir("/")
        os.setsid()
        os.umask(0)

        # do second fork
        try:
            pid = os.fork()
            if pid > 0:
                # exit from second parent, print eventual PID before
                print "Daemon PID %d" % pid
                sys.exit(0)
        except OSError, e:
            print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
            sys.exit(1)

        sys.stdout.flush()
        sys.stderr.flush()
        si = file(self.stdin, 'r')
        so = file(self.stdout, 'a+')
        se = file(self.stderr, 'a+', 0)
        os.dup2(si.fileno(), sys.stdin.fileno())
        os.dup2(so.fileno(), sys.stdout.fileno())
        os.dup2(se.fileno(), sys.stderr.fileno())

        pid = str(os.getpid())
        file(self.pidfile,'w+').write("%s\n" % pid)

        atexit.register(self.delpid)

    def delpid(self):
        os.remove(self.pidfile)

    def stop(self):
        try:
            pf = file(self.pidfile,'r')
            pid = int(pf.read().strip())
            pf.close()
        except IOError:
            pid = None

        if not pid:
            message = "pidfile %s does not exist. Daemon not running?\n"
            sys.stderr.write(message % self.pidfile)
            return # not an error in a restart

        try:
            while 1:
                os.kill(pid, SIGTERM)
                time.sleep(0.1)
        except OSError, err:
            err = str(err)
            if err.find("No such process") > 0:
                if os.path.exists(self.pidfile):
                    os.remove(self.pidfile)
            else:
                print str(err)
                sys.exit(1)

    def restart(self):
        self.stop()
        self.start()

    def start(self):
        try:
            pf = file(self.pidfile,'r')
            pid = int(pf.read().strip())
            pf.close()
        except IOError:
            pid = None

        if pid:
            message = "pidfile %s already exist. Daemon already running?\n"
            sys.stderr.write(message % self.pidfile)
            sys.exit(1)
        self.daemonize()
        self.run()

    def run(self):
        """Override while subclassing"""

#27 Andr

Nevermind

just stupid me forget to import a module required by the overriding run method...

#28 Sander Marechal (http://www.jejik.com)

I wrote a little workaround to help me debug issues like that. I added another option besides start, stop and restart called "foreground". When I use that, it calls daemon.run() instead of daemon.start(). That means the application will not fork to the background but run in the foreground instead. That means you can easily spot errors like this because exceptions get printed to the commandline instead of to a logfile or pipe. It's really useful for debugging :-)

#29 Andre

Thank you, sounds like a good idea ;-)

#30 Brian

Shouldn't your example MyDaemon provide an __init__ routine and call the parent class's __init__?

#31 Robert

Thank you for this code! I'm using it for a python based weather station logger.

#32 Sander Marechal (http://www.jejik.com)

@Brian: No. Because MyDaemon does not implement __init__, the parent's __init__ will be called automatically.

@Robert: That sounds like a great project. I always like hearing where my code ends up :-)

#33 Brian

Ah, yes. I'm still new to Python.

Thanks for the very useful code! I'm using it to implement an IRC bot. Works great.

#34 Vik

I am using a similar daemon and that daemon for me does call quite a lot of functions for me from various scripts. I also have a log file for errors and warnings. After the daemon is up for certain amount of time, I am getting the following error:

close failed: [Errno 10] No child processes

I am not sure as where I am going wrong, when I looked up on google to find out that this might be a bug in python:

http://mail.python.org/pipermail/python-bugs-list/2007-August/039194.html

I wanna keep my logs and get rid of this error, the problem is the error is continuously writing to a file and that way in a certain period of time my hard drive will be full with just one text file.

Please let me know if anyone has ever faced this error. Any help is appreciated.

Thanks
Vik

#35 Anonymous Coward

Very nice code. Here is an updated version for python 3.x series:


"""Generic linux daemon base class for python 3.x."""

import sys, os, time, atexit, signal

class daemon:
"""A generic daemon class.

Usage: subclass the daemon class and override the run() method."""

def __init__(self, pidfile): self.pidfile = pidfile

def daemonize(self):
"""Deamonize class. UNIX double fork mechanism."""

try:
pid = os.fork()
if pid > 0:
# exit first parent
sys.exit(0)
except OSError as err:
sys.stderr.write('fork #1 failed: {0}\n'.format(err))
sys.exit(1)

# decouple from parent environment
os.chdir('/')
os.setsid()
os.umask(0)

# do second fork
try:
pid = os.fork()
if pid > 0:
# exit from second parent
sys.exit(0)
except OSError as err:
sys.stderr.write('fork #2 failed: {0}\n'.format(err))
sys.exit(1)

# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
si = open(os.devnull, 'r')
so = open(os.devnull, 'a+')
se = open(os.devnull, 'a+')
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())

# write pidfile
atexit.register(self.delpid)
pid = str(os.getpid())
with open(self.pidfile,'w+') as f:
f.write(pid + '\n')

def delpid(self):
os.remove(self.pidfile)

def start(self):
"""Start the daemon."""

# Check for a pidfile to see if the daemon already runs
try:
with open(self.pidfile,'r') as pf:
pid = int(pf.read().strip())
except IOError:
pid = None

if pid:
message = "pidfile {0} already exist. " + \
"Daemon already running?\n"
sys.stderr.write(message.format(self.pidfile))
sys.exit(1)

# Start the daemon
self.daemonize()
self.run()

def stop(self):
"""Stop the daemon."""

# Get the pid from the pidfile
try:
with open(self.pidfile,'r') as pf:
pid = int(pf.read().strip())
except IOError:
pid = None

if not pid:
message = "pidfile {0} does not exist. " + \
"Daemon not running?\n"
sys.stderr.write(message.format(self.pidfile))
return # not an error in a restart

# Try killing the daemon process
try:
while 1:
os.kill(pid, signal.SIGTERM)
time.sleep(0.1)
except OSError as err:
e = str(err.args)
if e.find("No such process") > 0:
if os.path.exists(self.pidfile):
os.remove(self.pidfile)
else:
print (str(err.args))
sys.exit(1)

def restart(self):
"""Restart the daemon."""

self.stop()
self.start()

def run(self):
"""You should override this method when you subclass Daemon.

It will be called after the process has been daemonized by
start() or restart()."""


#36 Sander Marechal (http://www.jejik.com)

Vik, I don't think that error has anything to do with the daemon code. According to the post you linked, this is about threading and subprocesses.

I suggest you look at the backtraces for your error and see where they originate. Then simply catch the OSError and act appropriately (i.e. assume the thread has died, clean up and restart it for example).

#37 Sander Marechal (http://www.jejik.com)

@Anonymous: Thanks for the Python 3.x code. I have updated the main article with a link you your implementation.

#38 Vik

hey Sander, Thanks for your help. I shall try that. Might need your help again in case.

#39 Sander Marechal (http://www.jejik.com)

Oops! There were two comments posted here that I accidentally removed as spam (the spam/ham buttons are perhaps a tad too close together on my admin panel). Anyway, I will try to answer them from memory.

The first poster had a problem implementing his own daemon. He was implementing his code after the run() function. The answer is that your code should replace the run() function. That is what overriding a function means. You create your own function that replaces the original. You should make sure that the run() function never exits. That is why the example code has an infinite loop in it. If the run() function ever exits then the daemon stops.

The second poster had a question about the pidfile. IIRC he had a problem under Python 2.4 that the PID of the first fork was written to the pidfile and not the PID from the second fork, the one that keeps running. If I get this wrong, please re-post and I promise I will press the "ham" button this time :-)

Anyway, I have no idea what would cause this. I have tested this daemon code in 2.4, 2.5 and 2.6 and it works fine for me. The code that reads the pid and created the pidfile happens after the second fork. The original process and the first fork should already be dead by the time the pidfile is written.

It may be possible that there is a bug in the 2.4 version of os.getpid() but that is pure speculation. I know of no such bug, but one hypothetical scenario is that the second fork reaches os.getpid() before the first fork has exited completely (remember, forks run in parrallel). You could try a simple time.sleep(1) after the second fork but before calling the os.getpid() function. That sleep will make sure that the first fork is dead by the time the pidfile is created.

#40 Justin

Second poster here.

The issue I was wondering about is that the pid of the daemon is the same as the second parent's pid--that's what's actually running, and that's what's in pidfile. To see what I'm seeing, add 'print pid' after each occurrence of 'if pid > 0:' in daemonize(), just before 'sys.exit()'. If I'm understanding correctly, those should be the pids of the first and second parents, while the second child's pid is obtained further on with a call to 'os.getpid()'. In my environment, python 2.4 on CentOS 5.3, the second child's pid is the same as the second parent's pid.

It's not really a problem, but I'm obviously not getting something here. I thought the process went like this: first parent->fork->first child/second parent->fork->second child. So there should be three pids in all.

The purpose of the second fork is to make sure that the daemon isn't the process group leader of the new session, but if the second fork isn't taking a new pid, then it's still the process group leader, yes?

#41 Sander Marechal (http://www.jejik.com)

Ah, I see now. Your assumptions about pid are not correct.

The fork() command returns 0 in the child and returns the child's pid in the parent. So, the value of pid that you are printing just after 'if pid > 0:' is the pid of the child, not the pid of the parent. You are printing the pid of the first child and second child, not of the parent and first child.

You can see that by leaving those print statements in and adding a 'print os.getpid()' all the way at the beginning of the daemonize() method, before the first fork. Now you should see three different pid's printed. The original parent, the first child and the second child.

#42 Seth

Refined on the Python 3 version a bit. The original version has the attitude of creating a new daemon instance on every call (on stop calls too), which might have some unwanted side effects. I separated the daemon and control logic in different classes:

daemon.py:

"""\
Daemon base and control class.

This file contains the daemon base class for a UNIX daemon and the control
class for it. See test logic at the end of file and test_daemon.py to
understand how to use it.\
"""

import sys, os, time, atexit, signal

# -- generic daemon base class ------------------------------------------ #

class daemon_base:
	"""A generic daemon base class.
	
	Usage: subclass this class and override the run() method.
	"""
	def __init__(self, pidfile, workpath='/'):
		"""Constructor.

		We need the pidfile for the atexit cleanup method.
		The workpath is the path the daemon will operate
		in. Normally this is the root directory, but can be some
		data directory too, just make sure it exists.
		"""
		self.pidfile = pidfile
		self.workpath = workpath
	
	def perror(self, msg, err):
		"""Print error message and exit. (helper method)
		"""
		msg = msg + '\n'
		sys.stderr.write(msg.format(err))
		sys.exit(1)

	def daemonize(self):
		"""Deamonize calss process. (UNIX double fork mechanism).
		"""
		if not os.path.isdir(self.workpath):
			self.perror('workpath does not exist!', '')

		try: # exit first parent process
			pid = os.fork() 
			if pid > 0: sys.exit(0) 
		except OSError as err:
			self.perror('fork #1 failed: {0}', err)
	
		# decouple from parent environment
		try: os.chdir(self.workpath)
		except OSError as err:
			self.perror('path change failed: {0}', err)

		os.setsid() 
		os.umask(0) 
	
		try: # exit from second parent
			pid = os.fork() 
			if pid > 0: sys.exit(0) 
		except OSError as err:
			self.perror('fork #2 failed: {0}', err)
	
		# redirect standard file descriptors
		sys.stdout.flush()
		sys.stderr.flush()
		si = open(os.devnull, 'r')
		so = open(os.devnull, 'a+')
		se = open(os.devnull, 'a+')
		os.dup2(si.fileno(), sys.stdin.fileno())
		os.dup2(so.fileno(), sys.stdout.fileno())
		os.dup2(se.fileno(), sys.stderr.fileno())
	
		# write pidfile
		atexit.register(os.remove, self.pidfile)
		pid = str(os.getpid())
		with open(self.pidfile,'w+') as f:
			f.write(pid + '\n')
		self.run()
	
	def run(self):
		"""Worker method.
		
		It will be called after the process has been daemonized
		by start() or restart(). You'll have to overwrite this
		method with the daemon program logic.
		"""
		while True:
			time.sleep(1)

# -- daemon control class ----------------------------------------------- #

class daemon_ctl:
	"""Control class for a daemon.

	Usage:
	>>>	dc = daemon_ctl(daemon_base, '/tmp/foo.pid')
	>>>	dc.start()

	This class is the control wrapper for the above (daemon_base)
	class. It adds start/stop/restart functionality for it withouth
	creating a new daemon every time.
	"""
	def __init__(self, daemon, pidfile, workdir='/'):
		"""Constructor.

		@param daemon: daemon class (not instance)
		@param pidfile: daemon pid file
		@param workdir: daemon working directory
		"""
		self.daemon = daemon
		self.pidfile = pidfile
		self.workdir = workdir
	
	def start(self):
		"""Start the daemon.
		"""
		try: # check for pidfile to see if the daemon already runs
			with open(self.pidfile, 'r') as pf:
				pid = int(pf.read().strip())
		except IOError: pid = None
	
		if pid:
			message = "pidfile {0} already exist. " + \
					"Daemon already running?\n"
			sys.stderr.write(message.format(self.pidfile))
			sys.exit(1)
		
		# Start the daemon
		d = self.daemon(self.pidfile, self.workdir)
		d.daemonize()

	def stop(self):
		"""Stop the daemon.

		This is purely based on the pidfile / process control
		and does not reference the daemon class directly.
		"""
		try: # get the pid from the pidfile
			with open(self.pidfile,'r') as pf:
				pid = int(pf.read().strip())
		except IOError: pid = None
	
		if not pid:
			message = "pidfile {0} does not exist. " + \
					"Daemon not running?\n"
			sys.stderr.write(message.format(self.pidfile))
			return # not an error in a restart

		try: # try killing the daemon process	
			while 1:
				os.kill(pid, signal.SIGTERM)
				time.sleep(0.1)
		except OSError as err:
			e = str(err.args)
			if e.find("No such process") > 0:
				if os.path.exists(self.pidfile):
					os.remove(self.pidfile)
			else:
				print (str(err.args))
				sys.exit(1)

	def restart(self):
		"""Restart the daemon.
		"""
		self.stop()
		self.start()

# -- test logic --------------------------------------------------------- #

if __name__ == '__main__':
	"""Daemon test logic.

	This logic must be called as seperate executable (i.e. python3
	daemon.py start/stop/restart).	See test_daemon.py for
	implementation.
	"""
	usage = 'Missing parameter, usage of test logic:\n' + \
			' % python3 daemon.py start|restart|stop\n'
	if len(sys.argv) < 2:
		sys.stderr.write(usage)
		sys.exit(2)

	pidfile = '/tmp/test_daemon.pid'
	dc = daemon_ctl(daemon_base, pidfile)

	if sys.argv[1] == 'start':
		dc.start()
	elif sys.argv[1] == 'stop':
		dc.stop()
	elif sys.argv[1] == 'restart':
		dc.restart()

I also wrote an improvised test file for this file (still ugly, but does the job), test_daemon.py:

"""\
Base daemon test file.

This file tests the base daemon (runs it's test logic). The daemon is
started, then stopped, then started then restarted then stopped again.

The output of this file should look like this (with different pids):

starting daemon
listing processes
15554 ?        S      0:00 python3 daemon.py start
-rw-rw-rw- 1 seth seth 6 2009-06-13 14:55 /tmp/test_daemon.pid
stopping daemon
listing processes
ls: cannot access /tmp/test_daemon.pid: No such file or directory
starting daemon
listing processes
15572 ?        S      0:00 python3 daemon.py start
-rw-rw-rw- 1 seth seth 6 2009-06-13 14:55 /tmp/test_daemon.pid
restarting daemon
listing processes
15582 ?        S      0:00 python3 daemon.py restart
-rw-rw-rw- 1 seth seth 6 2009-06-13 14:55 /tmp/test_daemon.pid
stopping daemon
listing processes
ls: cannot access /tmp/test_daemon.pid: No such file or directory
"""

import os, time

if __name__ == '__main__':
	start_cmd = 'python3 daemon.py start'
	stop_cmd = 'python3 daemon.py stop'
	restart_cmd = 'python3 daemon.py restart'

	os.chdir(os.path.abspath(os.path.dirname(__file__)))
	
	print('starting daemon')
	os.system(start_cmd)
	time.sleep(0.2)
	print('listing processes')
	os.system('ps x | grep "python3 daemon.py" | grep -v grep')
	os.system('ls -l /tmp/test_daemon.pid')

	print('stopping daemon')
	os.system(stop_cmd)
	time.sleep(0.2)
	print('listing processes')
	os.system('ps x | grep "python3 daemon.py" | grep -v grep')
	os.system('ls -l /tmp/test_daemon.pid')
	
	print('starting daemon')
	os.system(start_cmd)
	time.sleep(0.2)
	print('listing processes')
	os.system('ps x | grep "python3 daemon.py" | grep -v grep')
	os.system('ls -l /tmp/test_daemon.pid')
	print('restarting daemon')
	os.system(restart_cmd)
	time.sleep(0.2)
	print('listing processes')
	os.system('ps x | grep "python3 daemon.py" | grep -v grep')
	os.system('ls -l /tmp/test_daemon.pid')
	print('stopping daemon')
	os.system(stop_cmd)
	time.sleep(0.2)
	print('listing processes')
	os.system('ps x | grep "python3 daemon.py" | grep -v grep')
	os.system('ls -l /tmp/test_daemon.pid')

#43 Vik

Will this daemon listen on a specific port which I would want it to and what should be done in order to do that? Any help

#44 Sander Marechal (http://www.jejik.com)

Hi Vik. No, this daemon does not listen to TCP. It's just a daemon. It runs in the background doing whatever you tell it to do. If you want a Python daemon that communicated over TCP then I highly recommend checking out the Twisted framework instead of this daemon.

#45 Vik

Hi Sander, Thanks for your help. I appreciate that.

#46 acid (http://blutrache.blogspot.com)

hi the next part never execute I think must be into a second fork in the if pid > 0: statement

# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
si = file(self.stdin, 'r')
so = file(self.stdout, 'a+')
se = file(self.stderr, 'a+', 0)
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())

# write pidfile
atexit.register(self.delpid)
pid = str(os.getpid())
file(self.pidfile,'w+').write("%s\n" % pid)

#47 Sander Marechal (http://www.jejik.com)

Acid: Are you sure it never executes? It is also possible that your daemon is exiting immediately after starting. In that case it looks like it is never starting because there is no output and no pidfile (the pidfile is removed when the program exits).

#48 acid (http://blutrache.blogspot.com)

:( I wasn't doing importing the atexit package thousand apologies

#49 Sander Marechal (http://www.jejik.com)

No problem. I'm glad you got it fixed :-)

#50 Doug

The daemon is not running self.delpid() at exit, take a look at: http://docs.python.org/library/atexit.html

"Note: the functions registered via this module are not called when the program is killed by a signal, when a Python fatal internal error is detected, or when os._exit() is called."

If you step through the code you'll see that the function is never called.

So, is there any way to get some cleanup function to run when Daemon.stop() is called?

#51 Sander Marechal (http://www.jejik.com)

I'll have to look into that, because the pidfile is removed on my system when I call the stop() function.

Anyway, one workaround would be to trap the SIGTERM in the daemon and call a clean sys.exit() so that the atexit function does work.

#52 Doug

It's removed on my system too, but not from delpid(). When you call stop(), the daemon calls 'os.kill(pid, SIGTERM)' in the try statement, which throws an exception when it's called a second time (since it's in an infinite while loop and the process gets killed the first time). Then 'os.remove(self.pidfile)' takes care of the 'pid' file and delpid() is never touched.


try:
while 1:
os.kill(pid, SIGTERM)
time.sleep(0.1)
except OSError, err:
err = str(err)
if err.find("No such process") > 0:
os.remove(self.pidfile)
else:
print str(err)
sys.exit(1)


Can you give an example of how to trap the SIGTERM? I can't seem to get it to work. Thanks for this writeup by the way, I use it all the time!

#53 Doug

Nevermind, figured it out ;)

#54 Chenix

I'm new to linux, and was wondering how daemons gets their permissions, specifically disk access permissions.
1. does a daemon needs to have a user attached to it in some way, so it gets the user's
access policy for itself too?
2. what would be a good practice for a disk location of application files designed to
run as a daemon? (/usr/local maybe?)
3. putting all this together, lets say my python application files and some data files
are located under /usr/local/myapp, what steps do i have to take in order that my daemon
will be able to access this location?

thanks

#55 Sander Marechal (http://www.jejik.com)

All applications, including daemons, run as a certain user. Often a daemon is started as root and then executes some special code to change itself to a different user. The Python daemon I wrote does not have such code. It simply inherits from the user who started it.

A daemon is no different from a regular application. Application files should be in the same place (/usr or /usr/local). Privileges work the same. The only thing that is different about a daemon is that you get your terminal back after starting it :-)

#56 Mike

Thanx alot for the script, it's incredibly useful! I'm using it for a project using MPD (music player daemon) and the serial port. Basically a microcontroller will tell the daemon to change song and such over the serial port.

So I got the script working and all, after starting the daemon I can see (and hear) that the song is playing. The problem is that it doesn't stop playing when I stop the daemon. I modified the delpid() method:


...
def delpid(self):
os.system('mpc stop')
os.remove(self.pidfile)
...


but somehow the command "os.system('mpc stop')" doesn't run. The "mpc" manual is very simple and I know that "mpc stop" works fine outside of the script. Any idea what I might be doing wrong? The script runs as root and everything else works perfectly (up to now :)

#57 Sander Marechal (http://www.jejik.com)

Mike, I think the problem is that a daemon does not have an environment anymore. So executing commands with os.system() does not work. There is no shell, to environmnet, no input/output streams.

Instead, try sending a signal to MPD. You need to know it's process ID (pid) though. Example:

# Tell MPD to terminate
os.kill (mpd_pid, signal.SIGTERM)
# wait a few seconds for MPD to finish
time.sleep(3)
# If MPD still lives, kill it the hard way
try:
    os.kill (mpd_pid, signal.SIGKILL)
except OSError:
    pass # SIGTERM already shut down MPD

#58 Mike

Thanks for the quick reply. I use os.system() to start, suffle, play, etc the songs with "mpc command" so I know it works, exept when I call it from delpid(). Also I just want to stop MPC (music player client) from playing, not to kill MPD (... daemon).

Hum the indentation didn't come right on my last post.

#59 Sander Marechal (http://www.jejik.com)

Oh wait. Duh. I see it now. Have a look at comment #50 above. The delpid() function isn't called because stop() uses os.kill() to terminate the daemon. Functions registered with atexit() are not called when a program is terminated with os.kill().

If you are terminating the daemon from within (e.g. based on serial port communication) you could use sys.exit() instead of Daemon.stop(). That way the atexit() will fire and execute delpid().

#60 Mike

Ah I missed #50, thanx alot! It's working now!

#61 Seth

Happened to come across the atexit issue too. The way to make it work correctly is to set up a signal handler while daemonizing:

import signal

[...]

def cleanup_handler(signum, frame):
	os.remove(pidfile)

signal.signal(signal.SIGTERM, cleanup_handler)

#62 Sander Marechal (http://www.jejik.com)

Perfect. Thanks!

#63 stelvis

firstly a huge thanks for this - it has really helped me out (I'm building a small server that will receive filepaths over TCP and automatically build symlinks to the files - its part of an asset management and versioning system where the file server is running linux but the files will be accessed via symlinks by numerous windows machines on the other side of a samba connection - hence why it has to run server side as a daemon)

couple of questions:

regarding Seth's post on using signal handlers to get cleaner exit behaviour - any chance of any elaboration on how that should work? (sorry if I'm missing the obvious)- eg I assume the def is another method declared in the Daemon parent class, but where exactly is the signal stuff called from? does it replace the stop method?

secondly I have been trying to get logging to work, and seem to have a working model but it ONLY works if i call the run() method directly (I implemented a 'debug' or 'foreground' mode as you suggested above) - if run() is called from start() after self.daemonize() the logging stuff just hangs the daemon (ie no PID file is created, run() isn't called)

code is below:


class SymServer(Daemon):
def run(self):

#logging.config.fileConfig("SymServer_logging.conf")
#logger = logging.getLogger("SymServer")

#create socket

s = socket(AF_INET, SOCK_STREAM) # create a TCP socket
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) # don't wait for port to be releaed
s.bind((myHost, myPort)) # bind it to the server
s.listen(5) # allow 5 simultaneous pending connections
#logger.info("server starting on port: %d" % (myPort))

# listen for clients

while 1:
connection, address = s.accept() # connection is a new
while 1:
data = connection.recv(1024) # receive up to 1K bytes
if data:
connection.send('echo -> ' + data)
#logger.info("echoing: " + data)
else:
#logger.info("Client exiting")
break
connection.close() # close socket


(the actual symlinking stuff will eventually replace the simple echo stuff there currently)

#64 stelvis

ok, dumped the logging.config stuff and the logging code works fine

:)

#65 Sander Marechal (http://www.jejik.com)

Stelvis: The problem probably is that you don't have the correct permission to write the logfile. Remember that when the daemon starts it detaches from it's environment and changes it's working directory to the root /. So, if you're trying to write your symserver.log it will try to write it to /symserver.log and not to /home/stelvis/symserver/symserver.log (or wherever your code is located). You probably don't have rights to write in /.

#66 stelvis

ahhh

actually the log file was being directed to /home/%user%/python/symserver/symserver.log anyway and I did make sure it had write permissions...as soon as I stopped using a seperate logging.config file it worked fine.

BUT

the logging conf file (which was also in that dir) didn't have an explicit path set (so your right in a way i think) - it just wasn't finding the config file

cheers

#67 Sander Marechal (http://www.jejik.com)

Glad to hear you solved it!

#68 dik123

Thanks for code!!

For me this code:
# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
si = file(self.stdin, 'r')
so = file(self.stdout, 'a+')
se = file(self.stderr, 'a+', 0)
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())

Dose not work for pythons 'print'
But this code works:

# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
sys.stdout = file(self.stdout, 'a+')
sys.stderr = file(self.stderr, 'a+', 0)
si = file(self.stdin, 'r')
os.dup2(si.fileno(), sys.stdin.fileno())

#69 prakash

Firstly thanx to the one who coded this.I tried using this to creat a daemon process.here is my run method

def run(self):
    while True:
        Date=date.today()
        open('/proc/net/dev','r')
        Lines=DevFile.readlines()
        Line1=Lines[3].split(':')
        Line2=Line1[1].split()
        Ty=Date.strftime("%y")
        Tm=Date.strftime("%m")
        Td=Date.strftime("%d")
        TH=time.strftime("%H")
        TM=time.strftime("%M")
        TS=time.strftime("%S")
        #L=open("log.txt",'a')
        #L.write("in run of my daemon")
        path=Ty+"/"+Tm+"/"+Td+"/"+TH+"/"+TM
        Ap=open(path,"a")   
        Ap.write(Ty+":"+Tm+":"+Td+":"+TH+":"+TM+":"+TS+":"+
                 Line2[0]+":"+Line2[8]+"\n")
        Ap.close()      
        time.sleep(1)

and my code is giving me trouble.This process is not starting(i checked using ps) .What i observed is if i remove all those file descriptors the code works.But it should work with those .Also i made sure that the directory structure is correct.so can someone plz sort this out

#70 Sander Marechal (http://www.jejik.com)

prakash: What happens if you call daemon.run() instead of daemon.start()? When you do that, you essentially run your code in the foreground instead of as a daemon. Does it work then without throwing any exceptions?

#71 Peter (http://sourceforge.net/projects/cozy/)

Hi Sanders, also from me a big thank you for this code! I'm using it in my backup program.

Following up on the discussion on exiting the daemon, I wondered which signal it receives if I log out. It seems like it's not a SIGTERM. I tried the SIGHUP, but it doesn't work either. So whenever I log out and log in (it's in the autostart), it doesn't start because the pid-file already exists. Note that everything's fine if I restart the complete machine. Any hints?

#72 solarn

Thanks! Great piece of code, I love it! But perhaps it would be good to update the code and include the signal handling stuff Seth shared with us in post #61, ie:

signal.signal(signal.SIGTERM, self.delpid)
atexit.register(self.delpid)
[....]
def delpid(self):
os.remove(self.pidfile)
self.cleanup()

I also added a call to self.cleanup(), to simplify customisations even more by allowing the user to override cleanup(), without having to touch delpid() (and perhaps the risk of forgetting to remove the pidfile?)

#73 solarn

Oh yeah, while I'm at it, perhaps add that foreground() method from #28, so the people who don't browse through the comments will not miss the, in my opinion, important changes? (nearly happened for me, heh..)

#74 solarn

(Arghh, I should stop spamming already)
Smal misstake in prev. post:
signal.signal(signal.SIGTERM, self.delpid)
atexit.register(self.delpid)
should instead be:
def cleanup_handler(signum, frame):
sys.exit(0)
signal.signal(signal.SIGTERM, cleanup_handler)
atexit.register(self.delpid)
Now, whenever a SIGTERM is caught, sys.exit() is called and atexit should kick in and cleanup.

#75 Sander Marechal (http://www.jejik.com)

@Peter (Comment #71): I'm not sure but I would not be surprised if you get a SIGKILL. Unfortunately you can't trap a SIGKILL.

@Solarn: I'll see if I have time to update the article in the coming few days. I've just moved so I have been a bit busy paiting, redecorating, shopping for furniture and yelling at my ISP for the low quality ADSL connection I get at my new house.

#76 eromirou

Hi Sander,
first of all, thanks a lot for this code, it was really helpful.

I'm having a weird problem...
I'm using '/var/log/mydaemon.log' and '/var/log/mydaemon.err' as stdout and stderr respectively, but I'm not getting anything on those files unless I print something before redirecting the output...
When I print something before redirecting, it goes to the terminal, the daemon goes in the background, and then anything that's printed during execution goes into the files I stated.

Any ideas on what might be causing this behaviour?

Thanks

#77 Girish Venkatachalam (http://spam-cheetah.com)

Brilliant page design and equally brilliant code.

Can you please explain the need for the Pid file and sleep(2)?

Without that it would be perfect I think.

Great work. Keep going!

Ever yours,
Girish

#78 Sander Marechal (http://www.jejik.com)

@eromirou: You could try Dik123's modifications from comment #68.

@Girish: After the daemon has started you still need to communicate with it from the outside. For example, in order to stop it. To do that you need to know the process ID, which is stored in the pidfile. The pidfile also acts as a lockfile, preventing multiple instances of the daemon running.

I don't see any sleep(2) in the code. Which sleep do you mean?

#79 eromirou

Thanks for your response. The proposed solution did not work for me though, but I found a fix for it.
I made stdout unbuffered like so:

sys.stdout = file(self.stdout, 'a+', 0)

Do you see any problems with this?

Thanks

#80 Sander Marechal (http://www.jejik.com)

I don't see any problem with that directly. The stderr is also unbuffered. You could probably still use buffered but you should flush it regularly to get the output in the file.

#81 Monte Davis

Hey --

I'm using this code to daemonize a number of different business processes that we have. It works very well and very cleanly. I have however run into a little problem.

I want to start and stop the daemon from a web-interface using django. I save a django object and on save I run the command(python daemon-file-name start or python daemon-file-name stop accordingly).

The stop works fine. However, the start results in the web page never "hanging". The process actually starts but I find it listening on port 80.

I have found in a forum someone who had a similar issue in Python Win32. The link is here -

http://mail.python.org/pipermail/python-win32/2005-May/003310.html

It seems that I need to close stdout in order to return control to the webserver and not inherit the handles of the process that launches the daemon.

This is consistent with some other comments that I have found on the web.

In this other daemon code,
http://code.activestate.com/recipes/278731/

os.close(1) and os.close(2) are called as are the sys.stdout.close()

Here is another incomplete post that deals with similar issues --

http://code.activestate.com/recipes/186101/

I have tried to add these lines in a number of places in your code but nothing works. Either the daemon doesn't start from the command line or the daemon does not release when called from the webserver.

Any help here would be very appreciated.

Thanks
Monte

#82 Sander Marechal (http://www.jejik.com)

Hi Monte. I think you not only need to close stdout and stderr, but also stdin. I would add it to the daemonize() function. Replace the part that duplicates the file descriptors with this:

# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()

# Close the old descriptors
sys.stdin.close()
sys.stdout.close()
sys.stderr.close()
os.close(0)
os.close(1)
os.close(2)

# Create the new file descriptors and attach them
si = file(self.stdin, 'r')
so = file(self.stdout, 'a+')
se = file(self.stderr, 'a+', 0)
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())

Note: I have not tested the above code. I don't know Django. If this does not work, please tell me how Django runs external scripts. Does it use subprocess.Popen?

#83 Megaman821

Why write your own daemon function instead of using something like, http://pypi.python.org/pypi/python-daemon/ ?

#84 Sander Marechal (http://www.jejik.com)

Customisability? Also, this article is almost two years older than that library and the PEP that goes along with it.

#85 Adam Collard

You should check out http://daemonize.sourceforge.net/ too

#86 CMFL

Appologies if this has already been answered but I read above where someone wanted to see an example of signal handling. Here is an example of a two classes SignalHandler and SigAction that uses polymorphism to handle trapped signals. This could be easily implemented in the Daemon class that Sander has wonderfully demonstrated. I used this in my Daemon class.

This was written in python 2.6. I am not sure if it will work with python 3.

To use this, you can either fork this script to the background or open another terminal and send the signals. The defined signals are SIGTERM (kill -15) and SIGINT (kill -2) -> CTRL-C. If you type CTRL-C on your keyboard and this script is not forked to the backgournd you will see that the class handled the SIGINT signal correctly.

#!/usr/bin/python                                  
#############################################################
# Script: sighandler.py                                      
# Description:                                               
#   Demonstrate generic polymorphism example of how to setup and
#   handle implemented signals through python.                  
#                                                               
# Author: Cody Lane                                             
# Date: 12-4-2009                                               
#############################################################   

import signal
import sys   
import os    

class SignalHandler:
    '''             
    This class is used to handle and trap a signal and associate
    it back to a python callable object. When a new SignalHandler
    object is created you must call the register() method to register
    the signal and callback.  See register() for details.            

    This superclass has an abstract method called 'handler' which
    should be defined in a sub-class.                            

    This class is a super class for SigAction. SigAction is the 
    class that performs the signal event handling.              
    '''                                                         
    SIGNALS = ()                                                
                                                                
    def register(self, signum, callback):                       
        '''                                                     
        Registers a new signal handler action object (SigAction) 
        for trapping a signal and then calling a python object.  

        When this method is called it also updates the static SIGNALS
        tuple appending the new SigAction class.  To get this tuple  
        see getActions().                                            

        @param signum: int -> Should be signal int.  Ex: signal.SIGTERM
        @param callback: pythonObj -> Should be python object.  Ex: somemethod
            NOTE: Don't put () around your callback function or the function  
                  will be called when the callback is initialized..           
        '''                                                                   
        self.SIGNALS += (SigAction(signum, callback), )                       

    def getActions(self):
        '''              
        Return a tuple of defined signal actions if any.  If defined, 
        the tuple will contain SigAction instances.                   
        '''                                                           
        return self.SIGNALS                                           

    def handler(self, signum, frame):
        '''                          
        Abstract method, which must be defined when sub-classing.
        If the method is not defined, an AssertionError will be raised.
        '''                                                            
        assert 0, "You must define a handler(signum, frame) method in %s" %(self)

    def __repr__(self):
        '''            
        Custom string method when printing the class object.  
            Example:                                          
                >>> print self                                
                <Class: SignalHandler>                        
        '''                                                   
        return "<Class:%s>" %(self.__class__.__name__)        

class SigAction(SignalHandler):
    '''                        
    This class defines how a signal should be traped and what do when
    a signal trap occurs.  This class should NOT be instantiated because
    the superclass SignalHandler will setup and create a SigAction object
    when the SignalHandler.register() method is called.                  

    '''
    def __init__(self, signum, callback):
        '''                              
        Create a SigAction object.  Also setup the signal trap.

        @param signum: int -> Should be signal int.  Ex: signal.SIGTERM
        @param callback: pythonObj -> Should be python object.  Ex: somemethod
            NOTE: Don't put () around your callback function or the function  
                  will be called when the callback is initialized..           
        '''                                                                   
        self.signum = signum                                                  
        self.callback = callback                                              
        signal.signal(self.signum, self.handler)                              

    def handler(self, signum, frame):
        '''                          
        Overrides the superclass definition so that each new signal performs
        it's own action.                                                    

        NOTE: You do not have to pass arguments to this method. See help(signal)
        for details.                                                            
        '''
        self.callback()

    def __repr__(self):
        '''
        Custom string message when class object is printed.

        Example:
            >>> print self
            <Class:SigAction signal:15>
        '''
        return "<Class:%s signal:%s>" %(self.__class__.__name__, self.signum)

def stop():
    print "Called the stop().... Exitting 0"
    sys.exit(0)

def abort():
    print "Called abort().... Exitting 1"
    sys.exit(1)

if __name__ == '__main__':
    # Create SignalHandler object
    sighandler = SignalHandler()

    # Now register a SIGTERM (kill -15) and callback
    # object when signal is trapped.
    sighandler.register( signal.SIGTERM, stop)

    # Register a SIGINT (kill -2) and callback
    # object when sign is trapped.
    # This is equivalent to CTRL-C
    sighandler.register( signal.SIGINT, abort )

    # Display the current registered signal events.
    # This is just for show and tell purposes.
    print "Implemented Signals: ", sighandler.getActions()

    while True:
        print "Waiting for signal.... Running pid: %s" %(os.getpid())
        signal.pause()

Here is the example not forking script to the background and issuing a CTRL-C in the terminal the script was run from.

$ ./sighandler.py
Implemented Signals:  (<Class:SigAction signal:15>, <Class:SigAction signal:2>)
Waiting for signal.... Running pid: 5276
^CCalled abort().... Exitting 1

Here is an example of the script being forked to the background, and then sending a kill command to the running proccess.

$ ./sighandler.py &
[1] 5277
Implemented Signals:  (<Class:SigAction signal:15>, <Class:SigAction signal:2>)
Waiting for signal.... Running pid: 5277

$ kill 5277
Called the stop().... Exitting 0
$

#87 Sander Marechal (http://www.jejik.com)

@CMFL: That looks very nice. Thank you!

#88 CMFL

Thanks Sander for demonstrating this, there is some really neat informatoin in this thread. I find myself visiting this thread often to review other user's comments. Perhaps I should register. :) I hope the example I provided helps out others as much as the information contained here.

#89 Anonymous Coward

I am a relatively newbie. Sorry for posting windows questions in a unix spot. I have a .py file that seems to be exactly what I need except I need to have it started on a shared network drive on NTFS and then just have it run indefinitely. I would prefer not to have the server admin start it. Nothing malicious..just our IT dept may never get to it. I need to avoid crashing any machines. (the .py file polls a folder for new .csv files and then reformats them and generates an email alert) I would want it to run.

#90 Sander Marechal (http://www.jejik.com)

I have no idea how to do that Anonymous. Daemons like this don't work on Windows.

#91 Batista

Really slick python coding there! This will be a huge help to one of my projects, thank you for sharing!

#92 Alfredo

Very useful snippet! Thanks!

#93 Josh R

Nice code. Should come handy. Thank a lot!

#94 Anonymous Coward

Very nice! I'm trying to writing a scipt like this.