The Design of kHTTPd

zuzhihui

The Design of kHTTPd
                                by Alessandro Rubini
Based on last month's article about invoking system calls from
within kernel code, this month's column shows how a complete network
server can be implemented as a kernel thread. The sample code
shown implements the skeleton of a simplified tftp server.
While normally you shouldn't bring user-space processes to kernel
space, there are times when this may be a good choice,
either for performance or for size. The former reason is what led to
kHTTPd, the latter may be relevant to avoid libc altogether in small
embedded systems devoted to a single task.
The discussion in this column is based in the kernel-based web
server released within version 2.4.0-test9 of the kernel, in the
net/khttpd directory. This is the kHTTPd program by
Arjan van de Ven, not the Tux reimplementation by Ingo
Molnar. The latter is both more performant and more complex, and can
be found at http://people.redhat.com/mingo/TUX-patches/.
Features of the Sample Code
Code excerpts included in this column are part of a ktftpd
skeletal module, available from
http://ar.linux.it/docs/khttpd/ktftpd.tar.gz.  The
kernel-space daemon loosely mimics what kHTTPd implements,
but with simplicity as a primary target. The choice to use UDP instead
of TCP is partly for simplicity and partly to avoid replicating too
much of kHTTPd. Serving a file through TCP can be performed
using do_generic_file_read, (``sendfile''), but this is yet
another can of worms.
Our tftp daemon shall be able to serve world readable files
from the /tftp file tree, where /tftp could
also be a symbolic link to another directory.
Even though the tftp protocol supports data transfers
in both directions, the sample daemon will refuse to write to its own
filesystem. Finally, it won't implement any packet retransmission
(contrary to what RFC 783 requires).
The daemon won't keep real logs. It'll merely print a little
information using conventional printk calls.
The structure of the skeletal module has been thought out to be
thouroughly understood in a reasonable time, but has not been
completely implemented. In my opinion, reading this column,
ktftpd, kHTTPd, and the Tux patch (in that
order) makes a good learning path.
Kernel Threads
The first step a programmer must take to run a server in kernel
space is forking a process. To create a new thread, you must
call the function kernel_thread. Since this is usually
performed by the initialization function of a kernel module, the
programmer must also ensure to detach the thread from the context
of the insmod or modprobe command that was
executing the initialization function.
Listing 1 shows how module initialization forks a new thread and
how the thread detaches itself from the user process that forked it.
    /*
     * in init_module(): fork the main thread
     */
    kernel_thread(ktftpd_main, NULL /* no arg */,
                  0 /* no clone flags */);
    /*
     * in ktftpd_main(): detach from the original process
     */
    sprintf(current->comm,"ktftpd-main"); /* comm is 16 bytes */
    lock_kernel();   /* This seems to be required for exit_mm */
    exit_mm(current);
    /* close open files too (stdin/out/err are open) */
    exit_files(current);
In order to handle several clients at the same time, a daemon
usually forks several copies of itself, each of them in charge of a
single connection. This is accomplished by calling
kernel_thread again from within the main loop each time a new
connection is accepted. This time, however, there is no need to do
special processing at the beginning of the thread, since there is no
user process to detach from.
You shouldn't be shy of forking copies of your kernel daemon, as
the resources consumed by each of them are almost negligible when
compared to the cost associated to forking a user-space server.
A kernel thread requires no memory-management overhead: it
only consumes a pair memory pages for the stack and a few
data structures that are replicated for each thread.
To count the number of running threads, an atomic_t
data item is used. I called it DaemonCount, the same
name used by kHTTPd.
Just before unloading the module you'll need to stop all the threads,
as the code their are executing is bound to disappear.
There are several ways to accomplish the task. The kHTTPd server uses
a sysctl entry point
(/proc/sys/net/khttpd/stop), so the user can tell kernel code
to stop server activity before unloading the module (and each thread
increments the module's usage count, to prevent users from unloading
it before the threads are all stopped.
Sysctl is an interesting feature, but would increase the
complexity of the sample ktftpd module.  Who's interested in
sysctl could refer to
http://www.linux.it/~rubini/docs/sysctl/sysctl.html
for more information.  To keep code shorter and simpler I chose a
different approach:
the individual thread doesn't add to the usage count for the module, and
the cleanup function sets a global flag and then waits for all the
threads to terminate.
Listing 2 shows the code that deals with thread termination.
   int ktftpd_shutdown = 0; /* set at unload time */
   DECLARE_WAIT_QUEUE_HEAD(ktftpd_wait_threads);
    /*
     * In the code of each thread, the main loop depends
     * on the value of ktftpd_shutdown
     */
    while (!signal_pending(current) && !ktftpd_shutdown) {
        /* .... */
    }
    /*
     * The following code is part of the cleanup function
     */
    /* tell all threads to quit */
    ktftpd_shutdown = 1;
    /* kill the one listening (it would take too much time to exit) */
    kill_proc(DaemonPid, SIGTERM, 1);
    /* and wait for them to terminate (no signals accepted) */
    wait_event(ktftpd_wait_threads, !atomic_read(&DaemonCount));
Additionally, the user is allowed to terminate each thread by
sending it a signal (as you may have imagined by looking at the
condition around the main loop above).  Trivially, when a signal is
pending the thread exits. This behavior is the same signal handling
implemented in the kernel web server, and boils down to the few lines
of code shown in listing 3. The instructions shown are part of the
initialization code of the main thread. Other threads are created with
the CLONE_SIGHAND flag, so sending a signal to any of
them will kill them all.
    /* Block all signals except SIGKILL, SIGTERM */
    spin_lock_irq(&current->sigmask_lock);
    siginitsetinv(&current->blocked, sigmask(SIGKILL) | sigmask(SIGTERM));
    recalc_sigpending(current);
    spin_unlock_irq(&current->sigmask_lock);
Managing Network Connections
The main task of a kHTTPd (and most similar network
services) consists in transferring data to/from the network and
from/to the local filesystem. While I'm going to describe network
access, I won't discuss filesystem access, as system calls like
open, read and close are accessible from
kernel space and last month we've seen how to invoke them.
As fas as network access is concerned, what a server should
generally do reduces to the following few system calls:
    fd = socket();
    bind(fd); listen(fd);
    while (1) {
        newfd = accept(fd);
        if (fork()) {
            close(newfd);
            /* .... */
            exit();
        } else {
            close(fd);
        }
Performiing the same task from kernel space reduces to similar
code, with fork replaced by kernel_thread. The main
difference lies in that there is no need to use file descriptors because
socket structures can be manipulated directly (thus avoiding the need
to look up the file descriptor table every time).
The file net/khttpd/main.c as found in the kernel source
shows how to handle a TCP session from kernel space. The
implementation for UDP is pretty similar and Listing 4 shows how the
first part of the task (preparing to receive packets) is implemented
in the sample code.
    /* Open and bind a listening socket */
    error = sock_create(PF_INET, SOCK_DGRAM, IPPROTO_UDP, &sock);
    if (error sk->reuse   = 1; ---  needed for multi-thread TCP servers */
    sin.sin_family       = AF_INET;
    sin.sin_addr.s_addr  = INADDR_ANY;
    sin.sin_port         = htons((unsigned short)KTFTPD_PORT);
    error = sock->ops->bind(sock,(struct sockaddr*)&sin,sizeof(sin));
    if (error ops->listen(sock,5);  /* "5" is the standard value */
    if (error
Next, a TCP server would sleep in the accept system call.
But unless you want to run a separate thread for any active network
connection, you'll need to multiplex operation of a single thread
across several file descriptors, by calling either select or
poll.
A kernel-space server shouldn't resort to select nor to
poll, as these calls feature non-negligible
overhead. Instead, what kHTTPd does is performing
non-blocking calls for all pending operation, counting the
successes. If no operation performed was successfull, then the daemon
sleeps for at least one timer tick, giving other processes the option
to run before polling its file descriptors again.
If the service is based on the UDP protocol, the thread will
usually sleep on recvfrom right after bind returns.
This is the what the sample server does. It avoids both
select and a polling loop similar to the one described above
by forking a new thread for each new connection being processed.
Sleeping on a system call invoked from kernel space is not different than
sleeping in user space: the system call handles its own wait queue.
The difference is, as outlined last month, in the need to use
set_fs get_fs if the system call is expected to
read or write a ``user'' buffer.
The kHTTPd daemon uses two functions to receive data. One
is used to collect all the HTTP headers, it is called
DecodeHeader; it uses the MSG_PEEK flag in order
not to flush the input queue until all headers are received.  The
other function is called ReadRest and it uses the
MSG_DONTWAIT flag in order not to block when no data is
there.
The procedure used by ktftpd to receive a data packet is
somehow simpler and uses no socket flags. It will sleep waiting for a
packet until one is received or a signal is caught. It is shown in
Listing 5. It is not as generic as recvfrom is, as it
requires to be using IP addresses.
/*
* This procedure is used as a replacement for recvfrom(). Actually it is
* is based on the one in kHTTPd which in turn is based on sys_recvfrom.
* The iov is passed by the caller since it hosts the peer's address,
* and the buffer is passed by the calles because it can't be global
* (all threads share the same address space)
*/
static inline int ktftpd_recvfrom(struct socket *sock,
                                  struct sockaddr_in *addr,
                                  unsigned char *buf)
{
    struct msghdr msg;
    struct iovec iov;
    int len;
    mm_segment_t oldfs;
    if (sock->sk==NULL) return 0;
    msg.msg_flags = 0;
    msg.msg_name = addr;
    msg.msg_namelen  = sizeof(struct sockaddr_in);
    msg.msg_control = NULL;
    msg.msg_controllen = 0;
    msg.msg_iov         = &iov;
    msg.msg_iovlen = 1;
    msg.msg_iov->iov_base = buf;
    msg.msg_iov->iov_len = PKTSIZE;
    oldfs = get_fs(); set_fs(KERNEL_DS);
    len = sock_recvmsg(sock,&msg,1024,0);
    set_fs(oldfs);
    return len;
}
Sending Packets
The code to transmit packets is similar to the code receiving them. It
exploits sock_sendmsg, which mimics sock_recvmsg.
The main difference is in how blocking is managed. The kernel
thread that pushes data to a TCP socket should prevent to find itself
with partially-written data, as that situation would require extra
data management.
In the implementation of kHTTPd (in the file called
datasending.c), the program requests how much free space
is there in the TCP window associated to the socket, and only
tries to push that many bytes. The following lines represent the
culprit of the code:
    int ReadSize,Space;
    int retval;
    Space = sock_wspace(sock->sk);
    ReadSize = min(4*4096, FileLength - BytesSent);
    ReadSize = min(ReadSize , Space );
    if (ReadSize>0) {
        oldfs = get_fs(); set_fs(KERNEL_DS);
        retval = filp->f_op->read(filp, buf, ReadSize, &filp->f_pos);
        set_fs(oldfs);
        if (retval>0) {
            retval = SendBuffer(sock, buf, (size_t)retval);
            if (retval>0) {
                BytesSent += retval;
            }
        }
    }
With UDP each packet is sent as an individual item, so there is no
need to check sock_wpace. The ktftpd sample daemon
reads 512 bytes at a time from the filesystem and builds a packet
accordingly to the amount read (512 or less). It then waits for the
acknowledgment packet using the same ktftpd_recvfrom function
shown above. A TCP server doesn't deal with acknowledgments since
reliability of the connection is build in the TCP protocol stack.
Alessandro is an independent consultant based in Italy. He learns
programming by reading free software and is usually late with his
deadlines.  He can be reached as rubini@gnu.org.
Thanks to Davide Ciminaghi  and
Andrea Glorioso
for helping revising this article.

               
               
               

本文来自ChinaUnix博客，如果查看原文请点：http://blog.chinaunix.net/u/1938/showart_132447.html