about linux scsi module error handling

ggggchen

                                                                               
此文直接由WORD贴过来.若想格式和图片效果更好,请下载PDF(附在此文最下面).( 此站第一篇文章,heh )
About Linux module



      

1 Linux error handling thread
When
can work

Linux have a error handling thread called
scsi_error_handler, it will call eh_strategy_handler function pointer if have
otherwise it call scsi_unjam_host to do error handling.

flow 1
scsi_error_handler
        |
        |
        |---------à  eh_strategy_handler  or
                  
scsi_unjam_host


The below is an important function
scsi_error_handler that handle error handling, the marked red part shows some
policy related to error handling

When error handling start
  
1)      host_failed == host_busy and host_failed != 0   or
2)      host_failed == host_busy and host_eh_scheduled != 0.


The above two conditions meet any one error
handling can start. host_failed means the failed scmd request count, the
host_busy means working request count. So the necessary condition is all
working requests failed it can start error handling.  The other two is if host_failed is not equal
0 or host_eh_scheduled is not 0.

host_eh_scheduled is a variable that to let
driver to motivate scsi_error_handler by call function scsi_schedule_eh.  When user call scsi_scheduler_eh function the
host_eh_sceduled would increase one. So the aim it to let scsi_error_handler
thread can run if failed request number equal to busy request number.

When doing recovery the normal command also
can’t be send to LLDD from mid layer. scsi_request_fn function would check the
shost status. when doing recovery the host status set to
1)            
SHOST_RECOVERY
2)            
SHOST_CANCEL_RECOVERY
3)            
SHOST_DEL_RECOVERY


function scsi_error_handler 1
int
scsi_error_handler(void *data)
{
       struct Scsi_Host *shost = data;

       /*
        *
We use TASK_INTERRUPTIBLE so that the thread is not
        *
counted against the load average as a running process.
        *
We never actually get interrupted because kthread_run
        *
disables singal delivery for the created thread.
        */
       set_current_state(TASK_INTERRUPTIBLE);
       while (!kthread_should_stop()) {
              if
((shost->host_failed == 0 && shost->host_eh_scheduled == 0) ||
                  shost->host_failed !=
shost->host_busy) {
                     SCSI_LOG_ERROR_RECOVERY(1,
                            printk("Error
handler scsi_eh_%d sleeping\n",
                                   shost->host_no));
                     schedule();
                     set_current_state(TASK_INTERRUPTIBLE);
                     continue;
              }

              __set_current_state(TASK_RUNNING);
              SCSI_LOG_ERROR_RECOVERY(1,
                     printk("Error handler
scsi_eh_%d waking up\n",
                            shost->host_no));

              /*
               * We have a host that is failing for some
reason.  Figure out
               * what we need to do to get it up and online
again (if we can).
               * If we fail, we end up taking the thing
offline.
               */
              if
(shost->transportt->eh_strategy_handler)
                     shost->transportt->eh_strategy_handler(shost);
              else
                     scsi_unjam_host(shost);

              /*
               * Note - if the above fails completely, the
action is to take
               * individual devices offline and flush the
queue of any
               * outstanding requests that may have been
pending.  When we
               * restart, we restart any I/O to any other
devices on the bus
               * which are still online.
               */
              scsi_restart_operations(shost);
              set_current_state(TASK_INTERRUPTIBLE);
       }
       __set_current_state(TASK_RUNNING);

       SCSI_LOG_ERROR_RECOVERY(1,
              printk("Error handler
scsi_eh_%d exiting\n", shost->host_no));
       shost->ehandler = NULL;
       return 0;
}

What
to do
The linux scsi error handler thread just
call a function pointer eh_strategy_handler or scsi_unjam_host. The
eh_strategy_handler function pointer common not implement by LLDD. So the
scsi_unjam_host stand out to do the job. This function is important so we would
talk about it later.

Because when doing failed request recovery
the normal operation is blocked so after doing recovery need to restart the
operation so function scsi_restart_operation function called.


scsi_unjam_host
In Linux document have a detail description
for this function. The link is

./Document/scsi/scsi_eh.txt

Below I also refer the function implement
and the kernel document to describe the function. I think it is very detail. I
don’t want to talk about the thing that the document already talked. However
there also may something else to talk.

In host template we have some function
pointer to implement that related to error handling. It includes
1)     
int (* eh_abort_handler)(struct
scsi_cmnd *);
2)     
int (*
eh_device_reset_handler)(struct scsi_cmnd *);
3)     
int (*
eh_bus_reset_handler)(struct scsi_cmnd *);
4)     
int (*
eh_host_reset_handler)(struct scsi_cmnd *);

These 4 function is different level error
handling from easy to complex. When doing error handling while no normal
request handling, it tries below layer error handler, if it can succeed handle
all error request no need more level handler stand out. And error handling can
finish. Otherwise it need to using higher level error handling function to
handle.

Here we should follow the function. Simply
saying that there are two type error request.
1)      
time out request
2)      
no succeed handle by LLDD
request but returned with error sense information.
For different type error request, though it
all need be handled in scsi_unjam_host function, however there error mark is
different so there error handle way is different.

The major different related mark including
1)       SCSI_EH_CANCEL_CMD.
2)       SENSE error request.
The SCSI_EH_CANCEL_CMD mark now is only add
to time out request. The sense error request is related the scmd that return
with not correct sense.

Below would talked about that under
paragraph. First let’s make sure on concept that what error handling aim. Just
like two type error request so error handling aim existed 2.
1)       clear time out scmd in LLDD. Because time out request is handling in
LLDD, the scmd is still remembered in LLDD driver. So the error handling want
the LLDD forgot the time out scmd in LLDD.
And cancel the jobs that doing for handling the time out scmd in LLDD driver.
2)       Found hardware have error, so reset hardware to let it work
normally.


function scsi_unjam_host 2
static void
scsi_unjam_host(struct Scsi_Host *shost)
{
       unsigned long flags;
       LIST_HEAD(eh_work_q);
       LIST_HEAD(eh_done_q);

       spin_lock_irqsave(shost->host_lock,
flags);
       list_splice_init(&shost->eh_cmd_q,
&eh_work_q);
       spin_unlock_irqrestore(shost->host_lock,
flags);

       SCSI_LOG_ERROR_RECOVERY(1,
scsi_eh_prt_fail_stats(shost, &eh_work_q));




   



   


       if (!scsi_eh_get_sense(&eh_work_q,
&eh_done_q))
              if
(!scsi_eh_abort_cmds(&eh_work_q, &eh_done_q))
                     scsi_eh_ready_devs(shost,
&eh_work_q, &eh_done_q);



       scsi_eh_flush_done_q(&eh_done_q);
}

  


  
   



reference scsi_eh.txt 1
>

    1. Lock shost->host_lock, splice_init
shost->eh_cmd_q into local
       eh_work_q and unlock host_lock.  Note that shost->eh_cmd_q is
       cleared by this action.

    2. Invoke scsi_eh_get_sense.

    >

       This action is taken for each
error-completed
       (!SCSI_EH_CANCEL_CMD) commands without
valid sense data.  Most
       SCSI transports/LLDDs automatically
acquire sense data on
       command failures (autosense).  Autosense is recommended for
       performance reasons and as sense
information could get out of
       sync inbetween occurrence of CHECK
CONDITION and this action.

       Note that if autosense is not supported,
scmd->sense_buffer
       contains invalid sense data when
error-completing the scmd
       with scsi_done().  scsi_decide_disposition() always returns
       FAILED in such cases thus invoking SCSI
EH.  When the scmd
       reaches here, sense data is acquired and
       scsi_decide_disposition() is called
again.

       1. Invoke scsi_request_sense() which
issues REQUEST_SENSE
           command.  If fails, no action.  Note that taking no action
           causes higher-severity recovery to
be taken for the scmd.

       2. Invoke scsi_decide_disposition() on
the scmd

         
- SUCCESS
              scmd->retries is set to
scmd->allowed preventing
              scsi_eh_flush_done_q() from
retrying the scmd and
              scsi_eh_finish_cmd() is invoked.

         
- NEEDS_RETRY
              scsi_eh_finish_cmd() invoked

         
- otherwise
              No action.

    3. If !list_empty(&eh_work_q), invoke
scsi_eh_abort_cmds().

    >

       This action is taken for each timed out
command.
       hostt->eh_abort_handler() is invoked
for each scmd.  The
       handler returns SUCCESS if it has
succeeded to make LLDD and
       all related hardware forget about the
scmd.

       If a timedout scmd is successfully
aborted and the sdev is
       either offline or ready, scsi_eh_finish_cmd()
is invoked for
       the scmd.
Otherwise, the scmd is left in eh_work_q for
       higher-severity actions.

       Note that both offline and ready status
mean that the sdev is
       ready to process new scmds, where
processing also implies
       immediate failing; thus, if a sdev is in
one of the two
       states, no further recovery action is
needed.

       Device readiness is tested using
scsi_eh_tur() which issues
       TEST_UNIT_READY command.  Note that the scmd must have been
       aborted successfully before reusing it
for TEST_UNIT_READY.

    4. If !list_empty(&eh_work_q), invoke
scsi_eh_ready_devs()

    >

       This function takes four increasingly
more severe measures to
       make failed sdevs ready for new commands.

       1. Invoke scsi_eh_stu()

       >

         
For each sdev which has failed scmds with valid sense data
         
of which scsi_check_sense()'s verdict is FAILED,
         
START_STOP_UNIT command is issued w/ start=1.  Note that
         
as we explicitly choose error-completed scmds, it is known
         
that lower layers have forgotten about the scmd and we can
         
reuse it for STU.

         
If STU succeeds and the sdev is either offline or ready,
         
all failed scmds on the sdev are EH-finished with
         
scsi_eh_finish_cmd().

         
*NOTE* If hostt->eh_abort_handler() isn't implemented or
         
failed, we may still have timed out scmds at this point
         
and STU doesn't make lower layers forget about those
         
scmds.  Yet, this function
EH-finish all scmds on the sdev
         
if STU succeeds leaving lower layers in an inconsistent
         
state.  It seems that STU action
should be taken only when
         
a sdev has no timed out scmd.

       2. If !list_empty(&eh_work_q), invoke
scsi_eh_bus_device_reset().

       >

         
This action is very similar to scsi_eh_stu() except that,
         
instead of issuing STU, hostt->eh_device_reset_handler()
         
is used.  Also, as we're not
issuing SCSI commands and
         
resetting clears all scmds on the sdev, there is no need
         
to choose error-completed scmds.

       3. If !list_empty(&eh_work_q), invoke
scsi_eh_bus_reset()

       >

         
hostt->eh_bus_reset_handler() is invoked for each channel
         
with failed scmds.  If bus reset
succeeds, all failed
         
scmds on all ready or offline sdevs on the channel are
         
EH-finished.

       4. If !list_empty(&eh_work_q), invoke
scsi_eh_host_reset()

       >

         
This is the last resort.
hostt->eh_host_reset_handler()
         
is invoked.  If host reset
succeeds, all failed scmds on
         
all ready or offline sdevs on the host are EH-finished.

       5. If !list_empty(&eh_work_q), invoke
scsi_eh_offline_sdevs()

       >

         
Take all sdevs which still have unrecovered scmds offline
         
and EH-finish the scmds.

    5. Invoke scsi_eh_flush_done_q().

       >

         
At this point all scmds are recovered (or given up) and
         
put on eh_done_q by scsi_eh_finish_cmd().  This function
         
flushes eh_done_q by either retrying or notifying upper
         
layer of failure of the scmds.




1.1 time out request error handle path
Previously I have said that error handle
aim for time out request is to let LLDD to forgot that scmd after an time out
scmd insert to error queue. (By the way time out request also may be handled by
other way, I would describe later. Here only talked error handling thread how
to handle time out request) the error handling aim is to let LLDD to
forgot.  So it mark SCSI_EH_CANCEL_CMD to
the time out scmd. And insert in to error queue.
In scsi_unjam_host function. It would tries
1)       abort the request ( to make it forgot ) if failed
2)       try to reset device to make hardware forgot scmd. ( because
scmd->done function already be ignore, so let hardware forgot that is the
left thing ), if failed
3)       try to reset bus, if failed
4)       try to reset host., if failed
5)       mark the device (that still have failing request)is offline.  

The other important function related to
time out request eh_timed_out function
1.2 error return request handle path
The error return request would not try to
abort request because it already return by LLDD driver. So it just try to get
correct sense.
1)       try to get sense
2)       try to send start stop unit to enable the device
3)       try to reset device to make hardware forgot scmd. ( because
scmd->done function already be ignore, so let hardware forgot that is the
left thing ), if failed
4)       try to reset bus, if failed
5)       try to reset host., if failed
6)       mark the device (that still have failing request)is offline.  

1.3 compare the two error request type handling
The two path is only different at previous time
out scmd need to let LLDD forgot and sense error request need to re-get sense.

If failed, they all try to reset hardware
until reset return succeed. Because they thinks if abort or get sense fail the
hardware may have problem so reset. hardware

 SHAPE 
\* MERGEFORMAT


  
  
   
   
   
   
   
   
   
   
   
   
   
   
  
  
  

  
  
  

  
   
   
   
     
     
     time
     out scmd
     
     
   
   
   

  

  
   
   
   
     
     
     SCSI_EH_CANCEL_CMD
     
     
   
   
   

  
   
   
   
     
     
     Abort
     request
     
     
   
   
   

  

  
   
   
   
     
     
     returned failed scmd
     
     
   
   
   

  

  
   
   
   
     
     
     sense is not valid
     
     
   
   
   

  
   
   
   
     
     
     get
     sense
     
     
   
   
   

  


   此处缺少图,若要看图请下载 pdf

1.4 The fate of error request
 SHAPE 
\* MERGEFORMAT


  
  
  

  
   
   
   
     
     
     any faile
     
     
   
   
   

  
   
   
   
     
     
     device
     on line
     
     
   
   
   

  

  
   
   
   
     
     
     have
     retry opptunity
     
     
   
   
   

  
   
   
   
     
     
     reinsert
     to request queue
     
     
   
   
   

  


  

  
   
   
   
     
     
     finish the request
     
     
   
   
   

  

  
   
   
   
     
     
     both true
     
     
   
   
   


  此处缺少图,若要看图请下载 pdf

After the error request handled finished,
it would insert to an done_q. When scsi_eh_flush_done_q called, all the scmd in
the done_q would be handled. The scmd in done_q have two results. one is to
reinsert to request queue to retry. If it retry count didn’t run out and the
device still on line. The other result is just finish the request.

So the error handling aim in that thread we
could see is just make hardware works fine, then may retry or fail the error
request.
1.5 others related to error handling thread
1) the scmd to do error handing such as
send START_STOP_UNIT cmd is reusing the failed scmd.



2 time out scmd before enter in error queue.
Time out is an very common error situation.
Above talked about how to handle time out scmd in scsi_error_handler thread. In
fact not all time out request would be handled by scsi_error_handler thread
handle.  Many of them would handled by
others.

In linux scsi host and transport templates
structure both have time out scmd handler function pointer. both name
eh_timed_out. Both of them are optional.

Simply saying, that
1)      
eh_timed_out function if not
available it would return  EH_NOT_HANDLED. And insert to time out scmd
into error queue.
2)      
if eh_timed_out avail, it have
3 return value, EH_HANDLED,EH_RESET_TIMER,EH_NOT_HANDLED
3)      
EH_HANDLED_HANDLED lead to
finish to command
4)      
EH_ RESET_TIMER lead to
enlarger the time out time value.
5)      
EH_NOT_HANDLED lead to the scmd
request insert to error queue

And when I check code I
found EH_RESET_TIMER is not cause retry count increase not like kernel Document
saying

So here shows that only EH_NOT_HANDLED time
out scmd would enter in error queue.

Below reference paragraphs have detail
introduction.

reference scmd time out related 2
[1-2-2]
Completing a scmd w/ timeout

The timeout handler is scsi_times_out().  When a timeout occurs, this
function

1. invokes optional hostt->eh_timed_out()
callback.  Return value can
    be one of

    - EH_HANDLED
        This indicates that eh_timed_out()
dealt with the timeout.  The
        scmd is passed to __scsi_done() and
thus linked into per-cpu
        scsi_done_q.  Normal command completion described in
[1-2-1]
        follows.

    - EH_RESET_TIMER
        This indicates that more time is
required to finish the
        command.  Timer is restarted.  This action is
counted as a
        retry and only
allowed scmd->allowed + 1(!) times.  Once the
        limit is reached, action for EH_NOT_HANDLED
is taken instead.

        *NOTE* This action is racy as the LLDD
could finish the scmd
        after the timeout has expired but
before it's added back.  In
        such cases, scsi_done() would think
that timeout has occurred
        and return without doing anything.  We lose completion and the
        command will time out again.

    - EH_NOT_HANDLED
        This is the same as when eh_timed_out()
callback doesn't exist.
        Step #2 is taken.

2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD)
is invoked for the
    command.
See [1-3] for more information.




3 the mpt2sas error handling introduction
LSI mpt2SAS driver is for the LSI message
passing technology related controller driver. I thinks it just like our Loki,
it includes embedded cpus.

For error handling part it have below
functions.
reference lsi error handling related
functions 3
        .eh_abort_handler               = scsih_abort,
        .eh_device_reset_handler        = scsih_dev_reset,
        .eh_host_reset_handler          = scsih_host_reset,

scsih_abort function do belows things
1)      if the scmd have no hardware related just call scmd->done();
2)     
if the scmd have hardware
related, get a message and set it to MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK and
send it to hba firmware. to cancel related command.

scsih_dev_reset function do belows things
1)      if the scmd have no hardware related just call scmd->done();
2)      find the related device to the scmd.
3)     
and get a message and set it to
MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET and send it to hba firmware to reset
the device.
scsih_host_reset function do below things
1)     
reset the hba controller.

During hba hardware reset. It would block
by using completion mechanism to wait for hba restart.

               

       
        文件:About Linux module error handling.pdf
        大小:72KB
        下载:
下载
       
               
               
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}

  Normal
  0
  
  7.8 pt
  0
  2
  
  false
  false
  false
  
   
   
   
   
   
   
   
   
   
   
   
   
  
  MicrosoftInternetExplorer4



/* Style Definitions */
table.MsoNormalTable
        {mso-style-name:"Table Normal";
        mso-tstyle-rowband-size:0;
        mso-tstyle-colband-size:0;
        mso-style-noshow:yes;
        mso-style-parent:"";
        mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
        mso-para-margin:0cm;
        mso-para-margin-bottom:.0001pt;
        mso-pagination:widow-orphan;
        font-size:10.0pt;
        font-family:"Times New Roman";
        mso-ansi-language:#0400;
        mso-fareast-language:#0400;
        mso-bidi-language:#0400;}
table.MsoTableGrid
        {mso-style-name:"Table Grid";
        mso-tstyle-rowband-size:0;
        mso-tstyle-colband-size:0;
        border:solid windowtext 1.0pt;
        mso-border-alt:solid windowtext .5pt;
        mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
        mso-border-insideh:.5pt solid windowtext;
        mso-border-insidev:.5pt solid windowtext;
        mso-para-margin:0cm;
        mso-para-margin-bottom:.0001pt;
        text-align:justify;
        text-justify:inter-ideograph;
        mso-pagination:none;
        font-size:10.0pt;
        font-family:"Times New Roman";
        mso-ansi-language:#0400;
        mso-fareast-language:#0400;
        mso-bidi-language:#0400;}


  

               
               
               
               
               
               
               
               
               
               

本文来自ChinaUnix博客，如果查看原文请点：http://blog.chinaunix.net/u3/102470/showart_2020067.html

liumangxiong

very good