【求助】kernel panic问题

mrfan

425的板子上跑一个netfilter的测试模块（为了省空间编进内核了），分析数据包内容并将需要的信息printk以记录日志，结果出现严重的panic错误（bad PC value），按照albcamus版主的oops调试方法，打开了kernel debug选项，幸好跑了一晚上重现到了问题，panic信息在最后（太长了，这里只给了前两次的，后面基本一样，不停在do_PrefetchAbort->asm_do_IRQ循环报错），都是backtrace到ipt_mydns_target函数再无法trace下去了。我个人的理解就是ipt_mydns_target+0x324这里调用printk的时候出了问题。做应用方面的多些，所以内核只是刚刚入门，希望各位大牛不吝赐教，这个问题郁闷狠了，google、baidu了一天都有点晕。

实现代码：

...... printk(KERN_INFO "test %s %s %u.%u.%u.%u %s %s\n", LOG_DNS_TEXT, LOG_SRC_TEXT, NIPQUAD(iph->saddr), LOG_DST_DNS, dns); dns_count++; ......

汇编代码：（objdump出来的ipt_mydns_target函数）

c065bc24:       1a000013        bne     c065bc78 <.text+0x178c78> c065bc28:       e51b6154        ldr     r6, [fp, #-340] c065bc2c:       e59f0544        ldr     r0, [pc, #1348] ; c065c178 <.text+0x179178> c065bc30:       e59f1544        ldr     r1, [pc, #1348] ; c065c17c <.text+0x17917c> c065bc34:       e5d6200d        ldrb    r2, [r6, #13] c065bc38:       e5d6300c        ldrb    r3, [r6, #12] c065bc3c:       e24b7f4a        sub     r7, fp, #296    ; 0x128 c065bc40:       e58d2000        str     r2, [sp] c065bc44:       e5d6200e        ldrb    r2, [r6, #14] c065bc48:       e58d2004        str     r2, [sp, #4] c065bc4c:       e5d6c00f        ldrb    ip, [r6, #15] c065bc50:       e59f2528        ldr     r2, [pc, #1320] ; c065c180 <.text+0x179180> c065bc54:       e58d7010        str     r7, [sp, #16] c065bc58:       e58dc008        str     ip, [sp, #8] c065bc5c:       e59fc520        ldr     ip, [pc, #1312] ; c065c184 <.text+0x179184> c065bc60:       e58dc00c        str     ip, [sp, #12] c065bc64:       ebfa7795        bl      c04f9ac0 <printk> c065bc68:       e59f2518        ldr     r2, [pc, #1304] ; c065c188 <.text+0x179188> c065bc6c:       e5923000        ldr     r3, [r2] c065bc70:       e2833001        add     r3, r3, #1      ; 0x1 c065bc74:       e5823000        str     r3, [r2] c065bc78:       e1a00008        mov     r0, r8

panic信息：

1. 
   
2. Unable to handle kernel NULL pointer dereference at virtual address 00000000
   
3. pgd = c0004000
   
4. [00000000] *pgd=00000000
   
5. Internal error: Oops: 0 [#1]
   
6. Modules linked in: ffw(P) ixp400_eth ixp400(P) rtc(P)
   
7. CPU: 0
   
8. PC is at __init_begin+0x3fff8000/0x30
   
9. LR is at asm_do_IRQ+0x48/0x60
   
10. pc : [<00000000>]    lr : [<c04e44c0>]    Tainted: P     
    
11. sp : c069bacc  ip : c069a000  fp : c069bae0
    
12. r10: f069cbe9  r9 : 000002c8  r8 : c06e2589
    
13. r7 : 60000013  r6 : 00000020  r5 : c069bf68  r4 : c06fa540
    
14. r3 : 00010200  r2 : c069c140  r1 : c069c140  r0 : 00000005
    
15. Flags: Nzcv  IRQs off  FIQs on  Mode SVC_32  Segment kernel
    
16. Control: 39FF
    
17. Table: 02F60000  DAC: 00000017
    
18. Process swapper (pid: 0, stack limit = 0xc069a250)
    
19. Stack: (0xc069bacc to 0xc069c000)
    
20. bac0:                            ffffffff 0000001f c069bb94 c069bae4 c04e3824
    
21. bae0: c04e4484 00000001 c33f3f64 c069f2f4 c069f2f4 00000000 c0686d10 c069bbac
    
22. bb00: 60000013 c06e2589 000002c8 f069cbe9 c069bb94 00000013 c069bb2c c069f2f4
    
23. bb20: c04f9a68 60000013 ffffffff 0000f324 00000cdb c069bb64 c2d91c80 c05ffbac
    
24. bb40: 0000f03f 00000fc0 c2d91c80 c04c7800 c069bb70 c069bb60 00000000 c04c7800
    
25. bb60: 00000000 c069bbb4 c069bb74 00000087 0000004c c29cc024 c069bc18 252349ea
    
26. bb80: 3a49fb55 f069cbe9 c069bba4 c069bb98 c04f9adc c04f97e4 c069bd40 c069bbb8
    
27. bba0: c065bc68 c04f9ad0 c0686d10 c0686d38 c0685f68 000000c0 000000a8 00000000
    
28. bbc0: 000000e6 c0686d48 c069bc18 000000d3 00000059 00000006 0000028e 0000028f
    
29. bbe0: c29cc2e0 c29cc040 c29cc038 c29cc024 00000002 c069be3c c069bc04 c05e2cd8
    
30. bc00: c05e15b8 4849357f 0006d5d2 c069bc38 c069bc1c c0663910 678381b2 e101d184
    
31. bc20: 9c0148c0 bb26c6f5 9fab37b7 f7d39048 e72e2667 1bb10b0d 16bdfe8a fe2faca0
    
32. bc40: cae403cb b2f0c558 b3146031 11e0c4b7 29a86e31 7f40b15e 74d61891 f39dde54
    
33. bc60: 716711f7 8e94ea05 2049e18b e03c2724 4a2ab3fd b3e34421 abf6b2ad 9742c728
    
34. bc80: c52bd1f7 b9b59773 5ca5a451 8a226c38 dcc4b849 8e18f955 c7edf4b5 d7a93854
    
35. bca0: 06ad663e 2e5926f6 47c90395 4d1a92f3 b69201fd d6146288 b4aa6e56 29b079d5
    
36. bcc0: 3f4de365 11499cf3 2eb79b6b 082b647b 54b49643 5dd01ba1 8bf83479 54f591a5
    
37. bce0: 9bb7774e b89be380 7be405b5 75743276 07b24c88 94ffbb5d bbdfbdaf 3532fb2c
    
38. bd00: a115477b 971f29cc e2e69f4a 8d47379b 368dea17 533cc286 1f90a732 542bbb48
    
39. bd20: 1f800cc7 9ec89a64 127c143c cddb192f 61c561da d58c1ca4 1f890bbe 17d0c09e
    
40. bd40: 523aaff0 e96a3520 4f3a77b1 6cbb85c8 2ec8bff8 474e8159 672a9c79 3ca186aa
    
41. bd60: a840db69 210fbbbd efa6eeea d4ec506b 50c4aa78 6d0f704d b63685df 30ec0870
    
42. bd80: 34cf1a56 3e91830a 9208ab4c 09d2f3d0 6a445120 7529d26e ed99b1ca 581996da
    
43. bda0: 8c347a28 51977cda 68122683 cd45b9d6 d0fb2e54 29238731 c0335a0f 90a2f403
    
44. bdc0: 7566ce33 780d076e 797f81df fa3ee4c4 d2ff6f68 b632c098 21320f4b 5672ae66
    
45. bde0: 8973a709 2b40f285 cd8cf885 ab2e9e87 33d9d47f 25b9f30d 2f8eb55b e810a4f4
    
46. be00: 838e601d ab62fd0d e2fca5a1 e1d1b4fb 15099804 7593373b a74fcb06 e6a230c3
    
47. be20: 720f11e2 2248409c 0968cea4 de233d41 a32304dc 304ed2b1 b937195e 76a63dc5
    
48. be40: 797b2108 3bd35b77 c72e1ca4 1072ddb2 b4a88506 15ee5bf8 500cfb90 8cbbc31b
    
49. be60: 0c72af03 04b715da 7da7e2db 521c21ca e6b7836a f37ca0ef 9cc96922 deb41644
    
50. be80: 8b2a1987 72bc52d4 352c832e 3086ead4 08344574 46cc5940 4004d387 1d5c4c77
    
51. bea0: 828b2113 62510032 04eeb46d 21fdeaa6 38d81479 8b95d5a1 7e9b6949 ece509c5
    
52. bec0: 82d15ea1 5c99b8fd e1d08c20 a3d1ea04 d294e99e 44451299 bc2562a1 5e103fb0
    
53. bee0: 83b4c061 3fe387a3 75cf60e0 7249aea7 7559fe2e 8348f4f4 6727702d feb9eaf8
    
54. bf00: cecf5a53 a3f5b09f e77346c7 9ba3b907 3065e0e8 cba0ec8a 42892ab0 2b3bc63e
    
55. bf20: eb126b1e 4ee160e1 cbfec02a 0f884157 bf0893d0 354b272f 03fd3304 1228d2cc
    
56. bf40: 8e57b51d 836b76ca fbe41e0f 30d22b49 fff7fb72 18d44181 2623440f 08bf1135
    
57. bf60: af2d56a0 72167b2b 17ca819e bc82d6d2 146ef2cd fd01186b 26d10f29 ea38b529
    
58. bf80: c5476da3 2bf3c02b 8cb25bc5 ec758baa 57380549 cd2b7e5b 4958d5ea 8f90f885
    
59. bfa0: 2b8a02a6 73b04254 84443032 192699b2 22ff1627 b11d49d0 26519b55 fef9eb98
    
60. bfc0: 08f929ed acc394ab 85b29938 2055ddea 9e812e80 5b45e8d9 c26aebd5 e05eeef4
    
61. bfe0: ed7b540e 314872bf 84404a6e 0b71a73f 6196bc8f f01088e7 57e4ee09 747f60dd
    
62. Backtrace:
    
63. [<c04e4478>] (asm_do_IRQ+0x0/0x60) from [<c04e3824>] (__irq_svc+0x24/0x60)
    
64. r5 = 0000001F  r4 = FFFFFFFF
    
65. [<c04f97d8>] (vprintk+0x0/0x2e8) from [<c04f9adc>] (printk+0x1c/0x20)
    
66. [<c04f9ac0>] (printk+0x0/0x20) from [<c065bc68>] (ipt_mydns_target+0x324/0x87c)
    
67. r3 = 000000C0  r2 = C0685F68  r1 = C0686D38  r0 = C0686D10
    
68. Backtrace aborted due to bad frame pointer <c069bd40>
    
69. Code: bad PC value.
    
70. <1>Unable to handle kernel NULL pointer dereference at virtual address 00000000
    
71. pgd = c0004000
    
72. [00000000] *pgd=00000000
    
73. Internal error: Oops: 0 [#2]
    
74. Modules linked in: ffw(P) ixp400_eth ixp400(P) rtc(P)
    
75. CPU: 0
    
76. PC is at __init_begin+0x3fff8000/0x30
    
77. LR is at asm_do_IRQ+0x48/0x60
    
78. pc : [<00000000>]    lr : [<c04e44c0>]    Tainted: P     
    
79. sp : c069b988  ip : c069a000  fp : c069b99c
    
80. r10: c069ba84  r9 : 00000008  r8 : 00000000
    
81. r7 : 00000000  r6 : 00000020  r5 : c069bae4  r4 : c06fa540
    
82. r3 : 00020200  r2 : c069c140  r1 : c069c140  r0 : 00000005
    
83. Flags: Nzcv  IRQs off  FIQs on  Mode SVC_32  Segment kernel
    
84. Control: 39FF
    
85. Table: 02F60000  DAC: 00000017
    
86. Process swapper (pid: 0, stack limit = 0xc069a250)
    
87. Stack: (0xc069b988 to 0xc069c000)
    
88. b980:                   ffffffff 0000001f c069b9f8 c069b9a0 c04e3824 c04e4484
    
89. b9a0: 00000004 c069f30c 00000000 60000013 00000000 c069a000 fffffffc 00000000
    
90. b9c0: 00000000 00000008 c069ba84 c069b9f8 60000093 c069b9e8 00000001 c04e7a68
    
91. b9e0: 60000013 ffffffff 00000000 c069ba18 c069b9fc c04e8988 c04e7774 ffffffff
    
92. ba00: c069d4d8 00000020 00000000 c069ba54 c069ba1c c04e8c6c c04e8928 c33c8440
    
93. ba20: bf090330 033c8480 00000000 ffffffff c069bab8 00000020 60000013 c06e2589
    
94. ba40: 80000093 f069cbe9 c069ba70 c069ba58 c04e8ca0 c04e8a7c ffffffff c069bab8
    
95. ba60: 00000020 c069ba80 c069ba74 c04e8dd4 c04e8c90 c069bae0 c069ba84 c04e3900
    
96. ba80: c04e8dc8 00000005 c069c140 c069c140 00010200 c06fa540 c069bf68 00000020
    
97. baa0: 60000013 c06e2589 000002c8 f069cbe9 c069bae0 c069a000 c069bacc c04e44c0
    
98. bac0: 00000000 80000093 ffffffff ffffffff 0000001f c069bb94 c069bae4 c04e3824
    
99. bae0: c04e4484 00000001 c33f3f64 c069f2f4 c069f2f4 00000000 c0686d10 c069bbac
    
100. bb00: 60000013 c06e2589 000002c8 f069cbe9 c069bb94 00000013 c069bb2c c069f2f4
     
101. bb20: c04f9a68 60000013 ffffffff 0000f324 00000cdb c069bb64 c2d91c80 c05ffbac
     
102. bb40: 0000f03f 00000fc0 c2d91c80 c04c7800 c069bb70 c069bb60 00000000 c04c7800
     
103. bb60: 00000000 c069bbb4 c069bb74 00000087 0000004c c29cc024 c069bc18 252349ea
     
104. bb80: 3a49fb55 f069cbe9 c069bba4 c069bb98 c04f9adc c04f97e4 c069bd40 c069bbb8
     
105. bba0: c065bc68 c04f9ad0 c0686d10 c0686d38 c0685f68 000000c0 000000a8 00000000
     
106. bbc0: 000000e6 c0686d48 c069bc18 000000d3 00000059 00000006 0000028e 0000028f
     
107. bbe0: c29cc2e0 c29cc040 c29cc038 c29cc024 00000002 c069be3c c069bc04 c05e2cd8
     
108. bc00: c05e15b8 4849357f 0006d5d2 c069bc38 c069bc1c c0663910 678381b2 e101d184
     
109. bc20: 9c0148c0 bb26c6f5 9fab37b7 f7d39048 e72e2667 1bb10b0d 16bdfe8a fe2faca0
     
110. bc40: cae403cb b2f0c558 b3146031 11e0c4b7 29a86e31 7f40b15e 74d61891 f39dde54
     
111. bc60: 716711f7 8e94ea05 2049e18b e03c2724 4a2ab3fd b3e34421 abf6b2ad 9742c728
     
112. bc80: c52bd1f7 b9b59773 5ca5a451 8a226c38 dcc4b849 8e18f955 c7edf4b5 d7a93854
     
113. bca0: 06ad663e 2e5926f6 47c90395 4d1a92f3 b69201fd d6146288 b4aa6e56 29b079d5
     
114. bcc0: 3f4de365 11499cf3 2eb79b6b 082b647b 54b49643 5dd01ba1 8bf83479 54f591a5
     
115. bce0: 9bb7774e b89be380 7be405b5 75743276 07b24c88 94ffbb5d bbdfbdaf 3532fb2c
     
116. bd00: a115477b 971f29cc e2e69f4a 8d47379b 368dea17 533cc286 1f90a732 542bbb48
     
117. bd20: 1f800cc7 9ec89a64 127c143c cddb192f 61c561da d58c1ca4 1f890bbe 17d0c09e
     
118. bd40: 523aaff0 e96a3520 4f3a77b1 6cbb85c8 2ec8bff8 474e8159 672a9c79 3ca186aa
     
119. bd60: a840db69 210fbbbd efa6eeea d4ec506b 50c4aa78 6d0f704d b63685df 30ec0870
     
120. bd80: 34cf1a56 3e91830a 9208ab4c 09d2f3d0 6a445120 7529d26e ed99b1ca 581996da
     
121. bda0: 8c347a28 51977cda 68122683 cd45b9d6 d0fb2e54 29238731 c0335a0f 90a2f403
     
122. bdc0: 7566ce33 780d076e 797f81df fa3ee4c4 d2ff6f68 b632c098 21320f4b 5672ae66
     
123. bde0: 8973a709 2b40f285 cd8cf885 ab2e9e87 33d9d47f 25b9f30d 2f8eb55b e810a4f4
     
124. be00: 838e601d ab62fd0d e2fca5a1 e1d1b4fb 15099804 7593373b a74fcb06 e6a230c3
     
125. be20: 720f11e2 2248409c 0968cea4 de233d41 a32304dc 304ed2b1 b937195e 76a63dc5
     
126. be40: 797b2108 3bd35b77 c72e1ca4 1072ddb2 b4a88506 15ee5bf8 500cfb90 8cbbc31b
     
127. be60: 0c72af03 04b715da 7da7e2db 521c21ca e6b7836a f37ca0ef 9cc96922 deb41644
     
128. be80: 8b2a1987 72bc52d4 352c832e 3086ead4 08344574 46cc5940 4004d387 1d5c4c77
     
129. bea0: 828b2113 62510032 04eeb46d 21fdeaa6 38d81479 8b95d5a1 7e9b6949 ece509c5
     
130. bec0: 82d15ea1 5c99b8fd e1d08c20 a3d1ea04 d294e99e 44451299 bc2562a1 5e103fb0
     
131. bee0: 83b4c061 3fe387a3 75cf60e0 7249aea7 7559fe2e 8348f4f4 6727702d feb9eaf8
     
132. bf00: cecf5a53 a3f5b09f e77346c7 9ba3b907 3065e0e8 cba0ec8a 42892ab0 2b3bc63e
     
133. bf20: eb126b1e 4ee160e1 cbfec02a 0f884157 bf0893d0 354b272f 03fd3304 1228d2cc
     
134. bf40: 8e57b51d 836b76ca fbe41e0f 30d22b49 fff7fb72 18d44181 2623440f 08bf1135
     
135. bf60: af2d56a0 72167b2b 17ca819e bc82d6d2 146ef2cd fd01186b 26d10f29 ea38b529
     
136. bf80: c5476da3 2bf3c02b 8cb25bc5 ec758baa 57380549 cd2b7e5b 4958d5ea 8f90f885
     
137. bfa0: 2b8a02a6 73b04254 84443032 192699b2 22ff1627 b11d49d0 26519b55 fef9eb98
     
138. bfc0: 08f929ed acc394ab 85b29938 2055ddea 9e812e80 5b45e8d9 c26aebd5 e05eeef4
     
139. bfe0: ed7b540e 314872bf 84404a6e 0b71a73f 6196bc8f f01088e7 57e4ee09 747f60dd
     
140. Backtrace:
     
141. [<c04e4478>] (asm_do_IRQ+0x0/0x60) from [<c04e3824>] (__irq_svc+0x24/0x60)
     
142. r5 = 0000001F  r4 = FFFFFFFF
     
143. [<c04e7768>] (die+0x0/0x370) from [<c04e8988>] (__do_kernel_fault+0x6c/0x80)
     
144. [<c04e891c>] (__do_kernel_fault+0x0/0x80) from [<c04e8c6c>] (do_page_fault+0x1fc/0x214)
     
145. r7 = 00000000  r6 = 00000020  r5 = C069D4D8  r4 = FFFFFFFF
     
146. [<c04e8a70>] (do_page_fault+0x0/0x214) from [<c04e8ca0>] (do_translation_fault+0x1c/0x80)
     
147. [<c04e8c84>] (do_translation_fault+0x0/0x80) from [<c04e8dd4>] (do_PrefetchAbort+0x18/0x1c)
     
148. r6 = 00000020  r5 = C069BAB8  r4 = FFFFFFFF
     
149. [<c04e8dbc>] (do_PrefetchAbort+0x0/0x1c) from [<c04e3900>] (__pabt_svc+0x40/0x80)
     
150. [<c04e4478>] (asm_do_IRQ+0x0/0x60) from [<c04e3824>] (__irq_svc+0x24/0x60)
     
151. r5 = 0000001F  r4 = FFFFFFFF
     
152. [<c04f97d8>] (vprintk+0x0/0x2e8) from [<c04f9adc>] (printk+0x1c/0x20)
     
153. [<c04f9ac0>] (printk+0x0/0x20) from [<c065bc68>] (ipt_mydns_target+0x324/0x87c)
     
154. r3 = 000000C0  r2 = C0685F68  r1 = C0686D38  r0 = C0686D10
     
155. Backtrace aborted due to bad frame pointer <c069bd40>
     
156. Code: bad PC value.
     

复制代码

[ 本帖最后由 mrfan 于 2008-6-9 00:53 编辑 ]

mrfan

而且这个panic问题不确定出现，我现在用大量数据包进行测试，基本上一天之内能够重现，但是正常网络中可能3、4天才出，但也可能连续出现，甚至重启就panic。感觉应该是某种特殊的数据包引起的？但是printk只是打印信息，并没有去尝试修改。

刚刚查看了ipt_LOG的实现，发现使用了自旋锁，为了防止日志重叠（overlap） 应该不会不锁就panic了吧

kwest

“Unable to handle kernel NULL pointer dereference at virtual address 00000000”
某个指针指向了NULL（0地址）。

waikontheroad

偶也很想知道???

mrfan

没有人回答，这个怎么结贴啊？

最终自己检查代码，应该是printk的参数dns字符串处理过程中没有进行严格的安全性检查，可能对堆栈造成了破坏，加上就OK了