seL4 Debugging Guide
When developing an operating system on top of seL4 you do not have
the luxury of using a debugger such as gdb. Your best bet is a combination
of dprintf and objdump . If you are doing the project on a
Linux machine (using our linux toolchain) you can also use the backtrace function,
combined with a script for parsing objdump output to obtain a primitive stack trace.
Debugging Faults
All exceptions result in a context switch to the seL4 kernel. The kernel exports
fault handling policies to user level by sending an IPC to the appropriate IPC endpoint
(see Chapter 5 of the seL4 manual). For applications such as tty_test, the assigned
endpoint will usually be the IPC endpoint that SOS waits on in the syscall_loop.
SOS on the other hand is initialised with no fault endpoint and hence is expected to ensure
that it does not cause a fault.
SOS faults
Below is an example of a page fault that is caused by SOS:
388 int main(void) {
389
390 dprintf(0, "\nSOS Starting...\n");
391 *((char*)0xDEADBEEF) = 0;
SOS Starting...
Caught cap fault in send phase at address 0x0
while trying to handle:
vm fault on data at address 0xdeadbeef with status 0xf5
in thread 0xf0063e00 at address 0x8ae4
A capability fault occurs while sending the fault IPC to the fault handler because
the capability address of SOS's fault endpoint is 0x0 (seL4_CapNull). 0x0 does not address
a valid synchronous endpoint in the capability space of the faulting thread (SOS).
From the provided information, we also see that SOS caused
a vm fault. The fault address register (FAR) was set to 0xdeadbeef while the fault status
register (FSR) was set to 0xf5 (see Section 3.6 of the
ARMv5 manual). Lastly, and most
importantly, we see the value of the instruction pointer at the time of the fault is 0x8ae4.
objdump
may then be used to determine the context of fault.
Application faults
When an application thread is created, an appropriate fault handler IPC endpoint should be
set in the thread's TCB (see Chapter 5 of the seL4 manual). If the applciation causes a fault
and IPC will be sent to the assigned fault endpoint. Message registers provide important
imformation regarding the cause and location of the fault.
objdump
may then be used to determine the context of
fault.
Using objdump
Whenever you cause a fault, you should be able to see the current program
counter printed out. You can use objdump to find where the offending code is.
% armeb-oe-linux-gnueabi-objdump -Dlx stage/arm/nslu2/bin/sos | less
You can now use the searching facility in less to search
for the faulting address. In this case I find the following fragment of
output:
/tmp_amd/ravel/export/ravel/2/akroh/aos-2012/apps/sos/src/main.c:391
8adc: e3a02000 mov r2, #0
8ae0: e59f367c ldr r3, [pc, #1660] ; 9164
8ae4: e5432010 strb r2, [r3, #-16]
We can see that the error has occured on code at line 391 of main.c. If the
file name and line number does not appear in your dump, run
make menuconfig , select Emit debugging information
in the Toolchain Options menu and recompile.
If the fault occurs in an application (such as tty_test or sosh) rather than SOS,
the following command should be used:
% armeb-oe-linux-gnueabi-objdump -Dlx stage/arm/nslu2/bin/appname | less
More on objdump
objdump is a very handy utility for working out exactly what is
where in an executable so you can work out what exactly is going wrong.
The two standard incantations for objdump are:
% armeb-oe-linux-gnueabi-objdump -dl my_elf.file | less
and
% armeb-oe-linux-gnueabi-objdump -lx my_elf.file | less
The first command (-dl) disassesmbles the text segment
and shows you all the instructions and at what address. Using this
information you can find out things such as:
- What does my function compile to?
eg. Am I compiling the right function? Do the instructions
make sense?
- Why is this instruction crashing?
Look at the fsr for the type of fault, and use objdump to find
the exact instruction causing the fault.
NB: In case it's not yet obvious, you will need to get
up to speed on your ARM assembly and not be afraid to get your hands
dirty if you want to minimise the time you spend debugging.
The second objdump command (-lx) is useful for when addresses
appear inside an object file but outside of the text segment. This is
especially useful when debugging ELF loading. The -lx option
displays section and symbol information. Further options can be added to
dump data segments etc. man objdump is your friend.
Generating a Stack Trace
Warning: Backtrace only works for the Linux toolchain
The backtrace() function is called by default by assert() and conditional_panic(). You can also
call it from within any arbitrary place in your code. Backtrace will traverse the stack and
print out the addresses of each function in the call stack (as long as you haven't done anything to
the stack). This can be useful if you need to know what your code is doing.
Backtrace output looks like this:
Assertion failed: 0, function _sos_init, file /home/qiang/aos-2012/apps/sos/src/main.c, line 385.
seL4 root server aborted
Backtracing stack PCs: 0xb06c 0xbe88 0x9124 <------------------------------the back trace
Debug halt syscall from user thread 0xf0063e00
halting...
We have also provided a script to parse the objdump output
and lookup the addresses in the stack trace and show the assembler and C code that they
correspond to. Copy the script into the root directory of you project and execute as follows:
USEAGE: backtrace_symbol [OPTIONS] <<EOF
PCs
<<EOF
OPTIONS: -l printout line number for disassembly
-S printout source code for disassembly
-w WIDTH extra lines of disassembly for each side of the target PC, between 4 and 10 (default is 4)
--start-address=0xADDR only process data whose address is <= ADDR
--stop-address=0xADDR only process data whose address is >= ADDR
--h printout this message
Format: the PCs should be started with a new line each
PCs are hex numbers, start with "0x"
Finally, here is an example usage of the backtrace script:
./backtrace_symbol.sh -l -S -w 5 <<EOF
0xb074
0xb034
0x91e0
EOF
This should output something like:
--------------------------------------------------------------------------------
# 0 0x0000b074 in __conditional_panic
b060: e59f00b8 ldr r0, [pc, #184] ; b120 <__conditional_panic+0xdc>
b064: eb0004aa bl c314 <printf>
b068: e58d5000 str r5, [sp]
b06c: e59f00b0 ldr r0, [pc, #176] ; b124 <__conditional_panic+0xe0>
b070: e1a01004 mov r1, r4
b074: e1a02006 mov r2, r6
b078: e59d3050 ldr r3, [sp, #80] ; 0x50
b07c: ebfffce7 bl a420 <plogf>
b080: e59f00a0 ldr r0, [pc, #160] ; b128 <__conditional_panic+0xe4>
b084: eb0004a2 bl c314 <printf>
/home/qiang/aos-2012/apps/sos/src/sys/panic.c:21
}
b088: e28d0008 add r0, sp, #8
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
# 1 0x0000b034 in _logf
b020: e24dd00c sub sp, sp, #12
/home/qiang/aos-2012/apps/sos/src/sys/network.c:300
va_list alist;
va_start(alist, msg);
b024: e28d1014 add r1, sp, #20
b028: e58d1004 str r1, [sp, #4]
/home/qiang/aos-2012/apps/sos/src/sys/network.c:301
plogf(msg, alist);
b02c: e59d0010 ldr r0, [sp, #16]
b030: ebfffcfa bl a420 <plogf>
/home/qiang/aos-2012/apps/sos/src/sys/network.c:303
va_end(alist);
}
b034: e28dd00c add sp, sp, #12
b038: e49de004 pop {lr} ; (ldr lr, [sp], #4)
b03c: e28dd010 add sp, sp, #16
b040: e12fff1e bx lr
0000b044 <__conditional_panic>:
__conditional_panic():
/home/qiang/aos-2012/apps/sos/src/sys/panic.c:13
#define verbose 1
inline void __conditional_panic(int condition, const char *message,
const char *file, const char *func, int line) {
if (condition) {
_dprintf(0, "\033[22;31m", "PANIC %s-%s:%d %s\n\n", file, func, line, message);
b044: e92d45f0 push {r4, r5, r6, r7, r8, sl, lr}
b048: e24dd034 sub sp, sp, #52 ; 0x34
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
# 2 0x000091e0 in main
91cc: e3a03f52 mov r3, #328 ; 0x148
91d0: e2833003 add r3, r3, #3
91d4: e58d3000 str r3, [sp]
91d8: e1a00621 lsr r0, r1, #12
91dc: e59f1178 ldr r1, [pc, #376] ; 935c <main+0x768>
91e0: e1a02006 mov r2, r6
91e4: e1a03008 mov r3, r8
91e8: eb000795 bl b044 <__conditional_panic>
/home/qiang/aos-2012/apps/sos/src/main.c:335
/* Create an endpoint for user application IPC */
ep_addr = ut_alloc(seL4_EndpointBits);
91ec: e3a00004 mov r0, #4
91f0: eb000274 bl 9bc8 <ut_alloc>
/home/qiang/aos-2012/apps/sos/src/main.c:336
conditional_panic(!ep_addr, "No memory for endpoint");
91f4: e1a05000 mov r5, r0
--------------------------------------------------------------------------------
Last modified: Wed Aug 06 11:00:11 EST 2008
|