When developing an operating system on top of seL4 you do not have
the luxury of using a debugger such as gdb. Your best bet is a combination
of dprintf and objdump. If you are doing the project on a
Linux machine (using our linux toolchain) you can also use the backtrace function,
combined with a script for parsing objdump output to obtain a primitive stack trace.
All exceptions result in a context switch to the seL4 kernel. The kernel exports fault handling policies to user level by sending an IPC to the appropriate IPC endpoint (see Chapter 5 of the seL4 manual). For applications such as tty_test, the assigned endpoint will usually be the IPC endpoint that SOS waits on in the syscall_loop. SOS on the other hand is initialised with no fault endpoint and hence is expected to ensure that it does not cause a fault.
Below is an example of a page fault that is caused by SOS:
388 int main(void) {
389
390 dprintf(0, "\nSOS Starting...\n");
391 *((char*)0xDEADBEEF) = 0;
SOS Starting... Caught cap fault in send phase at address 0x0 while trying to handle: vm fault on data at address 0xdeadbeef with status 0xf5 in thread 0xf0063e00 at address 0x8ae4
A capability fault occurs while sending the fault IPC to the fault handler because
the capability address of SOS's fault endpoint is 0x0 (seL4_CapNull). 0x0 does not address
a valid synchronous endpoint in the capability space of the faulting thread (SOS).
From the provided information, we also see that SOS caused
a vm fault. The fault address register (FAR) was set to 0xdeadbeef while the fault status
register (FSR) was set to 0xf5 (see Section 3.6 of the
ARMv5 manual). Lastly, and most
importantly, we see the value of the instruction pointer at the time of the fault is 0x8ae4.
objdump
may then be used to determine the context of fault.
When an application thread is created, an appropriate fault handler IPC endpoint should be
set in the thread's TCB (see Chapter 5 of the seL4 manual). If the applciation causes a fault
and IPC will be sent to the assigned fault endpoint. Message registers provide important
imformation regarding the cause and location of the fault.
objdump
may then be used to determine the context of
fault.
objdump
Whenever you cause a fault, you should be able to see the current program
counter printed out. You can use objdump to find where the offending code is.
% armeb-oe-linux-gnueabi-objdump -Dlx stage/arm/nslu2/bin/sos | less
You can now use the searching facility in less to search
for the faulting address. In this case I find the following fragment of
output:
/tmp_amd/ravel/export/ravel/2/akroh/aos-2012/apps/sos/src/main.c:391
8adc: e3a02000 mov r2, #0
8ae0: e59f367c ldr r3, [pc, #1660] ; 9164
8ae4: e5432010 strb r2, [r3, #-16]
We can see that the error has occured on code at line 391 of main.c. If the
file name and line number does not appear in your dump, run
make menuconfig, select Emit debugging information
in the Toolchain Options menu and recompile.
If the fault occurs in an application (such as tty_test or sosh) rather than SOS, the following command should be used:
% armeb-oe-linux-gnueabi-objdump -Dlx stage/arm/nslu2/bin/appname | less
objdump is a very handy utility for working out exactly what is
where in an executable so you can work out what exactly is going wrong.
The two standard incantations for objdump are:
% armeb-oe-linux-gnueabi-objdump -dl my_elf.file | lessand
% armeb-oe-linux-gnueabi-objdump -lx my_elf.file | lessThe first command (-dl) disassesmbles the text segment and shows you all the instructions and at what address. Using this information you can find out things such as:
eg. Am I compiling the right function? Do the instructions make sense?
Look at the fsr for the type of fault, and use objdump to find the exact instruction causing the fault.
NB: In case it's not yet obvious, you will need to get up to speed on your ARM assembly and not be afraid to get your hands dirty if you want to minimise the time you spend debugging.
The second objdump command (-lx) is useful for when addresses appear inside an object file but outside of the text segment. This is especially useful when debugging ELF loading. The -lx option displays section and symbol information. Further options can be added to dump data segments etc. man objdump is your friend.
Warning: Backtrace only works for the Linux toolchain
The backtrace() function is called by default by assert() and conditional_panic(). You can also call it from within any arbitrary place in your code. Backtrace will traverse the stack and print out the addresses of each function in the call stack (as long as you haven't done anything to the stack). This can be useful if you need to know what your code is doing.
Backtrace output looks like this:
Assertion failed: 0, function _sos_init, file /home/qiang/aos-2012/apps/sos/src/main.c, line 385. seL4 root server aborted Backtracing stack PCs: 0xb06c 0xbe88 0x9124 <------------------------------the back trace Debug halt syscall from user thread 0xf0063e00 halting...
We have also provided a script to parse the objdump output and lookup the addresses in the stack trace and show the assembler and C code that they correspond to. Copy the script into the root directory of you project and execute as follows:
USEAGE: backtrace_symbol [OPTIONS] <<EOF
PCs
<<EOF
OPTIONS: -l printout line number for disassembly
-S printout source code for disassembly
-w WIDTH extra lines of disassembly for each side of the target PC, between 4 and 10 (default is 4)
--start-address=0xADDR only process data whose address is <= ADDR
--stop-address=0xADDR only process data whose address is >= ADDR
--h printout this message
Format: the PCs should be started with a new line each
PCs are hex numbers, start with "0x"
Finally, here is an example usage of the backtrace script:
./backtrace_symbol.sh -l -S -w 5 <<EOF 0xb074 0xb034 0x91e0 EOFThis should output something like:
--------------------------------------------------------------------------------
# 0 0x0000b074 in __conditional_panic
b060: e59f00b8 ldr r0, [pc, #184] ; b120 <__conditional_panic+0xdc>
b064: eb0004aa bl c314 <printf>
b068: e58d5000 str r5, [sp]
b06c: e59f00b0 ldr r0, [pc, #176] ; b124 <__conditional_panic+0xe0>
b070: e1a01004 mov r1, r4
b074: e1a02006 mov r2, r6
b078: e59d3050 ldr r3, [sp, #80] ; 0x50
b07c: ebfffce7 bl a420 <plogf>
b080: e59f00a0 ldr r0, [pc, #160] ; b128 <__conditional_panic+0xe4>
b084: eb0004a2 bl c314 <printf>
/home/qiang/aos-2012/apps/sos/src/sys/panic.c:21
}
b088: e28d0008 add r0, sp, #8
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
# 1 0x0000b034 in _logf
b020: e24dd00c sub sp, sp, #12
/home/qiang/aos-2012/apps/sos/src/sys/network.c:300
va_list alist;
va_start(alist, msg);
b024: e28d1014 add r1, sp, #20
b028: e58d1004 str r1, [sp, #4]
/home/qiang/aos-2012/apps/sos/src/sys/network.c:301
plogf(msg, alist);
b02c: e59d0010 ldr r0, [sp, #16]
b030: ebfffcfa bl a420 <plogf>
/home/qiang/aos-2012/apps/sos/src/sys/network.c:303
va_end(alist);
}
b034: e28dd00c add sp, sp, #12
b038: e49de004 pop {lr} ; (ldr lr, [sp], #4)
b03c: e28dd010 add sp, sp, #16
b040: e12fff1e bx lr
0000b044 <__conditional_panic>:
__conditional_panic():
/home/qiang/aos-2012/apps/sos/src/sys/panic.c:13
#define verbose 1
inline void __conditional_panic(int condition, const char *message,
const char *file, const char *func, int line) {
if (condition) {
_dprintf(0, "\033[22;31m", "PANIC %s-%s:%d %s\n\n", file, func, line, message);
b044: e92d45f0 push {r4, r5, r6, r7, r8, sl, lr}
b048: e24dd034 sub sp, sp, #52 ; 0x34
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
# 2 0x000091e0 in main
91cc: e3a03f52 mov r3, #328 ; 0x148
91d0: e2833003 add r3, r3, #3
91d4: e58d3000 str r3, [sp]
91d8: e1a00621 lsr r0, r1, #12
91dc: e59f1178 ldr r1, [pc, #376] ; 935c <main+0x768>
91e0: e1a02006 mov r2, r6
91e4: e1a03008 mov r3, r8
91e8: eb000795 bl b044 <__conditional_panic>
/home/qiang/aos-2012/apps/sos/src/main.c:335
/* Create an endpoint for user application IPC */
ep_addr = ut_alloc(seL4_EndpointBits);
91ec: e3a00004 mov r0, #4
91f0: eb000274 bl 9bc8 <ut_alloc>
/home/qiang/aos-2012/apps/sos/src/main.c:336
conditional_panic(!ep_addr, "No memory for endpoint");
91f4: e1a05000 mov r5, r0
--------------------------------------------------------------------------------