Course Surveys 2010

Gernot's Comments on LiC Survey

Many thanks to all students for taking the time to answer the course survey. This year we achieved a 100% return rate!

The results mostly speak for themselves. The course seems to be in good shape, overall satsifaction was high (although not as high as in the last couple of years). Given the small size of the class, not too much can be interpreted in this.

Repeatedly-made comments:

• The biggest gripe was the quality of the documentation. Which is interesting, as previous years that was also the biggest gripe, but then we were using a research platform, while for the last three years we used the commercially-supported OKL4 platform. I guess part of the problem is that we're programming directly to the kernel API, which isn't the recommended way of using OKL4, and thus not so well supported by docs. (But making more use of the library API would have hidden some of the stuff I wanted you folks to learn).
  Next year we'll be moving to seL4. This is again a research system, so whether that will improve the the documentation situation. But in real-life, docs are usually much worse. Also, working things you for yourself is part of the learning exercise (although I accept that in some cases it's just a waste of time).
  In any case, we'll think carefully about documentation for the new project version.
• Other comments regarding the project environment (libraries, kenge, build system etc) are noted, but next year will be completely different.
• Three-hour lecture block sux: Agreed, but we're at the mercy of central time tabling. At least we could do it in the afternoon followed by drinks, rather than the alternative of Tue 12–3 which was offered by central.
  We did move the milestone deadline away from the lectures following last year's comments. By all indications, this helped.
• Chairs/tables in seminar room: This is the first time we've got this complaint (from two students), but we've been using the same room for at least 4 years. Are the chairs really worse than in the normal lecture theatres?
  Also, there were tables around you could have used. I'll make this point specifically next time.

Relevant one-off comments:

• Workload is heavy:
  sure. It's meant to be that way. You won't learn building real systems without that. But we're always very up-front about this, so students know what to expect.
  If anything it's good to see that only one student said this. Looks like we've managed to adjust expectations ;-)
• Time. Having the assignment due in week 12 limited my being able to do any bonuses, especially with other assessments due in week 12. It would be awesome if you submitted a working solution on time and were able to submit diffs to complete some bonus tasks afterwards.
  We did this in the past and went off it, as people used bonus tasks as an excuse to improve their basic system, which gave them an unfair advantage. But the suggestion of letting you provide a patch (to make it easier to confirm that you just added bonus features) is definitely worth considering for next time, thanks for the suggestion!
• Limited consultation time for the project.
  I'm rather surprised to read this, and have a hard time reconciling this comments with the facts. We ran a full hour of consults each working day, which seems pretty extensive for a class of seven! Furthermore, the tutors tell me that half the consults were not used at all. What's going on???
• Not enough emphasis on research paper critical reading and evaluation during the semester
  We listed 1—3 relevant papers for each week's lecture, so you were clearly directed to relevant literature. I also always encourage people to read these papers, but inevitably hardly anyone does. I can't read them for you, so what do you expect us to do?
  What we can do in the future is to provide the relevant papers a week or so before the lecture, in the (probably vain) hope that someone will read them. Worth trying...
• Booting process could be covered.
  I agree this would be useful. Will think about how we can fit this in somewhere, but as you know, the material is crowded...
• Some of the specifications were far too vague, and were assessed on criteria that were not revealed to the students.
  It is a core feature of this course that we do not spoon-feed students. We intentionally provide only rough guidance, combined with extensive feedback by tutors. We consider this a central part of the course experience.
  As far as assessment goes: No-one's milestone demo got marked down because they chose a non-ideal design or implementation, but the tutors might have told them why their choice was bad and advised them to change it. This is very different from assessing based on undisclosed criteria, and something we will not change!
• For milestone 0, having some choice methods pointed out would be nice...
  Same comment as above: it's a feature, not a bug.
• The biggest problem with the project is the loading time
  Noted, and possibly solved with a different platform, else we'll think about it with the project re-design.
• Some more exciting drivers would be nice.
  I agree, but hard to fit in without making the project even harder. Also, the platform is limited in that it only provides trivial and complex devices, not much in between.
  With a new project next year, and possibly a new hardware platform, we will revisit this point.
• Auto marking scripts are helpful to resolve conceptual problems
  Providing you with a test harness and test cases would only work if the milestones were defined much more narrowly. We do not want this, as constraining the project too much would compromise the learning experience as it would drastically limit your design space, and avoid having to think through all the trade-offs. This is core systems work.
  Furthermore, testing your system is your job, and when thinking about your design you should think about how to test it. Taking this off you would be counter-productive!

All up, that was a lot of useful feedback. Thanks to all of you!

Gernot