Text in red colour was added/modified after the start of the exam.
- This is a 24h take-home exam.
- The exam runs from 11:00 on Wed, 19 August 2020 to 11:00 on Thu, 20 August 2020
- The basic exam question is available now, but the papers you are asked to analyse will be made available only at 11:00 on Wed, 19 August 2020.
- The papers will be available electronically via this WWW page.
- The total exam is worth 35 marks. The two papers may carry differnet weights (to be revealed when the papers are available).
- You will lose 3.5 marks for each hour, or part thereof, your submission is late.
- You are not to get any help from anyone on the exam. You should not talk to anyone else about the exam from the time you receive the full details until you submit your solution.
- Submit an electronic copy (PDF), via the give system, mark name "exam", by the deadline.
- Please use separate PDF files for the two
), that makes my life easier as I mark all the reports for a paper in a batch.
- Please do not put your name on the submission, only your student ID. This makes it easier for me to do an unbiased assessment.
Note: There will be no option of a hardcopy submission this year, submission is only through give. There is also no need to electronically sign submissions.
You are to read, understand, and critically assess the papers. Questions you may want to ask yourself for each of the papers:
- What problem is it trying to address?
- How well does it address the issue?
- How well do they motivate the value of solving the problem. Is the problem a real problem? Is the problem/work significant?
- How does it relate to other work? Does it reference relevant other work (as far as you can tell), does it do the other work justice? Has it all been done before?
- How technically sound is it? Does their argumentation, the presented data convince you? Should they have been looking at other issues?
- How good are the results?
- How good/deep is their analysis?
- How easy would it be to reproduce their results?
- How general are their results? Can they be applied to other systems? Did we learn some general truth?
These are only hints, I am not asking you to explicitly answer all these for each paper. However, you may find those questions helpful in critically analysing the papers. Imagine you are a reviewer for a conference to which the papers have been submitted, and you are to judge their contribution to the field. In order to get an idea of what program committees at top systems conferences are looking for, have a look at this classic!
You can expect to get a bare pass if you demonstrate that you have understood what the authors are doing. Beyond that, I want to see a critical assessment, given the knowledge you gained in this course. Note that all papers are in fact published (and should therefore meet certain quality standards one hopes :-). Nevertheless, they may have flaws. If they do, then I'd like you to find them. Beyond that, you should critically assess the work against your understanding of OS issues. The more depth of analysis you demonstrate, the higher your mark will be.
What to submitYou are to submit for each paper a report which summarises the basic ideas behind their work. You are to give a critique of the technical merits, achievements and shortcomings (if any). The papers are not directly related, so you don't have to compare them.
I am intentionally not specifying a length limit. However, I strongly encourage you to be concise. Lengthy submissions will almost certainly be unfocussed and waffly. I cannot imagine a decent job in excess of 3000 words, and a very good submission should be possible to be written up in 2–3 pages. If your report gets longer than this you should step back and try to focus.
A good way to structure your review is the standard approach taken by conference program committees, which tend to use some variant of a basic structure which has the following sections:
Summary of the paper. This is about three paragraphs summarising
what the paper is trying to achieve and how it goes about it, and
how relevant the work is. Given that you're trying to convince me
that you got it, you may go into a bit more detail than the
typical reviewer would (who is selected as an expert in the field
and doesn't have to prove themselves). So you may want to write up
to a page here (but be concise!)
Also, this section should not focus on criticising the paper (although with a bad paper I frequently find it difficult to state what they are doing without noting that it is wrong...)
- Pros: What you like about the paper (list of bullet points or short sentences).
- Cons: What you don't like about the paper.
- Criticism of the work. Things which are wrong, insufficient, could be improved. But also detailed discussion of the strengths. If you were a reviewer whose job is to recommend whether the paper should be accepted or rejected, this is where you make your case. But even if you generally like the paper, discuss its shortcomings, and even if you would reject it, discuss its redeeming features. This is the most important part of the review.
- Minor issues that should be fixed (typos, grammar, etc). These are part of a formal paper review, but not really relevant to this exercise, so omit (unless something really irks you ;-).
- Questions to the authors. Many conferences have a rebuttal period, where authors get the chance to comment on the reviews before the decision is made. If you think there is something the authors could clarify to help you make a decision, this is the place. Again, given that the paper is already published, this section is obviously optional.
- Points that must be addressed if accepted. Many conferences use a shepherding process, where accepted papers are assigned a shepherd who supervises the revisions and ensure that the authors follow the requests of the reviewers. Again, in your case it's too late, but if you think that there are improvements that should have been made prior to publication, then this is the place to make your point.
Remember: In order to help us to perform an unbiased assessment of your report, please do not put your name on the report itself, only your student ID. And please submit the reports on the two papers as separate files, to make it easier for me to mark each paper in batch.
What I will be looking forYou will be marked on the level of understanding and critical analysis portrayed in you submission. All relative to what can be reasonably expected from you (I know that none of you have a PhD in OS yet :-)
Note: this is an exam, not betting on horses. It is dangerous to guess what I might think of the paper, or to guess that there'll be a good and a bad one. Papers are selected on other criteria.