Wed Nov  5 17:06:33 EST 2003

Done. All done. Completed thesis at nfd03.pdf in this directory.

Mon Oct 27 00:23:50 EST 2003
More type-writing. Callbacks should work now. Untested. Testing happens tomorrow! :)

Sat Oct 25 15:10:44 EST 2003
Basic MCS works and it's down to writing types. Worked out how to get reverse MCS going (ie callbacks into Python). Wrote a new list type, "mcssequence" to represent MCS sequences which are both homogenous and fixed-size. Python has gone out of its way to make this easy. Case in point: listobj[-index] is automatically converted to listobj[len(listobj)-index] and then passed to the extension function as an int. If they'd been lazy it could easily have been an unmodified PyObject *. It's the small things.

Thu Oct 23 23:51:25 EST 2003
Very close to getting basic mcs working. I keep getting st_prots. :(

Tue Oct 14 22:57:28 EST 2003
I would have finished the code for the Python module generator by now, but I have assignments due. Ironically the most pressing is the assignment for cs3131-compilers, implementing static semantics checking for the toy language we're writing - exactly the sort of static semantics checking that I won't be implementing for my Python-pax compiler. :-) There are two reasons that I'm not implementing static semantics checking:
	1) Everything compiled by the Python pax compiler will also be compiled by regular pax, which presumably does its own checks, and
	2) I don't have enough time. :) I haven't finished the compiler yet, but almost certainly there will be lots of valid IDL files which don't pass pypax, so the static semantics would be ad-hoc. Obviously all valid IDL files *should* pass pypax, so the semantics would change as new features were added to the compiler to make it more compatible with Mungi. All of which is undesirable.

Tue Oct 14 19:26:46 EST 2003
Writeup, mostly. Rushing to get something to Gernot before he leaves on his trip. Lots more to do.

Not much more on the implementation side. Benno had a go running the code on Alpha, and hit a problem: L4/Alpha doesn't support the floating-point registers. This is a major problem because, as it turns out, Python uses floating point everywhere: even to detect integer addition.

Two possible solutions:
	- Bite the bullet and support FP everywhere
	- Fix Python so FP support is optional.

I lean towards option 1, because the FP support is in there for a reason and removing it will be difficult to debug/prove correct (or even be reasonably sure about correctness) wrt things like overflow, etc.

Mon Oct 6, 4am :(
Got the major part of the AST generation out of the way. This is great.
The direction I will now take is: I will use the PLY-generated AST to generate a Python interface module in C. This interface module will rely on the C constructs generated by the normal C-style pax compiler.

The major disadvantage to this approach is the duplication of effort: if the IDL definition changes, it needs to be changed in two places. Fortunately it is unlikely to change significantly. The implementation of the parser in Python, which literally took a matter of hours, opens the way to reimplementing the entire pax compiler in Python, which would reduce the number of executable programs required to build Mungi (Python is already a requirement) and also allow rapid generation of new AST-walking backends.

The alternative route was for me to implement an additional backend for pax. The pax compiler has no design documents and few comments, and while it is not spaghetti my approach was obviously quicker. It also has several design advantages, the first being that the code can easily be incorporated, as-is, into pungi, paving the way for dynamic interfaces.
Sun Oct 5/Mon Oct 6
Converted the pax compiler grammar to a Python implementation of lex and yacc imaginatively named PLY (Python Lex-Yacc). Despite being a little more cumbersome than standard lex/yacc, I was able to produce a working lexer and parser (clagging the rules across) in a very short amount of time.

PLY was originally written to teach a course at the University of Chicago.

http://systems.cs.uchicago.edu/ply/

Thu Oct 2
Python now runs as a component inside Mungi. This is ... pretty good! Now working on the next step, which is to create a very simplistic script component, supporting just a single method, run(), which runs a supplied Python script.

Thu Oct 2
Well the 32-bit ELF loader seemed fine: the elf64 and elf32 formats are very very similar (as one would hope!) and so the elf32 loader is heavily based off the elf64 loader, with a couple of minor structure changes (and all the "64"s changed to "32"s. :)

Unfortunately Mungi on MIPS isn't ready to support dynamic relocation yet. Python ends up linking with two Mungi libraries and libl4, all of which need to be compiled to use a global offset table, which is the mips way of supporting dynamic relocation (all code references the GOT, the dynamic loader relocates the values in the GOT). Several attempts resulted in mystifying lock-ups, and tempting as it was to continue investigating, especially since I had come rather a long way down this path, I decided to revert to static linking.

Proceeding now:
- Python inside Mungi build process; as component.
- Component interface development.

Sat Sep 28
- Working on 32-bit elf loader. The "unable to kmalloc" was because the builtin elf loader was for 64-bit elf files only (!) but didn't check the ELF file type when loading - so it happily started loading the elf32, but ended up reading 32 bits too many for the image size, which resulted in an attempt to kmalloc an altogether-too-large area of memory. ;)

Tues Sep 16 cont'd
Fexec stuff

- Added include to env.h
- Added "loadexec" to conf.
- Set up Python to be added as a raw file to the end of the DIT.
- (6:45pm)Encountered problem with padding in dite (same problem as before; sulima "bails out")
- (7pm)Problem turns out to be size-related: anything larger than about 1.2 megabytes stored RAW in the dit fails.
- (7:30pm) Problem turned out to be with Sulima configuration: size of ROM was set to 2mb, which was not sufficient. This didn't show up before because dite strips unneeded segments from ELF files when it's processing them as ELF files (but I am now inserting the files raw).
- (8:30pm) Now unable to find files in the dit.
- (11pm) Using dit_find_file, now load_elf breaks (unable to kmalloc). Interesting... probably time to work on my graphics assignment though.

Comments from progress report:
	- First to port "real" (as in large) application to Mungi; chapter on howto would be good (not to mention experiences)
	- Benchmarks? Python on mungi vs Python on Linux, same hardware.


Tue Sep 16 16:44:19 EST 2003

Thesis write-up. Blah. :)

Python runs on CVS Mungi rather than a snapshot, which is ... kinda nice... in a way... to be honest I'd prefer not to have to hit a moving target but at least this way I have the option to upgrade easily if necessary. Ditto Pistachio.

Tue Sep  9 00:50:07 EST 2003
A little hiatus here. Python now runs on Mungi and is capable of executing simple (and not-so-simple) programs encoded into the executable. Next steps are to get "import" working within python, which requires a nice (at least a little bit nice) way of packing data into the dit.

Did some more work on actual thesis writeup; URL forthcoming.

Mon Aug 18 21:08:56 EST 2003
Sulima now boots on my Mac and presumably other big-endian systems. Yay!

Technically, there was very little to the port. The ROM was being byteswapped so I stopped byteswapping it, and there was some work to change the endianness of the main memory which I simply removed. Various other things such as all the devices save the serial device I simply haven't tested.

Also a couple of minor modifications to get the code compiling on a system with a BSD libc - I'm still nervous about the change I made to BSD pty support because I know next to nothing about how it's supposed to work.

Mon Aug 18 00:29:04 EST 2003
Got all of the stuff required to build Mungi compiling and working on my Mac. Unfortunately the simulator, Pat's opus "Sulima", isn't so easy because it does not support big-endian hosts. Spent some time changing that and now get it up to the early part of L4 booting. I shouldn't spent much time on it but the problem is now sufficfiently frustrating to me that I would like to see it through.

Thu Aug 14 21:51:29 EST 2003
Lots of good stuff.
	- Worked with various people (benno, cgray, ab) to iron out various loader problems in order to get Python running from within mungi. Basically the procedure is yucky right now, because the "build outside Mungi" thing isn't 100% done:
		1) Build Mungi
		2) Examine the dit image and find a suitable address after all the stuff in the image
		3) Hard-link Python to run from this address
		4) Manually insert Python into the dit.
		5) re-run "make" in the Mungi directory. This will notice that the dit image is more recent than everything else, and will recreate the bootimg from the dit.
		6) Now run sulima and everything should work fine.

The problem I had was that the link location initially suggested by cgray was a little too early on in the dit. The problem was that:
	- The final file in the dit is "conf"
	- This is stored as a raw file in the dit
	- Raw files are padded to the nearest 16kb by dite.
	- Dite apparently packs the files in reverse order.

The upshot of this is that Python was being added but the first few K of it were being clobbered by the extensive padding assigned to the next file up in the dit image, 'conf'. Moving Python further down in memory (by relinking it) solved the problem.

Now there are various issues to do with the fact that lots of bits of the C library that I was relying on are stubs. :)

Also, after much pain, I managed to get a cross-compiler built under OS X for MIPS. I will explain how on a Web page.


Wed Aug  6 19:42:33 EST 2003
We have linkage. :-)

I now have a MIPS program, 'python', with which I can do very little for the moment. The build system looks a little hacked-up at this point. Noticeable features:

	* Implemented a bunch of functions from math and stdlib as stubs. I compile these files into a library named "libpungistubs.a" in the root Python build directory, and then link those files against libpython.a. 
	* A couple of bits of Mungi don't get included in any library after linking. This means that building external programs requires adding a bunch of .o files into the build from $(MUNGIROOT)/build/*/, including one MIPS-specific one, which is a bit nasty. 
	* Removed dynamic module loading support (dynload_stub)
	* Made stubs for a few Python functions that simply had to be there - to do with enabling,disabling and handling signals for example. They are part of libpungistubs, for no good reason.
	* Use the Python strtod implementation (Hmm, I wonder what else it has...)
	* Build a special "host only" version of the parser generator, pgen.

Now to find out how to make an app and put it in the dit image.

Tue Aug  5 17:20:28 EST 2003
35 unique undefined references. They fall into the following categories:
	1 - libm failures
	2 - Parts of Mungi not in the standard Mungi libraries libmlib and libmungi
	3 - Various Python failures.

For 1, I need to implement various libm functions. Following Simon's suggestion I will simply do this using stub functions for now.

For 2, the failures are with various Mungi functions that wrap l4 functions - the "platlib" (?) functions. Ab and Balial's explanation is that these functions should actually be part of libmlib, but aren't. For now I will explicitly link against the object files containing these functions, but eventually the Mungi developers should incorporate the functions into one of the .a files. This situation arises because most Mungi applications are built within the Mungi build system, so the problem has gone unnoticed.

For 3, the failures are for various things that I simply don't want - signal handling etc. Once again stub functions should suffice for this.

Tue Jul 29 13:32:12 EST 2003

Passed the buck all the way down to the linker. Unfortunately I'm now at the "link" stage. Pulled out various things - apparently Python is linked with C++ (ie g++) if C++ is available. It is, but without libstdc++ we're not going to get anywhere, so we now link with C (ie gcc). Last week I hacked in some bits from math.h; this week I need to make a libm. :-)

Pulled out libutil wholesale - this is sure to cause hilarity later.

Python is annoying in that everything it uses is included by one magical include file, Python.h, and then other modules just include this file. This is good for them because they are free to implement emulation for various include files which aren't present on certain systems, but slightly annoying for me because it effectively adds a huge amount of global functions to every module, making finding out which module uses what includes more tricky. 

Finally did an actual link and got 1363 undefined references to things. Considering that I wasn't linking in any standard libs (so no malloc, str* etc) this is actually rather compact. 

Fri Jul 25 13:41:08 EST 2003
Learned a fair bit more about make. So-called "static pattern rules" allow the use of a specific set of compilation instructions for multiple targets. I was going to use this technique to build pgen using the host compiler and the rest using the cross-compiler, but have decided against it. Static pattern rules, courtesy http://www.cs.bu.edu/teaching/c/makefiles.html :

OBJECTS: something.o somethingelse.o
$(OBJECTS): %.o: %.c
	$(CC) -c $(CFLAGS) $< -o $@

The highly intuitive $< and $@ means "name of dependency" and "name of target". 

Am planning on using "recursive make" to generate the host build files. Read the paper "Recursive Make Considered Harmful" at http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html which, after some initial interesting work seems to degrade into an advertisement for gcc (!?). Incidentally a "recursive make" is not recursive in either sense - make doesn't recurse, it executes a child, and the Makefile doesn't recurse, a totally different makefile is read in. Either way, ensuring that host files are built first doesn't seem to attract any of the problems that are outlined in the paper.

=-=-=
Tue Jul 22 14:38:21 EST 2003
Getting closer to at least finishing the compile process, but two programs are built by Python during the build process, one to generate the parser (pgen) and one to tokenize and install modules (python itself). So I am working around that.
=-=-=

Thu Jul 17 17:04:45 EST 2003

Building a rudimentary Python for the sim. It's rather depressing; most of the build process so far has consisted of going through pyconfig.h and undefining stuff. Python build process after I get past the autoconf crap seems remarkably clean. Hopefully I can port it to Mungi without stuffing this up too much. :-)

=-=-=-=-=

Short guide to porting apps so far, courtesy Balial (cgray@cse):

1) You need a small patch to the Mungi makefile to create an 'extern.mk' target. "make extern.mk" produces extern.mk which among other things defines a target which includes all the paths to various include files in Mungi. There are heaps of them. :(

2) Add a couple of lines to the beginning of the Makefile for the port:
### Mungi Hack!
MUNGIROOT := <PATH TO MUNGI ROOT>
include $(MUNGIROOT)/extern.mk

3) Ensure that the compilation flags include:
CFLAGS := .... $(MCFLAGS)
INCLUDES := .... $(MINCLUDES)
LDFLAGS := .... $(MLDFLAGS)

where MCFLAGS, MINCLUDES and MLDFLAGS are defined in extern.mk which we included above.

4) Cross fingers, type make and wait for things to break. I encountered a problem where Mungi/lib/libc/std_include/machine/limits.h (included from ../limits.h) wasn't getting the definition of UINTPTR_BITS; putting #include <archtypes.h> at the top of the file seemed to work without too much ugliness. Perhaps there are more of these problems.

=-=-=-=

Now all I need to do is prevent Python from attempting to compile in, for example, socket support...