Better *is* better and worse is worse

I’ve read the original Worse Is Better article by Richard Gabriel a couple of times now. I discovered it via my interest in programming languages (Lisp in this case).

Jim Waldo reminds us that Better really is *better* in Artima article Worse is worse.

Movitz – a Lisp OS platform

Wow. You hear alot of rumours about Lisp-based operating systems but you rarely come across a project that appears active. Movitz looks like a promising *active* LispOS project! It’s goal is to provide a development platform to run CL-based kernels on x86 PCs “on the metal”. I guess these restrictions make it doable. I’d prefer to see CL or Scheme combined with Linux or FreeBSD kernel so that you could still have a viable usable system and also experiement with OS ideas using lisp. In this respect, Schemix looks interesting. It’s a Scheme interpreter (based on TinyScheme) patched into the Linux kernel. I haven’t looked at it closely but imagine that it’s interpreted nature means that it is probably only useful for early prototyping of ideas and certainly for exploring and learning about the kernel.

WCL and shared libraries

Just read about WCL, a Common Lisp (CL) implementation I wasn’t aware of. The paper talks about how CL is compiled to C and linked into a shared library. This allows a memory efficient delivery environment. i.e. CL application share code via shared libraries including the core system/libraries. I missed whether the CL compiler is available at runtime which would be a drawback. Many problems were solved but still afew remained such as relocated data with embedded pointers in the shared library (causing slower startup times), generational GC is not implemented, the compiler could be more sophisticated and there is no thread support. The project appears to be stalled.

I wonder whether other CL implementations such as GCL and ECL using the CL->C method are able to provide sharing through shared libraries?

Java programs have similar problems that WCL attempts to solve for CL. When the same Java program is loaded by separate JVMs (in different processes), they don’t share any code. i.e. the classes will be jitted multiple times …being stored in memory multiple times. I believe that this problem occurs even within the *same* JVM when the class files are loaded via different class loaders! I’ve yet to confirm that though. This is one of the reasons why Java application take up alot of memory. Seems to be a problem with 1.4.2 anyways. Perhaps 1.5 fixes this problem. Microsoft’s CLR deals with this situation by using the assembly is the unit of deployment and having each assembly contain a version number. I imagine that with the CLR (and other CLI implementations), that each assembly is only compiled once within the same process. This doesn’t however solve the “multiple process runnings CLRs” problem though I believe there is an ahead-of-time option which stores compiled assemblies in an on-disk cache – this would would likely solve the problem (as long as the compiled assemblies are loaded into shared memory like a shared library).

CPU Scheduling

I’m slowing working my way through the cs162 lectures from Berkeley. Just watched the CPU Scheduling lecture. I had the realisation that CPU scheduling is “just” resource sharing (der), like shoppers sharing the checkout operators on their way out of the store. Alan talked about optimal scheduling algorithms for “perfect” situation. I think it was FIFO was provably optimal when considering time to completion and SRTF (shortest running task first?) was provably optimal when considering average response time. FIFO works when all jobs are the same length, SRFT is more general but you have to guess about how long a job will run. I had the thought that if there are optimal schedulers for certain situations then wouldn’t it be great to be able to specify what scheduling algorithm to use for certain processes? You could even allow user-designed schedulers to which you assign your jobs. Then you could have a heirarchy of schedulers and a super-scheduler that moves jobs/tasks between the different schedulers…. or you could use the lottery algorithm that Alan talked about towards the end of the lecture. It has nice properties like avoiding starvation (in particular of long running tasks) and easy to understand fairness. Anyways I still think it would be great to have a Lisp-based operating system, perhaps as a layer above Linux (cause you don’t want to write all those device drivers do you) that would allow experimentation with custom scheduling algorithms and heirarchies of schedulers.

The other potentially interesting thing about a LispOS is that you could do away with reserving stack space for each thread. One of the problems with having lots of processes (and threads) is that it consumes alot of memory even if each thread only gets afew kilobytes it mounts up quickly. With Lisp you needn’t use the stack to hold “procedure activations”, you could put them in the garbage collected heap (actually you might want to optimise that a bit and put them in a special “stack” heap and only move/link them into the gc-heap if necessary). This way threads only take as much “stack” as they need. Also “stack space” grows as required (just as the heap grows as required). Potential problems would be efficiency of the gc-heap – particularly memory allocation speed – and slightly larger activation frames due to embedded pointers.

The Python Paradox

http://www.paulgraham.com/pypar.html

I learnt Python afew years ago. I’ve moved onto Ruby for day to day tasks and am studying Scheme and Common Lisp when I can spare a moment.

Today I was watching another couple of videos from the Abelson and Sussman collection. It’s about 9G of divx movies. I’m pretty dedicated to my lisp education – I’m only on 32k dialup connection. I watched 6a&b about streams (again) and then got onto the very amusing lectures on meta circular evaluation (7a&b). Very funny – a must see for any lisp enthusiast.

I was also reviewing multithreading issues in Common Lisp implementations via the slides from Roger Corman’s talk on the subject.