pagetable.com

Playstation 3 Hacking – Linux Is Inevitable

Michael Steil — Tue, 07 Sep 2010 07:45:02 +0000

In the talk “Why Silicon Security is still that hard” by Felix Domke at the 24th Chaos Communication Congress in 2007 (in which he described how he hacked the Xbox 360, and bushing had a cameo at the end explaining how they hacked the Wii), I had a little part, in which I argued that “Linux Is Inevitable”: If you lock down a system, it will eventually get hacked. In the light of the recent events happening with PlayStation 3 hacking, let’s revisit them.

This is the original slide from 2007:

device	y	security	hacked	for	effect
PS2	1999	?	?	piracy	-
dbox2	2000	signed kernel	3 months	Linux	pay TV decoding
GameCube	2001	encrypted boot	12 months	Homebrew	piracy
Xbox	2001	encrypted/signed bootup, signed executables	4 months	Linux Homebrew	piracy
iPod	2001	checksum	<12 months	Linux	-
DS	2004	signed/encrypted executables	6 months	Homebrew	piracy
PSP	2004	signed bootup/executables	2 months	Homebrew	piracy
Xbox 360	2005	encrypted/signed bootup,encrypted/signed executables, encrypted RAM, hypervisor, eFuses	12 months	Linux Homebrew	leaked keys
PS3	2006	encrypted/signed bootup,encrypted/signed executables, hypervisor, eFuses, isolated SPU	not yet	-	-
Wii	2006	encrypted bootup	1 month	Linux	piracy
AppleTV	2007	signed bootloader	2 weeks	Linux	Front Row piracy
iPhone	2007	?	1 month	Homebrew international	SIM-Lock revenue

The table shows the relationship between the quality of a device’s security system and the time it took to hack it, as well as the original motivation for hacking and the side effects (collateral damage) it caused.

Correlation security/time to hack

There is a pretty clear correlation betwen the quality of the security system and the time required for hacking it – with the notable exception being the GameCube, which had rather weak security, but since its release coincided with the much more powerful Xbox, much of the hacker community neglected the GameCube until the Xbox was done. What can also be seen is that recently, devices tend to get hacked more quickly; probably simply because there are more and more people interested in hacking.

Correlation Linux/time to hack

The other exception is the PlayStation 3, which was not hacked until about three and a half years after its introduction. I argued that this was because there was only very little motivation to hack it: Sony shipped the devices with the “Other OS” option and even sponsored a port of Linux to it, allowing any user to install Linux if they wanted. Although Linux was running on top of a hypervisor and did not have access to all of the features of the device, it seems to have been enough to take the enough motivation to hack it out of the hacker community.

Linux/homebrew is the primary motivation

This is supported by the by the fact that the motivation for hacking every system in the table was either homebrew (i.e. running unautorized hobbyist applications) or Linux. Hackers seem to love to convert their devices into Linux computers to run a big library of existing software, or to hack the device to make it possible to run versions of existing emulators and games on the native OS.

Piracy is a side effect

None of the hacks in the table was done with the motivation to allow running copied games – but whenever the point of the security system was to prevent piracy, hacking it inevitably enabled piracy as a side effect. Some security systems protected other things like pay TV keys and SIM-locks; these also fell as side effects.

2010 update

In September 2009, Sony started shipping the “slim” model of the PlayStation 3, with the “Other OS” feature removed. With firmware 3.21 in April 2010, the feature was also removed from existing original models that users chose to upgrade – which was required for using any of the online features. The missing “Other OS” feature on the slim model motivated George Hotz (geohot) to hack into hypervisor mode (Jan 2010), but this approach did not lead to a working hack of the security system. In August 2010, the Australian company OzMods announced the commercial “PSJailbreak” USB dongle that hacks into non-hypervisor mode, allowing piracy and homebrew (“Backup Manager” says “backups and homebrew”).

Although this is the first time that a commercial company is first to hack a system, and the first time that piracy seems to have been a key motivation, removal of “Other OS” might have been another motivation, and geohot’s previous attempts might have helped as an entry point for this hack.

Usually, an open hacker community develops a hack, and commercial companies convert them into modchips. This time, a company developed a hack and a modchip, and the community reverse engineered it and ported the exploit code onto several other devices, allowing people to hack the PlayStation 3 without a dedicated device. And I’m sure Linux will be adapted soon to run in the new environment.

Conclusion

What do we learn from this? Linux is inevitable. Or maybe it should be “Homebrew is inevitable”. In the history of mankind, there has yet to be a popular system that is locked down to only allow certain software to run, but does not get hacked to run arbitrary code. I still dare to say that if Sony had not removed “Other OS”, the PlayStation 3 would have been the first system to not get hacked. At all.

(Here is an updated 2010 version of the table:)

device	y	security	hacked	for	effect
PS2	1999	?	?	piracy	-
dbox2	2000	signed kernel	3 months	Linux	pay TV decoding
GameCube	2001	encrypted boot	12 months	Homebrew	piracy
Xbox	2001	encrypted/signed bootup, signed executables	4 months	Linux Homebrew	piracy
iPod	2001	checksum	<12 months	Linux	-
DS	2004	signed/encrypted executables	6 months	Homebrew	piracy
PSP	2004	signed bootup/executables	2 months	Homebrew	piracy
Xbox 360	2005	encrypted/signed bootup,encrypted/signed executables, encrypted RAM, hypervisor, eFuses	12 months	Linux Homebrew	leaked keys
PS3	2006	encrypted/signed bootup,encrypted/signed executables, hypervisor, eFuses, isolated SPU	4 years	Piracy Homebrew	-
Wii	2006	encrypted bootup	1 month	Linux	piracy
AppleTV	2007	signed bootloader	2 weeks	Linux	Front Row piracy
iPhone	2007	signed/encrypted bootup/executables	11 days	Homebrew SIM-Lock	piracy
iPad	2010	signed/encrypted bootup/executables	1 day	Homebrew	piracy

Why is there no CR1 – and why are control registers such a mess anyway?

Michael Steil — Sat, 03 Jul 2010 05:43:40 +0000

If you want to enable protected mode or paging on the i386/x86_64 architecture, you use CR0, which is short for control register 0. Makes sense. These are important system settings. But if you want to switch the pagetable format, you have to change a bit in CR4 (CR1 does not exist and CR2 and CR3 don’t hold control bits), if you want to switch to 64 bit mode, you have to change a bit in an MSR, oh, and if you want to turn on single stepping, that’s actually in your FLAGS. Also, have I mentioned that CR5 through CR15 don’t exist – except for CR8, of course?

Like many (but unfortunately not all) quirks of the i386/x86_64 architecture, this mess can be explained with history.

8086 – FLAGS

x86 history typically starts with the 16 bit 8086, but although it was not binary compatible with its predecessor, it was nevertheless a rather straightforward assembly-level compatible 16 bit extension of the 8 bit Intel 8080 with some ideas of the Zilog Z80. The 8086 is still a classic “home computer class” CPU, which was not meant for modern operating systems: It had no MMU of any kind, and no concept of privileged and unpriviliged modes. Therefore, control bits that we see as system state today were encoded into the 16 bit FLAGS register: The interrupt enable bit and the trap flag (which will cause a software interrupt after the next instruction and thus lets you single-step) are encoded into FLAGS right next to the ALU’s flags like Zero and Carry.

80286 – Machine Status Word

The 80286 then came with a simple form of memory management that allowed more sophisticated (but not yet “modern”) operating systems to run – like the original versions of OS/2. The 16 bit “Machine Status Word” was created to host the big switch between legacy mode (real mode) and the new memory-managed mode (protected mode) and a program could access it using the new instructions “lmsw” and “smsw”. The 80286 had more system state than just this bit: The GDT, the IDT and the TSS had its own registers and dedicated instructions to access them (“lgdt”/”sgdt”, “lidt”/”sidt”, “ltr”/”str”)

i386 – Control Registers

The i386 finally had a real MMU that allowed paging and thus modern operating systems. The MMU required two more registers in the system state, one for the base address of the pagetables, and one to read a fault address from. Intel decided against adding more special purpose registers with dedicated accessor instructions, but instead introduced eight indexed 32 bit wide “control registers” CR0 to CR7. The new accessors “mov crn, r32“/”mov r32, crn” allowed copying between registers and control registers and had the 3 bit CR index encoded in the opcode.

The old MSW was also wired into the lower 16 bits of CR0; but CR0 was also extended with new bits like the switch to turn on paging. CR1 was kept reserved, presumably as a second control register for miscellaneous control bits, and CR2 and CR3 were used for the aforementioned fault address and pagetable base pointer. The opcodes to access reserved control registers generated an “invalid opcode” fault, making it possible for Intel to reuse the opcodes later if they don’t use the control registers.

i486 – CR4

The i486 added a few more control bits, and some of them went into CR0. But instead of overflowing the new bits into CR1, Intel decided to skip it and open up CR4 instead – for unknown reasons.

Pentium – MSRs

On the Pentium, Intel added for the first time control bits that were a property of the implementation as opposed to the architecture, i.e. bits that are microarchitecture-specific and will therefore only work on certain CPUs and not necessarily be supported on later CPUs – like caching details and debug settings. In order not to waste the valuable CR space with throw-away control bits, Intel introduced the Model-specific Registers (MSRs). The MSR address space is 32 bits, and every MSR is 64 bits wide. The two new instructions “rdmsr” and “wrmsr” copy between an EDX-indexed MSR and the EAX register.

Pentium II – SYSENTER MSR

The SYSENTER instruction that got introduced on the on the Pentium II is a fast way to switch between unprivileged and privileged mode. Instead of looking up the destination segment, instruction pointer and stack pointer in memory, the CPU holds this information in three special-purpose system registers. CR space is valuable, so Intel decided against filling up CR5, CR6 and CR7, so they put it into the MSR address space instead – at 0×174 through 0×176. This was practically an abuse of the MSR concept.

AMD K6 – EFER MSR

Who can blame AMD for doing similar things then? With the K6, which was introduced at the same time as the Pentium II, AMD diverged from just copying Intel for the first time and actually added features of their own: They added the SYSCALL instruction, and with it, a control bit that turns it on and off, and an extra control register with the target location. Being afraid to collide with Intel extensions they they didn’t know about, they put the extra system registers into the MSR space: the control register “EFER” (Extended Feature Enable Register) at 0xC000_0080 and the Syscall Target Register (STAR) at 0xC000_0081. Intel had been nicely lining up MSRs counting up from 0, so AMD decided to start counting at 0xC000_0080. Understandable as this is, it is basically the same abuse of the MSR concept as Intel’s with SYSENTER.

A very similar thing happened in the CPUID space, by the way: While Intel encoded all its feature bits in leaf 0x0000_0001, AMD defined leaf 0x8000_0001 for its features.

x86_64 – Chaos!

So far everything looked like it was getting a little more controlled. Both Intel and AMD are only adding new control registers in the MSR space, and since this is a big address space and AMD and Intel extend it on rather opposite locations, it all looks nicer. But then came x86_64: For the first time, Intel was copying a feature that AMD introduced, and it needed to be compatible with all its details. AMD had encoded the availibility of x86_64 in its own CPUID leaf in 0x8000_0001, so Intel had to support this leaf as well. And since Long Mode was turned on in the EFER MSR, Intel had to support an MSR in the AMD space of 0xC000_0000. Long mode also required supporting SYSCALL, so Intel also supported the STAR MSR.

Since x86_64 introduced the REX prefix to double the number of available general-purpose registers, AMD decided to allow this prefix also for “mov cr”, doubling the number of control registers and therefore introducing CR8 through CR15 – also doubling their width. And since AMD introduced them, they owned them, and decided to use CR8 for the “Task Priority Register” feature.

VMX and SVM

The architecture is messy, sure, but does it matter? Maybe not… as long as CPUs didn’t have virtualization extensions! Both Intel VMX and AMD SVM are designed so that they can automatically switch the complete privileged machine state including control registers and certain MSRs. Intel for example special cases CR0, CR3, CR4 and CR8, leaves CR2 to the user. AMD on the other hand has 16 fields for all CRs in its switcher. And because of the two different starting points of the MSR space, Intel VMX required a whitelist bitmap for 8192 MSRs starting at 0x0000_0000 and for another 8192 MSRs starting at 0xC000_0000 – and of course SYSENTER_CS, EFER, STAR and friends are special-cased. If you want to have a lot of fun, read the VMCS layout reference of Intel’s manual 3B!

Future?

CR1 and CR5 to CR7 are still “owned” by Intel. AMD has shown that they don’t want to use them – and even Intel has not added a control register since 1989.
CR9 through CR15 are technically owned by AMD, since they introduced them with x86_64 and decided to use CR8. Intel adopted the reserved ones when adopting x86_64, but it is unlikely that Intel will ever adopt smaller changes to the architecture from AMD, and AMD is unlikely to use them if they won’t be part of the architecture, so these will probably never be used either. On the other hand, AMD added these to the auto-switcher list of their SVM Virtual Machine Control Block (VMCB), showing that they haven’t given up on them yet.
The MSR space is properly de-facto partitioned. Intel continues adding MSRs at 0 and AMD at 0xC000_0000 – but MSR have already lost their model-specificness in 1997. MSRs are the new CRs.

Dear Intel, dear AMD: I like the control registers, and I hate to see them wasted. Why don’t you finally define CR1 and give it a few control bits in the future? If you’re scared about collisions, I will be happy to be the arbiter. Ah, whatever: Intel, you get to define all even bits in CR1, and AMD, you get to define all odd bits. Okay? Cool.

High-Res Pictures of a MOS KIM-1

Michael Steil — Tue, 29 Jun 2010 06:10:25 +0000

The MOS KIM-1 is a quite rare collectorâs item today. So if you hold one in your hands, you better take some high resolution pictures of the board. Here they are:

Note that this is the original revision of the board (pre-Rev A), and the 6502 CPU is from week 51 of the year 1975 – so it has the ROR bug!

Does anyone know what the three digit numbers 002 and 003 on the 6530 RIOTs mean? Are these the indexes of the ROM images? If so, what is ROM #001 and was there a #000? Also, the back has the number “0372″ on it – is this a serial number? Looking at the dates of the chips, this seems to be the oldest KIM-1 of all those I could find on the internet.

How much change is in a vending machine?

Michael Steil — Thu, 24 Jun 2010 03:56:25 +0000

There is only one way to find out – all you need is a giant pile of money and a vending machine that sells soda for $1.25: If you put in a dollar note and press the “return change” button, you will get the dollar note back directly. If you put in two dollar notes (the maximum it takes) at a time, it will give you change for the two dollars.

Our machine had $29 of change. It started out with 25¢ coins, and after a while returned a combination of different coins (25¢ & 10¢, then 10¢ & 5¢), finally switching to all 5¢ coins at the end.

In total, it had

25¢ x 57 = $14.25
10¢ x 114 = $11.40
5¢ x 67= $3.35

After this, the machine refused to take bills, so it was either full with bills or, more likely, out of change.

But actually, it might still contain up to 95¢ of change – today’s homework is to find out how to measure the remaining change in the machine!

Intel VT VMCS Layout

Michael Steil — Mon, 24 May 2010 21:58:24 +0000

I understand that there might be a good reason for Intel to add virtualization extensions to their CPU architecture. Instead of fixing the x86 architecture to (optionally) make it Popek-Goldberg compliant and have all critial instructions trap if not run in Ring 0, they added non-root mode, a very big hammer that allows me to switch my CPU state completely to that of the guest and switches back to my original host state on a certain event in the guest. Well, it’s a great toy for people who want to play with CPU internals.

Therefore Intel had to add the VMCS, a 4 KB block in memory that holds the complete CPU state of both the host and the guest (segment registers, GDT and IDT pointer, certain MSRs etc.) as well as some control bits (for example, when to exit).

I also understand that Intel doesn’t allow me to just read and write memory in the VMCS, but abstracts accessing the virtualization state using a vmread/vmwrite interface. This way, the actual layout of this 4 KB page is an implementation detail and can be changed on later CPUs. It also allows for field indexes that are more spread out and encode what kind of field it is.

So I understand very well why Intel encodes into the VMCS field index whether it’s a control field (0), a read-only field (1), part of the guest state (2) or part of the host state (3), and whether it’s a 16 bit (0), 32 bit (2), 64 bit (1) or native-sized (3) field. This way, for example, all 16 bit guest state fields (like the guest’s CS) have indexes starting at 0×0800, and all 64 bit host state fields (like the hosts’s EFER MSR) start at 0x2C00.

Now what I don’t understand is what is so hard to be consistent with this convention (Intel Manual 3B, Appendix H).

VMCS Link Pointer (0×2800): In the first revision of VT, it had already been already decided that there should be a mechanism for having a second 4 KB page in case later versions of VT need more than 4 KB of state. For this, there is there “VMCS Link Pointer”, which is a 64 bit physical address. Guess what category this belongs to? Guest state.
“Guest Address Space Size” bit in the “VM Entry Controls” Field (0×4012): This is clearly guest state and not a control field.
“Host Address Space Size” bit in the “VM Exit Controls” Field (0x400C): This is clearly host state and not a control field.
VMX-preemption timer value (0x482E): This timer controls after how many ticks execution of the guest should end and control should be returned to the hypervisor. Intel put this into the “guest state” bucket: All other guest state fields are properties of the i386/x86_64 architecture that need to be switched, but not this one. This should really be a control field.

And here is another favorite of mine: the “Primary Execution Controls” field. The 32 bits specify which events in the guest will exit guest execution and trap into the hypervisor (Table 21-6). These events are, among others:

exit on HLT
exit on INVLPG
exit on MOV CR3
exit on PAUSE

Setting these bits to 1 enables the traps. So if you set all bits to 0, you basically have an unrestricted guest, and if you set all bits to 1, you have the most controlled guest, and you get a notification about every event in the guest. Or so you might think. Actually, there are two bits in the field that don’t work like this:

Use MSR bitmaps
Use I/O bitmaps

If these bits are set to 1, it checks a whitelist whether a certain MSR or I/O access is possible. If they are set to 0, all MSR and I/O accesses trap. Compared to all other bits, that’s backwards. Oh great.

Since Steve Jobs seems to be happy to explain his personal opinion on everything lately, I wrote him an email asking him about this, and he replied:

Return-path: 
Received: from bulkin002-bge351000.mac.com ([unknown] [10.150.69.129])
 by ms231.mac.com
 (Sun Java(tm) System Messaging Server 7u3-12.01 64bit (built Oct 15 2009))
 with ESMTP id <0L2X00HTAZ3Q6GF1@ms231.mac.com> for XXX@mac.com; Mon,
 24 May 2010 13:47:50 -0700 (PDT)
Original-recipient: rfc822;XXX@mac.com
Received: from relay13.apple.com ([17.128.113.29])
 by bulkin002.mac.com (Sun Java(tm) System Messaging Server 6.3-7.02 (built Jun
 27 2008; 32bit)) with ESMTP id <0L2X001EVZ3QKED0@bulkin002.mac.com> for
 XXX@mac.com (ORCPT XXX@mac.com); Mon, 24 May 2010 13:47:50 -0700 (PDT)
X-AuditID: 1180721d-b7c17fe00000693e-19-4bfae5f6545a
Received: from [17.201.27.84]
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)	by relay13.apple.com (Apple SCV relay)
 with SMTP id DB.14.26942.6F6EAFB4; Mon, 24 May 2010 13:47:50 -0700 (PDT)
From: Steve Jobs 
Content-type: text/plain
Content-transfer-encoding: 7bit
Subject: Re: Intel VT VMCS Layout
Date: Mon, 24 May 2010 13:47:48 -0700
Message-id: <3E789F1B-7E13-FFD2-80F6-8E8D4CDDE7FB@apple.com>
To: Michael Steil 
MIME-version: 1.0 (Apple Message framework v1077)
X-Mailer: Apple Mail (2.1077)
X-Brightmail-Tracker: AAAAAQAAAZE=

The whole VMCS is a big mess, I hate it.

> Hi Steve, what do you think about the ordering of the VMCS fields in
> Intel's VT extenions?
>
>   Michael

Michael Steil rocks

Michael Steil — Wed, 05 May 2010 04:19:37 +0000

Google is always right.

Exercise for the reader: Change it.

Microsoft vs. Standards

Michael Steil — Fri, 23 Apr 2010 03:04:29 +0000

Here is a fun game for long car rides: One person names a respected standard implemented by dozens of IT companies, and the other person names Microsoft’s competing technology. Example: MPEG Audio (MP3/AAC) – Windows Media Audio.

(When you have run out of examples, you can try this game with other major players in the IT business.)

Let’s play this game in the comments to this blog entry: Just add as many of these pairs that you can think of – extra points if Microsoft’s technology has a closed specification.

Who invented the computer?

Michael Steil — Tue, 20 Apr 2010 19:24:02 +0000

In 1837, Charles Babbage designed a general purpose computer, the Analytical Engine, but never built it.
Between 1934 and 1937, Church, Turing et al. defined the general purpose computer, but didn’t design one.
In 1941, Konrad Zuse built the first general purpose computer, the Z3, but didn’t know it was general purpose and didn’t use it that way.
From 1943 to 1946, Mauchly and Eckert finally built a computer, ENIAC, that was designed to be general-purpose.

Standards and Intellectual Property

Michael Steil — Sat, 10 Apr 2010 00:32:19 +0000

I am sitting here, working with my PC: My keyboard and my mouse are connected wirelessly via Bluetooth and my monitor is hooked up through DVI. The graphics card is sitting in a PCI slot, main memory is DDR-SDRAM, and my optical drive can do CDs and DVDs. While my internal hard disk speaks the SATA protocol, my home directory is actually sitting on an SD card that is connected through a USB reader. My internet connection is done through DSL. On the software side, I am using GNU/Linux and browsing the internet with Firefox. No way I would ever watch a video in H.264 format.

Buggy Drivers

Michael Steil — Thu, 01 Apr 2010 22:21:56 +0000

Announcement: ‘libcpu’ Binary Translator

Michael Steil — Tue, 29 Dec 2009 19:47:09 +0000

I just did a Lightning Talk at the 26th Chaos Communication Congress 26C3 about our new project “libcpu”, and it has already been picked up by Golem.de and reddit.com, so I might as well announce it here:

“libcpu” is an open source library that emulates several CPU architectures, allowing itself to be used as the CPU core for different kinds of emulator projects. It uses its own frontends for the different CPU types, and uses LLVM for the backend. libcpu is supposed to be able to do user mode and system emulation, and dynamic as well as static recompilation.

Here are my slides: 26C3-libcpu.pdf

Here is the video recording: 26C3-libcpu.mp4

Having Fun with Branch Delay Slots

Michael Steil — Mon, 23 Nov 2009 02:56:08 +0000

Branch Delay Slots are one of the awkward features of RISC architectures. RISC CPUs are pipelined by definition, so while the current instruction is in execution, the following instruction(s) will be in the pipeline already. If there is for example a conditional branch in the instruction stream, the CPU cannot know whether the next instruction is the one following the branch or the instruction at the target location until it has evaluated the branch. This would cause a bubble in the pipeline; therefore some RISC architectures have a branch delay slot: The instruction after the branch will always be executed, no matter whether the branch is taken or not.

So in practice, you can put the instruction that would be before the branch right after the branch, if this instruction is independent of the branch instruction, i.e. doesn’t access the same registers. Otherwise, you can fill it with a NOP. Out-of-order architectures can do this reordering at runtime, so there would be no need for a delay slot. Nevertheless, the delay slot is a feature of the architecture, not the implementation.

Some RISCs like PowerPC and ARM do not have a delay slot, but for example MIPS, SPARC, PA-RISC have it. But there are some variations: MIPS and PA-RISC have an annihilation/nullify/likely bit in the instruction, so the programmer can choose that the instruction in the delay slot only gets executed if the branch is taken.

Other CPUs, like SPARC, PA-RISC or the ill-fated Motorola M88K have optional delay slots: The programmer can set a bit in the opcode if he cannot come up with a good instruction for the delay slot, and save the wasted NOP in the program code this way – the CPU will put a bubble into the pipeline.

Now the interesting question is what happens if the branch and the delay instruction are not independent. What if the delay instruction writes r5 and the branch jumps to r5? What if it’s a branch-and-link, and the delay instruction modifies the link register? On MIPS, this is illegal, and undefined.

In practice, MIPS won’t halt and catch fire though. As you would expect from the design of a CPU pipeline, the CPU basically executes the branch and the delay instruction in order, as they are stored in the instruction stream, and it only delays the write to PC, i.e. the actual jump until after the delay instruction. So, for example, if you modify a register that the branch depends on, it will not influence the branch, but be in effect after the jump.

On the aforementioned Motorola M88K, this behaviour is documented, and GCC even makes use of it:

820:   7d ad 00 08   cmp   r13,r13,0x08    ; compare

824:   d4 6d 00 05   bb0.n 0x03,r13,0x834  ; cond. branch
828:   63 df 00 00   addu  r30,r31,0       ; delay slot

82c:   cc 00 00 7f   bsr.n 0xa28           ; function call
830:   60 21 01 ac   addu  r1,r1,0x1ac     ; delay slot: fix up link

834:   00 00 00 00   nop

The first three instructions are a compare/branch sequence. If the branch is taken, execution will continue at 0×834, otherwise at 0x82c, after the delay slot. The delay instruction is independent of the branch, nothing special here yet.

But now look at the following two instructions: 0x82c and 0×830 are not independent. r1 is the M88K’s link register, so the “bsr” writes the addres of the following instruction after the delay slot (0×834) into r1. The delay slot also writes into r1: It adds 0x1ac to it. These instructions are executed in order, but the actual branch to 0xa28 will only be done after the delay instruction.

So what these two instructions effectively do is call a function, but set up the return address to skip the next 0x1ac bytes (107 instructions) after return. If the conditional branch at 0×824 is taken, the code at 0×834 will be executed, otherwise, 0xa28 is called, and the “taken” case of the conditional branch is skipped. This trick can be used whenever you have C code with an “if” statement of which one case is a single function call:

if (...) {
    call_something();
} else {
    [...]
}

PCEPTPDPTE

Michael Steil — Sat, 31 Oct 2009 16:05:14 +0000

Here is a new pagetable entry.

I like Intel. I told you before how Intel messed up the x86 register nomenclature by extending A to AX (A extended) and then to EAX (extended A extended). Then AMD came and extended the register once more, giving it a more sane name: RAX.

I also told you before how Intel messed up the x86 pagetable nomenclature: There were pagetables (PT, level 1) and page directories (PD, level 2) on the i386, and for the Pentium Pro, they added page directory pointers (PDP, level 3). Then AMD came and extended it once more, giving it a more sane name: page map level 4 (PML4).

With the advent of virtualization, both Intel and AMD added a feature to get rid of the slow software shadow pagetables, and added hardware support for nested pagetables, i.e. the guest has 4 levels of pagetables, and the host has another 4 levels.

AMD called these – surprise, surprise! – nested pagetables, NPT. Intel was more creative. With a history of extending architectures, they went with the big E: extended pagetables, EPT.

Let’s practice a bit: A PD is a page directory, a PDE is a page directory entry. You can also call it a PDPTE, a page directory pagetable entry (level 2 PTE), because after all, all these entries on all levels are PTEs, because they share the same format. A PDPPTE is a page directory pointer pagetable entry, aka level 3 entry.

If we use nested paging – excuse me – extended paging on Intel, we need to prepend EPT to our nice little abbreviations. An EPTPTE is a level 1 entry, an EPTPDPTE is level 2 not to be confused with an EPTPDPPTE, which is level 3, and a level 4 entry is EPTPML4PTE.

It get even better. Oracle/Sun/Innotek VirtualBox uses Hungarian Notation for its variable names, so it prepends “P” for pointer and “C” for constant. So what would you call a variable, which is a pointer to a constant level 2 EPT entry?

Of course, PCEPTPDPTE.

/** Pointer to a const EPT Page Directory Pointer Entry. */
typedef const EPTPDPTE *PCEPTPDPTE;

I thought about this for a while, and considered patenting this brilliant idea of mine, but here it is, free of patents and free for everyone to use: Michael’s nomenclature for Intel/AMD pagetables:

new name	description	old name
P4	pagetable level 4 page	PML4
P3	pagetable level 3 page	PDP
P2	pagetable level 2 page	PD
P1	pagetable level 1 page	PT
P4E	pagetable level 4 entry	PML4E/PML4PTE
P3E	pagetable level 3 entry	PDPE/PDPPTE
P2E	pagetable level 2 entry	PDE/PDPTE
P1E	pagetable level 1 entry	PTE
NP4	nested pagetable level 4 page	EPTPML4
NP3	nested pagetable level 3 page	EPTPDP
NP2	nested pagetable level 2 page	EPTPD
NP1	nested pagetable level 1 page	EPTPT
NP4E	nested pagetable level 4 entry	EPTPML4E/EPTPML4PTE
NP3E	nested pagetable level 3 entry	EPTPDPE/EPTPDPPTE
NP2E	nested pagetable level 2 entry	EPTPDE/EPTPDPTE
NP1E	nested pagetable level 1 entry	EPTPTE

You are welcome.

A Standalone printf() for Early Bootup

Michael Steil — Tue, 08 Sep 2009 02:15:20 +0000

A while ago, I complained about operating systems with overly complicated startup code that spends too much time in assembly and does hot have printf() or framebuffer access until very late.

This second post is about printf(): Many systems use POST codes (on i386/x86_64, i.e. writes to port 0×80) or debug LED for debugging, or have complicated and cumbersome implementations for puts() and print_hex() – and printf() is only available very late, because it has some special requirement, like console channels being set up. But printf() is not rocket science: Al it needs is C and a stack. Whatever system you are bringing up on whatever platform: Having printf() as early as possible will prove very useful.

Today, I am presenting a full-featured standalone version of printf() that can be added to arbitrary 32 or 64 bit C code. The code has been taken from FreeBSD (sys/kern/subr_prf.c) and is therefore BSD-licensed. Unnecessary functions have been removed and all typedefs required have been added.

/*-
 * Copyright (c) 1986, 1988, 1991, 1993
 *	The Regents of the University of California.  All rights reserved.
 * (c) UNIX System Laboratories, Inc.
 * All or some portions of this file are derived from material licensed
 * to the University of California by American Telephone and Telegraph
 * Co. or Unix System Laboratories, Inc. and are reproduced herein with
 * the permission of UNIX System Laboratories, Inc.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 * 4. Neither the name of the University nor the names of its contributors
 *    may be used to endorse or promote products derived from this software
 *    without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 *
 *	@(#)subr_prf.c	8.3 (Berkeley) 1/21/94
 */

typedef unsigned long size_t;
typedef long ssize_t;
#ifdef __64BIT__
typedef unsigned long long uintmax_t;
typedef long long intmax_t;
#else
typedef unsigned int uintmax_t;
typedef int intmax_t;
#endif
typedef unsigned char u_char;
typedef unsigned int u_int;
typedef unsigned long u_long;
typedef unsigned short u_short;
typedef unsigned long long u_quad_t;
typedef long long quad_t;
typedef unsigned long uintptr_t;
typedef long ptrdiff_t;
#define NULL ((void*)0)
#define NBBY    8               /* number of bits in a byte */
char const hex2ascii_data[] = "0123456789abcdefghijklmnopqrstuvwxyz";
#define hex2ascii(hex)  (hex2ascii_data[hex])
#define va_list __builtin_va_list
#define va_start __builtin_va_start
#define va_arg __builtin_va_arg
#define va_end __builtin_va_end
#define toupper(c)      ((c) - 0x20 * (((c) >= 'a') && ((c) <= 'z')))
static size_t
strlen(const char *s)
{
	size_t l = 0;
	while (*s++)
		l++;
	return l;
}

/* Max number conversion buffer length: a u_quad_t in base 2, plus NUL byte. */
#define MAXNBUF	(sizeof(intmax_t) * NBBY + 1)

/*
 * Put a NUL-terminated ASCII number (base <= 36) in a buffer in reverse
 * order; return an optional length and a pointer to the last character
 * written in the buffer (i.e., the first character of the string).
 * The buffer pointed to by `nbuf' must have length >= MAXNBUF.
 */
static char *
ksprintn(char *nbuf, uintmax_t num, int base, int *lenp, int upper)
{
	char *p, c;

	p = nbuf;
	*p = '\0';
	do {
		c = hex2ascii(num % base);
		*++p = upper ? toupper(c) : c;
	} while (num /= base);
	if (lenp)
		*lenp = p - nbuf;
	return (p);
}

/*
 * Scaled down version of printf(3).
 *
 * Two additional formats:
 *
 * The format %b is supported to decode error registers.
 * Its usage is:
 *
 *	printf("reg=%b\n", regval, "*");
 *
 * where  is the output base expressed as a control character, e.g.
 * \10 gives octal; \20 gives hex.  Each arg is a sequence of characters,
 * the first of which gives the bit number to be inspected (origin 1), and
 * the next characters (up to a control character, i.e. a character <= 32),
 * give the name of the register.  Thus:
 *
 *	kvprintf("reg=%b\n", 3, "\10\2BITTWO\1BITONE\n");
 *
 * would produce output:
 *
 *	reg=3
 *
 * XXX:  %D  -- Hexdump, takes pointer and separator string:
 *		("%6D", ptr, ":")   -> XX:XX:XX:XX:XX:XX
 *		("%*D", len, ptr, " " -> XX XX XX XX ...
 */
int
kvprintf(char const *fmt, void (*func)(int, void*), void *arg, int radix, va_list ap)
{
#define PCHAR(c) {int cc=(c); if (func) (*func)(cc,arg); else *d++ = cc; retval++; }
	char nbuf[MAXNBUF];
	char *d;
	const char *p, *percent, *q;
	u_char *up;
	int ch, n;
	uintmax_t num;
	int base, lflag, qflag, tmp, width, ladjust, sharpflag, neg, sign, dot;
	int cflag, hflag, jflag, tflag, zflag;
	int dwidth, upper;
	char padc;
	int stop = 0, retval = 0;

	num = 0;
	if (!func)
		d = (char *) arg;
	else
		d = NULL;

	if (fmt == NULL)
		fmt = "(fmt null)\n";

	if (radix < 2 || radix > 36)
		radix = 10;

	for (;;) {
		padc = ' ';
		width = 0;
		while ((ch = (u_char)*fmt++) != '%' || stop) {
			if (ch == '\0')
				return (retval);
			PCHAR(ch);
		}
		percent = fmt - 1;
		qflag = 0; lflag = 0; ladjust = 0; sharpflag = 0; neg = 0;
		sign = 0; dot = 0; dwidth = 0; upper = 0;
		cflag = 0; hflag = 0; jflag = 0; tflag = 0; zflag = 0;
reswitch:	switch (ch = (u_char)*fmt++) {
		case '.':
			dot = 1;
			goto reswitch;
		case '#':
			sharpflag = 1;
			goto reswitch;
		case '+':
			sign = 1;
			goto reswitch;
		case '-':
			ladjust = 1;
			goto reswitch;
		case '%':
			PCHAR(ch);
			break;
		case '*':
			if (!dot) {
				width = va_arg(ap, int);
				if (width < 0) {
					ladjust = !ladjust;
					width = -width;
				}
			} else {
				dwidth = va_arg(ap, int);
			}
			goto reswitch;
		case '0':
			if (!dot) {
				padc = '0';
				goto reswitch;
			}
		case '1': case '2': case '3': case '4':
		case '5': case '6': case '7': case '8': case '9':
				for (n = 0;; ++fmt) {
					n = n * 10 + ch - '0';
					ch = *fmt;
					if (ch < '0' || ch > '9')
						break;
				}
			if (dot)
				dwidth = n;
			else
				width = n;
			goto reswitch;
		case 'b':
			num = (u_int)va_arg(ap, int);
			p = va_arg(ap, char *);
			for (q = ksprintn(nbuf, num, *p++, NULL, 0); *q;)
				PCHAR(*q--);

			if (num == 0)
				break;

			for (tmp = 0; *p;) {
				n = *p++;
				if (num & (1 << (n - 1))) {
					PCHAR(tmp ? ',' : '<');
					for (; (n = *p) > ' '; ++p)
						PCHAR(n);
					tmp = 1;
				} else
					for (; *p > ' '; ++p)
						continue;
			}
			if (tmp)
				PCHAR('>');
			break;
		case 'c':
			PCHAR(va_arg(ap, int));
			break;
		case 'D':
			up = va_arg(ap, u_char *);
			p = va_arg(ap, char *);
			if (!width)
				width = 16;
			while(width--) {
				PCHAR(hex2ascii(*up >> 4));
				PCHAR(hex2ascii(*up & 0x0f));
				up++;
				if (width)
					for (q=p;*q;q++)
						PCHAR(*q);
			}
			break;
		case 'd':
		case 'i':
			base = 10;
			sign = 1;
			goto handle_sign;
		case 'h':
			if (hflag) {
				hflag = 0;
				cflag = 1;
			} else
				hflag = 1;
			goto reswitch;
		case 'j':
			jflag = 1;
			goto reswitch;
		case 'l':
			if (lflag) {
				lflag = 0;
				qflag = 1;
			} else
				lflag = 1;
			goto reswitch;
		case 'n':
			if (jflag)
				*(va_arg(ap, intmax_t *)) = retval;
			else if (qflag)
				*(va_arg(ap, quad_t *)) = retval;
			else if (lflag)
				*(va_arg(ap, long *)) = retval;
			else if (zflag)
				*(va_arg(ap, size_t *)) = retval;
			else if (hflag)
				*(va_arg(ap, short *)) = retval;
			else if (cflag)
				*(va_arg(ap, char *)) = retval;
			else
				*(va_arg(ap, int *)) = retval;
			break;
		case 'o':
			base = 8;
			goto handle_nosign;
		case 'p':
			base = 16;
			sharpflag = (width == 0);
			sign = 0;
			num = (uintptr_t)va_arg(ap, void *);
			goto number;
		case 'q':
			qflag = 1;
			goto reswitch;
		case 'r':
			base = radix;
			if (sign)
				goto handle_sign;
			goto handle_nosign;
		case 's':
			p = va_arg(ap, char *);
			if (p == NULL)
				p = "(null)";
			if (!dot)
				n = strlen (p);
			else
				for (n = 0; n < dwidth && p[n]; n++)
					continue;

			width -= n;

			if (!ladjust && width > 0)
				while (width--)
					PCHAR(padc);
			while (n--)
				PCHAR(*p++);
			if (ladjust && width > 0)
				while (width--)
					PCHAR(padc);
			break;
		case 't':
			tflag = 1;
			goto reswitch;
		case 'u':
			base = 10;
			goto handle_nosign;
		case 'X':
			upper = 1;
		case 'x':
			base = 16;
			goto handle_nosign;
		case 'y':
			base = 16;
			sign = 1;
			goto handle_sign;
		case 'z':
			zflag = 1;
			goto reswitch;
handle_nosign:
			sign = 0;
			if (jflag)
				num = va_arg(ap, uintmax_t);
			else if (qflag)
				num = va_arg(ap, u_quad_t);
			else if (tflag)
				num = va_arg(ap, ptrdiff_t);
			else if (lflag)
				num = va_arg(ap, u_long);
			else if (zflag)
				num = va_arg(ap, size_t);
			else if (hflag)
				num = (u_short)va_arg(ap, int);
			else if (cflag)
				num = (u_char)va_arg(ap, int);
			else
				num = va_arg(ap, u_int);
			goto number;
handle_sign:
			if (jflag)
				num = va_arg(ap, intmax_t);
			else if (qflag)
				num = va_arg(ap, quad_t);
			else if (tflag)
				num = va_arg(ap, ptrdiff_t);
			else if (lflag)
				num = va_arg(ap, long);
			else if (zflag)
				num = va_arg(ap, ssize_t);
			else if (hflag)
				num = (short)va_arg(ap, int);
			else if (cflag)
				num = (char)va_arg(ap, int);
			else
				num = va_arg(ap, int);
number:
			if (sign && (intmax_t)num < 0) {
				neg = 1;
				num = -(intmax_t)num;
			}
			p = ksprintn(nbuf, num, base, &tmp, upper);
			if (sharpflag && num != 0) {
				if (base == 8)
					tmp++;
				else if (base == 16)
					tmp += 2;
			}
			if (neg)
				tmp++;

			if (!ladjust && padc != '0' && width
			    && (width -= tmp) > 0)
				while (width--)
					PCHAR(padc);
			if (neg)
				PCHAR('-');
			if (sharpflag && num != 0) {
				if (base == 8) {
					PCHAR('0');
				} else if (base == 16) {
					PCHAR('0');
					PCHAR('x');
				}
			}
			if (!ladjust && width && (width -= tmp) > 0)
				while (width--)
					PCHAR(padc);

			while (*p)
				PCHAR(*p--);

			if (ladjust && width && (width -= tmp) > 0)
				while (width--)
					PCHAR(padc);

			break;
		default:
			while (percent < fmt)
				PCHAR(*percent++);
			/*
			 * Since we ignore an formatting argument it is no
			 * longer safe to obey the remaining formatting
			 * arguments as the arguments will no longer match
			 * the format specs.
			 */
			stop = 1;
			break;
		}
	}
#undef PCHAR
}

static void
putchar(int c, void *arg)
{
	/* add your putchar here */
}

void
printf(const char *fmt, ...)
{
	/* http://www.pagetable.com/?p=298 */
	va_list ap;

	va_start(ap, fmt);
	kvprintf(fmt, putchar, NULL, 10, ap);
	va_end(ap);
}

One thing you will have to do is #define or #undef __64__BIT to enable or disable support for printing 64 bit values. If you are on a 64 bit architecture, you will need this to print addresses properly. If you are on 32 bit, this support will require external library functions to do 64 bit arithmetic, so you probably want to turn this off.

On ARM, which doesn't have an assembly statement for division, will always need a library routine. You can use FreeBSD's at sys/libkern/arm/divsi3.S, if you to add the following #defines to make the file compile:

#define __FBSDID(a)
#define RET bx lr
#define ENTRY_NP(a) _##a: .global _##a
#define _KERNEL

All you need to do now to get printf() working is to fill the putchar() function, for example with a call to a serial_putc() implementation, if you have a serial port. On a PC, serial_putc() could look like this:

static void outb(short p, unsigned char a)
{
        asm volatile("outb %b0, %w1" : : "a" (a), "Nd" (p));
}
static unsigned char inb(short p)
{
        unsigned char a;
        asm volatile("inb %w1, %b0" : "=a" (a) : "Nd" (p));
        return a;
}
#define SERIAL_PORT 0x3f8
static void
serial_putc(unsigned char c) {
	while (!(inb(SERIAL_PORT + 5) & 0x20));
	outb(SERIAL_PORT, c);
}

On an "ARM Integrator/CP" board, the default for QEMU emulating ARM, the follwing serial_putc() would apply (make sure you are running the MMU off, or with the serial port MMIO region mapped):

#define SERIAL_PORT 0x16000000
static void
serial_putc(unsigned char c) {
	while ((*(volatile unsigned int *)(SERIAL_PORT + 0x18)) & 0x20);
	*(volatile unsigned int *)(SERIAL_PORT) = (c);
}

Unfortunately, sometimes you are bringing up or debugging a system that does not have a serial port - or at least you haven't found one: Apple Macintosh and various game systems come to mind. In this case, you might be able to set up a framebuffer and print to that. In a later post, I will present a small standalone framebuffer library.

Pictures of Apple Lisa 2 Boards

Michael Steil — Tue, 01 Sep 2009 08:01:14 +0000

The Apple Lisa is a quite rare collector’s item today. So if you hold one in your hands, you better take some high resolution pictures of the boards. Here they are:

CPU Board

I/O Board

Memory Board

TI-99/4A BASIC as a Scripting Language

James Abbatiello — Tue, 25 Aug 2009 03:28:09 +0000

It is a good time for statically recompiled versions of BASIC from old computers.Â First there was Apple I BASIC.Â Then came Commodore BASIC.Â Now, due to overwhelming demand, we’re proud to release TI-99/4A BASIC. For those unfamiliar the TI-99/4A was a home computer by Texas Instruments released in 1981. Unusually for the time it had a 16-bit CPU: the TMS9900.

Download the program from the project page on SourceForge. Binaries for Windows and Mac OS X are available and the source should compile on most POSIX-like systems. If you port it to a new platform please drop us a line; we’d love hear about it.

It supports the same interfaces as the previous projects.Â You can use it interactively in direct mode:

$ tibasic
TI BASIC READY
>print 4*atn(1)
 3.141592654

>bye

Or you can write a line-numbered program:

$ tibasic
>10 FOR I=1 TO 10
>20 PRINT TAB(I);"Hello, world!"
>30 NEXT I
>RUN
Hello, world!
 Hello, world!
  Hello, world!
   Hello, world!
    Hello, world!
     Hello, world!
      Hello, world!
       Hello, world!
        Hello, world!
         Hello, world!

** DONE **

>BYE

You can also run programs from a file:

$ cat name.bas
10 INPUT "What is your name? ":N$
20 PRINT "Hello, ";N$;"!"

$ tibasic.exe name.bas
What is your name? James
Hello, James!

There are a few sample programs available on the homepage that you can run in this manner.

How it works

This program works much like the already mentioned Commodore BASIC project. The original program is statically recompiled to produce a new native binary. The recompiler is the work of Michael Steil and theÂ support for the TI-99/4A is by James Abbatiello.

The output of the recompiler is tibasic.c which is platform independent. The support functions are in runtime_functions.c and it is this file that you would edit to port to a new system or to add new features.

There are a couple quirks of the TI-99/4A which made this a bit trickier to support than the C64 version:

The BASIC interpreter was not written in assembly language as you might expect. Instead it was programmed in something called Graphics Programming Language (GPL). GPL was much like an assembly language with some high-level primitives for commonly needed functions. A program written in GPL would be assembled to bytecode and then run on an interpreter that was written in TMS9900 assembly language. This made BASIC programs on the machine rather slow. The CPU would execute the GPL interpreter. The GPL interpreter would run the BASIC interpreter. Finally, the BASIC interpreter would run your program. Only the TMS9900 code is statically recompiled in this program. The GPL still runs interpreted just as it did on the original machine. While it would theoretically be possible to statically recompile the GPL code as well this would be significantly more difficult since GPL was never officially documented by TI. The only definitive reference is the original interpreter.
C64 BASIC outputs characters to the screen via one function (CHROUT) which was easy to trap. TI-99/4A BASIC outputs to the screen with direct writes to video memory from multiple different places in the code. This requires some ugly hacks to get anything approximating reasonable on stdout.

Limitations

The Cassette, Disk, and Sound devices are not currently emulated. Contributions in these areas are most welcome.

Amiga intern (PDF)

Michael Steil — Tue, 18 Aug 2009 08:01:30 +0000

(German) Die QualitĂ¤t dieses Scans ist furchtbar, aber wenigstens ist die PDF durchsuchbar.

Dittrich, S., Gelfand, R., & Schemmel, J. (1988). Amiga intern. DĂźsseldorf: Data Becker. (PDF, 718 S., 11 MB)

DAS STEHT DRIN:

In der dritten Ăźberarbeiteten Auflage finden Sie alles von der Hardware Ăźber den Betriebssystemkern EXEC bis zum DOS, alle entscheidenden Informationen zum Amiga. Und zwar so verstĂ¤ndlich, daĂ auch die Nicht-Profis unter Ihnen die Arbeitsweise des Amiga-Betriebssystems schnell verstehen werden.

Aus dem Inhalt:

Die Chips des Amiga (68000, CIA, Gary, Agnus, Denise, Paula)
Die Schnittstellen (Video, Audio, RGB, Centronics, seriell, Floppy, Gameport, Expansionsport, Zorro-Bus, Tastatur)
Programmierung der Hardware (Speicherbelegung, Interrupts, Copper, Blitter, Disk-Controller)
Strukturen des EXEC (Node, List, Libraries, Tasks)
Funktion des Multitasking (Task-Switching, Kommunikation zwischen Tasks, Exceptions, Traps, Semaphoren, Speicherverwaltung)
I/O-Handhabung beim Amiga durch Devices und I/O-Request Interrupt-Handhabung und Verwaltung der Ressources
RESET-feste Programme und Strukturen, Dokumentation der RESET-Routine Programmierung eigener Handler, Devices und Libraries
EXEC-Base (Dokumentation und Nutzung der Systemvariablen) DOS-Bibliothek (Funktionen, Parameter, Fehlermeldungen)
Disketten (Boot-Vorgang, Datenstrukturen, Programmaufbau) Fast-Filing-System auf Diskette und Festplatte
Programmstart, Parameter, Aufruf von CU und Workbench, detaillierte Beschreibung des internen Aufbaus der CU-Befehle (Interne DOS-Bibliothek) Ein-/Ausgaben (Tastatur, Bildschirm, Diskette, parallele und serielle Schnittstelle)

UND GESCHRIEBEN HABEN DIESES BUCH:

Johannes Schemmel ist Hardware-Spezialist mit der FĂ¤higkeit, GesamtzusammenhĂ¤nge verstĂ¤ndlich darzustellen. Stefan Dittrich als 68000-Spezialist hat schon mit seinem Buch “Amiga Maschinensprache” dem interessierten AmigaAnwender gezeigt, welche FĂ¤higkeiten in dem Rechner stecken. Ralf Gelfand ist ausgefuchster Amiga-Programmierer, der spĂ¤testens seit dem groĂen Floppy-Buch zum Amiga ein Begriff ist.

ISBN 3-89011-104-1

Dangerous Xbox 360 Update Killing Homebrew

Michael Steil — Tue, 11 Aug 2009 17:24:28 +0000

On Tuesday, Microsoft has released an Xbox 360 software update that overwrites the first stage bootloader of the system. Although there have been numerous software updates for Microsoft’s gaming console in the past, this is the first one to overwrite the vital boot block. Any failure while updating this will break the Xbox 360 beyond repair. Statistics from other systems have shown that about one in a thousand bootloader updates goes wrong, and unless Microsoft has a novel solution to this problem, this puts tens of thousands of Xboxes at risk.

It seems that this update is being done to fix a vulnerability already known to the Free60 Project. This vulnerability has been successfully exploited to run arbitrary code, and a complete end user compatible hack has been in development for some time and is planned to be released on free60.org shortly. It will allow users to take back control of their Xboxes and run arbitrary code like homebrew applications or Linux right after turning on the console and without the need of a modchip, finally opening up the Xbox 360 to a level of hacking as the original Xbox.

Because of the dangerousness of the update and the homebrew lockout, the Free60 Project advises all Xbox 360 users to not update their systems to the latest software version. The Project website at http://free60.org/ will provide the latest information on this ongoing topic, including the final hack software.

Free60 (www.free60.org) is a project that aims to enable Xbox 360 users to run homebrew applications and operating systems like Linux on their consoles. The effort is headed by Felix Domke and Michael Steil, who have a background in dbox2, Xbox and GameCube hacking, and who have spoken at various conferences about their findings. Two years ago, Free60 released a hack that allowed arbitrary code execution using a game (“King Kong Hack”) as well as an adapted version of Linux, but this possibility has been disabled by Microsoft in subsequent updates of the Xbox 360 software.

Felix and Michael have repeatedly argued that game console manufacturers should open up their platforms to Linux and homebrew, similar to what Sony has done with the PlayStation 3.

(Felix Domke, Michael Steil, Free60 Project; 11 August 2009)

Minimizing the Assembly needed for Machine Initialization

Michael Steil — Tue, 11 Aug 2009 05:47:12 +0000

In many operating systems, I have seen overly complicated startup code. Too much is done in assembly, and printf() and framebuffer access is only available very late. In the next three blog posts, I will show how this can be avoided.

In this post, I am showing how little assembly code is needed for startup. Minimizing the assembly makes your code significantly more maintainable. Everything that really needs to be done is setting up the CPU state to support 64 bit (or 32 bit) C code running at physical addresses, and everything else, including setting up the final machine state, can be done in C with a little inline assembly. The following example switches from 16 bit (real mode) or 32 bit mode (flat protected mode) into 64 bit mode on an x86_64 CPU (NASM syntax):

PAGE_SIZE	equ	0x1000
STACK_SIZE	equ	16*1024

PML2		equ	0x1000
PML3		equ	PML2+PAGE_SIZE
PML4		equ	PML3+PAGE_SIZE

STACK_BOTTOM	equ	PML4+PAGE_SIZE
STACK_TOP	equ	STACK_BOTTOM+STACK_SIZE

CODE_START	equ	STACK_TOP

	org CODE_START

	[BITS 16]
	cli

	; clear 3 pages of pagetables
	mov edi, PML2
	xor eax, eax
	mov ecx, 3*4096/4
	rep stosd

	; set up pagetables
	mov dword [PML2], 0x87		; single 4 MB id mapping PML2
	mov dword [PML3], PML2 | 7	; single entry at PML3
	mov dword [PML4], PML3 | 7	; single entry at PML4

	; load the GDT
	lgdt [gdt_desc]

	; set PSE,  PAE
	mov eax, 0x30
	mov cr4, eax

	; long mode
	mov ecx, 0xc0000080
	rdmsr
	or eax, 0x100
	wrmsr

	; enable pagetables
	mov eax, PML4
	mov cr3, eax

	; turn on long mode and paging
	mov eax, 0x80010001
	mov cr0, eax

	jmp SEL_CS:code64

code64:
	[BITS 64]
	mov ax, SEL_DS
	mov ds, ax
	mov es, ax
	mov fs, ax
	mov gs, ax
	mov ss, ax

	mov sp, STACK_TOP
	call start

inf:	jmp inf

gdt_desc:
	dw  GDT_LEN-1
	dd  gdt
	align 8
gdt	db 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 ; 0x00 dummy
gdt_cs	db 0xff, 0xff, 0x00, 0x00, 0x00, 0x9b, 0xaf, 0x00 ; 0x08 code64
gdt_ds	db 0xff, 0xff, 0x00, 0x00, 0x00, 0x93, 0xaf, 0x00 ; 0x18 data64

GDT_LEN equ $ - gdt
SEL_CS equ gdt_cs - gdt
SEL_DS equ gdt_ds - gdt

While switching into 32 bit flat mode is trivial, switching into 64 bit mode requires setting up pagetables. This code sets up 4 MB of identity-mapped memory starting at address 0.

The code is designed to switch from 16 bit mode into 64 bit mode, but since 16 and 32 bit flat mode on i386 are assembly source compatible, you can replace “[BITS 16]” with “[BITS 32]“, and the code will switch from 32 bit to 64 bit mode. (Yes, it is possible to switch from real mode directly into 64 bit mode: osdev.org Forums)

If you use this code, be careful about the memory layout. The example leaves the first page in memory untouched (in case you need the original real mode BIOS IDT for something later), and occupies the next few pages for the pagetables and the stack. Your code should be above that, but, on a BIOS system, not be between 640 KB and 1 MB (this might be device memory and ROM), and also not above 1 MB before you have enabled the A20 gate.

While this code is enough to support 64 bit C code, this is not enough to set up the machine to support all aspects of an operating system. You probably want to set up your own GDT that has entries for 32 bit code and data too, you want to set up an IDT in order to be able to catch exceptions and interrupts, and you will need real pagetables to support virtual memory. Also, you will have to move your stack pointer once you have your final memory layout.

But it is possible to construct the overly complicated GDT, IDT and pagetable structures using readable C code, and the “lidt”, “lgdt” etc. instructions can be done in inline assembly. While this is not portable C code, it is possible to reuse large parts of the machine initialization for a 32 bit (i386) and a 64 bit (x86_64) version of the operating system, which is not as easy to get right in assembly.

In my next post, I am going to show how easy it is to get printf() working as soon as you reach C, so you don’t have to mess around with puts()- and print_hex()-like functions in early machine initialization.

Aggressive Tail Call Optimization

Michael Steil — Tue, 04 Aug 2009 08:01:33 +0000

In some i386/x86_64 assembly code my coworker was working on, there was a macro like this:

#define ENDFUNC leave \
                ret

Having forgotten about the exact implementation of the macro, he then wrote a function that ended like this:

        leave
        ENDFUNC

Now this obviously ended the function with

        leave
        leave
        ret

and he got very interesting results. What was happening?

LOAD”$”,8

Michael Steil — Tue, 28 Jul 2009 08:01:47 +0000

Commodore computers up to BASIC 2.0 (like the Commodore 64, the VIC-20 and the PET 2001) only had a very basic understanding of mass storage: There were physical device numbers that were mapped to the different busses, and the “KERNAL” library had “open”, “read”, “write” and “close” functions that worked on these devices. There were also higher-level “load” and “save” functions that could load and save arbitrary regions of memory: The first two bytes of the file would be the (little endian) start address of the memory block.

With no special knowledge of “block storage” devices like disk drives, BASIC 2.0, which was not only a programming laguage but basically the shell of Commodore computers, could not express commands like “format a disk”, “delete a file” or “show the directory”. All this functionality, as well as the file system implementation, was part of the firmware of the disk drives.

Sending a Command

Sending commands to the drive was done by using the “open” call with a “secondary address” of 15: The computer’s KERNAL just sent the file name and the secondary address over the IEC bus as if it were to open a file, but the floppy drive understood secondary address 15 as the command channel. So for example, deleting a file from BASIC looked like this:

OPEN 1,8,15,"S:FOO": CLOSE 1

“1″ is the KERNAL’s file descriptor, “8″ the device number and “15″ the secondary address. Experts omitted the close, because it blocked on the completion of the operation.

Getting Data Back

While the “OPEN” line for disk commands was pretty verbose, it was still doable. Getting the error message of the last operation back was more tricky: It required a loop in BASIC that read bytes from channel 15 until EOF was reached.

Getting a directory listing would be in the same class of problem, since it requires the computer to send a command (and a file name mask) to the floppy and receive the data. Neither BASIC nor KERNAL knew how to do this, and since this was such a common operation, it wouldn’t have been possible to have the user type in a 4 line BASIC program just to dump the directory contents.

The BASIC Program Hack

Here comes the trick: If the program to load started with a “$” (followed by an optional mask), the floppy drive just returned the directory listing – formatted as a BASIC program. The user could then just “LOAD” the directory and “LIST” it if it were a BASIC program:

LOAD"$",8

SEARCHING FOR $
LOADING
READY.
LIST

0 "TEST DISK       " 23 2A
20   "FOO"               PRG
3    "BAR"               PRG
641 BLOCKS FREE.

In this example, “TEST DISK” is the disk name, “23″ the disk ID and “2A” the filesystem format/version (always 2A on 1540/1541/1551/1570/1571 – but this was only a redundant copy of the version information which was never read and could be changed). There are two files, 20 and 3 blocks in size respecively (a block is a 256 byte allocation unit on disk – since blocks are stored as linked lists there are only 254 bytes of payload), and both are of the “PRG” type.

Encoding of Commodore BASIC Programs

The floppy was aware of the encoding that Commodore BASIC (a derivative of Microsoft BASIC for 6502) used and prepared the directory listing in that way. A BASIC program in memory is a linked list of lines. Every line starts with a 2-byte pointer to the next line. A 0-pointer marks the end of the program. The next two bytes are the line number, followed by the zero-terminated encoded line contents.

The LIST command decodes a BASIC program in memory by following the linked list from the start of BASIC RAM. It prints the line number, a space, and the line contents. These contents have BASIC keywords encoded as 1-byte tokens starting at 0×80. Character below 0×80 are printed verbatim. Here is what 10 PRINT"HELLO WORLD!" would look like:

0801  0E 08    - next line starts at 0x080E
0803  0A 00    - line number 10
0805  99       - token for PRINT
0806  "HELLO!" - ASCII text of line
080D  00       - end of line
080E  00 00    - end of program

The example directory listing from above would be encoded by the floppy like this:

0801  21 08    - next line starts at 0x0821
0803  00 00    - line number 0
0805  '"TEST DISK       " 23 2A '
0820  00       - end of line
0821  21 08    - next line starts at 0x0821
0823  14 00    - line number 20
0825  '  "FOO"               PRG '
0840  00       - end of line
[...]

A couple of things are interesting here:

The line with the disk name and the ID is actually printed in inverted letters, which is done by having the “revert” character code as the first character of the first line, i.e. the floppy makes the assumption that the computer understands this convention.
BASIC will print the file sizes as variable-with line numbers, so the floppy adds extra spaces to the beginning of the line contents to have all file names aligned.
The floppy needs to populate the next line pointers for the linked list.

The Link Pointer

The obvious question here is: How can the floppy know where in the computer’s memory the BASIC program will live? The answer is: It doesn’t. The BASIC interpreter supports having its program anywhere in memory, and loading programs that were saved from other locations on memory – or possibly other Microsoft BASIC compatible computers with a different memory layout. The VIC-20 had BASIC RAM at 0×0401, the C64 at 0×0801 and the C128 at 0x1C01. Therefore, BASIC “rebinds” a program on load, searching for the zero-terminator of the lines and filling the (redundant) link pointers.

The floppy therefore only has to send non-zero values as the link pointers for BASIC to accept the directory listing as a program. In fact, a 1541 sends the directory with a 0×0401-base, which would be valid on a VIC-20. The reason for this is that the 1541 is only a 1540 with minor timing fixes for C64 support, and the 1540 is the floppy drive that was designed for the VIC-20.

Therefore, if you do LOAD"$",8,1 on a C64, the extra “,1″ will be interpreted by the KERNAL LOAD code to load the file at its original address (as opposed to the beginning of BASIC RAM), and since there is screen RAM at 0×0400 on the C64, garbage will appear on the screen, because the character encoding of screen ram is incompatible with BASIC character encoding.

Directory Code in 61 Bytes

There are two problems with this “directory listing is a BASIC program” hack: Listing the directory overwrites a BASIC program in RAM, and listing the directory from inside an application is non-trivial.

Therefore, many many implementations to show a directory listing exist on the C64 – and I want to present my own one here, which is, to my knowledge, the shortest existing (and maybe shorted possible?) version. It is based on a 70 byte version published in “64′er Magazin” some time in the 80s, and I managed to get it down to 61 bytes.

,C000:  A9 01     LDA #$01     ; filename length
,C002:  AA        TAX
,C003:  A0 E8     LDY #$E8     ; there is a "$" at $E801 in ROM
,C005:  20 BD FF  JSR $FFBD    ; set filename
,C008:  A9 60     LDA #$60
,C00A:  85 B9     STA $B9      ; set secondary address
,C00C:  20 D5 F3  JSR $F3D5    ; OPEN (IEC bus version)
,C00F:  20 19 F2  JSR $F219    ; set default input device
,C012:  A0 04     LDY #$04     ; skip 4 bytes (load address and link pointer)
,C014:  20 13 EE  JSR $EE13    ; read byte
,C017:  88        DEY
,C018:  D0 FA     BNE $C014    ; loop
,C01A:  A5 90     LDA $90
,C01C:  D0 19     BNE $C037    ; check end of file
,C01E:  20 13 EE  JSR $EE13    ; read byte (block count low)
,C021:  AA        TAX
,C022:  20 13 EE  JSR $EE13    ; read byte (block count high)
,C025:  20 CD BD  JSR $BDCD    ; print 16 bit integer
,C028:  20 13 EE  JSR $EE13    ; read character
,C02B:  20 D2 FF  JSR $FFD2    ; print character to stdout
,C02E:  D0 F8     BNE $C028    ; loop until zero
,C030:  20 D7 AA  JSR $AAD7    ; print carriage return character
,C033:  A0 02     LDY #$02
,C035:  D0 DD     BNE $C014    ; skip 2 bytes next time (link pointer)
,C037:  20 42 F6  JSR $F642    ; CLOSE
,C03A:  4C F3 F6  JMP $F6F3    ; reset default input device

(There is a similar implementation here.)

There are two limitations of this code though: It omits the extra space between the block number and the filename, leading to a slightly different output, and it cannot be interrupted.

The Infinite Loop Mystery

Michael Steil — Tue, 21 Jul 2009 04:08:41 +0000

Today’s puzzle is about some code behaving horribly wrong.

Recently, I was working on some operating system project and hacking on the code to switch between privileged and non-privileged mode. I could switch modes successfully and intercept traps when in non-privileged mode.

Then I wanted to check whether I could handle timer interrupts correctly, so I added this to my non-privileged code, to give the timer interrupt a chance to fire:

    volatile int i;
    for (i=0; i<10000; i++);

Timer interrupts were handled correctly and eventually returned to the non-privileged code – but the delay code turned into an infinite loop!

I changed to loop to count only to 10, and I changed it to count down instead of up, but the result remained the same. I looked at the generated assembly. It looked like this:

    movl    $10, 0xfc(%ebp)	// i = 10
    jmp     1f			// goto 1
2:
    movl    0xfc(%ebp), %eax	// %eax = i
    decl    %eax		// %eax--
    movl    %eax,0xfc(%ebp)	// i = %eax
1:
    movl    0xfc(%ebp), %eax	// %eax = i
    testl   %eax, %eax		// if (%eax > 0)
    jg      2b			// goto 2

It looked fine. On every timer interrupt, I dumped %eax, and it was stuck at 10. I debugged my pusha/popa code to save and restore registers between modes, and it was okay. I debugged my flag handing code, and flags were fine.

Then I replaced my C code with the generated assembly code and added instructions that copied the value of %eax before the “decl” into %ebx, and after the “decl” into %ecx and added a trap instruction right after that to have privileged mode print out the values of the three registers.

    movl    $10, 0xfc(%ebp)	// i = 10
    jmp     1f			// goto 1
2:
    movl    0xfc(%ebp), %eax	// %eax = i
    movl    %eax, %ebx          // value before
    decl    %eax		// %eax--
    movl    %eax, %ecx          // value after
    TRAP
    movl    %eax,0xfc(%ebp)	// i = %eax
1:
    movl    0xfc(%ebp), %eax	// %eax = i
    testl   %eax, %eax		// if (%eax > 0)
    jg      2b			// goto 2

The result was %eax = %ebx = %ecx = 10. This is when I understood what was going on.

Please share your comments below. :-)

This is Copyright 1983 Microsoft – NOT!

Michael Steil — Tue, 14 Jul 2009 08:01:08 +0000

If you look at a hexdump of any version of the Logitech mouse driver for MS-DOS, you will see the following:

000007c0                                       2a 2a 2a 20  |            *** |
000007d0  54 68 69 73 20 69 73 20  43 6f 70 79 72 69 67 68  |This is Copyrigh|
000007e0  74 20 31 39 38 33 20 4d  69 63 72 6f 73 6f 66 74  |t 1983 Microsoft|
000007f0  20 2a 2a 2a                                       | ***            |

Microsoft introduced the mouse to MS-DOS, and they specified the mouse driver interface and implemented the first MS-DOS mouse driver. Did Logitech license their code? Or did they steal it? Let’s look closer:

00000770                           54 68 69 73 20 69 73 20  |        This is |
00000780  61 20 4c 4f 47 49 54 45  43 48 20 6d 6f 75 73 65  |a LOGITECH mouse|
00000790  20 64 72 69 76 65 72 2c  20 62 75 74 20 73 6f 6d  | driver, but som|
000007a0  65 20 73 6f 66 74 77 61  72 65 20 65 78 70 65 63  |e software expec|
000007b0  74 20 68 65 72 65 20 74  68 65 20 66 6f 6c 6c 6f  |t here the follo|
000007c0  77 69 6e 67 20 73 74 72  69 6e 67 3a 2a 2a 2a 20  |wing string:*** |
000007d0  54 68 69 73 20 69 73 20  43 6f 70 79 72 69 67 68  |This is Copyrigh|
000007e0  74 20 31 39 38 33 20 4d  69 63 72 6f 73 6f 66 74  |t 1983 Microsoft|
000007f0  20 2a 2a 2a                                       | ***            |

This string is located directly before the INT 0×33 API entry point, so it is easy to check for it. There’s a sucker born every minute, but this still makes you wonder what kind of programmer would really check for a string like this, even if some Microsoft API reference indeed suggested to do so. Maybe it was only Microsoft software to compare the string.

In either case, this is a very bad and unfair practice. If you define an interface, don’t add a call to ask for the version (or even the vendor!), but add feature bits instead, so that an alternative implementation can choose to be compatible with parts of it (or extend the interface independently). And if you are a developer that uses an interface, use feature bits if they are there, and resist the temptation to check for the vendor, even if the API documentation tells you to do so.

The Giant Pile of Money in My Office

Michael Steil — Wed, 08 Jul 2009 05:39:24 +0000

Corporate security thought it wasn’t the best idea:

(1:06 min, 176 KB)

Apple Copland Reference Documentation

Michael Steil — Wed, 01 Jul 2009 06:39:07 +0000

The Copland project was Apple’s ill-fated attempt in the mid 1990s to replace the aging classic Mac OS with a more modern operating system that had a microkernel, virtual memory and preemptive multitasking. Information on Copland is scarce, therefore I have compiled 20 hard to find Copland reference documents, as well as the 359 page book “Mac OS 8 Revealed”.

Note that Copland was supposed to be the next major OS release after System 7, so the while the first two beta releases D7E1 and D9 were called “Copland”, the final beta D11E4 was called “Mac OS 8″ everywhere. After the cancellation of the Copland project, Apple reused the term “Mac OS 8″ for a System 7 update.

Copland D9 (Copland Developer Release – Tools Edition)

November 1995

An Introduction to Copland. The Mac OS Foundation for the Next Generation of Personal Computers
Glossary of Copland Terminology. This Acrobat file contains the document âGlossary of Copland Terminology,â which defines many of the terms used throughout Copland developer documentation. Note, however, that the terms defined in this glossary are preliminary and subject to change.
The Copland Toolbox. This Acrobat file contains the two-chapter book The Copland Toolbox. The first chapter introduces the event-handling model and human interface elements supported by the Copland Toolbox, and it describes the programming concepts necessary to use the Toolbox in applications. This will help you plan frameworks and tools that take full advantage of Coplandâs Toolbox features. The second chapter compares the System 7 Toolbox with the equivalent Copland managers and capabilities, and this chapter includes information on backward compatibility with System 7 applications and requirements for Copland-savvy applications. This is especially useful for helping you adapt your existing System 7-based frameworks to work in Copland.
Software Extensibility. This Acrobat file contains the document âSoftware Extensibility and the System Object Model (SOM),â which describes where and why Copland uses SOM classes, dynamically linked libraries, and other mechanisms to support extension and modification of system software, and how you can use these to add extensibility to your own software.
Intro to Kernel and OS Services. This Acrobat file contains the document âIntroduction to Kernel and Operating System Services,â which provides conceptual information about how and when to use services provided by the microkernel and related portions of the Copland operating system. This document will help you plan frameworks and tools that take full advantage of Coplandâs operating-system features.
Microkernel White Paper. Containing the document âMicrokernel White Paper,â this file gives a conceptual overview of the Copland microkernel. This document discusses tasks, hardware and software interrupts, and privileged software execution; it describes address space management, including such topics as areas and memory reservations; and it describes messaging, such as how to use message objects, how to send and receive them, and how reply or cancel them. This version of the âMicrokernel White Paperâ differs only slightly from that delivered at the 1995 Worldwide Developerâs Conference.
Patch Manager. The document âPatch Managerâ contained in this Acrobat files describes the new Copland mechanism for patching the Mac OS. This will help you plan how to create patch-type system extensions for your development environment and to produce code that creates or uses extensions that patch Copland. The Patch Manager defines an application programming interface (API) that enables software to patch the system in a consistent, more easily supported manner. This is the only documentation provided for this release of Copland that describes an application programming interface. Note, however, that as with the rest of this release, all of the Patch Managerâs programming interfaces are subject to change.
I/O Architecture. This Acrobat file contains the document âAbout the Copland I/O Architecture,â which provides an overview of the architecture and features of Coplandâs new I/O system. This document will help you with any driver development that you might be planning for Copland, and will help you design or adapt software that writes to or otherwise directly manipulates devices. This document is identical to the chapter supplied in the Copland Technical Overview delivered at the 1995 Worldwide Developerâs Conference.
About the Copland file System. This Acrobat file contains the document âAbout the Copland file System,â which introduces the features of the new file system available with Copland. It describes the various components of the file system, including the file Manager, the Navigation Services API, and the file systems API. This document also describes how to get your program ready to use the new file Manager. finally, the Documentation for MPW folder contains information describing the Macintosh Programmerâs Workshop.
Copland Desktop. The Copland Desktop Picture is a very high quality version of the CD cover art. To use this as your desktop picture, place it on your Copland Drive. While running Copland, you can open the picture from the Appearance Control Panel.

Copland D11E4 (Mac OS 8 DDK 0.4)

June 1996 (WWDC)

This Inside Macintosh for Mac OS 8 Read Me provides a road map to the chapters within this folder, provides information on viewing the Acrobat files contained within this folder, and lists the methods you can use to give us your feedback on the Inside Macintosh documentation.
For an overview of new features supported by the Toolbox, such as human interface objects, panels, window groups, imaging objects, and themes, see the chapter “Introduction to the Mac OS 8 Toolbox” in Human Interface Toolbox. For reference information on human interface objects, see the chapter “HIObject Class Reference” in Human Interface Toolbox.
For an overview of how Apple events are used pervasively throughout Mac OS 8 by system services as well as other programs, see the chapters “Introduction to the Mac OS 8 Event Model” and “Event Model Reference” in Apple Events in Mac OS 8. For information on how Apple events are processed by the Toolbox, see “Toolbox Event Routing” and “Toolbox Events Reference” in Human Interface Toolbox.
For information on the Mac OS 8 File Manager, Navigation Services, and other file services, as well as reference information on the Mac OS 8 File Manager, see File Navigation and Access.
For information on the extensive support for text handling that enables you to develop your application for a worldwide market, such as support for Unicode, locales, and the use of text objects, see “Introduction to Text Handling and Internationalization” in Text Handling and Internationalization.
For information on the use of the System Object Model (SOM) in Mac OS 8, see Software Extensibility.
For information on creating resources that are new in Mac OS 8, see ResEdit 3.0 User’s Guide.
For an overview of the Mac OS 8 microkernel, including information on tasks, processes, scheduling, preemption, multithreading, allocation of memory, and other services, see the chapter “About Mac OS 8″ in Microkernel and Core System Services. See subsequent chapters in Microkernel and Core System Services for reference information on many of these topics.
For information on the Mac OS 8 I/O architecture, including information on families, plug-ins, and I/O services, see “About the I/O Architecture” in Modular I/O. For information on families for specific hardware devices, see subsequent chapters in Modular I/O.
For updated information on Open Transport, see Open Transport.
Please note that all presentations in this folder were created using Aldus Persuasion; an application we have not included on the DDK. (Displays Presentation, Keyboard Family Presentation, Mac OS 8 I/O Overview, OT General Presentation, OT Serial Presentation)

Tony Francis: Mac OS 8 Revealed

July 1996

With the next release of the Macintosh operating system, Apple will provide a state-of=the-art platform for developers to create new software products. At the core of Mac as 8 is a redesigned operating system foundation based on microkernel technology that will dramatically increase user productivity.

Author Tony Francis worked closely with the Mac as 8 engineering team to provide the inside story of the design and development of this innovative technology. The book describes the technical geography of Mac as 8 and illustrates how the system’s user benefits can be implemented in software and hardware products.

Written for developers, system administrators, and information systems professionals, Mac OS 8 Revealed explains the key technologies of Mac as 8:

How Mac as 8 exploits preemptive multitasking to perform many operations concurrently;
How the new memory protection model insulates the microkernel and other operating system services to provide system stability;
How to provide automatic, intelligent assistance to users;
How to allow the user to customize and scale the human interface.

The accompanying CD-ROM contains an electronic version of the book with animated illustrations that demonstrate the capabilities of Mac as 8. Simply click on a screen shot marked with a film strip in the book and see how a new feature in the operating system works.

Tony Francis has been with Apple for over 10 years as a lead writer on the Inside Macintosh team.

ISBN 0-201-47955-9

I publish these files for educational purposes, especially for all computer archeology enthusiasts like me. This data is copyrighted, but since the project has been canceled ~13 years ago, nobody should still care about it. If you disagree, contact me. Thanks to all the people who have worked on Copland and written these interesting documents!

Readable and Maintainable Bitfields in C

Michael Steil — Tue, 23 Jun 2009 08:01:46 +0000

Bitfields are very common in low level C programming. You can use them for efficient storage of a data structure with lots of flags, or to pass a set of flags between functions. Let us look at the different ways of doing this.

The straightforward way to deal with bitfields is to do the Boolean logic by hand:

Boolean Magic

#define FLAG_USER   (1 << 0)
#define FLAG_ZERO   (1 << 1)
#define FLAG_FORCE  (1 << 2)
/* bits 3-30 reserved */
#define FLAG_COMPAT (1 << 31)

int
create_object(int flags)
{
        int is_compat = (flags & FLAG_COMPAT);

        if (is_compat)
                flags &= ~FLAGS_FORCE;

        if (flags & FLAG_FORCE) {
                [...]
        }
        [...]
}

int
create_object_zero(int flags)
{
	create_object(flags | FLAGS_ZERO);
}

void
caller()
{
        create_object(FLAG_FORCE | FLAG_COMPAT);
}

You can see code like this everywhere, so most C programmers can probably read and understand this quite easily. But unfortunately, this method is very error-prone. Mixing up "&" and "&&" and omitting the "~" when doing "&=" are common oversights, and since the compiler only sees "int" types, this also doesn't give you any kind of type-safety.

Bitfields

Let us look at the same code using bitfields instead:

typedef unsigned int boolean_t;
#define FALSE 0
#define TRUE !FALSE
typedef union {
        struct {
                boolean_t user:1;
                boolean_t zero:1;
                boolean_t force:1;
                int :28;                /* unused */
                boolean_t compat:1;     /* bit 31 */
        };
        int raw;
} flags_t;

int
create_object(flags_t flags)
{
        boolean_t is_compat = flags.compat;

        if (is_compat)
                flags.force = FALSE;

        if (flags.force) {
                [...]
        }
        [...]
}

int
create_object_zero(flags_t flags)
{
	flags.zero = TRUE;
	create_object(flags);
}

void
caller()
{
        create_object((flags_t) { { .force = TRUE, .compat = TRUE } });
}

Flags can just be used like any variables. The compiler abstracts the Boolean logic away. The only downside is that the code with the static initializer requires some advanced syntax.

Endianness

Bitfields in C always start at bit 0. While this is the least significant bit (LSB) on Little Endian (bit 0 has a weight of 2^0), most compilers on Big Endian systems inconveniently consider the most significant bit (MSB) bit 0 (bit 0 has a weight of 2^31, 2^63 etc. depending on the word size), so in case your bitfield needs to be binary-compatible across machines with different endianness, you will need to define two versions of the bitfield.

The Raw Bitfield

Did you notice the "int raw" in the union? It lets us conveniently (and type-safely) export the raw bit value without having to use a cast.

	printf("raw flags: 0x%x\n", flags.raw);

We can use this to reconstruct the FLAG_* constants in the original example, in case some code needs it:

#define FLAG_USER   (((flags_t) { { .user   = TRUE } }).raw)
#define FLAG_ZERO   (((flags_t) { { .zero   = TRUE } }).raw)
#define FLAG_FORCE  (((flags_t) { { .force  = TRUE } }).raw)
#define FLAG_COMPAT (((flags_t) { { .compat = TRUE } }).raw)

This code constructs a temporary instance of the bitfield, sets one bit, and converts it into a raw integer - all at compile time.

Bitfield Access from Assembly

With the same trick, you can also access your bitfield from assembly, for example if the bitfield is part of the Thread Control Block in your operating system kernel, and the low level context switch code needs to access one of the bits. The "int raw" can be used to statically convert a flag into the corresponding raw mask:

typedef unsigned int boolean_t;

typedef union {
	struct {
		boolean_t bit0:1;
		boolean_t bit1:1;
		int :19;
		boolean_t bit31:1;
	};
	int raw;
} bitfield_t;

int test()
{
	int param = -1;
	int result;

	__asm__ volatile (
		"test    %2, %1    \n"
		"xor     %0, %0    \n"
		"setcc   %0        \n"
		: "=r" (result)
		: "r" (param),
		  "i" (((bitfield_t) { { .bit31 = TRUE } }).raw)
	);
	return result;
}

The corresponding x86 assembly code looks like this:

	.text
	.align	4,0x90
	.globl	_test
_test:
	pushl	%ebp
	movl	%esp, %ebp
	movl	$-1, %eax
	## InlineAsm Start
	test	$0x80000000, %eax
	xor	%eax, %eax
	setcc	%eax
	## InlineAsm End
	popl	%ebp
	ret

	.subsections_via_symbols

This works fine with LLVM, but unfortunately GCC (4.2.1) has problems detecting that the raw value is available at compile time, so the "i" has to be replaced with an "r": GCC will then pre-assign a register with the raw value instead of being able to use an immediate with the "test" instruction.

How to Not Do It

I have seen C++ code doing this:

enum {
	FLAG_USER,
	FLAG_ZERO,
	FLAG_FORCE,
	FLAG_COMPAT = 31
}

int
create_object(bitfield_t flags)
{
        bool is_compat = flags.is_set(FLAG_COMPAT);

        if (is_compat)
                flags -= FLAGS_FORCE;

        if (flags.is_set(FLAG_FORCE)) {
                [...]
        }
        [...]
}

int
create_object_zero(int flags)
{
	create_object(flags + FLAGS_ZERO);
}

void
caller()
{
        create_object(((bitfield_t)FLAG_FORCE) + FLAG_COMPAT);
}

This all looks quite weird. The constants are bit index values, and they are added and subtracted. The reason is C++ operator overloading:

class bitmask_t
{
    word_t      maskvalue;

public:
    [...]
    inline bitmask_t operator -= (int n)
        {
            maskvalue = maskvalue & ~(1UL << n);
            return (bitmask_t) maskvalue;
        }
    [...]
}

This is horrible. The code that uses this class makes no sense unless you read and understand the implementation of the class. And you have to be very careful: While it is possible to "add" a flag to an existing bitfield, you cannot just add two flags - it would do the arithmetic and add the two values.

Mapping the setting and clearing of bits onto the addition and subtraction operators is clearly wrong in the first place: Flags in a bitfield are equivalent to elements in a set. Setting a flag is equivalent to the "union" operation, which even in Mathematics has its own symbol instead of overloading the "+" operator.

Question

If you compile code that does something like "((bitfield_t) { { .bit31 = TRUE } }).raw" with GCC in C++ mode, it fails. Why?

A Lot of Security

Michael Steil — Tue, 16 Jun 2009 08:01:31 +0000

I happened to drive through Cupertino, CA, USA last Wednesday and ended up in this situation:

Oh-oh, they got me. But they were not after me, they escorted two vans onto some company’s campus.

Five police cars, two police motorcycles, and lots of people with suits and sunglasses. For some reason, this outfit doesn’t have the same effect on me any more since “The Matrix”.

The people in the vans went into the building through a side door:

Here are some details:

A blonde woman in white, a woman with a red dress, a man in a brown uniform with a suitcase, and lots of more men in suits. The vans had license places from Maryland.

The question of today’s security puzzle is: Who is the very important person?

Stamp Vending Machine DoS

Michael Steil — Tue, 09 Jun 2009 08:01:00 +0000

It makes sense for a stamp vending machine to have some limits and not print any amount of stamps or stamps of any value. The vending machines from the Deutsche Post are a little weird though:

You can only buy stamps worth 100 EUR at a time. Makes sense.

The maximum face value for a stamp is 36.75 EUR. Hmmm…

You can buy up to 100 stamps at a time, and the minimum face value is 1 cent. The poor machine will print worthless stamps for five minutes. The obvious question now is: What is the minimum amount of money that I have to invest to make it run out of paper.

Feed in Rogue Signal Here

Michael Steil — Tue, 02 Jun 2009 08:01:53 +0000

Sometimes it makes sense to label a surveillance camera.

But it rarely makes sense to label the cables that lead to a surveillance camera in a public area of Parking Garage A at the San Francisco airport.

This just invites you to do the “Splice and Dice” attack:

Apple Lisa Operating System Reference Manual (PDF, 1983)

Michael Steil — Wed, 27 May 2009 07:32:33 +0000

The Apple Lisa from 1983 was the first consumer-class computer with a graphical user interface and significantly more advanced than the 1984 Macintosh, which had a similar UI, but a comparatively primitive underlying OS. Here, I present a searchable PDF of the rare “Operating System Reference Manual for the Lisa” (1983), as well as a quick overview of the OS and how it compares to UNIX.

“Operating System Reference Manual for the Lisa” (1983)
(PDF, 188 pages, 6.2 MB)

The OS Reference Manual is actually volume 3 of 3 of the Lisa Pascal documentation. As the last page states, the book was typeset on a Lisa and printed on a dot matrix printer.

I also converted the typewriter-written draft version from lisa.sunder.net into a searchable PDF:

“Lisa Operating System Reference Manual” (Draft March 1982)
(PDF, 113 pages, 1.2 MB)

In its spirit, the Lisa Operating System resembles UNIX a lot, and its features and details were pretty much on par with UNIX systems from that time.

Scheduling: Executable files are statically linked with the Pascal runtime, and they make syscalls into the kernel. The system manages processes (with a single thread of execution) with their own address spaces that are cooperatively scheduled (255 levels of priorities). A syscall or a code segment switch can yield the CPU. Processes are managed in a tree, and the death of a parent will kill its children.

Memory Management: There is no paging, but segmented memory (up to 106 code segments and 16 data segments), and explicit swapping of code and data pages. It is possible to write protect data segments.

Inter-Process Communication: Two processes can communicate through shared files on the filesystem, named pipes, event channels (blocking or callback messaging with typed data) and shared data segments. Exceptions are delivered as events.

Filesystem: The filesystem supports 32 character filenames and allows all ASCII characters except for “-”, which is the path separator. A path can be up to 255 characters; absolute paths start with “-” (processes have a working directory). There are no enforced extensions, but the convention is to use extensions for file types. Volumes can be mounted and unmounted, and accessed at the top level by using their device name or their volume name, if they are mounted. The serial (RS232A/RS323B) and parallel (PARAPORT) ports, stdio (MAINCONSOLE) and /dev/null (BITBKT) are
like character-devices and also accessible at the top level that also provide an ioctl-like interface (DEVICE_CONTROL).

File access goes through open/read/write/close. A file is associated with cdate, mdate, adate, a delete protection flag and an up to 128 character label. Apart from regular files, there are pipes, disk-mapped data segments and event channel files.

For safety, the on-disk data structures are very redundant. Every block contains context data like a size, name, filesize, forward/backward link, inode and position in file. Directories (catalogs) and the “medium descriptor data file” are managed just like regular files.

Tripos, the Roots of AmigaDOS

Michael Steil — Tue, 19 May 2009 11:31:46 +0000

The core of the Amiga Operating systems consists of the three major components Exec (scheduling, memory management, IPC), Intuition (GUI library) and AmigaDOS (process and file management). AmigaDOS is based on the Tripos operating system which Commodore bought because development of their own DOS subsystem failed to meet deadlines. In this article, I am presenting searchable PDFs of the very rare Tripos manuals (638 pages) as well as the AmigaDOS manual (304 pages). Comparing the two documents will share some insight in the relationship between Tripos and Amiga OS.

Tripos Manuals
(PDF, 638 p., 20MB)

The AmigaDOS Manual
(PDF, 304 p., 14MB)

These are the tables of contents of the two documents:

Introduction to Tripos	AmigaDOS User’s Manual
1. Simple Use of Tripos	1. Introducing AmigaDOS
2. Editing Files
3. Further Use of Tripos
Tripos User’s Reference Manual
1. Tripos Commands	2. AmigaDOS Commands
2. ED – The Screen Editor	3. ED – The Screen Editor
3. EDIT – The Line Editor	4. EDIT – The Line Editor
Appendix A: Error Codes and Messages	Appendix: Error Codes and Messages
Tripos Programmer’s Reference Manual	AmigaDOS Developer’s Manual
1. Introduction to Programming	1. Programming on the Amiga
2. Calling the Kernel
3. Calling the DOS	2. Calling AmigaDOS
4. The Macro Assembler	3. The Macro Assembler
5. The Linker	4. The Linker
6. The System Debugger – DEBUG
7. Full Screen Support
8. Floating Point
	Appendix: Console Input and Output on the Amiga
Tripos Technical Reference Manual	AmigaDOS Technical Reference Manual
1. The Filing System	1. The Filing System
2. Binary File Structure	2. Amiga Binary File Structure
3. Tripos Data Structures	3. AmigaDOS Data Structures
4. Installation
	4. AmigaDOS Additional Information for the Advanced Developer

The Original DOS Module of AmigaOS

AmigaOS was architected in a very modular way. “Exec”, the operating system kernel, was responsible for tasks (and therefore scheduling), memory management and the messaging infrastructure. It knew nothing about filesystems or user I/O. The graphics library “Intuition” built on Exec, just like the DOS module, but they were independent of each other. But the DOS module fell behind schedule, so Commodore decided to license “Tripos“.

Tripos

Tripos is, according to “Introduction to Tripos”, “a multi-processing operating system designed for 68000 computers. Although you can use it as a multi-user system, you normally run Tripos for a single user. The multi-processing facility lets many jobs take place simultaneously. You can also use the multi-processing facility to suspend one job while you run another.“

Tripos was very similar to the Amiga OS design and therefore fit well. Tripos consisted of a kernel, a DOS library and a collection of user mode tools. Since the “Exec” was already working well, and Intuition depended on it, there was no sense in replacing it with the Tripos kernel. Instead, only the DOS part of Tripos (including the tools) was integrated into AmigaOS.

So the DOS API, the command line interface with its commands, the executable format (“hunk”) as well as the original Amiga filesystem were all basically identical to Tripos.

Comparing the Manuals

Commodore published a number of reference manuals about the Amiga operating system:

AMIGA ROM Kernel Reference Manual: Exec
Amiga ROM Kernel Reference Manual: Libraries and Devices
Amiga ROM Kernel Reference Manual: Includes and Autodocs
Amiga Intuition Reference Manual
The AmigaDOS Manual

While the former four books were all written from stratch describing new Amiga technology, the AmigaDOS Manual was basically just the Tripos manual set with “Tripos” replaced with “AmigaDOS” and all references to technologies that didn’t make it into the Amiga removed.

You can see from the tables of contents of the two books that The AmigaDOS User’s Manual omitted the chapters about the Tripos kernel (AmigaOS used “Exec”), the debugger (AmigaOS had “ROMWack”), full screen support (everything above simple character I/O was done by Intuition), floating point (Amiga OS contained Motorola’s “MC68343 FLOATING POINT FIRMWARE” instead) and Installation.

What is interesting about the AmigaDOS Manual is that very limited adaption to the Amiga has been performed. The previously quoted definition of Tripos can be found in the AmigaDOS manual as well: “AmigaDOS a multi-processing operating system designed for 68000 computers. [...]” While this would be true for AmigaOS, AmigaDOS was only a library and in no way a “multi-processing operating system”.

There are not many real differences between the two documents: The AmigaDOS Manual replaced references to “screen” with “current window” and mentions the new logical devices “LIBS:”, “DEVS:” and “FONTS:” when talking about the filesystem layout. There are also some smaller differences on the command line: There is no MOUNT command (AmigaOS detects disks and automounts), the “C” command has been renamed to “EXECUTE” as well as the “PAR:” device to “PRT:”. The AmigaDOS Manual also contains a section about cross-development on a Sun or MS-DOS machine.

Now I am looking forward to your comments: Did you any other interesting differences? In what way were Exec and the Tripos kernel different? Any other input? Thanks!

Reverse-Engineering DOS 1.0 – Part 2: IBMBIO.COM

Michael Steil — Tue, 12 May 2009 08:01:13 +0000

My last post was about the internals of the DOS 1.0 bootsector. This time, let’s look at the next stage of the DOS 1.0 boot process, the hardware abstraction library IBMBIO.COM.

CP/M and DOS History

Let us first look at the historical background: CP/M was an 8 bit operating system that existed for virtually every computer with an 8080/Z80 CPU. It consisted of the three core components: BIOS, BDOS and CCP. BIOS was the machine abstraction layer that allowed CP/M work on different platforms. BDOS was the platform agnostic core library code, and CCP the command line interpreter.

86-DOS by Seattle Computer Products was a clone of CP/M intended for 8086 computers. It shared the architecture of CP/M, having a separate machine abstraction layer (“DOSIO”). When Microsoft bought 86-DOS and ported it to the upcoming IBM PC (model 5150), they kept this architecture, although there was no need to implement custom drivers, since the IBM PC had all its drivers in its “BIOS” firmware. But IBM’s BIOS did not have the same interface as 86-DOS DOSIO, so PC-DOS 1.0 included a very small DOSIO which would just sit on top of BIOS and using its driver library (and work around some bugs).

So DOS 1.0 for the IBM PC consisted of the three parts IBMBIO.COM (machine abstraction), IBMDOS.COM (DOS library) and COMMAND.COM (command line).

Microsoft soon started licensing MS-DOS to other computer manufacturers that wanted to make IBM PC compatibles. Back then, IBM PC compatible meant being able to run DOS applications and not necessarily sharing the whole system design with the IBM PC, so for MS-DOS to run on other 8086-based systems, it was enough to adapt the hardware abstraction layer to these machines, and therefore Microsoft provided the source code of this part of MS-DOS to hardware vendors. In MS-DOS, the system files are called IO.SYS and MSDOS.SYS.

But since MS-DOS only provided a rather narrow API that did not include, for example, access to bitmap graphics, soon many DOS programs accesses hardware directly, forcing clone makers to make their PC compatibles more and more similar to the IBM PC, eventually using the same support chips and a binary compatible firmware interface. The separation of the kernel in two parts was less and less necessary, so that starting with MS-DOS 5.0, Microsoft only provided a single version of IO.SYS, and in MS-DOS 7.0 (Windows 95), IO.SYS and MSDOS.SYS were merged into IO.SYS.

IBMBIO

On DOS 1.0, IBMBIO.COM provides the following library functions to DOS (names taken from 86-DOS DOSIO source):

STATUS check for keypress

INP get key from keyboard

OUTP send character to screen

PRINT send character to printer

AUXIN get character from serial

AUXOUT send character to serial

READ read sector(s) from disk

WRITE write sector(s) to disk

DSKCHG check for disk change

(READ and WRITE will be directly hooked up by DOS into the INT 0×25/0×26 direct disk I/O API later.)

In addition to this, IBMBIO is the next step in the boot process after the bootloader and responsible for

initializing the serial and printer ports

building a list of floppy drives and its capabilities

setting up exception vectors (division by zero etc.)

call the IBMDOS init code (already in memory)

load and run COMMAND.COM

Let us now look at the library calls it provides:

Serial and Printer

The code to talk to the serial and printer ports is rather straightforward. There is support for a single serial port and a single printer port. IBMBIO sets up the port to 2400 8N1 and has no function to change this setting. I/O will just be passed to the respective BIOS functions, but errors will be evaluated and error messages will be printed in the error case.

Keyboard and Screen

While printing a character just passes the character to BIOS, character input is quite interesting: When reading a character, BIOS returns the ASCII code as well as the raw keyboard scancode. For keys that have no ASCII equivalent like the function or cursor keys, this returns zero as the ASCII code. IBMBIO always returns the ASCII code, but for special keys, it returns two bytes: A zero, indicating that it is a special key, and the BIOS scancode. Therefore in case of a special key, it caches the scancode, returns the zero, and will return the scancode the next time a character is read.

The Control+C/Control+Break handler uses this infrastructure to inject the code “3″ into the input stream.

Disk I/O: Virtual Disks

The library code for Disk I/O is the most interesting part, since it can simulate a virtual disk drive, and it works around two issues of the BIOS function.

DOS supports up to four disk drives, A:, B:, C: and D:, but in case there is only a single drive, it will present two drives to DOS. Since all disk access goes through the READ and WRITE functions of IBMBIO, it can compare the requested disk drive with the disk drive last used, and if it’s the other drive, it will print:

Insert diskette for driveÂ A: and strike any key when ready

After pressing a key, the actual I/O access is performed. This way, DOS can be completely agnostic about whether there are two physical drives, or a physical and a virtual drive, and “COPY A:FOO B:BAR” will just work. Note that without this feature, it would be impossible to get data from one disk onto another with standard DOS tools.

Disk I/O: Multiple Tracks

The first IBM PC BIOS issue IBMBIO works around is the fact that the original version of BIOS did not support a read or write of sectors across multiple heads. A track always had 8 sectors, and if your read starting at sector 1 of a track, it is possible to read up to 8 sectors, but starting at sector 8, only one sector can be read – if you want to continue reading sectors from the next track, you have to call BIOS again explicitly. The IBMBIO driver therefore breaks up longer reads if they span tracks.

Disk I/O: 8237 DMA Controller Bug

The second problem is actually a design issue with the Intel 8237 DMA controller in the PC. Although the Intel 8088 CPU was internally 16 bits, had 16 bit registers and could support up to 1 MB of RAM, it was the low cost version that was meant to interface to 8 bit support chips. The 8237 is such a support chip intended for 8 bit systems, and therefore only supports 64 KB of memory. Since this would have meant that data from the disk drive can only be read into the lower 64 KB of the PC, IBM extended the DMA controller by adding an external latch per channel to it: You set up the lower 16 bits of the DMA address in the DMA controller, and the upper 4 bits (20 bits correspond to 1 MB) in an external latch, and the upper four address lines will be provided by the latch when the DMA controller accesses memory.

Unfortunately, the 8237 had no “carry out”, so if you set up a DMA to 0x0FFFF (latch: 0×0, DMA controller: 0xFFFF), the address inside the DMA controller will wrap around to 0×0000, but it will not update the upper four bits of the address in the latch. So DMA that spans a 64 KB boundary will end up at the wrong location.

The idea of a device driver is to abstract away details of a device and work around device bugs, but the BIOS in the first PC failed to work around this quirk in the DMA hardware. Therefore IBMBIO works around it by detecting I/O that spans a 64 KB boundary and performing it in a temporary buffer inside IBMBIO.

What is interesting about these workarounds is that DOS 1.0 was the default operating system for the disk-equipped version of the IBM PC, and IBM shipped it with its first machines, so it should have been possible to include these workarounds in BIOS already. In fact, later versions of BIOS did not have these issues any more, but DOS kept supporting the workarounds for a long time.

Source

Here is the assembly source of IBMBIO 1.0. It can be compiled with NASM and produces a binary which is not 100% identical, because of variations in the instruction encoding of different assemblers. The original assembler wasted a few bytes in the encoding, so NOPs have been added to keep the layout identical. The binary is only 1920 bytes. IBMDOS.COM is 6400 bytes, and I might be looking into that one in the future as well.

;----------------------------------------------------------------------------- ; DOS 1.0 IBMBIO.COM (disk image MD5 73c919cecadf002a7124b7e8bfe3b5ba) ; Â http://www.pagetable.com/ ;----------------------------------------------------------------------------- SECTOR_SIZE Â Â equ Â Â 0x0200Â Â Â Â Â ; size of a sector DOS_SIZEÂ Â Â Â equ Â Â 10000 Â Â Â Â Â ; max size of IBMDOS.COM in bytes PAUSE_KEY Â Â Â equ Â Â 0x7200Â Â Â Â Â ; scancode + charcode of PAUSE key KEYBUF_NEXT Â Â equ Â Â 0x041AÂ Â Â Â Â ; next character in keyboard buffer KEYBUF_FREE Â Â equ Â Â 0x041CÂ Â Â Â Â ; next free slot in keyboard buffer KEYBUFÂ Â Â Â Â equ Â Â 0x041EÂ Â Â Â Â ; keyboard buffer data LOGICAL_DRIVE Â equ Â Â 0x0504Â Â Â Â Â ; linear address of logical drive byte SEG_DOS_TEMPÂ Â equ Â Â 0xE0Â Â Â Â Â Â ; segment in which DOS was loaded SEG_DOS Â Â Â Â equ Â Â 0xB1Â Â Â Â Â Â ; segment in which DOS will run SEG_BIO Â Â Â Â equ Â Â 0x60Â Â Â Â Â Â ; segment in which BIO is running ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â org 0x0000Â Â Â Â Â Â Â ; segment 0x0060 Â Â Â Â Â Â Â Â jmp Â Â INITÂ Â Â Â Â Â ; 0x0060:0x0000 entry point Â Â Â Â Â Â Â Â jmp Â Â STATUSÂ Â Â Â Â ; 0x0060:0x0003 check for keypress Â Â Â Â Â Â Â Â jmp Â Â INP Â Â Â Â Â Â ; 0x0060:0x0006 get key from keyboard Â Â Â Â Â Â Â Â jmp Â Â OUTPÂ Â Â Â Â Â ; 0x0060:0x0009 send character to screen Â Â Â Â Â Â Â Â jmp Â Â PRINT Â Â Â Â Â ; 0x0060:0x000C send character to printer Â Â Â Â Â Â Â Â jmp Â Â AUXIN Â Â Â Â Â ; 0x0060:0x000F get character from serial Â Â Â Â Â Â Â Â jmp Â Â AUXOUTÂ Â Â Â Â ; 0x0060:0x0012 send character to serial Â Â Â Â Â Â Â Â jmp Â Â READÂ Â Â Â Â Â ; 0x0060:0x0015 read sector(s) from disk (INT 0x25) Â Â Â Â Â Â Â Â jmp Â Â WRITE Â Â Â Â Â ; 0x0060:0x0018 write sector(s) to diskÂ (INT 0x26) Â Â Â Â Â Â Â Â jmp Â Â DSKCHGÂ Â Â Â Â ; 0x0060:0x001B check for disk change ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â dw SEG_DOSÂ Â Â Â Â Â Â ; ??? Â Â Â Â Â Â Â Â dw TXT_VERSIONÂ Â Â Â Â ; ??? TXT_VERSION Â Â db 'BIOS Version 1.00' Â Â Â Â Â Â Â Â db ' '+0x80 Â Â Â Â Â Â Â Â db '22-Jul-81',0 ;----------------------------------------------------------------------------- ERR_PAPER Â Â Â db 13,10,'Out of pape','r'+0x80,13,10,0 ERR_PRINTER Â Â db 13,10,'Printer faul','t'+0x80,13,10,0 ERR_AUX Â Â Â Â db 13,10,'Aux I/O erro','r'+0x80,13,10,0 ;----------------------------------------------------------------------------- ; check for keypress ;Â AL = character ;Â ZÂ = set if no character ;Â all other registers preserved ;----------------------------------------------------------------------------- STATUSÂ Â Â Â Â mov Â Â al, [cs:next_char]; check for waiting character Â Â Â Â Â Â Â Â orÂ Â Â al, al Â Â Â Â Â Â Â Â jnz Â Â char_availÂ Â Â ; yes, return it Â Â Â Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â xchgÂ Â ax, dx Â Â Â Â Â Â Â Â mov Â Â ah, 1 Â Â Â Â Â Â Â Â int Â Â 0x16Â Â Â Â Â Â ; otherwise get key (don't clear) Â Â Â Â Â Â Â Â jzÂ Â Â status_exit Â Â ; no key Â Â Â Â Â Â Â Â cmp Â Â ax, PAUSE_KEY Â ; PAUSE key? Â Â Â Â Â Â Â Â jnz Â Â status_exit Â Â Â Â Â Â Â Â mov Â Â al, 0x10Â Â Â Â ; convert into Ctrl+P Â Â Â Â Â Â Â Â orÂ Â Â al, al status_exit Â Â mov Â Â ah, dhÂ Â Â Â Â ; restore original AH Â Â Â Â Â Â Â Â pop Â Â dx char_availÂ Â Â retf ;----------------------------------------------------------------------------- ; Interrupt 0x1B handler: Control+Break handler ;----------------------------------------------------------------------------- int_1BÂ Â Â Â Â mov Â Â byte [cs:next_char], 3; put code for Ctrl+C Â Â Â Â Â Â Â Â iretÂ Â Â Â Â Â Â Â Â Â ; into keyboard queue ;----------------------------------------------------------------------------- ; Interrupt 0x00 handler: Division by Zero ;----------------------------------------------------------------------------- int_00Â Â Â Â Â sti Â Â Â Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â mov Â Â dx, ERR_DIVIDE Â Â Â Â Â Â Â Â callÂ Â print_string Â Â Â Â Â Â Â Â pop Â Â dx Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â int Â Â 0x23Â Â Â Â Â Â ; exit program through Ctrl+C path ;----------------------------------------------------------------------------- ; Interrupt 0x00 handler: Single Step ; Interrupt 0x03 handler: Breakpoint ; Interrupt 0x04 handler: Overflow ;----------------------------------------------------------------------------- iret1 Â Â Â Â Â iretÂ Â Â Â Â Â Â Â Â Â ; empty interrupt handler ;----------------------------------------------------------------------------- ERR_DIVIDEÂ Â Â db 13,10,'Divide overflo','w'+0x80,13,10,0 ;----------------------------------------------------------------------------- ; get key from keyboard ;Â AL = character ;Â all other registers preserved ;----------------------------------------------------------------------------- again Â Â Â Â Â xchgÂ Â ax, dx Â Â Â Â Â Â Â Â pop Â Â dx INP Â Â Â Â Â Â mov Â Â al, 0 Â Â Â Â Â Â Â Â xchgÂ Â al, [cs:next_char]; get and clear waiting character Â Â Â Â Â Â Â Â orÂ Â Â al, al Â Â Â Â Â Â Â Â jnz Â Â inp_exitÂ Â Â Â ; there is no character waiting Â Â Â Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â xchgÂ Â ax, dx Â Â Â Â Â Â Â Â mov Â Â ah, 0 Â Â Â Â Â Â Â Â int Â Â 0x16Â Â Â Â Â Â ; then read character from keyboard Â Â Â Â Â Â Â Â orÂ Â Â ax, ax Â Â Â Â Â Â Â Â jzÂ Â Â again Â Â Â Â Â Â Â Â cmp Â Â ax, PAUSE_KEY Â Â Â Â Â Â Â Â jnz Â Â not_pause2 Â Â Â Â Â Â Â Â mov Â Â al, 0x10Â Â Â Â ; Ctrl+P not_pause2Â Â Â cmp Â Â al, 0 Â Â Â Â Â Â Â Â jnz Â Â skip1 Â Â Â Â Â ; key with ASCII representation Â Â Â Â Â Â Â Â mov Â Â [cs:next_char], ah; return scancode next time skip1 Â Â Â Â Â mov Â Â ah, dhÂ Â Â Â Â ; restore AH Â Â Â Â Â Â Â Â pop Â Â dx inp_exitÂ Â Â Â retf ;----------------------------------------------------------------------------- ; send character to screen ;Â AL = character ;Â all registers preserved ;----------------------------------------------------------------------------- OUTPÂ Â Â Â Â Â pushÂ Â bp Â Â Â Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â pushÂ Â bx Â Â Â Â Â Â Â Â pushÂ Â si Â Â Â Â Â Â Â Â pushÂ Â di Â Â Â Â Â Â Â Â mov Â Â ah, 0x0E Â Â Â Â Â Â Â Â csÂ Â Â Â Â Â Â Â Â Â Â ; XXX makes no sense Â Â Â Â Â Â Â Â mov Â Â bx, 7 Â Â Â Â Â Â Â Â int Â Â 0x10Â Â Â Â Â Â ; print character Â Â Â Â Â Â Â Â pop Â Â di Â Â Â Â Â Â Â Â pop Â Â si Â Â Â Â Â Â Â Â pop Â Â bx Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â pop Â Â bp Â Â Â Â Â Â Â Â retf ;----------------------------------------------------------------------------- ; send character to printer ;Â AL = character ;Â all registers preserved ;----------------------------------------------------------------------------- PRINT Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â mov Â Â byte [cs:printer_retry], 0 printer_again Â mov Â Â dx, 0 Â Â Â Â Â ; printer port #0 Â Â Â Â Â Â Â Â mov Â Â ah, 0 Â Â Â Â Â Â Â Â int Â Â 0x17Â Â Â Â Â Â ; send character to printer Â Â Â Â Â Â Â Â mov Â Â dx, ERR_PAPER Â Â Â Â Â Â Â Â testÂ Â ah, 0x20 Â Â Â Â Â Â Â Â jnz Â Â printer_error Â ; out of paper error Â Â Â Â Â Â Â Â mov Â Â dx, ERR_PRINTER Â Â Â Â Â Â Â Â testÂ Â ah, 5 Â Â Â Â Â Â Â Â jzÂ Â Â pop_dx_ax_retfÂ ; no timeout error, return Â Â Â Â Â Â Â Â xor Â Â byte [cs:printer_retry], 1 Â Â Â Â Â Â Â Â jnz Â Â printer_again Â ; on a timeout, try twice printer_error Â callÂ Â print_string pop_dx_ax_retfÂ pop Â Â dx Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â retf ;----------------------------------------------------------------------------- ; print zero-terminated string at DS:DX ;----------------------------------------------------------------------------- print_stringÂ Â xchgÂ Â si, dx prints1 Â Â Â Â cs lodsb Â Â Â Â Â Â Â Â and Â Â al, 0x7FÂ Â Â Â ; clear bit 7 (XXX why?) Â Â Â Â Â Â Â Â jzÂ Â Â prints2 Â Â Â Â ; zero-terminated Â Â Â Â Â Â Â Â callÂ Â SEG_BIO:OUTPÂ Â ; print character Â Â Â Â Â Â Â Â jmp Â Â prints1 Â Â Â Â ; loop prints2 Â Â Â Â xchgÂ Â si, dx Â Â Â Â Â Â Â Â retn ;----------------------------------------------------------------------------- ; get character from serial ;Â AL = character ;Â all other registers preserved ;----------------------------------------------------------------------------- AUXIN Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â mov Â Â dx, 0 Â Â Â Â Â ; serial port #0 Â Â Â Â Â Â Â Â mov Â Â ah, 2 Â Â Â Â Â Â Â Â int Â Â 0x14Â Â Â Â Â Â ; get character from serial port Â Â Â Â Â Â Â Â mov Â Â dx, ERR_AUX Â Â Â Â Â Â Â Â testÂ Â ah, 0x0EÂ Â Â Â ; framing, parity or overrun? Â Â Â Â Â Â Â Â jzÂ Â Â aux_noerr Â Â Â ; no error Â Â Â Â Â Â Â Â callÂ Â print_string aux_noerr Â Â Â pop Â Â dx Â Â Â Â Â Â Â Â mov Â Â ah, dhÂ Â Â Â Â ; restore AH Â Â Â Â Â Â Â Â pop Â Â dx Â Â Â Â Â Â Â Â retf ;----------------------------------------------------------------------------- ; send character to serial ;Â AL = character ;Â all registers preserved ;----------------------------------------------------------------------------- AUXOUTÂ Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â mov Â Â ah, 1 Â Â Â Â Â Â Â Â mov Â Â dx, 0 Â Â Â Â Â Â Â Â int Â Â 0x14Â Â Â Â Â Â ; send character to serial port Â Â Â Â Â Â Â Â testÂ Â ah, 0x80Â Â Â Â ; timeout error? Â Â Â Â Â Â Â Â jzÂ Â Â pop_dx_ax_retfÂ ; no all fine Â Â Â Â Â Â Â Â mov Â Â dx, ERR_AUX Â Â Â Â Â Â Â Â jmp Â Â printer_error ;----------------------------------------------------------------------------- ; check for disk change ;Â AH = flag (1=changed) ;----------------------------------------------------------------------------- DSKCHGÂ Â Â Â Â mov Â Â ah, 0 Â Â Â Â Â ; the IBM PC can't detect disk change Â Â Â Â Â Â Â Â retf temp_sector: ;----------------------------------------------------------------------------- ; entry point from boot sector ;Â assumes DX = 0 ;----------------------------------------------------------------------------- INITÂ Â Â Â Â Â cli Â Â Â Â Â Â Â Â mov Â Â ax, cs Â Â Â Â Â Â Â Â mov Â Â ds, ax Â Â Â Â Â Â Â Â mov Â Â ss, ax Â Â Â Â Â Â Â Â mov Â Â sp, temp_sector_end; set stack used during init Â Â Â Â Â Â Â Â sti Â Â Â Â Â Â Â Â xor Â Â ah, ah Â Â Â Â Â Â Â Â int Â Â 0x13Â Â Â Â Â Â ; reset disk 0 (DX = 0) Â Â Â Â Â Â Â Â mov Â Â al, 0xA3Â Â Â Â ; 2400 8N1 Â Â Â Â Â Â Â Â int Â Â 0x14Â Â Â Â Â Â ; initialize serial port Â Â Â Â Â Â Â Â mov Â Â ah, 1 Â Â Â Â Â Â Â Â int Â Â 0x17Â Â Â Â Â Â ; initialize printer Â Â Â Â Â Â Â Â int Â Â 0x11Â Â Â Â Â Â ; get system info Â Â Â Â Â Â Â Â and Â Â ax, 0xC0Â Â Â Â ; number of floppies in bits 6 and 7 Â Â Â Â Â Â Â Â mov Â Â cx, 5 Â Â Â Â Â Â Â Â shr Â Â ax, clÂ Â Â Â Â ; (floppies-1) * 2 Â Â Â Â Â Â Â Â add Â Â ax, 2 Â Â Â Â Â ; floppies * 2 Â Â Â Â Â Â Â Â and Â Â ax, 6 Â Â Â Â Â ; will become 0 for 4 floppies Â Â Â Â Â Â Â Â jzÂ Â Â four_floppies Â ; 4 floppies (num_floppies pre-assigned with 4) Â Â Â Â Â Â Â Â cmp Â Â al, 2 Â Â Â Â Â ; one floppy? Â Â Â Â Â Â Â Â jnz Â Â multi_floppyÂ Â ; no Â Â Â Â Â Â Â Â shl Â Â ax, 1 Â Â Â Â Â ; pretend we have two, we'll emulate one Â Â Â Â Â Â Â Â mov Â Â byte [single_floppy], 1 multi_floppyÂ Â mov Â Â bx, floppy_list Â Â Â Â Â Â Â Â add Â Â bx, axÂ Â Â Â Â ; + floppies * 2 Â Â Â Â Â Â Â Â mov Â Â word [bx], 0Â Â ; terminate list with 2 zero words Â Â Â Â Â Â Â Â mov Â Â word [bx+2], 0 Â Â Â Â Â Â Â Â nop Â Â Â Â Â Â Â Â Â Â ; XXX original assembler wasted a byte Â Â Â Â Â Â Â Â shr Â Â ax, 1 Â Â Â Â Â ; =floppies Â Â Â Â Â Â Â Â mov Â Â [num_floppies], al four_floppies Â pushÂ Â ds Â Â Â Â Â Â Â Â mov Â Â ax, 0 Â Â Â Â Â Â Â Â mov Â Â ds, axÂ Â Â Â Â ; DS := 0x0000 Â Â Â Â Â Â Â Â mov Â Â ax, SEG_BIO Â Â ; target segment for interrupt vectors Â Â Â Â Â Â Â Â mov Â Â [0x6E], axÂ Â Â ; set INT 1Bh segment Â Â Â Â Â Â Â Â mov Â Â word [0x6C], int_1B; set INT 1Bh offset Â Â Â Â Â Â Â Â mov Â Â word [0x00], int_00; set INT 00h offset Â Â Â Â Â Â Â Â mov Â Â [0x02], axÂ Â Â ; set INT 00h segment Â Â Â Â Â Â Â Â mov Â Â bx, iret1 Â Â Â ; set INT 00h offset Â Â Â Â Â Â Â Â mov Â Â [0x04], bxÂ Â Â ; set INT 01h offset (empty) Â Â Â Â Â Â Â Â mov Â Â [0x06], axÂ Â Â ; set INT 01h segment Â Â Â Â Â Â Â Â mov Â Â [0x0C], bxÂ Â Â ; set INT 03h offset (empty) Â Â Â Â Â Â Â Â mov Â Â [0x0E], axÂ Â Â ; set INT 03h segment Â Â Â Â Â Â Â Â mov Â Â [0x10], bxÂ Â Â ; set INT 04h offset (empty) Â Â Â Â Â Â Â Â mov Â Â [0x12], axÂ Â Â ; set INT 04h segment Â Â Â Â Â Â Â Â mov Â Â ax, 0x50 Â Â Â Â Â Â Â Â mov Â Â ds, axÂ Â Â Â Â ; DS := 0x0050 Â Â Â Â Â Â Â Â mov Â Â word [0x0], 0 Â ; clear 0x0500 in DOS Comm. Area (???) Â Â Â Â Â Â Â Â pushÂ Â es Â Â Â Â Â Â Â Â mov Â Â ax, SEG_DOS Â Â ; target segment for IBMDOS.COM Â Â Â Â Â Â Â Â mov Â Â es, ax Â Â Â Â Â Â Â Â mov Â Â cx, DOS_SIZE/2Â ; size/2 of IBMDOS.COM Â Â Â Â Â Â Â Â cld Â Â Â Â Â Â Â Â mov Â Â ax, SEG_DOS_TEMP; source segment of IBMDOS.COM Â Â Â Â Â Â Â Â mov Â Â ds, axÂ Â Â Â Â ; the booloader read whole sectors and puts Â Â Â Â Â Â Â Â xor Â Â di, diÂ Â Â Â Â ; the IBMDOS.COM image right after this; Â Â Â Â Â Â Â Â mov Â Â si, diÂ Â Â Â Â ; so move it down a little Â Â Â Â Â Â Â Â rep movsw Â Â Â Â Â Â Â ; copy 10 000 bytes from 0xE00 to 0xB10 Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â mov Â Â si, num_floppies; pass in pointer to structure Â Â Â Â Â Â Â Â callÂ Â SEG_DOS:0 Â Â Â ; init DOS (returns DS = memory for COMMAND.COM) Â Â Â Â Â Â Â Â sti Â Â Â Â Â Â Â Â mov Â Â dx, 0x0100Â Â Â ; 0x0100 in COMMAND.COM segment Â Â Â Â Â Â Â Â mov Â Â ah, 0x1A Â Â Â Â Â Â Â Â int Â Â 0x21Â Â Â Â Â Â ; set disk transfer area address Â Â Â Â Â Â Â Â mov Â Â cx, [0x06]Â Â Â ; remaining memory size Â Â Â Â Â Â Â Â sub Â Â cx, 0x0100Â Â Â ; - Program Segment Prefix = bytes to read Â Â Â Â Â Â Â Â mov Â Â bx, ds Â Â Â Â Â Â Â Â mov Â Â ax, cs Â Â Â Â Â Â Â Â mov Â Â ds, ax Â Â Â Â Â Â Â Â mov Â Â dx, FCB_command_com; File Control Block Â Â Â Â Â Â Â Â mov Â Â ah, 0x0F Â Â Â Â Â Â Â Â int Â Â 0x21Â Â Â Â Â Â ; DOS: open COMMAND.COM Â Â Â Â Â Â Â Â orÂ Â Â al, al Â Â Â Â Â Â Â Â jnz Â Â error_command Â ; error opening COMMAND.COM Â Â Â Â Â Â Â Â mov Â Â word [FCB_command_com+0x21], 0; random record field Â Â Â Â Â Â Â Â mov Â Â word [FCB_command_com+0x23], 0;Â := 0x00000000 Â Â Â Â Â Â Â Â mov Â Â word [FCB_command_com+0x0E], 1; record length = 1 byte Â Â Â Â Â Â Â Â mov Â Â ah, 0x27 Â Â Â Â Â Â Â Â int Â Â 0x21Â Â Â Â Â Â ; DOS: read Â Â Â Â Â Â Â Â jcxzÂ Â error_command Â ; read 0 bytes -> error Â Â Â Â Â Â Â Â cmp Â Â al, 1 Â Â Â Â Â Â Â Â jnz Â Â error_command Â ; end of file not reached -> error Â Â Â Â Â Â Â Â mov Â Â ds, bx Â Â Â Â Â Â Â Â mov Â Â es, bxÂ Â Â Â Â ; DS := ES := SS := COMMAND.COM Â Â Â Â Â Â Â Â mov Â Â ss, bx Â Â Â Â Â Â Â Â mov Â Â sp, 0x40Â Â Â Â ; 64 byte stack in PSP (XXX interrupts are on!) Â Â Â Â Â Â Â Â xor Â Â ax, ax Â Â Â Â Â Â Â Â pushÂ Â axÂ Â Â Â Â Â Â ; push return address 0x0000 (int 0x20) Â Â Â Â Â Â Â Â mov Â Â dx, [0x80]Â Â Â ; get new DTA address Â Â Â Â Â Â Â Â mov Â Â ah, 0x1A Â Â Â Â Â Â Â Â int Â Â 0x21Â Â Â Â Â Â ; set disk transfer area address Â Â Â Â Â Â Â Â pushÂ Â bxÂ Â Â Â Â Â Â ; segment of COMMAND.COM Â Â Â Â Â Â Â Â mov Â Â ax, 0x0100Â Â Â ; offset of COMMAND.COM entry Â Â Â Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â retfÂ Â Â Â Â Â Â Â Â Â ; run COMMAND.COM error_command:Â mov Â Â dx, ERR_COMMANDCOM ; "\r\nBad or missing Command Interprete" Â Â Â Â Â Â Â Â callÂ Â print_string haltÂ Â Â Â Â Â jpÂ Â Â haltÂ Â ; XXX jp instead of jmp ;----------------------------------------------------------------------------- FCB_command_com db 1, 'COMMAND CO','M'+0x80 Â Â Â Â Â Â Â Â times 19h db 0 ;----------------------------------------------------------------------------- ERR_COMMANDCOMÂ db 13,10,'Bad or missing Command Interprete','r'+0x80,13,10,0 ;----------------------------------------------------------------------------- ; this is passed to IBMDOS.COM num_floppiesÂ Â db 4Â Â Â Â Â Â Â Â Â Â ; if there's 1 physical drive, this says 2 floppy_list Â Â dw parameters Â Â Â Â Â ; point to params for every floppy installed; 0-terminated Â Â Â Â Â Â Â Â dw parameters Â Â Â Â Â Â Â Â dw parameters Â Â Â Â Â Â Â Â dw parameters Â Â Â Â Â Â Â Â dw 0,0 parametersÂ Â Â dw SECTOR_SIZE Â Â Â Â Â Â Â Â db 1Â Â Â Â Â Â Â Â Â Â ; will be decremented by 1, then used Â Â Â Â Â Â Â Â dw 1 Â Â Â Â Â Â Â Â db 2 Â Â Â Â Â Â Â Â dw 0x0040 Â Â Â Â Â Â Â Â dw 320Â Â Â Â Â Â Â Â Â ; number of total sectors ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â times 512-($-temp_sector) db 0 temp_sector_end: ;----------------------------------------------------------------------------- printer_retry Â db 0Â Â Â Â Â Â Â Â Â Â ; count for printer retries next_char Â Â Â db 0Â Â Â Â Â Â Â Â Â Â ; extra character in keyboard queue Â Â Â Â Â Â Â Â db 0Â Â Â Â Â Â Â Â Â Â ; XXX unused single_floppy Â db 0Â Â Â Â Â Â Â Â Â Â ; true if we emulate a second logical floppy ;----------------------------------------------------------------------------- ; READÂ - read sector(s) from disk ; WRITE - write sector(s) to disk ;Â al Â Â drive number (0-3) ;Â ds:bxÂ buffer ;Â cx Â Â count ;Â dx Â Â logical block number ;----------------------------------------------------------------------------- READÂ Â Â Â Â Â mov Â Â ah, 2 Â Â Â Â Â ; BIOS code "read" Â Â Â Â Â Â Â Â jmp Â Â short read_write WRITE Â Â Â Â Â mov Â Â ah, 3 Â Â Â Â Â ; BIOS code "write" read_writeÂ Â Â pushÂ Â es Â Â Â Â Â Â Â Â pushÂ Â ds Â Â Â Â Â Â Â Â pushÂ Â ds Â Â Â Â Â Â Â Â pop Â Â esÂ Â Â Â Â Â Â ; ES := DS Â Â Â Â Â Â Â Â pushÂ Â cs Â Â Â Â Â Â Â Â pop Â Â dsÂ Â Â Â Â Â Â ; DS := CS Â Â Â Â Â Â Â Â mov Â Â [temp_sp], sp Â ; save sp for function abort Â Â Â Â Â Â Â Â mov Â Â [int_13_cmd], ah; save whether it was read or write ; logic to emulate a "logical" drive B: by prompting the user to change disk ; when the currently used drive is changed Â Â Â Â Â Â Â Â cmp Â Â byte [single_floppy], 1 Â Â Â Â Â Â Â Â jnz Â Â multi_drive Â Â ; more than one drive Â Â Â Â Â Â Â Â pushÂ Â ds Â Â Â Â Â Â Â Â xor Â Â si, si Â Â Â Â Â Â Â Â mov Â Â ds, siÂ Â Â Â Â ; DS := 0x0000 Â Â Â Â Â Â Â Â mov Â Â ah, al Â Â Â Â Â Â Â Â xchgÂ Â ah, [LOGICAL_DRIVE]; current logical drive Â Â Â Â Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â cmp Â Â al, ah Â Â Â Â Â Â Â Â jzÂ Â Â drive_unchanged Â Â Â Â Â Â Â Â pushÂ Â dxÂ Â Â Â Â Â Â ; save block number Â Â Â Â Â Â Â Â add Â Â al, 'A' Â Â Â Â Â Â Â Â mov Â Â [TXT_DRIVE], al Â Â Â Â Â Â Â Â mov Â Â dx, TXT_INSERTDISK Â Â Â Â Â Â Â Â callÂ Â print_stringÂ Â ; prompt for disk change Â Â Â Â Â Â Â Â pushÂ Â ds Â Â Â Â Â Â Â Â xor Â Â bp, bp Â Â Â Â Â Â Â Â mov Â Â ds, bp Â Â Â Â Â Â Â Â mov Â Â byte [KEYBUF_NEXT], KEYBUF & 0xFF Â Â Â Â Â Â Â Â mov Â Â byte [KEYBUF_FREE], KEYBUF & 0xFF; clear keyboard buffer Â Â Â Â Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â mov Â Â ah, 0 Â Â Â Â Â Â Â Â int Â Â 0x16Â Â Â Â Â Â ; wait for any key Â Â Â Â Â Â Â Â pop Â Â dxÂ Â Â Â Â Â Â ; block number drive_unchanged mov Â Â al, 0 Â Â Â Â Â ; for both logical A: or B: use drive A: multi_drive Â Â xchgÂ Â ax, dx Â Â Â Â Â Â Â Â mov Â Â dh, 8 Â Â Â Â Â ; convert LBA to CHS Â Â Â Â Â Â Â Â div Â Â dhÂ Â Â Â Â Â Â ; al = track (starts at 0) Â Â Â Â Â Â Â Â inc Â Â ahÂ Â Â Â Â Â Â ; ah = sector (starts at 1) Â Â Â Â Â Â Â Â xchgÂ Â al, ahÂ Â Â Â Â ; track and sector Â Â Â Â Â Â Â Â xchgÂ Â ax, cxÂ Â Â Â Â ; cx = t/s, ax = count Â Â Â Â Â Â Â Â mov Â Â [num_sectors], ax; count Â Â Â Â Â Â Â Â mov Â Â dh, 0 ; work around DMA hardware bug in case I/O spans a 64 KB boundary ; by using a temporary buffer Â Â Â Â Â Â Â Â mov Â Â di, esÂ Â Â Â Â ; destination segment Â Â Â Â Â Â Â Â shl Â Â di, 1 Â Â Â Â Â Â Â Â shl Â Â di, 1 Â Â Â Â Â ; make es:bx a linear address Â Â Â Â Â Â Â Â shl Â Â di, 1 Â Â Â Â Â ; (discard upper bits) Â Â Â Â Â Â Â Â shl Â Â di, 1 Â Â Â Â Â Â Â Â add Â Â di, bx Â Â Â Â Â Â Â Â add Â Â di, SECTOR_SIZE-1; last byte of sector (linear) Â Â Â Â Â Â Â Â jcÂ Â Â across_64kÂ Â Â ; sector overflows it Â Â Â Â Â Â Â Â xchgÂ Â bx, diÂ Â Â Â Â ; bx = last byte, di = buffer Â Â Â Â Â Â Â Â shr Â Â bh, 1 Â Â Â Â Â ; sector index in memory Â Â Â Â Â Â Â Â mov Â Â ah, 0x80Â Â Â Â ; 0x80 sectors fit into 64 KB Â Â Â Â Â Â Â Â sub Â Â ah, bhÂ Â Â Â Â ; sectors until 64 KB boundary Â Â Â Â Â Â Â Â mov Â Â bx, diÂ Â Â Â Â ; bx = buffer Â Â Â Â Â Â Â Â cmp Â Â ah, alÂ Â Â Â Â ; compare to number of sectors Â Â Â Â Â Â Â Â jbe Â Â skip2 Â Â Â Â Â ; they fit into 64 KB, cap num Â Â Â Â Â Â Â Â mov Â Â ah, alÂ Â Â Â Â ; don't cap number of sectors skip2 Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â mov Â Â al, ahÂ Â Â Â Â ; al = count Â Â Â Â Â Â Â Â callÂ Â rw_tracks Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â sub Â Â al, ahÂ Â Â Â Â ; requested = done? Â Â Â Â Â Â Â Â jzÂ Â Â rw_done Â Â Â Â ; yes, exit across_64kÂ Â Â dec Â Â alÂ Â Â Â Â Â Â ; one sector less Â Â Â Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â cld Â Â Â Â Â Â Â Â pushÂ Â bx Â Â Â Â Â Â Â Â pushÂ Â esÂ Â Â Â Â Â Â ; save data pointer Â Â Â Â Â Â Â Â cmp Â Â byte [int_13_cmd], 2 Â Â Â Â Â Â Â Â jzÂ Â Â across_64k_read ; write case follows Â Â Â Â Â Â Â Â mov Â Â si, bx Â Â Â Â Â Â Â Â pushÂ Â cx Â Â Â Â Â Â Â Â mov Â Â cx, SECTOR_SIZE/2; copy first sector Â Â Â Â Â Â Â Â pushÂ Â es Â Â Â Â Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â pushÂ Â cs Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â mov Â Â di, temp_sector Â Â Â Â Â Â Â Â mov Â Â bx, di Â Â Â Â Â Â Â Â rep movsw Â Â Â Â Â Â Â ; copy into IBMBIO local data Â Â Â Â Â Â Â Â pop Â Â cx Â Â Â Â Â Â Â Â pushÂ Â cs Â Â Â Â Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â callÂ Â rw_one_sector Â ; write last sector Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â pop Â Â bx Â Â Â Â Â Â Â Â jmp Â Â short across_64k_end across_64k_read mov Â Â bx, temp_sector Â Â Â Â Â Â Â Â pushÂ Â cs Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â callÂ Â rw_one_sector Â ; read last sector into temp buffer Â Â Â Â Â Â Â Â mov Â Â si, bx Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â pop Â Â bx Â Â Â Â Â Â Â Â mov Â Â di, bx Â Â Â Â Â Â Â Â pushÂ Â cx Â Â Â Â Â Â Â Â mov Â Â cx, SECTOR_SIZE/2 Â Â Â Â Â Â Â Â rep movsw Â Â Â Â Â Â Â ; copy out Â Â Â Â Â Â Â Â pop Â Â cx across_64k_endÂ add Â Â bh, 2 Â Â Â Â Â ; continue 0x0200 after that Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â callÂ Â rw_tracks rw_done Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â clc Â Â Â Â Â Â Â Â Â Â ; success Â Â Â Â Â Â Â Â retf ;----------------------------------------------------------------------------- ; read/write an arbirtary number of sectors ;----------------------------------------------------------------------------- rw_tracks Â Â Â orÂ Â Â al, al Â Â Â Â Â Â Â Â jzÂ Â Â ret2Â Â Â Â Â Â ; nothing to read Â Â Â Â Â Â Â Â mov Â Â ah, 9 Â Â Â Â Â Â Â Â sub Â Â ah, cl Â Â Â Â Â Â Â Â cmp Â Â ah, alÂ Â Â Â Â ; more sectors than left in track? Â Â Â Â Â Â Â Â jbe Â Â skip3 Â Â Â Â Â ; no Â Â Â Â Â Â Â Â mov Â Â ah, alÂ Â Â Â Â ; otherwise, read up to end of track skip3 Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â mov Â Â al, ah Â Â Â Â Â Â Â Â callÂ Â near rw_sectors ; reads/writes up to 8 sectors Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â sub Â Â al, ahÂ Â Â Â Â ; decrease sectors to read Â Â Â Â Â Â Â Â shl Â Â ah, 1 Â Â Â Â Â Â Â Â add Â Â bh, ahÂ Â Â Â Â ; advance pointer by sectors * 0x0200 Â Â Â Â Â Â Â Â jmp Â Â rw_tracks Â Â Â ; continue ;----------------------------------------------------------------------------- int_13_errÂ Â Â xchgÂ Â ax, di Â Â Â Â Â Â Â Â mov Â Â ah, 0 Â Â Â Â Â Â Â Â int Â Â 0x13Â Â Â Â Â Â ; disk reset Â Â Â Â Â Â Â Â dec Â Â si Â Â Â Â Â Â Â Â jzÂ Â Â translate Â Â Â ; retries exhausted Â Â Â Â Â Â Â Â mov Â Â ax, di Â Â Â Â Â Â Â Â cmp Â Â ah, 0x80Â Â Â Â ; in the "timeout (not ready)" case, Â Â Â Â Â Â Â Â jzÂ Â Â translate Â Â Â ; we don't retry (this would take forever) Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â jmp Â Â short retry translate Â Â Â pushÂ Â cs Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â mov Â Â ax, di Â Â Â Â Â Â Â Â mov Â Â al, ahÂ Â Â Â Â ; status Â Â Â Â Â Â Â Â mov Â Â cx, 0x0A Â Â Â Â Â Â Â Â mov Â Â di, conv_status Â Â Â Â Â Â Â Â repne scasb Â Â Â Â Â Â Â Â mov Â Â al, [di+9] Â Â Â Â Â Â Â Â nop Â Â Â Â Â Â Â Â Â Â ; XXX original assembler wasted a byte Â Â Â Â Â Â Â Â mov Â Â cx, [num_sectors] Â Â Â Â Â Â Â Â mov Â Â sp, [temp_sp] Â ; clean up stack Â Â Â Â Â Â Â Â pop Â Â ds Â Â Â Â Â Â Â Â pop Â Â es Â Â Â Â Â Â Â Â stc Â Â Â Â Â Â Â Â Â Â ; error Â Â Â Â Â Â Â Â retf rw_one_sector Â mov Â Â al, 1 ; reads/writes one or more sectors that are on the same track rw_sectorsÂ Â Â mov Â Â si, 5 Â Â Â Â Â ; number of retries Â Â Â Â Â Â Â Â mov Â Â ah, [int_13_cmd] retry Â Â Â Â Â pushÂ Â ax Â Â Â Â Â Â Â Â int Â Â 0x13Â Â Â Â Â Â ; perform the read/write Â Â Â Â Â Â Â Â jcÂ Â Â int_13_err Â Â Â Â Â Â Â Â pop Â Â ax Â Â Â Â Â Â Â Â sub Â Â [num_sectors], al Â Â Â Â Â Â Â Â add Â Â cl, alÂ Â Â Â Â ; calculate next sector number Â Â Â Â Â Â Â Â cmp Â Â cl, 8 Â Â Â Â Â ; exceeds track? Â Â Â Â Â Â Â Â jbe Â Â ret2Â Â Â Â Â Â ; no Â Â Â Â Â Â Â Â inc Â Â chÂ Â Â Â Â Â Â ; next track Â Â Â Â Â Â Â Â mov Â Â cl, 1 Â Â Â Â Â ; sector 1 ret2Â Â Â Â Â Â retn ;----------------------------------------------------------------------------- TXT_INSERTDISKÂ db 13,10,'Insert diskette for drive',' '+0x80 TXT_DRIVE Â Â Â db 'A: and strik','e'+0x80,13,10 Â Â Â Â Â Â Â Â db 'any key when read','y'+0x80,13,10,10,0 ;----------------------------------------------------------------------------- conv_status Â Â db 0x80,0x40,0x20,0x10,9,8,4,3,2; BIOS error codes Â Â Â Â Â Â Â Â db 1,2,6,0x0C,4,0x0C,4,8,0,0x0C,0x0C; IBMBIO error codes ;----------------------------------------------------------------------------- int_13_cmdÂ Â Â db 2 Â Â temp_sp Â Â Â Â dw 0 Â Â num_sectors Â Â db 0 Â Â ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â times 513 db 0 Â Â Â Â Â Â Â Â db 0xC9 Â Â Â Â Â Â Â Â times 126 db 0 ;-----------------------------------------------------------------------------

Reverse-Engineering DOS 1.0 – Part 1: The Boot Sector

Michael Steil — Thu, 07 May 2009 08:01:25 +0000

The bootsector of DOS 1.0 is celebtaring its 28th birthday today (it contains the timestamp “7-May-81″), so let’s look at it more closely.

Here it is:

00000000 eb 2f 14 00 00 00 60 00 20 37 2d 4d 61 79 2d 38 |........ 7-May-8| 00000010 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |1...............| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 00 fa 8c c8 8e d8 ba 00 00 8e d2 bc 00 7c fb a1 |................| 00000040 06 7c 8e d8 8e c0 ba 00 00 8b c2 cd 13 72 41 e8 |................| 00000050 58 00 72 fb 2e 8b 0e 02 7c 51 bb 00 00 33 d2 b9 |................| 00000060 08 00 be 01 00 56 b0 01 b4 02 cd 13 72 22 5e 58 |................| 00000070 e8 e7 00 2b c6 74 14 fe c5 b1 01 be 08 00 3b c6 |................| 00000080 73 04 8b f0 eb 01 96 56 50 eb dd 2e ff 2e 04 7c |................| 00000090 be 44 7d b8 42 7d 50 32 ff ac 24 7f 74 0b 56 b4 |................| 000000a0 0e bb 07 00 cd 10 5e eb f0 c3 bb 00 00 b9 04 00 |................| 000000b0 b8 01 02 cd 13 1e 72 34 8c c8 8e d8 bf 00 00 b9 |................| 000000c0 0b 00 26 80 0d 20 26 80 8d 20 00 20 47 e2 f3 bf |................| 000000d0 00 00 be 76 7d b9 0b 00 fc f3 a6 75 0f bf 20 00 |................| 000000e0 be 82 7d b9 0b 00 f3 a6 75 02 1f c3 be f9 7c e8 |................| 000000f0 a5 ff b4 00 cd 16 1f f9 c3 0d 0a 4e 6f 6e 2d 53 |...........Non-S| 00000100 79 73 74 65 6d 20 64 69 73 6b 20 6f 72 20 64 69 |ystem disk or di| 00000110 73 6b 20 65 72 72 6f f2 0d 0a 52 65 70 6c 61 63 |sk erro?..Replac| 00000120 65 20 61 6e 64 20 73 74 72 69 6b 65 20 61 6e 79 |e and strike any| 00000130 20 6b 65 79 20 77 68 65 6e 20 72 65 61 64 f9 0d | key when read?.| 00000140 0a 00 cd 18 0d 0a 44 69 73 6b 20 42 6f 6f 74 20 |......Disk Boot | 00000150 66 61 69 6c 75 72 e5 0d 0a 00 50 52 8b c6 bf 00 |failur?.........| 00000160 02 f7 e7 03 d8 5a 58 c3 52 6f 62 65 72 74 20 4f |........Robert O| 00000170 27 52 65 61 72 20 69 62 6d 62 69 6f 20 20 63 6f |'Rear ibmbio co| 00000180 6d b0 69 62 6d 64 6f 73 20 20 63 6f 6d b0 c9 00 |m.ibmdos com...| 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000200

DOS 1.0 shipped on a 160 KB single sided disk. The boot code in the IBM PC’s BIOS loaded the first sector into RAM at segment 0×0000, offset 0x7C00 and ran it. Later versions of BIOS checked for 0xAA55 in the last word of the bootsector, but the first version did not. Note that DOS 1.0 is also pre-BIOS Parameter Block, i.e. the bootsector does not contain any information about the physical layout of the disk, since there was only a single disk size.

What the boot sector is supposed to do is read “IBMBIO.COM” and “IBMDOS.COM” into RAM and run them – these are the DOS system files for machine abstraction and DOS API, respectively. In MS-DOS, they would be called “IO.SYS” and “MSDOS.SYS”.

But the DOS 1.0 bootsector takes quite a lot of shortcuts. It assumes it’s always a 40 track, 8 sectors single-sided disk and the two files occupy the first sectors of the data area contiguously – something that SYS.COM could guarantee when making a disk bootable.

So the bootsector first loads the first sector of the root directory (hardcoded to track 0, sector 4) and compares the first two entries with “IBMBIO.COM” and “IBMDOS.COM”. For some reason, the comparison is case-insensitive, although DOS only allows uppercase filenames.

00000600 49 42 4d 42 49 4f 20 20 43 4f 4d 06 00 00 00 00 |IBMBIO COM.....| 00000610 00 00 00 00 00 00 00 00 f7 02 02 00 80 07 00 00 |................| 00000620 49 42 4d 44 4f 53 20 20 43 4f 4d 06 00 00 00 00 |IBMDOS COM.....| 00000630 00 00 00 00 00 00 00 00 0d 03 06 00 00 19 00 00 |................|

If they are not there, it prompts the user to replace the disk and tries again. Otherwise, it loads 20 sectors starting from track 0, sector 8 to segment 0×0060, offset 0×0000 into memory and jumps there.

0×60:0×0000 is the same as the linear address 0×0600. On the IBM PC, 0×0000 to 0x03FF are occupied by the interrupts vectors, 0×0400 to 0x4FF is used by BIOS for its variables, and DOS 1.0 uses 0×500 to 0x5FF as the “DOS Communication Area”, so code can start at 0×0600.

00000e00 e9 62 01 e9 6d 00 e9 b2 00 e9 d8 00 e9 e8 00 e9 |................| 00000e10 24 01 e9 3a 01 e9 51 03 e9 52 03 e9 44 01 b1 00 |................| 00000e20 22 00 42 49 4f 53 20 56 65 72 73 69 6f 6e 20 31 |..BIOS Version 1| 00000e30 2e 30 30 a0 32 32 2d 4a 75 6c 2d 38 31 00 0d 0a |.00.22-Jul-81...|

These are the first few bytes of IBMBIO.COM. Its birthday is about six weeks from now, but I am going to post about its internals next week.

Let us finally look at the complete commented disassembly of the boot sector. It compiles with NASM, but will emit a few bytes differently because of variations in the assembly encoding – but variations in size have been compensated wit NOPs. It is actually quite easy to read.

;----------------------------------------------------------------------------- ; DOS 1.0 Boot Sector (disk image MD5 73c919cecadf002a7124b7e8bfe3b5ba) ; http://www.pagetable.com/ ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â org 0x7C00 Â Â Â Â Â Â Â Â jmpÂ Â short start ;----------------------------------------------------------------------------- os_numsectors Â dw 20 Â Â Â Â Â Â Â Â Â ; how many sectors to read os_offset Â Â Â dw 0Â Â Â Â Â Â Â Â Â Â ; segment to load code into os_segmentÂ Â Â dw 0x60 Â Â Â Â Â Â Â Â ; offset to load code into Â Â Â Â Â Â Â Â db " 7-May-81",0Â Â Â Â ; timestamp Â Â Â Â Â Â Â Â times 31 db 0 Â Â Â Â Â ; padding ;----------------------------------------------------------------------------- start Â Â Â Â Â cli Â Â Â Â Â Â Â Â movÂ Â ax, cs Â Â Â Â Â Â Â Â movÂ Â ds, axÂ Â Â Â Â ; DS := CS Â Â Â Â Â Â Â Â movÂ Â dx, 0 Â Â Â Â Â Â Â Â movÂ Â ss, dxÂ Â Â Â Â ; SS := 0000 Â Â Â Â Â Â Â Â movÂ Â sp, 0x7C00Â Â Â ; stack below code Â Â Â Â Â Â Â Â sti Â Â Â Â Â Â Â Â movÂ Â ax, [os_segment] Â Â Â Â Â Â Â Â movÂ Â ds, ax Â Â Â Â Â Â Â Â movÂ Â es, axÂ Â Â Â Â ; ES := DS := where to load DOS Â Â Â Â Â Â Â Â movÂ Â dx, 0 Â Â Â Â Â Â Â Â movÂ Â ax, dx Â Â Â Â Â Â Â Â intÂ Â 0x13Â Â Â Â Â Â ; reset drive 0 Â Â Â Â Â Â Â Â jcÂ Â Â disk_error again Â Â Â Â Â callÂ Â check_sys_files ; check for presence of IBMDOS/IBMBIO Â Â Â Â Â Â Â Â jcÂ Â Â again Â Â Â Â Â ; not found, try another disk Â Â Â Â Â Â Â Â movÂ Â cx, [cs:os_numsectors] Â Â Â Â Â Â Â Â pushÂ Â cxÂ Â Â Â Â Â Â ; remaining sectors Â Â Â Â Â Â Â Â movÂ Â bx, 0 Â Â Â Â Â Â Â Â xorÂ Â dx, dxÂ Â Â Â Â ; drive 0, head 0 Â Â Â Â Â Â Â Â movÂ Â cx, 8 Â Â Â Â Â ; track 0, sector 8 Â Â Â Â Â Â Â Â movÂ Â si, 1 Â Â Â Â Â ; read 1 sector in first found Â Â Â Â Â Â Â Â pushÂ Â si Â Â Â Â Â Â Â Â movÂ Â al, 1 Â Â Â Â Â ; 1 sector read_loop Â Â Â movÂ Â ah, 2 Â Â Â Â Â Â Â Â intÂ Â 0x13Â Â Â Â Â Â ; read sector(s) Â Â Â Â Â Â Â Â jcÂ Â Â disk_error Â Â Â Â Â Â Â Â popÂ Â siÂ Â Â Â Â Â Â ; sectors read Â Â Â Â Â Â Â Â popÂ Â axÂ Â Â Â Â Â Â ; remaining sectors Â Â Â Â Â Â Â Â callÂ Â add_si_sectorsÂ ; bx += si*512 Â Â Â Â Â Â Â Â subÂ Â ax, siÂ Â Â Â Â ; remaining -= read Â Â Â Â Â Â Â Â jzÂ Â Â doneÂ Â Â Â Â Â Â Â ; none left Â Â Â Â Â Â Â Â incÂ Â chÂ Â Â Â Â Â Â ; next track Â Â Â Â Â Â Â Â movÂ Â cl, 1 Â Â Â Â Â ; start at sector 1 Â Â Â Â Â Â Â Â movÂ Â si, 8 Â Â Â Â Â ; read up to 8 sectors Â Â Â Â Â Â Â Â cmpÂ Â ax, siÂ Â Â Â Â ; how many are left to read? Â Â Â Â Â Â Â Â jaeÂ Â at_least_8_left ; at least 8 Â Â Â Â Â Â Â Â movÂ Â si, axÂ Â Â Â Â ; only read remaining amount Â Â Â Â Â Â Â Â jmpÂ Â short skip at_least_8_left xchgÂ Â ax, siÂ Â Â Â Â ; read 8 sectors this time skipÂ Â Â Â Â Â pushÂ Â siÂ Â Â Â Â Â Â ; number of remaining sectors Â Â Â Â Â Â Â Â pushÂ Â axÂ Â Â Â Â Â Â ; number of sectors to read this time Â Â Â Â Â Â Â Â jmpÂ Â read_loop Â Â Â ; next read doneÂ Â Â Â Â Â jmpÂ Â far [cs:os_offset]; jump to IBMBIO.COM disk_errorÂ Â Â movÂ Â si, FAILUREÂ Â ; string to print Â Â Â Â Â Â Â Â movÂ Â ax, rom_basic Â ; put return address of "int 18" code Â Â Â Â Â Â Â Â pushÂ Â axÂ Â Â Â Â Â Â ; onto stack ;----------------------------------------------------------------------------- ; print zero-terminated string pointed to by DS:SI ;----------------------------------------------------------------------------- print Â Â Â Â Â xorÂ Â bh, bhÂ Â Â Â Â ; XXX unnecessary print_loopÂ Â Â lodsb Â Â Â Â Â Â Â Â andÂ Â al, 0x7FÂ Â Â Â ; clear bit 7 XXX why is it set? Â Â Â Â Â Â Â Â jzÂ Â Â ret0Â Â Â Â Â Â ; zero-termination Â Â Â Â Â Â Â Â pushÂ Â si Â Â Â Â Â Â Â Â movÂ Â ah, 0x0E Â Â Â Â Â Â Â Â movÂ Â bx, 7 Â Â Â Â Â ; light grey, text page 0 Â Â Â Â Â Â Â Â intÂ Â 0x10Â Â Â Â Â Â ; write character Â Â Â Â Â Â Â Â popÂ Â si Â Â Â Â Â Â Â Â jmpÂ Â print_loop ret0Â Â Â Â Â Â retn ;----------------------------------------------------------------------------- ; test for IBMBIO.COM and IBMDOS.COM in the first two directory entries ;----------------------------------------------------------------------------- check_sys_files movÂ Â bx, 0 Â Â Â Â Â ; read to address 0 in the DOS segment Â Â Â Â Â Â Â Â movÂ Â cx, 4 Â Â Â Â Â ; track 0, sector 4 Â Â Â Â Â Â Â Â movÂ Â ax, 0x0201 Â Â Â Â Â Â Â Â intÂ Â 0x13Â Â Â Â Â Â ; read 1 sector Â Â Â Â Â Â Â Â pushÂ Â ds Â Â Â Â Â Â Â Â jcÂ Â Â non_system_disk ; error case Â Â Â Â Â Â Â Â movÂ Â ax, cs Â Â Â Â Â Â Â Â movÂ Â ds, axÂ Â Â Â Â ; DS := CS Â Â Â Â Â Â Â Â movÂ Â di, 0 Â Â Â Â Â Â Â Â movÂ Â cx, 11Â Â Â Â Â ; convert 11 bytes of first two to_lowerÂ Â Â Â orÂ Â Â byte [es:di], 0x20; directory entries to lowercase Â Â Â Â Â Â Â Â orÂ Â Â byte [es:di+0x20], 0x20 Â Â Â Â Â Â Â Â nop Â Â Â Â Â ; XXX original assembler wasted a byte Â Â Â Â Â Â Â Â incÂ Â di Â Â Â Â Â Â Â Â loopÂ Â to_lower Â Â Â Â Â Â Â Â movÂ Â di, 0 Â Â Â Â Â ; first entry Â Â Â Â Â Â Â Â movÂ Â si, IBMBIO_COM Â Â Â Â Â Â Â Â movÂ Â cx, 11 Â Â Â Â Â Â Â Â cld Â Â Â Â Â Â Â Â rep cmpsb Â Â Â Â Â Â Â ; compare first entry with IBMBIO.COM Â Â Â Â Â Â Â Â jnzÂ Â non_system_disk Â Â Â Â Â Â Â Â movÂ Â di, 0x20Â Â Â Â ; second entry Â Â Â Â Â Â Â Â movÂ Â si, IBMDOS_COM Â Â Â Â Â Â Â Â movÂ Â cx, 11 Â Â Â Â Â Â Â Â rep cmpsb Â Â Â Â Â Â Â ; compare second entry with IBMDOS.COM Â Â Â Â Â Â Â Â jnzÂ Â non_system_disk Â Â Â Â Â Â Â Â popÂ Â ds Â Â Â Â Â Â Â Â retnÂ Â Â Â Â Â Â Â Â Â ; return with carry clear non_system_disk movÂ Â si, NON_SYSTEM_DISK Â Â Â Â Â Â Â Â callÂ Â print Â Â Â Â Â Â Â Â movÂ Â ah, 0 Â Â Â Â Â Â Â Â intÂ Â 0x16Â Â Â Â Â Â ; wait for key Â Â Â Â Â Â Â Â popÂ Â ds Â Â Â Â Â Â Â Â stc Â Â Â Â Â Â Â Â retnÂ Â Â Â Â Â Â Â Â Â ; return with carry set ;----------------------------------------------------------------------------- NON_SYSTEM_DISK db 13,10 Â Â Â Â Â Â Â Â db "Non-System disk or disk erro",'r'+0x80 Â Â Â Â Â Â Â Â db 13,10 Â Â Â Â Â Â Â Â db "Replace and strike any key when read",'y'+0x80 Â Â Â Â Â Â Â Â dbÂ 13,10,0 ;----------------------------------------------------------------------------- rom_basic Â Â Â intÂ Â 0x18Â Â Â Â Â Â Â Â ; ROM BASIC ;----------------------------------------------------------------------------- FAILURE Â Â Â Â db 13,10 Â Â Â Â Â Â Â Â db "Disk Boot failur",'e'+0x80 Â Â Â Â Â Â Â Â db 13,10,0 ;----------------------------------------------------------------------------- add_si_sectorsÂ pushÂ Â axÂ Â Â Â Â Â Â ; bx += si*512 Â Â Â Â Â Â Â Â pushÂ Â dx Â Â Â Â Â Â Â Â movÂ Â ax, si Â Â Â Â Â Â Â Â movÂ Â di, 512 Â Â Â Â Â Â Â Â mulÂ Â di Â Â Â Â Â Â Â Â addÂ Â bx, ax Â Â Â Â Â Â Â Â popÂ Â dx Â Â Â Â Â Â Â Â popÂ Â ax Â Â Â Â Â Â Â Â retn ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â db "Robert O'Rear " IBMBIO_COMÂ Â Â db "ibmbioÂ com" Â Â Â Â Â Â Â Â db 0xB0 Â Â Â Â Â Â Â Â ; XXX unused IBMDOS_COMÂ Â Â db "ibmdosÂ com" Â Â Â Â Â Â Â Â db 0xB0, 0xC9 Â Â Â Â Â ; XXX unused ;----------------------------------------------------------------------------- Â Â Â Â Â Â Â Â times 512-($-$$) db 0 ;-----------------------------------------------------------------------------

The 6502 in “The Terminator”

Michael Steil — Tue, 05 May 2009 08:01:54 +0000

In the first Terminator movie, the audience sees the world from the T-800′s view several times. It is well-known that in two instances, there is 6502 assembly code on the T-800′s HUD, and many sites have analyzed the contents: It’s Apple-II code taken from Nibble Magazine. Here are HD versions of the shots, thanks to Dominik Wagner:

This is the first assembly snippet:

This is the second assembly snippet:

There are some assembly equates:

On the left, these are the assembled opcodes of the second assembly listing, reaching from “LDY#10″ to “SEC”. On the right, there is output of a run of the checksum application Key Perfect on a file names “OVLY.OBJ”, which prints a 16 bit checksum for every 0×50 bytes:

The History of OS Migration

Michael Steil — Tue, 28 Apr 2009 08:01:17 +0000

Operating system vendors face this problem once or twice a decade: They need to migrate their user base from their old operating system to their very different new one, or they need to switch from one CPU architecture to another one, and they want to enable users to run old applications unmodified, and help developers port their applications to the new OS. Let us look at how this has been done in the last 3 decades, looking at DOS/Windows, Macintosh, Amiga and Palm.

CP/M to PC-DOS/MS-DOS

CP/M was an 8 bit operating system by Digital Research that ran on all kinds of Intel 8080-based systems. Seattle Computer Products “86-DOS”, which later became MS-DOS (called “PC-DOS” on IBM machines), was a clone of CP/M, but for the Intel 8086, much like DR’s own CP/M-86 (which later became DR-DOS).

While not binary compatible, the Intel 8086 was “assembly source compatible” with the 8080, which meant that it was easily possible to convert 8 bit 8080/Z80 assembly source into 8086 assembly, since the two architectures were very similar (backward-compatible memory model; one register set could be mapped onto the other) and only the instruction encoding was different.

Since MS-DOS implemented the same ABI and memory map, it was source-code compatible with CP/M. On a CP/M system, which could access a total of 64 KB of memory, the region from 0×0000 to 0×0100 (Zero Page) was reserved to the operating system and contained, among other things, the command line arguments. The running application was located from 0×0100 up, and the operating system was at the top of memory, with the application stack growing down from just below the start of the OS. The memory model of the 8086 partitions memory into (overlapping) chunks of contiguous 64 KB, so one of these segments is basically a virtual 8080 machine. MS-DOS “.COM” files are executables below 64 KB that are loaded at address 0×0100 of such a segment. 0×0000-0×0100 is called the Program Segment Prefix and is very similar to the CP/M zero page, the stack grows down from the end of code segment, and the operating system resides in a different segment.

Because of the high compatibility of the CPUs and the ABIs, a port of a program from CP/M to DOS was pretty painless. Such a direct port could only support up to 64 KB of memory (like WordStar 3.0 from 1982), but it was also possible to maintain a single source base for both CP/M and MS-DOS just by using a few macros and two different assemblers.

DOS 2.0 then introduced more powerful APIs (file handles instead of FCBs, subdirectories, relocatable .EXE files), obsoleting most of the CP/M API – but DOS kept CP/M compatibility until the last version.

CP/M to PC-DOS/MS-DOS

Change New CPU
New OS codebase

Running new applications Native

Running old applications Not supported

Running old drivers Not supported

Porting applications High level of source/ABI compatibility

DOS to Windows

Microsoft Windows was first architected as a graphical shell on top of MS-DOS: All device and filesystem access was done by making DOS API calls, so all MS-DOS drivers ran natively, and Windows could use them. DOS applications could still be used by just exiting Windows.

Windows/386 2.1 changed this model, as it was a real operating system kernel that ran a number of “virtual 8086 mode” (V86) virtual machines side by side: One for the MS-DOS operating system and one per Windows application. The DOS VM was used by Windows to call out to device drivers and the filesystem, so it was basically a driver compatibility environment running inside a VM. The user could start any number of additional DOS VMs to run DOS applications, and each of these contained a copy of DOS. Windows hooked memory accesses to screen RAM as well as some system calls to route them to the Windows graphics driver or through the “root” DOS VM.

Windows 3.x started using Windows-native drivers that replaced calls into the DOS VM, and had the DOS VM call up to Windows for certain device accesses. The standard Windows 95 installation didn’t use the DOS VM for drivers or the filesystem at all, but could do so if necessary.

DOS was not only a compatibilty environment for old drivers and applications, it was also the command line of Windows, so when Windows 95 introduced long file names, it trapped DOS API calls to provide a new interface for this functionality to command line tools.

MS-DOS to Windows

Change New OS

Running new applications Native

Running old applications Virtual machine with old OS

Running old drivers Virtual machine with old OS

Porting applications No migration path

DOS to Windows NT

Windows NT was never based on DOS, but still allowed running MS-DOS applications since its first version, NT 3.1. Like non-NT Windows, it runs DOS applications in V86 mode. But instead of running a copy of MS-DOS, using its logic and trapping its device accesses, NT just runs the application in V86 mode and traps all system calls and I/O accesses and maps them to NT API calls. It is not a virtual machine: V86 mode is merely used to provide the memory model necessary to support DOS applications.

One common misconception is that the Windows NT command line is a “DOS box”: The command line interpreter and its support tools are native NT applications, and a Virtual DOS Machine (NTVDM) is not started until a real DOS program is launched from the command line.

DOS to Windows NT

Change New OS

Running new applications Native

Running old applications API reimplementation

Running old drivers Not supported

Porting applications No migration path

Windows 3.1 (Win16) to Windows 95 (Win32)

Since the release of Windows NT 3.1 in 1993, it was clear that it would replace classic Windows eventually, but although it had the same look-and-feel and good Win16 and decent DOS compatibility, every current version of NT typically required quite high-end hardware. The migration from Windows to Windows NT was done by slowly making Windows more like Windows NT, and when they were similar enough, and even low-end computers were powerful enough to run NT well, switching the users to the new codebase.

The big step to make Windows more like Windows NT was supporting NT’s 32 bit Win32 API: The first step was the free “Win32S” update to Windows 3.1, which provided a subset (thus the “S”) of the Win32 API on classic Windows. Win32S extended the Windows kernel to create a 32 bit address space for all 32 bit applications (NT had a separtate address space for each application). It also provided ported versions of some new NT libraries (e.g. RICHED32.DLL), as well as 32 bit DLLs that accepted the low-level Win32 API calls (“GDI” and “USER”) and forwarded them to the Win16 system (“thunking”).

Windows 95 included this functionality by default, ran 32 bit applications in separate address spaces, supported more of the Win32 API and included several 32 bit core applications (like Explorer), but a good chunk of the core system was still 16 bit. With Windows 95, most developers switched to writing 32 bit applications, making them instantly available as native applications on Windows NT.

Windows 3.1 (Win16) to Windows 95 (Win32)

Change New CPU mode/bitness

Running new applications Thunking

Running old applications Native

Running old drivers Native

Porting applications High level of source compatibility

Windows 9X to Windows NT

The second step in the migration from 16 bit Windows to Windows NT was the switch from Windows ME to the NT-based Windows XP in 2001. Windows NT (2000/XP/…) was a fully 32 bit operating system with the Win32 API, but it also allowed running 16 bit Windows applications by forwarding their Win16 API calls to the Win32 libraries (thunking).

The driver model of Windows NT 3.1/3.5/4.0 (“Windows NT Driver Model”) and classic Windows (“VxD”) was different, so Windows 98 (successor of Windows 95) and Windows 2000 (successor to Windows NT 4.0) both supported the new “Windows Driver Model”. A single driver could now work on both operating systems, but each OS continued to support their original driver mode.

When Microsoft switched the home users to the NT codebase, most current applications, games and drivers worked on Windows XP as well. It was only the system tools that had to be rewritten.

Windows 9X to Windows NT

Change New OS

Running new applications Native

Running old applications Win16: Thunking
Win32: Native

Running old drivers Providing the same API for the old OS

Porting applications High level of source compatibility
Providing the same API for the old OS

Windows i386 (Win32) to Windows x64/x86_64 (Win64)

The switch from 32 bit Windows to 64 bit Windows is currently in progress: Windows XP was the first version to be available for AMD64/Intel64, and both Windows Vista and Windows 7 are available as both 32 bit and 64 bit editions. On the 64 bit edition, the kernel is 64 bit native, and so are all libraries and most applications. The 32 bit API is still supported using the “WOW64″ (Windows-on-Windows 64-bit) subsystem: A 32 bit application links against all 32 bit libraries, but the low-level API calls it wants to make get translated by the WOW64 DLL into their 64 bit counterparts.

Since drivers run in the same address space as the kernel, 32 bit drivers could not be easily supported on 64 bit Windows, and thus are not. Support for DOS and Win16 applications was dropped on 64 bit Windows.

Windows i386 (Win32) to Windows x64/x86_64 (Win64)

Change New CPU mode/bitness

Running new applications Native

Running old applications Thunking

Running old drivers Not supported

Porting applications High level of source compatibility

Macintosh on 68K to Macintosh on PowerPC

Apple switched their computers from using Motorola 68K processors to Motorola/IBM PowerPC processors between 1994 and 1996. Since the Macintosh operating system, called System 7 at that time, was mostly written in 68K assembly, it could not be easily converted into a PowerPC operating system. Instead, most of the system was run in emulation: The new “nanokernel” handled and dispatched interrupts and did some basic memory management to abstract away the PowerPC, and the tightly integrated 68K emulator ran the old operating system code, which was modified to hook into the nanokernel for interrupts and memory management. So System 7.1.2 for PowerPC was basically a paravirtualized operating system running inside emulation on top of a very thin hypervisor.

The first version of Mac OS for PowerPC ran most of the operating system inside 68K emulation, even drivers, but some performance-sensitive code was native. The executable loader detected binaries with PowerPC code in them and could run them natively inside the same context. Most communication to the OS APIs went back through the emulator. Later versions of Mac OS replaced more and more of the 68K code with PowerPC code.

Macintosh on 68K to Macintosh on PowerPC

Change New CPU

Running new applications Thunking

Running old applications Paravirtualized old OS in emulator

Running old drivers Paravirtualized old OS in emulator

Porting applications High level of source compatibility

Classic Mac OS to Mac OS X

Just like Microsoft switched from Windows to Windows NT, Apple switched from Classic Mac OS to Mac OS X. While Classic Mac OS was a hacky OS with cooperative multitasking and without memory protection that still ran some of the OS code in 68K emulation, Mac OS X was based on NEXTSTEP, a modern UNIX-like operating system with a completely different API.

When Apple decided to migrate towards a new operating system, they ported the system libraries of Classic Mac OS (“Toolbox”) to Mac OS X, omitting the calls that could not be supported on the modern OS (and replacing them with alternatives), and called the new API “Carbon”. They provided the same API for (Classic) Mac OS 8.1 in 1998, so developers could already update their applications for OS X, while maintaining compatibility with Classic Mac OS. When Mac OS X was introduced in 2001, binaries of “carbonized” applications would then run unmodified on both operating systems. This is similar to the “make Windows more like Windows NT” approach by Microsoft.

But since not all applications were expected to exist as carbonized versions with the introduction of OS X, the new operating system also contained a virtual machine called “Classic” or “Blue Box” in which the unmodified Mac OS 9 was run together with any number of legacy applications. Hooks were installed inside the VM to route network and filesystem requests to the host OS, and window manager integration allowed the two desktop environments to blend almost seamlessly together.

Classic Mac OS to Mac OS X

Change New OS

Running new applications Native

Running old applications Classic: Virtual machine with old OS
Carbon: Intermediate API for both systems

Running old drivers Virtual machine with old OS

Porting applications Intermediate API for both systems

Mac OS X on PowerPC to Mac OS X on Intel

In 2005, Apple announced that they would switch CPUs a second time, this time away from the PowerPC towards the Intel i386 architecture. Being a modern operating system mostly written in C and Objective C, it could be easily ported to i386 – actually, Apple claims that they have always maintained i386 versions of the whole operating systems since the first release.

In order to run legacy applications that had not yet been ported to i386, Apple included the emulator “Rosetta” with the operating system; but this time, it was not tighltly integrated into the kernel as with the 68K to PowerPC switch, but the kernel only added support to run an external recompiler with the application as a parameter whenever a PowerPC application was launched. Rosetta translated all application code as well as the libraries it linked against, and interfaced to the native OS kernel.

Mac OS X on PowerPC to Mac OS X on Intel

Change New CPU

Running new applications Native

Running old applications User mode emulator

Running old drivers Not supported

Porting applications High level of source compatibility

Mac OS X (32 bit) to Mac OS X (64 bit)

The next switch for Apple was the migration from 32 bit Intel (i386) to 64 bit (x86_64) in Mac OS X 10.4 in 2006. Although the whole operating system could have been ported to 64, as it was done with Windows, Apple decided to take an approach which is closer to the Windows 95 one: The kernel stayed 32 bit, but got support for 64 bit user applications. All applications and drivers on the system were still 32 bit, but some system libraries were also available as ported 64 bit versions. A 64 bit application thus linked against 64 bit libraries, and made 64 bit syscalls that were converted to 32 bit calls inside the kernel.

Mac OS X 10.5 then provided all libraries in 64 bit versions, but the kernel remained 32 bit. OS X 10.6 will be the first version with a 64 bit kernel, requiring new 64 bit drivers.

Mac OS X (32 bit) to Mac OS X (64 bit)

Change New CPU mode/bitness

Running new applications Thunking

Running old applications Native

Running old drivers Native

Porting applications Carbon: Not supported
Cocoa: High level of source compatibility

AmigaOS on 68K to AmigaOS on PowerPC

The Amiga platform was the same OS on the same 68K CPU architecture in the Commodore days between 1985 and 1994, but third-party manufacturers offered PowerPC CPU upgrade boards since 1997. The closed source operating system could not be ported to PowerPC by these third parties, so AmigaOS continued to run on the 68K CPU, and an extension in the binary loader detected PowerPC code and handed it off to the other CPU. All system calls then went though a thunking library back to the 68K.

AmigaOS 4 (2006) is a native port of AmigaOS to the PowerPC, which was a major effort, since a lot of operating system code had to be converted from BCPL to C first. 68K application support is done by emulating the binary code and interfacing it to the native API.

AmigaOS on 68K to AmigaOS on PowerPC (3.x)

Change New CPU

Running new applications Thunking (new CPU)

Running old applications Native (old CPU)

Running old drivers Native

Porting applications High level of source compatibility

AmigaOS on 68K to AmigaOS on PowerPC (4.0)

Change New CPU

Running new applications Native

Running old applications User mode emulator

Running old drivers Not supported

Porting applications High level of source compatibility

Palm OS on 68K to Palm OS on ARM

Palm switched from 68K processors to the ARM architecture with Palm OS 5 in 2002. The operating system was ported to ARM, and a the “Palm Application Compatibility Environment” (“PACE”) 68K emulator was included to run old applications. But Palm discouraged developers from switching to ARM code and did not even provide an environment in the OS to run native ARM applications. They claimed that most applications on Palm OS did most of their work in the native operating system code anyway, so they would not see a significant speedup.

But for applications that were heavily CPU bound and contained compression or crypto code, Palm provided a way to run small chunks of native ARM code inside a 68K application. These “ARMlets” (later called “PNOlets” for “Palm Native Object”) could be called from 68K code and provided a minimal interface with a single integer for input and output, so the developer had to write the code to pass extra parameters in structs and care about endianness and alignment. ARM code could neither call back to 68K code, nor could it call operating systems API directly.

Sticking with 68K code for most applications practically meant having a virtual architecture for user mode programs, not unlike Java or .NET. The switch to ARM was mostly unnoticed by developers, and this approach could have allowed Palm to switch architectures again in the future with little effort.

Palm OS on 68K to Palm OS on ARM

Change New CPU

Running new applications Not supported (PNOlet for subroutines)

Running old applications User mode emulation

Running old drivers Not supported

Porting applications Not supported (PNOlet for subroutines)

Summary

Let us summarize the OS and CPU switches and how the different vendors approached their respective problems.

Bitness

Switching to a new CPU mode is the easiest change in an operating system, since old applications code can run natively, and API calls can be translated. An operating system can either stay in the old bitness and convert calls from new applications for the old system, or move up to the new bitness and convert calls fom old applications. There are also two ways where to hook the calls: An operating systems could hook into high-level API calls like creating a GUI window, but this is hard to do, since the high-level API is typically very wide, and it is very hard to get a converter for so many calls correct and compatible. Alternatively, the OS can convert low-level system calls. With this solution, the interface is quite narrow. But since all old applications link against the old libraries and new applications against the new libraries, equivalent libraries will end up twice in memory if the user runs old and new applications concurrently.

New CPU mode/bitness

OS Old mode New mode Thunking direction Thunking level

Windows 16 bit 32 bit new to old library

Windows NT 32 bit 64 bit old to new kernel

Mac OS X 32 bit 64 bit new to old kernel

For the 16 to 32 bit switch in Windows, the operating system stayed 16 bit and converted 32 bit calls into 16 bit calls at the API level. When Windows NT switched from 32 bit to 64 bit, the whole OS became 64 bit, and low-level kernel calls were converted for old applications. The same switch was done differently on Mac OS X: The OS stayed 32 bit, and 64 bit calls were translated at the kernel level.

The solutions of Windows NT and Mac OS X are quite similar, as they both run all 32 bit code with 32 bit libraries, and all 64 bit code with 64 bit libraries, and it is just the kernel that is different. For Windows, this has the advantage of having access to more than 4 GB in kernel mode, as well as some speedup from the new registers in x86_64 long mode, and for Mac OS X, it has the advantage of running old 32 bit drivers unmodified. (In a second step, Mac OS X later switched to a 64 bit kernel.)

CPU

It is harder to switch to a new CPU, because the new CPU just cannot run the old application code any more, and some operating systems cannot be easily adapted to a new CPU.

New CPU

OS Old CPU New CPU Running old apps Thunking level

CP/M, DOS 8080/Z80 8086 Developer has to recompile -

Macintosh 68K PowerPC Run OS and app in emulation -

Mac OS X PowerPC i386 User mode emulation kernel

Amiga 68K PowerPC Dual-CPU thunking library

Palm 68K ARM User mode emulation library

Mac OS X Intel and Palm OS ARM were written in a platform independent enough way so that they could be ported to the new architecture. They both included recompilers that ran the old code. This is the easy way. Amiga OS could not be ported, because the source code was not available. So systems had both CPUs, the original operating system code ran on the old CPU, and new applications ran on the new CPU, switching back to the old CPU for system calls.

For Classic Macintosh (68K to PowerPC), the OS source code was available, but could not be ported easily, so it was done similarly to the Amiga, although with a single CPU: Most of the old operating system ran inside emulation, and new applications ran natively, calling back into the emulator for system calls.

DOS was a reimplementation of the old OS by a different company that did not support running old binary code. Instead, it made the developer recompile their code.

OS

Switching to a new operating system, but keeping your users and developers is the hardest of all switches.

New OS

Old OS New OS Running old apps

CP/M DOS Compatible API

DOS Windows Virtual machine with old OS

DOS Windows NT API emulation

Windows 9X Windows NT Compatible API

Mac OS Mac OS X Classic: Virtual machine with old OS
Carbon: Compatible API

The approach to take depends on the plans with the API of the old operating system. If the API is good enough to be worth supporting in the new OS, the new OS should just have the same API. This has been the case for the CP/M to DOS and the Windows 9X to Windows NT migrations. In a way, this was also true for Classic Mac OS to Mac OS X, but in this case, Carbon was not the main API of the new OS, but one of three APIs (Carbon, Cocoa, Java; while everything but Cocoa is pretty much deprectated today).

If the old API is not worth maintaining on the new OS, but it is important that old applications run very well, it makes sense to run the old operating system in a virtual machine, together with its applications. This was done by Windows to run DOS applications as well as Mac OS X to run old Mac OS applications.

If the OS interface of the old operating system is relatively small and easy, or perfect accuracy is not necessary, the best solution might be API emulation, i.e. hooking the system calls of the old application and mapping them into the new operating system. This was done by Windows NT to run DOS applications, and was only moderately compatible.

Conclusion

It is interesting how different the solutions for all these OS migrations were: There have been hardly two instances that followed the same approach. The reason for it might be that the situations were all subtly different, and a lot of time was spend to work out the perfect solution for the specific problem.

But there is a trend that can be seen: As systems are getting more modern, solutions tend to get less hacky, and migrations tend to happen in many small steps instead of few big steps. Modern operating systems like Windows NT and Mac OS X can be ported to new architectures quite easily, emulators help running old applications, and thunking can be used to interface with the native syscall interface. Because of the abstractions in a system, an operating system can be ported to a new architecture or a new CPU bitness in steps, with some parts in the new system, and others still in the old system. These abstractions also allow developers swapping out complete subsystems or rearchitecting parts of the operating system without much user impact. It is getting more and more convenient for OS developers – but unfortunately, it’s also getting less exciting.

Links

1 2 3 4

The Easiest Way to Reset an i386/x86_64 System

Michael Steil — Thu, 23 Apr 2009 08:01:46 +0000

Try this in kernel mode:

uint64_t null_idtr = 0; asm("xor %%eax, %%eax; lidt %0; int3" :: "m" (null_idtr));

This can be quite helpful when doing operating system development on an i386/x86_64 system. You can use this for the regular restart case or when a kernel panic is supposed to restart immediately and you cannot make any assumptions on what is still working in the system.

You can also use this for debugging very low-level code if you don’t have a serial port or even an LED to report the most basic information: First make sure your code is reached by putting the reset code there. Then remove it again and put this code in:

if (condition) reset(); else for(;;);

The system will either hang or reset, depending on the condition.

LodgeNet Reverse Engineering

Michael Steil — Tue, 21 Apr 2009 08:01:19 +0000

Many hotels (at least in the USA) equip their room TVs with a “LodgeNet” entertainment system. The TV will show regular free television channels, but also have an interactive channel controlled by the remote that features video on demand and video games.

The setup consists of a regular TV, a set-top-box and a Nintendo 64 controller.

If you play with the system a bit, it’s be quite easy to reverse engineer the architecture:

All TVs in the hotel are connected to the same internal TV cable. The cable carries one channel per room, and the TV is locked to that channel. This gives the TV a point-to-point channel to a multiplexer in the server room.

This analog multiplexer has the free-to-air channels as inputs as well as “channel 00″, which is the entertainment system. Another multiplexer switches channel 00 between the “welcome” screen and one of the servers. It is the servers that provide the image as soon as the user browses through the menu or selects video on demand or a game.

The remote is custom. The volume buttons will control the TV directly, and all other buttons are sent to the set-top-box.

The Nintendo 64 controller has extra buttons and a custom (RJ) connector.

The set-top-box has an IR receiver at the front, and sits on the cable between the multiplexer in the server room and the TV (“CABLE IN”, “TV”), in order to be able to send the commands from the remote. “DATA” connects to the Nintendo 64 controller. It is unknown what “IR” and “MTI” would be for.

If you switch channels, the set-top-box tunnels the command to the multiplexer. The on-screen-display is generated by the multiplexer. Channel “00″ is the entertainment system, which by default is connected to the “welcome” channel, a series of static screens that are hotel-global.

The volume button on the remote targets the TV directly. You can tell that the on-screen-display is generated by the TV and not the multiplexer.

Switching channels with the buttons on the TV itself does not succeed. Otherwise you could watch your neighbor’s channel.

If you press the MENU key, the second-level multiplexer will switch you away from the global “welcome” channel and connect you to one of the servers. Since most hotel guests do not use the entertainment system most of the time, the number of servers is only a fraction of the number of rooms. While the multiplexer is finding a free server, it displays a “Please Wait” picture for about a second.

The user is connected to the server that is now dedicated to his TV. Key presses on the remote (except volume and channel) will be sent to the server.

A very obvious attack on the system would be to connect the cable to a TV receiver that allows switching channels.

Does anyone know more about this system? How are the games done – emulation or dedicated N64 hardware? Are the movies streamed from disk? Is there a dedicated storage server? How is the system updated with new movies? What operating system are the servers running?

Bringup History of Mac OS X

Michael Steil — Tue, 14 Apr 2009 07:09:46 +0000

The heritage of different operating systems has been discussed many times. Mac OS X includes code from Mach and BSD, AmigaOS is based on TRIPOS, MS-DOS is a CP/M-86 clone and Windows NT is modeled after VMS. But what machines and operating systems were used for cross-compilation and bringup of these systems? In order to find this out about Mac OS X, I talked to a few people working at NEXT and Apple, and people that worked on Mach and BSD.

Currently, Apple only ships Intel-based machines. Mac OS X for Intel was released in 2006. The Intel version had been “leading a secret double life” since 2000, i.e. Mac OS X existed for Intel all the time, but was not released. In that time, Mac OS X was never self-hosting; instead, it was cross-compiled on PowerPC Macs. The first released version of Intel Mac OS X was a version of 10.4 for the Pentium 4 based “Developer Transition Kit” in 2005.

The first version of what would later be Mac OS X was “Rhapsody DR1″ released in 1997. It ran on PowerPC 604 Macintoshes (the 603 was not supported because it lacked a hardware pagetable walker) and was cross-compiled from OpenStep 4.2 running on Intel Pentium II CPUs. Rhapsody, which was basically OpenStep 5.0, also continued to run on Intel, but as mentioned before, Intel became a second-class architecture. Actually, somewhere between Rhapsody and OS X 10.0, there was a time when the GUI was not built for Intel.

Nextstep 3.1 from 1993 was the first version of Nextstep/OpenStep to support Intel CPUs (and also PA-RISC and SPARC) next to the existing Motorola 68K support. Bringup was done on a 25 MHz 68040 NeXTstation running NextStep 2.1, and the target CPU was a 486DX50.

At NEXT, the systems used for bringup of the original 68030 NeXT Computer were Sun machines running SunOS 3.0, a BSD-derivative. The NEXT engineers ported the Mach 2.0 kernel from VAX to M68K. So this port was not only cross-architecture, but cross-OS.

Mach 1.0 was first written for VAX and was brought up on VAX-based 4.2BSD machines at Carnegie-Mellon University. Coincidently, 4.2BSD evolved into SunOS 1.0 (and 4.2BSD/VAX machines were used for its bringup), and the BSD codebase also ended up in Nextstep.

Now that we arrived at BSD, we could go back the history of UNIX, which I am not going to do at this point.

So, to summarize, BSD/VAX was used to develop Mach/VAX, and SunOS/M68K was used to port Mach to 68K, Nextstep/68K was used to port itself to i386, and the i386 version to port itself to PowerPC, and then the PowerPC version to maintain the i386 version. Today, Mac OS X for Intel is self-hosting again.

Now I would love to see is the same kind of bringup history for Linux (Minix, …), AmigaOS (SunOS 1.x, …), BeOS, Copland (Mac OS?), Windows NT (VMS? DOS?), OS/2 (DOS? AIX?), MS-DOS (CP/M?), and of course the common bringup ancestors of so many systems, BSD and UNIX. It would also be interesting to extend this information with the respective compilers used. So if you can contribute something in the comments here or on your blogs, that would be great.

cbmbasic 1.0 with Plugins

Michael Steil — Wed, 08 Apr 2009 06:07:19 +0000

I moved cbmbasic development to SourceForge and released version 1.0, which has the following added features:

RDTIM/SETTIM support (George Talbot)

LOAD”$” on Win32 (Lorenzo)

RND() is now random (Wolfram Sang)

The C code now hooks into the cbmbasic plugin infrastructure. This lets developers add additional statements, functions etc. Right now, you can turn this on with “SYS 1″ (turn off with “SYS 0″), and use the new statements LOCATE y,x (set cursor position), SYSTEM string (run command line command) and the extended WAIT port,mask, which implements the Bill Gates easter egg.

Amiga/Lorraine Mugs

Michael Steil — Tue, 07 Apr 2009 02:56:26 +0000

Every touristy place has them: Souvenirs with given names on them. If you have an uncommon name, or a friend with an uncommon name, you might look through the whole collection – and notice that they have generic ones like “#1 FRIEND” (i case you really don’t find your friend’s name), and, sometimes, generic ones in Spanish.

Who can resist a Las Vegas souvenir mug with “AMIGA” on it? Especially if you can get “AMIGA” and “LORRAINE” together at twice the price?

Note to self: I travel too much.

Zuse Z1 at the Deutsches Technikmuseum

Michael Steil — Sat, 04 Apr 2009 03:11:20 +0000

My last blog post showed the Zuse Z3 (1939-1941), the world’s first working digital Turing-complete computer. Let’s go back two more steps: The Zuse Z1 (1936-1938) shared its design with the Z3: It read its program from punched film and used floating point as its internal representation of numbers. But since it was all mechanical, it never worked reliably.

Just like the Z3, the Z1 was destroyed in the second world war. The Deutsches Technikmuseum in Berlin has a replica built by Zuse in the 1980s – but it’s not working either.

(MP4/H.264 Video, 384×288, 00:42 min, 12.5 MB)

In 1925, Zuse receives an award document for his shovel excavator from the firm Walthers Metallbaukasten.

Patent application for mechanical memory (1937)

The Z1 in the living room of Zuse’s parents in Kreuzberg, Berlin (1937).

Zuse Z3 at the Deutsches Museum

Michael Steil — Sat, 31 Jan 2009 11:14:15 +0000

The Z3 by Konrad Zuse was the world’s first working digital Turing-complete computer. It did floating point arithmetic, had two registers accessible to the programmer, was microcoded, and clocked at about 5 Hz.

The original Z3 was destroyed in the second world war. The Deutsches Museum in Munich has the only working replica, built by Zuse in the 1960s.

Here is a video of a demonstration of the Z3, including a lot of biographical information on Zuse. If you speak German, you will be blown away by the wit of the presenter; if you don’t, enjoy the clicking relays!

(MP4/H.264 Video, 384×288, 14:30 min, 31 MB)

Zuse Apparatebau: Er kriegte sogar noch drei ArbeitskrĂ¤fte. Das waren die, die sonst keiner mehr haben wollte. Die Zuse-Firma bestand aus ihm selber, der Chef, dann ein vorbestrafter Buchhalter, ein blinder Mathelehrer – der hat die Filme gelocht… ja, der hat das Tag und Nacht gemacht – das war dem auch wurscht, ob’s Tag oder Nacht war. Und dann kam noch ein Konstrukteur aus einer Nervenheilanstalt. Was sehen wir an der Story? In der Branche waren immer schon die abgefahrensten Typen!

Archimedes Operating System (PDF)

Michael Steil — Mon, 26 Jan 2009 01:03:33 +0000

Van Someren, Alex, and Nic Van Someren.
Archimedes Operating System. A Dabhand guide.
Prestwich: Dabs Press, 1991.
ISBN 1-870336-48-8
(320 pages, 7.5 MB PDF)

For Archimedes users who take their computing seriously, this guide to the Operating System gives you a real insight into the micro’s inner workings. The book is applicable to any model of Archimedes whether running the Arthur or RISC OS Operating Systems.

The Relocatable Module system is one of the many areas covered â its format is explained, and the information necessary to enable you to write your own modules and applications is provided. This tutorial approach is repeated through the book.

The sound system is explained and the text includes much information never before published.

The discerning user will revel in the wealth of information covering many aspects of Arthur and RISC OS including:

The ARM Instruction Set

Writing Relocatable Modules

Writing Applications

VIDC, MEMC and IOC

Sound

The Voice Generator

SWIs

Vectors and Events

Command Line Interpreter

The FileSwitch Module

Floating Point Model

and much more…

Throughout the book, programs are used to provide practical examples to use side by side with the text, which go to make this publication the table-side companion for all Archimedes users.

Alex and Nic van Someren have both worked for Acorn Computers. Alex is a former Technical Editor of Acorn User magazine and author of numerous computer-related books.

In my quest to preserve retrocomputing documents, here is a very interesting reference of the orginal Acorn RISC Machine (ARM) computing platform and the RISC OS Operating System. Thanks to Dominik Wagner for the book. As always, my scanned books come with a table of contents and are fully searchable.

Amiga & Supergrips

Michael Steil — Thu, 15 Jan 2009 11:41:50 +0000

This article is in German, since it is about the German TV show “Supergrips” and how the scoreboard was implemented.

Supergrips war eine Spielshow des Bayerischen Rundfunks in Zusammenarbeit mit dem Mitteldeutschen Rundfunks, die von 1988 bis 1995 ausgestrahlt wurde.

Was besonders interessant war: Die Spielwand wurde auf einem Amiga gemacht, in einer Kombination aus Basic und Assembler. So sieht es aus, wenn eine neue Wand geöffnet wird:

(MP4/H.264 Video, 768×576)

So gewinnt man einen Block:

(MP4/H.264 Video, 768×576)

Und so sieht es aus, wenn man eine Runde gewinnt:

(MP4/H.264 Video, 768×576)

Programmiert wurde das ganze 1986/1987 von Stefani Seibold aus München, die mir auf meine Anfrage hin ein wenig über die Hintergründe erzählte:

Der BR wollte eine Spielshow aus GB übernehmen, leider war dafür so eine Tafel wie beim großen Preis oder Jeopardy erfoderlich, welche gut und gerne 100.000 DM und mehr kosten würde. Das war für eine billige Jugendsendung zu teuer.

So fragte mich ein Redaktuer vom BR, ob es nicht noch einen anderen Weg geben würde, z.B. mit dem Amiga.

So dachte ich ein bischen drüber nach um kam zum Schluß, daß das ganze durchaus mit dem Basic der Kiste zu realisieren sei. Nur die Steuerung machte etwas Probleme, wie sollte man die Felder möglichst Fehlerfrei anwählen? Tastatur-Kürzel, no way, das wäre nicht intuitiv und schnell zu zuordnen. Also ersann ich den 1 Pixel großen Maus Cursor.

Den sieht man normalerweise nur dann, wenn man weiß, daß er da ist und wo er ist. Auf den alten PAL Fernsehern von damals wurde der sowieso von fast allen Modellen verschmiert.

Aber wenn man genau hinsieht, dann kann man den 1 Pixel Cursor auch sehen ;-)

Leider war ich keine so gute Grafikerin, so wurde das Projekt dann nach den ersten Pilot-Aufnahmen noch einmal von einen mehr erfahren Entwickler übernommen, der dann noch die Sterne, die räumlich Tiefe u.a. einbaute.

Danke Stefani!

The Ultimate Commodore 64 Talk: Pushing Keynote to its Limits

Michael Steil — Mon, 12 Jan 2009 09:23:41 +0000

Download the Apple Keynote 08 presentation.

The Ultimate Commodore 64 Talk: Video Recording

Michael Steil — Wed, 31 Dec 2008 08:28:32 +0000

Link:

Thanks to F!XMBR for providing the flash video.

Additionally, here are a few links to the video recording of the talk:

Bittorrent (OGG), Mirror 1 (OGG), Mirror 1 (WMV), Mirror 2 (OGG), Mirror 2 (WMV), Mirror 3 (OGG), Mirror 3 (WMV), Mirror 4 (OGG), Mirror 4 (WMV), Mirror 5 (OGG)

The OGG version seems to be in better quality. I’ll post new links as soon as the talk is available in a standards-compliant version (MPEG-4).

The Ultimate Commodore 64 Talk @25C3

Michael Steil — Mon, 15 Dec 2008 22:15:23 +0000

Update: Video recording available.

I am going to present The Ultimate Commodore 64 Talk (Everything about the C64 in 64 Minutes) at the 25th Chaos Communication Congress 2008 (25C3) in Berlin on 29 Dec 2008. The following article, which will be printed in the Congress Proceedings, is supposed to give you an overview of what I am going to talk about. If you cannot attend 25C3, you will be able to watch a recoding afterwards, which I am going to link to. And yes, it is a lot of information, and it’ll be about 256 slides (in 64 minutes) – that is exactly the challenge. If you don’t believe I cannot present all this in a way that the audience can understand it: Watch it!

Retrocomputing is cool as never before. People play C64 games in emulators and listen to SID music, but few people know much about the C64 architecture and its limitations, and what programming was like back then. This paper attempts to give a comprehensive overview of the Commodore 64, including its internals and quirks, making the point that classic computer systems aren’t all that hard to understand – and that programmers today should be more aware of the art that programming once used to be.

Commodore History

The company that became Commodore Business Machines was founded in 1954 by Jack Tramiel. The company specialized on electronic calculators, and in 1976, Commodore bought the chip manufacturer MOS Technology and decided to have Chuck Peddle from MOS evolve their KIM-1 computer kit (a design that demos their new MOS 6502 8 bit CPU) into the Commodore PET series: computers with built-in monitors for the home, school and small business market that ended up competing with devices from Atari and Apple.

In 1981, Commodore introduced the VIC-20, a 5 KB stripped down monitorless computer-in-the-keyboard design based on the PET for the home computer market. This was followed by the (incompatible) higher-end Commodore 64 in 1982 that included more PET features, came with 64 KB of RAM (an immense amount compared to the rest of the market) and was very aggressively priced at US$595 beating the competition by a factor of two. This was made possible by designing and building most of the system in-house.

Although some features of the C64 were taken from the PET models of the time, it had to be connected to a TV set, which only made 40 columns of text possible (as opposed to then-current 80 columns on the PET). Also, the BASIC 4.0 codebase was stripped down to the old 2.0 feature set to make it fit into 8 KB.

In the beginning, the C64 did well in the competition. The superior but compatible C128 from 1985 did well, too, but was never more popular than the C64, which continued to be sold. The direct successor to the low-end VIC-20, the 1984 Plus/4 and its siblings, the C16 and the C116, failed, mostly because they were incompatible with the C64, which at that time already had a remarkable software library.

A few years after the introduction, the C64 was still offered as a low-end alternative to the Commodore Amiga, and while it became less popular in the USA, it gained more and more popularity in Europe. In the early 90s, when manufacturing costs of a C64 were as low as $25, it gained a second life in Eastern Europe. Production did not end until the liquidation of Commodore in 1994. According to the 1993 Annual Report, 17 million C64 had been produced in by then, as well as 4.5 million C128.

Look and Feel

A C64 only needs to be connected to power and a TV set (or monitor) to be fully functional. When turned on, it shows a blue-on-blue theme with a startup message and drops into a BASIC interpreter derived from Microsoft BASIC. In order to load and save BASIC programs or use third party software, the C64 requires mass storage – either a âdatasetteâ cassette tape drive or a disk drive like the 5.25″ Commodore 1541.

Unless the user really wanted to interact with the BASIC interpreter, he would typically only use the BASIC instructions LOAD, LIST and RUN in order to access mass storage. LOAD”$”,8 followed by LIST shows the directory of the disk in the drive, and LOAD”filename”,8 followed by RUN would load and start a program. If a tape drive is connected, LOAD and RUN will launch the first program on tape – pressing SHIFT and the RUN/STOP key has the same effect.

By default, typing characters without SHIFT will result in upper case characters being shown on the screen. This can be changed by pressing the Commodore key and SHIFT at the same time, which switches to the upper/lower character set. This behavior is due to the fact that the first PET only had uppercase characters, and that BASIC required keywords unshifted – but all tutorials taught them as uppercase. This is also the reason why in Commodore’s version of ASCII, called PETSCII, the codes for uppercase and lowercase characters are reversed.

The text based user interface is not a line editor, but a screen editor, i.e. the cursor can be moved freely on the screen, and pressing RETURN on a line with existing text will present the text to the application as if it were just typed in.

While a physical screen line is only 40 characters, the screen editor logic can extend it to a logical 80 characters – whenever a character is entered on the 40th column, the rest of the screen is moved down by one line, âopeningâ a line that extends the previous line to 80 characters.

The C64 screen editor supports selecting one out of 16 color for the foreground text by pressing the key combinations Ctrl+1 to Ctrl+8 and Commodore+1 to Commodore+8. Reverse text mode can be turned on and off with Ctrl+9 and Ctrl+0.

Ports and Connections

The C64 has a whole range of connection possibilities. On the side, it has two 9 pin Atari-style joystick connectors that can also be used for a mouse, light pens or paddles. On the back, there is the expansion port, which exports the complete processor bus, allowing not only game cartridges but also cartridges with I/O chips that map themselves into the CPU’s address space – or even cartridges that completely replace the CPU. For a conventional TV connection, there is an RCA connector that outputs an RF signal. For monitors, there is an extra DIN connector that carries separate chroma, luma and audio signals (although not 100% S-Video, it is compatible with a lot of S-Video equipment).

For connecting Commodore compatible printers and disk drives, there is a DIN connector for the IEC bus. There is also a dedicated connector for datasette drives. The âUser Portâ consists of several GPIO pins that can be used for custom hardware projects, or as a RS-232 port (with TTL levels), for which support exists in the ROM.

Board

On the C64 motherboard, there is a dedicated IC each for the main tasks. There is the MOS 6510 CPU, eight 64 KBit RAM chips (later consolidated into 2), three ROM chips with KERNAL (I/O library), BASIC and the character set (KERNAL and BASIC were later consolidated), two 6526 CIA I/O controllers (one for keyboard and joystick, one for the IEC bus and the user port), the 6581 SID sound chip, and the 6567/6569 VIC video chip, as well as the RAM chip that holds the 512 bytes of Color RAM.

All non-RAM chips are custom chips designed manufactured by MOS, Commodore’s inhouse chip company.

Address Space

The 8 bit C64 design has a 16 bit address bus, allowing the CPU to address 64 KB of memory. Since the C64 has 64 KB of RAM, filling the complete address space, ROM and I/O chips are mapped into regions of the address space that are shared with RAM: The CPU can switch these regions between RAM and a second or third mapping. These regions are as follows:

$0000-$9FFF: RAM
$A000-$BFFF: RAM or BASIC ROM
$C000-$CFFF: RAM
$D000-$DFFF: RAM or memory mapped I/O chips or character ROM
$E000-$FFFF: RAM or KERNAL ROM

In contrast to CPUs like the Z80 and the 8086, and like most modern CPUs, I/O devices are memory mapped on the C64′s 6510 CPU. The mapping is as follows:

$D000-$D3FF: VIC video controller
$D400-$D7FF: SID sound controller
$D800-$DBFF: Color RAM
$DC00-$DCFF: CIA 1 I/O controller
$DD00-$DDFF: CIA 2 I/O controller
$DE00-$DFFF: for extensions on the expansion port

6502 CPU

The CPU inside the C64 is a 0.985 MHz (on PAL) MOS 6510, which is a close derivative of the well-known 8 bit little-endian MOS 6502. The 6502 was introduced in 1975 by MOS Technology, a company formed earlier the same year by former Motorola engineers, with engineering headed by Chuck Peddle. The philosophy of the 6502 was to have a reduced instruction set and a small register file, making it simpler and faster than CPUs like the Z80 at the same clock speed, as well as cheaper to manufacture.

Unlike most modern CPUs, the 6502 does not have a set of general purpose registers. Instead, it has a single accumulator A (for arithmetic and logic), two index registers X and Y (for incrementing, decrementing and indexing memory) and a stack pointer. All these registers are 8 bits. The processor status, consisting of the negative (N), overflow (V), break (B), decimal (D), interrupt (I), zero (Z) and carry (C) flags is exposed as register P. The program counter (PC) is 16 bits wide. The fact that the stack pointer is 8 bit means that the stack is confined to the area between $0100 and $01FF in the address space, i.e. the upper half of the effective stack pointer is hard-coded to $01. There is another special area in the address space: The first 256 bytes, at $0000 to $00FF are referred to as the zero page (ZP). Many instructions support special encodings for zero page addresses, which saves one byte in the instruction encoding as well as at least one cycle of execution time. This can be seen as an extension of the register file to another 256 (though external) registers.

All opcodes are one byte, and have 0, 1 or 2 byte operands. The 8×8 opcode matrix is somewhat logical (e.g. branch instructions are encoded as $10, $30, $50, …), but there is no easy rule to construct the opcode table. Nevertheless, the opcode table is a minimal encoding for optimal decoding in the 6502′s internal PLA ROM.

Instruction Set

The instruction set is very streamlined, and avoids redundancies. There are load instructions (LDA/LDX/LDY to load A, X and Y respecively), store instructions (STA/STX/STY), read-modify-write instructions (logic: ASL/LSR/ROL/ROR, count: INC/DEC), arithmetic (ADC/SBC; note that these always include the carry: CLC/ADC is a regular addition, and SEC/SBC is a regular subtraction, because of the one’s complement logic), compare (CMP/CPX/CPY; these are subtractions without storing the result), logic (AND/ORA/EOR, and BIT, which is AND without storing the result), as well as branch instructions, flag manipulation, register transfer and stack manipulation.

Addressing Modes

Each instruction supports one or more addressing modes. Common instructions like LDA (load accumulator) support more addressing modes than less common ones (BIT).

The immediate addressing mode is indicated with a # sign: LDA #$17 loads the immediate value of $17 into the accumulator.
Absolute addressing specifies a 16 bit address as an operand: LDA $0314 loads from the memory address $0314.
Zero page addressing is an optimized version of absolute addressing: LDA $02 will read from address $0002 in memory, but the instruction can be encoded more tightly, and execution is faster.
Absolute-X-indexed addressing reads from a specified address, to which the contents of the X register is added. LDA $0200,X reads from the address $020A, in case X is $0A. This allows reading from tables.
Absolute-Y-indexed is the same thing, but with the Y register.
Zero-Page-X-indexed is an optimized version of Absolute-X-indexed. LDA $F0,X reads from the Xth location in a table stored starting at $00F0 in memory. Note that all zero page addresses will wrap around, so $F0 + $10 = $00.
Zero-Page-Y-indexed is the same thing, but with the Y register.
Zero-Page-X-indexed-indirect adds X to a specified zero page address, reads a 16 bit pointer from the resulting address and finally accesses memory at that address. So LDA ($80,X) will read from an address specified by the array of pointers at $0080 and the index X into the array. This addressing mode is rarely used.
Zero-Page-indirect-Y-indexed treats two consecutive bytes in zero page as an address and adds Y to the address. LDA ($14),Y will read from $E020, if the address stored at $14/$15 is $E000 and Y is $20. This addressing mode is the most convenient way to work with pointers, as no register can hold 16 bits.

Register Transfer and Stack

There are several 1 byte instructions without operands that move data between registers. TAX, TXA, TAY and TYA move between A, X and Y. TSX and TXS copies between X and the stack pointer.

The stack pointer always points to the next address that is written to. This means that an empty stack has a stack pointer of $FF, and pushing a value first writes the value and then decrements the stack pointer. The 6502 can move the accumulator from and to the stack (PHA/PLA), as well as the processor status P (PHP/PLP).

Control Transfer

Next to the absolute JMP instruction, there is an indirect version that jumps over a vector (e.g. JMP ($FFFC)). JSR (jump to subroutine) only has an absolute version, and stores the address of the next instruction minus one on the stack. RTS (return from subroutine) takes the address from the stack, adds one, and moves it into the program counter. The âminus oneâ logic was chosen because it could save one cycle in the implementation of JSR.

A hardware interrupt, unless disabled by a set interrupt (I) flag, pushes the address of the next instruction minus one (just like JSR), pushes the processor status afterwards, disables interrupts, and jumps over the vector at $FFFE/$FFFF. RTI return from interrupt) is the same as the combination of PLP and RTS. BRK causes a software interrupt and behaves the same as a hardware interrupt, except that it sets the B flag in the processor status on the stack to 1 (a hardware interrupt sets it to 0). Note that the B flag does not exist in the actual processor status register, the corresponding bit is only used in a status byte on the stack.

From a software perspective, NMIs behave the same as IRQs, but they cannot be masked, and they use the $FFFA/$FFFB vector. The reset vector is at $FFFC/$FFFD.

Flags and Branches

All load and logic instructions set N and Z accordingly, shift instructions also modify C, and arithmetic instructions touch N V, Z and C. The D (decimal), I (interrupt disable) and C flags can be set and cleared programmatically (CLD/SED, CLI/SEI, CLC/SEC), while the V flag can only be cleared (CLV). Conditional branches are possible based on the value of the negative (BPL/BMI), overflow (BVC/BVS), zero (BNE/BEQ) and carry (BCC/BCS) flags. Branches encode an 8 bit relative offset and can therefore reach code in the area of +127 and -128, counting from the byte after the branch instruction. Since a compare is the same as a subtraction, BCC is a branch on (unsigned) below, and BCS is a branch on above-or-equal.

NOP

NOP (no operation) does nothing. Its encoding is $EA.

Decimal Mode

If the D flag is set, all ADC and SBC operations will be BCD-adjusted afterwards, i.e. $09+$02 won’t be $0B, but $11, since 9+2=11. The BCD correction circuit has been patented in US patent 3,991,307.

Cycle Counting

It is quite straightforward to find out how many cycles an instruction takes. As a rule of thumb, an instruction takes as many cycles as the number of memory fetches it has to perform, but at least two.

Therefore, single-byte opcodes (one byte fetches; NOP/TAX/INX etc.) as well as instructions with immediate operands take two cycles. Zero page instructions take three memory accesses (opcode, address, data), so they are three cycles. Absolute instructions take four accesses (opcode, address low, address high, data), so they are four cycles.

Read-modify-write instructions (INC/DEC/shift/rotate) are an exception: They require 4 memory accesses for the zero page case and 5 otherwise, but they take 5 and 6 cycles, respectively.

Branches take 3 cycles if they are taken and two if they are not taken. And extra cycle has to be added if the branch crosses a page boundary. JMP is 3, push is 3, pull is 4, JSR and RTS are 6 each.

All other timings can be looked up in the 6502′s reference, but they are very easy to memorize.

Common Tricks

The BIT instruction exists in a two-byte (immediate operand) and three-byte (absolute operand) variant. Since BIT only changes the flags, it effectively skips one or two bytes in the instruction stream. This can be used to replace a two-byte branch or a three-byte JMP with a one-byte BIT if only one or two bytes have to be skipped.

The architecture allows safe self-modifying code, so a common optimization for copy loops is to use LDA $nn00,X and STA $mm00,X, looping X from $00 to $FF and then incrementing the bytes that encode nn and mm for the next page. Compared to a LDA (zp1),Y: STA (zp2),Y sequence, this gets the inner loop down from 16 cycles (5 LDA, 6 STA, 2 INY, 3 BNE) to 14 (4 LDA, 5 STA, 2 INY, 3 BNE).

A PHA/PLA combination is 7 cycles, while an STA/LDA combination in the zero page is 6 cycles, so unless there is no free zero page space, PHA/PLA should be avoided to quickly store a value. Using an absolute store to write the value into the operand of a future immediate load (i.e. self modification) is the same speed at the zero page solution, but does not waste zero page space.

An elegant way to store a flag is to have it in bit #7 of a zero page address. While a load/store combination has to be used to set the flag (or the slower, but register-preserving SEC/ROR combination), it can be cleared with a simple LSR (5 cycles) and tested with BIT (3 cycles), without affecting register contents.

Since the 6502 is so register starved, only 3 bytes can be passed to a subroutine in registers. Also, the stack is small, and accessing it is slow, so stack frames as seen on modern architectures are very uncommon. Many applications and libraries (e.g. GEOS) use a dedicated area in the zero page as virtual registers.

The 6502 has no instructions for multiplication, division or floating point arithmetic. Most 6502-based computers have a BASIC interpreter in ROM though, and they typically include a math and floating point library.

Bugs and Quirks

The original 6502 implementation has a series of bugs and other anomalies that have never been fixed in MOS chips (not counting the 65CE02, which was only used in Amiga peripherals).

The indirect version of JMP loads the program counter from the wrong address, if the vector’s address lies on a page boundary: JMP ($23FF) will read the address from $23FF and $2300 instead of $23FF and $2400.

When in decimal mode, the negative flag reflects the original binary result, not the effective decimal result.

If a software interrupt (BRK) and a hardware interrupt occur at the same time, the BRK is dropped.

STA with the absolute-indexed addressing mode takes first reads from the absolute address without the index register added, and then reads again from the correct address. LDX #$07 STA $DC0D,X will first read from $DC0D, discard the result, and then write to $DC14. On the C64, this read form $DC0D would ACK all pending CIA 1 interrupts, while it is only supposed to write to $DC14.

Read-modify-write instructions with absolute addresses first read the value, but one cycle before they store the result, they store the original value again. On a C64, this can be seen when incrementing the screen border color at a defined area of the screen, as every write to the register will cause a tiny gray dot on the screen (on later HMOS versions of the VIC). When this is used with certain I/O ports, this can have other side effects. The latter two quirks have been used heavily for obfuscating copy protection software.

Instruction decoding in the 6502 is done by a PLA that compares the current cycle number within the instruction and the current opcode against a ROM of 130 mask lines, of which any number can fire independently. The outputs of these lines are then fed into components like the ALU, bus control, register control and program counter logic. The instruction set only consists of 151 defined opcodes, and since handling the remaining 105 opcodes as NOPs or traps would have required extra lines in the PLA, they will match against some lines that were meant for instructions with similar opcodes. Some of these âillegal opcodesâ lead to useful results and are used in some software (SAX = store A & X), but most of the instructions make little sense (LAS = loads from memory, ANDs it with S and stores it into A, X and S), and some even lock up the CPU, disabling IRQs and NMIs (CRA/KIL).

The MOS 6510

Except for the pin layout, the MOS 6510 that is used in the C64 differs from the generic MOS 6502 in two ways: It can make the bus tri-state when not used, so the VIC can use it, and it has a 6 bit I/O port built in, which can be controlled using zero page locations 0 and 1. In register 0, each bit from 0-5 set it to output if 1, and to input if 0. Bits 0-5 in register 1 are the actual I/O pins. On the C64, bits 0-2 are outputs and control bank switching, they turn the ROMs and the I/O area on and off. Bits 3-5 go to the tape connector and control the motor and the data sent to the head, and detect whether a key on the tape deck is pressed.

BASIC

Microsoft had a strong position in the market for (mostly ROM) BASIC interpreters in 8080-based home computers when the MOS 6502 was released in 1975, so they rewrote their interpreter in 6502 assembly. Microsoft BASIC was pure 6502 code with a minimal character I/O interface to the machine’s âmonitorâ, i.e. I/O library.

Commodore decided to license the interpreter for the 1977 PET and extended it slightly to interface with their disk and tape libraries. Commodore BASIC was very buggy, so they went back to Microsoft for an update, which, with the Commodore changes re-applied, shipped in newer PETs as BASIC V2. For version 4, Commodore added several instructions for more conveniently dealing with disk.

Being a low-end machine, Commodore took the bugfixed BASIC V4 codebase and removed all features after V2, making it independent of the machine’s graphics and sound features again, and fitting it back into 8 KB, and shipped this version on the VIC-20. The Commodore 64 got the almost exact same version, but it runs at a different memory address ($A000-$BFFF).

Microsoft BASIC is a line-based editor, that is, lines can be shown with the LIST command, and they can be modified by re-typing them. This integrates nicely with the KERNAL screen editor: The cursor can be moved up to LISTed lines, the lines can be modified, and when RETURN is pressed, the whole line is fed into BASIC again.

A nice feature of this and later versions of Commodore BASIC is the fact that all important parts, like the tokenizer, the detokenizer and the interpreter loop jump over a jump table in RAM before they do their work, allowing the user to extend BASIC arbitrarily. The most well-known BASIC extension is Simons’ BASIC, a cartridge that maps 8 KB of extra ROM at $8000-$9FFF.

KERNAL

The C64 has an 8 KB I/O library at $E000-$FFFF which is utilized by BASIC, but is intended to be used by other applications as well. All Commodore 8 bit systems have a standardized library call interface in the form of jump tables at the very top of memory that call into machine-specific functions for I/O.

KERNAL is started form the RESET vector, initializes the machine, sets up an interrupt service routine that handles the keyboard, animates the cursor and does the real time clock. The C64 has a hardware clock in each of the CIA chips, but KERNAL has not been updated to use this feature since the VIC-20.

KERNAL provides an abstract character I/O interface to a number of devices. All devices support open, read, write and close. The open call takes three parameters: The logical file number (there is a maximum of 16 channels), the device number and the secondary address. There are 32 device numbers statically assigned to the devices. The 8 bit secondary address can signal something to the device, like speed or an operation mode. Some devices (tape and IEC) support an optional filename.

Device 0 is the keyboard. While KERNAL exports raw key presses, the keyboard can also be accessed through character I/O, which will go through the screen editor and replay all characters on the screen that are in the line of the cursor, regardless of whether the user typed them or they had been there before.

Device 1 is the tape drive. KERNAL reads and writes blocks of data at a time and buffers them for character I/O.

Device 2 is RS-232. KERNAL contains a very sophisticated (but rarely used) software RS-232 implementation that supports up to 2400 baud.

Device 3 is the screen. KERNAL interprets special codes, manages the cursor position and handles scrolling.

Devices 4 and greater will be directed to the IEC bus. By convention, devices 4 to 7 are printers and plotters, and devices 8 to 30 are floppy drives or hard disks.

KERNAL allows interacting with the IEC bus manually by sending TALK and LISTEN requests to the bus.

GEOS

While KERNAL is a minimal character-based operating system in ROM, there is also a disk-based operating system with a graphical user interface for the C64. GEOS was released by Berkeley Softworks in 1986 and Commodore bundled it with the C64 for some time. The GUI, which can be controlled by a joystick or a mouse, runs in 320×200 graphics mode and resembles early versions of MacOS. GEOS is a 16 KB library that includes an optimized disk interface (faster, support for timstamps, icons and multi-fork âVLIRâ files), library code for drawing to the screen, high level UI primitives for menus, buttons and dialog boxes (with callbacks) and a simple memory swapping facility. Furthermore GEOS allows input and printer driver plugins, as well as proportional fonts in different sizes. Internally, GEOS has a jump table to its library routines that consists of about 150 entries.

GEOS came with applications like GeoPaint and GeoWrite; Berkeley Softworks themselves offered solutions like GeoPublish and GeoCalc, and more software was available from third parties.

GEOS’ only requirement is a 1541 disk drive, but a 3.5″ 1581 drive, a RAM extension or one of the later hard drives helped speed it up a lot.

6526 CIA

The C64 has two identical 6526 CIAs (Complex Interface Adapter) that are mostly used for I/O. One CIA features 16 general purpose I/O pins (8 bit port A and 8 bit port B) that can be used either as an input or an output, two programmable timers and a real-time-clock.

The timers have 16 bit counters and count down by one either on each clock cycle, or on an external event, or on a timer A underflow (in the case of timer B). This allows concatenating the timers to one 32 bit timer. On an underflow, the CIA can be programmed to cause an IRQ or to send data through a serial shift register. The CIA also supports receiving data through a shift register.

The real-time-clock has a resolution of 1/10 of seconds and supports generating interrupts at a certain time.

CIA 1 is hooked up to the keyboard and the joystick ports. Since the keyboard consists of 64 keys (plus SHIFT LOCK, which is parallel to the left SHIFT key, and RESTORE, which is directly connected to the CPU’s NMI line), these can be laid out in a 8×8 matrix of lines, key presses connecting the intersections. One side of the matrix is connected to port A (output), and the perpendicular side is connected to port B (input). The keyboard driver can now write the values of $01, $02, $04 etc. in port A and test the input of port B to see which keys are pressed. The two joysticks are connected in parallel to port A and port B, so they can cause spurious keyboard events.

CIA 2 is hooked up to the IEC bus, and I/O lines control the VIC bank. The rest is exposed on the user port, and can be used for RS-232.

KERNAL uses CIA 1 for the 60 Hz system timer, but, apart from the ports, doesn’t use any of the extra features of either CIA.

6581 SID

The SID (Sound Interface Device) is a whole topic of its own.

6567/6569 VIC

The video chip inside the C64 is called the MOS 6567/6569 VIC-II (Video Interface Controller) – the video chip in the VIC-20 had been the original VIC, and was the reason for the marketing name of the VIC-20.

The VIC supports a 40×25 text mode, a 320×200 bitmap mode, 16 colors and 8 sprites – all of these features have lots of sub-modes and options. The amount of memory the VIC can address is 16 KB, and while by default, it accesses the first 16 KB of the C64 RAM, it can be configured to use any of the four banks.

The 16 colors of the VIC are divided into two sets of eight. The first eight are the more important colors, as some modes only support the first eight. The colors are, in the original order: black, white, red, cyan, purple, green, blue, yellow, orange, brown, pink, dark gray, gray, light green, light blue and light gray.

Character Mode

The C64 has two built-in character sets that the VIC can access. They can be shown on the screen by writing the numbers 0 to 255 into screen RAM (at $0400 by default). The default font has uppercase characters and lots of line-drawing symbols in the lower 128 characters, and the second half consists of the same characters, but inverse. The alternative font has upper- and lowercase characters and omits some of the symbols.

The foreground color of the characters can be changed by writing the color numbers into the Color RAM, which is located at $D800-$DBFF. There is one byte per character, but only the lower 4 bits are actually preserved by Color RAM.

Each character is 8×8 pixels, and stored as eight bytes in the character ROM (or RAM, if a user-defined character set is enabled), one line being one byte. A 1-bit will take the color from the Color RAM ($D800-$DBFF), and a 0 bit will take it from the global background color register ($D021). The pixel matrix is determined by looking up the character index in the screen RAM (at $0400-$07FF by default) and consequently looking up the pattern the current character set (the VIC sees the default font at $1000, although for the 6502 it is invisible there).

In Extended Color Mode (ECM), it is possible to chose between one of four background colors (registers $D021 to $D024) with the upper two bits of the character index, but then only 6 bits will be used to look up the character pattern, decreasing the number of possible characters to 64. The built-in (uppercase) character set is well-suited for this: While it is similar to the ASCII encoding, it has the uppercase characters mapped to codes $01 to $1A, so the most important characters are within the first $40.

Multi-Color Character Mode allows up to four colors per character and is intended for tile-based games, like platformers. If bit 3 of the value in Color RAM is 0, then the character gets displayed just like in non-multicolor mode, but colors are restricted to the first eight. If bit 3 is 1, then pairs of horizontally adjacent bits are combined in their meaning: 00 represents the screen background ($D021), 01 is the second background register ($D022), 10 is the third background register ($D023), and 11 is the color specified in bits 0-2 of the Color RAM. Pixels in these characters are twice as wide, so the resolution of a character is 4×8.

Bitmap Mode

In hi-res graphics mode, the VIC supports a resolution of 320×200, which uses the same pixel frequency as 40×25 character mode (40*8=320, 25*8=200). The bitmap can reside at $0000 or $2000, and the VIC reads one byte for 8 pixels. But hi-res mode does not only support monochrome graphics: The foreground and background colors of each 8×8 tile are taken from the high and low nibble of screen RAM, which would otherwise be unused. Color RAM is not used in this mode.

The encoding of the bitmap is identical to the encoding of a character set, making it a non-linear framebuffer: The first eight bytes of the bitmap represent the pixels in the tile at character position (0,0), the second eight bytes represent the tile at character position (1,0), which is pixel position (8,0), and so on. This layout makes pixel addressing in software slower.

In Multi-Color Bitmap Mode, the horizontal resolution is halved to 160×200, and pixels are twice the width. Every set of two bits in the encodes one of four colors per tile: 00 takes it from the global background register ($D021), 01 and 10 take it from the upper and lower nibble of screen RAM, respectively, and 11 takes it from Color RAM.

Scrolling

The VIC supports hardware X and Y scrolling by 0 to 7 pixels. Since the 40th column is half visible and another column left of the first column is half-visible when the horizontal shift register is set to e.g. 4, a 41th column would be needed. Instead, it is possible to switch the screen to 38 column mode, i.e. the whole screen is a little narrower, and more border is shown on the left and on the right. The screen can also be switched from 25 to 24 lines the same way.

Sprites

The VIC has eight hardware sprites (also called MOBs, movable objects). Each sprite is 24×21 pixels, which is encoded in 63 bytes. Set bits will be drawn in the sprite’s individual foreground color, and cleared bits will be transparent. The index to the sprite’s bitmap data in memory is an 8 bit value that is read from the last 8 bytes of screen RAM – since the screen is only 1000 characters, the last 24 characters of the $0400=1024 bytes area would otherwise be unused.

There is also a multicolor mode for sprites, which makes pixels twice as wide and decreases the horizontal resolution to 12 pixels. In this mode, 00 is still transparent, and 10 encodes the sprite’s individual color. The codes 01 and 11 take the color out of the sprite multicolor registers ($D025 and $D026), which are shared among all sprites.

Sprites can be positioned at arbitrary pixel positions on the screen, and overlap. In this case, sprites with lower numbers have priority over sprites with higher numbers. Each sprite can either be shown in front or behind background pixels. Sprites can be X- and Y-expanded by a factor of two, and collision of two sprites or of a sprite and background pixels is detected by hardware: Whenever two non-transparent sprite pixels are drawn at the same position on the screen, they have collided. Whenever a non-background pixel is drawn by the character generator at the same position where a non-background pixel of a sprite is draw, the sprite has collided with the background. An exception from this rule is the 01 code in the character data, which also counts as background. This way, a background picture can be constructed that does not cause collisions in certain areas. In practice, most newer games do not use the hardware functionality, but instead test for overlapping sprite bounding boxes in software.

Memory Layout

The VIC can address 16 KB at a time. All VIC data structures can be stored anywhere in these 16 KB, but they have to be aligned to their size.

The screen RAM is $0400 bytes in size and can be at $0000, $0400, …
The character set is $0800 bytes, and can be at $0000, $0800, …
The bitmap is $2000 bytes and can be at $0000 or $2000.
Sprites are $40 bytes, and can be at $0000, $0040, …

Two GPIO pins of the second 6526 CIA are connected to bits 6 and 7 of RAM when the VIC accesses it. By changing the lower 2 bits of $DD02, the VIC can be switched between banks $0000-$3FFF (11), $4000-$7FFF (10), $8000-$BFFF (01) and $C000-$FFFF (00).

If the VIC is set to banks $0000 or $8000, then the two built-in character sets shadow RAM in the area of $1000-$1FFF. This means that the built-in character set can be used on those banks without occupying RAM, but it also means that the area from $1000-$1FFF cannot be used for bitmap, screen RAM or sprite data either.

For timing reasons, color information is not taken from main RAM, but from a dedicated Color RAM. These $0400 half-bytes are accessible to the C64 at $D800-$DBFF and can not be bank switched.

Timing

For advanced VIC programming, it is necessary to not just set up a certain mode and have the VIC display it, but to reprogram the VIC while it is drawing the picture. For this, it is necessary, to understand its timing.

While the pixels within the screen area are 320 by 200, the VIC actively draws pixels in the border color outside of this area, which (on PAL) is 403×284 pixels. Analog TV standards specify an H blank area at the end of every line, and V blank area at the end of every screen. So counting this timing as pixels, this gives an absolute resolution of 504×312 pixels. The interesting and very useful connection about the pixel clock and the system clocks is that an 8 pixel character is drawn every system clock cycle, i.e. about 1 million times a second. The 504 horizontal pixels therefore mean that a line is drawn on the screen every 63 cycles. With this information, it is possible to do cycle-exact timing of assembly code to switch a VIC register at an 8 pixel granularity.

Further timing details (badlines, sprite timing), as well as the application of this information to do tricks like FLD, FLI and AGSP would go beyond the scope of this article, but are talked about in the presentation at the 25th Chaos Communication Congress.

Memory Configuration

In a running system with BASIC and KERNAL, the BASIC and KERNAL ROMs are turned on and visible at $A000-$BFFF and $E000-$FFFF respectively, and at $D000-$DFFF, the I/O area is visible. Using the 3 lowest bits in the processor port at address 1, this configuration can be altered. The ROMs can be turned off, revealing RAM instead, and the I/O area can be configured to show either RAM or the character set ROM. Note that writing to ROM will always direct the data to the underlying RAM.

In practice, many programs run in the default configuration and use both KERNAL library routines, as well as functions in BASIC, to keep their own code as small as possible. More recent programs and almost all games turn off all of ROM, to get direct control of the interrupt vector without having to go though the KERNAL handler first. The I/O area is typically configured to show the I/O registers and Color RAM, and only rarely switched to a different configuration to temporarily access the RAM underneath. A few applications read the character ROM at program start and modify the copy.

The C64 supports another ROM bank at $8000-$9FFF, which can only be serviced by an external cartridge connected to the expansion port. Also, the $A000-$BFFF bank can be overridden by an external ROM. If KERNAL detects the magic string âCBM80â at $8004 on startup, it will jump to the code of the cartridge right away.

Expansion port cartidges can also put the C64 into “ULTIMAX” mode, in which the it can play game cartridges for the failed C64-based stripped down “ULTIMAX” system (“Commodore MAX”, “VC-10″): All internal ROM is disabled, $8000-$9FFF and $E000-$FFFF show external ROM, RAM is only visible at $0000-$0FFF, and the VIC has a different view on the address space. This mode is used by freezer cartridges which can use it to replace the $E000-$FFFF bank and thus override the hardware vectors.

Tape Interface

The tape interface consists of a single line each for data input and output, motor control and key sense. The raw data is read from and written to the data lines, and all encoding and decoding of the data stream is done in software. 3 of the required lines are connected to the processor port at zero page location 1, and one (data input) is connected to CIA 2.

IEC Bus

The IEC bus is a serial version of the IEEE-488 bus used on the PET. Devices on the IEC bus are daisy-chained, and are all connected to the same three lines: ATN (attention), clock and data. IEC has a single bus master, which is the computer. It is the only device to ever raise ATN, while every device can output to clock and data, depending on the state of the bus.

If the computer raises ATN, every device on the bus listens for the command which includes the 5 bit device number (0-30) and compares it with its own. The protocol on who sends and who receives is determined by the computer sending TALK/UNTALK and LISTEN/UNLISTEN commands.

While KERNAL exports the interface at this level, it also allows high-level open, close, read and write operations on the IEC bus, as well as load and save operations. The BASIC LOAD and SAVE commands are directly hooked up to this interface.

The IEC bus was designed for the serial shift register in the VIA (Versatile Interface Adapter) of the VIC-20 and its disk drive, but it turned out that the VIA had a bug that made the shift register unusable, so the IEC protocol had to be implemented in software. While the C64 has CIAs, in which the bug has been fixed, the 1541 still used VIAs. It wasn’t until the C128 (in its native mode only) that the computer could talk to the floppy drive (Commodore 1570/1571/1581) in its intended speed.

1541 Disk Drive

The Commodore 1541 Disk Drive is the most common disk drive used with the C64. It uses 5.25″ SS/DD (single side, double density) disks, but disks can be flipped, and the other side can be used as well, if the disks are double sided. The 1541 does not use the index hole, and uses software markers (SYNC) instead to be able to tell the start of a sector. Due to reliability problems of early drives, the 1541 only uses 35 out of the 40 possible tracks on a 5.25″ disk. The tracks have a variable number of 256 byte sectors, ranging from 21 on the outside to 17 on the inside. The data is written in 4 different speeds. This makes an overall 683 sectors, or 174,848 bytes.

The file system is stored on track 18. Track 18, sector 0 contains the disk name and the BAM (block availability map), which stores one bit per sector (1 = free). Track 18, sector 1 is the first sector containing directory entries: There are eight 32 byte entries per sector, with a maximum filename length of 16 characters. The first two bytes of a directory sector point to the next directory entry sector.

The files on disk are also stored as a linked list. The first two bytes of every sector are either the track and sector number of the next block, or the first byte is 0 and the second byte is the number of valid bytes in this sector.

The 1541 is a stripped down version of the PET drive series, which had a parallel connection, and contained two 6502 CPUs: One for doing the filesystem and communicating with the computer, and one for reading data from disk and writing data to it, as well as encoding and decoding the data. The 1541 only has a single 6502 CPU running at 1 MHz, which (using timer IRQs) regularly switches itself between the two modes. The two virtual CPUs still communicate with each other using a messaging interface in the zero page. The 1541 has 2 KB of RAM at $0000-$07FF.

The 1541 has two VIA I/O controllers at $1800 (for the IEC bus) and at $1C00 (for the drive). The firmware is located at $C000-$FFFF.

Since loading an application or a game takes minutes on an unmodified C64, several âfloppy speedersâ appeared (either as software on disk or built into applications, as ROM extension cartridges, or as internal replacement ROMs), that consisted of implementations of more optimized protocols for the IEC bus for both the C64 and the 1541. The 1541 code was uploaded using the old bus protocol. Such a new protocol would for example not do a handshake on every bit using the clock line, but shift a complete byte through in 4 steps, two bits at a time, using the clock and data line at the same time. This would of course only work if both CPUs were not interrupted. VIC timing on the C64 side could already affect this, so many floppy speeders turned off the screen while loading.

Other Peripherals

The 1541 is the necessary companion to a C64. It can be replaced by a 1570 (1541 with fast bus routines for the C128) or a 1571 (double-sided 1570), since they include a 1541 compatibility mode. The 3.5″ Commodore 1581, which supports 880 KB per disk, can hardly be a replacement for a 1541, because most applications contain their own floppy speeders that make lots of assumptions on the exact on-disk format. For GEOS, it can be very useful though.

Creative Micro Devices sold and continues to sell a line of hard drives that have an IEC connection but contain a 3.5″ SCSI drive inside. Although they have a 6502 CPU built in and allow code execution on their CPU, they are not compatible enough to replace a 1541 either.

Several memory extension cartridges exist for the expansion port of the C64: The Commodore REU (contains a DMA chip that transfers data between itself and main RAM), as well as the third party GeoRAM (maps a block to the $DE00-$DFFF area) and RAMLink (battery backed, designed as RAM disk).

Freezer cartridges like the Action Replay and the Final Cartridge were not only popular because they could dump all of memory to disk and thus copy certain copy-protected games, but also because they featured floppy speeders that disabled the original routines directly at startup time, without any effort from the user.

In the mid 90s, CPU speeders for the expansion port became popular. The 8 MHz Flash 8 is rare today, but many enthusiasts have a SuperCPU, which replaces the onboard CPU with a 20 MHz 65C816, which has a native 16 bit mode that can address up to 16 MB or RAM. There are a few applications and games that require a SuperCPU. The speedup of GEOS with a SuperCPU is significant.

In the 2000s, the enthusiast scene created all kinds of peripherals, like ethernet interfaces, IDE interfaces and SD card readers.

And there are not only peripherals: In 2004, the Commodore 64 DTV, a reimplemented C64 appeared in the form of a Joystick. The device is fairly compatible and can be extended to connect to a keyboard and a 1541.

Copland D9 Booting

Michael Steil — Sat, 06 Dec 2008 08:31:05 +0000

Click on the image for a video of Copland D9 (the first developer release; the second one being D11E4), booting a bit, then hanging. It is running on a Power Macintosh 6100.

What Operating System Is This?

Michael Steil — Fri, 21 Nov 2008 05:50:48 +0000

Unfortunately, I have not been able to successfully boot this up all the way on this machine – it hangs on this screen.

Edit: Thanks for the great comments. It is indeed Copland. See my followup post for details.

Inside Macintosh Volumes I, II, III (PDF)

Michael Steil — Tue, 18 Nov 2008 09:10:58 +0000

Here are all three volumes of the original 1985 edition of Inside Macintosh as a searchable PDF:

Inside Macintosh Volumes I, II, III
(1284 pages, 18 MB)

Inside Macintosh consists of three volumes. Volume I begins with the following information of general interest:

a “road map” to the software and the rest of the documentation

the user interface guidelines

an introduction to memory management (the least you need to know, with a complete discussion following in Volume II)

some general information for assembly-language programmers

It then describes the various parts of the User Interface Toolbox, the software in ROM that helps you implement the standard Macintosh user interface in your application. This is followed by descriptions of other, RAM-based software that’s similar in function to the User Interface Toolbox. (The software overview in the Road Map chapter gives further details.)

Volume II describes the Operating System, the software in ROM that does basic tasks such as input and output, memory management, and interrupt handling. As in Volume I, some functionally similar RAM-based software is then described.

Volume III discusses your program’s interface with the Finder and then describes the Macintosh 128K and 512K hardware. A comprehensive summary of all the software is provided, followed by some useful appendices and a glossary of all terms defined in Inside Macintosh.

I have been told that there are several people in Apple’s operating system kernel team today that are younger than the original Macintosh, therefore I see this as act of preserving retrocomputing documents; the times when someone cared about the copyright of this must have long been gone.

STATUS	check for keypress
INP	get key from keyboard
OUTP	send character to screen
PRINT	send character to printer
AUXIN	get character from serial
AUXOUT	send character to serial
READ	read sector(s) from disk
WRITE	write sector(s) to disk
DSKCHG	check for disk change

CP/M to PC-DOS/MS-DOS
Change	New CPU New OS codebase
Running new applications	Native
Running old applications	Not supported
Running old drivers	Not supported
Porting applications	High level of source/ABI compatibility

MS-DOS to Windows
Change	New OS
Running new applications	Native
Running old applications	Virtual machine with old OS
Running old drivers	Virtual machine with old OS
Porting applications	No migration path

DOS to Windows NT
Change	New OS
Running new applications	Native
Running old applications	API reimplementation
Running old drivers	Not supported
Porting applications	No migration path

Windows 3.1 (Win16) to Windows 95 (Win32)
Change	New CPU mode/bitness
Running new applications	Thunking
Running old applications	Native
Running old drivers	Native
Porting applications	High level of source compatibility

Windows 9X to Windows NT
Change	New OS
Running new applications	Native
Running old applications	Win16: Thunking Win32: Native
Running old drivers	Providing the same API for the old OS
Porting applications	High level of source compatibility Providing the same API for the old OS

Windows i386 (Win32) to Windows x64/x86_64 (Win64)
Change	New CPU mode/bitness
Running new applications	Native
Running old applications	Thunking
Running old drivers	Not supported
Porting applications	High level of source compatibility

Macintosh on 68K to Macintosh on PowerPC
Change	New CPU
Running new applications	Thunking
Running old applications	Paravirtualized old OS in emulator
Running old drivers	Paravirtualized old OS in emulator
Porting applications	High level of source compatibility

Classic Mac OS to Mac OS X
Change	New OS
Running new applications	Native
Running old applications	Classic: Virtual machine with old OS Carbon: Intermediate API for both systems
Running old drivers	Virtual machine with old OS
Porting applications	Intermediate API for both systems

Mac OS X on PowerPC to Mac OS X on Intel
Change	New CPU
Running new applications	Native
Running old applications	User mode emulator
Running old drivers	Not supported
Porting applications	High level of source compatibility

Mac OS X (32 bit) to Mac OS X (64 bit)
Change	New CPU mode/bitness
Running new applications	Thunking
Running old applications	Native
Running old drivers	Native
Porting applications	Carbon: Not supported Cocoa: High level of source compatibility

AmigaOS on 68K to AmigaOS on PowerPC (3.x)
Change	New CPU
Running new applications	Thunking (new CPU)
Running old applications	Native (old CPU)
Running old drivers	Native
Porting applications	High level of source compatibility

Palm OS on 68K to Palm OS on ARM
Change	New CPU
Running new applications	Not supported (PNOlet for subroutines)
Running old applications	User mode emulation
Running old drivers	Not supported
Porting applications	Not supported (PNOlet for subroutines)

New CPU mode/bitness
OS	Old mode	New mode	Thunking direction	Thunking level
Windows	16 bit	32 bit	new to old	library
Windows NT	32 bit	64 bit	old to new	kernel
Mac OS X	32 bit	64 bit	new to old	kernel

New CPU
OS	Old CPU	New CPU	Running old apps	Thunking level
CP/M, DOS	8080/Z80	8086	Developer has to recompile	-
Macintosh	68K	PowerPC	Run OS and app in emulation	-
Mac OS X	PowerPC	i386	User mode emulation	kernel
Amiga	68K	PowerPC	Dual-CPU thunking	library
Palm	68K	ARM	User mode emulation	library

New OS
Old OS	New OS	Running old apps
CP/M	DOS	Compatible API
DOS	Windows	Virtual machine with old OS
DOS	Windows NT	API emulation
Windows 9X	Windows NT	Compatible API
Mac OS	Mac OS X	Classic: Virtual machine with old OS Carbon: Compatible API