Comparative C64 ROM Disassembly Study Guide

The Commodore 64 ROM has been subject to immense reverse engineering. Many commented disassemblies were published over the decades, scattered over different media such as books, magazines, disks, and later, the internet – and there are even some commentaries that apply to the C64 ROM, but were written with other systems in mind that shared Microsoft’s BASIC interpreter.

In the past weeks, I have collected and published several of these comentaries in a unified format:

Wouldn’t it be nice to see the comments of all these sources at the same time when looking up code in the C64 ROM?

At pagetable.com/c64rom, you can now see a cross-referenced HTML of the disassembled C64 ROM, with four commentaries side-by-side – the Comparative C64 ROM Disassembly Study Guide:

If you can’t fit all columns on your screen, try reducing the text size in your browser.

The raw txt files with the commentaries as well as the script to combine them are maintained at github.com/mist64/c64rom. Improvements welcome.

And, as mentioned previously, there are many more commentaries in existence, if you want to help me convert them into the canonical format, send me an email.

Fully Commented Commodore 64 BASIC ROM Disassembly – based on Microsoft’s Source

On my quest of collecting as many commentaries on the Commodore 64 ROM at pagetable.com/c64rom, we have gathered Lee Davison’s excellent commentary, the German de facto standard by Data Becker, and an adaptation of Bob Sander-Cederlof’s Apple II ROM commentary, all in the same cross-referenced HTML format.

Now that Microsoft’s original source of MOS 6502 BASIC is available, I’ve added it as the fourth commented disassembly, with the standard disassembly on the left, and the original source, both assembly and comments, lined up correctly on the right:

As always, the HTML version is available at pagetable.com/c64rom, while the raw txt files are maintained at github.com/mist64/c64rom.

While this may be the best set of comments for the BASIC part, we’re not done yet! There are many more (either direct or indirect) commentaries on the C64 ROM in existence:

Please contribute to our collection by helping convert one or more of these sources into the common format! Send me an email if you are interested!

Microsoft BASIC for 6502 Original Source Code [1978]

This is the original 1978 source code of Microsoft BASIC for 6502 with all original comments, documentation and easter eggs:

M6502.MAC (1978-07-27, 6955 lines, 161,685 bytes)

This is currently the oldest publicly available piece of source written by Bill Gates.

Language

Like the 8080 version, the 6502 version was developed on a PDP-10, using the MACRO-10 assembler. A set of macros developed by Paul Allen allowed MACRO-10 to understand and translate 6502 assembly, albeit in a modified format to fit the syntax of macros, for example:

MOS 6502 MACRO-10
LDA #0 LDAI 0
LDA (ADDR),Y LDADY ADDR

MACRO-10 did not support hex numbers, which is why most numbers are in decimal format. In the floating point code, all numbers are octal. The RADIX statement switches between the two. Octal can also be forced with a ^O prefix.

Conditional translation is done using the IFE and IFN statements, which test whether the argument is zero. The following only adds the string to the binary if REALIO is equal to 4:

IFE     REALIO-4,<DT"APPLE BASIC V1.1">

Macros

The source defines many macros that make development easier. There are some examples:

Macro Definition Comment
SYNCHK (Q) LDAI <Q>
JSR SYNCHR
Get the next character and make sure it’s Q, otherwise SYNTAX ERROR. This pattern is used a lot.
LDWD (WD) LDA WD
LDY <WD>+1
Most 16 bit constants are loaded into A/Y with this macro, but macros for A/X and X/Y also exist.
LDWDI (WD) LDAI <<WD>&^O377>
LDYI <<WD>/^O400>
This loads an immediate constant into A/Y.
PSHWD (WD) LDA <WD>+1
PHA
LDA WD
PHA
This pushes a 16 bit value from memory (absolute or zero page) onto the stack.
JEQ (WD) BNE .+5
JMP WD
A compact way to express out-of-bounds branches. Macros exist for all branches.
SKIP2 XWD ^O1000,^O054 This emits a byte value of 0x2C (BIT absolute), which skips the next instruction. (The ^O1000 part wraps the byte in a PDP-10 instruction – see below.)

Configurations

The BASIC source supports several compile-time configuration options:

Name Comment Description
INTPRC INTEGER ARRAYS
ADDPRC FOR ADDITIONAL PRECISION 40 bit (9 digit) vs 32 bit (7 digit) float
LNGERR LONG ERROR MESSAGES Error message strings instead of two-character codes
TIME CAPABILITY TO SET AND READ A CLK TI and TI$ support
EXTIO EXTERNAL I/O PRINT#, INPUT#, CMD, SYS (!), OPEN and CLOSE support
DISKO SAVE AND LOAD COMMANDS LOAD, SAVE (and on Commodore: VERIFY) support
NULCMD FOR THE "NULL" COMMAND NULL support, a command to configure the number of NUL characters to print to the terminal after each line break
GETCMD GET support
RORSW If 1, the ROR instruction is not used
ROMSW TELLS IF THIS IS ON ROM The RAM version can optionally jetison the SIN, COS, TAN and ATN commands at startup
CLMWID Column width for TAB()
LONGI LONG INITIALIZATION SWITCH
STKEND The top of stack at startup
BUFPAG Page of the input buffer; if 0, the buffer uses parts of the zero page
BUFLEN INPUT BUFFER SIZE
LINLEN TERMINAL LINE LENGTH
ROMLOC ADDRESS OF START OF PURE SEGMENT
KIMROM KIM-specific smaller config

Targets

The constant REALIO is used to configure what computer system to generate the binary for. It has one of the following values:

Value Comment Banner Machine
0 PDP-10 SIMULATING 6502 SIMULATED BASIC FOR THE 6502 V1.1 Paul Allen’s Simulator on PDP-10
1 MOS TECH,KIM KIM BASIC V1.1 MOS KIM-1
2 OSI OSI 6502 BASIC VERSION 1.1 OSI Model 500
3 COMMODORE ### COMMODORE BASIC ### Commodore PET 2001
4 APPLE APPLE BASIC V1.1 Apple II
5 STM STM BASIC V1.1 (unreleased)

All versions except Commodore also print “COPYRIGHT 1978 MICROSOFT” in a new line.

The target defines the setting of the configuration constants, but some code is also conditionally compiled depending on a specific target.

What is interesting is that initially it was Microsoft adapting their source for the different computers, instead of giving source to the different vendors and having them adapt it. Features like file I/O and time support seem to have been specifically developed for Commodore, for example. Later, the computer companies would get the source from Microsoft and develop themselves – source code of the Apple and Commodore derivatives is available; they both contain Microsoft comments.

By the way, the numbering of these targets probably indicated in which order Microsoft signed contracts with computer manufacturers. MOS was first (for the KIM), then OSI, then Commodore/MOS again (this time for the PET), then Apple.

The PDP-10 Target

Paul Allen’s additional macros for 6502 development made the MACRO-10 assembler output one 36 bit PDP-10 instruction word for every 6502 byte. When targeting a real 6502 machine, the 6502 binary could be created by simply extracting one byte from every PDP-10 word.

In the case of targeting the simulator, the code created by the assembler could just be run without modification, since every emitted PDP-10 instruction was constructed so that it would trap – the linked-in simulator would then extract the 6502 opcode from the instruction and emulate the 6502 behavior.

While this trick was mostly abstracted by the (unreleased) macro package, its workings can be seen in a few cases in the BASIC source. Here, it defines SKIP1 and SKIP2. Instead of just emitting 0×24 or 0x2C, respectively, it combines it with the octal value of 01000 to make it a PDP-10 instruction that traps:

DEFINE  SKIP1,  <XWD ^O1000,^O044>      ;BIT ZERO PAGE TRICK.
DEFINE  SKIP2,  <XWD ^O1000,^O054>      ;BIT ABS TRICK.

In the initialization code, it writes a JMP instruction into RAM. On the simulator, it has to patch up the opcode of JMP (0x4C, decimal 76) to be the correct PDP-10 instruction:

        LDAI    76              ;JMP INSTRUCTION.
IFE     REALIO,<HRLI 1,^O1000>  ;MAKE AN INST.

With this information, we can reconstruct what the set of 6502 macros, which is not part of this source, probably looked like. Here is LDAI (LDA immediate):

DEFINE  LDAI    (Q),<
        XWD ^O1000,^O251        ;EMIT OPCODE
        XWD ^O1000,<Q>          ;EMIT OPERAND
>

You can also see native TJSR PDP-10 assembly instructions for character I/O:

IFE     REALIO,<
        TJSR    INSIM##>        ;GET A CHARACTER FROM SIMULATOR
IFE     REALIO,<
        TJSR    OUTSIM##>       ;CALL SIMULATOR OUTPUT ROUTINE

The DDT command, which breaks into the PDP-10′s DDT debugger, only exists in this config:

IFE     REALIO,<
DDT:    PLA                     ;GET RID OF NEWSTT RETURN.
        PLA
        HRRZ    14,.JBDDT##
        JRST    0(14)>

The KIM and OSI Targets

The KIM target is meant for the MOS KIM-1 and Ohio Scientific OSI Model 500 single-board computers. These are the first ports to specific computers, and also the cleanest, i.e. except for the character I/O interface and the very simple LOAD/SAVE implementation for the KIM, there is nothing specific about these targets.

The Commodore Target

The Commodore target is meant for the Commodore PET 2001. It includes LOAD/SAVE/VERIFY (the commands jump directly to outside “KERNAL” ROM code), the I/O commands (SYS, PRINT#, OPEN etc.), the GET command and the π, ST, TI and TI$ symbols. CLEAR is renamed to CLR, "OK" is renamed to "READY.", the BEL character is not printed, and character I/O code behaves differently to account for the more featureful screen editor of the PET.

Oh, and the Commodore version of course includes the Bill Gates WAIT 6502,1 easter egg! This is the WAIT instruction:

; THE WAIT LOCATION,MASK1,MASK2 STATEMENT WAITS UNTIL THE CONTENTS
; OF LOCATION IS NONZERO WHEN XORED WITH MASK2
; AND THEN ANDED WITH MASK1. IF MASK2 IS NOT PRESENT, IT
; IS ASSUMED TO BE ZERO.

FNWAIT: JSR     GETNUM
        STX     ANDMSK
        LDXI    0
        JSR     CHRGOT
        BEQ     ZSTORDO
        JSR     COMBYT          ;GET MASK2.
STORDO: STX     EORMSK
        LDYI    0
WAITER: LDADY   POKER
        EOR     EORMSK
        AND     ANDMSK
        BEQ     WAITER
ZERRTS: RTS                     ;GOT A NONZERO.

Note how the BEQ instruction references ZSTORDO, not STORDO – execution sneaks out of this function here.

Well, on non-Commodore machines, ZSTORDO is assigned to be the same as STORDO, so everything is fine:

IFN     REALIO-3,<ZSTORDO=STORDO>

But on Commodore, we have this code hidden near the top of the floating point math package – close enough so the BEQ can reach it, but inside code that is least likely to get touched:

IFE     REALIO-3,<
ZSTORD:!        LDA     POKER
        CMPI    146
        BNE     STORDO
        LDA     POKER+1
        SBCI    31
        BNE     STORDO
        STA     POKER
        TAY
        LDAI    200
        STA     POKER+1
MRCHKR: LDXI    12
IF1,<
MRCHR:  LDA     60000,X,>
IF2,<
MRCHR:  LDA     SINCON+36,X,>
        ANDI    77
        STADY   POKER
        INY
        BNE     PKINC
        INC     POKER+1
PKINC:  DEX
        BNE     MRCHR
        DEC     ANDMSK
        BNE     MRCHKR
        RTS
IF2,<PURGE ZSTORD>>

(IF1 and IF2 are true on the first and the second assembler pass, respectively, so the conditional there is to hint to the assembler in the first pass that SINCON+36 is not a zero page address. Also note that all numbers here are octal, since this code is in the floating point package.)

First of all, the final line here removes ZSTORD from the list of symbols after the second pass, so that Commodore would not notice it in a printout of all symbols – very smart!

As has been discussed before, this code writes the string “MICROSOFT!” into the PET’s screen RAM if the argument to WAIT is “6502″. The encoded string is hidden as two extra 40 bit floating point numbers appended to the coefficients used by the SIN function:

IFN     ADDPRC,<
SINCON: 5               ;DEGREE-1.
        204     ; -14.381383816
        346
        032
        055
        033
        206     ; 42.07777095
        050
        007
        373
        370
        207     ; -76.704133676
        231
        150
        211
        001
        207     ; 81.605223690
        043
        065
        337
        341
        206     ; -41.34170209
        245
        135
        347
        050
        203     ; 6.2831853070
        111
        017
        332
        242
        241     ; 7.2362932E7
        124
        106
        217
        23
        217     ; 73276.2515
        122
        103
        211
        315>

These last ten bytes, nicely disguised as octal values of floating point constants, spell out “MICROSOFT!” backwards after clearing the upper two bits. What’s interesting is that the floating point values next to them are actually incorrect: They should be 7.12278788E9 and 26913.7691 instead.

Also note that these constants are not conditionally assembled! All versions built since the Commodore easter egg was introduced also contained these 10 bytes – including BASIC for the Motorola 6800!

The Apple Target

The Apple target is meant for the Apple II, and contains no customizations other than some changes around I/O handling (which calls into the monitor ROM). Note that this is not yet the “AppleSoft” version of BASIC, which was a more customized version modified by Apple later.

The STM Target

“STM” most likely stands for “Semi-Tech Microelectronics” – a company that never shipped a 6502-based computer. Their first machine was the “Pied Piper”, a Z80-based system, and they later made a PC clone. It seems they had a 6502-based computer in development that never shipped – or at least they were considering making one, and Microsoft added the target; this target doesn’t actually change any of the defaults.

Organization of the Source

The source uses the PAGE and SUBTTL keywords for organization. Here are the headings:

SUBTTL  SWITCHES,MACROS.
SUBTTL  INTRODUCTION AND COMPILATION PARAMETERS.
SUBTTL  SOME EXPLANATION.
SUBTTL  PAGE ZERO.
SUBTTL  RAM CODE.
SUBTTL  DISPATCH TABLES, RESERVED WORDS, AND ERROR TEXTS.
SUBTTL  GENERAL STORAGE MANAGEMENT ROUTINES.
SUBTTL  ERROR HANDLER, READY, TERMINAL INPUT, COMPACTIFY, NEW, REINIT.
SUBTTL  THE "LIST" COMMAND.
SUBTTL  THE "FOR" STATEMENT.
SUBTTL  NEW STATEMENT FETCHER.
SUBTTL  RESTORE,STOP,END,CONTINUE,NULL,CLEAR.
SUBTTL  LOAD AND SAVE SUBROUTINES.
SUBTTL  RUN,GOTO,GOSUB,RETURN.
SUBTTL  "IF ... THEN" CODE.
SUBTTL  "ON ... GO TO ..." CODE.
SUBTTL  LINGET -- READ A LINE NUMBER INTO LINNUM
SUBTTL  "LET" CODE.
SUBTTL  PRINT CODE.
SUBTTL  INPUT AND READ CODE.
SUBTTL  THE NEXT CODE IS THE "NEXT CODE"
SUBTTL  DIMENSION AND VARIABLE SEARCHING.
SUBTTL  MULTIPLE DIMENSION CODE.
SUBTTL  INTEGER ARITHMETIC ROUTINES.
SUBTTL  FRE FUNCTION AND INTEGER TO FLOATING ROUTINES.
SUBTTL  SIMPLE-USER-DEFINED-FUNCTION CODE.
SUBTTL  STRING FUNCTIONS.
SUBTTL  PEEK, POKE, AND FNWAIT.
SUBTTL  FLOATING POINT ADDITION AND SUBTRACTION.
SUBTTL  NATURAL LOG FUNCTION.
SUBTTL  FLOATING MULTIPLICATION AND DIVISION.
SUBTTL  FLOATING POINT MOVEMENT ROUTINES.
SUBTTL  SIGN, SGN, FLOAT, NEG, ABS.
SUBTTL  COMPARE TWO NUMBERS.
SUBTTL  GREATEST INTEGER FUNCTION.
SUBTTL  FLOATING POINT INPUT ROUTINE.
SUBTTL  FLOATING POINT OUTPUT ROUTINE.
SUBTTL  EXPONENTIATION AND SQUARE ROOT FUNCTION.
SUBTTL  EXPONENTIATION FUNCTION.
SUBTTL  POLYNOMIAL EVALUATOR AND THE RANDOM NUMBER GENERATOR.
SUBTTL  SINE, COSINE AND TANGENT FUNCTIONS.
SUBTTL  ARCTANGENT FUNCTION.
SUBTTL  SYSTEM INITIALIZATION CODE.

Paul Allen vs. Bill Gates

The source of the 8080 version states:

PAUL ALLEN WROTE THE NON-RUNTIME STUFF.
BILL GATES WROTE THE RUNTIME STUFF.
MONTE DAVIDOFF WROTE THE MATH PACKAGE.

People have since wondered what runtime vs. non-runtime meant, especially since Paul Allen’s recent debate on whether the company’s ownership was faily split.

The BASIC for 6502 source sheds some light on this:

NON-RUNTIME STUFF
        THE CODE TO INPUT A LINE, CRUNCH IT, GIVE ERRORS,
        FIND A SPECIFIC LINE IN THE PROGRAM,
        PERFORM A "NEW", "CLEAR", AND "LIST" ARE
        ALL IN THIS AREA. [...]

So by “runtime” they just literally mean “at run time”: all code that is active when the program runs, as opposed to non-runtime, which is all code that assists editing the program.

By this understanding, we can assume this:

  • Paul Allen wrote the macro package for the MACRO-10 assembler, the 6502 simulator, the tokenizer, the detokenizer, as well as finding, inserting and deleting BASIC lines.
  • Bill Gates implemented all BASIC statements, functions, operators, expression evaluation, stack management for FOR and GOSUB, the memory manager, as well as the array and string library.
  • Monte Davidoff wrote the floating point math package.

Version and Date

The last entry in the change log has a date of 1978-07-27. Both the comment in the first line of the file and the message printed at startup call it version 1.1.

What does this say about the version of the source? Is it the last version? Let’s look at the last bug fix and compare which BASIC binaries contain this fix, and let’s see whether there are fixes in BASIC binaries that are not in the source.

I have previously compared binaries of derivatives of BASIC for 6502 and compiled the information at github.com/mist64/msbasic. The last entry in the log of this source is about a bug that failed to correctly invalidate a pointer in the RETURN statement. According to my analysis of BASIC 6502 versions, this is fixed in the BASIC binaries for AIM-65, SYM-1, Commodore v2, KBD BASIC and MicroTAN, i.e. on everything my previous analysis calls CONFIG_2A and higher.

The same analysis also came to the conclusion that there were two successors, CONFIG_2B and CONFIG_2C. At least the two CONFIG_2B fixes exist in two BASIC binaries: KBD BASIC and MicroTAN, but they don’t exist in this source. It’s very unlikely that both these bugs (and only these!) got fixed by the two computer manufacturers independently, so it’s safe to assume that this source is not the final version – but pretty close to it!

Interesting Finds

  • This code is comparing a keyboard input character to the BEL code. Bob Albrecht is a computer educator that “was instrumental in helping bring about a public-domain version of Basic (called Tiny Basic) for early microcomputers.”.
    CMPI    7               ;IS IT BOB ALBRECHT RINGING THE BELL
                            ;FOR SCHOOL KIDS?
    
  • External documentation usually calls the conversion of ASCII BASIC text into the compressed format “tokenizing”. The source calls this “crunching”.
  • Microsoft is still spelled “Micro-Soft”.
  • Apparently the multiplication function could use some performance improvements:
    		
    BNE     MLTPL2          ;SLOW AS A TURTLE !
    
  • The NEW command is actually called SCRATCH in labels and comments – maybe other BASIC dialects called it that, and they decided to rename it to NEW later?
  • The math package documentation says:
    MATH PACKAGE
            THE MATH PACKAGE CONTAINS FLOATING INPUT (FIN),
            FLOATING OUTPUT (FOUT), FLOATING COMPARE (FCOMP)
            ... AND ALL THE NUMERIC OPERATORS AND FUNCTIONS.
            THE FORMATS, CONVENTIONS AND ENTRY POINTS ARE ALL
            DESCRIBED IN THE MATH PACKAGE ITSELF.
    

    Commodore’s derived source changes this to:

    ; MATH PACKAGE
    ;       THE MATH PACKAGE CONTAINS FLOATING INPUT FIN, OUTPUT
    ;       FOUT, COMPARE FCOMP...AND ALL THE NUMERIC OPERATORS
    ;       AND FUNCTIONS.  THE FORMATS, CONVENTIONS AND ENTRY
    ;       POINTS ARE ALL DESCRIBED IN THE MATH PACKAGE ITSELF.
    ;       (HA,HA...)
    
  • CHRGET is a central piece of BASIC for 6502. Here it is in its entirety:
    ; THIS CODE GETS CHANGED THROUGHOUT EXECUTION.
    ; IT IS MADE TO BE FAST THIS WAY.
    ; ALSO, [X] AND [Y] ARE NOT DISTURBED
    ;
    ; "CHRGET" USING [TXTPTR] AS THE CURRENT TEXT PNTR
    ; FETCHES A NEW CHARACTER INTO ACCA AFTER INCREMENTING [TXTPTR]
    ; AND SETS CONDITION CODES ACCORDING TO WHAT'S IN ACCA.
    ;       NOT C=  NUMERIC   ("0" THRU "9")
    ;       Z=      ":" OR END-OF-LINE (A NULL)
    ;
    ; [ACCA] = NEW CHAR.
    ; [TXTPTR]=[TXTPTR]+1
    ;
    ; THE FOLLOWING EXISTS IN ROM IF ROM EXISTS AND IS LOADED
    ; DOWN HERE BY INIT. OTHERWISE IT IS JUST LOADED INTO THIS
    ; RAM LIKE ALL THE REST OF RAM IS LOADED.
    ;
    CHRGET: INC     CHRGET+7        ;INCREMENT THE WHOLE TXTPTR.
            BNE     CHRGOT
            INC     CHRGET+8
    CHRGOT: LDA     60000           ;A LOAD WITH AN EXT ADDR.
    TXTPTR= CHRGOT+1
            CMPI    " "             ;SKIP SPACES.
            BEQ     CHRGET
    QNUM:   CMPI    ":"             ;IS IT A ":"?
            BCS     CHRRTS          ;IT IS .GE. ":"
            SEC
            SBCI    "0"             ;ALL CHARS .GT. "9" HAVE RET'D SO
            SEC
            SBCI    256-"0"         ;SEE IF NUMERIC.
                                    ;TURN CARRY ON IF NUMERIC.
                                    ;ALSO, SETZ IF NULL.
    CHRRTS: RTS                     ;RETURN TO CALLER.
    

    Did you ever wonder why all versions have $EA60 encoded into the LDA instruction that later gets overwritten? Because it’s 60000 decimal. That’s why! The source actually uses 60000 as a placeholder for 16 bit values in several places.

  • The handling of π, ST, TI and TI$ (all Commodore-specific) looks wonky: Instead of making them tokens, they are special cased in several places. I always assumed it was Commodore adding this without understanding (or wanting to disrupt) the existing code, but it was Microsoft adding these features. Maybe they were added by someone other than the original developers?

Origin of the File

The source was posted on the Korean-language blog 6502.tistory.com without further comment, in a marked-up format:

================================================================================================
FILE: "david mac g5 b:m6502.asm"
================================================================================================

000001  TITLE   BASIC M6502 8K VER 1.1 BY MICRO-SOFT
[...]
006955          END     $Z+START

End of File -- Lines: 6955 Characters: 154740

SUMMARY:

  Total number of files : 1
  Total file lines      : 6955
  Total file characters : 154740

This formatting was created by an unpublished tool by David T. Craig, who published a lot of Apple-related soure code (Apple II, Apple III, Lisa) in this format in as early as 1993, first anonymously, later with his name).

The filename “david mac g5 b:m6502.asm” (disk name “david mac g5 b”, file name “m6502.asm”, since it was a classic Mac OS tool) confirms David Craig’s involvement, and it means the line numbers were added no earlier than 2003.

Given all this, it is safe to assume the file with the Microsoft BASIC for 6502 source originated at Apple, and was given to David Craig together with the other source be published.

The version I posted is a reconstruction of the original file, with the header, the footer and the line numbers removed, and the spaces converted back into tabs. I chose the name “M6502.MAC” to be consistent with the MACRO-10 file extension used by the Microsoft BASIC for 8080 sources.

Using the OS X 10.10 Hypervisor Framework: A Simple DOS Emulator

Since Version 10.10 (Yosemite), OS X contains Hypervisor.framework, which provides a thin user mode abstraction of the Intel VT features. It enables apps to use virtualization without the need of a kernel extension (KEXT) – which makes them compatible with the OS X App Store guidelines.

The idea is that the OS takes care of memory management (including nested paging) as well as scheduling virtual CPUs like normal threads. All we have to do is create a virtual CPU (or more!), set up all its state, assign it some memory, and run it… and then handle all “VM exits” – Intel lingo for hypervisor traps.

There is no real documentation, but the headers contain a decent amount of information. Here are some declarations from Hypervisor/hv.h:

/*!

 * @function   hv_vm_create

 * @abstract   Creates a VM instance for the current task

 * @param      flags  RESERVED

 * @result     0 on success or error code

 */

extern hv_return_t hv_vm_create(hv_vm_options_t flags) __HV_10_10;

/*!

 * @function   hv_vm_map

 * @abstract   Maps a region in the virtual address space of the current task

 *             into the guest physical address space of the VM

 * @param      uva    Page aligned virtual address in the current task

 * @param      gpa    Page aligned address in the guest physical address space

 * @param      size   Size in bytes of the region to be mapped

 * @param      flags  READ, WRITE and EXECUTE permissions of the region

 * @result     0 on success or error code

 */

extern hv_return_t hv_vm_map(hv_uvaddr_t uva, hv_gpaddr_t gpa, size_t size,

hv_memory_flags_t flags) __HV_10_10;

/*!

 * @function   hv_vcpu_create

 * @abstract   Creates a vCPU instance for the current thread

 * @param      vcpu   Pointer to the vCPU ID (written on success)

 * @param      flags  RESERVED

 * @result     0 on success or error code

 */

extern hv_return_t hv_vcpu_create(hv_vcpuid_t *vcpu,

hv_vcpu_options_t flags) __HV_10_10;

/*!

 * @function   hv_vcpu_run

 * @abstract   Executes a vCPU

 * @param      vcpu  vCPU ID

 * @result     0 on success or error code

 * @discussion

 *             Call blocks until the next VMEXIT of the vCPU

 *

 *             Must be called by the owning thread

 */

extern hv_return_t hv_vcpu_run(hv_vcpuid_t vcpu) __HV_10_10;

So let’s create a virtual machine that runs simple DOS applications in 16 bit real mode, and trap all “int” DOS system calls – similar to DOSBox.

First, we need to create a VM:

hv_vm_create(HV_VM_DEFAULT);

This creates a VM for the current Mach task (i.e. UNIX process). It’s implicit, so it doesn’t return anything. Then we allocate some memory and assign it to the VM:

#define VM_MEM_SIZE (1 * 1024 * 1024)
void *vm_mem = valloc(VM_MEM_SIZE);
hv_vm_map(vm_mem, 0, VM_MEM_SIZE, HV_MEMORY_READ | 
                                  HV_MEMORY_WRITE | 
                                  HV_MEMORY_EXEC);

And we need to create a virtual CPU:

hv_vcpuid_t vcpu;
hv_vcpu_create(&vcpu, HV_VCPU_DEFAULT);

Now comes the annoying part: Set up the CPU state. If the state is illegal or inconsistent, the CPU will refuse to run. You will need to refer to the Intel Manual 3C for all the context. Luckily, most virtual machines start from 16 bit real mode, and mode changes will be done by the boot loader or operating system inside the VM, so you won’t have to worry about setting up any other state than real mode state. Real mode state setup looks something like this:

hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CS_SELECTOR, 0);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CS_LIMIT, 0xffff);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CS_ACCESS_RIGHTS, 0x9b);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CS_BASE, 0);

hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_DS_SELECTOR, 0);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_DS_LIMIT, 0xffff);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_DS_ACCESS_RIGHTS, 0x93);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_DS_BASE, 0);

[...]

hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CR0, 0x20);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CR3, 0x0);
hv_vmx_vcpu_write_vmcs(vcpu, VMCS_GUEST_CR4, 0x2000);

After that, we should populate RAM with the code we want to execute:

FILE *f = fopen(argv[1], "r");
fread((char *)vm_mem + 0x100, 1, 64 * 1024, f);
fclose(f);

…and assign the GPRs the proper initial state – including the instruction pointer, which will point to the code:

hv_vcpu_write_register(vcpu, HV_X86_RIP, 0x100);
hv_vcpu_write_register(vcpu, HV_X86_RFLAGS, 0x2);
hv_vcpu_write_register(vcpu, HV_X86_RSP, 0x0);

The virtual CPU is fully set up, we can now run it!

hv_vcpu_run(vcpu);

This call runs the virtual CPU (while blocking the calling thread) until its time slice expires or a “VM exit” happens. A VM exit is a hypervisor-class exception, i.e. an event in the VM that the hypervisor wants to trap. We can trap events like exceptions, certain privileged instructions (CPUID, HLT, RDTSC, RDMSR, …) and control register (CR0, CR2, CR3, CR4, …) accesses.

After hv_vcpu_run() returns, we need to read the exit reason and act upon it, and run the virtual CPU again. Here is a minimal loop to handle VM exits:

for (;;) {
	hv_vcpu_run(vcpu);

	uint64_t exit_reason = hv_vmx_vcpu_read_vmcs(vcpu, VMCS_EXIT_REASON);

	switch (exit_reason) {
		case EXIT_REASON_EXCEPTION:
			[...]
			break;
		case EXIT_REASON_EXT_INTR:
		case EXIT_REASON_EPT_FAULT:
			break;
		default:
			exit(1);
	}
}

EXIT_REASON_EXT_INTR is caused by host interrupts (usually it means that the time slice is up), so we will just ignore it. EXIT_REASON_EPT_FAULT happens every time the guest accesses a page for the first time, or when the guest accesses an unmapped page – this way we can emulate MMIO. In our case, we can also ignore those.

For emulating DOS, we are catching EXIT_REASON_EXCEPTION, which is caused by the int instruction (if caught). We can get the number of the interrupt from the virtual CPU state without decoding instructions:

uint8_t interrupt_number = hv_vmx_vcpu_read_vmcs(vcpu, VMCS_IDT_VECTORING_INFO) & 0xFF;

…and emulate the system call. We can read and write GPRs using the hv_vcpu_read_register() and hv_vcpu_write_register() calls.

hvdos – a simple DOS Emulator for OS X

The full source of hvdos, a simple DOS emulator using the OS X Hypervisor framework, is available at github.com/mist64/hvdos.

It contains an adapted version of the libcpu DOS system call library and manages to run (parts of) some .COM files. A good demo is the pkunzjr.com ZIP decompression tool.

Creating your own Hypervisor

hvdos can serve as a template for your own Hypervisor.framework experiments. It contains wrapper functions for error handling, a header that defines all Intel VT constants (taken from FreeBSD), complete 16 bit real mode initialization, as well as a few helper functions to set up the fields VMCS_PIN_BASED_CTLS, VMCS_PRI_PROC_BASED_CTLS, VMCS_SEC_PROC_BASED_CTLS and VMCS_ENTRY_CTLS properly. These are needed to define, among other things, which events cause VM exits.

You can easily add more CPUs by creating one POSIX thread per virtual CPU. For every thread, you create a virtual CPU and run a VM exit main loop.

You can for example start writing an IBM PC emulator by running Bochs BIOS and trapping I/O accesses, or running MS-DOS without BIOS by trapping BIOS int calls.

Or you could bridge an existing open source solution (QEMU, QEMU+KVM, VirtualBox, DOSBox, …) to use Hypervisor.framework…

Fully Commented Commodore 64 BASIC ROM Disassembly – based on Applesoft!

In our series about C64 ROM commentaries (English version by Lee Davison, German version by Data Becker), I’m now presenting a most unusual C64 ROM commentary – based on a commented disassembly of the Apple II ROM.

S-C DocuMentor for Applesoft” is a commented disassembly of the BASIC ROM of the Apple II computer. Like Commodore BASIC, “Applesoft” BASIC is based on Microsoft BASIC for 6502, but on an older revision. Since the two BASIC interpreters are almost the same instruction for instruction (modulo some command extensions on both sides), the commentary translated over very nicely.

The cross-referenced HTML version of the “S-C C64 BASIC Disassembly” is available here at pagetable.com/c64rom.

The raw txt files of all commentaries are maintained at github.com/mist64/c64rom. Fixes and additions happily accepted!

Fully Commented Commodore 64 ROM Disassembly (English)

After last week’s German C64 ROM disassembly from the “64 intern” book, I have now also converted Lee Davison’s commented disassembly into the same format.

The cross-referenced HTML version is available here at pagetable.com/c64rom.

The raw txt files of both the German and the English commented disassemblies are maintained at github.com/mist64/c64rom. The two files seem to have been independently developed, which gives us the opportunity to compare, find mistakes, and merge missing information.

I will happily accept additions and corrections to either file – let’s create the one true source of C64 ROM information!

Fully Commented Commodore 64 ROM Disassembly (German)

Whenever I need to look up some code in the ROM of the Commodore 64, I have the choice of the commented disassembly by Marko Mäkelä, the one by Ninja/The Dreams, or the one by Lee Davison – or I can just use my paper copy of “Das neue Commodore-64-intern-Buch“, an excellent line-by-line commentary in German.

That’s why I scanned, OCRed, cleaned up and cross-referenced it.

The raw txt file is maintained at github.com/mist64/c64rom. Corrections, additions and translations welcome.

The cross-referenced HTML version is available here at pagetable.com/c64rom.

Wikileaks Movie “The Fifth Estate” pirated my “Xbox Hacking” Slides

Xbox hacking has made it to the silver screen, and Felix Domke and me (Michael Steil) are movie stars! …and so are at least 14 of my presentation slides!

This a picture from the Julian Assange and Wikileaks movie The Fifth Estate (2013), starring Benedict Cumberbatch and Daniel Brühl, directed by Bill Condon:

“Linux is Inevitable”? Sounds like something I would say. In fact, looks like a slide from my presentation at the 24th Chaos Communication Congress in Berlin in December 2007:

Coincidence? Let’s get some context. The scene in the movie is indeed set at the 24th Chaos Communication Congress (24C3), where Julian Assange (Cumberbatch) presents his vision about Wikileaks in the break between two talks. From the movie’s screenplay (ironically leaked by Wikileaks):

                      OTTO 
           I'm afraid the small conference
           rooms are all booked.

                       DANIEL
                 (off the schedule)
           What about the auditorium? It's
           empty 'til the X-Box Security talk.

At the 24C3, Felix Domke and me indeed presented Why Silicon-Based Security is still that hard: Deconstructing Xbox 360 Security on day 2 at 16:00 in the main auditorium. (In reality, Julian Assange did not present in the main auditorium, but in a workshop area.)

 To one side of the stage, THE NEXT SPEAKER sets up a few
 deconstructed X-BOX 360s beside a CORKBOARD covered with
 exhibits for his talk: 'Deconstructing Xbox 360 Security.'


To be clear: These are pictures from the movie, not actual pictures from the conference. They really deconstructed an Xbox 360 for the movie!

 Daniel, also on stage, watches as Julian pulls out a WAD of
 TWINE, moves to the corkboard.

 The X-Box guy looks up, CONCERNED, as Julian wraps the twine
 around a PUSH PIN holding up part of the X-Box exhibit,
 STRINGING THE TWINE to another pin holding up another part.

 Julian POINTS to the two pins and the twine. ILLUSTRATING.

                       JULIAN 
                 (CONTINUED) 
           Two people and a secret. The
           beginning of any conspiracy, of all
           corruption. As it grows...

 Daniel watches, RIVETED, as Julian STRETCHES the twine to
 another pin. And another. And another...

                       JULIAN 
                 (CONTINUED) 
           More people... and more secrets.

 CLOSE ON THE BOARD as Julian RAPIDLY RUNS TWINE AROUND PINS,
 ensnaring more of the exhibit in his web. It's MESMERIZING.

                       JULIAN 
                 (CONTINUED) 
           But. If we can find one moral man,
           one whistleblower...

 Julian focuses on a PIN at the CENTER of his web of twine.
 CLOSE ON THE PIN as he LOOPS MORE TWINE AROUND IT...

                       JULIAN 
                 (CONTINUED) 
           Someone willing to expose these
           secrets --

 Julian PULLS THE TWINE TAUT... then YANKS THE CENTER PIN...
 PULLING THE TWINE SO THAT ALL THE PINS POP OUT OF THE BOARD.
 THE FLYERS FALL, SCATTERING ALL AROUND JULIAN.

                       JULIAN 
                 (CONTINUED) 
           That man... can topple the most
           repressive of regimes.

 Julian pins a WIKILEAKS FLYER on the NOW EMPTY CORKBOARD.

                       X BOX GUY 
           Was zur hoelle!

                       JULIAN 
           And there's the problem.
           Retribution.

                       X BOX GUY 
           Otto, my talk is in ten minutes.

 The X-Box guy, PISSED, looks to Otto who's with a CUTE
 HACKER GIRL in back of the EMPTY AUDITORIUM, maybe A DOZEN
 HACKERS.

In the movie, the “Xbox Guy” actually shouts “What the hell!” with a German accent.

I assume the person with the voltmeter is supposed to be Felix, and the angry German with the long hair (called “Game Console Hacker” in the credits, played by Christoph Franken) is supposed to be me.

On the corkboard in the movie, there are about two dozen printed slides.

This is a reconstructed version created from many individual frames of the scene:

It looks like the “Xbox Guy” is named “Denis Schnegg”, and the name of the talk is “Hacking Game Consoles”.

Most slides I could decipher are direct copies from slides from either our 24C3 talk, or our Google Tech Talk in 2008. Here are the screen captures of the slides in the movie, and the corresponding slides from our talks:

1 24c3, slide 6
2 Google, slide 6
3 24c3, slide 8
4 24c3, slide 7
5 Google, slide 24
6 Google, slide 26
7 Google, slide 28
8 Google, slide 36
9 Google, slide 8
10 Google, slide 9
11 Google, slide 15
12 Google, slide 11
13 Google, slide 14
25 24c3, slide 2

For reference, here are the full presentations the scene in the movie is based on:

Why Silicon-Based Security is still that hard: Deconstructing Xbox 360 Security (24C3)

The Xbox 360 Security System and its Weaknesses (Google TechTalk)

And since the producers of the movie consider it fair use to copy 14 of my slides without giving me credit, it must also be fair use to quote the scene of the movie here:

Rhapsody Developer’s Guide [PDF, 1997]


Feiler, Jesse.
Rhapsody Developer’s Guide.
Boston: AP Professional, 1997.
ISBN 0-12-251334-7

(528 pages, 13.3 MB PDF)

Rhapsody Developer’s Guide provides a road map to Rhapsody technology and the ways it can be used. Based on a modern microkernel, Rhapsody runs on PowerPC and Intel processors, and supports traditional Mac OS applications (in the Blue Box) as well as modern applications in the Yellow Box. Totally object-oriented, the Yellow Box platform offers an unparalleled development environment that permits rapid implementation of functionality ranging from traditional personal computer applications to media-rich, Internet-enabled, and database-driven applications for the next century.

This book describes the architecture of Rhapsody, including its cross-platform implementation on PowerPC and Intel. It details the Yellow Box platform (based on OpenStep) and provides a complete description of the core API, as well as a description of the architecture that will be enriched in the future with additional functionality from Apple. The languages of Rhapsody are discussed, and the API is presented in a language-neutral way that will be convenient for C++ developers, classic and modem Objective-C users, and Java programmers. Throughout, there is an emphasis on how Rhapsody relates to existing investments in code and programming expertise. Screen shots and code samples from products shipping today using Rhapsody technology provide opportunities and challenges to new Rhapsody developers.

About the Author

Jesse Feiler is software director of the Philmont Software Mill. He is also the author of Cyberdog and Real World Apple Guide. He has served as a consultant, author, and speaker for many prestigious businesses, including the Federal Reserve Bank ofNew York, Prodigy, Kodak, Young & Rubicam, and Apple Computer, Inc.

Why is my TI-99/4A in Black and White?

by James Abbatiello

My first computer was a Texas Instruments TI-99/4A. Longtime readers may remember a previous article where we implemented TI-99/4A BASIC as a Scripting Language for modern computers. Recently I got nostalgic for the actual hardware so I got my 99 out of the closet where it had been for a decade or more. I hooked it up to the TV and turned it on. I was expecting to see something like this:TI-99/4a title screen

Instead I was greeted by this:

Well, that’s not right! It is in black and white. And what’s with all these vertical black lines? Clearly something’s wrong but what could it be?

New Meets Old

At first I suspected that it was my TV, which is a fairly new LCD. Old computers or game consoles sometimes played a bit fast and loose with the NTSC standard and it seemed unlikely that a new TV would ever have been tested with something as old as a TI-99/4A. Perhaps the TV just couldn’t interpret output from the 99. So I tried with a CRT TV:

Well the black bars are gone (or at least not as apparent) but it is still in black and white. Something must be wrong with the computer itself.

All About Video Signals

The output from the back of the computer is a composite video signal but using a 5-pin DIN connector (that also carries audio and power) instead of the usual RCA jack. Back when this computer was new that signal would usually to go an RF modulator which was then connected to a TV via a 300-ohm connector. Nowadays you can still do the same thing but since most TVs don’t have screw terminals on the back anymore it can be more convenient to take the composite video signal and hook it directly into the composite input on the TV. All that is required is a simple adapter cable that can be created yourself or purchased online.

I thought that perhaps the video circuitry was generating separate Luminance (Y) and Chrominance (C) signals and then combining them into the final composite output. If this were the case then it would suggest something was wrong in the C amplifier or the final combining stage. It turns out that this is not the case. The video chip in the TI-99/4A is referred to as the Video Display Processor (VDP) and is a TMS9918A, TMS9928A or TMS9929A depending on the region the computer was originally intended for and the television standard in use there (e.g. NTSC or PAL). My computer was made for the US market and outputs NTSC signals using the TMS9918A. This chip has a single video output pin that supplies composite video directly with the Y and C already mixed. So if something was wrong with just the C generation circuitry then it was something broken inside the VDP and my only recourse would be to try to find a replacement chip.

Mad Scientist Equipment

The VDP still seemed to work correctly in all other respects so I was hopeful that the true problem lay elsewhere. I thought I’d take a look at the signal on an oscilloscope. We’d expect to the see the NTSC colorburst and if it was missing that would explain why no color was showing up on the TVs. Here’s what it looked like:

And here’s a closeup of the interesting portion:

I’m no expert but that looks like a horizontal sync pulse followed by a colorburst to me. But there was still no color on the TV.

The composite video signal that the VDP generates is sent to a simple 2-transistor amplifier and then to the output jack. I didn’t think it was likely but perhaps something in the amplifier had given out and Y was still strong enough to get picked up by the TV but C wasn’t. To test this I took the computer apart and tapped the signal right as it came out of the chip and before it went through the amplifier. It was still black-and-white. This suggested that the problem was not in the amplifier.

The Healing Power of Crystals

At this point I knew that the VDP was mostly working correctly. It generated the right pattern on the screen so it must be able to communicate with both the video RAM and the CPU. That accounts for most of the pins on the VDP, the ones handling digital signals. The remaining pins are mostly for power and the connection to the quartz crystal that provides the timing. I checked the power and that seemed fine. So let’s take a closer look at the crystal:

The crystal is the gray-colored component in the middle. To the right is the VDP, covered in thermal paste. Just behind the crystal is a variable inductor.

A variable inductor: now that’s interesting! It is connected to the crystal and apparently used for fine tuning the frequency. Could the fix be as simple as turning an adjustment screw?

Alas, no. I turned it as far as it would go in both directions with no improvement to the video output. If the frequency was off it was beyond the ability of this adjustment to correct. I don’t have any equipment to allow precision measuring of the actual frequency this crystal was producing, but I do have the internet. A little Googling brought me to this post on the TI-99/4A mailing list. Yes, there’s still an active mailing list for a computer that hasn’t been manufactured in almost 30 years!

That post describes the symptoms that I was experiencing and indicated that the solution was to replace the crystal. This was somewhat surprising to me. I’d heard of electrolytic capacitors going bad in old equipment but a quartz crystal? They’re usually quite reliable. But you can’t argue with real-world experience.

The VDP takes the frequency of this crystal (10.738635 MHz) and divides it by 3 to produce the NTSC colorburst frequency (3.579545 MHz). If the frequency of the crystal was off then the generated colorburst would also be off and the TV wouldn’t be able to sync to it. Without seeing a valid colorburst the TV isn’t going to produce any color. That would certainly explain our symptoms!

So after deciding to replace this crystal we have to actually find a replacement part. We want a crystal that runs at exactly 10.738635 MHz. We also need it rated for the proper “load capacitance”. Running a crystal with the wrong capacitance will shift the frequency from the rated frequency. That would be bad since our entire goal is to get the frequency back to the ideal. The original crystal was rated for a load capacitance of 32pF (you can just make out the 32 in the above picture although it is partially obscured by the blue wire). So we want a replacement crystal that’s also rated for 32pF.

Let’s go internet shopping for 10.738635 MHz crystals. Jameco doesn’t carry any. Digikey has some but didn’t have any in stock with a 32pF load capacitance. Luckily Mouser came through for me! A few days later and I had a replacement crystal:

And after a little surgery on the motherboard:

Now for the moment of truth:

Success! Now to play some Parsec!

Bonus Oscilloscope Image

If you’re wondering what the colorburst looks like with the new crystal then wonder no longer.

Looks pretty similar to my eyes but apparently it makes a world of difference to a TV.