Major GEOS applications on the Commodore 64 protect themselves from unauthorized duplication by keying themselves to the operating system's serial number. To avoid tampering with this mechanism, the system contains some elaborate traps, which will be discussed in this article.
GEOS Copy Protection
The GEOS boot disk protects itself with a complex copy protection scheme, which uses code uploaded to the disk drive to verify the authenticity of the boot disk. Berkeley Softworks, the creators of GEOS, found it necessary to also protect their applications like geoCalc and geoPublish from unauthorized duplication. Since these applications were running inside the GEOS "KERNAL" environment, which abstracted most hardware details away, these applications could not use the same kind of low-level tricks that the system was using to protect itself.
Serial Numbers for Protection
The solution was to use serial numbers. On the very first boot, the GEOS system created a 16 bit random number, the "serial number", and stored it in the KERNAL binary. (Since the system came with a "backup" boot disk, the system asked for that disk to be inserted, and stored the same serial in the backup's KERNAL.) Now whenever an application was run for the first time, it read the system's serial number and stored it in the application's binary. On subsequent runs, it read the system's serial number and compared it with the stored version. If the serial numbers didn't match, the application knew it was running on a different GEOS system than the first time – presumably as a copy on someone else's system: Since the boot disk could not be copied, two different people had to buy their own copies of GEOS, and different copies of GEOS had different serial numbers.
Serial Numbers in Practice
The code to verify the serial number usually looked something like this:
.,D5EF 20 D8 C1 JSR $C1D8 ; GetSerialNumber
.,D5F2 A5 03 LDA $03 ; read the hi byte
.,D5F4 CD 2F D8 CMP $D82F ; compare with stored version
.,D5F7 F0 03 BEQ $D5FC ; branch if equal
.,D5F9 EE 18 C2 INC $C218 ; sabotage LdDeskAcc syscall: increment vector
.,D5FC A0 00 LDY #$00 ; ...
If the highest 8 bits of the serial don't match the value stored in the application's binary, it increments the pointer of the LdDeskAcc vector. This code was taken from the "DeskTop" file manager, which uses this subtle sabotage to make loading a "desk accessory" (a small helper program that can be run from within an application) unstable. Every time DeskTop gets loaded, the pointer gets incremented, and while LdDeskAcc might still work by coincidence the first few times (because it only skips a few instructions), it will break eventually. Other applications used different checks and sabotaged the system in different ways, but they all had in common that they called GetSerialNumber.
(DeskTop came with every GEOS system and didn't need any extra copy protection, but it checked the serial anyway to prevent users from permanantly changing their GEOS serial to match one specific pirated application.)
A Potential Generic Hack
The downside of this scheme is that all applications are protected the same way, and a single hack could potentially circumvent the protection of all applications.
A generic hack would change the system's GetSerialNumber implementation to return exactly the serial number expected by the application by reading the saved value from the application's binary. The address where the saved valus is stored is different for every application, so the hack could either analyze the instructions after the GetSerialNumber call to detect the address, or come with a small table that knows these addresses for all major applications.
GEOS supports auto-execute applications (file type $0E) that will be executed right after boot – this would be the perfect way to make this hack available at startup without patching the (encrypted) system files.
Trap 1: Preventing Changing the Vector
Such a hack would change the GetSerialNumber vector in the system call jump table to point to new code in some previously unused memory. But the GEOS KERNAL has some code to counter this:
; (Y = $FF from the code before)
.,EE59 B9 98 C0 LDA $C098,Y ; read lo byte of GetSerialNumber vector
.,EE5C 18 CLC
.,EE5D 69 5A ADC #$5A ; add $5A
.,EE5F 99 38 C0 STA $C038,Y ; overwrite low byte GraphicsString vector
In the middle of code that deals with the menu bar and menus, it uses this obfuscated code to sabotage the GraphicsString system call if the GetSerialNumber vector was changed. If the GetSerialNumber vector is unchanged, these instructions are effectively a no-op: The lo byte of the system's GetSerialNumber vector ($F3) plus $5A equals the lo byte of the GraphicsString vector ($4D). But if the GetSerialNumber vector was changed, then GraphicsString will point to a random location and probably crash.
Berkely Softworks was cross-developing GEOS on UNIX machines with a toolchain that supported complex expressions, so they probably used code like this to express this:
; Y = $FF
lda GetSerialNumber + 1 - $FF,y
clc
adc #<(_GraphicsString - _GetSerialNumber)
sta GraphicsString + 1 - $FF,y
In fact, different variations of GEOS (like the GeoRAM version) were separate builds with different build time arguments, so because of different memory layouts, they were using different ADC values here.
Note that the pointers to the GetSerialNumber and GraphicsString have been obfuscated, so that an attacker that has detected the trashed GraphicsString vector won't be able to find the sabotage code by looking for the address.
Trap 2: Preventing Changing the Implementation
If the hack can't change the GetSerialNumber vector, it could put a JMP instruction at the beginning of the implementation to the new code. But the GEOS KERNAL counters this as well. The GetSerialNumber implementation looks like this:
.,CFF3 AD A7 9E LDA $9EA7 ; load lo byte of serial
.,CFF6 85 02 STA $02 ; into return value (lo)
.,CFF8 AD A8 9E LDA $9EA8 ; load hi byte of serial
.,CFFB 85 03 STA $03 ; into return value (hi)
.,CFFD 60 RTS ; return
At the end of the system call function UseSystemFont, it does this:
.,E6C9 AD 2F D8 LDA $D82F ; read copy of hi byte of serial
.,E6CC D0 06 BNE $E6D4 ; non-zero? done this before already
.,E6CE 20 F8 CF JSR $CFF8 ; call second half of GetSerialNumber
.,E6D1 8D 2F D8 STA $D82F ; and store the hi byte in our copy
.,E6D4 60 RTS ; ...
And in the middle of the system call function FindFTypes, it does this:
.,D5EB A2 C1 LDX #$C1
.,D5ED A9 96 LDA #$96 ; public GetSerialNumber vector ($C196)
.,D5EF 20 D8 C1 JSR $C1D8 ; "CallRoutine": call indirectly (obfuscation)
.,D5F2 A5 03 LDA $03 ; read hi byte of serial
.,D5F4 CD 2F D8 CMP $D82F ; compare with copy
.,D5F7 F0 03 BEQ $D5FC ; if identical, skip next instruction
.,D5F9 EE 18 C2 INC $C218 ; sabotage LdDeskAcc by incrementing its vector
.,D5FC A0 00 LDY #$00 ; ...
So UseSystemFont makes a copy of the hi byte of the serial, and FindFTypes compares the copy with the serial – so what's the protection? The trick is that one path goes through the proper GetSerialNumber vector, while the other one calls into the bottom half of the original implementation. If the hack overwrites the first instruction of the implementation (or managed to disable the first trap and changed the system call vector directly), calling though the vector will reach the hack, while calling into the middle of the original implementation will still reach the original code. If the hack returns a different value than the original code, this will sabotage the system in a subtle way, by incrementing the LdDeskAcc system call vector.
Note that this code calls a KERNAL system function that will call GetSerialNumber indirectly, so the function pointer is split into two 8 bit loads and can't be found by just searching for the constant. Since the code in UseSystemFont doesn't call GetSerialNumber either, an attacker won't find a call to that function anywhere inside KERNAL.
Summary
I don't know whether anyone has ever created a generic serial number hack for GEOS, but it would have been a major effort – in a time before emulators allowed for memory watch points. An attacker would have needed to suspect the memory corruption, compared memory dumps before and after, and then found the two places that change code. The LdDeskAcc sabotage would have been easy to find, because it encodes the address as a constant, but the GraphicsString sabotage would have been nearly impossible to find, because the trap neither uses the verbatim GraphicsString address nor GetSerialNumber function.
Usual effective hacks were much more low-tech and instead "un-keyed" applications, i.e. removed the cached serial number from their binaries to revert them into their original, out-of-the-box state.
