{"id":1425,"date":"2020-09-01T22:42:53","date_gmt":"2020-09-01T20:42:53","guid":{"rendered":"https:\/\/www.pagetable.com\/?p=1425"},"modified":"2020-09-01T22:42:53","modified_gmt":"2020-09-01T20:42:53","slug":"inside-geowrite-1-the-overlay-system","status":"publish","type":"post","link":"https:\/\/www.pagetable.com\/?p=1425","title":{"rendered":"Inside geoWrite \u2013 1: The Overlay System"},"content":{"rendered":"<p>geoWrite is a WYSIWYG rich text editor for the Commodore 64 GEOS operating system, which runs with a total of just 64 KB of RAM. In the series about the internals of geoWrite, this article discusses how it manages to fit 52 KB of code into the available 23 KB of application RAM.<\/p>\n<p><img decoding=\"async\" src=\"docs\/geowrite\/geowrite_1_overlays.png\" alt=\"\" \/><\/p>\n<h2 id=\"introduction\">Introduction<\/h2>\n<p><a href=\"https:\/\/github.com\/mist64\/geos\">GEOS<\/a> is a disk-based graphical operating system for the Commodore 64 that provides the following features:<\/p>\n<ul>\n<li>applications, desk accessories<\/li>\n<li>disk, printer and mouse drivers<\/li>\n<li>loadable proportional fonts<\/li>\n<li>menu bars, dialogs, file picker<\/li>\n<li>multi-fork filesystem API<\/li>\n<li>misc. library code (math, memory, strings, &hellip;)<\/li>\n<\/ul>\n<p>But since the OS kernel (called the &ldquo;GEOS KERNAL&rdquo;) is only 20 KB in size, some of the APIs are very limited, or cumbersome to use, so applications had to do a lot of work one would expect from the OS these days.<\/p>\n<p>The GEOS authors &ldquo;Berkeley Softworks&rdquo; also wrote several applications \u2013 the OS-included deskTop, geoWrite and geoPaint, as well as geoPublish, geoCalc, geoChart, geoFile and geoDex \u2013\u00a0which all share low-level functionality that is not in fact part of the operating system, but shared code between these apps, like:<\/p>\n<ul>\n<li>code overlays<\/li>\n<li>custom screen recovery<\/li>\n<li>create\/open\/exit startup dialog<\/li>\n<li>desk accessory enumeration<\/li>\n<li>font management<\/li>\n<li>zero page management<\/li>\n<li>copy protection<\/li>\n<\/ul>\n<p>In this series of articles, we will discuss some of the lower-level features that are implemented on the application side, using the example of geoWrite.<\/p>\n<ol>\n<li><strong>The Overlay System<\/strong> \u2190 this article<\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1428\">Screen Recovery<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1436\">Font Management<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1442\">Zero Page<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1449\">Copy Protection<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1460\">Localization<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1471\">File Format and Pagination<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1481\">Copy &amp; Paste<\/a><\/li>\n<li><a href=\"https:\/\/www.pagetable.com\/?p=1490\">Keyboard Handling<\/a><\/li>\n<\/ol>\n<h2 id=\"geos-memory-map\">GEOS Memory Map<\/h2>\n<p>GEOS and its apps need to fit into the 64 KB of RAM of the C64. Here is a rough overview of the memory map, mostly to scale:<\/p>\n<pre><code>-----------------------------------------\n$0000  Zero page, stack, system variables\n-----------------------------------------\n$0400\n\n\n\n\n       Application memory\n\n\n\n\n\n-----------------------------------------\n$6000\n       Background bitmap\n\n-----------------------------------------\n$8000  System buffers and variables\n-----------------------------------------\n$9000  Disk driver\n-----------------------------------------\n$A000\n       Screen bitmap\n\n-----------------------------------------\n$C000  GEOS KERNAL\n\n\n-----------------------------------------\n<\/code><\/pre>\n<p>The screen bitmap is an 8 KB RAM area for the 320&#215;200 monochrome screen bitmap. There is a full-size &ldquo;background&rdquo; copy used for recovering the main screen contents after closing a menu or a dialog without needing a slow redraw.<\/p>\n<p>The application has 23 KB of memory that it can use for code and data. geoWrite is 52 KB of code, and it needs 2 KB for variables, 7 KB for the current page of text and 6 KB for bitmap fonts. That would be 67 KB&hellip;<\/p>\n<h2 id=\"vlir-files\">VLIR Files<\/h2>\n<p>In the UNIX world, a file is a mapping of a filename to a sequence of bytes (and some metadata). Some operating systems extend this concept: On classic MacOS, files consist of two of these sequences (the &ldquo;resource fork&rdquo; and the &ldquo;data fork&rdquo;). NextSTEP and MacOS X can present folders with a tree of individual files as a single &ldquo;bundle&rdquo;.<\/p>\n<p>GEOS extends the UNIX-like Commodore filesystem with &ldquo;VLIR&rdquo; files, which stands for &ldquo;Variable Length Index Record&rdquo;. A VLIR file maps a filename to a 256 byte &ldquo;file header&rdquo; (for the icon and extra metadata) and up to 127 records, numbered 0-126. A record is a variable-length sequence of bytes, much like a traditional UNIX file.<\/p>\n<p>You can imagine a VLIR file as a folder with several files in it. The following is a visualization of a typical geoWrite document:<\/p>\n<pre><code>geoWrite Doc\n\\--- File Header\n\\--- 0\n\\--- 1\n\\--- 2\n\\--- 61\n\\--- 62\n\\--- 64\n<\/code><\/pre>\n<p>As you can see, record numbers don&rsquo;t have to be contiguous. (geoWrite for example stores pages in records 0-60, the header and footer into records 61 and 62, and image data in records 64-126.)<\/p>\n<p>The file header contains the icon, the file type, and some other generic as well as application-specific metadata.<\/p>\n<h2 id=\"vlir-applications\">VLIR Applications<\/h2>\n<p>GEOS applications can be VLIR files as well. When running an app, the system loads record 0 into memory and executes it. It&rsquo;s entirely up to the application what to do with the other records.<\/p>\n<p>This is what the geoWrite app looks like:<\/p>\n<pre><code>GEOWRITE           size\n\\--- File Header    256\n\\--- 0            10335\n\\--- 1             2552\n\\--- 2             3999\n\\--- 3             2328\n\\--- 4             1965\n\\--- 5             3870\n\\--- 6             3998\n\\--- 7             3897\n\\--- 8             1194\n<\/code><\/pre>\n<p>The geoWrite main code in record 0 is about 10 KB in size. The operating system loads it to $0400-$2C5E into application RAM.<\/p>\n<h2 id=\"code-overlays\">Code Overlays<\/h2>\n<p>This is the memory layout of application RAM for geoWrite:<\/p>\n<pre><code>-----------------------------------------\n$0400  Main code (record 0)\n\n\n-----------------------------------------\n$2C5F  Variables\n-----------------------------------------\n$3244  Overlay code (records 1-7)\n\n-----------------------------------------\n$41E4  Variables, page data, font data\n\n\n\n\n\n-----------------------------------------\n<\/code><\/pre>\n<p>The record 0 code loaded by the OS always remains in its slot. It contains the core editing functionality, the overlay manager, the font manager and other library code.<\/p>\n<p>There is a 4 KB slot for &ldquo;overlay&rdquo; code, meaning that the record 0 code can swap in any of the records from 1 through 7.<\/p>\n<ul>\n<li>[0 library code, core text editing]<\/li>\n<li>1 initialization, copy protection<\/li>\n<li>2 core text editing<\/li>\n<li>3 cut, copy, paste<\/li>\n<li>4 ruler editing<\/li>\n<li>5 startup\/about, create, open, paste text, run desk accessory<\/li>\n<li>6 navigation, search\/replace, header\/footer, reflow<\/li>\n<li>7 printing<\/li>\n<li>[8 print settings]<\/li>\n<\/ul>\n<p>(Record 8 is handled differently and is discussed at the end of this article.)<\/p>\n<p>Every record is linked to the same address ($3244) and starts with a jump table, like this one:<\/p>\n<pre><code>CODE5:\n    jmp recover            ; 0\n    jmp showStartupMenu    ; 1\n    jmp renameDocument     ; 2\n    jmp openDocument       ; 3\n    jmp showAboutDialog    ; 4\n    jmp loadDeskAcc        ; 5\n    jmp exitToDesktop      ; 6\n    jmp readReservedRecord ; 7\n    jmp makeFullPageWide   ; 8\n<\/code><\/pre>\n<p>The jump table means that the record 0 code can be assembled independently of the overlays. While the overlays access symbols in the record 0 code, record 0 code only calls through these jump table entries.<\/p>\n<h2 id=\"vlir-api\">VLIR API<\/h2>\n<p>The GEOS KERNAL has the following calls for working with VLIR files:<\/p>\n<ul>\n<li><code>OpenRecordFile<\/code> \u2013\u00a0Open an existing VLIR file given its name<\/li>\n<li><code>UpdateRecordFile<\/code> \u2013\u00a0Flush the VLIR&rsquo;s metadata to disk<\/li>\n<li><code>CloseRecordFile<\/code> \u2013 Flush and close VLIR file<\/li>\n<li><code>PointRecord<\/code> \u2013\u00a0Set current record<\/li>\n<li><code>PreviousRecord<\/code> \u2013\u00a0Move to previous record<\/li>\n<li><code>NextRecord<\/code> \u2013 Move to next record<\/li>\n<li><code>ReadRecord<\/code> \u2013\u00a0Read complete record into memory<\/li>\n<li><code>WriteRecord<\/code> \u2013\u00a0Write\/overwrite complete record from memory image<\/li>\n<li><code>DeleteRecord<\/code> \u2013\u00a0Delete current record<\/li>\n<\/ul>\n<p>Reading overlay code should therefore be as simple as this:<\/p>\n<pre><code>    ; startup\n    LoadW   r0, fnBuffer\n    jsr     OpenRecordFile\n\n    ; load overlay code\n    lda     #n\nloadCode:\n    jsr     PointRecord\n    LoadW   r7, OVERLAY_ADDRESS\n    LoadW   r2, OVERLAY_SIZE\n    jsr     ReadRecord\n<\/code><\/pre>\n<p>Unfortunately, GEOS can only have one VLIR file open at a time, and a geoWrite document is also a VLIR file. Opening and closing the two files would cause too much disk activity, which is why geoWrite comes with a simple read-only VLIR implementation on the side.<\/p>\n<h2 id=\"loading-overlays-manually\">Loading Overlays Manually<\/h2>\n<p>On disk, a VLIR file&rsquo;s directory entry points to it 256 bytes index table. Here is an example:<\/p>\n<pre><code>00 FF  06 13  08 10  09 14  0A 01  0A 12  0A 13  0B 00\n0C 02  0D 04  00 00  00 00  00 00  00 00  00 00  00 00\n[...]\n<\/code><\/pre>\n<p>The <code>00 FF<\/code> at the beginning is the Commodore DOS sector header and not part of the data. The remaining pairs of bytes point to the track and sector of the start of each record on disk.<\/p>\n<p>GEOS has an API for loading a file given a track and a sector (<code>ReadFile<\/code>), so all geoWrite needs to do is read its own index table on startup, and call <code>ReadFile<\/code> on items of this table when loading an overlay.<\/p>\n<p>Here&rsquo;s a shortened version of the code to get a copy of the app&rsquo;s index table:<\/p>\n<pre><code>    ; find application\n    LoadW   r6, fnBuffer\n    lda     #APPLICATION\n    sta     r7L\n    lda     #1 ; find max. 1 file\n    sta     r7H\n    LoadW   r10, appname\n    jsr     FindFTypes\n\n    LoadW   r0, fnBuffer\n    jsr     OpenRecordFile\n\n    jsr     i_MoveData ; copy index table\n    .word   fileHeader+2\n    .word   appIndexTable\n    .word   2 * NUM_APP_RECORDS\n    LoadB   curCodeRecord, $FF\n    rts\n\nappname:\n    .byte   \"geoWrite    V2.1\",0\n<\/code><\/pre>\n<p><code>OpenRecordFile<\/code> reads the VLIR file&rsquo;s index table info <code>fileHeader<\/code>. geoWrite then copies <code>NUM_APP_RECORDS<\/code> into its own table <code>appIndexTable<\/code>, skipping the first two bytes (<code>00 FF<\/code>).<\/p>\n<p>And here is a shortened version of the code to read a record:<\/p>\n<pre><code>    ; load overlay code\n    lda     #n\nloadCode:\n    cmp     curCodeRecord\n    beq     @rts ; already loaded\n    sta     curCodeRecord\n    asl     a\n    tay\n    lda     appIndexTable,y\n    sta     r1L\n    lda     appIndexTable+1,y\n    sta     r1H\n    LoadW   r7, OVERLAY_ADDRESS\n    LoadW   r2, OVERLAY_SIZE\n    jsr     _ReadFile\n@rts:\n    rts\n<\/code><\/pre>\n<p>You can see that the code keeps track of the currently loaded record, so it does not re-load the same code if it&rsquo;s already in memory.<\/p>\n<h2 id=\"managing-overlays\">Managing Overlays<\/h2>\n<p>All overlay functionality is implemented in the record 0 code, because it always needs to be accessible.<\/p>\n<p>There is a set of functions for loading the different records:<\/p>\n<pre><code>loadCode1:\n    lda     #1\n    .byte   $2C ; skip next\nloadCode2:\n    lda     #2\n    .byte   $2C ; skip next\nloadCode3:\n    lda     #3\n    .byte   $2C ; skip next\nloadCode4:\n    lda     #4\n    .byte   $2C ; skip next\nloadCode5:\n    lda     #5\n    .byte   $2C ; skip next\nloadCode6:\n    lda     #6\n    .byte   $2C ; skip next\nloadCode7:\n    lda     #7\nloadCode:\n    [...]\n<\/code><\/pre>\n<p>The record 0 code can then load an overlay and call a function through its jump table<\/p>\n<pre><code>    jsr     loadCode5\n    jsr     J5_showStartupMenu ; OVERLAY_ADDRESS + 3 * 1\n<\/code><\/pre>\n<p>Code inside an overlay can&rsquo;t call code from a different overlay this way, because the <code>loadCode<\/code> call would overwrite the caller. For this case, the record 0 code has functions like this one:<\/p>\n<pre><code>_showCantAddPages:\n    ldy     #&lt;J3_showCantAddPages ; OVERLAY_ADDRESS + 3 * 8\n    .byte   $2C\n_showTooManyPages:\n    ldy     #&lt;J3_showTooManyPages ; OVERLAY_ADDRESS + 3 * 7\n    .byte   $2C\n_splitTooBigPage:\n    ldy     #&lt;J3_splitTooBigPage  ; OVERLAY_ADDRESS + 3 * 5\n    ldx     #BANK_3\ncallRestore:\n    lda     curCodeRecord\n    pha\n    sty     @1\n    txa\n    jsr     loadCode\n@1 = * + 1\n    jsr     OVERLAY_ADDRESS\n    pla\n    jmp     loadCode\n<\/code><\/pre>\n<p>The function <code>showCantAddPages<\/code> is implemented on overlay 3. Code in overlay 2 can call <code>_showCantAddPages<\/code> in the record 0 code, which will load overlay 3, call the function, load the original overlay 2 again, and return.<\/p>\n<h2 id=\"splitting-the-logic\">Splitting the Logic<\/h2>\n<p>With helper functions in the record 0 code, it is possible to arbitrarily split logic into the different records. But since loading an overlay takes about 2-3 seconds on a 1541 disk drive, this should be minimized.<\/p>\n<h3 id=\"central-library-code\">Central Library Code<\/h3>\n<p>The overlay code that we have seen above has to live in the record 0 code, so it&rsquo;s directly callable by any record.<\/p>\n<p>While in theory any other library code could live in any other record, keeping the most-used functionality in the record 0 code will reduce disk accesses. Here are some examples:<\/p>\n<ul>\n<li>generic dialogs<\/li>\n<li>error dialogs<\/li>\n<li>drive switching (app vs. document)<\/li>\n<li>disk full testing<\/li>\n<li>screen recovery<\/li>\n<li>font management<\/li>\n<li>zero page management<\/li>\n<li>some common text strings<\/li>\n<\/ul>\n<h3 id=\"one-time-logic\">One-time logic<\/h3>\n<p>Furthermore, there is code that is only ever needed once. On startup, the following is done:<\/p>\n<ul>\n<li>enumerate fonts and desk accessories on disk<\/li>\n<li>get the page size from the printer<\/li>\n<li>initialize the menu bar<\/li>\n<li>draw the ruler and the page indicator<\/li>\n<li>prepare a file opened for printing only<\/li>\n<li>do the copy protection dance<\/li>\n<\/ul>\n<p>All this code lives in record 1. It is loaded immediately after the app is started. When it returns, it never gets loaded again.<\/p>\n<h3 id=\"main-mode-code\">Main Mode Code<\/h3>\n<p>Then, there is code that is needed when the app is in its main mode, like the text renderer and the handlers for navigating on the page, typing text and deleting text.<\/p>\n<p>The main mode code lives in the remainder of record 0 as well as in record 2: During normal text editing, geoWrite always keeps record 2 loaded.<\/p>\n<p>Since functions for menu items, keyboard shortcuts and mouse triggers in main mode are called by the GEOS KERNAL directly, which does not know about banking, at least the entry points of these handlers also have to live in record 0 (or, with restrictions, record 2).<\/p>\n<h3 id=\"grouped-functionality\">Grouped Functionality<\/h3>\n<p>The remaining records contain further functionality, grouped by topic, so they can use common code inside the same record. Here is the list again:<\/p>\n<ul>\n<li>3 cut, copy, paste<\/li>\n<li>4 ruler editing<\/li>\n<li>5 startup\/about, create, open, paste text, run desk accessory<\/li>\n<li>6 navigation, search\/replace, header\/footer, reflow<\/li>\n<li>7 printing<\/li>\n<\/ul>\n<p>On app launch, the record 0 entry code immediately loads initialization code in record 1. After its return, the record 0 code runs the startup UI code in record 5, which creates a new file or opens an existing file and returns to the core text editor in the record 0\/2 code.<\/p>\n<h3 id=\"duplicate-code\">Duplicate Code<\/h3>\n<p>Common code needed by very different functionality groups usually lives in record 0, so that any record can call it without causing swapping in and out overlays. This is true for most common code, but since space in record 0 is at a premium, a different solution was necessary especially for bigger reusable components.<\/p>\n<p>The startup code for example needs to talk to the printer driver to query the page size, and the print code needs to talk to the printer for printing. They both need to look up and load the printer driver. This common code is too big to fit into record 0, and having the startup code (#1) call out into the printing record (#7) would increase startup time by at least 5 seconds, swapping in #7 and then swapping in #1 again.<\/p>\n<p>The solution is to just duplicate the common code into different records, e.g. by using <code>.include<\/code> statements to reuse the same code in multiple places. As long as the individual records don&rsquo;t overflow their 4 KB maximum, this is a reasonable tradeoff.<\/p>\n<p>Other examples are the document file version check, the string-to-int conversion code and common &ldquo;text scrap&rdquo; (clipboard) code. Parts of the latter are even included in three different records.<\/p>\n<h2 id=\"functionality->-4-kb&#8221;>Functionality > 4 KB<\/h2>\n<p>The printing logic is about 5 KB in size, so it doesn&rsquo;t fit into a 4 KB overlay record. Swapping in and out records during printing is not an option.<\/p>\n<p>Since printing is a different mode altogether, 1 KB of record 0 code in RAM is overwritten by the additional printing code from record #8. The code in record 0 was arranged in a way that there is a contiguous 1 KB chunk ($0680-$0B29) that does not need to be called by the printing functionality. This is, among other things, the data structures for drawing the main menu. After printing, the record #7 code re-loads all of record 0.<\/p>\n<h2 id=\"conclusion\">Conclusion<\/h2>\n<p>The overlay system has been a feature since the very first version of GEOS and is used by all major applications. It is not without its limits though: An application&rsquo;s main mode should be responsive and do most of its work without swapping overlays, so the core logic (record 0 and one overlay record) has a certain limit in complexity. The individual overlays have a very tight size limit as well.<\/p>\n<p>For the printing functionality, geoWrite already has to work around the overlay code size, which shows that an app like geoWrite is truly at the limit of what a GEOS application can do on a 64 KB system.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>geoWrite is a WYSIWYG rich text editor for the Commodore 64 GEOS operating system, which runs with a total of just 64 KB of RAM. In the series about the internals of geoWrite, this article discusses how it manages to fit 52 KB of code into the available 23 KB of application RAM. Introduction GEOS &#8230; <a title=\"Inside geoWrite \u2013 1: The Overlay System\" class=\"read-more\" href=\"https:\/\/www.pagetable.com\/?p=1425\" aria-label=\"Read more about Inside geoWrite \u2013 1: The Overlay System\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,5,41,8,15,22],"tags":[],"class_list":["post-1425","post","type-post","status-publish","format-standard","hentry","category-2","category-archeology","category-c64","category-commodore","category-geos","category-operating-systems"],"_links":{"self":[{"href":"https:\/\/www.pagetable.com\/index.php?rest_route=\/wp\/v2\/posts\/1425","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pagetable.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pagetable.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pagetable.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pagetable.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1425"}],"version-history":[{"count":0,"href":"https:\/\/www.pagetable.com\/index.php?rest_route=\/wp\/v2\/posts\/1425\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.pagetable.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1425"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pagetable.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1425"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pagetable.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1425"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}