Note: This article is a work-in-progress.
A few years ago I did a talk about ToaruOS in which I traced through the system and explained how typing in a terminal worked at each layer. ToaruOS has changed a lot since then, and the slides from that talk have become outdated. Additionally, the level of detail to which a 40-minute talk can delve is limited, but an article can go much deeper. With the recent development of ToaruOS-NIH, a completely in-house distribution of ToaruOS with all code under the NCSA license, I figured it was time to approach that challenge again.
This article is a followup to my talk at Yelp, which itself was inspired by an article that posed the following question:
What happens when you open a browser, type "google.com", and hit Enter?
ToaruOS doesn't really have a web browser (main-line ToaruOS does have something that may possibly look like a web browser if you squint, but we'll skip that for now), so we'll adjust this question slightly. My previous talk was "What happens when you type fetch http://toaruos.org/docs/talk.pdf in a terminal and hit Enter?" which was reflective of the tools available in main-line ToaruOS at the time. As we will be using ToaruOS-NIH, one additional modification is necessary: The fetch tool is built on a third-party HTTP library, so we will instead use http-get, which is a rudimentary in-house HTTP client. Additionally, we'll use a different URL. Thus, ultimately, the question we pose is: "What happens when you type http-get http://toaruos.org/test in a terminal, and hit Enter?".
As ToaruOS-NIH includes its own bootloader, I believe we should start long before we type anything in our terminal - let's start from the beginning: right out of the BIOS.
ToaruOS-NIH's bootloader is an El Torito "no-emulation" CD bootloader. In supported BIOSes, an binary of 20 512-byte sectors is loaded from the CD into memory. This is our complete bootloader, saving us the need to have multiple stages that load larger binaries.
The first thing the bootloader does when the BIOS jumps to it is scan for available memory. It does this using two BIOS facilities while still in real mode: INT 12h, and E820.
[bits 16]
main:
mov ax, 0x0000
mov ds, ax
mov ax, 0x0500
mov es, ax
cli
clc
int 0x12
mov [lower_mem], ax
; memory scan
mov di, 0x0
call do_e820
jc hangNext, the bootloader enables the A20 line - a necessary step in jumping to protected mode.
Initializing a simple GDT, the bootlaoder then jums to protected mode.
; a20
in al, 0x92
or al, 2
out 0x92, al
; basic flat GDT
xor eax, eax
mov ax, ds
shl eax, 4
add eax, gdt_base
mov [gdtr+2], eax
mov eax, gdt_end
sub eax, gdt_base
mov [gdtr], ax
lgdt [gdtr]
; protected mode enable flag
mov eax, cr0
or eax, 1
mov cr0, eax
; set segments
mov ax, 0x10
mov ds, ax
mov es, ax
mov fs, ax
mov gs, ax
mov ss, ax
; jump to protected mode entry
extern kmain
jmp far 0x08:(kmain)We are now in C, and this is where the real fun begins. The bootloader provides a visual menu for selection options. It does this by writing directly the VGA text mode video memory, as the BIOS methods for writing to the screen are no longer available. It also reads the keyboard through simple polling, which is different from how the kernel will read the keyboard later. Arrow keys select between options. The screen is redrawn with each key press. Enter selects an option, either toggling a setting or continuing the boot process.
int s = read_scancode();
if (s == 0x50) {
sel = (sel + 1) % sel_max;
continue;
} else if (s == 0x48) {
sel = (sel_max + sel - 1) % sel_max;
continue;
} else if (s == 0x1c) {
...
}The next step is to detect the boot device, which is assumed to be an ATAPI CD ROM drive, due to the nature of the bootloader. The CD itself contains an ISO9660 filesystem, which the bootloader includes a simple driver for. We locate the kernel, load it into memory, then begin loading modules specified by the boot configuration, and finally load the ramdisk.
ata_device_detect(&ata_primary_master);
ata_device_detect(&ata_primary_slave);
ata_device_detect(&ata_secondary_master);
ata_device_detect(&ata_secondary_slave);
...
if (ata_primary_slave.is_atapi) {
do_it(&ata_primary_slave);
}
...static void do_it(struct ata_device * _device) {
...
if (navigate("KERNEL.")) {
print("Found kernel.\n");
print_hex(dir_entry->extent_start_LSB); print(" ");
print_hex(dir_entry->extent_length_LSB); print("\n");
long offset = 0;
for (int i = dir_entry->extent_start_LSB; i < dir_entry->extent_start_LSB + dir_entry->extent_length_LSB / 2048 + 1; ++i, offset += 2048) {
ata_device_read_sector_atapi(device, i, (uint8_t *)KERNEL_LOAD_START + offset);
}
...With the kernel, modules, and ramdisk loaded into memory we now load the kernel as an ELF binary, using its Phdr information.
for (uintptr_t x = 0; x < (uint32_t)header->e_phentsize * header->e_phnum; x += header->e_phentsize) {
Elf32_Phdr * phdr = (Elf32_Phdr *)((uint8_t*)KERNEL_LOAD_START + header->e_phoff + x);
if (phdr->p_type == PT_LOAD) {
//read_fs(file, phdr->p_offset, phdr->p_filesz, (uint8_t *)phdr->p_vaddr);
print("Loading a Phdr... ");
print_hex(phdr->p_vaddr);
print(" ");
print_hex(phdr->p_offset);
print(" ");
print_hex(phdr->p_filesz);
print("\n");
memcpy((uint8_t*)phdr->p_vaddr, (uint8_t*)KERNEL_LOAD_START + phdr->p_offset, phdr->p_filesz);
long r = phdr->p_filesz;
while (r < phdr->p_memsz) {
*(char *)(phdr->p_vaddr + r) = 0;
r++;
}
}
}Then we use the memory map information we collected during the E820 phase to produce a Multiboot-compatible memory map.
print("Setting up memory map...\n");
print_hex(mmap_ent);
print("\n");
memset((void*)KERNEL_LOAD_START, 0x00, 1024);
mboot_memmap_t * mmap = (void*)KERNEL_LOAD_START;
multiboot_header.mmap_addr = (uintptr_t)mmap;
struct mmap_entry * e820 = (void*)0x5000;
uint64_t upper_mem = 0;
for (int i = 0; i < mmap_ent; ++i) {
print("entry "); print_hex(i); print("\n");
print("base: "); print_hex((uint32_t)e820[i].base); print("\n");
print("type: "); print_hex(e820[i].type); print("\n");
mmap->size = sizeof(uint64_t) * 2 + sizeof(uintptr_t);
mmap->base_addr = e820[i].base;
mmap->length = e820[i].len;
mmap->type = e820[i].type;
if (mmap->type == 1 && mmap->base_addr >= 0x100000) {
upper_mem += mmap->length;
}
mmap = (mboot_memmap_t *) ((uintptr_t)mmap + mmap->size + sizeof(uintptr_t));
}
print("lower "); print_hex(lower_mem); print("KB\n");
multiboot_header.mem_lower = 1024;
print("upper ");
print_hex(upper_mem >> 32);
print_hex(upper_mem);
print("\n");
multiboot_header.mem_upper = upper_mem / 1024;Finally, we jump to the loaded kernel. Some assembly ensures we use the expected calling convention.
_eax = MULTIBOOT_EAX_MAGIC;
_ebx = (unsigned int)&multiboot_header;
_xmain = entry;
jump_to_main();[bits 32]
global jump_to_main
jump_to_main:
extern _eax
extern _ebx
extern _xmain
mov eax, [_eax]
mov ebx, [_ebx]
jmp [_xmain]