marv7000/zinnia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: marv7000/zinnia:rewrite
Branches Tags
marv7000/zinnia:rewrite marv7000/zinnia:microkernel marv7000/zinnia:main
..
pull from: marv7000/zinnia:main
Branches Tags
marv7000/zinnia:rewrite marv7000/zinnia:microkernel marv7000/zinnia:main

4 Commits
rewrite .. main

Author SHA1 Message Date
marv7000 fce4d38045 meson: Fix noexecstack on clang 2026-04-25 15:00:22 +02:00
marv7000 7d1b867c36 fixes 2026-02-05 00:58:38 +01:00
marv7000 d22b96885a rename logo 2026-02-04 02:00:17 +01:00
marv7000 0e01345abf Initial commit 2026-02-04 01:58:49 +01:00
383 changed files with 4255 additions and 28116 deletions
Expand all files Collapse all files
.clang-format
+8 -12
View File
@@ -51,16 +51,12 @@ SpaceInEmptyBlock: false
SpacesBeforeTrailingComments: 1 SpacesBeforeTrailingComments: 1
IncludeBlocks: Merge IncludeBlocks: Merge
IncludeCategories: IncludeCategories:
- Regex: "^<zinnia/.*>" - Regex: "^<zinnia/.*>"
Priority: 1 Priority: 1
- Regex: "^<common/.*>" - Regex: "^<kernel/.*>"
Priority: 2 Priority: 2
- Regex: "^<kernel/.*>" - Regex: '^<.*\.h>'
Priority: 3 Priority: 3
- Regex: "^<vdso/.*>" - Regex: '^".*\.h"'
Priority: 4 Priority: 4
- Regex: '^<.*\.h>'
Priority: 5
- Regex: '^".*\.h"'
Priority: 6
SortIncludes: CaseInsensitive SortIncludes: CaseInsensitive
.clangd
+1 -1
View File
@@ -1,2 +1,2 @@
Style: Style:
AngledHeaders: [ "zinnia/.*", "uapi/.*" ] AngledHeaders: ["zinnia/.*", " kernel/.*"]
.gitignore
+9 -4
View File
@@ -1,8 +1,13 @@
.config* # IDE Files
build/ .vscode/
.idea/
.cache/ .cache/
.wraplock
# Build artifacts
*.o *.o
*.a *.a
*.so *.so
__pycache__/ compile_commands.json
# Default build dir
build/
.gitmodules
-3
View File
@@ -1,3 +0,0 @@
[submodule "subprojects/uacpi"]
path = subprojects/uacpi
url = https://github.com/uACPI/uacpi
Kconfig
-10
View File
@@ -1,10 +0,0 @@
mainmenu "Zinnia Kernel Configuration"
config KERNEL_FLAVOR
string "Kernel flavor name"
default "vanilla"
source "arch/$(ARCH)/Kconfig"
source "mem/Kconfig"
source "sched/Kconfig"
source "dev/Kconfig"
LICENSE
-1
View File
@@ -1 +0,0 @@
LICENSES/ISC.txt
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Marvin Friedrich
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
LICENSES/ISC.txt
-7
View File
@@ -1,7 +0,0 @@
ISC License
Copyright 2026 Marvin Friedrich and contributors
Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
LICENSES/Unlicense.txt
-10
View File
@@ -1,10 +0,0 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.
In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <http://unlicense.org/>
README.md
+23 -13
View File
@@ -1,26 +1,30 @@
<img src="logo.svg" width="10%"/> <img src="zinnia.svg" width="10%"/>
# Zinnia ![GitHub License](https://img.shields.io/github/license/zinnia-os/zinnia?style=flat&color=blue)
![GitHub License](https://img.shields.io/github/license/zinnia-os/zinnia?style=flat&color=green)
![GitHub top language](https://img.shields.io/github/languages/top/zinnia-os/zinnia)
![GitHub Repo stars](https://img.shields.io/github/stars/zinnia-os/zinnia?style=flat) ![GitHub Repo stars](https://img.shields.io/github/stars/zinnia-os/zinnia?style=flat)
![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/zinnia-os/zinnia/ci.yml) ![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/zinnia-os/zinnia/ci.yml)
![GitHub Issues or Pull Requests](https://img.shields.io/github/issues/zinnia-os/zinnia?style=flat) ![GitHub Issues or Pull Requests](https://img.shields.io/github/issues/zinnia-os/zinnia?style=flat)
![Discord](https://img.shields.io/discord/1305893510704267327?logo=discord&logoColor=white&label=discord)
Zinnia is a lightweight Unix-like monolithic kernel targeting 64-bit devices. # zinnia
## Building Zinnia zinnia is a lightweight Unix-like microkernel targeting 64-bit devices.
To build the kernel you need: > [!IMPORTANT]
- `meson` > Please note that this project is a work in progress.
- a `kconfig` frontend (like `kconfiglib` via `pip`) > Some parts may not work at all and/or will drastically change over time.
- a GNU-like C compiler with support for C23 (e.g. `gcc` or `clang`)
## Building
> [!NOTE]
> This repository contains only the kernel and drivers.
> If you want to get a bootable image, you might want to check out
> **https://github.com/zinnia-os/bootstrap** instead.
To build the kernel you need `meson` and a C23-compatible GNU-like compiler.
Currently supported are GCC and Clang toolchains.
To configure, run: To configure, run:
```sh ```sh
alldefconfig
meson setup $build_dir meson setup $build_dir
``` ```
@@ -28,3 +32,9 @@ And to build:
```sh ```sh
meson compile -C $build_dir meson compile -C $build_dir
``` ```
To cross-compile, you should follow the Meson cross-compilation guide.
## Debugging
Follow the debugging guide at **https://github.com/zinnia-os/bootstrap**
arch/x86_64/Kconfig
-17
View File
@@ -1,17 +0,0 @@
menu "x86_64 Architecture options"
config X86_64
bool
default y
select ACPI
select HAVE_MMU
select HAVE_MMU_GENERIC
select HAVE_TLB_SHOOTDOWN_IPI
help
64-bit x86 CPU support for AMD and Intel CPUs.
config TSC
bool "Invariant Timestamp Counter support"
default y
endmenu
arch/x86_64/acpi.c
-36
View File
@@ -1,36 +0,0 @@
#include <stdckdint.h>
#include <stdint.h>
#include <uacpi/kernel_api.h>
#include <uacpi/status.h>
#include <x86_64/asm.h>
uacpi_status uacpi_kernel_io_map(uacpi_io_addr base, uacpi_size len, uacpi_handle* out_handle) {
uintptr_t end;
if (ckd_add(&end, base, len) || end > 0x10000)
return UACPI_STATUS_INVALID_ARGUMENT;
*out_handle = (uacpi_handle)(uintptr_t)base;
return UACPI_STATUS_OK;
}
void uacpi_kernel_io_unmap(uacpi_handle handle) {
// Left intentionally blank.
}
#define IO_READ(width, suffix) \
uacpi_status uacpi_kernel_io_read##width(uacpi_handle handle, uacpi_size offset, uacpi_u##width* out_value) { \
*out_value = asm_in##suffix((uintptr_t)handle + offset); \
return UACPI_STATUS_OK; \
}
#define IO_WRITE(width, suffix) \
uacpi_status uacpi_kernel_io_write##width(uacpi_handle handle, uacpi_size offset, uacpi_u##width in_value) { \
asm_out##suffix((uintptr_t)handle + offset, in_value); \
return UACPI_STATUS_OK; \
}
IO_READ(8, b)
IO_READ(16, w)
IO_READ(32, l)
IO_WRITE(8, b)
IO_WRITE(16, w)
IO_WRITE(32, l)
arch/x86_64/apic.c
-163
View File
@@ -1,163 +0,0 @@
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/clock.h>
#include <zinnia/mmio.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/utils.h>
#include <zinnia/vma.h>
#include <arch/cpu.h>
#include <arch/timer.h>
#include <stdint.h>
#include <x86_64/apic.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
#include <x86_64/percpu.h>
static uint32_t lapic_msr(size_t offset) {
return 0x800 + (offset >> 4);
}
static uint32_t lapic_read_reg(struct local_apic* lapic, mmio_reg32 reg) {
if (lapic->xapic_regs)
return mmio_reg_read(lapic->xapic_regs, reg);
else
return asm_rdmsr(lapic_msr(reg.offset));
}
static void lapic_write_reg(struct local_apic* lapic, mmio_reg32 reg, uint32_t value) {
if (lapic->xapic_regs)
mmio_reg_write(lapic->xapic_regs, reg, value);
else
asm_wrmsr(lapic_msr(reg.offset), value);
}
static uint64_t lapic_read_icr(struct local_apic* lapic) {
if (lapic->xapic_regs) {
uint64_t lo = mmio_reg_read(lapic->xapic_regs, APIC_REG_ICR_LO);
uint64_t hi = mmio_reg_read(lapic->xapic_regs, APIC_REG_ICR_HI);
return (hi << 32) | lo;
} else {
return asm_rdmsr(lapic_msr(APIC_REG_ICR.offset));
}
}
static void lapic_write_icr(struct local_apic* lapic, uint64_t value) {
if (lapic->xapic_regs) {
mmio_reg_write(lapic->xapic_regs, APIC_REG_ICR_HI, (uint32_t)(value >> 32));
mmio_reg_write(lapic->xapic_regs, APIC_REG_ICR_LO, (uint32_t)value);
} else {
asm_wrmsr(lapic_msr(APIC_REG_ICR.offset), value);
}
}
void lapic_init(struct local_apic* lapic) {
uint64_t apic_msr = asm_rdmsr(0x1B);
apic_msr |= 1 << 11; // Enable APIC.
struct cpuid cpuid = asm_cpuid(1, 0);
if (cpuid.ecx & CPUID_1C_X2APIC)
apic_msr |= 1 << 10;
else {
uintptr_t vaddr = vm_alloc(0x1000);
pmap_map(&kernel_vma.pmap, vaddr, apic_msr & 0xFFFF'F000, PTE_READ | PTE_WRITE, CACHE_MMIO);
lapic->xapic_regs = (void*)vaddr;
assert(lapic->xapic_regs, "Failed to allocate virtual memory!");
}
asm_wrmsr(0x1B, apic_msr);
lapic->id = lapic_read_reg(lapic, APIC_REG_ID);
if (lapic->xapic_regs)
lapic->id = mmio_field_get(APIC_ID_XAPIC, lapic->id);
// Reset the TPR.
lapic_write_reg(lapic, APIC_REG_TPR, 0);
// Enable APIC bit in the SIVR.
uint32_t sivr = lapic_read_reg(lapic, APIC_REG_SIVR);
sivr |= mmio_field(APIC_SIVR_VECTOR, IDT_SPURIOUS);
sivr |= mmio_field(APIC_SIVR_ENABLE, 1);
lapic_write_reg(lapic, APIC_REG_SIVR, sivr);
if (lapic->xapic_regs) {
lapic_write_reg(lapic, APIC_REG_DFR, 0xF000'0000);
// Logical destination = LAPIC ID.
lapic_write_reg(lapic, APIC_REG_LDR, lapic_read_reg(lapic, APIC_REG_ID));
}
// TODO: Parse MADT and setup NMI sources.
// Tell the APIC timer to divide by 16.
lapic_write_reg(lapic, APIC_REG_DCR, 3);
// Set the timer to the highest value possible.
lapic_write_reg(lapic, APIC_REG_ICR_TIMER, 0xFFFF'FFFF);
// Sleep for 10 milliseconds.
clock_spin_ns(10'000'000);
lapic->ticks_per_10ms = 0xFFFF'FFFF - lapic_read_reg(lapic, APIC_REG_CCR);
lapic_write_reg(lapic, APIC_REG_LVT_TR, mmio_field(APIC_LVT_TR_VECTOR, IDT_TIMER));
lapic_write_reg(lapic, APIC_REG_ICR_TIMER, 0);
}
void arch_timer_program(uint64_t deadline_ns) {
struct local_apic* lapic = &percpu_ref(arch_data)->lapic;
const uint64_t now = clock_get_elapsed_ns();
const uint64_t delta = MIN(deadline_ns > now ? deadline_ns - now : 0, 1'000'000'000);
const uint64_t ticks = delta * lapic->ticks_per_10ms / 10'000'000;
lapic_write_reg(lapic, APIC_REG_ICR_TIMER, MIN(MAX(ticks, 1), 0xFFFF'FFFF));
}
void lapic_eoi(struct local_apic* lapic) {
lapic_write_reg(lapic, APIC_REG_EOI, 0);
}
void lapic_send_ipi(
struct local_apic* lapic,
uint32_t target,
uint8_t vector,
enum apic_delivery_mode delivery_mode,
enum apic_dest_mode dest_mode,
enum apic_level level,
enum apic_trigger_mode trigger_mode
) {
uint64_t icr = mmio_field(APIC_ICR_VECTOR, vector);
icr |= mmio_field(APIC_ICR_DELIVERY_MODE, delivery_mode);
icr |= mmio_field(APIC_ICR_DEST_MODE, dest_mode);
icr |= mmio_field(APIC_ICR_LEVEL, level);
icr |= mmio_field(APIC_ICR_TRIGGER_MODE, trigger_mode);
if (lapic->xapic_regs)
icr |= mmio_field(APIC_ICR_DEST_XAPIC, target);
else
icr |= mmio_field(APIC_ICR_DEST_X2APIC, target);
lapic_write_icr(lapic, icr);
if (lapic->xapic_regs) {
while (mmio_field_get(APIC_ICR_DELIVERY_STATUS, lapic_read_icr(lapic)) == APIC_DELIVERY_STATUS_PENDING)
arch_cpu_pause();
}
}
constexpr uint16_t PIC1_COMMAND_PORT = 0x20;
constexpr uint16_t PIC1_DATA_PORT = 0x21;
constexpr uint16_t PIC2_COMMAND_PORT = 0xA0;
constexpr uint16_t PIC2_DATA_PORT = 0xA1;
void pic_disable() {
// We initialize the PIC properly, but completely disable it and use the APIC in favor of it.
// Remap IRQs so they start at 0x20 since interrupts 0x00..0x1F are used by CPU exceptions.
asm_outb(PIC1_COMMAND_PORT, 0x11);
asm_outb(PIC1_DATA_PORT, 0x20);
asm_outb(PIC1_DATA_PORT, 0x04);
asm_outb(PIC1_DATA_PORT, 0x01);
asm_outb(PIC1_DATA_PORT, 0xFF);
// Same for the slave PIC.
asm_outb(PIC2_COMMAND_PORT, 0x11);
asm_outb(PIC2_DATA_PORT, 0x28);
asm_outb(PIC2_DATA_PORT, 0x02);
asm_outb(PIC2_DATA_PORT, 0x01);
asm_outb(PIC2_DATA_PORT, 0xFF);
}
arch/x86_64/archctl.c
-32
View File
@@ -1,32 +0,0 @@
#include <zinnia/errno.h>
#include <zinnia/syscall.h>
#include <arch/archctl.h>
#include <arch/mem.h>
#include <uapi/archctl.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
struct syscall_ret arch_archctl(uintptr_t op, uintptr_t arg) {
struct syscall_ret ret = {};
switch (op) {
case __ARCHCTL_SET_FSBASE:
if (!arch_mem_addr_is_user(arg)) {
ret.error = EINVAL;
return ret;
}
asm_wrmsr(MSR_FS_BASE, arg);
return ret;
case __ARCHCTL_SET_GSBASE:
if (!arch_mem_addr_is_user(arg)) {
ret.error = EINVAL;
return ret;
}
// Update the inactive GSBASE MSR, not ours.
asm_wrmsr(MSR_KERNEL_GS_BASE, arg);
return ret;
default:
ret.error = EINVAL;
return ret;
}
}
arch/x86_64/gdt.c
-77
View File
@@ -1,77 +0,0 @@
#include <zinnia/percpu.h>
#include <stddef.h>
#include <string.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
#include <x86_64/gdt.h>
#include <x86_64/percpu.h>
#include <x86_64/tss.h>
const struct gdt gdt_base = {
.null = {0},
.kernel_code32 = {.val = 0x00cf9b000000ffff},
.kernel_data32 = {.val = 0x00cf93000000ffff},
.kernel_code64 = {.val = 0x00a09b0000000000},
.kernel_data64 = {.val = 0x0000930000000000},
.user_code32 = {.val = 0x00cffb000000ffff},
.user_data = {.val = 0x0000f30000000000},
.user_code64 = {.val = 0x00a0fb0000000000},
.tss = {
.limit0 = sizeof(struct tss) - 1,
.access = 0x89, // Present | DPL=0 | System | 64-bit TSS
.limit1_flags = ((sizeof(struct tss) - 1) >> 16) & 0x0F,
.reserved = 0,
},
};
void gdt_load() {
struct gdt* gdt = &percpu_ref(arch_data)->gdt;
memcpy(gdt, &gdt_base, sizeof(gdt_base));
struct tss* tss = &percpu_ref(arch_data)->tss;
tss->iopb = sizeof(struct tss);
uintptr_t tss_addr = (uintptr_t)tss;
gdt->tss.base0 = tss_addr;
gdt->tss.base1 = tss_addr >> 16;
gdt->tss.base2 = tss_addr >> 24;
gdt->tss.base3 = tss_addr >> 32;
struct gdtr gdtr = {
.limit = sizeof(struct gdt) - 1,
.base = gdt,
};
asm volatile("lgdt %0" ::"m"(gdtr));
// Save the contents of MSR_GS_BASE, as they get cleared by a write to `gs`.
uintptr_t gs = asm_rdmsr(MSR_GS_BASE);
// Flush and reload the segment registers.
asm volatile(
"push %0\n"
"lea rax, [rip + 1f]\n"
"push rax\n"
"retfq\n"
"1:\n"
"mov ax, %1\n"
"mov ds, ax\n"
"mov es, ax\n"
"mov fs, ax\n"
"mov gs, ax\n"
"mov ss, ax\n"
:
: "i"(offsetof(struct gdt, kernel_code64)), "i"(offsetof(struct gdt, kernel_data64))
: "rax", "memory"
);
asm_wrmsr(MSR_GS_BASE, gs);
asm volatile(
"mov ax, %c0\n"
"ltr ax"
:
: "i"(offsetof(struct gdt, tss))
: "rax", "memory"
);
}
arch/x86_64/hpet.c
-85
View File
@@ -1,85 +0,0 @@
#include <zinnia/acpi.h>
#include <zinnia/assert.h>
#include <zinnia/clock.h>
#include <zinnia/cmdline.h>
#include <zinnia/initgraph.h>
#include <zinnia/mmio.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/print.h>
#include <zinnia/utils.h>
#include <zinnia/vma.h>
#include <uacpi/acpi.h>
#include <uacpi/status.h>
#include <uacpi/tables.h>
#include <uacpi/uacpi.h>
static bool use_hpet = true;
static void hpet_option(bool use) {
use_hpet = use;
}
CMDLINE_OPTION("x86_64.hpet", "Enable High Precision Event Timer clock", hpet_option);
struct hpet {
struct clock clock;
void* regs;
uint32_t period;
};
constexpr mmio_reg64 HPET_REG_CAP = {.offset = 0x0};
constexpr mmio_field64 HPET_CAP_PERIOD = {HPET_REG_CAP, 32, 32};
constexpr mmio_reg64 HPET_REG_CFG = {.offset = 0x10};
constexpr mmio_field64 HPET_CFG_ENABLE = {HPET_REG_CFG, 0, 1};
constexpr mmio_reg64 HPET_REG_COUNTER = {.offset = 0xF0};
static void hpet_reset(struct clock* c) {
struct hpet* hpet = containerof(c, struct hpet, clock);
mmio_reg_write(hpet->regs, HPET_REG_COUNTER, 0);
}
static uint64_t hpet_get_elapsed_ns(struct clock* c) {
struct hpet* hpet = containerof(c, struct hpet, clock);
uint64_t counter = mmio_reg_read(hpet->regs, HPET_REG_COUNTER);
return (uint64_t)((__uint128_t)counter * (__uint128_t)hpet->period / 1'000'000);
}
static struct hpet hpet_clock = {
.clock = {
.name = "hpet",
.priority = 128,
.reset = hpet_reset,
.get_elapsed_ns = hpet_get_elapsed_ns,
},
};
static void hpet_init(struct boot_info*) {
if (!use_hpet)
return;
uacpi_table table = {0};
if (uacpi_table_find_by_signature("HPET", &table) != UACPI_STATUS_OK) {
log("x86_64/hpet: No HPET table found\n");
return;
}
struct acpi_hpet* hpet = table.ptr;
uintptr_t vaddr = vm_alloc(0x1000);
pmap_map(&kernel_vma.pmap, vaddr, hpet->address.address, PTE_READ | PTE_WRITE, CACHE_MMIO);
hpet_clock.regs = (void*)vaddr;
uacpi_table_unref(&table);
log("x86_64/hpet: HPET registers mapped at %p\n", hpet_clock.regs);
uint64_t cfg = mmio_reg_read(hpet_clock.regs, HPET_REG_CFG);
cfg |= mmio_field(HPET_CFG_ENABLE, 1);
mmio_reg_write(hpet_clock.regs, HPET_REG_CFG, cfg);
uint64_t cap = mmio_reg_read(hpet_clock.regs, HPET_REG_CAP);
hpet_clock.period = mmio_field_get(HPET_CAP_PERIOD, cap);
assert(clock_switch(&hpet_clock.clock), "Unable to switch to HPET!");
}
INIT_TASK(hpet_stage, "arch.x86_64.hpet", hpet_init);
INIT_DEPENDS(hpet_stage, acpi_stage);
INIT_ENTAILS(hpet_stage, percpu_stage);
arch/x86_64/idt.c
-270
View File
@@ -1,270 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/assert.h>
#include <zinnia/compiler.h>
#include <zinnia/irq.h>
#include <zinnia/panic.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/print.h>
#include <zinnia/sched.h>
#include <zinnia/timer.h>
#include <zinnia/usercopy.h>
#include <zinnia/utils.h>
#include <zinnia/vma.h>
#include <arch/sched.h>
#include <bits/sched.h>
#include <stddef.h>
#include <stdint.h>
#include <x86_64/apic.h>
#include <x86_64/defs.h>
#include <x86_64/gdt.h>
#include <x86_64/percpu.h>
struct [[__packed]] idt_entry {
uint16_t base0;
uint16_t selector;
uint8_t ist;
uint8_t attrib;
uint16_t base1;
uint32_t base2;
uint32_t reserved;
};
static_assert(sizeof(struct idt_entry) == 16);
struct [[__packed]] idtr {
uint16_t limit;
struct idt_entry* base;
};
static_assert(sizeof(struct idtr) == 10);
void interrupt_handler(struct arch_context* ctx) {
struct percpu* cpu = percpu_get();
struct arch_percpu* arch_cpu = percpu_ref(arch_data);
uint64_t isr = ctx->isr;
switch (isr) {
case IDT_PF:
uint64_t cr2;
asm volatile("mov %0, cr2" : "=r"(cr2));
const struct vma_fault_info info = {
.ip = ctx->rip,
.addr = cr2,
.caused_by_user = (ctx->cs & CPL_USER) == CPL_USER,
.caused_by_write = ctx->error & (1 << 1),
.caused_by_fetch = ctx->error & (1 << 4),
.page_was_present = ctx->error & (1 << 0),
};
struct task* current = cpu->sched.current;
const bool irqs_were_on = ctx->rflags & RFLAGS_IF;
if (irqs_were_on)
irq_set_state(true);
const bool handled = current && current->space && vma_page_fault(current->space, &info);
if (irqs_were_on)
irq_set_state(false);
if (handled)
break;
if (current && current->usercopy_region) {
struct usercopy_region* region = current->usercopy_region;
if (ctx->rip >= (uintptr_t)region->start_ip && ctx->rip < (uintptr_t)region->end_ip) {
ctx->rip = (uintptr_t)region->fault_ip;
break;
}
}
panic(
"Unrecoverable page fault. Attempted to %s a %s page at %#lx (IP: %#lx)\n",
info.caused_by_write ? "write to"
: info.caused_by_fetch ? "execute on"
: "read from",
info.page_was_present ? "present" : "non-present",
info.addr,
info.ip
);
case IDT_SPURIOUS:
break;
case IDT_IPI_RESCHED:
lapic_eoi(&arch_cpu->lapic);
arch_sched_preempt_disable();
if (arch_sched_preempt_enable())
sched_reschedule(&cpu->sched);
break;
case IDT_IPI_SHOOTDOWN:
lapic_eoi(&arch_cpu->lapic);
pmap_shootdown_handle();
break;
case IDT_TIMER:
lapic_eoi(&arch_cpu->lapic);
timer_interrupt();
if (cpu->sched.need_resched) {
arch_sched_preempt_disable();
if (arch_sched_preempt_enable())
sched_reschedule(&cpu->sched);
}
break;
default:
if (isr < 0x20)
panic("Unhandled exception %lu at %#lx, error: %#lx\n", isr, ctx->rip, ctx->error);
assert(isr - 0x20 < ARRAY_SIZE(arch_cpu->irq_lines), "Unhandled interrupt on ISR %lu", isr);
struct irq_line* line = arch_cpu->irq_lines[isr - 0x20];
assert(line, "Unhandled interrupt on ISR %lu", isr);
irq_line_raise(line);
}
}
#define CS_OFFSET (sizeof(struct arch_context) - sizeof(uint64_t) - offsetof(struct arch_context, cs))
[[__naked]]
void interrupt_return() {
asm volatile(
"pop r15\n"
"pop r14\n"
"pop r13\n"
"pop r12\n"
"pop r11\n"
"pop r10\n"
"pop r9\n"
"pop r8\n"
"pop rsi\n"
"pop rdi\n"
"pop rbp\n"
"pop rdx\n"
"pop rcx\n"
"pop rbx\n"
"pop rax\n"
// Change GS back if we came from user mode.
"cmp word ptr [rsp + %c0], %c1\n"
"je 2f\n"
"swapgs\n"
"2:\n"
// Skip .error and .isr fields.
"add rsp, 0x10\n"
"iretq\n"
:
: "i"(CS_OFFSET), "i"(offsetof(struct gdt, kernel_code64))
);
}
[[__naked]]
void interrupt_stub_internal() {
asm volatile(
// Load the kernel GS base if we're coming from user space.
"cmp word ptr [rsp + %c0], %c1\n"
"je 2f\n"
"swapgs\n"
"2:\n"
"push rax\n"
"push rbx\n"
"push rcx\n"
"push rdx\n"
"push rbp\n"
"push rdi\n"
"push rsi\n"
"push r8\n"
"push r9\n"
"push r10\n"
"push r11\n"
"push r12\n"
"push r13\n"
"push r14\n"
"push r15\n"
"cld\n"
// Zero out the base pointer since we can't trust it.
"xor rbp, rbp\n"
// Load the frame as first argument.
"mov rdi, rsp\n"
"call %c2\n"
"jmp %c3\n"
:
: "i"(CS_OFFSET), "i"(offsetof(struct gdt, kernel_code64)), "i"(interrupt_handler), "i"(interrupt_return)
);
}
// clang-format off
#define REPEAT_256(M) \
M(0) M(1) M(2) M(3) M(4) M(5) M(6) M(7) \
M(8) M(9) M(10) M(11) M(12) M(13) M(14) M(15) \
M(16) M(17) M(18) M(19) M(20) M(21) M(22) M(23) \
M(24) M(25) M(26) M(27) M(28) M(29) M(30) M(31) \
M(32) M(33) M(34) M(35) M(36) M(37) M(38) M(39) \
M(40) M(41) M(42) M(43) M(44) M(45) M(46) M(47) \
M(48) M(49) M(50) M(51) M(52) M(53) M(54) M(55) \
M(56) M(57) M(58) M(59) M(60) M(61) M(62) M(63) \
M(64) M(65) M(66) M(67) M(68) M(69) M(70) M(71) \
M(72) M(73) M(74) M(75) M(76) M(77) M(78) M(79) \
M(80) M(81) M(82) M(83) M(84) M(85) M(86) M(87) \
M(88) M(89) M(90) M(91) M(92) M(93) M(94) M(95) \
M(96) M(97) M(98) M(99) M(100) M(101) M(102) M(103) \
M(104) M(105) M(106) M(107) M(108) M(109) M(110) M(111) \
M(112) M(113) M(114) M(115) M(116) M(117) M(118) M(119) \
M(120) M(121) M(122) M(123) M(124) M(125) M(126) M(127) \
M(128) M(129) M(130) M(131) M(132) M(133) M(134) M(135) \
M(136) M(137) M(138) M(139) M(140) M(141) M(142) M(143) \
M(144) M(145) M(146) M(147) M(148) M(149) M(150) M(151) \
M(152) M(153) M(154) M(155) M(156) M(157) M(158) M(159) \
M(160) M(161) M(162) M(163) M(164) M(165) M(166) M(167) \
M(168) M(169) M(170) M(171) M(172) M(173) M(174) M(175) \
M(176) M(177) M(178) M(179) M(180) M(181) M(182) M(183) \
M(184) M(185) M(186) M(187) M(188) M(189) M(190) M(191) \
M(192) M(193) M(194) M(195) M(196) M(197) M(198) M(199) \
M(200) M(201) M(202) M(203) M(204) M(205) M(206) M(207) \
M(208) M(209) M(210) M(211) M(212) M(213) M(214) M(215) \
M(216) M(217) M(218) M(219) M(220) M(221) M(222) M(223) \
M(224) M(225) M(226) M(227) M(228) M(229) M(230) M(231) \
M(232) M(233) M(234) M(235) M(236) M(237) M(238) M(239) \
M(240) M(241) M(242) M(243) M(244) M(245) M(246) M(247) \
M(248) M(249) M(250) M(251) M(252) M(253) M(254) M(255)
// clang-format on
#define INTERRUPT_STUB(n) \
[[__naked]] \
static void interrupt_stub_##n() { \
asm volatile( \
".if (%c0 == 8 || (%c0 >= 10 && %c0 <= 14) || %c0 == 17 || %c0 == 21 || %c0 == 29 || %c0 == 30)\n" \
".else\n" \
"push 0\n" \
".endif\n" \
"push %c0\n" \
"jmp %c1\n" \
: \
: "i"(n), "i"(interrupt_stub_internal) \
); \
}
#define INTERRUPT_SETUP(n) \
do { \
uint64_t addr = (uint64_t)interrupt_stub_##n; \
idt[n].base0 = addr & 0xFFFF; \
idt[n].base1 = (addr >> 16) & 0xFFFF; \
idt[n].base2 = (addr >> 32) & 0xFFFF'FFFF; \
idt[n].ist = 0; \
idt[n].attrib = (1 << 7) | 0xE; \
idt[n].selector = offsetof(struct gdt, kernel_code64); \
idt[n].reserved = 0; \
} while (0);
REPEAT_256(INTERRUPT_STUB)
static struct idt_entry idt[256] = {};
void idt_init() {
REPEAT_256(INTERRUPT_SETUP)
idt[IDT_BP].attrib |= CPL_USER << 5;
}
void idt_load() {
struct idtr idtr = {
.limit = sizeof(idt) - 1,
.base = idt,
};
asm volatile("lidt %0" ::"m"(idtr));
}
arch/x86_64/include/bits/cpu.h
-9
View File
@@ -1,9 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
static inline void arch_cpu_pause() {
asm volatile("pause");
}
#define ARCH_CACHE_LINE_SIZE 64
arch/x86_64/include/bits/elf.h
-36
View File
@@ -1,36 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/elf.h>
#define ELF_ARCH_CLASS ELFCLASS64
#define ELF_ARCH_DATA ELFDATA2LSB
#define ELF_ARCH_MACHINE EM_X86_64
#define ELF_R_SYM ELF64_R_SYM
#define ELF_R_TYPE ELF64_R_TYPE
#define ELF_ST_BIND ELF64_ST_BIND
#define ELF_ST_TYPE ELF64_ST_TYPE
typedef elf64_ehdr_t elf_ehdr_t;
typedef elf64_phdr_t elf_phdr_t;
typedef elf64_dyn_t elf_dyn_t;
typedef elf64_addr_t elf_addr_t;
typedef elf64_off_t elf_off_t;
typedef elf64_nhdr_t elf_nhdr_t;
typedef elf64_auxv_t elf_auxv_t;
typedef elf64_sym_t elf_sym_t;
typedef elf64_rela_t elf_rela_t;
typedef elf64_rel_t elf_rel_t;
typedef elf64_relr_t elf_relr_t;
enum {
R_NONE = 0,
R_64 = 1,
R_COPY = 5,
R_GLOB_DAT = 6,
R_JUMP_SLOT = 7,
R_RELATIVE = 8,
R_IRELATIVE = 37,
};
arch/x86_64/include/bits/pmap.h
-7
View File
@@ -1,7 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <stdint.h>
typedef uint64_t pte_t;
arch/x86_64/include/x86_64/apic.h
-83
View File
@@ -1,83 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/mmio.h>
#include <stdint.h>
struct local_apic {
uint32_t id;
uint32_t ticks_per_10ms;
void* xapic_regs;
};
enum apic_delivery_mode {
APIC_DELIVERY_FIXED = 0b000,
APIC_DELIVERY_LOWESTPRIO = 0b001,
APIC_DELIVERY_SMI = 0b010,
APIC_DELIVERY_NMI = 0b100,
APIC_DELIVERY_INIT = 0b101,
APIC_DELIVERY_STARTUP = 0b110,
};
enum apic_dest_mode {
APIC_DEST_PHYSICAL = 0,
APIC_DEST_LOGICAL = 1,
};
enum apic_delivery_status {
APIC_DELIVERY_STATUS_IDLE = 0,
APIC_DELIVERY_STATUS_PENDING = 1,
};
enum apic_level {
APIC_LEVEL_DEASSERT = 0,
APIC_LEVEL_ASSERT = 1,
};
enum apic_trigger_mode {
APIC_TRIGGER_EDGE = 0,
APIC_TRIGGER_LEVEL = 1,
};
constexpr mmio_reg32 APIC_REG_ID = {.offset = 0x20};
constexpr mmio_field32 APIC_ID_XAPIC = {APIC_REG_ID, 24, 8};
constexpr mmio_reg32 APIC_REG_TPR = {.offset = 0x80};
constexpr mmio_reg32 APIC_REG_EOI = {.offset = 0xB0};
constexpr mmio_reg32 APIC_REG_LDR = {.offset = 0xD0};
constexpr mmio_reg32 APIC_REG_DFR = {.offset = 0xE0};
constexpr mmio_reg32 APIC_REG_SIVR = {.offset = 0xF0};
constexpr mmio_field32 APIC_SIVR_VECTOR = {APIC_REG_SIVR, 0, 8};
constexpr mmio_field32 APIC_SIVR_ENABLE = {APIC_REG_SIVR, 8, 1};
constexpr mmio_reg32 APIC_REG_ESR = {.offset = 0x280};
constexpr mmio_reg64 APIC_REG_ICR = {.offset = 0x300};
constexpr mmio_field64 APIC_ICR_VECTOR = {APIC_REG_ICR, 0, 8};
constexpr mmio_field64 APIC_ICR_DELIVERY_MODE = {APIC_REG_ICR, 8, 3};
constexpr mmio_field64 APIC_ICR_DEST_MODE = {APIC_REG_ICR, 11, 1};
constexpr mmio_field64 APIC_ICR_DELIVERY_STATUS = {APIC_REG_ICR, 12, 1};
constexpr mmio_field64 APIC_ICR_LEVEL = {APIC_REG_ICR, 14, 1};
constexpr mmio_field64 APIC_ICR_TRIGGER_MODE = {APIC_REG_ICR, 15, 1};
constexpr mmio_field64 APIC_ICR_DEST_XAPIC = {APIC_REG_ICR, 56, 8};
constexpr mmio_field64 APIC_ICR_DEST_X2APIC = {APIC_REG_ICR, 32, 32};
constexpr mmio_reg32 APIC_REG_ICR_LO = {.offset = 0x300};
constexpr mmio_reg32 APIC_REG_ICR_HI = {.offset = 0x310};
constexpr mmio_reg32 APIC_REG_LVT_TR = {.offset = 0x320};
constexpr mmio_field32 APIC_LVT_TR_VECTOR = {APIC_REG_LVT_TR, 0, 8};
constexpr mmio_reg32 APIC_REG_ICR_TIMER = {.offset = 0x380};
constexpr mmio_reg32 APIC_REG_CCR = {.offset = 0x390};
constexpr mmio_reg32 APIC_REG_DCR = {.offset = 0x3E0};
void lapic_init(struct local_apic* lapic);
void lapic_eoi(struct local_apic* lapic);
void lapic_send_ipi(
struct local_apic* lapic,
uint32_t target,
uint8_t vector,
enum apic_delivery_mode delivery_mode,
enum apic_dest_mode dest_mode,
enum apic_level level,
enum apic_trigger_mode trigger_mode
);
void pic_disable();
arch/x86_64/include/x86_64/defs.h
-257
View File
@@ -1,257 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <stdint.h>
enum : uint32_t {
CPUID_1C_SSE3 = 1u << 0,
CPUID_1C_PCLMUL = 1u << 1,
CPUID_1C_DTES64 = 1u << 2,
CPUID_1C_MONITOR = 1u << 3,
CPUID_1C_DS_CPL = 1u << 4,
CPUID_1C_VMX = 1u << 5,
CPUID_1C_SMX = 1u << 6,
CPUID_1C_EST = 1u << 7,
CPUID_1C_TM2 = 1u << 8,
CPUID_1C_SSSE3 = 1u << 9,
CPUID_1C_CID = 1u << 10,
CPUID_1C_SDBG = 1u << 11,
CPUID_1C_FMA = 1u << 12,
CPUID_1C_CX16 = 1u << 13,
CPUID_1C_XTPR = 1u << 14,
CPUID_1C_PDCM = 1u << 15,
CPUID_1C_PCID = 1u << 17,
CPUID_1C_DCA = 1u << 18,
CPUID_1C_SSE4_1 = 1u << 19,
CPUID_1C_SSE4_2 = 1u << 20,
CPUID_1C_X2APIC = 1u << 21,
CPUID_1C_MOVBE = 1u << 22,
CPUID_1C_POPCNT = 1u << 23,
CPUID_1C_TSC = 1u << 24,
CPUID_1C_AES = 1u << 25,
CPUID_1C_XSAVE = 1u << 26,
CPUID_1C_OSXSAVE = 1u << 27,
CPUID_1C_AVX = 1u << 28,
CPUID_1C_F16C = 1u << 29,
CPUID_1C_RDRAND = 1u << 30,
CPUID_1C_HYPERVISOR = 1u << 31,
};
enum : uint32_t {
CPUID_1D_FPU = 1u << 0,
CPUID_1D_VME = 1u << 1,
CPUID_1D_DE = 1u << 2,
CPUID_1D_PSE = 1u << 3,
CPUID_1D_TSC = 1u << 4,
CPUID_1D_MSR = 1u << 5,
CPUID_1D_PAE = 1u << 6,
CPUID_1D_MCE = 1u << 7,
CPUID_1D_CX8 = 1u << 8,
CPUID_1D_APIC = 1u << 9,
CPUID_1D_SEP = 1u << 11,
CPUID_1D_MTRR = 1u << 12,
CPUID_1D_PGE = 1u << 13,
CPUID_1D_MCA = 1u << 14,
CPUID_1D_CMOV = 1u << 15,
CPUID_1D_PAT = 1u << 16,
CPUID_1D_PSE36 = 1u << 17,
CPUID_1D_PSN = 1u << 18,
CPUID_1D_CLFLUSH = 1u << 19,
CPUID_1D_DS = 1u << 21,
CPUID_1D_ACPI = 1u << 22,
CPUID_1D_MMX = 1u << 23,
CPUID_1D_FXSR = 1u << 24,
CPUID_1D_SSE = 1u << 25,
CPUID_1D_SSE2 = 1u << 26,
CPUID_1D_SS = 1u << 27,
CPUID_1D_HTT = 1u << 28,
CPUID_1D_TM = 1u << 29,
CPUID_1D_IA64 = 1u << 30,
CPUID_1D_PBE = 1u << 31,
};
enum : uint32_t {
CPUID_7B_FSGSBASE = 1u << 0,
CPUID_7B_SGX = 1u << 2,
CPUID_7B_BMI = 1u << 3,
CPUID_7B_HLE = 1u << 4,
CPUID_7B_AVX2 = 1u << 5,
CPUID_7B_SMEP = 1u << 7,
CPUID_7B_BMI2 = 1u << 8,
CPUID_7B_RTM = 1u << 11,
CPUID_7B_AVX512F = 1u << 16,
CPUID_7B_AVX512DQ = 1u << 17,
CPUID_7B_RDSEED = 1u << 18,
CPUID_7B_ADX = 1u << 19,
CPUID_7B_SMAP = 1u << 20,
CPUID_7B_AVX512IFMA = 1u << 21,
CPUID_7B_CLFLUSHOPT = 1u << 23,
CPUID_7B_CLWB = 1u << 24,
CPUID_7B_AVX512CD = 1u << 28,
CPUID_7B_SHA = 1u << 29,
CPUID_7B_AVX512BW = 1u << 30,
CPUID_7B_AVX512VL = 1u << 31,
};
enum : uint32_t {
CPUID_7C_AVX512VBMI = 1u << 1,
CPUID_7C_UMIP = 1u << 2,
CPUID_7C_PKU = 1u << 3,
CPUID_7C_OSPKE = 1u << 4,
CPUID_7C_WAITPKG = 1u << 5,
CPUID_7C_AVX512VBMI2 = 1u << 6,
CPUID_7C_SHSTK = 1u << 7,
CPUID_7C_GFNI = 1u << 8,
CPUID_7C_VAES = 1u << 9,
CPUID_7C_VPCLMULQDQ = 1u << 10,
CPUID_7C_AVX512VNNI = 1u << 11,
CPUID_7C_AVX512BITALG = 1u << 12,
CPUID_7C_TME_EN = 1u << 13,
CPUID_7C_AVX512VPOPCNTDQ = 1u << 14,
CPUID_7C_RDPID = 1u << 22,
CPUID_7C_KL = 1u << 23,
CPUID_7C_CLDEMOTE = 1u << 25,
CPUID_7C_MOVDIRI = 1u << 27,
CPUID_7C_MOVDIR64B = 1u << 28,
CPUID_7C_ENQCMD = 1u << 29,
};
enum : uint32_t {
CPUID_7D_UINTR = 1u << 5,
CPUID_7D_AVX512VP2INTERSECT = 1u << 8,
CPUID_7D_SERIALIZE = 1u << 14,
CPUID_7D_TSXLDTRK = 1u << 16,
CPUID_7D_PCONFIG = 1u << 18,
CPUID_7D_IBT = 1u << 20,
CPUID_7D_AMX_BF16 = 1u << 22,
CPUID_7D_AVX512FP16 = 1u << 23,
CPUID_7D_AMX_TILE = 1u << 24,
CPUID_7D_AMX_INT8 = 1u << 25,
};
enum : uint64_t {
RFLAGS_CF = 1ull << 0,
RFLAGS_BRKI = 1ull << 1,
RFLAGS_PF = 1ull << 2,
RFLAGS_AF = 1ull << 4,
RFLAGS_ZF = 1ull << 6,
RFLAGS_SF = 1ull << 7,
RFLAGS_TF = 1ull << 8,
RFLAGS_IF = 1ull << 9,
RFLAGS_DF = 1ull << 10,
RFLAGS_OF = 1ull << 11,
RFLAGS_IOPL = 3ull << 12,
RFLAGS_NT = 1ull << 14,
RFLAGS_RF = 1ull << 16,
RFLAGS_VM = 1ull << 17,
RFLAGS_AC = 1ull << 18,
RFLAGS_VIF = 1ull << 19,
RFLAGS_VIP = 1ull << 20,
RFLAGS_ID = 1ull << 21,
};
enum : uint64_t {
XCR0_X87 = 1ull << 0,
XCR0_SSE = 1ull << 1,
XCR0_AVX = 1ull << 2,
XCR0_AXV512_OPMASK = 1ull << 5,
XCR0_AVX512_ZMM_HI256 = 1ull << 6,
XCR0_AVX512_HI16_ZMM = 1ull << 7,
};
enum : uint64_t {
CR0_PE = 1ull << 0,
CR0_MP = 1ull << 1,
CR0_EM = 1ull << 2,
CR0_TS = 1ull << 3,
CR0_ET = 1ull << 4,
CR0_NE = 1ull << 5,
CR0_WP = 1ull << 16,
CR0_AM = 1ull << 18,
CR0_NW = 1ull << 29,
CR0_CD = 1ull << 30,
CR0_PG = 1ull << 31,
};
enum : uint64_t {
CR4_VME = 1ull << 0,
CR4_PVI = 1ull << 1,
CR4_TSD = 1ull << 2,
CR4_DE = 1ull << 3,
CR4_PSE = 1ull << 4,
CR4_PAE = 1ull << 5,
CR4_MCE = 1ull << 6,
CR4_PGE = 1ull << 7,
CR4_PCE = 1ull << 8,
CR4_OSFXSR = 1ull << 9,
CR4_OSXMMEXCPT = 1ull << 10,
CR4_UMIP = 1ull << 11,
CR4_VMXE = 1ull << 13,
CR4_SMXE = 1ull << 14,
CR4_FSGSBASE = 1ull << 16,
CR4_PCIDE = 1ull << 17,
CR4_OSXSAVE = 1ull << 18,
CR4_SMEP = 1ull << 20,
CR4_SMAP = 1ull << 21,
CR4_PKE = 1ull << 22,
CR4_CET = 1ull << 23,
CR4_PKS = 1ull << 24,
};
enum : uint64_t {
MSR_EFER_SCE = 1ull << 0,
MSR_EFER_LME = 1ull << 8,
MSR_EFER_LMA = 1ull << 10,
MSR_EFER_NXE = 1ull << 11,
MSR_EFER_SVME = 1ull << 12,
MSR_EFER_LMSLE = 1ull << 13,
MSR_EFER_FFXSR = 1ull << 14,
MSR_EFER_TCE = 1ull << 15,
};
enum : uint32_t {
MSR_PAT = 0x277,
MSR_EFER = 0xC0000080,
MSR_STAR = 0xC0000081,
MSR_LSTAR = 0xC0000082,
MSR_CSTAR = 0xC0000083,
MSR_SFMASK = 0xC0000084,
MSR_FS_BASE = 0xC0000100,
MSR_GS_BASE = 0xC0000101,
MSR_KERNEL_GS_BASE = 0xC0000102,
};
enum : uint8_t {
PAT_UC = 0x0,
PAT_WC = 0x1,
PAT_WT = 0x4,
PAT_WP = 0x5,
PAT_WB = 0x6,
PAT_UCM = 0x7,
};
enum : uint8_t {
CPL_USER = 0x3,
CPL_KERNEL = 0x0,
};
enum : uint8_t {
IDT_DE = 0x0,
IDT_DB = 0x1,
IDT_BP = 0x3,
IDT_OF = 0x4,
IDT_BR = 0x5,
IDT_UD = 0x6,
IDT_NM = 0x7,
IDT_SS = 0xC,
IDT_GP = 0xD,
IDT_PF = 0xE,
IDT_MF = 0x10,
IDT_AC = 0x11,
IDT_XM = 0x13,
IDT_IPI_SHOOTDOWN = 0xFC,
IDT_TIMER = 0xFD,
IDT_SPURIOUS = 0xFE,
IDT_IPI_RESCHED = 0xFF,
};
arch/x86_64/include/x86_64/gdt.h
-58
View File
@@ -1,58 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/compiler.h>
#include <stdint.h>
union gdt_desc {
uint64_t val;
struct {
uint16_t limit0;
uint16_t base0;
uint8_t base1;
uint8_t access;
uint8_t limit1_flags;
uint8_t base2;
};
};
static_assert(sizeof(union gdt_desc) == 0x8);
union gdt_long_desc {
uint64_t val[2];
struct {
uint16_t limit0;
uint16_t base0;
uint8_t base1;
uint8_t access;
uint8_t limit1_flags;
uint8_t base2;
uint32_t base3;
uint32_t reserved;
};
};
static_assert(sizeof(union gdt_long_desc) == 0x10);
struct [[__packed]] gdt {
union gdt_desc null;
union gdt_desc kernel_code32;
union gdt_desc kernel_data32;
union gdt_desc kernel_code64;
union gdt_desc kernel_data64;
union gdt_desc user_code32;
union gdt_desc user_data;
union gdt_desc user_code64;
union gdt_long_desc tss;
};
struct [[__packed]] gdtr {
uint16_t limit;
struct gdt* base;
};
extern const struct gdt gdt_base;
// Initializes a GDT on the local CPU.
void gdt_load();
arch/x86_64/include/x86_64/idt.h
-13
View File
@@ -1,13 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/init.h>
[[__init]]
void idt_init();
// Loads the IDT on this CPU.
void idt_load();
void interrupt_return();
arch/x86_64/include/x86_64/percpu.h
-26
View File
@@ -1,26 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/irq.h>
#include <zinnia/percpu.h>
#include <stddef.h>
#include <x86_64/apic.h>
#include <x86_64/gdt.h>
#include <x86_64/tss.h>
struct arch_percpu {
struct gdt gdt;
struct tss tss;
struct local_apic lapic;
struct irq_line* irq_lines[128];
size_t fpu_size;
void (*fpu_save)(void*);
void (*fpu_restore)(const void*);
bool can_smap;
};
extern struct arch_percpu arch_data;
// Initializes the calling CPU and makes `cpu` its per-CPU block.
void x86_64_cpu_setup(struct percpu* cpu);
arch/x86_64/include/x86_64/pmap.h
-21
View File
@@ -1,21 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/pmap.h>
enum : pte_t {
ARCH_FLAG_PRESENT = 1 << 0,
ARCH_FLAG_READ_WRITE = 1 << 1,
ARCH_FLAG_USER_MODE = 1 << 2,
ARCH_FLAG_WRITE_THROUGH = 1 << 3,
ARCH_FLAG_CACHE_DISABLE = 1 << 4,
ARCH_FLAG_ACCESSED = 1 << 5,
ARCH_FLAG_DIRTY = 1 << 6,
ARCH_FLAG_SIZE = 1 << 7,
ARCH_FLAG_GLOBAL = 1 << 8,
ARCH_FLAG_AVAILABLE = 1 << 9,
ARCH_FLAG_ATTRIBUTE_TABLE = 1 << 7,
ARCH_FLAG_EXECUTE_DISABLE = 1LU << 63,
ARCH_PTE_ADDR_MASK = 0x000F'FFFF'FFFF'F000,
};
arch/x86_64/include/x86_64/smp.h
-26
View File
@@ -1,26 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/atomic.h>
#include <zinnia/compiler.h>
#include <zinnia/utils.h>
#include <stdint.h>
#include <x86_64/gdt.h>
struct [[__packed]] smp_info {
struct gdt gdt;
struct gdtr gdtr;
uint32_t farjmp_offset;
uint32_t farjmp_segment;
uint32_t temp_stack;
uint32_t temp_cr3;
uint64_t entry;
uint64_t hhdm_offset;
uint64_t percpu;
uint32_t lapic_id;
atomic_bool booted;
};
// Discovers all APs and starts them.
void x86_64_smp_init();
arch/x86_64/include/x86_64/syscall.h
-5
View File
@@ -1,5 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
void arch_syscall_stub();
arch/x86_64/include/x86_64/tsc.h
-5
View File
@@ -1,5 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
void tsc_init();
arch/x86_64/irq.c
-95
View File
@@ -1,95 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/percpu.h>
#include <zinnia/utils.h>
#include <arch/irq.h>
#include <stdint.h>
#include <x86_64/apic.h>
#include <x86_64/defs.h>
#include <x86_64/percpu.h>
bool arch_irq_get_state() {
uint64_t flags;
asm volatile("pushf; pop %0" : "=r"(flags)::"memory");
return flags & RFLAGS_IF;
}
bool arch_irq_set_state(bool state) {
bool old = arch_irq_get_state();
if (state)
asm volatile("sti" ::: "memory");
else
asm volatile("cli" ::: "memory");
return old;
}
void arch_irq_wait() {
asm volatile("hlt");
}
struct apic_msi_line {
struct msi_line msi;
uint8_t vector;
uint32_t lapic;
};
static phys_t apic_msi_msg_addr(struct msi_line* self) {
struct apic_msi_line* line = containerof(self, struct apic_msi_line, msi);
return 0xFEE0'0000 | ((phys_t)line->lapic << 12);
}
static uint32_t apic_msi_msg_data(struct msi_line* self) {
return containerof(self, struct apic_msi_line, msi)->vector;
}
static enum irq_mode apic_msi_set_config(
struct irq_line* self,
enum irq_trigger_mode trigger,
enum irq_polarity polarity
) {
assert(trigger == IRQ_TRIGGER_EDGE, "MSIs must be edge triggered");
return IRQ_MODE_EOI;
}
static void apic_msi_mask(struct irq_line* self) {}
static void apic_msi_unmask(struct irq_line* self) {}
static void apic_msi_eoi(struct irq_line* self) {
lapic_eoi(&percpu_ref(arch_data)->lapic);
}
errno_t arch_irq_alloc_msi(struct msi_line** out) {
struct apic_msi_line* line = mem_alloc(sizeof(*line), 0);
if (!line)
return ENOMEM;
irq_lock();
struct arch_percpu* arch_cpu = percpu_ref(arch_data);
size_t slot;
for (slot = 0; slot < ARRAY_SIZE(arch_cpu->irq_lines); slot++) {
if (!arch_cpu->irq_lines[slot])
break;
}
if (slot == ARRAY_SIZE(arch_cpu->irq_lines)) {
irq_unlock();
mem_free(line);
return ENOSPC;
}
line->msi.line.set_config = apic_msi_set_config;
line->msi.line.mask = apic_msi_mask;
line->msi.line.unmask = apic_msi_unmask;
line->msi.line.eoi = apic_msi_eoi;
line->msi.msg_addr = apic_msi_msg_addr;
line->msi.msg_data = apic_msi_msg_data;
line->vector = slot + 0x20;
line->lapic = arch_cpu->lapic.id;
arch_cpu->irq_lines[slot] = &line->msi.line;
irq_unlock();
*out = &line->msi;
return 0;
}
arch/x86_64/mem.c
-35
View File
@@ -1,35 +0,0 @@
#include <arch/mem.h>
#include <stddef.h>
#include <stdint.h>
size_t arch_mem_page_size() {
return 1 << 12;
}
size_t arch_mem_page_bits() {
return 12;
}
size_t arch_mem_level_bits() {
return 9;
}
size_t arch_mem_num_levels() {
return 4;
}
uintptr_t arch_mem_hhdm_addr() {
return 0xFFFF'8000'0000'0000;
}
uintptr_t arch_mem_pfndb_addr() {
return 0xFFFF'A000'0000'0000;
}
uintptr_t arch_mem_mapping_addr() {
return 0xFFFF'C000'0000'0000;
}
bool arch_mem_addr_is_user(uintptr_t addr) {
return addr < 0x0000'7FFF'FFFF'F000;
}
arch/x86_64/meson.build
-33
View File
@@ -1,33 +0,0 @@
kernel_c_args += [
'-mgeneral-regs-only',
'-mno-red-zone',
'-masm=intel',
]
kernel_sources += files(
'apic.c',
'archctl.c',
'gdt.c',
'hpet.c',
'idt.c',
'irq.c',
'mem.c',
'percpu.c',
'pmap.c',
'random.c',
'sched.c',
'serial.c',
'smp.c',
'start.c',
'syscall.c',
'tsc.c',
'usercopy.c',
)
if kconfig.get('CONFIG_PCI', 'n') == 'y'
kernel_sources += files('pci.c')
endif
if kconfig.get('CONFIG_ACPI', 'n') == 'y'
kernel_sources += files('acpi.c')
endif
arch/x86_64/pci.c
-50
View File
@@ -1,50 +0,0 @@
#include <zinnia/init.h>
#include <zinnia/pci.h>
#include <zinnia/spin.h>
#include <x86_64/asm.h>
constexpr uint16_t PCI_CONFIG_ADDRESS = 0xcf8;
constexpr uint16_t PCI_CONFIG_DATA = 0xcfc;
static struct spinlock port_lock = {};
static void port_select(const struct pci_address* addr, uint16_t offset) {
asm_outl(
PCI_CONFIG_ADDRESS,
1u << 31 | (uint32_t)addr->bus << 16 | (uint32_t)(addr->device & 0x1f) << 11 |
(uint32_t)(addr->function & 0x7) << 8 | (offset & 0xfc)
);
}
static uint32_t port_read32(struct pci_access*, const struct pci_address* addr, uint16_t offset) {
spin_lock(&port_lock);
port_select(addr, offset);
uint32_t value = asm_inl(PCI_CONFIG_DATA);
spin_unlock(&port_lock);
return value;
}
static void port_write32(struct pci_access*, const struct pci_address* addr, uint16_t offset, uint32_t value) {
spin_lock(&port_lock);
port_select(addr, offset);
asm_outl(PCI_CONFIG_DATA, value);
spin_unlock(&port_lock);
}
static struct pci_access port_access = {
.name = "port",
.domain = 0,
.start_bus = 0,
.end_bus = 255,
.priority = PCI_ACCESS_PRIO_LEGACY,
.read32 = port_read32,
.write32 = port_write32,
};
[[__init]]
static void pci_x86_stage_init(struct boot_info*) {
pci_access_register(&port_access);
}
INIT_TASK(pci_x86_stage, "arch.x86_64.pci", pci_x86_stage_init);
INIT_DEPENDS(pci_x86_stage, mem_stage);
INIT_ENTAILS(pci_x86_stage, pci_stage);
arch/x86_64/percpu.c
-145
View File
@@ -1,145 +0,0 @@
#include <zinnia/initgraph.h>
#include <zinnia/percpu.h>
#include <zinnia/print.h>
#include <config.h>
#include <stddef.h>
#include <x86_64/apic.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
#include <x86_64/gdt.h>
#include <x86_64/idt.h>
#include <x86_64/percpu.h>
#include <x86_64/smp.h>
#include <x86_64/syscall.h>
#include <x86_64/tsc.h>
struct percpu* arch_percpu_get() {
struct percpu* result;
asm volatile("mov %0, gs:0" : "=r"(result)::"memory");
return result;
}
void arch_percpu_bsp_init() {
asm_wrmsr(MSR_GS_BASE, (uint64_t)&percpu_bsp);
asm_wrmsr(MSR_FS_BASE, 0);
asm_wrmsr(MSR_KERNEL_GS_BASE, 0);
pic_disable();
gdt_load();
idt_init();
idt_load();
}
union ia32_star {
uint64_t value;
struct {
uint32_t rsvd;
uint16_t kernel_cs_ss;
uint16_t user_cs_ss;
};
};
static_assert(sizeof(union ia32_star) == 8);
[[__percpu]]
struct arch_percpu arch_data = {0};
void x86_64_cpu_setup(struct percpu* cpu) {
asm_wrmsr(MSR_KERNEL_GS_BASE, 0);
asm_wrmsr(MSR_GS_BASE, (uint64_t)cpu);
asm_wrmsr(MSR_FS_BASE, 0);
struct arch_percpu* arch_cpu = percpu_ref(arch_data);
gdt_load();
// TODO: not if FRED
idt_load();
asm_wrmsr(
MSR_PAT,
PAT_WB | PAT_WT << 8 | PAT_UCM << 16 | PAT_UC << 24 | (uint64_t)PAT_WP << 32 | (uint64_t)PAT_WC << 40 |
(uint64_t)PAT_UCM << 48 | (uint64_t)PAT_UC << 56
);
// Syscall extension.
asm_wrmsr(MSR_EFER, asm_rdmsr(MSR_EFER) | MSR_EFER_SCE);
union ia32_star star = {0};
star.kernel_cs_ss = offsetof(struct gdt, kernel_code64);
star.user_cs_ss = offsetof(struct gdt, user_code32) | CPL_USER;
asm_wrmsr(MSR_STAR, star.value);
asm_wrmsr(MSR_LSTAR, (uint64_t)arch_syscall_stub);
// Mask all RFLAGS except for BRKI which is reserved.
asm_wrmsr(MSR_SFMASK, ~RFLAGS_BRKI);
// TODO: FRED
uint64_t cr0, cr4;
asm volatile("mov %0, cr0" : "=r"(cr0));
asm volatile("mov %0, cr4" : "=r"(cr4));
struct cpuid cpuid1 = asm_cpuid(1, 0);
struct cpuid cpuid7 = asm_cpuid(7, 0);
struct cpuid cpuid13 = asm_cpuid(13, 0);
// Enable SSE.
cr0 &= ~CR0_EM;
cr0 |= CR0_MP | CR0_WP;
cr4 |= CR4_OSFXSR | CR4_OSXMMEXCPT;
if (cpuid1.ecx & CPUID_1C_XSAVE) {
cr4 |= CR4_OSXSAVE;
asm volatile("mov cr4, %0" ::"r"(cr4));
uint64_t xcr0 = XCR0_X87 | XCR0_SSE;
// AVX
if (cpuid1.ecx & CPUID_1C_AVX)
xcr0 |= XCR0_AVX;
// AVX-512
if (cpuid7.ebx & CPUID_7B_AVX512F)
xcr0 |= XCR0_AXV512_OPMASK | XCR0_AVX512_ZMM_HI256 | XCR0_AVX512_HI16_ZMM;
asm_wrxcr(0, xcr0);
arch_cpu->fpu_size = cpuid13.ecx;
arch_cpu->fpu_save = asm_xsave;
arch_cpu->fpu_restore = asm_xrstor;
} else {
arch_cpu->fpu_size = 512;
arch_cpu->fpu_save = asm_fxsave;
arch_cpu->fpu_restore = asm_fxrstor;
}
if (cpuid7.ecx & CPUID_7C_UMIP)
cr4 |= CR4_UMIP;
if (cpuid7.ebx & CPUID_7B_SMEP)
cr4 |= CR4_SMEP;
if (cpuid7.ebx & CPUID_7B_SMAP) {
cr4 |= CR4_SMAP;
arch_cpu->can_smap = true;
}
if (cpuid7.ebx & CPUID_7B_FSGSBASE)
cr4 |= CR4_FSGSBASE;
asm volatile("mov cr0, %0" ::"r"(cr0));
asm volatile("mov cr4, %0" ::"r"(cr4));
lapic_init(&arch_cpu->lapic);
cpu->online = true;
}
void arch_percpu_init() {
#if CONFIG_TSC
tsc_init();
#endif
// Init the BSP.
x86_64_cpu_setup(&percpu_bsp);
// Init any APs.
x86_64_smp_init();
}
arch/x86_64/pmap.c
-95
View File
@@ -1,95 +0,0 @@
#include <zinnia/irq.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/types.h>
#include <x86_64/apic.h>
#include <x86_64/defs.h>
#include <x86_64/percpu.h>
#include <x86_64/pmap.h>
void pte_clear(pte_t* pte) {
*pte = 0;
}
void pte_write(pte_t* pte, pte_t val) {
*pte = val;
}
void pte_flush(uintptr_t vaddr) {
asm volatile("invlpg [%0]" ::"r"(vaddr) : "memory");
}
void pte_flush_all() {
uint64_t cr3;
asm volatile("mov %0, cr3" : "=r"(cr3));
asm volatile("mov cr3, %0" ::"r"(cr3) : "memory");
}
void pte_shootdown_kick(struct percpu* cpu) {
irq_lock();
lapic_send_ipi(
&percpu_ref(arch_data)->lapic,
percpu_ref_on(cpu, arch_data)->lapic.id,
IDT_IPI_SHOOTDOWN,
APIC_DELIVERY_FIXED,
APIC_DEST_PHYSICAL,
APIC_LEVEL_ASSERT,
APIC_TRIGGER_EDGE
);
irq_unlock();
}
pte_t pte_build(phys_t addr, enum pte_flags flags, enum cache_mode cache) {
pte_t result = ((pte_t)addr & ARCH_PTE_ADDR_MASK) | ARCH_FLAG_PRESENT;
if (flags & PTE_USER)
result |= ARCH_FLAG_USER_MODE;
if (flags & PTE_DIR)
result |= ARCH_FLAG_READ_WRITE;
else {
if (flags & PTE_WRITE)
result |= ARCH_FLAG_READ_WRITE;
if (!(flags & PTE_EXEC))
result |= ARCH_FLAG_EXECUTE_DISABLE;
switch (cache) {
case CACHE_ANY:
case CACHE_WRITE_BACK:
break;
case CACHE_WRITE_THROUGH:
result |= ARCH_FLAG_WRITE_THROUGH;
break;
case CACHE_MMIO:
result |= ARCH_FLAG_CACHE_DISABLE | ARCH_FLAG_WRITE_THROUGH;
break;
case CACHE_WRITE_COMBINE:
result |= ARCH_FLAG_ATTRIBUTE_TABLE | ARCH_FLAG_WRITE_THROUGH;
break;
}
}
return result;
}
pte_t pte_protect(pte_t old, enum pte_flags flags) {
pte_t result = old & ~(ARCH_FLAG_READ_WRITE | ARCH_FLAG_USER_MODE | ARCH_FLAG_EXECUTE_DISABLE);
if (flags & PTE_USER)
result |= ARCH_FLAG_USER_MODE;
if (flags & PTE_WRITE)
result |= ARCH_FLAG_READ_WRITE;
if (!(flags & PTE_EXEC))
result |= ARCH_FLAG_EXECUTE_DISABLE;
return result;
}
bool pte_is_present(pte_t* pte) {
return *pte & ARCH_FLAG_PRESENT;
}
phys_t pte_address(pte_t* pte) {
return (phys_t)(*pte & ARCH_PTE_ADDR_MASK);
}
void pmap_set(struct pmap* pt) {
asm volatile("mov cr3, %0" ::"r"(pt->root) : "memory");
}
arch/x86_64/random.c
-36
View File
@@ -1,36 +0,0 @@
#include <zinnia/initgraph.h>
#include <zinnia/print.h>
#include <zinnia/random.h>
#include <x86_64/asm.h>
#define CPUID_1C_RDRAND (1u << 30)
#define RDRAND_RETRIES 10
static bool rdrand_get_u64(struct random_source*, uint64_t* out) {
for (int i = 0; i < RDRAND_RETRIES; i++) {
uint64_t value;
uint8_t ok;
asm volatile("rdrand %0\n setc %1" : "=r"(value), "=qm"(ok) : : "cc");
if (ok) {
*out = value;
return true;
}
}
return false;
}
static struct random_source rdrand_source = {
.name = "x86_64.rdrand",
.quality = 100,
.get_u64 = rdrand_get_u64,
};
static void rdrand_init(struct boot_info*) {
if (asm_cpuid(1, 0).ecx & CPUID_1C_RDRAND) {
random_register(&rdrand_source);
} else {
log_warn("random: This CPU doesn't support RDRAND, ignoring\n");
}
}
INIT_TASK(rdrand_stage, "arch.x86_64.rdrand", rdrand_init);
INIT_ENTAILS(rdrand_stage, random_stage);
arch/x86_64/sched.c
-272
View File
@@ -1,272 +0,0 @@
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/compiler.h>
#include <zinnia/errno.h>
#include <zinnia/irq.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/sched.h>
#include <zinnia/vma.h>
#include <arch/sched.h>
#include <bits/sched.h>
#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include <x86_64/apic.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
#include <x86_64/gdt.h>
#include <x86_64/idt.h>
#include <x86_64/percpu.h>
// The task frame consists of registers that the C ABI marks as callee-saved.
// If we don't save them, these registers are lost during a context switch.
struct task_frame {
uint64_t rbx, rbp, r12, r13, r14, r15, rip;
};
[[__naked]]
static struct task* perform_switch(uint64_t* old_rsp, uint64_t new_rsp, struct task* prev) {
asm volatile(
"sub rsp, 0x30\n" // Make room for all regs (except RIP).
"mov [rsp + %c[rbx]], rbx\n"
"mov [rsp + %c[rbp]], rbp\n"
"mov [rsp + %c[r12]], r12\n"
"mov [rsp + %c[r13]], r13\n"
"mov [rsp + %c[r14]], r14\n"
"mov [rsp + %c[r15]], r15\n"
"mov [rdi], rsp\n" // rdi = old_rsp
"mov rsp, rsi\n" // rsi = new_rsp
"mov rbx, [rsp + %c[rbx]]\n"
"mov rbp, [rsp + %c[rbp]]\n"
"mov r12, [rsp + %c[r12]]\n"
"mov r13, [rsp + %c[r13]]\n"
"mov r14, [rsp + %c[r14]]\n"
"mov r15, [rsp + %c[r15]]\n"
"add rsp, 0x30\n"
"mov rax, rdx\n"
"ret" // This will conveniently move us to the RIP we put at this stack entry.
:
: [rbx] "i"(offsetof(struct task_frame, rbx)),
[rbp] "i"(offsetof(struct task_frame, rbp)),
[r12] "i"(offsetof(struct task_frame, r12)),
[r13] "i"(offsetof(struct task_frame, r13)),
[r14] "i"(offsetof(struct task_frame, r14)),
[r15] "i"(offsetof(struct task_frame, r15))
: "memory"
);
}
struct task* arch_sched_switch(struct task* from, struct task* to) {
struct arch_percpu* cpu = percpu_ref(arch_data);
struct arch_task_context* from_ctx = &from->context;
struct arch_task_context* to_ctx = &to->context;
cpu->tss.rsp0 = kernel_stack_top(&to->kernel_stack);
if (from_ctx->fpu_region != nullptr) {
cpu->fpu_save(from_ctx->fpu_region);
from_ctx->ds = asm_read_ds();
from_ctx->es = asm_read_es();
from_ctx->fs = asm_read_fs();
from_ctx->gs = asm_read_gs();
from_ctx->fs_base = asm_rdmsr(MSR_FS_BASE);
from_ctx->gs_base = asm_rdmsr(MSR_KERNEL_GS_BASE);
}
if (to_ctx->fpu_region != nullptr) {
cpu->fpu_restore(to_ctx->fpu_region);
asm_write_ds(to_ctx->ds);
asm_write_es(to_ctx->es);
asm_write_fs(to_ctx->fs);
// If we have to change the GS segment we need to reload the MSR, otherwise we lose its value.
if (to_ctx->gs != asm_read_gs()) {
uint64_t percpu = asm_rdmsr(MSR_GS_BASE);
asm_write_gs(to_ctx->gs);
asm_wrmsr(MSR_GS_BASE, percpu);
}
asm_wrmsr(MSR_FS_BASE, to_ctx->fs_base);
asm_wrmsr(MSR_KERNEL_GS_BASE, to_ctx->gs_base);
}
return perform_switch(&from_ctx->rsp, to_ctx->rsp, from);
}
constexpr size_t FPU_FCW_OFFSET = 0;
constexpr size_t FPU_MXCSR_OFFSET = 24;
constexpr uint16_t FPU_DEFAULT_FCW = 0x37F;
constexpr uint32_t FPU_DEFAULT_MXCSR = 0x1F80;
// Calls task_entry with the correct parameters.
[[__naked]]
void task_entry_thunk() {
asm volatile(
"mov rdi, rbx\n"
"mov rsi, r12\n"
"mov rdx, rax\n"
"jmp task_entry"
:
:
: "memory", "rdi", "rsi", "rdx"
);
}
errno_t arch_task_init(
struct arch_task_context* context,
void* entry,
uintptr_t arg0,
uintptr_t stack_start,
bool is_user
) {
struct arch_percpu* cpu = percpu_ref(arch_data);
assert((stack_start & 0xF) == 0, "Task stack is not 16-byte aligned!");
// Prepare a dummy stack with an entry point function to return to.
struct task_frame* frame = (struct task_frame*)(stack_start - sizeof(uint64_t)) - 1;
frame->rbx = (uint64_t)entry;
frame->r12 = arg0;
frame->rip = (uint64_t)task_entry_thunk;
context->rsp = (uint64_t)frame;
if (is_user) {
// Allocate FPU space.
const size_t fpu_size = cpu->fpu_size;
uintptr_t fpu_base = vm_alloc(fpu_size);
if (!fpu_base)
return ENOMEM;
phys_t pages;
errno_t s = mem_phys_alloc(ALIGN_UP(fpu_size, arch_mem_page_size()) / arch_mem_page_size(), 0, &pages);
if (s) {
vm_free(fpu_base, fpu_size);
return s;
}
for (size_t i = 0; i < fpu_size; i += arch_mem_page_size())
pmap_map(&kernel_vma.pmap, fpu_base + i, pages + i, PTE_WRITE | PTE_READ, CACHE_WRITE_BACK);
context->fpu_region = (void*)fpu_base;
context->fpu_phys = pages;
const uint16_t fcw = FPU_DEFAULT_FCW;
const uint32_t mxcsr = FPU_DEFAULT_MXCSR;
memcpy((uint8_t*)fpu_base + FPU_FCW_OFFSET, &fcw, sizeof(fcw));
memcpy((uint8_t*)fpu_base + FPU_MXCSR_OFFSET, &mxcsr, sizeof(mxcsr));
context->ds = asm_read_ds();
context->es = asm_read_es();
context->fs = asm_read_fs();
context->gs = asm_read_gs();
context->fs_base = asm_rdmsr(MSR_FS_BASE);
context->gs_base = asm_rdmsr(MSR_KERNEL_GS_BASE);
}
return 0;
}
void arch_task_fini(struct arch_task_context* context) {
if (context->fpu_region == nullptr)
return;
const size_t page_size = arch_mem_page_size();
const size_t fpu_size = percpu_ref(arch_data)->fpu_size;
const uintptr_t fpu_base = (uintptr_t)context->fpu_region;
for (size_t i = 0; i < fpu_size; i += page_size)
pmap_unmap(&kernel_vma.pmap, fpu_base + i);
pmap_shootdown(&kernel_vma.pmap);
mem_phys_free(context->fpu_phys, ALIGN_UP(fpu_size, page_size) / page_size);
vm_free(fpu_base, fpu_size);
context->fpu_region = nullptr;
}
void arch_task_fork(struct arch_task_context* dst) {
if (dst->fpu_region)
percpu_ref(arch_data)->fpu_save(dst->fpu_region);
}
void arch_context_init_user(struct arch_context* ctx, uintptr_t ip, uintptr_t sp) {
*ctx = (struct arch_context){
.rip = ip,
.rsp = sp,
.rflags = 0x202,
.cs = offsetof(struct gdt, user_code64) | CPL_USER,
.ss = offsetof(struct gdt, user_data) | CPL_USER,
};
}
[[__naked]]
void arch_sched_preempt_disable() {
asm volatile(
"inc qword ptr gs:%c0\n"
"ret"
:
: "i"(offsetof(struct percpu, sched.preempt_level))
);
}
[[__naked]]
bool arch_sched_preempt_enable() {
asm volatile(
"dec qword ptr gs:%c0\n"
"setz al\n"
"movzx eax, al\n"
"ret\n"
:
: "i"(offsetof(struct percpu, sched.preempt_level))
: "cc"
);
}
void arch_sched_kick(struct percpu* cpu) {
if (!cpu->online)
return;
// The xAPIC ICR takes two writes, which must not be interleaved with an IPI sent by an interrupt handler.
irq_lock();
lapic_send_ipi(
&percpu_ref(arch_data)->lapic,
percpu_ref_on(cpu, arch_data)->lapic.id,
IDT_IPI_RESCHED,
APIC_DELIVERY_FIXED,
APIC_DEST_PHYSICAL,
APIC_LEVEL_ASSERT,
APIC_TRIGGER_EDGE
);
irq_unlock();
}
[[__naked]]
void arch_sched_jump_to_context(struct arch_context* context) {
asm volatile(
"mov rsp, rdi\n"
"jmp %0\n" ::"r"(interrupt_return)
);
}
void arch_sched_jump_to_user(uintptr_t ip, uintptr_t sp, uintptr_t arg) {
struct arch_context context = {
.rip = ip,
.rsp = sp,
.rdi = arg,
.rflags = 0x202,
.cs = offsetof(struct gdt, user_code64) | CPL_USER,
.ss = offsetof(struct gdt, user_data) | CPL_USER,
};
uintptr_t percpu = asm_rdmsr(MSR_GS_BASE);
asm volatile(
"mov ds, %0\n"
"mov es, %0\n"
"mov fs, %0\n"
"mov gs, %0\n"
:
: "a"(0)
: "memory"
);
asm_wrmsr(MSR_FS_BASE, 0);
asm_wrmsr(MSR_GS_BASE, percpu);
asm_wrmsr(MSR_KERNEL_GS_BASE, 0);
arch_sched_jump_to_context(&context);
}
arch/x86_64/smp.c
-359
View File
@@ -1,359 +0,0 @@
#define pr_fmt(fmt) "smp: " fmt
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/atomic.h>
#include <zinnia/clock.h>
#include <zinnia/compiler.h>
#include <zinnia/irq.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/print.h>
#include <zinnia/sched.h>
#include <zinnia/types.h>
#include <zinnia/utils.h>
#include <zinnia/vma.h>
#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include <uacpi/acpi.h>
#include <uacpi/tables.h>
#include <x86_64/apic.h>
#include <x86_64/defs.h>
#include <x86_64/percpu.h>
#include <x86_64/pmap.h>
#include <x86_64/smp.h>
extern uint8_t smp_trampoline_start[];
extern uint8_t smp_trampoline_data[];
extern uint8_t smp_trampoline_end[];
[[__naked, __used]]
static void smp_trampoline() {
asm volatile(
".pushsection .rodata\n"
".code16\n"
".global smp_trampoline_start\n"
"smp_trampoline_start:\n"
" cli\n"
" cld\n"
" jmp .Lmode16\n"
".org smp_trampoline_start + %c[data_off]\n"
".global smp_trampoline_data\n"
"smp_trampoline_data:\n"
" .skip %c[info_size]\n"
".set .Ldata_offset, %c[data_off]\n"
".set .Lgdtr_offset, (.Ldata_offset + %c[gdtr])\n"
".set .Lfarjmp_offset, (.Ldata_offset + %c[farjmp])\n"
".set .Ltemp_stack_offset, (.Ldata_offset + %c[temp_stack])\n"
".set .Ltemp_cr3_offset, (.Ldata_offset + %c[temp_cr3])\n"
".set .Lentry_offset, (.Ldata_offset + %c[entry])\n"
".set .Lhhdm_offset, (.Ldata_offset + %c[hhdm])\n"
".set .Linvalid_idtr_offset, (.Linvalid_idtr - smp_trampoline_start)\n"
".set .Lmode32_offset, (.Lmode32 - smp_trampoline_start)\n"
".set .Lmode64_offset, (.Lmode64 - smp_trampoline_start)\n"
".Lmode16:\n"
" mov bx, cs\n"
" shl ebx, 4\n"
" lidtd cs:.Linvalid_idtr_offset\n"
" lgdtd cs:.Lgdtr_offset\n"
" lea eax, [ebx + .Lmode32_offset]\n"
" mov dword ptr cs:.Lfarjmp_offset, eax\n"
" mov eax, %c[cr0]\n"
" mov cr0, eax\n"
" jmp fword ptr cs:.Lfarjmp_offset\n"
".code32\n"
".Lmode32:\n"
" mov ax, %c[kernel32_ds]\n"
" mov ds, ax\n"
" mov es, ax\n"
" mov fs, ax\n"
" mov gs, ax\n"
" mov ss, ax\n"
" xor eax, eax\n"
" lldt ax\n"
" mov esp, dword ptr [ebx + .Ltemp_stack_offset]\n"
" xor eax, eax\n"
" mov cr4, eax\n"
" or eax, %c[cr4_pae]\n"
" mov cr4, eax\n"
" mov ecx, %c[msr_efer]\n"
" mov eax, %c[efer_lme]\n"
" xor edx, edx\n"
" wrmsr\n"
" mov eax, dword ptr [ebx + .Ltemp_cr3_offset]\n"
" mov cr3, eax\n"
" mov eax, cr0\n"
" or eax, %c[cr0_pg]\n"
" mov cr0, eax\n"
" lea eax, [ebx + .Lmode64_offset]\n"
" push %c[kernel64_cs]\n"
" push eax\n"
" retf\n"
".code64\n"
".Lmode64:\n"
" mov ax, %c[kernel64_ds]\n"
" mov ds, ax\n"
" mov es, ax\n"
" mov fs, ax\n"
" mov gs, ax\n"
" mov ss, ax\n"
" mov ecx, %c[msr_efer]\n"
" rdmsr\n"
" or eax, %c[efer_nxe]\n"
" wrmsr\n"
" xor ebp, ebp\n"
" lea rdi, [rbx + .Ldata_offset]\n"
" mov rax, [rbx + .Lentry_offset]\n"
" add rsp, [rbx + .Lhhdm_offset]\n"
" jmp rax\n"
".Linvalid_idtr:\n"
" .word 0\n"
" .quad 0\n"
".global smp_trampoline_end\n"
"smp_trampoline_end:\n"
".popsection\n"
" ud2\n"
:
: [data_off] "i"(8),
[info_size] "i"(sizeof(struct smp_info)),
[gdtr] "i"(offsetof(struct smp_info, gdtr)),
[farjmp] "i"(offsetof(struct smp_info, farjmp_offset)),
[temp_stack] "i"(offsetof(struct smp_info, temp_stack)),
[temp_cr3] "i"(offsetof(struct smp_info, temp_cr3)),
[entry] "i"(offsetof(struct smp_info, entry)),
[hhdm] "i"(offsetof(struct smp_info, hhdm_offset)),
[kernel32_ds] "i"(offsetof(struct gdt, kernel_data32)),
[kernel64_cs] "i"(offsetof(struct gdt, kernel_code64)),
[kernel64_ds] "i"(offsetof(struct gdt, kernel_data64)),
[cr0] "i"(CR0_PE | CR0_ET),
[cr0_pg] "i"((uint32_t)CR0_PG),
[cr4_pae] "i"(CR4_PAE),
[msr_efer] "i"(MSR_EFER),
[efer_lme] "i"(MSR_EFER_LME),
[efer_nxe] "i"(MSR_EFER_NXE)
);
}
constexpr size_t AP_BOOT_STACK_SIZE = 0x8000;
constexpr uint64_t AP_BOOT_TIMEOUT_NS = 1'000'000'000;
[[noreturn]]
static void ap_entry(phys_t info_addr) {
// We're still running on the temporary page table.
pmap_set(&kernel_vma.pmap);
struct smp_info* info = HHDM_PTR(info_addr);
struct percpu* cpu = (struct percpu*)info->percpu;
x86_64_cpu_setup(cpu);
irq_lock();
// This task keeps running on the boot stack, but is never scheduled again after the first yield.
sched_init(&cpu->sched);
// Let the BSP know that we're alive.
// `info` must not be touched after this point.
atomic_store(&info->booted, 1, ATOMIC_RELEASE);
log("CPU %zu online\n", cpu->id);
sched_yield(&cpu->sched);
unreachable();
}
static bool start_ap(phys_t trampoline, uint32_t temp_cr3, uint32_t lapic_id) {
const size_t trampoline_size = smp_trampoline_end - smp_trampoline_start;
phys_t stack;
if (mem_phys_alloc(AP_BOOT_STACK_SIZE / arch_mem_page_size(), ALLOC_MEM32 | ALLOC_NOZERO, &stack)) {
log_err("Failed to allocate a boot stack for LAPIC %u\n", lapic_id);
return false;
}
struct percpu* cpu = percpu_new();
if (!cpu) {
log_err("Failed to allocate per-CPU data for LAPIC %u\n", lapic_id);
mem_phys_free(stack, AP_BOOT_STACK_SIZE / arch_mem_page_size());
return false;
}
log("Starting CPU %zu\n", cpu->id);
memcpy(HHDM_PTR(trampoline), smp_trampoline_start, trampoline_size);
const phys_t info_addr = trampoline + (smp_trampoline_data - smp_trampoline_start);
struct smp_info* info = HHDM_PTR(info_addr);
memcpy(&info->gdt, &gdt_base, sizeof(info->gdt));
info->gdtr.limit = sizeof(struct gdt) - 1;
info->gdtr.base = (struct gdt*)(info_addr + offsetof(struct smp_info, gdt));
info->farjmp_offset = 0; // Filled in by the trampoline.
info->farjmp_segment = offsetof(struct gdt, kernel_code32);
info->temp_stack = stack + AP_BOOT_STACK_SIZE;
info->temp_cr3 = temp_cr3;
info->entry = (uint64_t)ap_entry;
info->hhdm_offset = mem_hhdm_base;
info->percpu = (uint64_t)cpu;
info->lapic_id = lapic_id;
atomic_init(&info->booted, 0);
struct local_apic* lapic = &percpu_ref(arch_data)->lapic;
lapic_send_ipi(lapic, lapic_id, 0, APIC_DELIVERY_INIT, APIC_DEST_PHYSICAL, APIC_LEVEL_ASSERT, APIC_TRIGGER_EDGE);
clock_spin_ns(10'000'000);
lapic_send_ipi(
lapic,
lapic_id,
trampoline >> 12,
APIC_DELIVERY_STARTUP,
APIC_DEST_PHYSICAL,
APIC_LEVEL_ASSERT,
APIC_TRIGGER_EDGE
);
clock_spin_ns(10'000'000);
for (uint64_t waited = 0; !atomic_load(&info->booted, ATOMIC_ACQUIRE); waited += 1'000'000) {
if (waited >= AP_BOOT_TIMEOUT_NS) {
// The per-CPU data and boot stack are leaked on purpose, in case the AP still comes up.
log_err("CPU %zu (LAPIC %u) did not respond\n", cpu->id, lapic_id);
return false;
}
clock_spin_ns(1'000'000);
}
return true;
}
static errno_t build_temp_table(phys_t* out) {
const size_t pgsz = arch_mem_page_size();
phys_t mem;
errno_t status = mem_phys_alloc(6, ALLOC_MEM32, &mem);
if (status)
return status;
uint64_t* l4 = HHDM_PTR(mem);
uint64_t* l3 = HHDM_PTR(mem + pgsz);
const phys_t l2_base = mem + 2 * pgsz;
// Use 2 MiB pages.
l4[0] = (mem + pgsz) | ARCH_FLAG_PRESENT | ARCH_FLAG_READ_WRITE;
for (size_t i = 0; i < 4; i++) {
uint64_t* l2 = HHDM_PTR(l2_base + i * pgsz);
l3[i] = (l2_base + i * pgsz) | ARCH_FLAG_PRESENT | ARCH_FLAG_READ_WRITE;
for (size_t j = 0; j < 512; j++)
l2[j] = ((i << 30) + (j << 21)) | ARCH_FLAG_PRESENT | ARCH_FLAG_READ_WRITE | ARCH_FLAG_SIZE;
}
// Copy over the higher half from the kernel page table.
const uint64_t* kernel_l4 = HHDM_PTR(kernel_vma.pmap.root);
for (size_t i = 256; i < 512; i++)
l4[i] = kernel_l4[i];
*out = mem;
return 0;
}
void x86_64_smp_init() {
const size_t trampoline_size = smp_trampoline_end - smp_trampoline_start;
assert(trampoline_size <= arch_mem_page_size(), "SMP trampoline doesn't fit in a page");
const size_t max_cpus = percpu_max_cpus();
if (max_cpus <= 1)
return;
uacpi_table table;
if (uacpi_table_find_by_signature(ACPI_MADT_SIGNATURE, &table) != UACPI_STATUS_OK) {
log_warn("No MADT found, not starting any APs\n");
return;
}
phys_t trampoline;
if (mem_phys_alloc(1, ALLOC_MEM20, &trampoline)) {
log_err("Failed to allocate 20-bit memory for the AP trampoline\n");
uacpi_table_unref(&table);
return;
}
phys_t temp_table;
if (build_temp_table(&temp_table)) {
log_err("Failed to allocate a temporary page table\n");
mem_phys_free(trampoline, 1);
uacpi_table_unref(&table);
return;
}
const uint32_t bsp_id = percpu_ref(arch_data)->lapic.id;
bool all_booted = true;
size_t num_cpus = 1;
const struct acpi_madt* madt = table.ptr;
const uint8_t* entry = (const uint8_t*)madt->entries;
const uint8_t* end = (const uint8_t*)madt + madt->hdr.length;
while (entry + sizeof(struct acpi_entry_hdr) <= end && num_cpus < max_cpus) {
const struct acpi_entry_hdr* hdr = (const struct acpi_entry_hdr*)entry;
// Don't trust malformed firmware tables.
if (hdr->length < sizeof(struct acpi_entry_hdr) || entry + hdr->length > end) {
log_warn("MADT contains a malformed entry\n");
break;
}
uint32_t lapic_id = 0;
bool enabled = false;
switch (hdr->type) {
case ACPI_MADT_ENTRY_TYPE_LAPIC:
const struct acpi_madt_lapic* lapic = (const struct acpi_madt_lapic*)hdr;
lapic_id = lapic->id;
enabled = lapic->flags & ACPI_PIC_ENABLED;
break;
case ACPI_MADT_ENTRY_TYPE_LOCAL_X2APIC:
const struct acpi_madt_x2apic* x2apic = (const struct acpi_madt_x2apic*)hdr;
lapic_id = x2apic->id;
enabled = x2apic->flags & ACPI_PIC_ENABLED;
break;
}
if (enabled && lapic_id != bsp_id) {
if (start_ap(trampoline, temp_table, lapic_id))
num_cpus++;
else
all_booted = false;
}
entry += hdr->length;
}
uacpi_table_unref(&table);
// An AP which didn't respond might still be using these.
if (all_booted) {
mem_phys_free(temp_table, 6);
mem_phys_free(trampoline, 1);
}
log("%zu CPU(s) online\n", num_cpus);
}
arch/x86_64/start.c
-19
View File
@@ -1,19 +0,0 @@
#include <zinnia/compiler.h>
#include <zinnia/errno.h>
#include <zinnia/init.h>
#include <zinnia/percpu.h>
#include <uapi/archctl.h>
#include <x86_64/apic.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
#include <x86_64/gdt.h>
[[__init, __naked, __used]]
void _start() {
asm volatile(
"mov rsp, %0\n"
"jmp %1"
:
: "r"(__ld_stack_top), "r"(kernel_entry)
);
}
arch/x86_64/syscall.c
-73
View File
@@ -1,73 +0,0 @@
#include <zinnia/compiler.h>
#include <zinnia/percpu.h>
#include <zinnia/syscall.h>
#include <bits/sched.h>
#include <stddef.h>
#include <x86_64/defs.h>
#include <x86_64/gdt.h>
// TODO: Refactor to only push callee-saved registers.
[[__naked]]
void arch_syscall_stub() {
asm volatile(
" swapgs\n"
" mov gs:%c[user_stack], rsp\n"
" mov rsp, gs:%c[kernel_stack]\n"
" cld\n"
"1:\n"
" push %c[user_data]\n" // ss
" push gs:%c[user_stack]\n"
" push r11\n" // rflags
" push %c[user_code64]\n" // cs
" push rcx\n" // rip
" push 0x00\n" // arch_context::error
" push 0x00\n" // arch_context::isr
" push rax\n"
" push rbx\n"
" push rcx\n"
" push rdx\n"
" push rbp\n"
" push rdi\n"
" push rsi\n"
" push r8\n"
" push r9\n"
" push r10\n"
" push r11\n"
" push r12\n"
" push r13\n"
" push r14\n"
" push r15\n"
" xor rbp, rbp\n"
" mov rdi, rsp\n" // SysV ABI first argument.
" call syscall_dispatch\n" // Call syscall handler.
" cli\n"
" pop r15\n"
" pop r14\n"
" pop r13\n"
" pop r12\n"
" pop r11\n"
" pop r10\n"
" pop r9\n"
" pop r8\n"
" pop rsi\n"
" pop rdi\n"
" pop rbp\n"
" pop rdx\n"
" pop rcx\n"
" pop rbx\n"
" pop rax\n"
" add rsp, 0x10\n" // Skip .error and .isr fields
" pop rcx\n" // rip
" add rsp, 0x08\n" // Skip cs
" pop r11\n" // rflags
" pop rsp\n" // rsp
" swapgs\n"
" sysretq\n"
:
: [user_stack] "i"(offsetof(struct percpu, user_stack)),
[kernel_stack] "i"(offsetof(struct percpu, kernel_stack)),
[user_code64] "i"(offsetof(struct gdt, user_code64) | CPL_USER),
[user_data] "i"(offsetof(struct gdt, user_data) | CPL_USER)
);
}
arch/x86_64/tsc.c
-88
View File
@@ -1,88 +0,0 @@
#define pr_fmt(fmt) "x86_64/tsc: " fmt
#include <zinnia/clock.h>
#include <zinnia/cmdline.h>
#include <zinnia/print.h>
#include <zinnia/utils.h>
#include <stdint.h>
#include <x86_64/asm.h>
#include <x86_64/defs.h>
static bool use_tsc = true;
static void tsc_option(bool opt) {
use_tsc = opt;
}
CMDLINE_OPTION("x86_64.tsc", "Enable timestamp counter", tsc_option);
constexpr uint64_t nanos_per_sec = 1'000'000'000;
static uint64_t tsc_freq = 0;
static uint64_t tsc_base_cycles = 0;
static uint64_t cycles_to_ns(uint64_t cycles, uint64_t freq) {
uint64_t seconds = cycles / freq;
uint64_t remaining_cycles = cycles % freq;
uint64_t fractional_ns = (uint64_t)((__uint128_t)remaining_cycles * (__uint128_t)nanos_per_sec / (__uint128_t)freq);
uint64_t ns = seconds * nanos_per_sec + fractional_ns;
return MIN(ns, UINT64_MAX);
}
static void tsc_reset(struct clock* clock) {
tsc_base_cycles = asm_rdtsc();
}
static uint64_t tsc_get_elapsed_ns(struct clock* clock) {
uint64_t cycles = asm_rdtsc() - tsc_base_cycles;
return cycles_to_ns(cycles, tsc_freq);
}
static struct clock tsc_clock = {
.name = "tsc",
.priority = 255,
.reset = tsc_reset,
.get_elapsed_ns = tsc_get_elapsed_ns,
};
void tsc_init() {
if (!use_tsc) {
log_warn("TSC disabled on the command line");
return;
}
if (!(asm_cpuid(1, 0).edx & CPUID_1D_TSC) || !(asm_cpuid(0x8000'0007, 0).edx & (1 << 8))) {
log_warn("No invariant TSC detected!\n");
return;
}
// Check for the TSC info leaf.
bool has_tsc_leaf = false;
struct cpuid tsc_leaf = {0};
if (asm_cpuid(0, 0).eax >= 0x15) {
tsc_leaf = asm_cpuid(0x15, 0);
has_tsc_leaf = true;
}
if (clock_available()) {
// Calibrate using existing clock.
uint64_t t1 = asm_rdtsc();
clock_spin_ns(10'000'000);
uint64_t t2 = asm_rdtsc();
tsc_freq = (t2 - t1) * 100;
} else if (has_tsc_leaf) {
// If we have no timer (yet), the only way we can calibrate the TSC is if CPUID gives us the frequency.
// On a normal system, this should usually never be called and is a last resort
// since at this point we have at least the HPET timer.
if (tsc_leaf.ecx != 0 && tsc_leaf.ebx != 0 && tsc_leaf.eax != 0) {
log("Calibrating TSC using CPUID 0x15");
tsc_freq = (uint64_t)tsc_leaf.ecx * tsc_leaf.ebx / tsc_leaf.eax;
} else {
log_err("Unable to calibrate TSC using CPUID frequency information");
return;
}
} else {
log_err("Unable to calibrate the TSC without a clock or static frequency info!\n");
return;
}
log("Timer frequency is %lu MHz\n", tsc_freq / 1'000'000);
clock_switch(&tsc_clock);
}
arch/x86_64/usercopy.c
-142
View File
@@ -1,142 +0,0 @@
#include <zinnia/compiler.h>
#include <zinnia/percpu.h>
#include <arch/usercopy.h>
#include <x86_64/percpu.h>
void arch_usercopy_allow(bool state) {
if (!percpu_ref(arch_data)->can_smap)
return;
if (state) {
asm volatile("stac" ::: "memory");
} else {
asm volatile("clac" ::: "memory");
}
}
extern void x86_64_read_start();
extern void x86_64_read_end();
extern void x86_64_read_fault();
static const struct usercopy_region read_region = {
.start_ip = x86_64_read_start,
.end_ip = x86_64_read_end,
.fault_ip = x86_64_read_fault,
};
[[__naked]]
bool arch_usercopy_read(void* dst, const __user void* src, size_t len, struct usercopy_region** region) {
asm volatile(
// Setup regs for `rep movsb`.
" xchg rcx, rdx\n"
" lea rax, [rip + %0]\n"
" xchg [rdx], rax\n"
".global x86_64_read_start\n"
" x86_64_read_start:\n"
" rep movsb\n"
".global x86_64_read_end\n"
"x86_64_read_end:\n"
" xor rax, rax\n"
" xchg [rdx], rax\n"
" mov rax, 1\n"
" ret\n"
".global x86_64_read_fault\n"
"x86_64_read_fault:\n"
" xor rax, rax\n"
" mov [rdx], rax\n"
" ret\n"
:
: "i"(&read_region)
: "memory"
);
}
extern void x86_64_write_start();
extern void x86_64_write_end();
extern void x86_64_write_fault();
static const struct usercopy_region write_region = {
.start_ip = x86_64_write_start,
.end_ip = x86_64_write_end,
.fault_ip = x86_64_write_fault,
};
[[__naked]]
bool arch_usercopy_write(__user void* dst, const void* src, size_t len, struct usercopy_region** region) {
asm volatile(
// Setup regs for `rep movsb`.
" xchg rcx, rdx\n"
" lea rax, [rip + %0]\n"
" mov [rdx], rax\n"
".global x86_64_write_start\n"
" x86_64_write_start:\n"
" rep movsb\n"
".global x86_64_write_end\n"
" x86_64_write_end:\n"
" xor rax, rax\n"
" xchg [rdx], rax\n"
" mov rax, 1\n"
" ret\n"
".global x86_64_write_fault\n"
"x86_64_write_fault:\n"
" xor rax, rax\n"
" mov [rdx], rax\n"
" ret\n"
:
: "i"(&write_region)
: "memory"
);
}
extern void x86_64_strlen_start();
extern void x86_64_strlen_end();
extern void x86_64_strlen_fault();
static const struct usercopy_region strlen_region = {
.start_ip = x86_64_strlen_start,
.end_ip = x86_64_strlen_end,
.fault_ip = x86_64_strlen_fault,
};
[[__naked]]
bool arch_usercopy_strlen(const __user char* str, size_t max, size_t* len, struct usercopy_region** region) {
asm volatile(
" lea rax, [rip + %0]\n"
" xchg [rcx], rax\n"
".global x86_64_strlen_start\n"
" x86_64_strlen_start:\n"
" xor r8, r8\n"
".Lloop:\n"
" cmp r8, rsi\n"
" je .Lleave\n"
" cmp byte ptr [rdi + r8], 0\n"
" je .Lleave\n"
" inc r8\n"
" jmp .Lloop\n"
".Lleave:\n"
" mov [rdx], r8\n"
".global x86_64_strlen_end\n"
"x86_64_strlen_end:\n"
" xor rax, rax\n"
" xchg [rcx], rax\n"
" mov rax, 1\n"
" ret\n"
".global x86_64_strlen_fault\n"
"x86_64_strlen_fault:\n"
" xor rax, rax\n"
" mov [rcx], rax\n"
" ret\n"
:
: "i"(&strlen_region)
: "memory"
);
}
boot/cmdline.c
-122
View File
@@ -1,122 +0,0 @@
#include <zinnia/cmdline.h>
#include <zinnia/init.h>
#include <zinnia/stdlib.h>
#include <stddef.h>
#include <string.h>
// Defined in linker script.
extern const struct cmdline_option* __ld_cmdline_start[];
extern const struct cmdline_option* __ld_cmdline_end[];
static char cmdline_buf[0x1000];
static char cmdline_raw[0x1000];
const char* cmdline_get() {
return cmdline_raw;
}
void cmdline_parse_buf(
char* cmdline_buf,
size_t len,
const struct cmdline_option* const* start,
const struct cmdline_option* const* end
) {
size_t idx = 0;
while (1) {
const struct cmdline_option* const* opt = start;
char* name = nullptr;
char* value = nullptr;
// Skip all leading spaces.
while (idx < len && cmdline_buf[idx] == ' ')
idx++;
if (idx >= len)
break;
size_t name_idx = idx;
name = cmdline_buf + name_idx;
// Find the next equal sign or space.
while (idx < len && cmdline_buf[idx] != '=' && cmdline_buf[idx] != ' ')
idx++;
if (idx > len)
break;
// Check if the option has a value (=foo).
char seperator = cmdline_buf[idx];
cmdline_buf[idx++] = 0;
if (seperator == '=') {
// Check if we need to escape the value.
char check;
if (cmdline_buf[idx] == '"') {
check = '"';
cmdline_buf[idx++] = 0;
} else {
check = ' ';
}
value = cmdline_buf + idx;
// Skip the value.
while (idx < len && cmdline_buf[idx] != check)
idx++;
if (idx > len)
break;
cmdline_buf[idx++] = 0;
}
// Find the corresponding option.
while (opt < end) {
const struct cmdline_option* current = *opt;
if (!strcmp(current->name, name)) {
if (value == nullptr)
break;
switch (current->option_type) {
case CMDLINE_STRING:
current->func_str(value);
break;
case CMDLINE_BOOL:
if (!strcmp(value, "on") || !strcmp(value, "yes") || !strcmp(value, "true") || !strcmp(value, "1"))
current->func_bool(true);
else
current->func_bool(false);
break;
case CMDLINE_INTPTR: {
int base = 10;
if (!strncmp(value, "0x", 2)) {
base = 0x10;
value += 2;
}
current->func_intptr(atol(value, base));
break;
}
case CMDLINE_UINTPTR: {
int base = 10;
if (!strncmp(value, "0x", 2)) {
base = 0x10;
value += 2;
}
current->func_uintptr(atolu(value, base));
break;
}
}
break;
}
opt++;
}
if (idx >= len)
break;
}
}
[[__init]]
void cmdline_parse(const char* cmdline) {
if (cmdline == nullptr)
cmdline = "";
const size_t len = strnlen(cmdline, sizeof(cmdline_buf) - 1);
memcpy(cmdline_buf, cmdline, len);
memcpy(cmdline_raw, cmdline, len);
cmdline_parse_buf(cmdline_buf, len, __ld_cmdline_start, __ld_cmdline_end);
}
boot/limine/entry.c
-135
View File
@@ -1,135 +0,0 @@
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/cmdline.h>
#include <zinnia/fdt.h>
#include <zinnia/init.h>
#include <zinnia/print.h>
#include <zinnia/utils.h>
#include <string.h>
#include "limine.h"
[[__initdata_sorted("limine.0")]]
static volatile uint64_t limine_start[] = LIMINE_REQUESTS_START_MARKER;
[[__initdata_sorted("limine.1")]]
static volatile uint64_t limine_revision[] = LIMINE_BASE_REVISION(6);
[[__initdata_sorted("limine.2")]]
static volatile uint64_t limine_end[] = LIMINE_REQUESTS_END_MARKER;
#define LIMINE_REQUEST(request, tag, rev) \
[[__initdata_sorted("limine.1")]] \
static volatile struct limine_##request request = { \
.id = tag, \
.revision = rev, \
.response = nullptr, \
}
LIMINE_REQUEST(memmap_request, LIMINE_MEMMAP_REQUEST_ID, 0);
LIMINE_REQUEST(hhdm_request, LIMINE_HHDM_REQUEST_ID, 0);
LIMINE_REQUEST(executable_address_request, LIMINE_EXECUTABLE_ADDRESS_REQUEST_ID, 0);
LIMINE_REQUEST(executable_file_request, LIMINE_EXECUTABLE_FILE_REQUEST_ID, 0);
LIMINE_REQUEST(framebuffer_request, LIMINE_FRAMEBUFFER_REQUEST_ID, 1);
LIMINE_REQUEST(module_request, LIMINE_MODULE_REQUEST_ID, 0);
LIMINE_REQUEST(dtb_request, LIMINE_DTB_REQUEST_ID, 0);
LIMINE_REQUEST(rsdp_request, LIMINE_RSDP_REQUEST_ID, 0);
[[__initdata]]
static struct phys_mem mem[128];
[[__initdata]]
static struct boot_file files[8];
[[__init]]
void kernel_entry() {
kernel_early_init();
struct boot_info info = {};
const struct limine_memmap_response* mm_res = memmap_request.response;
const struct limine_hhdm_response* hhdm_res = hhdm_request.response;
const struct limine_module_response* module_res = module_request.response;
const struct limine_executable_address_response* exec_res = executable_address_request.response;
const struct limine_executable_file_response* exec_file_res = executable_file_request.response;
const struct limine_rsdp_response* rsdp_res = rsdp_request.response;
assert(mm_res, "memmap_request returned no response!");
assert(hhdm_res, "hhdm_request returned no response!");
info.cmdline = exec_file_res->executable_file->string;
info.kernel_file.data = (phys_t)((uintptr_t)exec_file_res->executable_file->address - hhdm_res->offset);
info.kernel_file.length = exec_file_res->executable_file->size;
info.num_mem_maps = MIN(ARRAY_SIZE(mem), mm_res->entry_count);
for (size_t i = 0; i < info.num_mem_maps; i++) {
mem[i].address = mm_res->entries[i]->base;
mem[i].length = mm_res->entries[i]->length;
switch (mm_res->entries[i]->type) {
case LIMINE_MEMMAP_USABLE:
mem[i].usage = PHYS_USABLE;
break;
case LIMINE_MEMMAP_ACPI_RECLAIMABLE:
case LIMINE_MEMMAP_BOOTLOADER_RECLAIMABLE:
mem[i].usage = PHYS_RECLAIMABLE;
break;
case LIMINE_MEMMAP_EXECUTABLE_AND_MODULES:
mem[i].usage = PHYS_STATIC;
break;
default:
mem[i].usage = PHYS_RESERVED;
break;
}
}
if (module_res) {
info.files = files;
info.num_files = MIN(ARRAY_SIZE(files), module_res->module_count);
for (size_t i = 0; i < info.num_files; i++) {
files[i].data = (phys_t)module_res->modules[i]->address - hhdm_res->offset;
files[i].length = module_res->modules[i]->size;
strncpy(files[i].path, module_res->modules[i]->path, sizeof(files[i].path) - 1);
files[i].path[sizeof(files[i].path) - 1] = 0;
}
}
info.mem_map = mem;
info.phys_base = (phys_t)exec_res->physical_base;
info.virt_base = (uintptr_t)exec_res->virtual_base;
info.hhdm_base = (uintptr_t)hhdm_res->offset;
const struct limine_dtb_response* dtb_res = dtb_request.response;
if (dtb_res && dtb_res->dtb_ptr) {
info.dtb = (phys_t)((uintptr_t)dtb_res->dtb_ptr - hhdm_res->offset);
info.dtb_size = fdt_read_u32((const uint8_t*)dtb_res->dtb_ptr + 4);
}
const struct limine_framebuffer_response* fb_res = framebuffer_request.response;
if (fb_res) {
// Only use the first framebuffer.
if (fb_res->framebuffer_count >= 1) {
const struct limine_framebuffer* fb = framebuffer_request.response->framebuffers[0];
info.fb.base = (phys_t)((uintptr_t)fb->address - hhdm_res->offset);
info.fb.width = fb->width;
info.fb.height = fb->height;
info.fb.pitch = fb->pitch;
info.fb.bytes_per_pixel = fb->bpp / 8;
info.fb.red.shift = fb->red_mask_shift;
info.fb.red.size = fb->red_mask_size;
info.fb.green.shift = fb->green_mask_shift;
info.fb.green.size = fb->green_mask_size;
info.fb.blue.shift = fb->blue_mask_shift;
info.fb.blue.size = fb->blue_mask_size;
}
}
#if CONFIG_ACPI
if (rsdp_request.response) {
info.rsdp = (phys_t)rsdp_request.response->address;
if (info.rsdp >= info.hhdm_base)
info.rsdp -= info.hhdm_base;
}
#endif
kernel_main(&info);
}
boot/limine/limine.h
-669
View File
@@ -1,669 +0,0 @@
/* SPDX-License-Identifier: 0BSD */
/* Copyright (C) 2022-2026 Mintsuki and contributors.
*
* Permission to use, copy, modify, and/or distribute this software for any
* purpose with or without fee is hereby granted.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
* SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
* OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
* CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#ifndef LIMINE_H
#define LIMINE_H 1
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
/* Misc */
#ifdef LIMINE_NO_POINTERS
# define LIMINE_PTR(TYPE) uint64_t
#else
# define LIMINE_PTR(TYPE) TYPE
#endif
#define LIMINE_REQUESTS_START_MARKER { 0xf6b8f4b39de7d1ae, 0xfab91a6940fcb9cf, \
0x785c6ed015d3e316, 0x181e920a7852b9d9 }
#define LIMINE_REQUESTS_END_MARKER { 0xadc0e0531bb10d03, 0x9572709f31764c62 }
#define LIMINE_BASE_REVISION(N) { 0xf9562b2d5c95a6c8, 0x6a7b384944536bdc, (N) }
#define LIMINE_BASE_REVISION_SUPPORTED(VAR) ((VAR)[2] == 0)
#define LIMINE_LOADED_BASE_REVISION_VALID(VAR) ((VAR)[1] != 0x6a7b384944536bdc)
#define LIMINE_LOADED_BASE_REVISION(VAR) ((VAR)[1])
#define LIMINE_COMMON_MAGIC 0xc7b1dd30df4c8b88, 0x0a82e883a194f07b
struct limine_uuid {
uint32_t a;
uint16_t b;
uint16_t c;
uint8_t d[8];
};
#define LIMINE_MEDIA_TYPE_GENERIC 0
#define LIMINE_MEDIA_TYPE_OPTICAL 1
#define LIMINE_MEDIA_TYPE_TFTP 2
struct limine_file {
uint64_t revision;
LIMINE_PTR(void *) address;
uint64_t size;
LIMINE_PTR(char *) path;
LIMINE_PTR(char *) string;
uint32_t media_type;
uint32_t unused;
uint8_t tftp_ipv4[4];
uint32_t tftp_port;
uint32_t partition_index;
uint32_t mbr_disk_id;
struct limine_uuid gpt_disk_uuid;
struct limine_uuid gpt_part_uuid;
struct limine_uuid part_uuid;
};
/* Boot info */
#define LIMINE_BOOTLOADER_INFO_REQUEST_ID { LIMINE_COMMON_MAGIC, 0xf55038d8e2a1202f, 0x279426fcf5f59740 }
struct limine_bootloader_info_response {
uint64_t revision;
LIMINE_PTR(char *) name;
LIMINE_PTR(char *) version;
};
struct limine_bootloader_info_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_bootloader_info_response *) response;
};
/* Executable command line */
#define LIMINE_EXECUTABLE_CMDLINE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x4b161536e598651e, 0xb390ad4a2f1f303a }
struct limine_executable_cmdline_response {
uint64_t revision;
LIMINE_PTR(char *) cmdline;
};
struct limine_executable_cmdline_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_executable_cmdline_response *) response;
};
/* Firmware type */
#define LIMINE_FIRMWARE_TYPE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x8c2f75d90bef28a8, 0x7045a4688eac00c3 }
#define LIMINE_FIRMWARE_TYPE_X86BIOS 0
#define LIMINE_FIRMWARE_TYPE_EFI32 1
#define LIMINE_FIRMWARE_TYPE_EFI64 2
#define LIMINE_FIRMWARE_TYPE_SBI 3
struct limine_firmware_type_response {
uint64_t revision;
uint64_t firmware_type;
};
struct limine_firmware_type_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_firmware_type_response *) response;
};
/* Stack size */
#define LIMINE_STACK_SIZE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x224ef0460a8e8926, 0xe1cb0fc25f46ea3d }
struct limine_stack_size_response {
uint64_t revision;
};
struct limine_stack_size_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_stack_size_response *) response;
uint64_t stack_size;
};
/* HHDM */
#define LIMINE_HHDM_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x48dcf1cb8ad2b852, 0x63984e959a98244b }
struct limine_hhdm_response {
uint64_t revision;
uint64_t offset;
};
struct limine_hhdm_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_hhdm_response *) response;
};
/* Framebuffer */
#define LIMINE_FRAMEBUFFER_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x9d5827dcd881dd75, 0xa3148604f6fab11b }
#define LIMINE_FRAMEBUFFER_RGB 1
struct limine_video_mode {
uint64_t pitch;
uint64_t width;
uint64_t height;
uint16_t bpp;
uint8_t memory_model;
uint8_t red_mask_size;
uint8_t red_mask_shift;
uint8_t green_mask_size;
uint8_t green_mask_shift;
uint8_t blue_mask_size;
uint8_t blue_mask_shift;
};
struct limine_framebuffer {
LIMINE_PTR(void *) address;
uint64_t width;
uint64_t height;
uint64_t pitch;
uint16_t bpp;
uint8_t memory_model;
uint8_t red_mask_size;
uint8_t red_mask_shift;
uint8_t green_mask_size;
uint8_t green_mask_shift;
uint8_t blue_mask_size;
uint8_t blue_mask_shift;
uint8_t unused[7];
uint64_t edid_size;
LIMINE_PTR(void *) edid;
/* Response revision 1 */
uint64_t mode_count;
LIMINE_PTR(struct limine_video_mode **) modes;
};
struct limine_framebuffer_response {
uint64_t revision;
uint64_t framebuffer_count;
LIMINE_PTR(struct limine_framebuffer **) framebuffers;
};
struct limine_framebuffer_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_framebuffer_response *) response;
};
/* Flanterm FB init params */
#define LIMINE_FLANTERM_FB_INIT_PARAMS_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x3259399fe7c5f126, 0xe01c1c8c5db9d1a9 }
#define LIMINE_FLANTERM_FB_ROTATE_0 0
#define LIMINE_FLANTERM_FB_ROTATE_90 1
#define LIMINE_FLANTERM_FB_ROTATE_180 2
#define LIMINE_FLANTERM_FB_ROTATE_270 3
struct limine_flanterm_fb_init_params {
LIMINE_PTR(uint32_t *) canvas;
uint64_t canvas_size;
uint32_t ansi_colours[8];
uint32_t ansi_bright_colours[8];
uint32_t default_bg;
uint32_t default_fg;
uint32_t default_bg_bright;
uint32_t default_fg_bright;
LIMINE_PTR(void *) font;
uint64_t font_width;
uint64_t font_height;
uint64_t font_spacing;
uint64_t font_scale_x;
uint64_t font_scale_y;
uint64_t margin;
uint64_t rotation;
};
struct limine_flanterm_fb_init_params_response {
uint64_t revision;
uint64_t entry_count;
LIMINE_PTR(struct limine_flanterm_fb_init_params **) entries;
};
struct limine_flanterm_fb_init_params_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_flanterm_fb_init_params_response *) response;
};
/* Paging mode */
#define LIMINE_PAGING_MODE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x95c1a0edab0944cb, 0xa4e5cb3842f7488a }
#define LIMINE_PAGING_MODE_X86_64_4LVL 0
#define LIMINE_PAGING_MODE_X86_64_5LVL 1
#define LIMINE_PAGING_MODE_X86_64_MIN LIMINE_PAGING_MODE_X86_64_4LVL
#define LIMINE_PAGING_MODE_X86_64_DEFAULT LIMINE_PAGING_MODE_X86_64_4LVL
#define LIMINE_PAGING_MODE_AARCH64_4LVL 0
#define LIMINE_PAGING_MODE_AARCH64_5LVL 1
#define LIMINE_PAGING_MODE_AARCH64_MIN LIMINE_PAGING_MODE_AARCH64_4LVL
#define LIMINE_PAGING_MODE_AARCH64_DEFAULT LIMINE_PAGING_MODE_AARCH64_4LVL
#define LIMINE_PAGING_MODE_RISCV_SV39 0
#define LIMINE_PAGING_MODE_RISCV_SV48 1
#define LIMINE_PAGING_MODE_RISCV_SV57 2
#define LIMINE_PAGING_MODE_RISCV_MIN LIMINE_PAGING_MODE_RISCV_SV39
#define LIMINE_PAGING_MODE_RISCV_DEFAULT LIMINE_PAGING_MODE_RISCV_SV48
#define LIMINE_PAGING_MODE_LOONGARCH_4LVL 0
#define LIMINE_PAGING_MODE_LOONGARCH_MIN LIMINE_PAGING_MODE_LOONGARCH_4LVL
#define LIMINE_PAGING_MODE_LOONGARCH_DEFAULT LIMINE_PAGING_MODE_LOONGARCH_4LVL
struct limine_paging_mode_response {
uint64_t revision;
uint64_t mode;
};
struct limine_paging_mode_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_paging_mode_response *) response;
uint64_t mode;
uint64_t max_mode;
uint64_t min_mode;
};
/* MP */
#define LIMINE_MP_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x95a67b819a1b857e, 0xa0b61b723b6a73e0 }
struct limine_mp_info;
typedef void (*limine_goto_address)(struct limine_mp_info *);
#if defined (__x86_64__) || defined (__i386__)
#define LIMINE_MP_RESPONSE_X86_64_X2APIC (1 << 0)
struct limine_mp_info {
uint32_t processor_id;
uint32_t lapic_id;
uint64_t reserved;
LIMINE_PTR(limine_goto_address) goto_address;
uint64_t extra_argument;
};
struct limine_mp_response {
uint64_t revision;
uint32_t flags;
uint32_t bsp_lapic_id;
uint64_t cpu_count;
LIMINE_PTR(struct limine_mp_info **) cpus;
};
#elif defined (__aarch64__)
struct limine_mp_info {
uint32_t processor_id;
uint32_t reserved1;
uint64_t mpidr;
uint64_t reserved;
LIMINE_PTR(limine_goto_address) goto_address;
uint64_t extra_argument;
};
struct limine_mp_response {
uint64_t revision;
uint64_t flags;
uint64_t bsp_mpidr;
uint64_t cpu_count;
LIMINE_PTR(struct limine_mp_info **) cpus;
};
#elif defined (__riscv) && (__riscv_xlen == 64)
struct limine_mp_info {
uint64_t processor_id;
uint64_t hartid;
uint64_t reserved;
LIMINE_PTR(limine_goto_address) goto_address;
uint64_t extra_argument;
};
struct limine_mp_response {
uint64_t revision;
uint64_t flags;
uint64_t bsp_hartid;
uint64_t cpu_count;
LIMINE_PTR(struct limine_mp_info **) cpus;
};
#elif defined (__loongarch__) && (__loongarch_grlen == 64)
struct limine_mp_info {
uint64_t processor_id;
uint64_t phys_id;
uint64_t reserved;
LIMINE_PTR(limine_goto_address) goto_address;
uint64_t extra_argument;
};
struct limine_mp_response {
uint64_t revision;
uint64_t flags;
uint64_t bsp_phys_id;
uint64_t cpu_count;
LIMINE_PTR(struct limine_mp_info **) cpus;
};
#else
#error Unknown architecture
#endif
#define LIMINE_MP_REQUEST_X86_64_X2APIC (1 << 0)
struct limine_mp_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_mp_response *) response;
uint64_t flags;
};
/* Memory map */
#define LIMINE_MEMMAP_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x67cf3d9d378a806f, 0xe304acdfc50c3c62 }
#define LIMINE_MEMMAP_USABLE 0
#define LIMINE_MEMMAP_RESERVED 1
#define LIMINE_MEMMAP_ACPI_RECLAIMABLE 2
#define LIMINE_MEMMAP_ACPI_NVS 3
#define LIMINE_MEMMAP_BAD_MEMORY 4
#define LIMINE_MEMMAP_BOOTLOADER_RECLAIMABLE 5
#define LIMINE_MEMMAP_EXECUTABLE_AND_MODULES 6
#define LIMINE_MEMMAP_FRAMEBUFFER 7
#define LIMINE_MEMMAP_RESERVED_MAPPED 8
struct limine_memmap_entry {
uint64_t base;
uint64_t length;
uint64_t type;
};
struct limine_memmap_response {
uint64_t revision;
uint64_t entry_count;
LIMINE_PTR(struct limine_memmap_entry **) entries;
};
struct limine_memmap_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_memmap_response *) response;
};
/* Entry point */
#define LIMINE_ENTRY_POINT_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x13d86c035a1cd3e1, 0x2b0caa89d8f3026a }
typedef void (*limine_entry_point)(void);
struct limine_entry_point_response {
uint64_t revision;
};
struct limine_entry_point_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_entry_point_response *) response;
LIMINE_PTR(limine_entry_point) entry;
};
/* Executable File */
#define LIMINE_EXECUTABLE_FILE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0xad97e90e83f1ed67, 0x31eb5d1c5ff23b69 }
struct limine_executable_file_response {
uint64_t revision;
LIMINE_PTR(struct limine_file *) executable_file;
};
struct limine_executable_file_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_executable_file_response *) response;
};
/* Module */
#define LIMINE_MODULE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x3e7e279702be32af, 0xca1c4f3bd1280cee }
#define LIMINE_INTERNAL_MODULE_REQUIRED (1 << 0)
#define LIMINE_INTERNAL_MODULE_COMPRESSED (1 << 1)
struct limine_internal_module {
LIMINE_PTR(const char *) path;
LIMINE_PTR(const char *) string;
uint64_t flags;
};
struct limine_module_response {
uint64_t revision;
uint64_t module_count;
LIMINE_PTR(struct limine_file **) modules;
};
struct limine_module_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_module_response *) response;
/* Request revision 1 */
uint64_t internal_module_count;
LIMINE_PTR(struct limine_internal_module **) internal_modules;
};
/* RSDP */
#define LIMINE_RSDP_REQUEST_ID { LIMINE_COMMON_MAGIC, 0xc5e77b6b397e7b43, 0x27637845accdcf3c }
struct limine_rsdp_response {
uint64_t revision;
LIMINE_PTR(void *) address;
};
struct limine_rsdp_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_rsdp_response *) response;
};
/* SMBIOS */
#define LIMINE_SMBIOS_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x9e9046f11e095391, 0xaa4a520fefbde5ee }
struct limine_smbios_response {
uint64_t revision;
LIMINE_PTR(void *) entry_32;
LIMINE_PTR(void *) entry_64;
};
struct limine_smbios_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_smbios_response *) response;
};
/* EFI system table */
#define LIMINE_EFI_SYSTEM_TABLE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x5ceba5163eaaf6d6, 0x0a6981610cf65fcc }
struct limine_efi_system_table_response {
uint64_t revision;
LIMINE_PTR(void *) address;
};
struct limine_efi_system_table_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_efi_system_table_response *) response;
};
/* TPM event log */
#define LIMINE_TPM_EVENT_LOG_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x98e094fc7e76e979, 0xee8d8775c54e1d1f }
#define LIMINE_TPM_EVENT_LOG_FORMAT_TCG_1_2 1
#define LIMINE_TPM_EVENT_LOG_FORMAT_TCG_2 2
struct limine_tpm_event_log_response {
uint64_t revision;
uint64_t format;
uint64_t size;
LIMINE_PTR(void *) address;
};
struct limine_tpm_event_log_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_tpm_event_log_response *) response;
};
/* EFI memory map */
#define LIMINE_EFI_MEMMAP_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x7df62a431d6872d5, 0xa4fcdfb3e57306c8 }
struct limine_efi_memmap_response {
uint64_t revision;
LIMINE_PTR(void *) memmap;
uint64_t memmap_size;
uint64_t desc_size;
uint64_t desc_version;
};
struct limine_efi_memmap_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_efi_memmap_response *) response;
};
/* Date at boot */
#define LIMINE_DATE_AT_BOOT_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x502746e184c088aa, 0xfbc5ec83e6327893 }
struct limine_date_at_boot_response {
uint64_t revision;
int64_t timestamp;
};
struct limine_date_at_boot_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_date_at_boot_response *) response;
};
/* Executable address */
#define LIMINE_EXECUTABLE_ADDRESS_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x71ba76863cc55f63, 0xb2644a48c516a487 }
struct limine_executable_address_response {
uint64_t revision;
uint64_t physical_base;
uint64_t virtual_base;
};
struct limine_executable_address_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_executable_address_response *) response;
};
/* Device Tree Blob */
#define LIMINE_DTB_REQUEST_ID { LIMINE_COMMON_MAGIC, 0xb40ddb48fb54bac7, 0x545081493f81ffb7 }
struct limine_dtb_response {
uint64_t revision;
LIMINE_PTR(void *) dtb_ptr;
};
struct limine_dtb_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_dtb_response *) response;
};
/* RISC-V Boot Hart ID */
#define LIMINE_RISCV_BSP_HARTID_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x1369359f025525f9, 0x2ff2a56178391bb6 }
struct limine_riscv_bsp_hartid_response {
uint64_t revision;
uint64_t bsp_hartid;
};
struct limine_riscv_bsp_hartid_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_riscv_bsp_hartid_response *) response;
};
/* Bootloader Performance */
#define LIMINE_BOOTLOADER_PERFORMANCE_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x6b50ad9bf36d13ad, 0xdc4c7e88fc759e17 }
struct limine_bootloader_performance_response {
uint64_t revision;
uint64_t reset_usec;
uint64_t init_usec;
uint64_t exec_usec;
};
struct limine_bootloader_performance_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_bootloader_performance_response *) response;
};
#define LIMINE_X86_64_KEEP_IOMMU_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x8ebaabe51f490179, 0x2aa86a59ffb4ab0f }
struct limine_x86_64_keep_iommu_response {
uint64_t revision;
};
struct limine_x86_64_keep_iommu_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_x86_64_keep_iommu_response *) response;
};
/* TSC (Timestamp Counter) Frequency */
#define LIMINE_TSC_FREQUENCY_REQUEST_ID { LIMINE_COMMON_MAGIC, 0x10f2ee1d87d195e4, 0xf747a2b78f6ddb31 }
struct limine_tsc_frequency_response {
uint64_t revision;
uint64_t frequency;
};
struct limine_tsc_frequency_request {
uint64_t id[4];
uint64_t revision;
LIMINE_PTR(struct limine_tsc_frequency_response *) response;
};
#ifdef __cplusplus
}
#endif
#endif
boot/main.c
-90
View File
@@ -1,90 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/clock.h>
#include <zinnia/cmdline.h>
#include <zinnia/compiler.h>
#include <zinnia/console.h>
#include <zinnia/elf.h>
#include <zinnia/init.h>
#include <zinnia/initgraph.h>
#include <zinnia/initramfs.h>
#include <zinnia/irq.h>
#include <zinnia/module.h>
#include <zinnia/percpu.h>
#include <zinnia/print.h>
#include <zinnia/process.h>
#include <zinnia/sched.h>
#include <zinnia/utils.h>
#include <zinnia/vfs.h>
#include <zinnia/vma.h>
#include <zinnia/vmo.h>
#include <config.h>
#include <stdint.h>
static const char zinnia_banner[] = "Zinnia " ZINNIA_VERSION " " CONFIG_KERNEL_FLAVOR " (" ZINNIA_ARCH
", " ZINNIA_COMPILER_ID ", " ZINNIA_LINKER_ID ")";
// Path to the init executable.
static const char* init_path = "/init";
static void init_option(const char* value) {
init_path = value;
}
CMDLINE_OPTION("init", "Path to the init executable", init_option);
static void kernel_main_task(uintptr_t arg0) {
errno_t status = 0;
struct boot_info* info = (struct boot_info*)arg0;
log("Hello from main task!\n");
// Finish the initgraph.
initgraph_run(nullptr, info);
// After everything is loaded, load all builtin drivers.
module_init();
// Load all initramfs files.
initramfs_load_boot_modules(info);
const char* init_argv[] = {init_path, nullptr};
const char* init_envp[] = {"HOME=/", "TERM=console", nullptr};
log("Starting init \"%s\"\n", init_path);
log("With arguments:\n");
for (const char** arg = init_argv; *arg; arg++)
log(" %s\n", *arg);
log("With environment:\n");
for (const char** env = init_envp; *env; env++)
log(" %s\n", *env);
status = process_spawn_init(init_path, init_argv, init_envp);
assert(status == 0, "Failed to start init \"%s\" (%i)", init_path, status);
}
[[__init]]
void kernel_early_init() {
percpu_bsp_init();
percpu_get()->online = true;
irq_lock();
}
[[noreturn]]
void kernel_main(struct boot_info* info) {
cmdline_parse(info->cmdline);
log("%s\n", zinnia_banner); // Say hello!
log("Command line: %s\n", info->cmdline);
initgraph_run(sched_stage, info);
struct task* main_task;
task_create("main", process_kernel(), kernel_main_task, (uintptr_t)info, &main_task);
sched_add_task(&percpu_get()->sched, main_task);
irq_unlock();
sched_yield(&percpu_get()->sched);
unreachable();
}
dev/Kconfig
-19
View File
@@ -1,19 +0,0 @@
menu "Device Drivers"
config OF
bool "Support for OpenFirmware"
default y
help
Adds support for flattened device trees and OF platform drivers.
config ACPI
bool "Support for ACPI firmware"
default y
help
Adds support for ACPI firmware, tables and devices.
source "dev/block/Kconfig"
source "dev/pci/Kconfig"
source "dev/video/Kconfig"
endmenu
dev/acpi/include/uacpi_types.h
-41
View File
@@ -1,41 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <stdarg.h>
#include <stddef.h>
#include <stdint.h>
typedef uint8_t uacpi_u8;
typedef uint16_t uacpi_u16;
typedef uint32_t uacpi_u32;
typedef uint64_t uacpi_u64;
typedef int8_t uacpi_i8;
typedef int16_t uacpi_i16;
typedef int32_t uacpi_i32;
typedef int64_t uacpi_i64;
#define UACPI_TRUE true
#define UACPI_FALSE false
typedef bool uacpi_bool;
#define UACPI_NULL nullptr
typedef uintptr_t uacpi_uintptr;
typedef uacpi_uintptr uacpi_virt_addr;
typedef size_t uacpi_size;
typedef va_list uacpi_va_list;
#define uacpi_va_start va_start
#define uacpi_va_end va_end
#define uacpi_va_arg va_arg
typedef char uacpi_char;
#define uacpi_offsetof offsetof
#define UACPI_PRIu64 "llu"
#define UACPI_PRIx64 "llx"
#define UACPI_PRIX64 "llX"
#define UACPI_FMT64(val) ((unsigned long long)(val))
dev/acpi/include/zinnia/acpi.h
-19
View File
@@ -1,19 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/init.h>
#include <zinnia/initgraph.h>
#include <zinnia/types.h>
#include <stdint.h>
INIT_EXTERN(acpi_stage);
extern phys_t acpi_rsdp_addr;
struct acpi_match {
const char* hid;
uintptr_t data;
};
#define ACPI_MATCH_HID(id) .hid = (id)
dev/acpi/init.c
-47
View File
@@ -1,47 +0,0 @@
// SPDX-License-Identifier: ISC
#define pr_fmt(fmt) "acpi: " fmt
#include <zinnia/acpi.h>
#include <zinnia/alloc.h>
#include <zinnia/cmdline.h>
#include <zinnia/percpu.h>
#include <zinnia/print.h>
#include <zinnia/types.h>
#include <uacpi/status.h>
#include <uacpi/tables.h>
#include <uacpi/uacpi.h>
phys_t acpi_rsdp_addr = 0;
static char table_buf[0x1000];
static bool use_acpi = true;
void use_acpi_option(bool use) {
use_acpi = use;
}
CMDLINE_OPTION("acpi", "Enable ACPI firmware subsystem", use_acpi_option);
[[__init]]
static bool acpi_stage_init(struct boot_info* info) {
if (!use_acpi) {
log_warn("ACPI disabled on the command line!\n");
return false;
}
if (!info->rsdp) {
log_warn("No RSDP was passed, unable to start ACPI subsystem!\n");
return false;
}
acpi_rsdp_addr = info->rsdp;
if (uacpi_setup_early_table_access(table_buf, sizeof(table_buf)) != UACPI_STATUS_OK) {
log_err("Failed to set up early table access!\n");
return false;
}
return true;
}
INIT_TASK(acpi_stage, "dev.acpi", acpi_stage_init);
INIT_DEPENDS(acpi_stage, mem_stage);
INIT_ENTAILS(acpi_stage, percpu_stage);
dev/acpi/mcfg.c
-32
View File
@@ -1,32 +0,0 @@
#define pr_fmt(fmt) "acpi: " fmt
#include <zinnia/acpi.h>
#include <zinnia/pci.h>
#include <zinnia/print.h>
#include <uacpi/acpi.h>
#include <uacpi/status.h>
#include <uacpi/tables.h>
[[__init]]
static void acpi_mcfg_stage_init(struct boot_info*) {
uacpi_table table = {};
if (uacpi_table_find_by_signature(ACPI_MCFG_SIGNATURE, &table) != UACPI_STATUS_OK)
return;
struct acpi_mcfg* mcfg = table.ptr;
size_t count = 0;
if (mcfg->hdr.length > sizeof(*mcfg))
count = (mcfg->hdr.length - sizeof(*mcfg)) / sizeof(struct acpi_mcfg_allocation);
for (size_t i = 0; i < count; i++) {
struct acpi_mcfg_allocation* entry = &mcfg->entries[i];
errno_t status = pci_ecam_register(entry->address, entry->segment, entry->start_bus, entry->end_bus);
if (status != 0)
log_warn("Failed to register ECAM for segment %hu: %i\n", entry->segment, status);
}
uacpi_table_unref(&table);
}
INIT_TASK(acpi_mcfg_stage, "dev.acpi.mcfg", acpi_mcfg_stage_init);
INIT_DEPENDS(acpi_mcfg_stage, acpi_stage);
INIT_ENTAILS(acpi_mcfg_stage, pci_stage);
dev/acpi/meson.build
-20
View File
@@ -1,20 +0,0 @@
uacpi = subproject('uacpi')
kernel_sources += uacpi.get_variable('sources')
kernel_sources += files(
'init.c',
'uacpi_glue.c',
)
if kconfig.get('CONFIG_PCI', 'n') == 'y'
kernel_sources += files('mcfg.c')
endif
kernel_includes += uacpi.get_variable('includes')
kernel_includes += include_directories('include')
kernel_c_args += [
'-DUACPI_FORMATTED_LOGGING',
'-DUACPI_BAREBONES_MODE', # TODO: Remove
'-DUACPI_OVERRIDE_TYPES',
]
dev/acpi/uacpi_glue.c
-75
View File
@@ -1,75 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/acpi.h>
#include <zinnia/alloc.h>
#include <zinnia/console.h>
#include <zinnia/init.h>
#include <zinnia/mmio.h>
#include <zinnia/pmap.h>
#include <zinnia/print.h>
#include <zinnia/utils.h>
#include <zinnia/virt.h>
#include <zinnia/vma.h>
#include <arch/mem.h>
#include <stdarg.h>
#include <uacpi/kernel_api.h>
#include <uacpi/status.h>
#include <uacpi/types.h>
uacpi_status uacpi_kernel_get_rsdp(uacpi_phys_addr* out_rsdp_address) {
*out_rsdp_address = (uacpi_phys_addr)acpi_rsdp_addr;
return UACPI_STATUS_OK;
}
void* uacpi_kernel_map(uacpi_phys_addr addr, uacpi_size len) {
uintptr_t aligned_addr = ALIGN_DOWN(addr, arch_mem_page_size());
uintptr_t difference = addr - aligned_addr;
size_t aligned_len = ALIGN_UP(len + difference, arch_mem_page_size());
uintptr_t vaddr = vm_alloc(aligned_len);
if (!vaddr)
return nullptr;
for (size_t i = 0; i < aligned_len; i += arch_mem_page_size()) {
pmap_map(&kernel_vma.pmap, vaddr + i, aligned_addr + i, PTE_READ | PTE_WRITE, CACHE_ANY);
}
return (void*)(vaddr + difference);
}
void uacpi_kernel_unmap(void* addr, uacpi_size len) {
uintptr_t aligned_addr = ALIGN_DOWN((uintptr_t)addr, arch_mem_page_size());
uintptr_t difference = (uintptr_t)addr - aligned_addr;
size_t aligned_len = ALIGN_UP(len + difference, arch_mem_page_size());
for (size_t i = 0; i < aligned_len; i += arch_mem_page_size())
pmap_unmap(&kernel_vma.pmap, aligned_addr + i);
vm_free(aligned_addr, aligned_len);
}
static enum log_level acpi_log_level(uacpi_log_level lvl) {
switch (lvl) {
case UACPI_LOG_ERROR:
return LOG_ERR;
case UACPI_LOG_WARN:
return LOG_WARN;
case UACPI_LOG_INFO:
return LOG_INFO;
default:
return LOG_DEBUG;
}
}
void uacpi_kernel_log(uacpi_log_level lvl, const uacpi_char* msg, ...) {
va_list args;
va_start(args, msg);
log_vwrite_prefixed(acpi_log_level(lvl), "acpi: ", msg, args);
va_end(args);
}
void uacpi_kernel_vlog(uacpi_log_level lvl, const uacpi_char* msg, uacpi_va_list args) {
log_vwrite_prefixed(acpi_log_level(lvl), "acpi: ", msg, args);
}
dev/block/Kconfig
-5
View File
@@ -1,5 +0,0 @@
menu "Block device drivers"
source "dev/block/nvme/Kconfig"
endmenu
dev/block/bio.c
-160
View File
@@ -1,160 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/blockdev.h>
#include <zinnia/utils.h>
struct bio* bio_alloc(enum bio_op op, size_t num_segs) {
struct bio* bio = mem_alloc(sizeof(struct bio) + num_segs * sizeof(struct bio_seg), 0);
if (!bio)
return nullptr;
bio->op = op;
bio->num_segs = num_segs;
event_init(&bio->event);
atomic_init(&bio->pending, 0);
atomic_init(&bio->short_offset, UINT64_MAX);
atomic_init(&bio->child_status, 0);
return bio;
}
void bio_free(struct bio* bio) {
mem_free(bio);
}
static void record_short(struct bio* parent, uint64_t offset, errno_t status) {
uint64_t current = atomic_load(&parent->short_offset, ATOMIC_RELAXED);
while (offset < current) {
if (atomic_compare_exchange_weak(&parent->short_offset, &current, offset, ATOMIC_RELAXED, ATOMIC_RELAXED))
break;
}
if (status) {
errno_t expected = 0;
atomic_compare_exchange(&parent->child_status, &expected, status, ATOMIC_RELAXED, ATOMIC_RELAXED);
}
}
static void parent_put(struct bio* parent) {
if (atomic_fetch_sub(&parent->pending, 1, ATOMIC_ACQ_REL) != 1)
return;
const uint64_t offset = atomic_load(&parent->short_offset, ATOMIC_RELAXED);
const size_t done = offset == UINT64_MAX ? parent->num_lbas : offset;
const errno_t status = atomic_load(&parent->child_status, ATOMIC_RELAXED);
bio_complete(parent, done == 0 ? status : 0, done);
}
void bio_complete(struct bio* bio, errno_t status, size_t done_lbas) {
bio->status = status;
bio->done_lbas = done_lbas;
struct bio* parent = bio->parent;
if (parent) {
if (status || done_lbas < bio->num_lbas)
record_short(parent, bio->parent_offset + done_lbas, status);
bio_free(bio);
parent_put(parent);
return;
}
if (bio->end)
bio->end(bio);
else
event_set(&bio->event);
}
errno_t bio_wait(struct bio* bio, size_t* out_lbas) {
event_wait(&bio->event);
if (out_lbas)
*out_lbas = bio->done_lbas;
return bio->status;
}
// Submits one bio and completes early if it failed.
static void submit_one(struct blockdev* dev, struct bio* bio) {
errno_t status = dev->ops->submit(dev, bio);
if (status)
bio_complete(bio, status, 0);
}
void bio_submit(struct blockdev* dev, struct bio* bio) {
bio->dev = dev;
if (bio->op == BIO_FLUSH) {
submit_one(dev, bio);
return;
}
if (bio->op == BIO_READ) {
if (bio->lba >= dev->lba_count) {
bio_complete(bio, 0, 0);
return;
}
bio->num_lbas = MIN(bio->num_lbas, dev->lba_count - bio->lba);
}
if (bio->num_lbas == 0) {
bio_complete(bio, 0, 0);
return;
}
const size_t lba_size = dev->lba_size;
const size_t lbas_per_page = MAX(arch_mem_page_size() / lba_size, 1);
size_t max_lbas = dev->limits.max_lbas ? dev->limits.max_lbas : SIZE_MAX;
if (max_lbas > lbas_per_page)
max_lbas -= max_lbas % lbas_per_page;
const size_t max_segs = dev->limits.max_segs ? dev->limits.max_segs : SIZE_MAX;
if (bio->num_lbas <= max_lbas && bio->num_segs <= max_segs) {
submit_one(dev, bio);
return;
}
atomic_store(&bio->pending, 1, ATOMIC_RELAXED);
atomic_store(&bio->short_offset, UINT64_MAX, ATOMIC_RELAXED);
atomic_store(&bio->child_status, 0, ATOMIC_RELAXED);
size_t seg = 0;
size_t seg_off = 0;
uint64_t consumed = 0;
while (consumed < bio->num_lbas) {
const size_t cap = MIN(max_segs, bio->num_segs - seg);
struct bio* child = cap ? bio_alloc(bio->op, cap) : nullptr;
if (!child) {
record_short(bio, consumed, cap ? ENOMEM : EINVAL);
break;
}
size_t count = 0;
size_t lbas = 0;
while (lbas < max_lbas && count < cap && consumed + lbas < bio->num_lbas) {
const struct bio_seg* src = &bio->segs[seg];
const size_t take =
MIN(MIN(src->len - seg_off, (max_lbas - lbas) * lba_size),
(bio->num_lbas - consumed - lbas) * lba_size);
if (take < lba_size)
break;
child->segs[count++] = (struct bio_seg){.addr = src->addr + seg_off, .len = take};
lbas += take / lba_size;
seg_off += take;
if (seg_off == src->len) {
seg++;
seg_off = 0;
}
}
if (lbas == 0) {
bio_free(child);
record_short(bio, consumed, EINVAL);
break;
}
child->num_segs = count;
child->num_lbas = lbas;
child->lba = bio->lba + consumed;
child->parent = bio;
child->parent_offset = consumed;
child->dev = dev;
consumed += lbas;
atomic_fetch_add(&bio->pending, 1, ATOMIC_RELAXED);
submit_one(dev, child);
}
parent_put(bio);
}
dev/block/blockdev.c
-490
View File
@@ -1,490 +0,0 @@
// SPDX-License-Identifier: ISC
#define pr_fmt(fmt) "block: " fmt
#include <zinnia/alloc.h>
#include <zinnia/blockdev.h>
#include <zinnia/chardev.h>
#include <zinnia/devctl.h>
#include <zinnia/initgraph.h>
#include <zinnia/print.h>
#include <zinnia/tailq.h>
#include <zinnia/usercopy.h>
#include <zinnia/utils.h>
#include <zinnia/vec.h>
#include <zinnia/vma.h>
#include <string.h>
#include <uapi/block.h>
static size_t lbas_per_page(struct blockdev* dev) {
return arch_mem_page_size() / dev->lba_size;
}
static errno_t cache_get_pages(struct vmo* vmo, uint64_t index, struct page** pages, size_t count, size_t* out_valid) {
struct blockdev* dev = containerof(vmo, struct blockdev, cache);
const size_t page_size = arch_mem_page_size();
const size_t per_page = lbas_per_page(dev);
const uint64_t lba = index * per_page;
if (lba >= dev->lba_count) {
*out_valid = 0;
return 0;
}
const size_t num_lbas = MIN((uint64_t)count * per_page, dev->lba_count - lba);
size_t done;
errno_t status = blockdev_rw_pages(dev, BIO_READ, lba, num_lbas, pages, count, &done);
if (status)
return status;
const size_t valid = done / per_page + (done % per_page != 0);
if (done % per_page != 0) {
const size_t tail = (done % per_page) * dev->lba_size;
memset((uint8_t*)HHDM_PTR(vm_page_addr(pages[valid - 1])) + tail, 0, page_size - tail);
}
*out_valid = valid;
return valid == 0 && num_lbas > 0 ? EIO : 0;
}
static errno_t cache_put_pages(struct vmo* vmo, uint64_t index, struct page** pages, size_t count) {
struct blockdev* dev = containerof(vmo, struct blockdev, cache);
const size_t per_page = lbas_per_page(dev);
const uint64_t lba = index * per_page;
if (lba >= dev->lba_count)
return 0;
const size_t num_lbas = MIN((uint64_t)count * per_page, dev->lba_count - lba);
size_t done;
errno_t status = blockdev_rw_pages(dev, BIO_WRITE, lba, num_lbas, pages, count, &done);
if (status)
return status;
return done < num_lbas ? EIO : 0;
}
static void cache_destroy(struct vmo* vmo) {
struct blockdev* dev = containerof(vmo, struct blockdev, cache);
if (dev->ops->destroy)
dev->ops->destroy(dev);
}
static const struct vmo_pager_ops cache_pager = {
.get_pages = cache_get_pages,
.put_pages = cache_put_pages,
.max_cluster = 64,
.destroy = cache_destroy,
};
void blockdev_init(
struct blockdev* dev,
const struct blockdev_ops* ops,
uint32_t major,
uint32_t minor,
size_t lba_size,
uint64_t lba_count
) {
vmo_init(&dev->cache, &cache_pager);
dev->ops = ops;
dev->dev = dev_make(major, minor);
dev->lba_size = lba_size;
dev->lba_count = lba_count;
atomic_init(&dev->claimed, false);
}
void blockdev_ref(struct blockdev* dev) {
vmo_ref(&dev->cache);
}
void blockdev_unref(struct blockdev* dev) {
vmo_unref(&dev->cache);
}
errno_t blockdev_rw(
struct blockdev* dev,
enum bio_op op,
uint64_t lba,
size_t num_lbas,
const struct bio_seg* segs,
size_t num_segs,
size_t* out_lbas
) {
struct bio* bio = bio_alloc(op, num_segs);
if (!bio)
return ENOMEM;
memcpy(bio->segs, segs, num_segs * sizeof(struct bio_seg));
bio->lba = lba;
bio->num_lbas = num_lbas;
bio_submit(dev, bio);
errno_t status = bio_wait(bio, out_lbas);
bio_free(bio);
return status;
}
errno_t blockdev_rw_pages(
struct blockdev* dev,
enum bio_op op,
uint64_t lba,
size_t num_lbas,
struct page** pages,
size_t count,
size_t* out_lbas
) {
const size_t page_size = arch_mem_page_size();
size_t bytes = num_lbas * dev->lba_size;
if (bytes > count * page_size)
return EINVAL;
struct bio* bio = bio_alloc(op, count);
if (!bio)
return ENOMEM;
size_t num_segs = 0;
for (size_t i = 0; i < count && bytes > 0; i++) {
const size_t len = MIN(bytes, page_size);
bio->segs[num_segs++] = (struct bio_seg){.addr = vm_page_addr(pages[i]), .len = len};
bytes -= len;
}
bio->num_segs = num_segs;
bio->lba = lba;
bio->num_lbas = num_lbas;
bio_submit(dev, bio);
errno_t status = bio_wait(bio, out_lbas);
bio_free(bio);
return status;
}
errno_t blockdev_flush(struct blockdev* dev) {
struct bio* bio = bio_alloc(BIO_FLUSH, 0);
if (!bio)
return ENOMEM;
bio_submit(dev, bio);
errno_t status = bio_wait(bio, nullptr);
bio_free(bio);
return status;
}
static struct blockdev* file_dev(struct vfs_file* file) {
return file->priv;
}
static uint64_t dev_size(struct blockdev* dev) {
return dev->lba_count * dev->lba_size;
}
static errno_t blockdev_file_read(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
struct blockdev* dev = file_dev(file);
*out_read = 0;
if (offset >= dev_size(dev))
return 0;
size_t progress;
errno_t status =
vmo_read_iter(&dev->cache, offset, buffer, MIN(iovec_iter_len(buffer), dev_size(dev) - offset), &progress);
*out_read = progress;
return progress > 0 ? 0 : status;
}
static errno_t blockdev_file_write(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_written) {
struct blockdev* dev = file_dev(file);
*out_written = 0;
if (iovec_iter_len(buffer) == 0)
return 0;
if (offset >= dev_size(dev))
return ENOSPC;
size_t progress;
errno_t status =
vmo_write_iter(&dev->cache, offset, buffer, MIN(iovec_iter_len(buffer), dev_size(dev) - offset), &progress);
*out_written = progress;
return progress > 0 ? 0 : status;
}
static errno_t blockdev_file_ioctl(struct vfs_file* file, uintptr_t request, uintptr_t arg, uintptr_t* out_result) {
struct blockdev* dev = file_dev(file);
switch (request) {
case __BLKGETSIZE64: {
const uint64_t size = dev_size(dev);
if (!usercopy_write((__user void*)arg, &size, sizeof(size)))
return EFAULT;
*out_result = 0;
return 0;
}
case __BLKSSZGET: {
const unsigned int lba_size = dev->lba_size;
if (!usercopy_write((__user void*)arg, &lba_size, sizeof(lba_size)))
return EFAULT;
*out_result = 0;
return 0;
}
case __BLKRRPART: {
errno_t status = blockdev_rescan(dev);
if (!status)
*out_result = 0;
return status;
}
default:
if (dev->ops->ioctl)
return dev->ops->ioctl(dev, file, request, arg, out_result);
return ENOTTY;
}
}
static errno_t blockdev_file_sync(struct vfs_file* file, bool data_only) {
struct blockdev* dev = file_dev(file);
errno_t status = vmo_sync(&dev->cache, 0, UINT64_MAX);
if (status)
return status;
return blockdev_flush(dev);
}
static errno_t blockdev_file_mmap(
struct vfs_file* file,
struct vma* vma,
uintptr_t addr,
size_t len,
enum prot_flags prot,
enum map_flags flags,
off_t offset,
uintptr_t* out_addr
) {
errno_t status =
vma_map(vma, &file_dev(file)->cache, addr, len, prot, (flags & MAP_SHARED) ? VMA_SHARED : 0, offset);
if (status)
return status;
*out_addr = addr;
return 0;
}
static void blockdev_file_release(struct vfs_file* file) {
blockdev_unref(file_dev(file));
}
static const struct vfs_file_ops blockdev_file_ops = {
.read = blockdev_file_read,
.write = blockdev_file_write,
.ioctl = blockdev_file_ioctl,
.sync = blockdev_file_sync,
.mmap = blockdev_file_mmap,
.release = blockdev_file_release,
};
static errno_t blockdev_node_open(struct vfs_node* node, struct vfs_file* file) {
struct blockdev* dev = node->priv;
blockdev_ref(dev);
file->priv = dev;
file->ops = &blockdev_file_ops;
return 0;
}
static void blockdev_node_destroy(struct vfs_node* node) {
blockdev_unref(node->priv);
}
static const struct vfs_node_ops blockdev_node_ops = {
.destroy = blockdev_node_destroy,
.device = {
.open = blockdev_node_open,
},
};
errno_t devtmpfs_add_blockdev(const char* name, struct blockdev* dev, mode_t mode) {
struct vfs_mknod_target target = {
.type = VNODE_BLOCKDEV,
.ops = &blockdev_node_ops,
.priv = dev,
.rdev = dev->dev,
};
blockdev_ref(dev);
struct vfs_path root = devtmpfs_root();
errno_t status = vfs_mknod(root, root, name, mode, &target, ident_kernel(), nullptr);
vfs_path_drop(root);
if (status) {
blockdev_unref(dev);
return status;
}
devctl_notify_create(name);
return 0;
}
errno_t blockdev_from_node(struct vfs_node* node, struct blockdev** out) {
if (node->type != VNODE_BLOCKDEV || node->ops != &blockdev_node_ops)
return ENOTBLK;
struct blockdev* dev = node->priv;
blockdev_ref(dev);
*out = dev;
return 0;
}
errno_t blockdev_claim(struct blockdev* dev) {
bool expected = false;
if (!atomic_compare_exchange(&dev->claimed, &expected, true, ATOMIC_ACQUIRE, ATOMIC_RELAXED))
return EBUSY;
return 0;
}
void blockdev_release(struct blockdev* dev) {
atomic_store(&dev->claimed, false, ATOMIC_RELEASE);
}
struct disk_node {
char name[64];
struct blockdev* dev;
};
struct disk {
char name[32];
struct blockdev* dev;
vec(struct disk_node) nodes;
tailq_link(struct disk) link;
};
static tailq_head(struct disk) disks;
static mutex_t disks_lock;
static void add_node(struct disk* disk, const char* name, struct blockdev* dev) {
struct disk_node node = {.dev = dev};
ksnprintf(node.name, sizeof(node.name), "%s", name);
if (vec_push(&disk->nodes, node) && dev)
blockdev_unref(dev);
}
static void scan_partitions(struct disk* disk) {
struct gpt_partition* parts;
size_t count;
if (gpt_scan(disk->dev, &parts, &count))
return;
for (size_t i = 0; i < count; i++) {
const struct gpt_partition* part = &parts[i];
if (part->end_lba < part->start_lba || part->end_lba >= disk->dev->lba_count) {
log_warn("Ignoring partition %zu of \"%s\" with invalid range\n", i + 1, disk->name);
continue;
}
char part_name[48];
ksnprintf(part_name, sizeof(part_name), "%sp%zu", disk->name, i + 1);
struct blockdev* dev;
errno_t status = partition_new(disk->dev, part->start_lba, part->end_lba - part->start_lba + 1, &dev);
if (status) {
log_warn("Unable to create partition \"%s\" (%i)\n", part_name, status);
continue;
}
char path[64];
ksnprintf(path, sizeof(path), "block/%s", part_name);
status = devtmpfs_add_blockdev(path, dev, 0660);
if (status) {
blockdev_unref(dev);
log_warn("Unable to register \"%s\" (%i)\n", path, status);
continue;
}
add_node(disk, path, dev);
char type_str[GUID_STR_LEN];
char uuid_str[GUID_STR_LEN];
guid_format(&part->type_guid, type_str);
guid_format(&part->unique_guid, uuid_str);
ksnprintf(path, sizeof(path), "block/parttype-%s", type_str);
status = devtmpfs_symlink(path, part_name);
if (!status)
add_node(disk, path, nullptr);
else if (status == EEXIST)
log("Partition type %s already claimed, no \"%s\" link for \"%s\"\n", type_str, path, part_name);
ksnprintf(path, sizeof(path), "block/partuuid-%s", uuid_str);
status = devtmpfs_symlink(path, part_name);
if (!status)
add_node(disk, path, nullptr);
log("Partition %zu: \"%s\" Type: %s UUID: %s\n", i + 1, part_name, type_str, uuid_str);
}
mem_free(parts);
}
errno_t blockdev_register(const char* name, struct blockdev* dev) {
struct disk* disk = mem_alloc(sizeof(struct disk), 0);
if (!disk)
return ENOMEM;
ksnprintf(disk->name, sizeof(disk->name), "%s", name);
vec_init(&disk->nodes);
char path[64];
ksnprintf(path, sizeof(path), "block/%s", name);
errno_t status = devtmpfs_add_blockdev(path, dev, 0660);
if (status) {
mem_free(disk);
return status;
}
blockdev_ref(dev);
disk->dev = dev;
log("Registered block device \"%s\"\n", name);
mutex_lock(&disks_lock);
tailq_insert_tail(&disks, disk, link);
scan_partitions(disk);
mutex_unlock(&disks_lock);
return 0;
}
errno_t blockdev_rescan(struct blockdev* dev) {
mutex_lock(&disks_lock);
struct disk* disk;
for (disk = tailq_first(&disks); disk; disk = tailq_next(disk, link)) {
if (disk->dev == dev)
break;
}
if (!disk) {
mutex_unlock(&disks_lock);
return ENOTTY;
}
for (size_t i = 0; i < vec_length(&disk->nodes); i++) {
struct disk_node* node = &vec_elem_get(&disk->nodes, i);
errno_t status = devtmpfs_remove(node->name);
if (status)
log_warn("Unable to remove stale node \"%s\" (%i)\n", node->name, status);
if (node->dev)
blockdev_unref(node->dev);
}
disk->nodes.length = 0;
scan_partitions(disk);
mutex_unlock(&disks_lock);
return 0;
}
errno_t blockdev_sync_all() {
errno_t result = 0;
mutex_lock(&disks_lock);
for (struct disk* disk = tailq_first(&disks); disk; disk = tailq_next(disk, link)) {
for (size_t i = 0; i < vec_length(&disk->nodes); i++) {
struct blockdev* part = vec_elem_get(&disk->nodes, i).dev;
errno_t status = part ? vmo_sync(&part->cache, 0, UINT64_MAX) : 0;
if (status && !result)
result = status;
}
errno_t status = vmo_sync(&disk->dev->cache, 0, UINT64_MAX);
if (!status)
status = blockdev_flush(disk->dev);
if (status && !result)
result = status;
}
mutex_unlock(&disks_lock);
return result;
}
static void block_stage_init(struct boot_info* info) {
tailq_init(&disks);
mutex_init(&disks_lock);
errno_t status = devtmpfs_mkdir("block", 0755);
if (status && status != EEXIST)
log_warn("Unable to create /dev/block (%i)\n", status);
}
INIT_TASK(block_stage, "dev.block", block_stage_init);
INIT_DEPENDS(block_stage, devtmpfs_stage);
dev/block/gpt.c
-70
View File
@@ -1,70 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/blockdev.h>
#include <string.h>
struct [[__packed]] gpt_header {
char signature[8];
uint32_t revision;
uint32_t header_size;
uint32_t header_crc32;
uint32_t reserved;
uint64_t my_lba;
uint64_t alternate_lba;
uint64_t first_usable_lba;
uint64_t last_usable_lba;
struct guid disk_guid;
uint64_t partition_entry_lba;
uint32_t num_partition_entries;
uint32_t partition_entry_size;
uint32_t partition_entry_array_crc32;
};
struct [[__packed]] gpt_entry {
struct guid type_guid;
struct guid unique_guid;
uint64_t start_lba;
uint64_t end_lba;
uint64_t attributes;
uint16_t name[36];
};
constexpr size_t GPT_MAX_ENTRIES = 1024;
errno_t gpt_scan(struct blockdev* dev, struct gpt_partition** out, size_t* out_count) {
struct gpt_header header;
size_t read;
errno_t status = vmo_read(&dev->cache, dev->lba_size, &header, sizeof(header), &read);
if (status)
return status;
if (read != sizeof(header) || memcmp(header.signature, "EFI PART", 8) != 0)
return ENODATA;
if (header.partition_entry_size < sizeof(struct gpt_entry) || header.num_partition_entries > GPT_MAX_ENTRIES)
return EINVAL;
struct gpt_partition* parts = mem_alloc(MAX(header.num_partition_entries, 1) * sizeof(struct gpt_partition), 0);
if (!parts)
return ENOMEM;
size_t count = 0;
const uint64_t base = header.partition_entry_lba * dev->lba_size;
for (size_t i = 0; i < header.num_partition_entries; i++) {
struct gpt_entry entry;
status = vmo_read(&dev->cache, base + i * header.partition_entry_size, &entry, sizeof(entry), &read);
if (status || read != sizeof(entry))
break;
if (guid_is_zero(&entry.type_guid))
continue;
struct gpt_partition* part = &parts[count++];
part->type_guid = entry.type_guid;
part->unique_guid = entry.unique_guid;
part->start_lba = entry.start_lba;
part->end_lba = entry.end_lba;
}
*out = parts;
*out_count = count;
return 0;
}
dev/block/meson.build
-11
View File
@@ -1,11 +0,0 @@
kernel_sources += files(
'bio.c',
'blockdev.c',
'gpt.c',
'partition.c',
'ram.c',
)
if kconfig.get('CONFIG_NVME_CORE', 'n') != 'n'
subdir('nvme')
endif
dev/block/nvme/Kconfig
-13
View File
@@ -1,13 +0,0 @@
menu "NVMe Support"
config NVME_CORE
bool
default y
config NVME_PCI
bool "NVMe via PCI transport"
default y
depends on PCI
depends on NVME_CORE
endmenu
dev/block/nvme/core.c
-41
View File
@@ -1,41 +0,0 @@
#define pr_fmt(fmt) "nvme: " fmt
#include <zinnia/fdt.h>
#include <zinnia/module.h>
#include <zinnia/pci.h>
#include <zinnia/print.h>
#include <zinnia/utils.h>
#include <config.h>
#if CONFIG_NVME_PCI
#include "nvme_pci.h"
#endif
struct nvme_dev {
#if CONFIG_NVME_PCI
struct pci_device pci_dev;
#endif
};
errno_t nvme_init(struct module* mod) {
errno_t status = 0;
#if CONFIG_NVME_PCI
status = pci_driver_register(&nvme_driver);
#endif
return status;
}
errno_t nvme_exit(struct module* mod) {
// TODO: Unregister
return 0;
}
MODULE_ENTRY(nvme_init);
MODULE_EXIT(nvme_exit);
MODULE_DESC("NVMe Block Devices");
MODULE_AUTHOR("Marvin Friedrich");
MODULE_LICENSE("ISC");
MODULE_VERSION("1.0");
dev/block/nvme/meson.build
-12
View File
@@ -1,12 +0,0 @@
nvme_sources = files('core.c')
if kconfig.get('CONFIG_NVME_PCI', 'n') == 'y'
nvme_sources += files('pci.c')
endif
modules += {
'nvme': {
'modular': kconfig.get('CONFIG_NVME_CORE') == 'm',
'sources': nvme_sources,
},
}
dev/block/nvme/nvme_pci.h
-5
View File
@@ -1,5 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
extern struct pci_driver nvme_driver;
dev/block/nvme/pci.c
-25
View File
@@ -1,25 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/errno.h>
#include <zinnia/module.h>
#include <zinnia/pci.h>
static errno_t nvme_pci_probe(struct pci_device* dev, const struct pci_match* match) {
pci_log(dev, "Detected controller \n");
return 0;
}
static const struct pci_match nvme_match[] = {
// Generic NVMe device. Assume no quirks or any funny business.
{PCI_MATCH_CLASS(1, 8, 2), .data = 0},
};
MATCH_TABLE("pci", nvme_match);
struct pci_driver nvme_driver = {
.name = "nvme_pci",
.match = nvme_match,
.num_matches = ARRAY_SIZE(nvme_match),
.probe = nvme_pci_probe,
};
dev/block/nvme/spec.h
-3
View File
@@ -1,3 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
dev/block/partition.c
-78
View File
@@ -1,78 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/atomic.h>
#include <zinnia/blockdev.h>
#include <stdint.h>
struct partition {
struct blockdev blk;
struct blockdev* parent;
uint64_t start;
};
static _Atomic(uint32_t) next_minor;
static errno_t partition_submit(struct blockdev* blk, struct bio* bio) {
struct partition* part = containerof(blk, struct partition, blk);
struct blockdev* parent = part->parent;
if (bio->op != BIO_FLUSH) {
if (bio->lba >= blk->lba_count) {
bio_complete(bio, bio->op == BIO_READ ? 0 : ENOSPC, 0);
return 0;
}
if (bio->op == BIO_WRITE && bio->num_lbas > blk->lba_count - bio->lba) {
bio_complete(bio, ENOSPC, 0);
return 0;
}
bio->lba += part->start;
}
return parent->ops->submit(parent, bio);
}
static errno_t partition_ioctl(
struct blockdev* blk,
struct vfs_file* file,
uintptr_t request,
uintptr_t arg,
uintptr_t* out
) {
struct blockdev* parent = containerof(blk, struct partition, blk)->parent;
if (!parent->ops->ioctl)
return ENOTTY;
return parent->ops->ioctl(parent, file, request, arg, out);
}
static void partition_destroy(struct blockdev* blk) {
struct partition* part = containerof(blk, struct partition, blk);
blockdev_unref(part->parent);
mem_free(part);
}
static const struct blockdev_ops partition_ops = {
.submit = partition_submit,
.ioctl = partition_ioctl,
.destroy = partition_destroy,
};
errno_t partition_new(struct blockdev* parent, uint64_t start_lba, uint64_t lba_count, struct blockdev** out) {
struct partition* part = mem_alloc(sizeof(struct partition), 0);
if (!part)
return ENOMEM;
blockdev_init(
&part->blk,
&partition_ops,
259,
atomic_fetch_add(&next_minor, 1, ATOMIC_RELAXED),
parent->lba_size,
lba_count
);
part->blk.limits = parent->limits;
blockdev_ref(parent);
part->parent = parent;
part->start = start_lba;
*out = &part->blk;
return 0;
}
dev/block/ram.c
-71
View File
@@ -1,71 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/atomic.h>
#include <zinnia/blockdev.h>
#include <string.h>
constexpr uint32_t RAM_MAJOR = 1;
constexpr size_t RAM_LBA_SIZE = 512;
struct ramdisk {
struct blockdev blk;
phys_t base;
};
static atomic_uint next_minor;
static errno_t ramdisk_submit(struct blockdev* blk, struct bio* bio) {
struct ramdisk* ram = containerof(blk, struct ramdisk, blk);
if (bio->op == BIO_FLUSH) {
bio_complete(bio, 0, 0);
return 0;
}
if (bio->lba >= blk->lba_count || bio->num_lbas > blk->lba_count - bio->lba) {
bio_complete(bio, bio->op == BIO_READ ? 0 : ENOSPC, 0);
return 0;
}
uint64_t lba = bio->lba;
for (size_t i = 0; i < bio->num_segs; i++) {
const struct bio_seg* seg = &bio->segs[i];
void* store = HHDM_PTR(ram->base + lba * RAM_LBA_SIZE);
void* buffer = HHDM_PTR(seg->addr);
if (bio->op == BIO_READ)
memcpy(buffer, store, seg->len);
else
memcpy(store, buffer, seg->len);
lba += seg->len / RAM_LBA_SIZE;
}
bio_complete(bio, 0, bio->num_lbas);
return 0;
}
static void ramdisk_destroy(struct blockdev* blk) {
mem_free(containerof(blk, struct ramdisk, blk));
}
static const struct blockdev_ops ramdisk_ops = {
.submit = ramdisk_submit,
.destroy = ramdisk_destroy,
};
errno_t ramdisk_new(phys_t base, size_t length, struct blockdev** out) {
struct ramdisk* ram = mem_alloc(sizeof(struct ramdisk), 0);
if (!ram)
return ENOMEM;
blockdev_init(
&ram->blk,
&ramdisk_ops,
RAM_MAJOR,
atomic_fetch_add(&next_minor, 1, ATOMIC_RELAXED),
RAM_LBA_SIZE,
length / RAM_LBA_SIZE
);
ram->base = base;
*out = &ram->blk;
return 0;
}
dev/char/chardev.c
-89
View File
@@ -1,89 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/chardev.h>
#include <zinnia/devctl.h>
void chardev_init(
struct chardev* dev,
const struct chardev_ops* ops,
const struct vfs_file_ops* file_ops,
uint32_t major,
uint32_t minor
) {
refcount_init(&dev->rc);
dev->ops = ops;
dev->file_ops = file_ops;
dev->dev = dev_make(major, minor);
}
void chardev_ref(struct chardev* dev) {
refcount_inc(&dev->rc);
}
void chardev_unref(struct chardev* dev) {
if (refcount_dec(&dev->rc) && dev->ops && dev->ops->destroy)
dev->ops->destroy(dev);
}
static errno_t chardev_node_open(struct vfs_node* node, struct vfs_file* file) {
struct chardev* dev = node->priv;
file->ops = dev->file_ops;
if (dev->ops && dev->ops->open)
return dev->ops->open(dev, file);
return 0;
}
static void chardev_node_destroy(struct vfs_node* node) {
chardev_unref(node->priv);
}
static const struct vfs_node_ops chardev_node_ops = {
.destroy = chardev_node_destroy,
.device = {
.open = chardev_node_open,
},
};
errno_t devtmpfs_add_chardev(const char* name, struct chardev* dev, mode_t mode) {
struct vfs_mknod_target target = {
.type = VNODE_CHARDEV,
.ops = &chardev_node_ops,
.priv = dev,
.rdev = dev->dev,
};
chardev_ref(dev);
struct vfs_path root = devtmpfs_root();
errno_t status = vfs_mknod(root, root, name, mode, &target, ident_kernel(), nullptr);
vfs_path_drop(root);
if (status) {
chardev_unref(dev);
return status;
}
devctl_notify_create(name);
return 0;
}
errno_t devtmpfs_mkdir(const char* name, mode_t mode) {
struct vfs_path root = devtmpfs_root();
errno_t status = vfs_mkdir(root, root, name, mode, ident_kernel());
vfs_path_drop(root);
return status;
}
errno_t devtmpfs_remove(const char* name) {
struct vfs_path root = devtmpfs_root();
errno_t status = vfs_unlink(root, root, name, ident_kernel());
vfs_path_drop(root);
if (status)
return status;
devctl_notify_destroy(name);
return 0;
}
errno_t devtmpfs_symlink(const char* name, const char* target) {
struct vfs_path root = devtmpfs_root();
errno_t status = vfs_symlink(root, root, name, target, ident_kernel());
vfs_path_drop(root);
return status;
}
dev/char/devctl.c
-262
View File
@@ -1,262 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/atomic.h>
#include <zinnia/chardev.h>
#include <zinnia/devctl.h>
#include <zinnia/mutex.h>
#include <zinnia/poll.h>
#include <zinnia/tailq.h>
#include <string.h>
constexpr size_t DEVCTL_QUEUE_CAPACITY = 256;
constexpr size_t DEVCTL_BACKLOG_CAPACITY = 256;
struct devctl_line {
tailq_link(struct devctl_line) link;
size_t len;
char data[];
};
struct devctl_queue {
tailq_head(struct devctl_line) lines;
size_t count;
};
struct devctl_reader {
tailq_link(struct devctl_reader) link;
struct devctl_queue queue;
};
static mutex_t devctl_lock;
static tailq_head(struct devctl_reader) devctl_readers;
static struct devctl_queue devctl_backlog;
static bool devctl_had_reader;
static struct poll_event devctl_event;
static atomic_bool devctl_ready;
static struct chardev devctl_dev;
static void queue_init(struct devctl_queue* queue) {
tailq_init(&queue->lines);
queue->count = 0;
}
static struct devctl_line* queue_pop(struct devctl_queue* queue) {
struct devctl_line* line = tailq_first(&queue->lines);
if (line) {
tailq_remove(&queue->lines, line, link);
queue->count--;
}
return line;
}
static void queue_push(struct devctl_queue* queue, struct devctl_line* line, size_t capacity) {
if (queue->count >= capacity)
mem_free(queue_pop(queue));
tailq_insert_tail(&queue->lines, line, link);
queue->count++;
}
static void queue_clear(struct devctl_queue* queue) {
struct devctl_line* line;
while ((line = queue_pop(queue)))
mem_free(line);
}
static struct devctl_line* line_new(const char* data, size_t len) {
struct devctl_line* line = mem_alloc(sizeof(*line) + len, 0);
if (!line)
return nullptr;
line->len = len;
memcpy(line->data, data, len);
return line;
}
static void devctl_notify(const char* type, const char* name) {
if (!atomic_load(&devctl_ready, ATOMIC_ACQUIRE))
return;
while (*name == '/')
name++;
const char prefix[] = "!system=DEVFS subsystem=CDEV type=";
const char middle[] = " cdev=";
const size_t prefix_len = sizeof(prefix) - 1;
const size_t type_len = strlen(type);
const size_t middle_len = sizeof(middle) - 1;
const size_t name_len = strlen(name);
const size_t len = prefix_len + type_len + middle_len + name_len + 1;
char* buf = mem_alloc(len, 0);
if (!buf)
return;
char* pos = buf;
memcpy(pos, prefix, prefix_len);
pos += prefix_len;
memcpy(pos, type, type_len);
pos += type_len;
memcpy(pos, middle, middle_len);
pos += middle_len;
memcpy(pos, name, name_len);
pos += name_len;
*pos = '\n';
mutex_lock(&devctl_lock);
if (tailq_empty(&devctl_readers)) {
if (!devctl_had_reader) {
struct devctl_line* line = line_new(buf, len);
if (line)
queue_push(&devctl_backlog, line, DEVCTL_BACKLOG_CAPACITY);
}
} else {
for (struct devctl_reader* reader = tailq_first(&devctl_readers); reader; reader = tailq_next(reader, link)) {
struct devctl_line* line = line_new(buf, len);
if (line)
queue_push(&reader->queue, line, DEVCTL_QUEUE_CAPACITY);
}
}
mutex_unlock(&devctl_lock);
mem_free(buf);
poll_event_notify(&devctl_event);
}
void devctl_notify_create(const char* name) {
devctl_notify("CREATE", name);
}
void devctl_notify_destroy(const char* name) {
devctl_notify("DESTROY", name);
}
static bool file_nonblocking(struct vfs_file* file) {
return atomic_load(&file->flags, ATOMIC_RELAXED) & O_NONBLOCK;
}
static errno_t drain_lines(struct devctl_queue* queue, iovec_iter_t* buffer, size_t* out_total) {
size_t total = 0;
errno_t status = 0;
struct devctl_line* line;
while ((line = tailq_first(&queue->lines))) {
if (line->len > iovec_iter_len(buffer)) {
if (total == 0)
status = EINVAL;
break;
}
queue_pop(queue);
size_t copied;
status = iovec_iter_copy_to(buffer, line->data, line->len, &copied);
mem_free(line);
total += copied;
if (status)
break;
}
*out_total = total;
return status;
}
static errno_t devctl_read(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
struct devctl_reader* reader = file->priv;
struct poll_table table;
poll_table_init(&table);
poll_wait(&table, &devctl_event);
errno_t status = 0;
size_t read = 0;
mutex_lock(&devctl_lock);
while (true) {
if (reader->queue.count) {
status = drain_lines(&reader->queue, buffer, &read);
break;
}
if (file_nonblocking(file)) {
status = EAGAIN;
break;
}
if (poll_table_sleep(&table, &devctl_lock, nullptr) == EINTR) {
status = ERESTART;
break;
}
}
mutex_unlock(&devctl_lock);
poll_table_fini(&table);
*out_read = read;
return status;
}
static errno_t devctl_write(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_written) {
return EBADF;
}
static enum poll_flags devctl_poll(struct vfs_file* file, enum poll_flags events, struct poll_table* table) {
struct devctl_reader* reader = file->priv;
if (events & (POLLIN | POLLRDNORM))
poll_wait(table, &devctl_event);
enum poll_flags revents = 0;
mutex_lock(&devctl_lock);
if (reader->queue.count)
revents |= POLLIN | POLLRDNORM;
mutex_unlock(&devctl_lock);
return revents;
}
static void devctl_release(struct vfs_file* file) {
struct devctl_reader* reader = file->priv;
mutex_lock(&devctl_lock);
tailq_remove(&devctl_readers, reader, link);
mutex_unlock(&devctl_lock);
queue_clear(&reader->queue);
mem_free(reader);
}
static const struct vfs_file_ops devctl_file_ops = {
.read = devctl_read,
.write = devctl_write,
.poll = devctl_poll,
.release = devctl_release,
};
static errno_t devctl_open(struct chardev* dev, struct vfs_file* file) {
struct devctl_reader* reader = mem_alloc(sizeof(*reader), 0);
if (!reader)
return ENOMEM;
queue_init(&reader->queue);
mutex_lock(&devctl_lock);
if (!devctl_had_reader) {
devctl_had_reader = true;
struct devctl_line* line;
while ((line = queue_pop(&devctl_backlog)))
queue_push(&reader->queue, line, DEVCTL_QUEUE_CAPACITY);
}
tailq_insert_tail(&devctl_readers, reader, link);
mutex_unlock(&devctl_lock);
file->priv = reader;
vfs_stream_open(file);
return 0;
}
static const struct chardev_ops devctl_dev_ops = {.open = devctl_open};
static void devctl_stage_init(struct boot_info* info) {
mutex_init(&devctl_lock);
tailq_init(&devctl_readers);
queue_init(&devctl_backlog);
poll_event_init(&devctl_event);
atomic_store(&devctl_ready, true, ATOMIC_RELEASE);
chardev_init(&devctl_dev, &devctl_dev_ops, &devctl_file_ops, 5, 3);
errno_t status = devtmpfs_add_chardev("devctl", &devctl_dev, 0600);
assert(status == 0, "Unable to create /dev/devctl (%i)", status);
}
INIT_TASK(devctl_stage, "dev.devctl", devctl_stage_init);
INIT_DEPENDS(devctl_stage, devtmpfs_stage);
dev/char/memfiles.c
-102
View File
@@ -1,102 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/assert.h>
#include <zinnia/chardev.h>
#include <zinnia/random.h>
#include <zinnia/utils.h>
constexpr size_t MEMFILE_CHUNK = 64;
static const uint8_t zero_chunk[MEMFILE_CHUNK];
static errno_t memfile_discard(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_written) {
*out_written = iovec_iter_len(buffer);
return 0;
}
static errno_t memfile_read_zero(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
size_t total = 0;
errno_t status = 0;
while (iovec_iter_len(buffer)) {
size_t copied;
status = iovec_iter_copy_to(buffer, zero_chunk, MIN(iovec_iter_len(buffer), MEMFILE_CHUNK), &copied);
total += copied;
if (status)
break;
}
*out_read = total;
return status;
}
static errno_t null_read(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
*out_read = 0;
return 0;
}
static const struct vfs_file_ops null_file_ops = {
.read = null_read,
.write = memfile_discard,
};
static const struct vfs_file_ops zero_file_ops = {
.read = memfile_read_zero,
.write = memfile_discard,
};
static errno_t full_write(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_written) {
*out_written = 0;
return ENOSPC;
}
static const struct vfs_file_ops full_file_ops = {
.read = memfile_read_zero,
.write = full_write,
};
static errno_t random_read(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
uint8_t chunk[MEMFILE_CHUNK];
size_t total = 0;
errno_t status = 0;
while (iovec_iter_len(buffer)) {
const size_t len = MIN(iovec_iter_len(buffer), sizeof(chunk));
// TODO: This function sucks and is probably not useful for cryptography.
random_fill(chunk, len);
size_t copied;
status = iovec_iter_copy_to(buffer, chunk, len, &copied);
total += copied;
if (status)
break;
}
*out_read = total;
return status;
}
static const struct vfs_file_ops random_file_ops = {
.read = random_read,
.write = memfile_discard,
};
static struct chardev null_dev;
static struct chardev zero_dev;
static struct chardev full_dev;
static struct chardev random_dev;
static struct chardev urandom_dev;
static void memfiles_add(const char* name, struct chardev* dev, const struct vfs_file_ops* ops, uint32_t minor) {
chardev_init(dev, nullptr, ops, 1, minor);
errno_t status = devtmpfs_add_chardev(name, dev, 0666);
assert(status == 0, "Unable to create /dev/%s (%i)", name, status);
}
static void memfiles_stage_init(struct boot_info* info) {
memfiles_add("null", &null_dev, &null_file_ops, 3);
memfiles_add("zero", &zero_dev, &zero_file_ops, 5);
memfiles_add("full", &full_dev, &full_file_ops, 7);
memfiles_add("random", &random_dev, &random_file_ops, 8);
memfiles_add("urandom", &urandom_dev, &random_file_ops, 9);
}
INIT_TASK(memfiles_stage, "dev.memfiles", memfiles_stage_init);
INIT_DEPENDS(memfiles_stage, devtmpfs_stage, random_stage);
dev/char/meson.build
-5
View File
@@ -1,5 +0,0 @@
kernel_sources += files(
'chardev.c',
'devctl.c',
'memfiles.c',
)
dev/dt/driver.c
-96
View File
@@ -1,96 +0,0 @@
#define pr_fmt(fmt) "fdt: " fmt
#include <zinnia/alloc.h>
#include <zinnia/fdt.h>
#include <zinnia/init.h>
#include <zinnia/print.h>
#include <zinnia/spin.h>
#include <zinnia/vec.h>
#include <string.h>
static struct fdt tree;
static bool tree_valid = false;
static struct spinlock driver_lock = {};
static vec(struct fdt_driver*) drivers = {};
struct fdt* fdt_get() {
return tree_valid ? &tree : nullptr;
}
static const struct fdt_match* fdt_match_table(
const struct fdt_match* table,
size_t num_matches,
struct fdt_node node
) {
for (size_t i = 0; i < num_matches; i++) {
if (fdt_node_is_compatible(node, table[i].compatible))
return &table[i];
}
return nullptr;
}
errno_t fdt_driver_register(struct fdt_driver* driver) {
if (driver->match == nullptr || driver->num_matches == 0) {
log_warn("Driver \"%s\" does not define any matches, ignoring\n", driver->name);
return 0;
}
if (!tree_valid)
return ENODEV;
spin_lock(&driver_lock);
vec_foreach(&drivers, i) {
if (strcmp(vec_elem_get(&drivers, i)->name, driver->name) == 0) {
spin_unlock(&driver_lock);
log_warn("Driver \"%s\" is already registered\n", driver->name);
return EEXIST;
}
}
errno_t status = vec_push(&drivers, driver);
spin_unlock(&driver_lock);
if (status != 0)
return status;
log("Registered platform driver \"%s\" with %zu match%s\n",
driver->name,
driver->num_matches,
driver->num_matches == 1 ? "" : "es");
for (size_t i = 0; i < driver->num_matches; i++)
log(" [%zu] = \"%s\"\n", i, driver->match[i].compatible);
for (struct fdt_node node = fdt_root(&tree); node.tree; node = fdt_next_node(node)) {
const struct fdt_match* match = fdt_match_table(driver->match, driver->num_matches, node);
if (match == nullptr)
continue;
log("Matched driver \"%s\" on rule [%zu]\n", driver->name, (size_t)(match - driver->match));
status = driver->probe(node, match);
if (status != 0)
log_warn("%s failed to probe: %i\n", driver->name, status);
}
return 0;
}
[[__init]]
static bool fdt_stage_init(struct boot_info* info) {
if (!info->dtb)
return false;
const void* blob = HHDM_PTR(info->dtb);
if (!fdt_init(&tree, blob, info->dtb_size)) {
log_warn("Unable to parse the device tree blob\n");
return false;
}
tree_valid = true;
struct fdt_prop model = fdt_property(fdt_root(&tree), "model");
if (model.tree && model.length)
log("Running on \"%.*s\"\n", (int)model.length, (const char*)model.data);
return true;
}
INIT_TASK(fdt_stage, "dev.dt", fdt_stage_init);
INIT_DEPENDS(fdt_stage, mem_stage);
dev/dt/fdt.c
-281
View File
@@ -1,281 +0,0 @@
#include <zinnia/fdt.h>
#include <string.h>
uint32_t fdt_read_u32(const void* data) {
const uint8_t* p = data;
return (uint32_t)p[0] << 24 | (uint32_t)p[1] << 16 | (uint32_t)p[2] << 8 | p[3];
}
uint64_t fdt_read_u64(const void* data) {
const uint8_t* p = data;
return (uint64_t)fdt_read_u32(p) << 32 | fdt_read_u32(p + 4);
}
static bool range_valid(size_t offset, size_t length, size_t total) {
return offset <= total && length <= total - offset;
}
struct token {
uint32_t kind;
size_t next;
const char* name;
const void* data;
size_t length;
};
static bool read_token(struct fdt* tree, size_t offset, struct token* token) {
if (!range_valid(offset, 4, tree->structs_len) || offset % 4)
return false;
*token = (struct token){.kind = fdt_read_u32(tree->structs + offset), .next = offset + 4};
if (token->kind == FDT_BEGIN_NODE) {
token->name = (const char*)tree->structs + token->next;
size_t left = tree->structs_len - token->next;
size_t length = strnlen(token->name, left);
if (length == left)
return false;
token->next += (length + 1 + 3) & ~(size_t)3;
} else if (token->kind == FDT_PROP) {
if (!range_valid(token->next, 8, tree->structs_len))
return false;
token->length = fdt_read_u32(tree->structs + token->next);
size_t name_offset = fdt_read_u32(tree->structs + token->next + 4);
token->next += 8;
if (name_offset >= tree->strings_len || !range_valid(token->next, token->length, tree->structs_len))
return false;
token->name = (const char*)tree->strings + name_offset;
size_t left = tree->strings_len - name_offset;
if (strnlen(token->name, left) == left || !*token->name)
return false;
token->data = tree->structs + token->next;
token->next += (token->length + 3) & ~(size_t)3;
} else if (token->kind != FDT_END_NODE && token->kind != FDT_NOP && token->kind != FDT_END) {
return false;
}
return token->next <= tree->structs_len;
}
bool fdt_init(struct fdt* tree, const void* data, size_t length) {
if (!tree)
return false;
*tree = (struct fdt){};
if (!data || length < sizeof(struct fdt_header))
return false;
const uint8_t* blob = data;
size_t total = fdt_read_u32(blob + 4);
size_t structs = fdt_read_u32(blob + 8);
size_t strings = fdt_read_u32(blob + 12);
size_t reserves = fdt_read_u32(blob + 16);
size_t strings_len = fdt_read_u32(blob + 32);
size_t structs_len = fdt_read_u32(blob + 36);
uint32_t version = fdt_read_u32(blob + 20);
uint32_t compat = fdt_read_u32(blob + 24);
if (fdt_read_u32(blob) != FDT_MAGIC || total > length || total < sizeof(struct fdt_header) || version < 17 ||
compat > 17 || compat > version || structs % 4 || reserves % 8 || structs < sizeof(struct fdt_header) ||
strings < sizeof(struct fdt_header) || reserves < sizeof(struct fdt_header) ||
!range_valid(structs, structs_len, total) || !range_valid(strings, strings_len, total) ||
(structs < strings + strings_len && strings < structs + structs_len))
return false;
size_t reserved_end = reserves;
for (;;) {
if (!range_valid(reserved_end, 16, total))
return false;
bool end = !fdt_read_u64(blob + reserved_end) && !fdt_read_u64(blob + reserved_end + 8);
reserved_end += 16;
if (end)
break;
}
if ((reserves < structs + structs_len && structs < reserved_end) ||
(reserves < strings + strings_len && strings < reserved_end))
return false;
struct fdt candidate = {
.blob = blob,
.total_size = total,
.version = version,
.compat_version = compat,
.boot_cpuid = fdt_read_u32(blob + 28),
.structs = blob + structs,
.structs_len = structs_len,
.strings = blob + strings,
.strings_len = strings_len,
};
size_t depth = 0;
bool started = false;
bool children[64] = {};
for (size_t offset = 0; offset < structs_len;) {
struct token token;
if (!read_token(&candidate, offset, &token))
return false;
if (token.kind == FDT_BEGIN_NODE) {
if (depth == 64 || (!depth && (started || *token.name)) || (depth && !*token.name))
return false;
if (depth)
children[depth - 1] = true;
children[depth++] = false;
started = true;
} else if (token.kind == FDT_END_NODE) {
if (!depth)
return false;
depth--;
} else if (token.kind == FDT_PROP) {
if (!depth || children[depth - 1])
return false;
} else if (token.kind == FDT_END) {
if (!started || depth || token.next != structs_len)
return false;
*tree = candidate;
return true;
}
offset = token.next;
}
return false;
}
struct fdt_node fdt_root(struct fdt* tree) {
struct token token;
size_t offset = 0;
if (tree && tree->blob) {
while (read_token(tree, offset, &token)) {
if (token.kind == FDT_BEGIN_NODE)
return (struct fdt_node){tree, offset};
if (token.kind != FDT_NOP)
break;
offset = token.next;
}
}
return (struct fdt_node){};
}
static struct fdt_node find_child(struct fdt_node node, const char* name, size_t length) {
struct token token;
if (!node.tree || !read_token(node.tree, node.offset, &token) || token.kind != FDT_BEGIN_NODE)
return (struct fdt_node){};
size_t depth = 0;
for (size_t offset = token.next; read_token(node.tree, offset, &token); offset = token.next) {
if (token.kind == FDT_BEGIN_NODE) {
if (!depth && strlen(token.name) == length && !memcmp(token.name, name, length))
return (struct fdt_node){node.tree, offset};
depth++;
} else if (token.kind == FDT_END_NODE) {
if (!depth)
break;
depth--;
}
}
return (struct fdt_node){};
}
struct fdt_node fdt_find(struct fdt* tree, const char* path) {
if (!path || *path != '/')
return (struct fdt_node){};
struct fdt_node node = fdt_root(tree);
path++;
while (*path && node.tree) {
size_t length = 0;
while (path[length] && path[length] != '/')
length++;
if (!length)
return (struct fdt_node){};
node = find_child(node, path, length);
path += length;
if (*path == '/')
path++;
}
return node;
}
struct fdt_prop fdt_property(struct fdt_node node, const char* name) {
struct token token;
if (!node.tree || !name || !read_token(node.tree, node.offset, &token) || token.kind != FDT_BEGIN_NODE)
return (struct fdt_prop){};
for (size_t offset = token.next; read_token(node.tree, offset, &token); offset = token.next) {
if (token.kind == FDT_PROP && !strcmp(token.name, name))
return (struct fdt_prop){node.tree, token.data, token.length};
if (token.kind == FDT_BEGIN_NODE || token.kind == FDT_END_NODE)
break;
}
return (struct fdt_prop){};
}
bool fdt_property_u32(struct fdt_node node, const char* name, uint32_t* value) {
struct fdt_prop property = fdt_property(node, name);
if (!property.tree || property.length != 4 || !value)
return false;
*value = fdt_read_u32(property.data);
return true;
}
bool fdt_property_u64(struct fdt_node node, const char* name, uint64_t* value) {
struct fdt_prop property = fdt_property(node, name);
if (!property.tree || property.length != 8 || !value)
return false;
*value = fdt_read_u64(property.data);
return true;
}
struct fdt_node fdt_next_node(struct fdt_node node) {
struct token token;
if (!node.tree || !read_token(node.tree, node.offset, &token))
return (struct fdt_node){};
for (size_t offset = token.next; read_token(node.tree, offset, &token); offset = token.next) {
if (token.kind == FDT_BEGIN_NODE)
return (struct fdt_node){node.tree, offset};
if (token.kind == FDT_END)
break;
}
return (struct fdt_node){};
}
bool fdt_node_is_compatible(struct fdt_node node, const char* compatible) {
struct fdt_prop property = fdt_property(node, "compatible");
const char* data = property.data;
for (size_t i = 0; i < property.length;) {
size_t length = strnlen(data + i, property.length - i);
if (length == property.length - i)
break;
if (!strcmp(data + i, compatible))
return true;
i += length + 1;
}
return false;
}
static uint64_t read_cells(const uint8_t* data, uint32_t cells) {
uint64_t value = 0;
for (uint32_t i = 0; i < cells; i++)
value = value << 32 | fdt_read_u32(data + i * 4);
return value;
}
bool fdt_node_reg(struct fdt_node node, size_t index, phys_t* addr, size_t* size) {
if (!node.tree)
return false;
struct fdt_node root = fdt_root(node.tree);
uint32_t addr_cells = 2;
uint32_t size_cells = 2;
fdt_property_u32(root, "#address-cells", &addr_cells);
fdt_property_u32(root, "#size-cells", &size_cells);
if (addr_cells > 2 || size_cells > 2)
return false;
struct fdt_prop reg = fdt_property(node, "reg");
const size_t stride = (addr_cells + size_cells) * 4;
if (!reg.tree || index >= reg.length / stride)
return false;
const uint8_t* entry = (const uint8_t*)reg.data + index * stride;
if (addr)
*addr = read_cells(entry, addr_cells);
if (size)
*size = read_cells(entry + addr_cells * 4, size_cells);
return true;
}
struct fdt_node fdt_find_compatible(struct fdt* tree, const char* compatible) {
if (!tree || !tree->blob || !compatible)
return (struct fdt_node){};
for (struct fdt_node node = fdt_root(tree); node.tree; node = fdt_next_node(node)) {
if (fdt_node_is_compatible(node, compatible))
return node;
}
return (struct fdt_node){};
}
dev/dt/meson.build
-4
View File
@@ -1,4 +0,0 @@
kernel_sources += files(
'driver.c',
'fdt.c',
)
dev/meson.build
-14
View File
@@ -1,14 +0,0 @@
subdir('block')
subdir('char')
subdir('dt')
subdir('video')
if kconfig.get('CONFIG_PCI', 'n') == 'y'
subdir('pci')
endif
if kconfig.get('CONFIG_ACPI', 'n') == 'y'
subdir('acpi')
endif
module_lds = meson.current_source_dir() / 'module.ld'
dev/module.ld
-114
View File
@@ -1,114 +0,0 @@
/* Linker script for kernel modules */
ENTRY(__module_entry)
PHDRS
{
text PT_LOAD FLAGS(5);
rodata PT_LOAD FLAGS(4);
data PT_LOAD FLAGS(6);
dynamic PT_DYNAMIC FLAGS(6);
modversion 0x60000001;
modauthor 0x60000002;
moddesc 0x60000003;
modlicense 0x60000004;
modcmdline 0x60000005;
modexit 0x60000006;
modmatch 0x60000007;
}
SECTIONS
{
. = ALIGN(CONSTANT(MAXPAGESIZE));
/* Code */
.text : {
*(.init .init.*)
*(.text .text.*)
*(.fini .fini.*)
*(.plt .plt.*)
} :text
. = ALIGN(CONSTANT(MAXPAGESIZE));
/* Read-only data */
.rodata : {
*(.rodata .rodata.*)
*(.hash .hash.*)
*(.got .got.*)
*(.got.plt .got.plt.*)
} :rodata
.mod.version : {
KEEP(*(.mod.version))
} :rodata :modversion
.mod.author : {
KEEP(*(.mod.author))
} :rodata :modauthor
.mod.desc : {
KEEP(*(.mod.desc))
} :rodata :moddesc
.mod.license : {
KEEP(*(.mod.license))
} :rodata :modlicense
.cmdline : {
KEEP(*(.cmdline))
} :rodata :modcmdline
.mod.exit : {
KEEP(*(.mod.exit))
} :rodata :modexit
.mod.match : {
KEEP(*(.mod.match))
} :rodata :modmatch
. = ALIGN(CONSTANT(MAXPAGESIZE));
/* Read-Write data */
.data : {
*(.data)
*(.data.*)
} :data
/* Zero-initialized memory */
.bss : {
*(.*bss .*bss.*)
*(COMMON)
} :data
.dynsym : {
*(.dynsym .dynsym.*)
} :data
.dynstr : {
*(.dynstr .dynstr.*)
} :data
.rela.dyn : {
*(.rela.dyn .rela.dyn.*)
} :data
.relr.dyn : {
*(.relr.dyn .relr.dyn.*)
} :data
/* Dynamic tags */
.dynamic : {
*(.dynamic)
} :data :dynamic
/DISCARD/ :
{
*(.comment*)
*(.eh_frame*)
*(.rel.eh_frame*)
*(.gcc_except_table*)
*(.note*)
*(.interp)
}
}
dev/pci/Kconfig
-7
View File
@@ -1,7 +0,0 @@
menu "PCI bus drivers"
config PCI
bool "PCI core drivers"
default y
endmenu
dev/pci/core.c
-377
View File
@@ -1,377 +0,0 @@
// SPDX-License-Identifier: ISC
#define pr_fmt(fmt) "pci: " fmt
#include <zinnia/alloc.h>
#include <zinnia/init.h>
#include <zinnia/module.h>
#include <zinnia/pci.h>
#include <zinnia/print.h>
#include <zinnia/spin.h>
#include <zinnia/vec.h>
#include <string.h>
static struct spinlock access_lock = {};
static vec(struct pci_access*) accesses = {};
static struct spinlock device_lock = {};
static vec(struct pci_device*) devices = {};
static vec(struct pci_driver*) drivers = {};
errno_t pci_access_register(struct pci_access* access) {
spin_lock(&access_lock);
errno_t status = vec_push(&accesses, access);
spin_unlock(&access_lock);
return status;
}
static bool pci_access_decodes(const struct pci_access* access, const struct pci_address* addr) {
return access->domain == addr->domain && addr->bus >= access->start_bus && addr->bus <= access->end_bus;
}
struct pci_access* pci_access_find(const struct pci_address* addr) {
struct pci_access* found = nullptr;
spin_lock(&access_lock);
vec_foreach(&accesses, i) {
struct pci_access* access = vec_elem_get(&accesses, i);
if (!pci_access_decodes(access, addr))
continue;
if (found == nullptr || access->priority > found->priority)
found = access;
}
spin_unlock(&access_lock);
return found;
}
static uint32_t cfg_read32(struct pci_access* access, const struct pci_address* addr, uint16_t offset) {
return access->read32(access, addr, offset & ~3);
}
static void cfg_write32(struct pci_access* access, const struct pci_address* addr, uint16_t offset, uint32_t value) {
access->write32(access, addr, offset & ~3, value);
}
uint32_t pci_read32(struct pci_device* dev, uint16_t offset) {
return cfg_read32(dev->access, &dev->address, offset);
}
uint16_t pci_read16(struct pci_device* dev, uint16_t offset) {
return pci_read32(dev, offset) >> ((offset & 2) * 8);
}
uint8_t pci_read8(struct pci_device* dev, uint16_t offset) {
return pci_read32(dev, offset) >> ((offset & 3) * 8);
}
void pci_write32(struct pci_device* dev, uint16_t offset, uint32_t value) {
cfg_write32(dev->access, &dev->address, offset, value);
}
void pci_write16(struct pci_device* dev, uint16_t offset, uint16_t value) {
uint32_t shift = (offset & 2) * 8;
uint32_t reg = pci_read32(dev, offset);
reg &= ~(0xffffu << shift);
reg |= (uint32_t)value << shift;
pci_write32(dev, offset, reg);
}
void pci_write8(struct pci_device* dev, uint16_t offset, uint8_t value) {
uint32_t shift = (offset & 3) * 8;
uint32_t reg = pci_read32(dev, offset);
reg &= ~(0xffu << shift);
reg |= (uint32_t)value << shift;
pci_write32(dev, offset, reg);
}
errno_t pci_bar_get(struct pci_device* dev, size_t index, struct pci_bar* out) {
if (index >= PCI_NUM_BARS)
return EINVAL;
uint16_t bar_offset = PCI_BAR0 + index * sizeof(uint32_t);
uint16_t high_offset = bar_offset + sizeof(uint32_t);
uint32_t bar = pci_read32(dev, bar_offset);
bool is_mmio = (bar & 0x1) == 0;
bool is_mmio64 = is_mmio && ((bar >> 1) & 0x3) == 0x2;
bool is_prefetchable = is_mmio && (bar & (1 << 3));
bool is_64bit_pair = is_mmio64 && index + 1 < PCI_NUM_BARS;
uint16_t command = pci_read16(dev, PCI_COMMAND);
pci_write16(dev, PCI_COMMAND, command & ~(PCI_COMMAND_IO | PCI_COMMAND_MEM));
uint32_t high = is_64bit_pair ? pci_read32(dev, high_offset) : 0;
pci_write32(dev, bar_offset, 0xffff'ffff);
if (is_64bit_pair)
pci_write32(dev, high_offset, 0xffff'ffff);
uint32_t mask_low = pci_read32(dev, bar_offset);
uint32_t mask_high = is_64bit_pair ? pci_read32(dev, high_offset) : 0;
pci_write32(dev, bar_offset, bar);
if (is_64bit_pair)
pci_write32(dev, high_offset, high);
pci_write16(dev, PCI_COMMAND, command);
uint64_t addr_mask = is_mmio ? ~0xfull : ~0x3ull;
uint64_t raw = (uint64_t)mask_high << 32 | mask_low;
uint64_t masked = raw & addr_mask & (is_64bit_pair ? UINT64_MAX : UINT32_MAX);
struct pci_bar result = {
.size = masked == 0 ? 0 : (size_t)1 << __builtin_ctzll(masked),
.prefetchable = is_prefetchable,
};
if (is_mmio && is_64bit_pair) {
result.type = PCI_BAR_MMIO64;
result.address = (phys_t)high << 32 | (bar & 0xffff'fff0);
} else if (is_mmio) {
result.type = PCI_BAR_MMIO32;
result.address = bar & 0xffff'fff0;
} else {
result.type = PCI_BAR_IO;
result.address = (uint16_t)(bar & 0xffff'fffc);
}
if (result.address == 0)
return ENOENT;
*out = result;
return 0;
}
uint8_t pci_cap_next(struct pci_device* dev, uint8_t cur) {
uint8_t ptr;
if (cur == 0) {
if (!(pci_read16(dev, PCI_STATUS) & PCI_STATUS_CAP_LIST))
return 0;
ptr = pci_read8(dev, PCI_CAP_PTR) & ~3;
} else {
ptr = (pci_read32(dev, cur) >> 8) & ~3;
}
if (ptr == 0)
return 0;
if ((pci_read32(dev, ptr) & 0xff) == 0xff)
return 0;
return ptr;
}
uint8_t pci_cap_find(struct pci_device* dev, uint8_t id) {
for (uint8_t cap = pci_cap_next(dev, 0); cap != 0; cap = pci_cap_next(dev, cap)) {
if (pci_read8(dev, cap) == id)
return cap;
}
return 0;
}
static void pci_device_name(const struct pci_device* dev, char buf[static 18]) {
ksnprintf(buf, 18, "pci-" PCI_ADDR_FMT, PCI_ADDR_ARGS(&dev->address));
}
static bool scan_function(struct pci_access* access, struct pci_address addr) {
uint32_t reg0 = cfg_read32(access, &addr, PCI_VENDOR_ID);
uint16_t vendor_id = reg0;
uint16_t device_id = reg0 >> 16;
if (vendor_id == 0xffff || vendor_id == 0)
return false;
if (device_id == 0xffff || device_id == 0)
return false;
struct pci_device* dev = mem_alloc(sizeof(*dev), 0);
if (dev == nullptr) {
log_err("Out of memory while scanning " PCI_ADDR_FMT "\n", PCI_ADDR_ARGS(&addr));
return false;
}
dev->address = addr;
dev->access = access;
dev->id.vendor_id = vendor_id;
dev->id.device_id = device_id;
dev->class.prog_if = pci_read8(dev, PCI_PROG_IF);
dev->class.subclass = pci_read8(dev, PCI_SUBCLASS);
dev->class.class = pci_read8(dev, PCI_CLASS);
dev->header_type = pci_read8(dev, PCI_HEADER_TYPE);
if ((dev->header_type & PCI_HEADER_TYPE_MASK) == 0) {
dev->id.subvendor_id = pci_read16(dev, PCI_SUBSYS_VENDOR_ID);
dev->id.subdevice_id = pci_read16(dev, PCI_SUBSYS_ID);
}
pci_log(
dev,
"<id = %.4hx:%.4hx, class = %.2hhx:%.2hhx:%.2hhx>\n",
dev->id.vendor_id,
dev->id.device_id,
dev->class.class,
dev->class.subclass,
dev->class.prog_if
);
if (vec_push(&devices, dev) != 0) {
mem_free(dev);
return false;
}
return true;
}
static void scan_slot(struct pci_access* access, struct pci_address addr) {
if (!scan_function(access, addr))
return;
if (!(vec_elem_get(&devices, vec_length(&devices) - 1)->header_type & PCI_HEADER_TYPE_MULTIFUNCTION))
return;
for (addr.function = 1; addr.function < PCI_NUM_FUNCTIONS; addr.function++)
scan_function(access, addr);
}
static void format_match_id(char buf[static 5], uint16_t value) {
if (value == PCI_ANY_ID)
ksnprintf(buf, 5, "****");
else
ksnprintf(buf, 5, "%.4hx", value);
}
static void format_match_class(char buf[static 3], uint8_t value) {
if (value == PCI_ANY_CLASS)
ksnprintf(buf, 3, "**");
else
ksnprintf(buf, 3, "%.2hhx", value);
}
static void format_match(const struct pci_match* match, char* buf, size_t size) {
char vendor[5], device[5], class[3], subclass[3], prog_if[3];
format_match_id(vendor, match->vendor_id);
format_match_id(device, match->device_id);
format_match_class(class, match->class);
format_match_class(subclass, match->subclass);
format_match_class(prog_if, match->prog_if);
ksnprintf(buf, size, "<id = %s:%s, class = %s:%s:%s>", vendor, device, class, subclass, prog_if);
}
static bool pci_match_device(const struct pci_match* m, const struct pci_device* dev) {
return (m->vendor_id == PCI_ANY_ID || m->vendor_id == dev->id.vendor_id) &&
(m->device_id == PCI_ANY_ID || m->device_id == dev->id.device_id) &&
(m->class == PCI_ANY_CLASS || m->class == dev->class.class) &&
(m->subclass == PCI_ANY_CLASS || m->subclass == dev->class.subclass) &&
(m->prog_if == PCI_ANY_CLASS || m->prog_if == dev->class.prog_if);
}
// Returns the first entry that matches, or nullptr.
static const struct pci_match* pci_match_table(
const struct pci_match* table,
size_t num_matches,
const struct pci_device* dev
) {
for (size_t i = 0; i < num_matches; i++) {
if (pci_match_device(&table[i], dev))
return &table[i];
}
return nullptr;
}
errno_t pci_driver_register(struct pci_driver* driver) {
if (driver->match == nullptr || driver->num_matches == 0) {
log_warn("Driver \"%s\" does not define any matches, ignoring\n", driver->name);
return 0;
}
spin_lock(&device_lock);
vec_foreach(&drivers, i) {
if (strcmp(vec_elem_get(&drivers, i)->name, driver->name) == 0) {
spin_unlock(&device_lock);
log_warn("Driver \"%s\" is already registered\n", driver->name);
return EEXIST;
}
}
errno_t status = vec_push(&drivers, driver);
spin_unlock(&device_lock);
if (status != 0)
return status;
log("Registered PCI driver \"%s\" with %zu match%s\n",
driver->name,
driver->num_matches,
driver->num_matches == 1 ? "" : "es");
for (size_t i = 0; i < driver->num_matches; i++) {
char rule[48];
format_match(&driver->match[i], rule, sizeof(rule));
log(" [%zu] = %s\n", i, rule);
}
for (size_t i = 0;; i++) {
spin_lock(&device_lock);
if (i >= vec_length(&devices)) {
spin_unlock(&device_lock);
break;
}
struct pci_device* dev = vec_elem_get(&devices, i);
const struct pci_match* match = nullptr;
if (dev->driver == nullptr) {
match = pci_match_table(driver->match, driver->num_matches, dev);
if (match != nullptr)
dev->driver = driver;
}
spin_unlock(&device_lock);
if (match == nullptr)
continue;
pci_log(dev, "Matched driver \"%s\" on rule [%zu]\n", driver->name, (size_t)(match - driver->match));
status = driver->probe(dev, match);
if (status != 0) {
pci_log_warn(dev, "%s failed to probe: %i\n", driver->name, status);
spin_lock(&device_lock);
dev->driver = nullptr;
spin_unlock(&device_lock);
}
}
return 0;
}
[[__init]]
static void pci_stage_init(struct boot_info*) {
if (vec_is_empty(&accesses)) {
log_warn("No configuration space access method available\n");
return;
}
log("Scanning PCI devices\n");
spin_lock(&device_lock);
vec_foreach(&accesses, i) {
struct pci_access* access = vec_elem_get(&accesses, i);
for (uint32_t bus = access->start_bus; bus <= access->end_bus; bus++) {
for (uint8_t slot = 0; slot < PCI_NUM_SLOTS; slot++) {
struct pci_address addr = {
.domain = access->domain,
.bus = bus,
.device = slot,
};
if (pci_access_find(&addr) != access)
continue;
scan_slot(access, addr);
}
}
}
spin_unlock(&device_lock);
log("Found %zu device(s)\n", vec_length(&devices));
}
INIT_TASK(pci_stage, "dev.pci", pci_stage_init);
INIT_DEPENDS(pci_stage, mem_stage);
dev/pci/ecam.c
-76
View File
@@ -1,76 +0,0 @@
#define pr_fmt(fmt) "pci: " fmt
#include <zinnia/alloc.h>
#include <zinnia/mmio.h>
#include <zinnia/pci.h>
#include <zinnia/pmap.h>
#include <zinnia/print.h>
#include <zinnia/utils.h>
#include <zinnia/virt.h>
#include <zinnia/vma.h>
#include <arch/mem.h>
struct pci_ecam {
struct pci_access access;
void* base;
};
static void* ecam_ptr(struct pci_access* self, const struct pci_address* addr, uint16_t offset) {
struct pci_ecam* ecam = containerof(self, struct pci_ecam, access);
size_t index = (size_t)(addr->bus - self->start_bus) << 20 | (size_t)(addr->device & 0x1f) << 15 |
(size_t)(addr->function & 0x7) << 12 | (offset & 0xffc);
return (uint8_t*)ecam->base + index;
}
static uint32_t ecam_read32(struct pci_access* self, const struct pci_address* addr, uint16_t offset) {
return mmio_read32(ecam_ptr(self, addr, offset));
}
static void ecam_write32(struct pci_access* self, const struct pci_address* addr, uint16_t offset, uint32_t value) {
mmio_write32(ecam_ptr(self, addr, offset), value);
}
errno_t pci_ecam_register(phys_t base, uint16_t domain, uint8_t start_bus, uint8_t end_bus) {
if (end_bus < start_bus)
return EINVAL;
size_t length = ((size_t)(end_bus - start_bus) + 1) << 20;
phys_t phys = base + ((phys_t)start_bus << 20);
uintptr_t vaddr = vm_alloc(length);
if (!vaddr)
return ENOMEM;
for (size_t i = 0; i < length; i += arch_mem_page_size()) {
errno_t status = pmap_map(&kernel_vma.pmap, vaddr + i, phys + i, PTE_READ | PTE_WRITE, CACHE_MMIO);
if (status != 0)
return status;
}
struct pci_ecam* ecam = mem_alloc(sizeof(*ecam), 0);
if (ecam == nullptr)
return ENOMEM;
*ecam = (struct pci_ecam){
.access =
{
.name = "ecam",
.domain = domain,
.start_bus = start_bus,
.end_bus = end_bus,
.priority = PCI_ACCESS_PRIO_ECAM,
.read32 = ecam_read32,
.write32 = ecam_write32,
},
.base = (void*)vaddr,
};
errno_t status = pci_access_register(&ecam->access);
if (status != 0) {
mem_free(ecam);
return status;
}
log("ECAM for %.4hx:%.2hhx - %.4hx:%.2hhx at %p\n", domain, start_bus, domain, end_bus, (void*)phys);
return 0;
}
dev/pci/meson.build
-4
View File
@@ -1,4 +0,0 @@
kernel_sources += files(
'core.c',
'ecam.c',
)
dev/video/Kconfig
-9
View File
@@ -1,9 +0,0 @@
menu "Video Drivers"
config FBCON
bool "Framebuffer console"
default y
help
Adds a basic console that can draw on a framebuffer passed by firmware.
endmenu
dev/video/fbcon/fbcon.c
-207
View File
@@ -1,207 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/console.h>
#include <zinnia/init.h>
#include <zinnia/pmap.h>
#include <zinnia/sched.h>
#include <zinnia/utils.h>
#include <zinnia/vma.h>
#include <stdint.h>
#include <string.h>
static const uint8_t fbcon_font[] = {
#embed "fbcon_font.bin"
};
constexpr size_t FONT_WIDTH = 8;
constexpr size_t FONT_HEIGHT = 12;
constexpr uint32_t FG_COLOR = 0xaaaaaa;
constexpr uint32_t BG_COLOR = 0x000000;
enum esc_state {
ESC_NONE,
ESC_START,
ESC_CSI,
};
static struct {
struct boot_framebuffer fb;
uint32_t* back;
uint32_t fg, bg;
size_t cols, rows;
size_t offset_x, offset_y;
size_t cx, cy;
enum esc_state esc;
} fbcon;
static inline uint8_t* fb_ptr() {
return HHDM_PTR(fbcon.fb.base);
}
static inline uint32_t* back_row(size_t y) {
return fbcon.back + y * fbcon.fb.width;
}
static void flush_rect(size_t x, size_t y, size_t w, size_t h) {
for (size_t j = 0; j < h; j++)
memcpy((uint32_t*)(fb_ptr() + (y + j) * fbcon.fb.pitch) + x, back_row(y + j) + x, w * sizeof(uint32_t));
}
static uint32_t pack_color(uint32_t rgb) {
const struct boot_framebuffer* fb = &fbcon.fb;
uint32_t r = (rgb >> 16) & 0xff;
uint32_t g = (rgb >> 8) & 0xff;
uint32_t b = rgb & 0xff;
return ((r >> (8 - fb->red.size)) << fb->red.shift) | ((g >> (8 - fb->green.size)) << fb->green.shift) |
((b >> (8 - fb->blue.size)) << fb->blue.shift);
}
static void fill_rect(size_t x, size_t y, size_t w, size_t h, uint32_t color) {
for (size_t j = 0; j < h; j++) {
uint32_t* row = back_row(y + j) + x;
for (size_t i = 0; i < w; i++)
row[i] = color;
}
flush_rect(x, y, w, h);
}
static void clear_screen() {
fill_rect(0, 0, fbcon.fb.width, fbcon.fb.height, fbcon.bg);
fbcon.cx = fbcon.cy = 0;
}
static void draw_glyph(size_t col, size_t row, uint8_t ch) {
const uint8_t* glyph = &fbcon_font[ch * FONT_HEIGHT];
size_t x = fbcon.offset_x + col * FONT_WIDTH;
size_t y = fbcon.offset_y + row * FONT_HEIGHT;
for (size_t j = 0; j < FONT_HEIGHT; j++) {
uint32_t* line = back_row(y + j) + x;
for (size_t i = 0; i < FONT_WIDTH; i++)
line[i] = (glyph[j] & (0x80 >> i)) ? fbcon.fg : fbcon.bg;
}
flush_rect(x, y, FONT_WIDTH, FONT_HEIGHT);
}
static void scroll_up() {
const size_t width = fbcon.fb.width;
const size_t text_y = fbcon.offset_y;
const size_t text_h = fbcon.rows * FONT_HEIGHT;
memmove(back_row(text_y), back_row(text_y + FONT_HEIGHT), (text_h - FONT_HEIGHT) * width * sizeof(uint32_t));
for (size_t j = text_y + text_h - FONT_HEIGHT; j < text_y + text_h; j++) {
uint32_t* row = back_row(j);
for (size_t i = 0; i < width; i++)
row[i] = fbcon.bg;
}
flush_rect(0, text_y, width, text_h);
}
static void newline() {
fbcon.cx = 0;
if (++fbcon.cy >= fbcon.rows) {
fbcon.cy = fbcon.rows - 1;
scroll_up();
}
}
static void put_char(uint8_t ch) {
switch (fbcon.esc) {
case ESC_START:
fbcon.esc = ch == '[' ? ESC_CSI : ESC_NONE;
if (ch == 'c')
clear_screen();
return;
case ESC_CSI:
if (ch >= 0x40 && ch <= 0x7e)
fbcon.esc = ESC_NONE;
return;
case ESC_NONE:
break;
}
switch (ch) {
case 0x1b:
fbcon.esc = ESC_START;
break;
case '\n':
newline();
break;
case '\r':
fbcon.cx = 0;
break;
case '\b':
if (fbcon.cx > 0)
fbcon.cx--;
break;
case '\t':
fbcon.cx = (fbcon.cx + 8) & ~(size_t)7;
if (fbcon.cx >= fbcon.cols)
newline();
break;
default:
if (fbcon.cx >= fbcon.cols)
newline();
draw_glyph(fbcon.cx++, fbcon.cy, ch);
break;
}
}
static void fbcon_write(struct console* con, const char* buf, size_t count) {
for (size_t i = 0; i < count; i++)
put_char((uint8_t)buf[i]);
}
static bool fbcon_setup(struct console* con) {
const struct boot_framebuffer* fb = &fbcon.fb;
if (fb->bytes_per_pixel != 4 || fb->red.size > 8 || fb->green.size > 8 || fb->blue.size > 8)
return false;
fbcon.fg = pack_color(FG_COLOR);
fbcon.bg = pack_color(BG_COLOR);
fbcon.cols = fb->width / FONT_WIDTH;
fbcon.rows = fb->height / FONT_HEIGHT;
fbcon.offset_x = (fb->width % FONT_WIDTH) / 2;
fbcon.offset_y = (fb->height % FONT_HEIGHT) / 2;
if (!fbcon.cols || !fbcon.rows)
return false;
clear_screen();
return true;
}
static struct console fbcon_con = {
.name = "fbcon",
.init = fbcon_setup,
.write = fbcon_write,
};
[[__init]]
void fbcon_init(struct boot_info* info) {
const struct boot_framebuffer* fb = &info->fb;
if (!fb->base)
return;
fbcon.fb = *fb;
const size_t pgsz = arch_mem_page_size();
const phys_t start = ALIGN_DOWN(fb->base, pgsz);
const phys_t end = ALIGN_UP(fb->base + fb->pitch * fb->height, pgsz);
for (phys_t p = start; p < end; p += pgsz) {
errno_t status =
pmap_map(&kernel_vma.pmap, (uintptr_t)HHDM_PTR(p), p, PTE_READ | PTE_WRITE, CACHE_WRITE_COMBINE);
if (status)
return;
}
fbcon.back = mem_alloc(fb->width * fb->height * sizeof(uint32_t), ALLOC_NOZERO);
if (!fbcon.back)
return;
console_register(&fbcon_con);
}
INIT_TASK(fbcon_stage, "core.fbcon", fbcon_init);
INIT_DEPENDS(fbcon_stage, sched_stage);
dev/video/fbcon/fbcon_font.bin
BIN
View File
Binary file not shown.
dev/video/fbcon/meson.build
-1
View File
@@ -1 +0,0 @@
kernel_sources += files('fbcon.c')
dev/video/meson.build
-3
View File
@@ -1,3 +0,0 @@
if kconfig.get('CONFIG_FBCON', 'n') == 'y'
subdir('fbcon')
endif
fs/anon.c
-40
View File
@@ -1,40 +0,0 @@
// SPDX-License-Identifier: ISC
// anonfs is a virtual file system. It backs files that don't live in any other
// file system, like pipes or unbound sockets.
#include <zinnia/atomic.h>
#include <zinnia/vfs.h>
static const struct vfs_super_block_ops anon_super_ops = {};
static struct vfs_super_block anon_super = {
.rc = {.count = 1}, // There's always just this exact super block.
.ops = &anon_super_ops,
};
static const struct vfs_node_ops anon_node_ops = {};
static _Atomic(ino_t) anon_inode_counter = 1;
errno_t vfs_create_anon_inode(
const struct vfs_file_ops* file_ops,
enum vfs_node_type type,
mode_t mode,
const struct ident* identity,
struct vfs_node** out
) {
struct vfs_node* node;
errno_t status = vfs_node_new(&anon_super, type, &anon_node_ops, mode, &node);
if (status)
return status;
node->file_ops = file_ops;
// Every anon node gets its own inode counter.
node->id = atomic_fetch_add(&anon_inode_counter, 1, ATOMIC_RELAXED);
node->uid = identity->euid;
node->gid = identity->egid;
*out = node;
return 0;
}
fs/cache.c
-411
View File
@@ -1,411 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/vfs.h>
#include <string.h>
#include "internal.h"
constexpr size_t MAX_SYMLINK_DEPTH = 40;
// If `path` is a mount point, crosses into the root of the topmost mounted file system.
// Returns a new reference.
static struct vfs_path cross_mounts(struct vfs_path path) {
struct vfs_mount* mount = path.mount;
struct vfs_entry* entry = path.entry;
again:
mutex_lock(&entry->lock);
vec_foreach(&entry->mounts, i) {
struct vfs_mount* child = vec_elem_get(&entry->mounts, i);
if (child != mount) {
mutex_unlock(&entry->lock);
mount = child;
entry = child->root;
goto again;
}
}
mutex_unlock(&entry->lock);
return vfs_path_clone((struct vfs_path){.mount = mount, .entry = entry});
}
errno_t vfs_path_lookup_parent(struct vfs_path path, struct vfs_path* out) {
struct vfs_mount* mount = path.mount;
struct vfs_entry* entry = path.entry;
// Get the top mount point.
while (mount->has_mount_point && entry == mount->root) {
entry = mount->mount_point.entry;
mount = mount->mount_point.mount;
}
if (!entry->parent)
return ENOENT;
*out = vfs_path_clone((struct vfs_path){.mount = mount, .entry = entry->parent});
return 0;
}
errno_t vfs_path_lookup_child(
struct vfs_path path,
struct vfs_path* root,
const char* name,
size_t name_len,
struct vfs_path* out
) {
errno_t status = 0;
// Traverse the mounts on the current entry.
struct vfs_path crossed = cross_mounts(path);
// A lookup only makes sense on directories.
struct vfs_node* parent = vfs_entry_get_inode(crossed.entry);
if (!parent) {
status = ENOENT;
goto fail;
}
if (parent->type != VNODE_DIRECTORY) {
status = ENOTDIR;
goto fail;
}
if (name_len == 1 && name[0] == '.') {
vfs_node_unref(parent);
*out = crossed;
return 0;
}
if (name_len == 2 && name[0] == '.' && name[1] == '.') {
vfs_node_unref(parent);
struct vfs_path crossed_root = cross_mounts(*root);
bool at_root = vfs_path_equal(crossed, crossed_root);
vfs_path_drop(crossed_root);
if (at_root) {
*out = crossed;
return 0;
}
status = vfs_path_lookup_parent(crossed, out);
if (status == ENOENT) {
*out = crossed;
return 0;
}
vfs_path_drop(crossed);
return status;
}
mutex_lock(&crossed.entry->lock);
// If this entry has already been looked up before, return that.
struct vfs_entry* child = nullptr;
for (struct vfs_entry* it = tailq_first(&crossed.entry->children); it; it = tailq_next(it, child_link)) {
if (it->name_len == name_len && !memcmp(it->name, name, name_len)) {
child = it;
vfs_entry_ref(child);
break;
}
}
if (!child) {
status = vfs_entry_new(name, name_len, crossed.entry, &child);
if (status) {
mutex_unlock(&crossed.entry->lock);
goto fail;
}
struct vfs_path child_path = {.mount = crossed.mount, .entry = child};
status = parent->ops->dir.lookup ? parent->ops->dir.lookup(parent, &child_path) : ENOENT;
if (status && status != ENOENT) {
mutex_unlock(&crossed.entry->lock);
vfs_entry_unref(child);
goto fail;
}
// Allow lookup failures so we can cache it as a negative entry.
if (status == ENOENT)
vfs_entry_set_negative(child);
status = 0;
// The children list takes over the initial reference.
tailq_insert_tail(&crossed.entry->children, child, child_link);
vfs_entry_ref(child);
}
mutex_unlock(&crossed.entry->lock);
*out = cross_mounts((struct vfs_path){.mount = crossed.mount, .entry = child});
vfs_entry_unref(child);
vfs_node_unref(parent);
vfs_path_drop(crossed);
return 0;
fail:
if (parent)
vfs_node_unref(parent);
vfs_path_drop(crossed);
return status;
}
static errno_t lookup_inner(
struct vfs_path root,
struct vfs_path start,
const char* path,
const struct ident* identity,
enum vfs_lookup_flags flags,
size_t* symlink_depth,
struct vfs_path* out
);
// Resolves `current` in place until it no longer points to a symbolic link.
static errno_t resolve_symlink(
struct vfs_path root,
struct vfs_path* current,
const struct ident* identity,
enum vfs_lookup_flags flags,
size_t* symlink_depth
) {
errno_t status = 0;
char* link_buf = nullptr;
while (true) {
struct vfs_node* node = vfs_entry_get_inode(current->entry);
if (!node)
break;
if (node->type != VNODE_SYMLINK) {
vfs_node_unref(node);
break;
}
if (*symlink_depth >= MAX_SYMLINK_DEPTH || !node->ops->symlink.read_link) {
vfs_node_unref(node);
status = ELOOP;
break;
}
(*symlink_depth)++;
if (!link_buf) {
link_buf = mem_alloc(__PATH_MAX + 1, ALLOC_NOZERO);
if (!link_buf) {
vfs_node_unref(node);
status = ENOMEM;
break;
}
}
iovec_t vec = {.base = (uintptr_t)link_buf, .len = __PATH_MAX};
iovec_iter_t iter = iovec_iter_kernel(&vec, 1);
size_t link_len = 0;
status = node->ops->symlink.read_link(node, &iter, &link_len);
vfs_node_unref(node);
if (status)
break;
link_buf[link_len] = '\0';
struct vfs_path parent;
status = vfs_path_lookup_parent(*current, &parent);
if (status)
break;
struct vfs_path result;
status = lookup_inner(
root,
parent,
link_buf,
identity,
flags & ~(VFS_LOOKUP_MUST_EXIST | VFS_LOOKUP_MUST_NOT_EXIST),
symlink_depth,
&result
);
vfs_path_drop(parent);
if (status)
break;
if (vfs_path_equal(result, *current)) {
vfs_path_drop(result);
status = ELOOP;
break;
}
vfs_path_drop(*current);
*current = result;
}
mem_free(link_buf);
return status;
}
static errno_t lookup_inner(
struct vfs_path root,
struct vfs_path start,
const char* path,
const struct ident* identity,
enum vfs_lookup_flags flags,
size_t* symlink_depth,
struct vfs_path* out
) {
errno_t status = 0;
if (path[0] == '\0')
return ENOENT;
// If a path starts with '/', it's an absolute path.
// In that case, skip the first character and use the current root as a starting point.
struct vfs_path current;
if (path[0] == '/') {
current = vfs_path_clone(root);
path++;
} else {
current = vfs_path_clone(start);
}
// Parse each component.
while (true) {
while (*path == '/')
path++;
if (*path == '\0')
break;
const char* component = path;
size_t component_len = 0;
while (path[component_len] != '/' && path[component_len] != '\0')
component_len++;
path += component_len;
if (component_len > __NAME_MAX) {
status = ENAMETOOLONG;
goto fail;
}
status = resolve_symlink(root, &current, identity, flags, symlink_depth);
if (status)
goto fail;
struct vfs_node* node = vfs_entry_get_inode(current.entry);
if (!node) {
status = ENOENT;
goto fail;
}
if (node->type != VNODE_DIRECTORY) {
vfs_node_unref(node);
status = ENOTDIR;
goto fail;
}
status = vfs_node_try_access(node, identity, X_OK, flags & VFS_LOOKUP_REAL_ID);
vfs_node_unref(node);
if (status)
goto fail;
struct vfs_path next;
status = vfs_path_lookup_child(current, &root, component, component_len, &next);
if (status)
goto fail;
vfs_path_drop(current);
current = next;
}
if (flags & VFS_LOOKUP_FOLLOW_SYMLINKS) {
status = resolve_symlink(root, &current, identity, flags, symlink_depth);
if (status)
goto fail;
}
struct vfs_node* node = vfs_entry_get_inode(current.entry);
if (node)
vfs_node_unref(node);
if ((flags & VFS_LOOKUP_MUST_EXIST) && !node) {
status = ENOENT;
goto fail;
}
if ((flags & VFS_LOOKUP_MUST_NOT_EXIST) && node) {
status = EEXIST;
goto fail;
}
*out = current;
return 0;
fail:
vfs_path_drop(current);
return status;
}
errno_t vfs_path_lookup(
struct vfs_path root,
struct vfs_path cwd,
const char* path,
const struct ident* identity,
enum vfs_lookup_flags flags,
struct vfs_path* out
) {
size_t symlink_depth = 0;
return lookup_inner(root, cwd, path, identity, flags, &symlink_depth, out);
}
errno_t vfs_path_absolute(struct vfs_path path, struct vfs_path* root, char* buf, size_t len, size_t* out_len) {
if (len == 0)
return ENAMETOOLONG;
struct vfs_path global_root;
if (!root) {
global_root = vfs_get_root();
root = &global_root;
}
errno_t status = 0;
struct vfs_mount* mount = path.mount;
struct vfs_entry* entry = path.entry;
// Build the path back to front, leaving room for the NUL terminator.
size_t cursor = len - 1;
buf[cursor] = '\0';
while (entry != root->entry || mount != root->mount) {
// At the root of a mounted file system we must step into the parent file system.
if (entry == mount->root) {
if (!mount->has_mount_point) {
// Global root with no parent.
status = ENOENT;
goto out;
}
entry = mount->mount_point.entry;
mount = mount->mount_point.mount;
continue;
}
if (entry->name_len + 1 > cursor) {
status = ENAMETOOLONG;
goto out;
}
cursor -= entry->name_len;
memcpy(buf + cursor, entry->name, entry->name_len);
buf[--cursor] = '/';
if (!entry->parent) {
status = ENOENT;
goto out;
}
entry = entry->parent;
}
// The root itself resolves to "/".
if (cursor == len - 1) {
if (cursor == 0) {
status = ENAMETOOLONG;
goto out;
}
buf[--cursor] = '/';
}
size_t result_len = len - 1 - cursor;
memmove(buf, buf + cursor, result_len + 1);
if (out_len)
*out_len = result_len;
out:
if (root == &global_root)
vfs_path_drop(global_root);
return status;
}
fs/devtmpfs.c
-64
View File
@@ -1,64 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/assert.h>
#include <zinnia/chardev.h>
#include <zinnia/ident.h>
#include <zinnia/vfs.h>
static struct vfs_mount* dev_mount;
struct vfs_path devtmpfs_root() {
return vfs_path_clone((struct vfs_path){.mount = dev_mount, .entry = dev_mount->root});
}
static errno_t devtmpfs_mount(
const struct vfs_filesystem* fs,
enum mount_flags flags,
void* arg,
struct vfs_mount** out
) {
return vfs_mount_new(flags, dev_mount->root, out);
}
static const struct vfs_filesystem devtmpfs = {
.name = "devtmpfs",
.mount = devtmpfs_mount,
};
static void devtmpfs_stage_init(struct boot_info* info) {
errno_t status = vfs_mount("tmpfs", 0, nullptr, &dev_mount);
assert(status == 0, "Unable to create the devtmpfs (%i)", status);
status = vfs_register_filesystem(&devtmpfs);
assert(status == 0, "Unable to register the devtmpfs (%i)", status);
}
INIT_TASK(devtmpfs_stage, "fs.devtmpfs", devtmpfs_stage_init);
INIT_DEPENDS(devtmpfs_stage, vfs_stage);
static void devtmpfs_mount_stage_init(struct boot_info* info) {
// Create /dev.
struct vfs_path root = vfs_get_root();
errno_t status = vfs_mkdir(root, root, "/dev", 0755, ident_kernel());
assert(status == 0 || status == EEXIST, "Unable to create /dev (%i)", status);
// Get /dev.
struct vfs_path target;
status = vfs_path_lookup(root, root, "/dev", ident_kernel(), VFS_LOOKUP_MUST_EXIST, &target);
vfs_path_drop(root);
assert(status == 0, "Unable to look up /dev (%i)", status);
// Prepare a devtmpfs for mounting.
struct vfs_mount* mount;
status = vfs_mount("devtmpfs", 0, nullptr, &mount);
assert(status == 0, "Unable to mount the devtmpfs (%i)", status);
// Mount it on /dev.
status = vfs_path_mount(target, mount);
assert(status == 0, "Unable to mount the devtmpfs on /dev (%i)", status);
vfs_mount_unref(mount);
vfs_path_drop(target);
}
INIT_TASK(devtmpfs_mount_stage, "fs.devtmpfs.mount", devtmpfs_mount_stage_init);
INIT_DEPENDS(devtmpfs_mount_stage, devtmpfs_stage);
fs/exec/elf.c
-370
View File
@@ -1,370 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/elf.h>
#include <zinnia/errno.h>
#include <zinnia/exec.h>
#include <zinnia/initgraph.h>
#include <zinnia/mman.h>
#include <zinnia/percpu.h>
#include <zinnia/pmap.h>
#include <zinnia/print.h>
#include <zinnia/sched.h>
#include <zinnia/utils.h>
#include <zinnia/vfs.h>
#include <zinnia/vma.h>
#include <zinnia/vmo.h>
#include <bits/elf.h>
#include <stdckdint.h>
#include <uapi/limits.h>
constexpr uintptr_t ELF_DYN_BASE = 0x10000;
struct elf_image {
uintptr_t base;
uintptr_t entry;
uintptr_t phdr;
size_t phnum;
size_t phent;
};
static bool elf_identify(struct vmo* obj) {
elf_ehdr_t ehdr = {};
size_t read;
if (vmo_read(obj, 0, &ehdr, sizeof(ehdr), &read) || read != sizeof(ehdr))
return false;
return memcmp(ehdr.e_ident, ELF_MAG, sizeof(ELF_MAG)) == 0;
}
static errno_t elf_read_interp(struct vmo* obj, const elf_phdr_t* phdr, char** out) {
if (phdr->p_filesz < 2 || phdr->p_filesz > __PATH_MAX)
return ENOEXEC;
char* path = mem_alloc(phdr->p_filesz, ALLOC_NOZERO);
if (!path)
return ENOMEM;
size_t read;
errno_t err = vmo_read(obj, phdr->p_offset, path, phdr->p_filesz, &read);
if (!err && (read != phdr->p_filesz || path[phdr->p_filesz - 1] != '\0'))
err = ENOEXEC;
if (err) {
mem_free(path);
return err;
}
*out = path;
return 0;
}
static errno_t elf_map(
struct vma* space,
struct vmo* obj,
uintptr_t dyn_base,
struct elf_image* out,
char** out_interp
) {
errno_t err;
size_t read;
elf_ehdr_t ehdr = {};
err = vmo_read(obj, 0, &ehdr, sizeof(ehdr), &read);
if (err)
return err;
if (read != sizeof(ehdr))
return ENOEXEC;
// Check if the file is an ELF.
if (memcmp(ehdr.e_ident, ELF_MAG, sizeof(ELF_MAG)) != 0)
return EINVAL;
if (ehdr.e_ident[EI_VERSION] != EV_CURRENT)
return EINVAL;
if (ehdr.e_ident[EI_CLASS] != ELF_ARCH_CLASS)
return EINVAL;
if (ehdr.e_ident[EI_DATA] != ELF_ARCH_DATA)
return EINVAL;
if (ehdr.e_machine != ELF_ARCH_MACHINE)
return EINVAL;
if (ehdr.e_type != ET_EXEC && ehdr.e_type != ET_DYN)
return EINVAL;
const uintptr_t base = ehdr.e_type == ET_DYN ? dyn_base : 0;
uintptr_t entry;
if (ckd_add(&entry, ehdr.e_entry, base) || !arch_mem_addr_is_user(entry))
return ENOEXEC;
const size_t phdrs_size = (size_t)ehdr.e_phnum * ehdr.e_phentsize;
uintptr_t phdrs_end;
if (ehdr.e_phentsize < sizeof(elf_phdr_t) || ehdr.e_phnum == 0 || phdrs_size > 64 * 1024)
return ENOEXEC;
if (ckd_add(&phdrs_end, ehdr.e_phoff, phdrs_size))
return ENOEXEC;
const size_t page_size = arch_mem_page_size();
uintptr_t phdr_addr = 0;
for (size_t i = 0; i < ehdr.e_phnum; i++) {
elf_phdr_t phdr = {};
err = vmo_read(obj, (ehdr.e_phoff + (i * ehdr.e_phentsize)), &phdr, sizeof(phdr), &read);
if (err)
return err;
if (read != sizeof(phdr))
return ENOEXEC;
if (phdr.p_type == PT_PHDR)
phdr_addr = phdr.p_vaddr + base;
if (phdr.p_type == PT_INTERP && out_interp) {
if (*out_interp)
return ENOEXEC;
err = elf_read_interp(obj, &phdr, out_interp);
if (err)
return err;
}
if (phdr.p_type == PT_LOAD) {
if (phdr.p_memsz == 0)
continue;
if (phdr.p_filesz > phdr.p_memsz)
return ENOEXEC;
uintptr_t vaddr, seg_end;
if (ckd_add(&vaddr, phdr.p_vaddr, base) || ckd_add(&seg_end, vaddr, phdr.p_memsz))
return ENOEXEC;
if (!arch_mem_addr_is_user(vaddr) || !arch_mem_addr_is_user(seg_end - 1))
return ENOEXEC;
if (phdr.p_align > 1 && phdr.p_offset % phdr.p_align != phdr.p_vaddr % phdr.p_align)
return ENOEXEC;
if (phdr_addr == 0 && ehdr.e_phoff >= phdr.p_offset && phdrs_end - phdr.p_offset <= phdr.p_filesz)
phdr_addr = vaddr + (ehdr.e_phoff - phdr.p_offset);
enum prot_flags prot = 0;
if (phdr.p_flags & PF_R)
prot |= PROT_READ;
if (phdr.p_flags & PF_W)
prot |= PROT_WRITE;
if (phdr.p_flags & PF_X)
prot |= PROT_EXEC;
const uintptr_t misalign = vaddr & (page_size - 1);
const uintptr_t map_address = vaddr - misalign;
const size_t backed_map_size = (phdr.p_filesz + misalign + page_size - 1) & ~(page_size - 1);
const size_t total_map_size = (phdr.p_memsz + misalign + page_size - 1) & ~(page_size - 1);
if (backed_map_size) {
err = vma_map(space, obj, map_address, backed_map_size, prot, 0, phdr.p_offset - misalign);
if (err)
return err;
}
if (total_map_size > backed_map_size) {
err = vma_map(
space,
nullptr,
map_address + backed_map_size,
total_map_size - backed_map_size,
prot,
0,
0
);
if (err)
return err;
}
const uintptr_t file_end = vaddr + phdr.p_filesz;
if (phdr.p_memsz > phdr.p_filesz && file_end % page_size != 0) {
const uintptr_t zero_end = MIN(ALIGN_UP(file_end, page_size), seg_end);
err = vma_zero(space, file_end, zero_end - file_end);
if (err)
return err;
}
}
}
*out = (struct elf_image){
.base = base,
.entry = entry,
.phdr = phdr_addr,
.phnum = ehdr.e_phnum,
.phent = ehdr.e_phentsize,
};
return 0;
}
static errno_t elf_load(struct process* proc, struct exec_info* info, uintptr_t* out_ip, uintptr_t* out_sp) {
struct elf_image exe = {};
char* interp_path = nullptr;
errno_t err = elf_map(info->space, info->file_obj, ELF_DYN_BASE, &exe, &interp_path);
uintptr_t entry = exe.entry;
uintptr_t interp_base = 0;
if (!err && interp_path) {
struct vmo* interp;
err = exec_open(proc, interp_path, &interp);
if (!err) {
struct elf_image ld;
if (!elf_identify(interp))
err = ELIBBAD;
else
err = elf_map(info->space, interp, (uintptr_t)1 << (mem_high_shift() - 2), &ld, nullptr);
if (!err) {
entry = ld.entry;
interp_base = ld.base;
}
vmo_unref(interp);
}
}
mem_free(interp_path);
if (err)
return err;
const size_t page_size = arch_mem_page_size();
const uintptr_t highest = ((uintptr_t)1 << (mem_high_shift() - 1)) - page_size;
const size_t stack_size = 2 * 1024 * 1024; // 2MiB
const uintptr_t stack_start = highest - stack_size;
err = vma_map(info->space, nullptr, stack_start, stack_size, PROT_READ | PROT_WRITE, 0, 0);
if (err)
return err;
// Fill in stack info.
uintptr_t stack_off = stack_size;
uintptr_t execfn = 0;
if (info->exec_path) {
const char nul = 0;
stack_off -= 1;
vma_write(info->space, stack_start + stack_off, &nul, sizeof(nul));
const size_t len = strlen(info->exec_path);
stack_off -= len;
vma_write(info->space, stack_start + stack_off, info->exec_path, len);
execfn = stack_start + stack_off;
}
// TODO: Instead of allocating, calculate it on the fly.
size_t num_envp = 0;
for (const char** ptr = info->envp; *ptr != nullptr; ptr++) {
num_envp++;
}
size_t num_argv = 0;
for (const char** ptr = info->argv; *ptr != nullptr; ptr++) {
num_argv++;
}
size_t strings_size = 0;
for (size_t env = 0; env < num_envp; env++)
strings_size += strlen(info->envp[env]) + 1;
for (size_t arg = 0; arg < num_argv; arg++)
strings_size += strlen(info->argv[arg]) + 1;
uintptr_t* envp_offsets = mem_alloc(num_envp * sizeof(uintptr_t), 0);
uintptr_t* argv_offsets = mem_alloc(num_argv * sizeof(uintptr_t), 0);
if ((num_envp && !envp_offsets) || (num_argv && !argv_offsets)) {
mem_free(envp_offsets);
mem_free(argv_offsets);
return ENOMEM;
}
for (size_t env = 0; env < num_envp; env++) {
const char nul = 0;
stack_off -= 1;
vma_write(info->space, stack_start + stack_off, &nul, sizeof(nul));
const size_t len = strlen(info->envp[env]);
stack_off -= len;
vma_write(info->space, stack_start + stack_off, info->envp[env], len);
envp_offsets[env] = stack_start + stack_off;
}
for (size_t arg = 0; arg < num_argv; arg++) {
const char nul = 0;
stack_off -= 1;
vma_write(info->space, stack_start + stack_off, &nul, sizeof(nul));
const size_t len = strlen(info->argv[arg]);
stack_off -= len;
vma_write(info->space, stack_start + stack_off, info->argv[arg], len);
argv_offsets[arg] = stack_start + stack_off;
}
stack_off = ALIGN_DOWN(stack_off, 16);
// Align the stack if argc + argv + envp does not add up to 16 byte alignment.
if ((1 + num_argv + num_envp) & 1) {
stack_off -= sizeof(uintptr_t);
uintptr_t zero = 0;
vma_write(info->space, stack_start + stack_off, &zero, sizeof(zero));
}
#define WRITE_AUXV(auxv, value) \
do { \
stack_off -= sizeof(uintptr_t); \
uintptr_t auxv_val = value; \
vma_write(info->space, stack_start + stack_off, &auxv_val, sizeof(uintptr_t)); \
stack_off -= sizeof(uintptr_t); \
auxv_val = auxv; \
vma_write(info->space, stack_start + stack_off, &auxv_val, sizeof(uintptr_t)); \
} while (0)
// Write auxiliary values.
WRITE_AUXV(AT_NULL, 0);
WRITE_AUXV(AT_SECURE, 0);
WRITE_AUXV(AT_PAGESZ, page_size);
WRITE_AUXV(AT_PHNUM, exe.phnum);
WRITE_AUXV(AT_PHENT, exe.phent);
WRITE_AUXV(AT_ENTRY, exe.entry);
if (exe.phdr)
WRITE_AUXV(AT_PHDR, exe.phdr);
if (interp_base)
WRITE_AUXV(AT_BASE, interp_base);
if (execfn)
WRITE_AUXV(AT_EXECFN, execfn);
// envp pointers.
stack_off -= sizeof(uintptr_t);
const uintptr_t zero = 0;
vma_write(info->space, stack_start + stack_off, &zero, sizeof(zero));
for (size_t env = num_envp; env > 0; env--) {
stack_off -= sizeof(uintptr_t);
vma_write(info->space, stack_start + stack_off, &envp_offsets[env - 1], sizeof(envp_offsets[env - 1]));
}
// argv pointers.
stack_off -= sizeof(uintptr_t);
vma_write(info->space, stack_start + stack_off, &zero, sizeof(zero));
for (size_t arg = num_argv; arg > 0; arg--) {
stack_off -= sizeof(uintptr_t);
vma_write(info->space, stack_start + stack_off, &argv_offsets[arg - 1], sizeof(argv_offsets[arg - 1]));
}
mem_free(envp_offsets);
mem_free(argv_offsets);
// argc
stack_off -= sizeof(uintptr_t);
uintptr_t argc = num_argv;
vma_write(info->space, stack_start + stack_off, &argc, sizeof(argc));
*out_ip = entry;
*out_sp = highest - stack_size + stack_off;
return 0;
}
static struct exec_format elf_format = {
.name = "elf",
.identify = elf_identify,
.load = elf_load,
};
static void elf_stage_init(struct boot_info*) {
exec_register(&elf_format);
}
INIT_TASK(elf_stage, "fs.exec.elf", elf_stage_init);
INIT_DEPENDS(elf_stage, mem_stage);
INIT_ENTAILS(elf_stage, vfs_stage);
fs/exec/exec.c
-53
View File
@@ -1,53 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/exec.h>
#include <zinnia/mutex.h>
#include <zinnia/print.h>
#include <zinnia/process.h>
#include <zinnia/tailq.h>
#include <zinnia/vfs.h>
static mutex_t formats_lock;
static tailq_head(struct exec_format) formats = {nullptr, &formats.tqh_first};
struct exec_format* exec_identify(struct vmo* obj) {
mutex_lock(&formats_lock);
struct exec_format* format = tailq_first(&formats);
while (format != nullptr && !format->identify(obj))
format = tailq_next(format, link);
mutex_unlock(&formats_lock);
return format;
}
void exec_register(struct exec_format* format) {
mutex_lock(&formats_lock);
tailq_insert_tail(&formats, format, link);
mutex_unlock(&formats_lock);
log("exec: Registered new exec format \"%s\"\n", format->name);
}
errno_t exec_open(struct process* proc, const char* path, struct vmo** out) {
mutex_lock(&proc->lock);
struct vfs_path root = vfs_path_clone(proc->root);
struct vfs_path cwd = vfs_path_clone(proc->cwd);
mutex_unlock(&proc->lock);
struct vfs_file* file;
errno_t status = vfs_file_open(root, cwd, path, O_RDONLY | O_EXEC, 0, &proc->identity, &file);
vfs_path_drop(root);
vfs_path_drop(cwd);
if (status)
return status;
status = exec_file_object(proc, file, out);
vfs_file_unref(file);
return status;
}
errno_t exec_file_object(struct process* proc, struct vfs_file* file, struct vmo** out) {
errno_t status = vfs_node_check_exec(file->vnode, &proc->identity);
if (!status)
status = vfs_file_get_object(file, out);
return status;
}
fs/file.c
-628
View File
@@ -1,628 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/atomic.h>
#include <zinnia/print.h>
#include <zinnia/vfs.h>
#include <stdckdint.h>
#include <string.h>
#include "internal.h"
errno_t vfs_file_new(struct vfs_node* node, const struct vfs_path* path, enum open_flags flags, struct vfs_file** out) {
assert(node != nullptr, "Every open file must be backed by an inode");
struct vfs_file* file = mem_alloc(sizeof(*file), 0);
if (!file)
return ENOMEM;
refcount_init(&file->rc);
file->ops = node->file_ops;
file->vnode = node;
vfs_node_ref(node);
atomic_init(&file->flags, flags);
mutex_init(&file->lock);
switch (node->type) {
case VNODE_REGULAR:
case VNODE_DIRECTORY:
file->position_kind = VFS_POSITION_ATOMIC;
break;
default:
file->position_kind = VFS_POSITION_POSITION;
break;
}
if (path) {
file->path = vfs_path_clone(*path);
file->has_path = true;
}
*out = file;
return 0;
}
void vfs_stream_open(struct vfs_file* file) {
file->position_kind = VFS_POSITION_STREAM;
}
// Resolves the absolute path of `path` as seen from `root` into a fitting allocation.
static char* make_absolute_path(struct vfs_path path, struct vfs_path* root) {
char* buf = mem_alloc(__PATH_MAX, ALLOC_NOZERO);
if (!buf)
return nullptr;
size_t len;
char* result = nullptr;
if (vfs_path_absolute(path, root, buf, __PATH_MAX, &len) == 0) {
result = mem_alloc(len + 1, ALLOC_NOZERO);
if (result)
memcpy(result, buf, len + 1);
}
mem_free(buf);
return result;
}
static errno_t open_special(
struct vfs_path* root,
struct vfs_path* file_path,
struct vfs_node* node,
enum open_flags flags,
struct vfs_file** out
) {
if ((node->type == VNODE_CHARDEV || node->type == VNODE_BLOCKDEV) && (!node->ops || !node->ops->device.open))
return ENXIO;
struct vfs_file* file;
errno_t status = vfs_file_new(node, file_path, flags, &file);
if (status)
return status;
file->absolute_path = make_absolute_path(*file_path, root);
status = node->type == VNODE_FIFO ? vfs_fifo_open(node, file) : node->ops->device.open(node, file);
if (status) {
file->ops = nullptr;
vfs_file_unref(file);
return status;
}
*out = file;
return 0;
}
static enum access_flags open_access(enum open_flags flags) {
if (flags & O_PATH)
return F_OK;
enum access_flags access = F_OK;
if (flags & O_RDONLY)
access |= R_OK;
if (flags & (O_WRONLY | O_TRUNC))
access |= W_OK;
return access;
}
static errno_t do_open_inode(
struct vfs_path* root,
struct vfs_path* file_path,
struct vfs_node* node,
enum open_flags flags,
bool created,
const struct ident* identity,
struct vfs_file** out
) {
// If we want to open as a directory, make sure this is actually a directory.
if ((flags & O_DIRECTORY) && node->type != VNODE_DIRECTORY)
return ENOTDIR;
if (node->type == VNODE_DIRECTORY && (flags & (O_WRONLY | O_CREAT | O_TRUNC)))
return EISDIR;
errno_t status = created ? 0 : vfs_node_try_access(node, identity, open_access(flags), false);
if (status)
return status;
switch (node->type) {
case VNODE_REGULAR: {
if (flags & O_TRUNC) {
if (!(flags & O_WRONLY))
return EINVAL;
if (!node->ops->regular.truncate)
return EPERM;
status = node->ops->regular.truncate(node, 0);
if (status)
return status;
}
struct vfs_file* file;
status = vfs_file_new(node, file_path, flags, &file);
if (status)
return status;
file->absolute_path = make_absolute_path(*file_path, root);
*out = file;
return 0;
}
case VNODE_DIRECTORY:
if (node->ops->dir.open)
return node->ops->dir.open(node, file_path, flags, identity, out);
return vfs_file_new(node, file_path, flags, out);
case VNODE_SYMLINK:
return ELOOP;
case VNODE_CHARDEV:
case VNODE_BLOCKDEV:
case VNODE_FIFO:
return open_special(root, file_path, node, flags, out);
case VNODE_SOCKET:
return ENXIO;
default:
return ENOTSUP;
}
}
errno_t vfs_file_open(
struct vfs_path root,
struct vfs_path cwd,
const char* path,
enum open_flags flags,
mode_t mode,
const struct ident* identity,
struct vfs_file** out
) {
if ((flags & O_DIRECTORY) && (flags & (O_CREAT | O_TMPFILE)))
return EINVAL;
const bool exclusive = (flags & (O_CREAT | O_EXCL)) == (O_CREAT | O_EXCL);
enum vfs_lookup_flags lookup_flags = 0;
// If we don't want to create a file when opening, the lookup must succeed.
// If we want to create a temporary unnamed file, `path` is the directory of the owning FS.
// In that case the path also has to exist.
if (!(flags & O_CREAT) || (flags & O_TMPFILE))
lookup_flags |= VFS_LOOKUP_MUST_EXIST;
if (exclusive)
lookup_flags |= VFS_LOOKUP_MUST_NOT_EXIST;
if (!(flags & O_NOFOLLOW) && !exclusive)
lookup_flags |= VFS_LOOKUP_FOLLOW_SYMLINKS;
struct vfs_path file_path;
errno_t status = vfs_path_lookup(root, cwd, path, identity, lookup_flags, &file_path);
if (status)
return status;
bool created = false;
struct vfs_node* node = vfs_entry_get_inode(file_path.entry);
if (!node) {
// If the lookup was successful, we expect that the entry is positive.
if (!(flags & O_CREAT)) {
log_warn("vfs: Tried opening a file without O_CREAT and backing inode, this is a bug!\n");
status = ENOENT;
goto out;
}
struct vfs_path parent_path;
status = vfs_path_lookup_parent(file_path, &parent_path);
if (status)
goto out;
struct vfs_node* parent = vfs_entry_get_inode(parent_path.entry);
vfs_path_drop(parent_path);
if (!parent) {
status = ENOENT;
goto out;
}
status = parent->type == VNODE_DIRECTORY ? 0 : ENOTDIR;
if (!status)
status = vfs_node_try_access(parent, identity, W_OK | X_OK, false);
if (!status)
status = parent->ops->dir.create ? parent->ops->dir.create(parent, file_path.entry, mode, identity) : EPERM;
vfs_node_unref(parent);
if (status)
goto out;
node = vfs_entry_get_inode(file_path.entry);
if (!node) {
status = EIO;
goto out;
}
created = true;
}
status = do_open_inode(&root, &file_path, node, flags, created, identity, out);
vfs_node_unref(node);
out:
vfs_path_drop(file_path);
return status;
}
static errno_t advance_offset(off_t* offset, ssize_t amount) {
if (amount > 0 && ckd_add(offset, *offset, amount))
return EOVERFLOW;
return 0;
}
errno_t vfs_file_read(struct vfs_file* file, iovec_iter_t* buffer, ssize_t* out_read) {
*out_read = 0;
if (iovec_iter_len(buffer) == 0)
return 0;
if (!file->ops || !file->ops->read)
return EINVAL;
errno_t status;
ssize_t read = 0;
switch (file->position_kind) {
case VFS_POSITION_STREAM:
status = file->ops->read(file, buffer, 0, &read);
break;
case VFS_POSITION_ATOMIC:
mutex_lock(&file->lock);
status = file->ops->read(file, buffer, file->position, &read);
if (!status)
status = advance_offset(&file->position, read);
mutex_unlock(&file->lock);
break;
case VFS_POSITION_POSITION: {
mutex_lock(&file->lock);
off_t pos = file->position;
mutex_unlock(&file->lock);
status = file->ops->read(file, buffer, pos, &read);
if (!status)
status = advance_offset(&pos, read);
if (!status) {
mutex_lock(&file->lock);
file->position = pos;
mutex_unlock(&file->lock);
}
break;
}
default:
return EINVAL;
}
if (status)
return status;
*out_read = read;
return 0;
}
errno_t vfs_file_pread(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
*out_read = 0;
if (iovec_iter_len(buffer) == 0)
return 0;
if (file->position_kind == VFS_POSITION_STREAM)
return ESPIPE;
if (!file->ops || !file->ops->read)
return EINVAL;
return file->ops->read(file, buffer, offset, out_read);
}
static errno_t check_write_progress(ssize_t written) {
if (written == 0) {
log_warn("vfs: write reported zero progress for a non-empty buffer\n");
return EIO;
}
return 0;
}
static errno_t sync_after_write(struct vfs_file* file, ssize_t written) {
if (written <= 0)
return 0;
if ((atomic_load(&file->flags, ATOMIC_RELAXED) & O_SYNC) == O_SYNC)
return vfs_file_sync(file, false);
if ((atomic_load(&file->flags, ATOMIC_RELAXED) & O_DSYNC) == O_DSYNC)
return vfs_file_sync(file, true);
return 0;
}
errno_t vfs_file_write(struct vfs_file* file, iovec_iter_t* buffer, ssize_t* out_written) {
*out_written = 0;
if (iovec_iter_len(buffer) == 0)
return 0;
if (!file->ops || !file->ops->write)
return EINVAL;
// TODO: Appending writes through different files of the same inode aren't serialized.
const bool append = atomic_load(&file->flags, ATOMIC_RELAXED) & O_APPEND;
errno_t status;
ssize_t written = 0;
switch (file->position_kind) {
case VFS_POSITION_STREAM:
status = file->ops->write(file, buffer, 0, &written);
break;
case VFS_POSITION_ATOMIC:
mutex_lock(&file->lock);
if (append)
file->position = file->vnode->size;
status = file->ops->write(file, buffer, file->position, &written);
if (!status)
status = advance_offset(&file->position, written);
mutex_unlock(&file->lock);
break;
case VFS_POSITION_POSITION: {
mutex_lock(&file->lock);
off_t pos = append ? (off_t)file->vnode->size : file->position;
mutex_unlock(&file->lock);
status = file->ops->write(file, buffer, pos, &written);
if (!status)
status = advance_offset(&pos, written);
if (!status) {
mutex_lock(&file->lock);
file->position = pos;
mutex_unlock(&file->lock);
}
break;
}
default:
return EINVAL;
}
if (!status)
status = check_write_progress(written);
if (!status)
status = sync_after_write(file, written);
if (status)
return status;
*out_written = written;
return 0;
}
errno_t vfs_file_pwrite(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_written) {
*out_written = 0;
if (iovec_iter_len(buffer) == 0)
return 0;
if (file->position_kind == VFS_POSITION_STREAM)
return ESPIPE;
if (!file->ops || !file->ops->write)
return EINVAL;
ssize_t written = 0;
errno_t status = file->ops->write(file, buffer, offset, &written);
if (!status)
status = check_write_progress(written);
if (!status)
status = sync_after_write(file, written);
if (status)
return status;
*out_written = written;
return 0;
}
errno_t vfs_file_ioctl(struct vfs_file* file, uintptr_t request, uintptr_t arg, uintptr_t* out_result) {
if (!file->ops || !file->ops->ioctl)
return ENOTTY;
return file->ops->ioctl(file, request, arg, out_result);
}
errno_t vfs_file_sync(struct vfs_file* file, bool data_only) {
if (!file->ops || !file->ops->sync)
return 0;
return file->ops->sync(file, data_only);
}
errno_t vfs_file_seek(struct vfs_file* file, enum vfs_seek_anchor anchor, off_t offset, off_t* out_position) {
if (file->position_kind == VFS_POSITION_STREAM)
return ESPIPE;
errno_t status = 0;
mutex_lock(&file->lock);
off_t base;
switch (anchor) {
case VFS_SEEK_START:
base = 0;
break;
case VFS_SEEK_CURRENT:
base = file->position;
break;
case VFS_SEEK_END:
base = file->vnode->size;
break;
default:
status = EINVAL;
goto out;
}
off_t new_position;
if (ckd_add(&new_position, base, offset)) {
status = EOVERFLOW;
goto out;
}
if (new_position < 0) {
status = EINVAL;
goto out;
}
file->position = new_position;
if (out_position)
*out_position = new_position;
out:
mutex_unlock(&file->lock);
return status;
}
off_t vfs_file_tell(struct vfs_file* file) {
mutex_lock(&file->lock);
off_t position = file->position;
mutex_unlock(&file->lock);
return position;
}
errno_t vfs_file_mmap(
struct vfs_file* file,
struct vma* vma,
uintptr_t addr,
size_t len,
enum prot_flags prot,
enum map_flags flags,
off_t offset,
uintptr_t* out_addr
) {
if (!file->ops || !file->ops->mmap)
return ENODEV;
return file->ops->mmap(file, vma, addr, len, prot, flags, offset, out_addr);
}
enum poll_flags vfs_poll(struct vfs_file* file, enum poll_flags events, struct poll_table* table) {
const enum poll_flags always = POLLERR | POLLHUP | POLLNVAL;
if (!file->ops || !file->ops->poll)
return events & (POLLIN | POLLOUT | POLLRDNORM | POLLWRNORM);
return file->ops->poll(file, events | POLLERR | POLLHUP, table) & (events | always);
}
errno_t vfs_file_get_object(struct vfs_file* file, struct vmo** out) {
struct vfs_node* node = file->vnode;
if (node->type != VNODE_REGULAR || !node->ops || !node->ops->regular.get_vmo)
return ENODEV;
return node->ops->regular.get_vmo(node, out);
}
static uint8_t dirent_type(enum vfs_node_type type) {
switch (type) {
case VNODE_REGULAR:
return __DT_REG;
case VNODE_DIRECTORY:
return __DT_DIR;
case VNODE_SYMLINK:
return __DT_LNK;
case VNODE_FIFO:
return __DT_FIFO;
case VNODE_BLOCKDEV:
return __DT_BLK;
case VNODE_CHARDEV:
return __DT_CHR;
case VNODE_SOCKET:
return __DT_SOCK;
default:
// Should be unreachable, but just as a failsafe in case someone messes up.
return __DT_UNKNOWN;
}
}
errno_t vfs_dir_get_cached_entries(
struct vfs_node* node,
struct vfs_entry* entry,
off_t offset,
struct __dirent* buffer,
size_t buffer_len,
const struct ident* identity,
size_t* out_count
) {
size_t read = 0;
off_t current = 0;
mutex_lock(&entry->lock);
for (struct vfs_entry* child = tailq_first(&entry->children); child && read < buffer_len;
child = tailq_next(child, child_link)) {
struct vfs_node* child_node = vfs_entry_get_inode(child);
if (!child_node)
continue;
if (current < offset) {
current++;
vfs_node_unref(child_node);
continue;
}
struct __dirent* d = &buffer[read];
d->d_ino = child_node->id;
d->d_off = current;
d->d_reclen = sizeof(*d);
d->d_type = dirent_type(child_node->type);
memcpy(d->d_name, child->name, child->name_len);
d->d_name[child->name_len] = '\0';
vfs_node_unref(child_node);
read++;
current++;
}
mutex_unlock(&entry->lock);
*out_count = read;
return 0;
}
static void fill_dot_entry(struct __dirent* d, ino_t ino, off_t off, const char* name) {
d->d_ino = ino;
d->d_off = off;
d->d_reclen = sizeof(*d);
d->d_type = __DT_DIR;
memset(d->d_name, 0, sizeof(d->d_name));
memcpy(d->d_name, name, strlen(name));
}
errno_t vfs_get_dir_entries(
struct vfs_file* file,
struct __dirent* buffer,
size_t buffer_len,
const struct ident* identity,
size_t* out_count
) {
*out_count = 0;
if (buffer_len == 0)
return 0;
struct vfs_node* node = file->vnode;
if (!file->has_path || node->type != VNODE_DIRECTORY)
return ENOTDIR;
if (file->position_kind == VFS_POSITION_STREAM)
return EBADF;
ino_t parent_ino = node->id;
struct vfs_path parent;
if (vfs_path_lookup_parent(file->path, &parent) == 0) {
struct vfs_node* parent_node = vfs_entry_get_inode(parent.entry);
if (parent_node) {
parent_ino = parent_node->id;
vfs_node_unref(parent_node);
}
vfs_path_drop(parent);
}
errno_t status = 0;
mutex_lock(&file->lock);
const off_t start_offset = file->position;
size_t num_read = 0;
// Read `.` and `..` entries.
if (start_offset == 0)
fill_dot_entry(&buffer[num_read++], node->id, 0, ".");
if ((start_offset == 1 || num_read == 1) && num_read < buffer_len)
fill_dot_entry(&buffer[num_read++], parent_ino, 1, "..");
// Read normal dir entries.
if (num_read < buffer_len && node->ops->dir.get_dir_entries) {
size_t count = 0;
status = node->ops->dir.get_dir_entries(
node,
file->path.entry,
start_offset > 2 ? start_offset - 2 : 0,
buffer + num_read,
buffer_len - num_read,
identity,
&count
);
if (status)
goto out;
num_read += count;
}
file->position += num_read;
for (size_t i = 0; i < num_read; i++)
buffer[i].d_off = start_offset + i + 1;
*out_count = num_read;
out:
mutex_unlock(&file->lock);
return status;
}
fs/ident.c
-21
View File
@@ -1,21 +0,0 @@
#include <zinnia/ident.h>
static const struct ident kernel_identity = {0};
const struct ident* ident_kernel() {
return &kernel_identity;
}
bool ident_in_group(const struct ident* identity, gid_t gid, bool use_real) {
if (gid == (use_real ? identity->gid : identity->egid))
return true;
vec_foreach(&identity->groups, i) {
if (vec_elem_get(&identity->groups, i) == gid)
return true;
}
return false;
}
bool ident_is_privileged(const struct ident* identity, bool use_real) {
return (use_real ? identity->uid : identity->euid) == 0;
}
fs/init.c
-21
View File
@@ -1,21 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/initgraph.h>
#include <zinnia/vfs.h>
#include "internal.h"
static void vfs_stage_init(struct boot_info* info) {
// Mount a tmpfs as root.
struct vfs_mount* tmpfs;
errno_t status = vfs_mount("tmpfs", 0, nullptr, &tmpfs);
assert(status == 0, "Unable to mount the root tmpfs (%i)", status);
struct vfs_path root = {.mount = tmpfs, .entry = tmpfs->root};
vfs_set_root(root);
vfs_mount_unref(tmpfs);
}
INIT_TASK(vfs_stage, "fs.vfs", vfs_stage_init);
INIT_DEPENDS(vfs_stage, mem_stage);
fs/initramfs.c
-309
View File
@@ -1,309 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/blockdev.h>
#include <zinnia/errno.h>
#include <zinnia/init.h>
#include <zinnia/initramfs.h>
#include <zinnia/print.h>
#include <zinnia/vfs.h>
#include <uapi/limits.h>
#include <uapi/stat.h>
enum ustar_type : uint8_t {
USTAR_REGULAR = 0,
USTAR_NORMAL = '0',
USTAR_SYM_LINK = '2',
USTAR_DIRECTORY = '5',
USTAR_CONTIGUOUS = '7',
USTAR_LONG_LINK = 'L',
};
struct ustar_header {
char name[100];
char mode[8];
char uid[8];
char gid[8];
char size[12];
char mtime[12];
char checksum[8];
enum ustar_type type;
char linkname[100];
char signature[6];
char version[2];
char owner[32];
char group[32];
char devmajor[8];
char devminor[8];
char prefix[155];
char pad[12];
};
static_assert(sizeof(struct ustar_header) == 512, "USTAR header must be 512 bytes");
static size_t oct2bin(const char* str, size_t len) {
size_t value = 0;
for (size_t i = 0; i < len; i++) {
if (str[i] == '\0' || str[i] == ' ')
break;
value *= 8;
value += (size_t)(str[i] - '0');
}
return value;
}
static void copy_field(char* out, const char* field, size_t field_len, size_t out_len) {
size_t i = 0;
while (i < field_len && field[i] != '\0' && i + 1 < out_len) {
out[i] = field[i];
i++;
}
out[i] = '\0';
}
static errno_t create_dirs(
struct vfs_path root,
struct vfs_path at,
const char* path,
mode_t mode,
struct vfs_path* out_dir,
char* name_out,
size_t name_out_len
) {
size_t path_len = strlen(path);
size_t dir_len = 0;
const char* name = path;
for (size_t i = path_len; i > 0; i--) {
if (path[i - 1] == '/') {
dir_len = i - 1;
name = path + i;
break;
}
}
strncpy(name_out, name, name_out_len - 1);
name_out[name_out_len - 1] = '\0';
struct vfs_path current = vfs_path_clone(at);
size_t index = 0;
while (index < dir_len) {
while (index < dir_len && path[index] == '/')
index++;
if (index >= dir_len)
break;
size_t start = index;
while (index < dir_len && path[index] != '/')
index++;
char component[__NAME_MAX + 1];
size_t length = MIN(index - start, __NAME_MAX);
memcpy(component, path + start, length);
component[length] = '\0';
errno_t status = vfs_mkdir(root, current, component, mode, ident_kernel());
if (status && status != EEXIST) {
vfs_path_drop(current);
return status;
}
struct vfs_file* dir;
status = vfs_file_open(root, current, component, O_DIRECTORY, 0, ident_kernel(), &dir);
if (status) {
vfs_path_drop(current);
return status;
}
struct vfs_path next = vfs_path_clone(dir->path);
vfs_file_unref(dir);
vfs_path_drop(current);
current = next;
}
*out_dir = current;
return 0;
}
bool initramfs_is_valid(const void* data, size_t len) {
const struct ustar_header* header = data;
return len >= sizeof(*header) && !memcmp(header->signature, "ustar", 5);
}
// Sets all timestamps of the node behind `file` to `mtime`.
static void set_times(struct vfs_file* file, size_t mtime) {
struct vfs_node* node = file->vnode;
timespec_t time = time_from_sec(mtime);
mutex_lock(&node->lock);
node->atime = time;
node->mtime = time;
node->ctime = time;
mutex_unlock(&node->lock);
}
errno_t initramfs_load(struct vfs_path root, struct vfs_path target, const void* data, size_t len) {
const uint8_t* base = data;
size_t offset = 0;
size_t files_loaded = 0;
errno_t status = 0;
// Paths can be much longer than what fits on the stack.
char* override = mem_alloc(__PATH_MAX + 1, ALLOC_NOZERO);
char* name = mem_alloc(__PATH_MAX + 1, ALLOC_NOZERO);
if (!override || !name) {
status = ENOMEM;
goto out;
}
bool has_override = false;
while (offset + sizeof(struct ustar_header) <= len) {
const struct ustar_header* header = (const struct ustar_header*)(base + offset);
if (memcmp(header->signature, "ustar", 5) != 0)
break;
mode_t file_mode = (mode_t)oct2bin(header->mode, sizeof(header->mode));
size_t file_size = oct2bin(header->size, sizeof(header->size));
size_t mtime = oct2bin(header->mtime, sizeof(header->mtime));
size_t data_offset = offset + sizeof(struct ustar_header);
if (file_size > len - data_offset) {
log_warn("vfs: Truncated initramfs entry at offset %zu\n", offset);
status = EINVAL;
goto out;
}
if (has_override) {
memcpy(name, override, __PATH_MAX + 1);
has_override = false;
} else {
copy_field(name, header->name, sizeof(header->name), __PATH_MAX + 1);
}
// Directory entries usually end with a slash, which would leave us with an empty file name.
size_t name_len = strlen(name);
while (name_len > 1 && name[name_len - 1] == '/')
name[--name_len] = '\0';
struct vfs_path dir;
char file_name[__NAME_MAX + 1];
switch (header->type) {
case USTAR_REGULAR:
case USTAR_NORMAL:
case USTAR_CONTIGUOUS: {
status = create_dirs(root, target, name, file_mode, &dir, file_name, sizeof(file_name));
if (status)
goto out;
struct vfs_file* file;
status = vfs_file_open(root, dir, file_name, O_CREAT, file_mode, ident_kernel(), &file);
vfs_path_drop(dir);
if (status)
goto out;
if (file_size > 0) {
iovec_t vec = {
.base = (uintptr_t)(base + data_offset),
.len = file_size,
};
iovec_iter_t iter = iovec_iter_kernel(&vec, 1);
ssize_t written = 0;
status = vfs_file_pwrite(file, &iter, 0, &written);
if (status) {
vfs_file_unref(file);
goto out;
}
}
set_times(file, mtime);
vfs_file_unref(file);
files_loaded++;
break;
}
case USTAR_SYM_LINK: {
status = create_dirs(root, target, name, file_mode, &dir, file_name, sizeof(file_name));
if (status)
goto out;
char link_target[101];
copy_field(link_target, header->linkname, sizeof(header->linkname), sizeof(link_target));
status = vfs_symlink(root, dir, file_name, link_target, ident_kernel());
vfs_path_drop(dir);
if (status)
goto out;
files_loaded++;
break;
}
case USTAR_DIRECTORY: {
status = create_dirs(root, target, name, file_mode, &dir, file_name, sizeof(file_name));
if (status)
goto out;
status = vfs_mkdir(root, dir, file_name, file_mode, ident_kernel());
vfs_path_drop(dir);
if (status && status != EEXIST)
goto out;
status = 0;
files_loaded++;
break;
}
case USTAR_LONG_LINK: {
size_t copy_len = MIN(file_size, __PATH_MAX);
memcpy(override, base + data_offset, copy_len);
override[copy_len] = '\0';
has_override = true;
break;
}
default:
break;
}
offset += sizeof(struct ustar_header) + ALIGN_UP(file_size, 512);
}
log("vfs: Loaded %zu files from initramfs\n", files_loaded);
out:
mem_free(override);
mem_free(name);
return status;
}
void initramfs_load_boot_modules(struct boot_info* info) {
struct vfs_path root = vfs_get_root();
size_t ramdisk_index = 0;
for (size_t i = 0; i < info->num_files; i++) {
struct boot_file* file = &info->files[i];
const void* data = HHDM_PTR(file->data);
if (!initramfs_is_valid(data, file->length)) {
char name[16];
ksnprintf(name, sizeof(name), "ram%zu", ramdisk_index);
struct blockdev* disk;
errno_t status = ramdisk_new(file->data, file->length, &disk);
if (!status) {
status = blockdev_register(name, disk);
blockdev_unref(disk);
}
if (status) {
log_warn("initramfs: Unable to expose module \"%s\" as a RAM disk (%i)\n", file->path, status);
continue;
}
log("initramfs: Exposed module \"%s\" as RAM disk /dev/block/%s\n", file->path, name);
ramdisk_index++;
continue;
}
errno_t status = initramfs_load(root, root, data, file->length);
assert(status == 0, "Failed to load initramfs \"%s\" (%i)", file->path, status);
}
vfs_path_drop(root);
}
fs/internal.h
-16
View File
@@ -1,16 +0,0 @@
// SPDX-License-Identifier: ISC
#pragma once
#include <zinnia/vfs.h>
// Sets the global root of the VFS. Takes a new reference to `root`.
void vfs_set_root(struct vfs_path root);
// Returns true if both paths reference the same entry on the same mount.
static inline bool vfs_path_equal(struct vfs_path a, struct vfs_path b) {
return a.mount == b.mount && a.entry == b.entry;
}
void pipe_destroy(struct pipe* pipe);
errno_t vfs_fifo_open(struct vfs_node* node, struct vfs_file* file);
fs/meson.build
-17
View File
@@ -1,17 +0,0 @@
kernel_sources += files(
'anon.c',
'cache.c',
'devtmpfs.c',
'file.c',
'ident.c',
'init.c',
'initramfs.c',
'mount.c',
'ops.c',
'pipe.c',
'poll.c',
'refs.c',
'tmpfs.c',
)
subdir('exec')
fs/mount.c
-226
View File
@@ -1,226 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/print.h>
#include <zinnia/spin.h>
#include <zinnia/vfs.h>
#include <string.h>
#include "internal.h"
// All known and registered file systems.
static struct spinlock fs_table_lock;
static vec(const struct vfs_filesystem*) fs_table;
// Super blocks of all mounted file systems. Each holds a reference.
static struct spinlock supers_lock;
static vec(struct vfs_super_block*) mounted_supers;
// The global root.
static struct vfs_path vfs_root;
static bool vfs_root_set;
void vfs_set_root(struct vfs_path root) {
assert(!vfs_root_set, "The VFS root can only be set once");
vfs_root = vfs_path_clone(root);
vfs_root_set = true;
}
struct vfs_path vfs_get_root() {
assert(vfs_root_set, "The VFS root hasn't been set yet");
return vfs_path_clone(vfs_root);
}
errno_t vfs_register_filesystem(const struct vfs_filesystem* fs) {
errno_t status = 0;
spin_lock(&fs_table_lock);
vec_foreach(&fs_table, i) {
if (!strcmp(vec_elem_get(&fs_table, i)->name, fs->name)) {
status = EEXIST;
goto out;
}
}
status = vec_push(&fs_table, fs);
out:
spin_unlock(&fs_table_lock);
if (!status)
log("vfs: Registered new file system \"%s\"\n", fs->name);
return status;
}
// Returns a new reference to the super block of `mount`, if it has one.
static struct vfs_super_block* mount_get_super(struct vfs_mount* mount) {
struct vfs_node* root = vfs_entry_get_inode(mount->root);
if (!root)
return nullptr;
struct vfs_super_block* sb = root->sb;
if (sb)
vfs_super_block_ref(sb);
vfs_node_unref(root);
return sb;
}
errno_t vfs_mount(const char* fs_name, enum mount_flags flags, void* arg, struct vfs_mount** out) {
const struct vfs_filesystem* fs = nullptr;
spin_lock(&fs_table_lock);
vec_foreach(&fs_table, i) {
if (!strcmp(vec_elem_get(&fs_table, i)->name, fs_name)) {
fs = vec_elem_get(&fs_table, i);
break;
}
}
spin_unlock(&fs_table_lock);
if (!fs) {
log_warn("Attempted to mount unknown file system type \"%s\"\n", fs_name);
return ENODEV;
}
struct vfs_mount* mount;
errno_t status = fs->mount(fs, flags, arg, &mount);
if (status)
return status;
struct vfs_super_block* sb = mount_get_super(mount);
if (sb) {
bool known = false;
spin_lock(&supers_lock);
vec_foreach(&mounted_supers, i) {
if (vec_elem_get(&mounted_supers, i) == sb) {
known = true;
break;
}
}
// The list takes over our reference.
if (!known && vec_push(&mounted_supers, sb) == 0)
sb = nullptr;
spin_unlock(&supers_lock);
if (sb)
vfs_super_block_unref(sb);
}
*out = mount;
return 0;
}
errno_t vfs_unmount(struct vfs_mount* mount) {
struct vfs_super_block* sb = mount_get_super(mount);
if (!sb)
return 0;
spin_lock(&supers_lock);
vec_foreach(&mounted_supers, i) {
if (vec_elem_get(&mounted_supers, i) == sb) {
vec_swap_remove(&mounted_supers, i);
vfs_super_block_unref(sb);
break;
}
}
spin_unlock(&supers_lock);
errno_t status = sb->ops && sb->ops->sync ? sb->ops->sync(sb) : 0;
vfs_super_block_unref(sb);
return status;
}
errno_t vfs_sync_all() {
// Take a snapshot so we don't call into file systems while holding a spinlock.
spin_lock(&supers_lock);
size_t count = vec_length(&mounted_supers);
struct vfs_super_block** supers = count ? mem_alloc(count * sizeof(*supers), ALLOC_NOZERO) : nullptr;
if (count && !supers) {
spin_unlock(&supers_lock);
return ENOMEM;
}
for (size_t i = 0; i < count; i++) {
supers[i] = vec_elem_get(&mounted_supers, i);
vfs_super_block_ref(supers[i]);
}
spin_unlock(&supers_lock);
errno_t result = 0;
for (size_t i = 0; i < count; i++) {
struct vfs_super_block* sb = supers[i];
if (sb->ops && sb->ops->sync) {
errno_t status = sb->ops->sync(sb);
if (status)
result = status;
}
vfs_super_block_unref(sb);
}
mem_free(supers);
return result;
}
errno_t vfs_mount_new(enum mount_flags flags, struct vfs_entry* root, struct vfs_mount** out) {
struct vfs_mount* mount = mem_alloc(sizeof(*mount), 0);
if (!mount)
return ENOMEM;
refcount_init(&mount->rc);
mount->flags = flags;
mount->root = root;
vfs_entry_ref(root);
mount->has_mount_point = false;
*out = mount;
return 0;
}
errno_t vfs_path_mount(struct vfs_path at, struct vfs_mount* mount) {
if (mount->has_mount_point)
return EBUSY;
mutex_lock(&at.entry->lock);
errno_t status = vec_push(&at.entry->mounts, mount);
mutex_unlock(&at.entry->lock);
if (status)
return status;
vfs_mount_ref(mount);
mount->mount_point = vfs_path_clone(at);
mount->has_mount_point = true;
return 0;
}
errno_t vfs_path_unmount(struct vfs_path path) {
struct vfs_mount* mount = path.mount;
if (path.entry != mount->root || !mount->has_mount_point)
return EINVAL;
vfs_entry_prune(mount->root);
mutex_lock(&mount->root->lock);
const bool busy = !tailq_empty(&mount->root->children);
mutex_unlock(&mount->root->lock);
if (busy)
return EBUSY;
struct vfs_entry* point = mount->mount_point.entry;
bool found = false;
mutex_lock(&point->lock);
vec_foreach(&point->mounts, i) {
if (vec_elem_get(&point->mounts, i) == mount) {
vec_remove(&point->mounts, i);
found = true;
break;
}
}
mutex_unlock(&point->lock);
if (!found)
return EINVAL;
struct vfs_path mount_point = mount->mount_point;
mount->mount_point = (struct vfs_path){};
mount->has_mount_point = false;
vfs_path_drop(mount_point);
errno_t status = vfs_unmount(mount);
vfs_mount_unref(mount);
return status;
}
fs/ops.c
-656
View File
@@ -1,656 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/clock.h>
#include <zinnia/vfs.h>
#include <string.h>
static bool is_read_only(struct vfs_node* node) {
return node->sb && (node->sb->flags & MNT_RDONLY);
}
errno_t vfs_node_try_access(
struct vfs_node* node,
const struct ident* identity,
enum access_flags access,
bool use_real
) {
mutex_lock(&node->lock);
const mode_t mode = node->mode;
const uid_t owner = node->uid;
const gid_t group = node->gid;
mutex_unlock(&node->lock);
if ((access & W_OK) && is_read_only(node) &&
(node->type == VNODE_REGULAR || node->type == VNODE_DIRECTORY || node->type == VNODE_SYMLINK))
return EROFS;
if ((access & X_OK) && node->type == VNODE_REGULAR && node->sb && (node->sb->flags & MNT_NOEXEC))
return EACCES;
if (ident_is_privileged(identity, use_real)) {
if ((access & X_OK) && node->type != VNODE_DIRECTORY && !(mode & (__S_IXUSR | __S_IXGRP | __S_IXOTH)))
return EACCES;
return 0;
}
mode_t granted;
if ((use_real ? identity->uid : identity->euid) == owner)
granted = (mode >> 6) & 7;
else if (ident_in_group(identity, group, use_real))
granted = (mode >> 3) & 7;
else
granted = mode & 7;
return (access & (R_OK | W_OK | X_OK) & ~granted) ? EACCES : 0;
}
static errno_t sync_node(struct vfs_node* node) {
return node->ops && node->ops->sync ? node->ops->sync(node) : 0;
}
static timespec_t now() {
return time_from_nano(clock_get_realtime_ns());
}
// Looks up the parent directory of `path` and checks that `identity` may write to it.
// Returns a new reference to the parent inode in `out`.
static errno_t get_writable_parent(struct vfs_path path, const struct ident* identity, struct vfs_node** out) {
struct vfs_path parent;
errno_t status = vfs_path_lookup_parent(path, &parent);
if (status)
return status;
struct vfs_node* node = vfs_entry_get_inode(parent.entry);
vfs_path_drop(parent);
if (!node)
return ENOENT;
if (node->type != VNODE_DIRECTORY) {
vfs_node_unref(node);
return ENOTDIR;
}
status = vfs_node_try_access(node, identity, W_OK | X_OK, false);
if (status) {
vfs_node_unref(node);
return status;
}
*out = node;
return 0;
}
errno_t vfs_mkdir(
struct vfs_path root,
struct vfs_path cwd,
const char* path,
mode_t mode,
const struct ident* identity
) {
struct vfs_path target;
errno_t status = vfs_path_lookup(root, cwd, path, identity, VFS_LOOKUP_MUST_NOT_EXIST, &target);
if (status)
return status;
struct vfs_node* parent;
status = get_writable_parent(target, identity, &parent);
if (status)
goto out;
status = parent->ops->dir.mkdir ? parent->ops->dir.mkdir(parent, &target, mode, identity) : EPERM;
vfs_node_unref(parent);
out:
vfs_path_drop(target);
return status;
}
errno_t vfs_symlink(
struct vfs_path root,
struct vfs_path cwd,
const char* path,
const char* target_path,
const struct ident* identity
) {
struct vfs_path target;
errno_t status = vfs_path_lookup(root, cwd, path, identity, VFS_LOOKUP_MUST_NOT_EXIST, &target);
if (status)
return status;
struct vfs_node* parent;
status = get_writable_parent(target, identity, &parent);
if (status)
goto out;
status = parent->ops->dir.symlink
? parent->ops->dir.symlink(parent, &target, target_path, strlen(target_path), identity)
: EPERM;
vfs_node_unref(parent);
out:
vfs_path_drop(target);
return status;
}
static bool last_component_is(const char* path, const char* name) {
size_t len = strlen(path);
while (len > 1 && path[len - 1] == '/')
len--;
size_t begin = len;
while (begin > 0 && path[begin - 1] != '/')
begin--;
return len - begin == strlen(name) && !memcmp(path + begin, name, len - begin);
}
static errno_t check_sticky(struct vfs_node* dir, struct vfs_node* victim, const struct ident* identity) {
if (identity->euid == 0)
return 0;
mutex_lock(&dir->lock);
const bool sticky = dir->mode & __S_ISVTX;
const uid_t dir_owner = dir->uid;
mutex_unlock(&dir->lock);
if (!sticky)
return 0;
mutex_lock(&victim->lock);
const uid_t owner = victim->uid;
mutex_unlock(&victim->lock);
return identity->euid == owner || identity->euid == dir_owner ? 0 : EPERM;
}
errno_t vfs_unlink(struct vfs_path root, struct vfs_path cwd, const char* path, const struct ident* identity) {
struct vfs_path target;
errno_t status = vfs_path_lookup(root, cwd, path, identity, VFS_LOOKUP_MUST_EXIST, &target);
if (status)
return status;
struct vfs_node* node = vfs_entry_get_inode(target.entry);
if (!node) {
status = ENOENT;
goto out;
}
if (node->type == VNODE_DIRECTORY) {
status = EISDIR;
goto out_node;
}
struct vfs_node* parent;
status = get_writable_parent(target, identity, &parent);
if (status)
goto out_node;
status = check_sticky(parent, node, identity);
if (!status)
status = parent->ops->dir.unlink ? parent->ops->dir.unlink(parent, &target, identity) : EPERM;
vfs_node_unref(parent);
out_node:
vfs_node_unref(node);
out:
vfs_path_drop(target);
return status;
}
// Probe for a single entry to check if the directory is empty.
static errno_t check_dir_empty(struct vfs_node* node, struct vfs_entry* entry, const struct ident* identity) {
struct __dirent probe;
size_t count = 0;
if (node->ops->dir.get_dir_entries) {
errno_t status = node->ops->dir.get_dir_entries(node, entry, 0, &probe, 1, identity, &count);
if (status)
return status;
}
return count > 0 ? ENOTEMPTY : 0;
}
errno_t vfs_rmdir(struct vfs_path root, struct vfs_path cwd, const char* path, const struct ident* identity) {
if (last_component_is(path, "."))
return EINVAL;
if (last_component_is(path, ".."))
return ENOTEMPTY;
struct vfs_path target;
errno_t status = vfs_path_lookup(root, cwd, path, identity, VFS_LOOKUP_MUST_EXIST, &target);
if (status)
return status;
struct vfs_node* node = vfs_entry_get_inode(target.entry);
if (!node) {
status = ENOENT;
goto out;
}
if (node->type != VNODE_DIRECTORY) {
status = ENOTDIR;
goto out_node;
}
if (target.entry == target.mount->root) {
status = EBUSY;
goto out_node;
}
status = check_dir_empty(node, target.entry, identity);
if (status)
goto out_node;
struct vfs_node* parent;
status = get_writable_parent(target, identity, &parent);
if (status)
goto out_node;
status = check_sticky(parent, node, identity);
if (!status) {
if (parent->ops->dir.rmdir)
status = parent->ops->dir.rmdir(parent, &target, identity);
else if (parent->ops->dir.unlink)
status = parent->ops->dir.unlink(parent, &target, identity);
else
status = EPERM;
}
vfs_node_unref(parent);
out_node:
vfs_node_unref(node);
out:
vfs_path_drop(target);
return status;
}
errno_t vfs_mknod(
struct vfs_path root,
struct vfs_path cwd,
const char* path,
mode_t mode,
struct vfs_mknod_target* target,
const struct ident* identity,
struct vfs_node** out
) {
struct vfs_path new_path;
errno_t status = vfs_path_lookup(root, cwd, path, identity, VFS_LOOKUP_MUST_NOT_EXIST, &new_path);
if (status)
return status;
struct vfs_node* parent;
status = get_writable_parent(new_path, identity, &parent);
if (status)
goto out;
status = parent->ops->dir.mknod ? parent->ops->dir.mknod(parent, &new_path, mode, target, identity) : ENODEV;
vfs_node_unref(parent);
if (!status && out) {
*out = vfs_entry_get_inode(new_path.entry);
if (!*out)
status = ENOENT;
}
out:
vfs_path_drop(new_path);
return status;
}
static bool entry_is_ancestor(struct vfs_entry* ancestor, struct vfs_entry* entry) {
for (struct vfs_entry* it = entry; it; it = it->parent) {
if (it == ancestor)
return true;
}
return false;
}
static bool entry_points_to(struct vfs_entry* entry, struct vfs_node* node) {
struct vfs_node* current = vfs_entry_get_inode(entry);
if (current)
vfs_node_unref(current);
return current == node;
}
static errno_t check_rename_target(struct vfs_node* moved, struct vfs_path target, const struct ident* identity) {
struct vfs_node* existing = vfs_entry_get_inode(target.entry);
if (!existing)
return 0;
errno_t status = 0;
if (moved->type == VNODE_DIRECTORY && existing->type != VNODE_DIRECTORY)
status = ENOTDIR;
else if (moved->type != VNODE_DIRECTORY && existing->type == VNODE_DIRECTORY)
status = EISDIR;
else if (existing->type == VNODE_DIRECTORY)
status = check_dir_empty(existing, target.entry, identity);
vfs_node_unref(existing);
return status;
}
static errno_t check_rename_perms(
struct vfs_node* moved,
struct vfs_path new,
struct vfs_node* old_parent,
struct vfs_node* new_parent,
const struct ident* identity
) {
errno_t status = check_sticky(old_parent, moved, identity);
if (status)
return status;
struct vfs_node* existing = vfs_entry_get_inode(new.entry);
if (existing) {
status = check_sticky(new_parent, existing, identity);
vfs_node_unref(existing);
if (status)
return status;
}
if (moved->type == VNODE_DIRECTORY && old_parent != new_parent)
return vfs_node_try_access(moved, identity, W_OK, false);
return 0;
}
static errno_t do_rename(
struct vfs_path old,
struct vfs_path new,
struct vfs_node* moved,
const struct ident* identity
) {
struct vfs_node* old_parent;
errno_t status = get_writable_parent(old, identity, &old_parent);
if (status)
return status;
struct vfs_node* new_parent;
status = get_writable_parent(new, identity, &new_parent);
if (status) {
vfs_node_unref(old_parent);
return status;
}
status = check_rename_perms(moved, new, old_parent, new_parent, identity);
if (!status) {
if (old_parent->ops->dir.rename)
status = old_parent->ops->dir.rename(old_parent, &old, new_parent, &new, identity);
else
status = EPERM;
}
vfs_node_unref(new_parent);
vfs_node_unref(old_parent);
return status;
}
errno_t vfs_rename(
struct vfs_path root,
struct vfs_path old_cwd,
const char* old_path,
struct vfs_path new_cwd,
const char* new_path,
const struct ident* identity
) {
if (last_component_is(old_path, ".") || last_component_is(old_path, "..") || last_component_is(new_path, ".") ||
last_component_is(new_path, ".."))
return EINVAL;
struct vfs_path old;
errno_t status = vfs_path_lookup(root, old_cwd, old_path, identity, VFS_LOOKUP_MUST_EXIST, &old);
if (status)
return status;
struct vfs_path new;
status = vfs_path_lookup(root, new_cwd, new_path, identity, 0, &new);
if (status)
goto out_old;
if (old.mount != new.mount) {
status = EXDEV;
goto out_new;
}
if (old.entry == old.mount->root || new.entry == new.mount->root) {
status = EBUSY;
goto out_new;
}
if (old.entry == new.entry)
goto out_new;
struct vfs_node* moved = vfs_entry_get_inode(old.entry);
if (!moved) {
status = ENOENT;
goto out_new;
}
if (moved->type == VNODE_DIRECTORY && entry_is_ancestor(old.entry, new.entry))
status = EINVAL;
if (!status && !entry_points_to(new.entry, moved)) {
status = check_rename_target(moved, new, identity);
if (!status)
status = do_rename(old, new, moved, identity);
}
vfs_node_unref(moved);
out_new:
vfs_path_drop(new);
out_old:
vfs_path_drop(old);
return status;
}
errno_t vfs_link(
struct vfs_path root,
struct vfs_path old_cwd,
const char* old_path,
struct vfs_path new_cwd,
const char* new_path,
bool follow_symlinks,
const struct ident* identity
) {
enum vfs_lookup_flags flags = VFS_LOOKUP_MUST_EXIST;
if (follow_symlinks)
flags |= VFS_LOOKUP_FOLLOW_SYMLINKS;
struct vfs_path old;
errno_t status = vfs_path_lookup(root, old_cwd, old_path, identity, flags, &old);
if (status)
return status;
struct vfs_node* target = vfs_entry_get_inode(old.entry);
if (!target) {
status = ENOENT;
goto out_old;
}
if (target->type == VNODE_DIRECTORY) {
status = EPERM;
goto out_target;
}
struct vfs_path new;
status = vfs_path_lookup(root, new_cwd, new_path, identity, VFS_LOOKUP_MUST_NOT_EXIST, &new);
if (status)
goto out_target;
if (new.mount != old.mount) {
status = EXDEV;
goto out_new;
}
struct vfs_node* parent;
status = get_writable_parent(new, identity, &parent);
if (status)
goto out_new;
status = parent->ops->dir.link ? parent->ops->dir.link(parent, &new, target, identity) : EPERM;
vfs_node_unref(parent);
out_new:
vfs_path_drop(new);
out_target:
vfs_node_unref(target);
out_old:
vfs_path_drop(old);
return status;
}
static mode_t node_type_mode(enum vfs_node_type type) {
switch (type) {
case VNODE_DIRECTORY:
return __S_IFDIR;
case VNODE_SYMLINK:
return __S_IFLNK;
case VNODE_FIFO:
return __S_IFIFO;
case VNODE_BLOCKDEV:
return __S_IFBLK;
case VNODE_CHARDEV:
return __S_IFCHR;
case VNODE_SOCKET:
return __S_IFSOCK;
default:
return __S_IFREG;
}
}
void vfs_stat(struct vfs_node* node, struct __stat* out) {
mutex_lock(&node->lock);
*out = (struct __stat){
.st_ino = node->id,
.st_mode = node->mode | node_type_mode(node->type),
.st_nlink = node->nlink,
.st_uid = node->uid,
.st_gid = node->gid,
.st_rdev = node->rdev,
.st_size = node->size,
.st_atim = node->atime,
.st_mtim = node->mtime,
.st_ctim = node->ctime,
.st_blksize = 4096, // TODO: Get from file system
.st_blocks = node->blocks,
};
mutex_unlock(&node->lock);
}
static bool is_owner(struct vfs_node* node, const struct ident* identity) {
return identity->euid == 0 || identity->euid == node->uid;
}
errno_t vfs_chmod(struct vfs_node* node, mode_t mode, const struct ident* identity) {
if (is_read_only(node))
return EROFS;
mutex_lock(&node->lock);
if (!is_owner(node, identity)) {
mutex_unlock(&node->lock);
return EPERM;
}
mode &= 07777;
if (identity->euid != 0 && node->type == VNODE_REGULAR && !ident_in_group(identity, node->gid, false))
mode &= ~__S_ISGID;
node->mode = mode;
node->ctime = now();
mutex_unlock(&node->lock);
return sync_node(node);
}
errno_t vfs_chown(struct vfs_node* node, uid_t uid, gid_t gid, const struct ident* identity) {
if (is_read_only(node))
return EROFS;
mutex_lock(&node->lock);
if (identity->euid != 0) {
const bool uid_ok = uid == (uid_t)-1 || uid == node->uid;
const bool gid_ok = gid == (gid_t)-1 || gid == node->gid || ident_in_group(identity, gid, false);
if (identity->euid != node->uid || !uid_ok || !gid_ok) {
mutex_unlock(&node->lock);
return EPERM;
}
}
if (uid == (uid_t)-1 && gid == (gid_t)-1) {
mutex_unlock(&node->lock);
return 0;
}
if (uid != (uid_t)-1)
node->uid = uid;
if (gid != (gid_t)-1)
node->gid = gid;
if (identity->euid != 0 && node->type == VNODE_REGULAR && (node->mode & (__S_IXUSR | __S_IXGRP | __S_IXOTH)))
node->mode &= ~(__S_ISUID | __S_ISGID);
node->ctime = now();
mutex_unlock(&node->lock);
return sync_node(node);
}
static bool valid_utime(const timespec_t* ts) {
return ts->tv_nsec == __UTIME_NOW || ts->tv_nsec == __UTIME_OMIT ||
(ts->tv_nsec >= 0 && ts->tv_nsec < 1'000'000'000);
}
errno_t vfs_utimens(struct vfs_node* node, const timespec_t* times, const struct ident* identity) {
const timespec_t current = now();
timespec_t ts[2] = {
{.tv_nsec = __UTIME_NOW},
{.tv_nsec = __UTIME_NOW},
};
if (times) {
ts[0] = times[0];
ts[1] = times[1];
if (!valid_utime(&ts[0]) || !valid_utime(&ts[1]))
return EINVAL;
}
if (ts[0].tv_nsec == __UTIME_OMIT && ts[1].tv_nsec == __UTIME_OMIT)
return 0;
if (is_read_only(node))
return EROFS;
const bool only_now = (ts[0].tv_nsec == __UTIME_NOW || ts[0].tv_nsec == __UTIME_OMIT) &&
(ts[1].tv_nsec == __UTIME_NOW || ts[1].tv_nsec == __UTIME_OMIT);
mutex_lock(&node->lock);
const bool owner = is_owner(node, identity);
mutex_unlock(&node->lock);
if (!owner) {
if (!only_now)
return EPERM;
errno_t status = vfs_node_try_access(node, identity, W_OK, false);
if (status)
return status;
}
mutex_lock(&node->lock);
for (size_t i = 0; i < 2; i++) {
if (ts[i].tv_nsec == __UTIME_OMIT)
continue;
timespec_t* target = i == 0 ? &node->atime : &node->mtime;
*target = ts[i].tv_nsec == __UTIME_NOW ? current : ts[i];
}
node->ctime = current;
mutex_unlock(&node->lock);
return sync_node(node);
}
errno_t vfs_node_check_exec(struct vfs_node* node, const struct ident* identity) {
if (node->type != VNODE_REGULAR)
return EACCES;
return vfs_node_try_access(node, identity, X_OK, false);
}
errno_t vfs_truncate(struct vfs_node* node, uint64_t length) {
if (node->type == VNODE_DIRECTORY)
return EISDIR;
if (node->type != VNODE_REGULAR)
return EINVAL;
if (!node->ops->regular.truncate)
return EPERM;
return node->ops->regular.truncate(node, length);
}
errno_t vfs_read_link(struct vfs_node* node, iovec_iter_t* iter, size_t* out_read) {
if (node->type != VNODE_SYMLINK)
return EINVAL;
if (!node->ops->symlink.read_link)
return EINVAL;
return node->ops->symlink.read_link(node, iter, out_read);
}
errno_t vfs_statvfs(struct vfs_node* node, struct __statvfs* out) {
if (!node->sb || !node->sb->ops || !node->sb->ops->statvfs)
return ENOSYS;
return node->sb->ops->statvfs(node->sb, out);
}
fs/pipe.c
-325
View File
@@ -1,325 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/atomic.h>
#include <zinnia/percpu.h>
#include <zinnia/pipe.h>
#include <zinnia/poll.h>
#include <zinnia/ringbuf.h>
#include <zinnia/sched.h>
#include <zinnia/usercopy.h>
#include <zinnia/vfs.h>
#include <uapi/ioctls.h>
#include "internal.h"
constexpr size_t PIPE_CAPACITY = 0x10000;
struct pipe {
mutex_t lock;
struct ringbuf buffer;
size_t readers;
size_t writers;
uint64_t read_opens;
uint64_t write_opens;
struct poll_event read_event;
struct poll_event write_event;
struct poll_event open_event;
};
static const struct vfs_file_ops pipe_file_ops;
static errno_t pipe_new(struct pipe** out) {
struct pipe* pipe = mem_alloc(sizeof(*pipe), 0);
if (!pipe)
return ENOMEM;
errno_t status = ringbuf_init(&pipe->buffer, PIPE_CAPACITY);
if (status) {
mem_free(pipe);
return status;
}
mutex_init(&pipe->lock);
poll_event_init(&pipe->read_event);
poll_event_init(&pipe->write_event);
poll_event_init(&pipe->open_event);
*out = pipe;
return 0;
}
void pipe_destroy(struct pipe* pipe) {
ringbuf_fini(&pipe->buffer);
mem_free(pipe);
}
static bool file_readable(struct vfs_file* file) {
return atomic_load(&file->flags, ATOMIC_RELAXED) & O_RDONLY;
}
static bool file_writable(struct vfs_file* file) {
return atomic_load(&file->flags, ATOMIC_RELAXED) & O_WRONLY;
}
static bool file_nonblocking(struct vfs_file* file) {
return atomic_load(&file->flags, ATOMIC_RELAXED) & O_NONBLOCK;
}
static void notify_all(struct pipe* pipe) {
poll_event_notify(&pipe->read_event);
poll_event_notify(&pipe->write_event);
poll_event_notify(&pipe->open_event);
}
static uint64_t attach_locked(struct pipe* pipe, struct vfs_file* file) {
file->ops = &pipe_file_ops;
file->priv = pipe;
vfs_stream_open(file);
if (file_readable(file)) {
pipe->readers++;
pipe->read_opens++;
}
if (file_writable(file)) {
pipe->writers++;
pipe->write_opens++;
}
return file_readable(file) ? pipe->write_opens : pipe->read_opens;
}
static void pipe_release(struct vfs_file* file) {
struct pipe* pipe = file->priv;
mutex_lock(&pipe->lock);
if (file_readable(file))
pipe->readers--;
if (file_writable(file))
pipe->writers--;
mutex_unlock(&pipe->lock);
notify_all(pipe);
}
static errno_t pipe_read(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_read) {
struct pipe* pipe = file->priv;
const size_t want = iovec_iter_len(buffer);
struct poll_table table;
poll_table_init(&table);
poll_wait(&table, &pipe->read_event);
errno_t status = 0;
size_t read = 0;
mutex_lock(&pipe->lock);
while (true) {
if (ringbuf_len(&pipe->buffer)) {
status = ringbuf_read_iter(&pipe->buffer, buffer, want, &read);
if (read)
poll_event_notify(&pipe->write_event);
break;
}
if (pipe->writers == 0)
break;
if (file_nonblocking(file)) {
status = EAGAIN;
break;
}
if (poll_table_sleep(&table, &pipe->lock, nullptr) == EINTR) {
status = ERESTART;
break;
}
}
mutex_unlock(&pipe->lock);
poll_table_fini(&table);
*out_read = read;
return status;
}
static errno_t pipe_write(struct vfs_file* file, iovec_iter_t* buffer, uint64_t offset, ssize_t* out_written) {
struct pipe* pipe = file->priv;
const size_t total = iovec_iter_len(buffer);
const bool atomic = total <= __PIPE_BUF;
struct poll_table table;
poll_table_init(&table);
poll_wait(&table, &pipe->write_event);
errno_t status = 0;
size_t done = 0;
mutex_lock(&pipe->lock);
while (done < total) {
if (pipe->readers == 0) {
status = EPIPE;
break;
}
const size_t remaining = total - done;
const size_t space = ringbuf_space(&pipe->buffer);
if (space && (!atomic || space >= remaining)) {
size_t written;
status = ringbuf_write_iter(&pipe->buffer, buffer, remaining, &written);
done += written;
if (written)
poll_event_notify(&pipe->read_event);
if (status)
break;
continue;
}
if (file_nonblocking(file)) {
status = EAGAIN;
break;
}
if (poll_table_sleep(&table, &pipe->lock, nullptr) == EINTR) {
status = ERESTART;
break;
}
}
mutex_unlock(&pipe->lock);
poll_table_fini(&table);
*out_written = done;
return done ? 0 : status;
}
static errno_t pipe_ioctl(struct vfs_file* file, uintptr_t request, uintptr_t arg, uintptr_t* out_result) {
struct pipe* pipe = file->priv;
switch ((uint32_t)request) {
case __FIONREAD: {
mutex_lock(&pipe->lock);
int len = ringbuf_len(&pipe->buffer);
mutex_unlock(&pipe->lock);
return usercopy_write((__user void*)arg, &len, sizeof(len)) ? 0 : EFAULT;
}
default:
return ENOTTY;
}
}
static enum poll_flags pipe_poll(struct vfs_file* file, enum poll_flags events, struct poll_table* table) {
struct pipe* pipe = file->priv;
if (file_readable(file))
poll_wait(table, &pipe->read_event);
if (file_writable(file))
poll_wait(table, &pipe->write_event);
enum poll_flags revents = 0;
mutex_lock(&pipe->lock);
if (file_readable(file)) {
if (ringbuf_len(&pipe->buffer))
revents |= POLLIN | POLLRDNORM;
if (pipe->writers == 0)
revents |= POLLHUP;
}
if (file_writable(file)) {
if (ringbuf_space(&pipe->buffer) >= __PIPE_BUF)
revents |= POLLOUT | POLLWRNORM;
if (pipe->readers == 0)
revents |= POLLERR;
}
mutex_unlock(&pipe->lock);
return revents;
}
static const struct vfs_file_ops pipe_file_ops = {
.read = pipe_read,
.write = pipe_write,
.ioctl = pipe_ioctl,
.poll = pipe_poll,
.release = pipe_release,
};
errno_t pipe_create(
enum open_flags flags,
const struct ident* identity,
struct vfs_file** out_read,
struct vfs_file** out_write
) {
struct vfs_node* node;
errno_t status = vfs_create_anon_inode(&pipe_file_ops, VNODE_FIFO, 0600, identity, &node);
if (status)
return status;
status = pipe_new(&node->pipe);
if (status) {
vfs_node_unref(node);
return status;
}
struct pipe* pipe = node->pipe;
const enum open_flags extra = flags & O_NONBLOCK;
struct vfs_file* read_end;
struct vfs_file* write_end;
status = vfs_file_new(node, nullptr, O_RDONLY | extra, &read_end);
if (!status) {
mutex_lock(&pipe->lock);
attach_locked(pipe, read_end);
mutex_unlock(&pipe->lock);
status = vfs_file_new(node, nullptr, O_WRONLY | extra, &write_end);
if (status)
vfs_file_unref(read_end);
}
if (!status) {
mutex_lock(&pipe->lock);
attach_locked(pipe, write_end);
mutex_unlock(&pipe->lock);
}
vfs_node_unref(node);
if (status)
return status;
*out_read = read_end;
*out_write = write_end;
return 0;
}
errno_t vfs_fifo_open(struct vfs_node* node, struct vfs_file* file) {
const bool readable = file_readable(file);
const bool writable = file_writable(file);
const bool nonblock = atomic_load(&file->flags, ATOMIC_RELAXED) & O_NONBLOCK;
if (!readable && !writable)
return EINVAL;
errno_t status = 0;
mutex_lock(&node->lock);
if (!node->pipe)
status = pipe_new(&node->pipe);
struct pipe* pipe = node->pipe;
mutex_unlock(&node->lock);
if (status)
return status;
mutex_lock(&pipe->lock);
if (writable && !readable && nonblock && pipe->readers == 0) {
mutex_unlock(&pipe->lock);
return ENXIO;
}
const uint64_t peer_opens = attach_locked(pipe, file);
mutex_unlock(&pipe->lock);
notify_all(pipe);
if (nonblock || (readable && writable))
return 0;
struct poll_table table;
poll_table_init(&table);
poll_wait(&table, &pipe->open_event);
mutex_lock(&pipe->lock);
while (true) {
const size_t peers = readable ? pipe->writers : pipe->readers;
const uint64_t opens = readable ? pipe->write_opens : pipe->read_opens;
if (peers || opens != peer_opens)
break;
if (poll_table_sleep(&table, &pipe->lock, nullptr) == EINTR) {
status = ERESTART;
break;
}
}
mutex_unlock(&pipe->lock);
poll_table_fini(&table);
return status;
}
fs/poll.c
-125
View File
@@ -1,125 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/atomic.h>
#include <zinnia/percpu.h>
#include <zinnia/poll.h>
#include <zinnia/sched.h>
#include <zinnia/signal.h>
#include <zinnia/timer.h>
struct poll_timeout {
struct timer timer;
struct task* task;
atomic_bool expired;
};
void poll_event_init(struct poll_event* event) {
spin_init(&event->lock);
tailq_init(&event->waiters);
}
void poll_event_notify(struct poll_event* event) {
spin_lock(&event->lock);
for (struct poll_waiter* w = tailq_first(&event->waiters); w; w = tailq_next(w, link)) {
if (w->table->notify) {
w->table->notify(w->table);
continue;
}
atomic_store(&w->table->triggered, true, ATOMIC_RELEASE);
sched_wake(w->table->task);
}
spin_unlock(&event->lock);
}
void poll_table_init(struct poll_table* table) {
table->task = percpu_get()->sched.current;
table->notify = nullptr;
atomic_init(&table->triggered, false);
table->waiters = nullptr;
table->inline_used = false;
table->interruptible = true;
table->error = 0;
}
void poll_table_init_callback(struct poll_table* table, void (*notify)(struct poll_table* table)) {
poll_table_init(table);
table->task = nullptr;
table->notify = notify;
}
void poll_table_fini(struct poll_table* table) {
struct poll_waiter* w = table->waiters;
while (w) {
struct poll_waiter* next = w->next;
spin_lock(&w->event->lock);
tailq_remove(&w->event->waiters, w, link);
spin_unlock(&w->event->lock);
if (w != &table->inline_waiter)
mem_free(w);
w = next;
}
table->waiters = nullptr;
table->inline_used = false;
}
void poll_wait(struct poll_table* table, struct poll_event* event) {
if (!table)
return;
struct poll_waiter* w;
if (!table->inline_used) {
w = &table->inline_waiter;
table->inline_used = true;
} else if (!(w = mem_alloc(sizeof(*w), 0))) {
table->error = ENOMEM;
return;
}
w->event = event;
w->table = table;
w->next = table->waiters;
table->waiters = w;
spin_lock(&event->lock);
tailq_insert_tail(&event->waiters, w, link);
spin_unlock(&event->lock);
}
static void timeout_fn(struct timer* timer) {
struct poll_timeout* timeout = containerof(timer, struct poll_timeout, timer);
atomic_store(&timeout->expired, true, ATOMIC_RELEASE);
sched_wake(timeout->task);
}
errno_t poll_table_wait(struct poll_table* table, const uint64_t* deadline_ns) {
struct poll_timeout timeout = {.task = table->task};
atomic_init(&timeout.expired, false);
timer_init(&timeout.timer, timeout_fn);
if (deadline_ns)
timer_arm(&timeout.timer, *deadline_ns);
errno_t status = 0;
while (!atomic_load(&table->triggered, ATOMIC_ACQUIRE)) {
if (atomic_load(&timeout.expired, ATOMIC_ACQUIRE)) {
status = ETIMEDOUT;
break;
}
if (table->interruptible && signal_pending(table->task)) {
status = EINTR;
break;
}
sched_block(&percpu_get()->sched);
}
if (deadline_ns)
timer_cancel(&timeout.timer);
return status;
}
errno_t poll_table_sleep(struct poll_table* table, mutex_t* lock, const uint64_t* deadline_ns) {
mutex_unlock(lock);
errno_t status = poll_table_wait(table, deadline_ns);
mutex_lock(lock);
atomic_store(&table->triggered, false, ATOMIC_RELAXED);
return status;
}
fs/refs.c
-249
View File
@@ -1,249 +0,0 @@
// SPDX-License-Identifier: ISC
#include <zinnia/alloc.h>
#include <zinnia/assert.h>
#include <zinnia/clock.h>
#include <zinnia/vfs.h>
#include <string.h>
#include "internal.h"
errno_t vfs_node_new(
struct vfs_super_block* sb,
enum vfs_node_type type,
const struct vfs_node_ops* ops,
mode_t mode,
struct vfs_node** out
) {
struct vfs_node* node = mem_alloc(sizeof(*node), 0);
if (!node)
return ENOMEM;
refcount_init(&node->rc);
node->type = type;
node->ops = ops;
node->file_ops = ops ? ops->file_ops : nullptr;
node->sb = sb;
if (sb)
vfs_super_block_ref(sb);
mutex_init(&node->lock);
node->mode = mode & 07777;
node->nlink = 1;
const timespec_t now = time_from_nano(clock_get_realtime_ns());
node->atime = now;
node->mtime = now;
node->ctime = now;
*out = node;
return 0;
}
void vfs_node_ref(struct vfs_node* node) {
refcount_inc(&node->rc);
}
void vfs_node_unref(struct vfs_node* node) {
if (!refcount_dec(&node->rc))
return;
if (node->ops && node->ops->destroy)
node->ops->destroy(node);
if (node->type == VNODE_FIFO && node->pipe)
pipe_destroy(node->pipe);
if (node->sb)
vfs_super_block_unref(node->sb);
mem_free(node);
}
errno_t vfs_entry_new(const char* name, size_t name_len, struct vfs_entry* parent, struct vfs_entry** out) {
struct vfs_entry* entry = mem_alloc(sizeof(*entry), 0);
if (!entry)
return ENOMEM;
entry->name = mem_alloc(name_len + 1, 0);
if (!entry->name) {
mem_free(entry);
return ENOMEM;
}
memcpy(entry->name, name, name_len);
entry->name[name_len] = '\0';
entry->name_len = name_len;
refcount_init(&entry->rc);
mutex_init(&entry->lock);
entry->state = VFS_ENTRY_NOT_CACHED;
entry->inode = nullptr;
entry->parent = parent;
if (parent)
vfs_entry_ref(parent);
tailq_init(&entry->children);
vec_init(&entry->mounts);
*out = entry;
return 0;
}
void vfs_entry_ref(struct vfs_entry* entry) {
refcount_inc(&entry->rc);
}
void vfs_entry_unref(struct vfs_entry* entry) {
if (!refcount_dec(&entry->rc))
return;
// Children and mounts hold references to this entry, so both must be gone by now.
assert(tailq_empty(&entry->children), "Destroying an entry that still has children");
assert(vec_is_empty(&entry->mounts), "Destroying an entry that is still a mount point");
if (entry->inode)
vfs_node_unref(entry->inode);
if (entry->parent)
vfs_entry_unref(entry->parent);
vec_destroy(&entry->mounts);
mem_free(entry->name);
mem_free(entry);
}
void vfs_entry_set_inode(struct vfs_entry* entry, struct vfs_node* node) {
vfs_node_ref(node);
mutex_lock(&entry->lock);
struct vfs_node* old = entry->inode;
entry->inode = node;
entry->state = VFS_ENTRY_PRESENT;
mutex_unlock(&entry->lock);
if (old)
vfs_node_unref(old);
}
void vfs_entry_set_negative(struct vfs_entry* entry) {
mutex_lock(&entry->lock);
struct vfs_node* old = entry->inode;
entry->inode = nullptr;
entry->state = VFS_ENTRY_NOT_PRESENT;
mutex_unlock(&entry->lock);
if (old)
vfs_node_unref(old);
}
void vfs_entry_replace_children(struct vfs_entry* dst, struct vfs_entry* src) {
tailq_head(struct vfs_entry) stale;
tailq_init(&stale);
mutex_lock(&src->lock);
mutex_lock(&dst->lock);
struct vfs_entry* child;
while ((child = tailq_first(&dst->children))) {
tailq_remove(&dst->children, child, child_link);
tailq_insert_tail(&stale, child, child_link);
}
while ((child = tailq_first(&src->children))) {
tailq_remove(&src->children, child, child_link);
child->parent = dst;
vfs_entry_ref(dst);
vfs_entry_unref(src);
tailq_insert_tail(&dst->children, child, child_link);
}
mutex_unlock(&dst->lock);
mutex_unlock(&src->lock);
while ((child = tailq_first(&stale))) {
tailq_remove(&stale, child, child_link);
vfs_entry_unref(child);
}
}
void vfs_entry_prune(struct vfs_entry* entry) {
tailq_head(struct vfs_entry) stale;
tailq_init(&stale);
mutex_lock(&entry->lock);
struct vfs_entry* child = tailq_first(&entry->children);
while (child) {
struct vfs_entry* next = tailq_next(child, child_link);
vfs_entry_prune(child);
mutex_lock(&child->lock);
const bool idle = tailq_empty(&child->children) && vec_is_empty(&child->mounts);
mutex_unlock(&child->lock);
if (idle && atomic_load(&child->rc.count, ATOMIC_RELAXED) == 1) {
tailq_remove(&entry->children, child, child_link);
tailq_insert_tail(&stale, child, child_link);
}
child = next;
}
mutex_unlock(&entry->lock);
while ((child = tailq_first(&stale))) {
tailq_remove(&stale, child, child_link);
vfs_entry_unref(child);
}
}
struct vfs_node* vfs_entry_get_inode(struct vfs_entry* entry) {
struct vfs_node* node = nullptr;
mutex_lock(&entry->lock);
if (entry->state == VFS_ENTRY_PRESENT) {
node = entry->inode;
vfs_node_ref(node);
}
mutex_unlock(&entry->lock);
return node;
}
void vfs_mount_ref(struct vfs_mount* mount) {
refcount_inc(&mount->rc);
}
void vfs_mount_unref(struct vfs_mount* mount) {
if (!refcount_dec(&mount->rc))
return;
if (mount->has_mount_point)
vfs_path_drop(mount->mount_point);
vfs_entry_unref(mount->root);
mem_free(mount);
}
void vfs_super_block_ref(struct vfs_super_block* sb) {
refcount_inc(&sb->rc);
}
void vfs_super_block_unref(struct vfs_super_block* sb) {
if (!refcount_dec(&sb->rc))
return;
if (sb->ops && sb->ops->destroy)
sb->ops->destroy(sb);
}
void vfs_file_ref(struct vfs_file* file) {
refcount_inc(&file->rc);
}
void vfs_file_unref(struct vfs_file* file) {
if (!refcount_dec(&file->rc))
return;
if (file->ops && file->ops->release)
file->ops->release(file);
if (file->has_path)
vfs_path_drop(file->path);
vfs_node_unref(file->vnode);
mem_free(file->absolute_path);
mem_free(file);
}
struct vfs_path vfs_path_clone(struct vfs_path path) {
vfs_mount_ref(path.mount);
vfs_entry_ref(path.entry);
return path;
}
void vfs_path_drop(struct vfs_path path) {
vfs_entry_unref(path.entry);
vfs_mount_unref(path.mount);
}

Some files were not shown because too many files have changed in this diff Show More