name

hiv - hecc's standard library

synopsis

hiv [--two-variants] [--the-default-allocator] [--allocator-t] [--malloc-and-free] [--sized-operations] [--strings] [--memory-safety] [--hecc] [--library-functions] [--objects-and-allocators] [--portability] [--why-reimplement-libc]

description

hiv (as opposed to std) is hecc's standard library. it reimplements libc and makes two hidden things about memory explicit. it's written in hecc, so its functions have ownership decorators built in. hecc transpiles to plain c, so plain c projects can use hiv too.

hecc is an experimental c compiler that adds static ownership, borrows, and bounds analysis to c.

hiv handles the two hidden things separately. take strdup().

char* copy = strdup(string);

the first is ownership. nothing in the call says the caller owns the result and has to free it. the second is the allocator. strdup uses the global heap, and a program with its own arena or pool can't redirect it without rewriting it.

two variants

every hiv function that touches memory comes in two forms. the default form keeps the c signature and adds ownership information.

char* copy = strdup(string);

it compiles in any c program. built with hecc, it also says the caller owns the result and must free it, which hecc can check.

the _with form takes an allocator as its first argument.

char* copy = strdup_with(&allocator, string);

now the caller also picks where the memory comes from. every function that allocates or frees has both forms (malloc and malloc_with, strdup and strdup_with, free and free_with, and so on).

so ownership is always explicit, and the allocator is explicit when you ask. like in hecc, the default drops in and the explicit form is opt-in.

the default allocator

a default call uses a program-wide default allocator. the allocation is still visible, since you called malloc or strdup. so is ownership, since the decorators say who owns a result and what a free consumes, and hecc checks them. only the allocator is implicit, and _with can name it anywhere.

the program sets the default once. a hosted program can keep the usual heap. a freestanding one (a kernel, or bare metal) has no global heap. it installs a default before using the default forms, or uses _with everywhere.

allocator_t

an allocator_t provides the operations for allocating, resizing, and freeing memory.

struct allocator_t
{
    void* (*alloc)(struct allocator_t* allocator, size_t size);
    void* (*realloc)( struct allocator_t* allocator, void* buffer, size_t size);
    void (*free)( struct allocator_t* allocator, void* buffer);

    void* (*realloc_sized)(struct allocator_t* allocator, void* buffer, size_t old_size, size_t new_size);
    void (*free_sized)(struct allocator_t* allocator, void* buffer, size_t size);
};

to write your own allocator, you write those callbacks.

void* my_alloc(struct allocator_t* allocator, size_t size)
{
    // ...
}

then you build an allocator from them.

struct allocator_t allocator =
{
    .alloc = &my_alloc
};

other code doesn't normally call the callbacks. it either uses the _with form when it wants this allocator

void* buffer = malloc_with(&allocator, 1024);

or installs it as the program default, so a plain malloc(1024) goes through it. the calling code stays the same whether the allocator is a heap, a fixed region of memory, an arena, a pool, or something environment-specific. hiv only needs the operations implemented.

malloc() and free()

the default forms look like plain c and carry ownership.

void* buffer = malloc(1024);   /* default allocator; the result is owned */

/* ... */

free(buffer);                  /* consumes ownership */

the _with forms name the allocator.

void* buffer = malloc_with(&arena, 1024);

/* ... */

free_with(&arena, buffer);

this way several allocators can live side by side, with no global state.

void* temporary = malloc_with(&temporary_allocator, 1024);
void* permanent = malloc_with(&permanent_allocator, 1024);

/* ... */

free_with(&temporary_allocator, temporary);
free_with(&permanent_allocator, permanent);

they can work completely differently. temporary_allocator might use an arena and permanent_allocator a normal heap. neither is special to the rest of the program.

memory has to go back to the allocator it came from. default malloc pairs with default free, and malloc_with(&a, …) with free_with(&a, …). hiv doesn't store the allocator in the allocation. matching them is your job, like knowing where a pointer came from, and the decorators let hecc check the free happens.

sized operations

a program often knows a buffer's size when the function getting the pointer doesn't. say you have this.

char buffer[6] = "hello";

the program knows buffer is six bytes. a normal string function only gets a pointer

strlen(buffer);

and reads until a null terminator, not knowing where the object ends.

hiv has _sized variants for operations where knowing the size helps.

strdup_sized(buffer, sizeof(buffer));

the size is the buffer's exact size in bytes. it isn't a string length and says nothing about the contents. a six-byte buffer might hold this

h e l l o \0

or six bytes with no null. a sized string operation won't look past those six bytes, so it can check for a terminator without running into unrelated memory.

_sized and _with combine. the default sized form takes the size, and the explicit one also takes the allocator.

free_sized(buffer, size);
free_with_sized(&allocator, buffer, size);

resizing can pass the old and new size the same way.

buffer = realloc_sized(buffer, old_size, new_size);
buffer = realloc_with_sized(&allocator, buffer, old_size, new_size);

the plain versions stay for when you don't know the size. when you do, the allocator can use it directly.

strings

a c string is a char* plus unstated rules. every c string bug comes from what the char* leaves out, which is whether it's terminated, how big the buffer is, and who frees it. hiv keeps the plain char*, so existing functions and the representation don't change. it adds the missing facts using what hecc already has.

hecc already tracks three of the four facts, since they aren't specific to strings. ownership (an owned string is freed when its owner ends, a borrowed one isn't) works like any char* from strdup. capacity, the buffer's size in bytes, is bounds, as in the _sized forms. maybe-null is nullability.

the fourth, nul-termination, is specific to strings. it means the buffer is known to have a terminator inside its bounds. c never checks this, and hecc tracks it like the others.

string is hiv's name for a char* with those facts (owned, nul-terminated, known capacity), and it erases back to char*.

string s = strdup("hello");   /* owned, NUL-terminated */

a read-only function takes a borrow and can require the terminator it relies on.

size_t length(const string s);   /* borrows s; requires it terminated */

passing length a buffer not known to be terminated is then a provable mistake, instead of a read off the end. hecc flags it like a use-after-free.

the distinction that catches the most bugs is capacity versus length. capacity is the buffer's size, a static bound. length is the bytes up to the nul, a runtime value. hiv tracks capacity at compile time and termination as yes or no. a safe build checks the length, and that length <= capacity, at runtime.

string literals need no checks. "hello" is const, terminated, and has a known length and static origin.

a growable string is the same owned char*, resized through the round-trip owner parameter.

bool append(string <-> s, const char* tail);

ownership leaves and comes back through s, so a reallocation is tracked like any growing allocation.

hiv's string is deliberately not a fat pointer. a { ptr, len } value would change the representation, break the c abi, and stop it being a char*. the length lives in the bounds facts and the terminator instead.

it's also not a unicode type. a string is a byte buffer that must stay terminated, and staying at the byte level keeps it c. utf-8 is left to libraries.

memory safety

explicit memory operations alone don't make c memory-safe. a program can still dereference a bad pointer, write past an array, use memory after freeing it, or break c's rules in other ways. hiv doesn't change c's pointer model or add compile-time ownership tracking. that's hecc's job, and hiv plugs into it (see hecc below).

what hiv does give you is one interface you control for all the standard library's memory operations. a normal libc uses its own allocator internally, so strdup(string) ignores your custom allocator. in hiv it's always the program's choice, either the default or the one passed to _with, and it can do more than hand out memory.

for example, a debugging allocator can track every allocation it owns. when memory is freed

free_with(&debug_allocator, buffer);

it can check that buffer is one of its allocations. that catches invalid frees, double frees, and frees through the wrong allocator. installed as the default, it also checks plain free(buffer) calls, so you don't have to rewrite code to use _with.

sized operations give it more to check. given this call

free_with_sized(&allocator, buffer, 128);

the allocator gets the address and the size the caller thinks the allocation has. if it tracks its allocations, it can verify that size. realloc_with_sized() lets it confirm that old_size matches the allocation being resized.

to make bugs easier to catch, an allocator can poison freed memory or quarantine it instead of reusing it right away. on systems with virtual memory, it can put that memory behind inaccessible pages, so an out-of-bounds access faults instead of quietly corrupting another allocation.

it can enforce other policy too. a limited allocator caps how much memory part of a program uses. an arena gives everything allocated during one operation the same lifetime, and separate allocators can stand for separate memory domains.

char* temporary = strdup_with(&temporary_allocator, string);
char* request_copy = strdup_with(&request_allocator, string);

what matters is that the standard library takes part instead of escaping the program's memory model. the same interface works with any of these allocators. strdup() keeps its meaning, and the allocator decides where its memory comes from and what gets checked. so memory-safety tools work with the standard library without rewriting it for each allocator.

hecc

hiv and hecc are two halves of one experiment. because hiv is written in hecc, its decorators are written into the source from the start. they say which arguments are consumed, which returns are owned, and which are borrows and from where.

hecc's analysis is only as good as the contracts on the functions a program calls, and most of those are in the standard library. a libc hecc can't see into is a blind spot, and hiv makes that boundary explicit.

hecc tracks who owns what and who frees it. hiv states each call's ownership, exposes the allocator through _with, and gives hecc's bounds analysis sizes through _sized. an hiv guarding allocator is the runtime safety net for code hecc can't prove anything about statically. neither needs the other, but together they cover memory from both sides.

library functions

the two forms only exist where they mean something. functions that don't allocate or free memory stay the same, with no _with variant.

memcpy(destination, source, size);
memset(destination, value, size);
strlen(string);
strcmp(left, right);

copying bytes between existing buffers involves no allocator, so there's nothing to make explicit. keeping the extra form to functions that touch memory stops hiv from becoming a different flavour of c. most of the standard library still looks like the standard library.

objects and allocators

passing an allocator to a function doesn't attach it to the returned object. say a file is opened with a particular allocator.

FILE* file = fopen_with(&allocator, "example.txt", "r");

hiv doesn't keep allocator inside file to use quietly later. a later operation that frees memory gets the allocator passed again.

fclose_with(&allocator, file);

if the object remembered its allocator, fclose(file) could free memory with nothing at the call site showing which allocator was used. memory operations would be implicit again. so a _with open pairs with the matching _with close, and a default open with a default close.

portability

hiv aims to implement the iso c standard library, not posix. it should work in hosted programs, but also in custom operating systems, kernels, freestanding programs, and bare metal without a posix-compatible runtime.

that's another reason the allocator model matters, since hiv can't assume a normal heap. a bare-metal program might have one fixed block of ram, and a kernel its own page allocator. another environment might want an arena for everything.

in those cases there's no sensible default, so the default forms aren't available until one is installed. code uses _with to name the allocator that fits its memory. hiv doesn't get to decide what that memory looks like.

why reimplement libc?

there are way easier ways to use a custom allocator, like writing allocator-aware functions, using your own allocation api, or swapping the global allocator. reimplementing a whole standard library is overkill for that.

the interesting part is applying the idea consistently. writing this much

char* strdup(const char* string);                          /* default: carries ownership */
char* strdup_with(struct allocator_t* allocator, const char* string);

is trivial. doing it across a whole standard library isn't. some functions obviously allocate, since they return a fresh pointer. others create objects with internal state, allocate midway through some work, or free memory while tearing something down.

hiv is mostly an attempt to find those less obvious cases and see if making them explicit gives a nicer library. it may be more annoying than a normal libc in places, and finding that out is part of the experiment.

i'm not after verbosity for its own sake. i want to see whether a standard library can be upfront about memory (ownership always, the allocator when you ask for it) and keep the simplicity that made c's library useful.

ownership is never hidden, and the allocator is never out of reach.

see also

hecc(7), iridis(7), style-guide(7), blabbing(7), hachem(1)

author

hachem <im@hachem.wtf>