Changes to 16-FEB-2011: A simple "append-only" database between r2 and r3
'''A proposed API and implementation for an append-only database'''
1. Journal (multiple concurrent writers) -- all writes go here, acts as a buffer
2. Database (flat text file, looks like a log file) -- written to by single thread reading from Journal; order of inserts is not guaranteed to be the same as order of inserts into journal
3. Index (binary tree of keys) -- values are kept in sorted order with a count of the number of values; values are pointers to offsets in the Database
Support disabling the journal or the index at run-time by not specifying them during an `amdb_open()` call
Journal format:
1. [[Magic (32-bits)]]
1. [[Journal Bypass Flag (8-bits)]]
2. [[Header Size (`HDR_SZ`; 32-bits)]]
3. [[Header (`HDR_SZ` * 8-bits)]]:
33. Tag-Length-Value (32b, 32b, Variable)
33. Tags:
333. `0x00`: Number of tracks (`NUM_TRACKS`)
333. `0x01`: Track size (`TRACK_SIZE`)
333. `0x02`: Track 0 offset from start of file (`TRACK0_START`)
4. [[Tracks (`NUM_TRACKS` * `TRACK_SIZE` * 8-bits)]]
(blank line)
44. Tracks are an array of bytes `TRACK_SIZE` (which must be a multiple of the page size) bytes long
44. Each track is a circular buffer for a particular writer
44. Each entry in the buffer is of variable length and called a message
44. Each message is in the format:
444. [[Removed (boolean; 8-bits)]]
444. [[Complete (boolean; 8-bits)]]
444. [[Message Length (`MSG_LEN`; 16-bits)]]
444. [[Message (`MSG_LEN` * 8-bits)]]
(blank line)
Journal bypass can be `mmap()`'d and checked before all writes. This can be used to resize the journal with the application running. Writes can either be held, or go straight to the database (mutex).
The number of tracks will determine the number of concurrent writers. Each track will be a range of bytes that are exclusively written to by a single thread.
API:
1. void *amdb_open(const char *db, const char *journal, const char *index);
11. All parameters are strings containing the pathnames to files to store their respective components
2. void amdb_insert(void *handle, time_t date, ...);
22. All additional arguments are required to be strings in pairs, the first of the pair being the field name and the second being the field contents
22. Terminate with NULL
3. void amdb_consumejournal(void *handle);
33. Starts a process that consumes entries from the journal and writes to database and index files.
Only in r2
Only in r3 --> Modified slightly between r2 and r3