The Gecko Log Structured Storage System

Introduction

Disk contention is increasingly a significant problem for cloud storage, as applications are forced to co-exist on machines and share physical disk resources. Disks are notoriously sensitive to contention; a single application’s random I/O is sufficient to reduce the throughput of a disk array by an order of magnitude, disrupting every other application running on the same array. Log-structured storage designs can alleviate write-write contention between applications by sequentializing all writes, but have historically suffered from read-write contention triggered by garbage collection (GC) as well as application reads. Gecko is a novel log-structured design that eliminates read-write contention by chaining together a small number of drives into a single log, effectively separating the tail of the log (where writes are appended) from its body. As a result, writes proceed to the tail drive without contention from either GC reads or first-class reads, which are restricted to the body of the log with the help of a tail-specific caching policy. Gecko trades-off maximum contention-free sequential throughput from multiple drives in exchange for a stable and predictable maximum throughput from a single uncontended drive, and achieves better performance compared to native log-structured or RAID based systems for most cases. Our in-kernel implementation provides random write bandwidth to applications of 60 to 120MB/s, despite concurrent GC activity, application reads, and an adversarial workload.


Publications

People

Links

Gecko early concept slides

Download dm-gecko (implemented using the Linux device-mapper)

  • Alpha version (do not use to store your critical data): repository snapshot.

  • You can explore the source code by pointing your browser here (may be slightly out of date).

Instructions

  1. Download and build the snapshot (type 'make' in the gecko folder---all subsequent instructions assume gecko folder to be the current directory). Make sure you have the Linux headers installed, e.g. in Ubuntu the package is called linux-headers-2.6.xx-xx where xx-xx should match your current kernel (type 'uname -r' to find it out). The kernel module was tested on kernel versions 2.6.35 and 2.6.36.

  2. Edit the in.sh file to reflect the disks attached to your system and run it (alternatively, you can edit the in-loop-devs.sh file if you plan to use loop devices, say in a virtual machine).

    • The in.sh script takes two parameters as input. The first parameter is a boolean value that indicates if the gecko metadata (e.g. block level relocation maps) should be loaded from a fixed file location (currently /tmp/foo). The second parameter is the disk layout, and gecko currently supports linear and raid1 layouts. (Common usage example: bash in.sh 1 raid1)
    • To unload the gecko module (only after you unmounted any prevously mounted file system), you can use the out.sh script (bash out.sh). Note that it is hardcoded to always write the gecko metadata to the fixed file location /tmp/foo, hence you should edit the script accordingly.

  3. You can start fresh; e.g. by setting the first parameter of in.sh to 0, which will not load the gecko metadata from a persistent location (e.g. /tmp/foo) effectively acting like a block-level quick format. Further, you can also zap the existing file system and create a fresh one (sudo mkfs.ext2 -b 4096 /dev/mapper/gecko).

  4. Mount the existing file system (sudo mount -t ext2 -o noatime /dev/mapper/gecko /mnt/). Note that if you don't need a file system, you can interact directly with the /dev/mapper/gecko block device as with any regular block device (e.g. blockdev, or POSIX operations like open / read / write / close).

  5. The file system is mounted as root, hence you will need to create some folders that your regular user has access to (e.g. sudo mkdir /mnt/tudorm && sudo chown -R tudorm:tudorm /mnt/tudorm).

  6. Perform work on the file system.

    • To monitor gecko continuously, you can use iostat (e.g. iostat 1 and ignore the report that appears during the first second) as well as the gecko exported device-mapper counters and stats (while [ 1 ]; do NAME=gecko; clear && sudo dmsetup table "${NAME}" && echo "" && sudo dmsetup status "${NAME}"; sleep 1 ; done).
    • To control some of the gecko knobs, you can use dmsetup message, e.g.:
      • To turn the gc (garbage collection / block relocation) on sudo dmsetup message gecko 0 "gc-on", and sudo dmsetup message gecko 0 "gc-off" to turn it off.
      • To toggle between max-throughput and power-conserving modes: sudo dmsetup message gecko 0 "set-high-read-tput" and sudo dmsetup message gecko 0 "set-low-power" respectively.
      • To toggle between more detailed dmsetup table/status counters: sudo dmsetup message gecko 0 "detail-on" and sudo dmsetup message gecko 0 "detail-off".
      • To set the high and low watermarks for the gc: sudo dmsetup message gecko 0 "set-gc-watermarks" low-watermark-integer-value high-watermark-integer-value (the watermarks are given as number of blocks, or pages---Gecko currently works with block size == page size == 4K).