Easily Create In-Memory Directory

In Memory

If you have ever had to run a task which performs a lot of on-disk file operations you know how much of a bottleneck the hard drive could be. It’s just not fast enough to keep up with the demands of the program running in the CPU.

So what I usually do when I need to run some very disk intensive tasks is to move it all to an in-memory filesystem. Linux comes with a module called tmpfs, which allows you to mount such a filesystem.

So, assuming my file operations would be happening at /opt/work. What I would do is

1. Move /opt/work out of the way, to /opt/work.tmp
2. Make a new directory called /opt/work
3. Mount the in-memory filesystem at /opt/work
   sudo mount -t tmpfs none /opt/work
4. Copy the contents of /opt/work.tmp into /opt/work.
5. Start the program.
6. Then when the program completes I would copy everything back, unmount the in-memory filesystem and clean up.

Now, since the directory where all the operations will be happening isn’t actually on disk (but in memory), it can happen without any hard drive bottleneck.

Here are some benchmark results showing how much faster this is. I was writing 1GB of data to each filesystem. Specifically look at the last bit of the last line.

For the memory filesystem:
quintin@quintin-VAIO:/dev/shm$ dd if=/dev/zero of=test bs=$((1024 * 1024)) count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.984447 s, 1.1 GB/s

And for the hard drive filesystem:
quintin@quintin-VAIO:/tmp$ dd if=/dev/zero of=test bs=$((1024 * 1024)) count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 24.2657 s, 44.2 MB/s

As you can see, the memory filesystem was more than 24 times faster than the hard drive filesystem. I was able to write at only 44.2MB/s on the hard drive, but 1.1 GB/s on the memory filesystem. Note that this isn’t it’s maximum, since I had some other bottlenecks here. If you were to optimize this you would be able to write even faster to the memory filesystem. The fact remains that filesystem intensive tasks can run much faster when done this way.

There are some risks involved, in that loosing power to the system will cause everything that was in the memory filesystem to be lost. Keep this in mind when using it. In other words, don’t store any critical/important data only in an in-memory filesystem.

The Core of it All

So in the end it all comes down the the fact that you can easily create a in-memory filesystem. All you need to do to achieve this is decide on a directory you want to be in-memory, and mount it as such.

For example, if we were to choose /home/mem as an in-memory directory, we can mount it as follows:
sudo mount -t tmpfs none /home/mem

If we need it to be persistently mounted as such (across system boots), we can add the following line to /etc/fstab:
none /home/mem tmpfs defaults 0 0

Conclusion

So Why Love Linux? Because with a single command you can get a temporary in-memory filesystem running to help you speed up a filesystem intensive task.