Anyone using RAM Disks these days?

Last week I was chasing a “builds on my machine” problem where the builds were failing on our build agents (:chair: :man_facepalming: :keyboard: ) - and was getting frustrated with the time the builds were taking. A few minutes doesn’t seem long when you are doing 1 build, but when you are running it over and over trying to resolve an issue it soon becomes really annoying!

Our build servers are pretty old (7 yrs) and using spinning rust (6Gps SAS raid 10) - we do have a spiffy new all nvme dual amd epyc server (Gen 3, g4 is :cry: :dollar: :moneybag:) on order - but that is still several weeks away (and then we have to install/migrate everything etc) - so I was thinking about other ways to speed up our build process (we have already optimised the cr@p out of the the CI server/agent code).

Builds on Continua CI are mostly I/O bound - each build happens in a clean workspace folder, so the source code has to be checked out into the workspace - checking out thousands of small files onto spinning rust is slow.

One thought I had was to use a RAM disk - I had to google to see if they are still a thing, I remember using one in the late 90’s but it was tiny… these days machines have a bit more ram so it might be more practical. I remember back then the main issue was stability - they tended to crash windows pretty often.

Anyone using RAM Disks (in particular with delphi builds)? Any issues? Product recommendations?.

I’m not worried about the fact that they are non persistent - the build workspace folders on our CI agents get cleaned up automatically anyway so nothing would be lost by a reboot.

I’m sure you’ll notice the performance improvement with the new server :slight_smile: I haven’t got anything to say about using them… but here is a list of them

Yes, I’m using a RAM disk - not for Delphi builds but for temporary storage when doing TDD - some of my tests involve reading/writing files and the log file from the tests is also written to the RAM disk.

I’m using ImDisk:

No issues with stability that I’ve found. There’s no limit to the size of your RAM disks other than available RAM - I did some testing with a RAM disk that was over 100GB at one point…

Since I’ve got plenty of RAM on my dev system, I’ve got the system set up to automatically create the RAM disk during system startup, so that I’ve always got it available.

Slight :smirk: I sure hope so, cost an arm and a leg!

Thanks - hadn’t seen that and there’s a few on there I hadn’t found so far.

Does it work even when not logged in? I need something that can be used by a windows service.

Our new server will have 512GB ram - so there’s plenty of scope for large ram disks! We actually only need probably a 10GB ram disk to make things work - Continua CI cleans up the workspace when the build is done (unless instructed otherwise) so we can tune the size to our needs quite easily.

Hi Vincent

Yes. I think ImDisk is the one.very stable. been doing this for a decade or more.

I run 16 concurrent Delphi jobs builds all with their own git clone workspace full checkout and delete after each build. concurrency # depends on CPU cores and IO bus bandwidth… the size of the drive depends on how much ram the server has and how much is needed. my drive is around 200GB. enough to host all the concurrent jobs. selected build artifacts go to a persistent drive.

Nvme might do a good job too but it’s temporary so why bother. RAM is cheap enough now.

Delphi command line compiler has no issues so long as you don’t choose certain drive letters which interact with command options due to ambiguity. K: from memory was problematic.

Hope that helps.

Jared

Thanks @jdavison

I’ll give it a go.

I’m curious - what are you using to run those builds?

Not that cheap! 512GB DDR4 for our new server was $4500!

LOL - I though I knew most of the command line compilers foibles but that’s a new one for me. We generally use I: for our build workspace drives (no idea why) and haven’t had any issues.

So I tried it on my dev machine (all nvme) - I ran several builds with and without the ram disk, and this is what I saw (to checkout the files).

No Ram Disk - 34s

Ram Disk - 41s

Not sure why it is slower… perhaps the nvme drive is doing some optimisations as the hg repo it is checking out from is on the same drive.

I’ll test it tomorrow on our production server and see if it makes a difference.

I would presume so, but it’s not something that I’ve tried - sorry.

Nice - with that amount of RAM, you could have a decent number of builds running at the same time.

Out of curiosity, which specific Epyc processors did you go with? Since you’ve gone with a dual processor system, I’m guessing that you haven’t gone for the bottom of the range. What are you going to do to make use of all those cores - even with 512GB of RAM, I’m guessing that that isn’t going to be enough to be able to max out all of your cores with parallel builds - at least on RAM disks?

That’s quite strange - I certainly wouldn’t have expected that. I wouldn’t think that checking out to the same drive would make it any faster.

What happens if you check it out to a different NVMe drive?

I went totally overboard for todays needs - dual 7543 - 32 core 64 threads (partly because of what’s available).

I don’t buy servers every year, our youngest server is 7yrs old and the oldest is 15 yrs old. I’ll be retiring 3 old servers and repurposing the 7yr old one as a truenas storage server (with a new lsi controller card, sas3 backplane, 12Gbps sas hd’s and optane ssd’s for zfs slog). When budget permits I will purchase another server for offsite truenas/zfs replication.

The new server is from https://www.hyperscalers.com.au/ - based in Canberra. I used to buy supermicro but they have gone nuts with their pricing lately.

We run a bunch of VM’s for a CI server and CI agents, plus vm’s for testing various versions of TFS, Mattermost, Octopus Deploy, plus domain controllers and various other vms that come and go. We’re also looking at getting Continua CI going on Linux (agents first, then server) so need more VM’s for that.

At the moment, we have to be really mindful of what else is running on the servers to make sure we don’t slow builds down… an over speccd server will give us the freedom to experiment a bit more without impacting on production builds.

We also have 2 servers colocated at a DC in LA - those are for website/mail etc.

Nice - a little overboard, but not quite as overboard as a dual AMD EPYC™ 9654P that I was imagining - assuming such a beast is possible, it would support 384 threads.

We’ve come a long way since the 10 MHz 8088 in my first PC. I did dabble with a Z80 single board computer for a bit though as well.

No difference.

I suspect the issue is using the mount point. CrystalDiskMark is showing the RAM disk slower than the NVME drive!

Not sure why that is, machine is a recent Ryzen 5950x, 64GB Ram. Need to investigate more.

Edit : windows says my ram speed is 2133Mhz - where it should be 3200MHz - so probably need to look at the bios settings.

Well Ram speed made zero difference.

I tried another ram disk - Primo Ram Disk - which with CrystalDiskMark benchmarks a lot faster than ImDisk - but in practical use it was still a few seconds slower than the nvme drive.

CrystalDiskMark_20230202110108-samsung
CrystalDiskMark_20230202110122-ImDisk
CrystalDiskMark_20230202110153-primo

I suspect the limitations here are really the read speed of the (other) nvme drive the repository is on.

I installed ImDisk on a CI Agent VM, and instead of taking 2minutes to checkout the source, it took 3m30s.

I give up… waiting on the new server!

Yes we are using RAM disks in our build servers.
Since some people are cheap and we are putting consumer quality SSDs into them they usually start to give I/O errors after about 2 years or even earlier.

Which ram disk are you using, I wasn’t able to get any better performance out of any the ones I tried.

I don’t know - I am not the person doing DevOps - the goal was not better perf than SSD but to not trash them in record time with tons of write operations.

Understandable - consumer ssd’s are not good for that.

I spent a lot of time researching this before buying a new server. I’m also bulding a storage server (probably Truenas) and will use ZFS - it’s slog feature will trash consumer ssd’s in no time - so went with intel optane 900P ssd’s for the slog - they are from 2018 (new in the box though) but have an incredible write endurance of 10 DWPD - by comparison, the samsung 990 Pro 4TB consumer ssd has a TBW rating of just 2400TB

Optane : 10 Drive Writes Per Day = 10 x 280GB * 365 * 5 (yrs warranty) = 5110TB
Consumer : Terra Bytes Written (over warranty period of 5yrs) : 2400

I use a FAT32 1GB RAM Disk not because it’s fast but because it’s volatile.

It’s my junk drive. Anything I create or download that doesn’t have a permanent home goes on the RAM Disk.

When I restart (once every month or two these days), all the junk just goes away, and I don’t have years worth of cruft accumulating on my system.

Lachlan

2 Likes

Hm, using an already discontinued tech?