Instant boot Linux?

submitted by

Hi all,

Is there a method out there that can “save a hibernate state” to disk, and just permanently boot immediately into that state every time?

It would be OK to lose data used after this point. And ot would also be OK if it required building a custom image based on the hardware, similar to embedded systems.

The goal is to be able to, as quickly as possible, spin up a GPU server when needed, but without using suspend/resume.

Thanks.

21
33

Log in to comment

Hot Top New Old
edited

I was really curious about the possibilities here for a re-usable “saved hibernation state”, and did a lot of digging around and testing in exploration of that idea. Please note that the following is wacky as hell and is not a good way to run a server at all, but is possible:

  1. Set up a swap partition that can fit your full system memory.
  2. Enable swap and set up hibernation (https://wiki.archlinux.org/title/Power_management/Suspend_and_hibernate#Hibernation )
  3. Hibernate your system to the swap partition in the state you want to use as your perpetual checkpoint.
  4. Boot from a live usb, NOT your regular system (when the linux kernel boots with the resume=uuid=xxxx parameter which enables hibernation/resume, it checks that partition for a hibernation signature and immediately clears it, preventing multiple boots of the same hibernated state. You need to preserve this signature).
  5. From the live usb system, copy the swap partition that contains your main system’s hibernated state to an .img file on a filesystem mounted from one of your main disks.
  6. Reboot into your main system.

  7. Set up a scheduled job that triggers every time your system resumes (could use a systemd service with After=hibernate.target and WantedBy=hibernate.target) that does the following:

    1. Disables swap on the swap partition you set up (keeping the resume=uuid=xxxxx kernel parameter that points to it).
    2. Writes the .img file to that partition

  8. Use your system as you want, and power it off when done (do not suspend or hibernated again).

The next time you boot, the kernel will load the saved hibernation state from the partition specified in the resume= parameter, as if it had just been hibernated instead of halted.

Now for the downsides:

  • It’s very dangerous to resume a system’s memory to one state with storage in a different state (the official kernel docs for this have a giant red warning banner saying “If you touch anything on disk between suspend and resume… …kiss your data goodbye.” https://docs.kernel.org/power/swsusp.html ).
  • Resuming from hibernation isn’t really that much faster than cold booting a lean system, if speed is your primary goal.
  • The scheduled write job will hit you with a bunch of heavy io every time you boot/resume, probably negating any speed benefits of the whole thing anyway.

Anyway, now that we know it’s possible, if you really can’t suspend/resume in a non-wacky way I recommend instead just booting normally and trying to make the boot process as fast as possible. A good guide here: https://wiki.archlinux.org/title/Improving_performance/Boot_process

If booting quickly to a static state is the goal, alpine diskless mode might be a good option: https://wiki.alpinelinux.org/wiki/Diskless_Mode

 reply
20
OP

First, that’s super smart. I’ve got hibernate to LUKS in LVM working already, so saving the state males sense. However, speed of resume being the issue doesn’t he’ll much, to your point.

The Diskless Alpine looks very interesting. Do you know if there is a Debian version?

 reply
4
edited

Pretty sure you just have to enable hibernation and give it a file to write to? Then just set your PC to go into hibernate on idle

I kind of doubt you would have to change distros. What distro are you on now?

 reply
12
OP

Oh hibernate to LUKS I have working. What I want is a permanent static version of that. I.E. boot to the same spot every time. This way I’m not writing 64GB to disk every time.

 reply
1
edited

For my two systems (pc and laptop)with 32GB RAM each it takes ~4 minutes to boot from hibernation.

Which is expexted afaik.

Thats the opposite of “instant”.

 reply
0
edited

Firstly, suspend might be an option compared to hibernation for your subsequent startups if you can suffer a slow startup the very first time.

Otherwise look up “Linux from scratch” .

A lot of boot time is decompressing and mounting initrd, loading modules there, mounting the root filesystem, jumping to that.

Build a kernel that has exactly the minimum modules needed for your system, skip initrd and jump straight to a root filesystem and call your init binary (it can be BusyBox even) do exactly what’s needed there to get your network+graphics card up and running with a single shell script, done.

Also look at COW filesystems. Read only base with an overlay file for writes. Discard that file on each boot and you basically have a system that is always at the same state on startup.

 reply
3
OP edited

Yea, I was hoping to not have to go the scratch route, but we shall see. I do already run ZFS.

Edit: And I’m using suspend now, but am unhappy without encryption at rest. Also, full memory GPUs don’t suspend very quickly.

 reply
1

Baring post, my pc cold boots in 3 seconds. And does a full shutdown in 3 seconds as well.

Unfortunately my mobo’s slow to post so that takes around 5-6 seconds.

I moved from arch Linux to void and its init system called runit is amazing. Mega simple and fast.

 reply
2
OP

Interesting. Did you do any tuning or out of the box?

 reply
1

Out of the box. Void is a very Diy distro though, so don’t expect to install it and be up and running. But I have a fully functional pc with sway / Wayland, steam and video editing, so no compromises. I was previously on arch Linux for 15 years and tuned it as hard as I could and couldn’t get anywhere near this boot time.

 reply
3

Are we talking physical servers like Poweredge or Proliant? Those take a long time to boot, especially with lots of RAM. Why can’t you use suspend/resume?

And modern servers don’t really consume a lot of power while idle. You can configure them to downclock the CPU and GPU while idle just like a desktop or laptop.

But if you really wanted it, you could boot an immutable distro and mount user partitions tmpfs. Or use a portable distro and discard the state. Or a filesystem with snapshots and roll back on each shutdown or boot. But none of those are really going to get you instant boot. Linux boots pretty quick, but not instant.

 reply
4
edited

Linux boots pretty quick

I never managed to make linux boot faster than windows (on the same device ofc).

And to this day I don’t understand why.

Out of the 3 desktop OSes (lin, win, mac) linux was always the slowest one in my experience.

 reply
1
OP

Fast boot most likely. Windows can shut down user processes and then hibernate the kernel/system processes. Because the system only changes on updates, it doesn’t have to wrote the hibernate file every time, only on update. Never understood why distros didn’t look to replicate that. If they had this would probably be a non-issue for me.

 reply
2
OP edited

Desktop with GPUs. I’ve tried immutable, but that really doesn’t impact boot speed. It’s still the same startup procedure. And I use ZFS, so I could roll back (or use tmpfs), but that still requires the full boot process.

 reply
1

What’s your use case or broader goal? In some specific cases you might be able to use a VM instead of a bare metal server. The host would be idling all the time, but the VM can start from a snapshot near instantly

 reply
1
OP

Power saving with encryption at rest. I want the GPUs offline power wise.

 reply
1
edited

Just spitballing here, but if you use a VM with gpu passthrough, the host system doesn’t even see the GPU anymore, so if the VM is off I imagine the GPU is basically off as well. In terms of encryption at rest, you can probably make a separate LUKS volume for the VM disk image, and then lock it when the VM is off.

Edit: for example I have a VM with GPU passthrough and when the VM is off, I have no way to check the power consumption of the GPU (aside fom using external hardware) since the host doesn’t have access. The host uses a sort of dummy driver for the GPU, so that the hypervisor can pass it on to the VM, so it is actually possible for the host to switch the driver and take control of the GPU when the VM is off. But until then I imagine power consumption is minimal.

 reply
1
OP

I used to do this for my virtual gaming rig, and the catch is still idle draw. Just one B70 pulls over 30W idle, then there’s the nvidia card and then CPU etc, too. As this machine is solely for remote development and inference, I want it fully off.

Now, I guess I do have all the confidential stuff in encrypted ZFS, so I can shutdown the docker containers, unmount ZFS, and suspend. This gives me encryption at rest. The problem is the startup/resume is actually slower than you might think.

 reply
1

You might check out coreboot and PCs that support it https://www.coreboot.org/end_users.html

 reply
3
OP

Interesting, I don’t have this hardware now, but will track this one.

 reply
2
Insert image