Home > linux, management, software, wtf > The joys of btrfs and OpenSuSE – or “no space left on device”

The joys of btrfs and OpenSuSE – or “no space left on device”

At one point during the installation of OpenSuSE 12.1 on my Thinkpad I got a little adventurous and selected “btrfs” as the file system of choice. I wanted to have one large partition, but usually like /home to be apart from the rest so that I can keep all my data while doing a reinstall, upgrade or whatever. btrfs seemed a great choice to combine the two as it supports “subvolumes” which can be handled almost like own file systems. But alas, the YaST installer kindly noted that a “home” subvolume for “/” was not yet supported – so there I go again: how do you split up a rather small, albeit very fast, 128 GB SSD? I went for 20 GB for “/” and about 100 GB for  “/home”, so swap and a ext2 “/boot” (since grub supposedly can boot a kernel from btrfs, but it was not really recommended, it would furthermore allow me to crypt both “/” and “/home” later on if I wanted to). This is where trouble actually started, although I was unaware of it.

No space left on device – What GNU tools say

Together with btrfs OpenSuSE installs a tool called “snapper” which manages another feature of btrfs: snapshots. While this is actually great, it  really sucks if you don’t know that your system does them and got me into very deep sh!@$&. After working with the system for about 3 months and really enjoying it, I noticed my root partition filling up. I thought it normal, since I kept adding some software, but once Gnome keeps telling you that there is less than 1 GB left in “/” you try to do something. Luckily the pop-up with the warning also offers an inspection of the file system which showed that “/usr” was using over 8 GB of disk space. Hmm, ok – well that’s where the software lives… But why on earth is the next largest directory “/usr/share” with only 3 GB of disk usage? Where are those other 5 GB?!? First thought: Well, btrfs does inodes differently, since they are actually part of the normal file system and not set aside like in other file systems. A “df -i” actually looks like this:

juckel@denkbrett:~> df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 0 0 0 - /
devtmpfs 1012789 515 1012274 1% /dev
tmpfs 1014799 18 1014781 1% /dev/shm
tmpfs 1014799 507 1014292 1% /run
/dev/sda3 0 0 0 - /
tmpfs 1014799 12 1014787 1% /sys/fs/cgroup
tmpfs 1014799 1 1014798 1% /media
tmpfs 1014799 507 1014292 1% /var/lock
tmpfs 1014799 507 1014292 1% /var/run
/dev/sda1 38000 43 37957 1% /boot
/dev/sda4 0 0 0 - /home

But then again, 5 GB is a LOT of blocks that are used for metadata… But the whole thing was still working, and I did not have time to worry about it, until I was actually pushed into it yesterday. The whole session froze and only a very hard reboot was possible. After this one it took forever to boot and once the X-login appears the keyboard is not working. Well, a mouse is nice and a on-screen keyboard would also do the trick, but then again, no thanks. I tried to power the whole thing off, and there was my new friend for the next hour: “No space left on device”. In this case “Could not load xklavier keyboard support: No space left on device” – well that at least explains my inability to type. I need to fix this NOW! Let’s add a “1” to the grub command line to get into rescue mode. Shit is really piling up, when your runlevel 1 refuses to boot because “Cannot insmod XYZ: No space left on device”. Luckily, the second try got me a shell and I tried to get some breathing room to fix my system.

No space left on device – What’s really happening

This is where things got even more interesting. “df -h” said that I had about 350 MB left on “/”. So far so good, but then why does everything I try result in “no space left on device”? Weird… To top it all off, as I tried to get rid of some unpacked installation sources in “/root”, I got “cannot rm XYZ: No space left on device”. Now, come on – are you kidding me? You are supposed to delete the stuff and not add to it – somebody must be watching me right now and laughing his or her ass off. At some point I resigned and went to using the “btrfs” command which allows some modifications to the btrfs file system. It is actually an awesome tool since it allows to work with a mounted file system, but it is yet incomplete. What got me to the bottom of all this is the filesystem info of “btrfs”.

denkbrett:~ # btrfs filesystem show
Label: none uuid: 40507634-cb2c-4678-8b3d-d014e1b88d78
 Total devices 1 FS bytes used 20.00GB
 devid 1 size 20.00GB used 20.00GB path /dev/sda3

Hmm – somebody is not telling the truth. “df” insists that there is some space left… But since everything I see (“no space left on device”) indicates that “btrfs” is actually right and “df” might be wrong, let’s go down this path for some time. I finally am able to remove a large tar-ball lying around – that should have cleared up some room, but why is it not showing up in “btrfs filesystem show”? And wait a second, why is my home file system also showing two different numbers for usage?

denkbrett:~ # btrfs filesystem show
Label: none uuid: b3b42cba-c08e-4401-9382-6db379176a1f
 Total devices 1 FS bytes used 90.21GB
 devid 1 size 100.00GB used 85.29GB path /dev/sda4

This is weird. This is almost like on our large NAS for the home-directories of the ZIH users. BINGO! That NAS uses snapshots to keep old data blocks to be able to revert changes. I remember reading the article about btrfs in “c’t” (https://www.heise.de/artikel-archiv/ct/2011/23/174 and https://www.heise.de/artikel-archiv/ct/2011/24/190) – btrfs can also do snapshots. Maybe my OS is trying to be smart?! Let’s take “snapper” and see what’s there:

denkbrett:~ # snapper list
Type | # | Pre # | Date | Cleanup | Description | Userdata
single | 0 | | | | current |
single | 1 | | Thu Nov 10 14:00:01 2011 | timeline | timeline |
single | 607 | | Thu Dec 1 00:00:01 2011 | timeline | timeline |
single | 1381 | | Sun Jan 1 00:00:01 2012 | timeline | timeline |
pre | 1609 | | Mon Jan 9 15:13:34 2012 | number | zypp(zypper) |
post | 1610 | 1609 | Mon Jan 9 15:15:48 2012 | number | |
pre | 1611 | | Mon Jan 9 15:16:35 2012 | number | zypp(zypper) |
post | 1612 | 1611 | Mon Jan 9 15:16:49 2012 | number | |
... (lots more)

Well this is nice, my OS does snapshots from time to time and before and after a system update. Actually really nice to revert to an old state if something goes wrong. But now I need to get rid of you folks. Unfortunately a “snapper delete all” is not implemented – I have to run the command for every single one of them. Are you kidding me again? Well, let’s “fake” the “all”:

denkbrett:~ # for i in `seq 1 3656`; do snapper delete $i; done

Take this! Well almost – after getting rid of about 20 snapshots I find another known friend  (https://bugzilla.novell.com/show_bug.cgi?id=733843):

kernel BUG at
invalid opcode: 0000 [#1] PREEMPT SMP

Ok – reboot, runlevel 1 again. On the second try I get rid of all snapshots and am back to 49 % file system usage!!

Lesson learned

Don’t trust “df” or “du” when using btrfs! Keep those snapshots at bay!


To prevent this from happening again, I tried to find out how my system is deciding to do snapshots and why to keep them. A “man snapper” reveals that the snapper configs are in “/etc/snapper/configs” and sure there is a config there for “root”. But what the heck are all those undocumented parameters?

denkbrett:~ # cat /etc/snapper/configs/root
# subvolume to snapshot
# filesystem type
# run daily number cleanup
# limit for number cleanup
# create hourly snapshots
# cleanup hourly snapshots after some time
# limits for timeline cleanup
# cleanup empty pre-post-pairs
# limits for empty pre-post-pair cleanup

Some research on the web led me to this great article http://doc.opensuse.org/products/draft/SLES/SLES-admin_sd_draft/cha.snapper.html. I am now running with the following setup – let’s see how it goes:

denkbrett:~ # cat /etc/snapper/configs/root
# subvolume to snapshot
# filesystem type
# run daily number cleanup
# limit for number cleanup
NUMBER_LIMIT="20"    # let's only keep 20 snapshots around
# create hourly snapshots
# cleanup hourly snapshots after some time
# limits for timeline cleanup
TIMELINE_LIMIT_DAILY="2"  # let's only keep daily snapshots around for two days
TIMELINE_LIMIT_MONTHLY="0" # no I don't want to have things around for 10 months
TIMELINE_LIMIT_YEARLY="0"  # or even 10 years! Who came up with those defaults?!?
# cleanup empty pre-post-pairs
# limits for empty pre-post-pair cleanup

Categories: linux, management, software, wtf Tags:
  1. March 13th, 2012 at 15:15 | #1

    Without intent to start a flame war:

    That’s why I moved away from linux on the desktop 🙂

    Cool story, though, I guess you’re lucky to have been spared removing the hard drive and doing the repair on another machine.

  2. March 13th, 2012 at 15:20 | #2

    I just wished my system would tell me, what it does 🙂 I mean, how hard can it be, the snapper config for “root” comes from YaST – so the little house elfs should also be able to tell me that me disk is full because of the snapshots…

    I guess, the POSIX tools have to be extended to cope with this “shadow” disk usage somehow. Let’s see how this goes. At least it seems that everybody agrees that btrfs is the future for the Linux enviroment.

  3. March 16th, 2012 at 01:45 | #3

    @jupp: time machine eats up disk-space as well without telling the user explicitly. So your mac does not do much better (in my opinion) . . . 😉

    • March 16th, 2012 at 22:07 | #4

      @willi: Yes, but time machine (as well as the Windows7 internal backup) use an additional partition, so that they do not fill up your work file systems. While snapshoting is actually quite sweet, it has its disadvantages, too.

  4. March 20th, 2012 at 10:13 | #5

    Thanks for sharing! Do I need to restart snapper when I change the config file?

    • March 20th, 2012 at 11:23 | #6

      I did not, but then again it seems to me that the snapper is invoked via cron for the timebased snapshots on via zypper for snapshots during updates – but I also had to shut down my system shortly after the change…

  5. March 20th, 2012 at 13:36 | #7

    I replaced my superdrive with an additional hard drive that I also use as backup drive – and TimeMachine already managed to create 600GB of BackUps for my 128GB SSD the OS runs on . . .

  6. Nodrog
    May 4th, 2012 at 10:23 | #8

    Good write up! I was experimenting with btrfs on SLED 11 SP2 setup. All was going perfect for about 10 days when it just didn’t boot. I tried all the options to repair system etc. The error about no disk space didn’t make sense, but it does now…snapshots!! I was just about to wipe the system and revert to ext3 when the Google gods and Nerdyroom saved the day!
    – Gordon (Australia)

  7. Jake
    December 17th, 2012 at 16:22 | #9

    Thanks for the article, it got me moving in the right direction to solve my ‘no space left on device’ issues. However I would suggest a slightly modified approach. Rather than manually delete all the snaps, I just set the values in snapper config file as you did and then ran the snapper cleanup commands that are covered on the document you linked above in order to delete the old snapshots.

    snapper cleanup number; snapper cleanup timeline; snapper cleanup empty-pre-post

  8. Daniel
    February 12th, 2013 at 21:38 | #10

    Thanks for this post and for Jake’s streamlined solution, this saved my OpenSUSE desktop as well.

  9. robin
    May 2nd, 2013 at 21:34 | #11

    great article! is there anything I can do if I adding the 1 after the line linux …. in grub2 still does not get me to a shell (eg rescue discs or anything else)?



  10. Remus
    December 20th, 2013 at 16:26 | #12

    This is exactly what happened to me yesterday and managed to solve it with this article. Thanks!

  11. Artur
    December 26th, 2013 at 00:24 | #13

    Yupp, same here (today). Fixed the problem with this article within 5 minutes.

  12. Pingy
    December 8th, 2014 at 10:11 | #14

    Hi, you really saved me with this article! Thank you very much, have a nice day and be sure that this is still a problem with openSUSE 13.2, btrfs and a small SSD drive.

  13. gallier2
    January 4th, 2015 at 11:57 | #15

    Thank you also from me. It saved also my OpenSuSE 13.1. The thing was that I didn’t even identify the problem. My system would only refuse to start GUI. Console was still working (sort of). Only when trying an upgrade with OpenSuSE 13.2 was I surprized that the installer couldn’t even mount the btrfs partition.

  14. LinuxJack
    January 7th, 2015 at 19:28 | #16

    Just had the same problem here, with OpenSuSE 13.2. System started to crash a little at a time, first plasma-desktop, then kwin, then system locked up. I used SysRq to flush, unmount and reboot, but got no GUI login screen. I switched to a virtual terminal, logged in as root and checked dmesg, which gave no indication as to the problem. Having seen the results of a filled root partition before, this went to the top of my list of suspects. A quick check with ncdu revealed over a dozen 7+ GiB directories in the .snapshots directory. I then used snapper to delete the snapshots and the system rebooted just fine and now functions normally again. I have implemented the modified config file solutions posted here and am watching to see what happens.

  15. Kevin
    February 5th, 2015 at 07:53 | #17

    I too just visited this particular painful place, and it really came at the worst possible time, as I had a growing MySQL DB that I _really_ needed to work in order to perform that job that puts bread in the mouths of my children.

    This is a nightmare! Does anyone know how to put snapshots someplace where it won’t wedge your root directory (and, at least until I moved /var/lib/mysql to a separate ext4 partition today, your databases and the rest of your non- /home/ life)? Putting them in /.snapshots just seems like a bad joke…

  16. LinuxJack
    February 11th, 2015 at 01:50 | #18

    Hello again folks, thought you might appreciate an update…I now have one old snapshot left in the .snapshots directory that I (working as root) cannot delete! This snapshot does not appear on the list of snapshots in the snapper tool or in the YaST module. When I attempt to delete the snapshot manually using mc, I get the following error: “Cannot remove directory “/.snapshots/245~/grub2/i386-pc” Read-only file system (30)” I checked the file permissions and their all set to 755 so root should be able to delete them but can’t. I’ve since disabled snapshots all together but the snapshot is still there, choking my root (no that’s not a euphemism for something else.) The snapshot is consuming over 7 gigs of space on my root! I gotta give this one a great big WTF?! Has anyone seen this on their system or have any idea why this is happening?

  17. Michael Holopainen
    March 3rd, 2015 at 12:00 | #19


    Your loss. Growning pains of a single cutting edge distribution on using filesystem that lightyears ahead of old ones.

    At the moment of writing this, the OpenSUSE latest update has already fixed this.

    Now the Open SUSE default limit is 20.

  18. Michael Holopainen
    March 3rd, 2015 at 12:28 | #20


    You can not delete snapshots with normal file operation (using mc). They are NOT files, even though mc shows them as such. They are filesystem metadata outside of the formatted disk parttition.

    Forget snapper, use btrfs command.

    List subvolumes with

    btrfs subvolume list -s /

    then delete one at time with

    btrfs subvolume delete -c /.snapshots/NNN/snapshot

    where NNN is the number from list command output. Leaving one or two you feel might be best ones.

    Then check that space was freed with
    btrfs fi show

    One time I deleted snapshots on totally full drive there was no space freed. And running the below maintenance commands failed on “no space left of device”. I kept deleting more snapshots until the commands:
    btrfs fi sync /
    btrfs fi balance /

    succeeded. Then reboot and run
    btrfs fi defragment /

  19. March 22nd, 2015 at 22:34 | #21

    Thanks! Now I understand, similar steps from the opensuse forum solved my problem, but they are better explained here. We are learning with OpenSuse 13.2, with SystemD and btrfs.

  20. Sriram Reddy
    June 10th, 2015 at 18:10 | #22

    Sir/Bro, You gave me a reason to enjoy my drink this evening. Now I can finally get back to doing something I enjoy on my desktop. Cheers

  21. Ge,jiliang
    October 10th, 2015 at 07:30 | #23

    This morning, my opensuse13.2 didn’t boot up with the ordinary GUI interface, but the console could be used. I tried the recovery mode and found the error “no space left on device”. There is no help by moving files to free disk space. Thanks to your article, I fixed the problem according to your method. Thank you very much.

  22. Petr
    November 9th, 2015 at 20:02 | #24

    @Michael Holopainen

    Thank you!!!

  23. July 22nd, 2016 at 05:39 | #25

    Have you ever thought about publishing an ebook or guest authoring on other websites?
    I have a blog based upon on the same topics you discuss and
    would really like to have you share some stories/information. I know my
    visitors would enjoy your work. If you’re even remotely interested, feel free to send me an email.

  24. July 24th, 2016 at 04:52 | #26

    I almost never leave comments, but after reading through some of the
    comments on NerdyRoom™The joys of btrfs and OpenSuSE – or
    "no space left on device" – NerdyRoom™.

    I actually do have a couple of questions for you if it’s okay.
    Could it be only me or does it look like a few of the comments come across like they are written by brain dead individuals?
    😛 And, if you are posting on additional sites, I’d like to keep up with anything new you have to post.
    Could you post a list of all of all your public
    pages like your Facebook page, twitter feed, or linkedin profile?

  25. October 9th, 2016 at 03:13 | #27

    The link in the article to Snapper config file information has changed. If I’m not mistaken, the current URL is https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha.snapper.html

  26. February 24th, 2017 at 21:40 | #28

    You saved me with this…mad props and thank you!

  27. linux-frickler
    July 28th, 2017 at 08:51 | #29

    Please note, at least in recent versions of snapper on suse enterprise linux you cannot use “#” comments at the end of the snapper config entries. If you do that the config line will be ignored. Was a real pain in the neck to track this down…

  28. October 19th, 2019 at 00:53 | #30

    Touche. Great arguments. Keep up the amazing work.

  29. December 10th, 2019 at 13:48 | #31

    Howdy! I know this is kind of off-topic however I had to ask.
    Does running a well-established website like ours take a large
    amount of work? I’m brand new to operating a
    blog however I doo write in my diary daily. I’d lik to start a blog so
    I can easily hare my own experience and thoughts online.
    Please let me know if you have any recommendations or tips for new aspiring blog
    owners. Appreciate it!

  30. December 17th, 2019 at 11:30 | #32

    This is a very good tip especially to those
    new to the blogosphere. Simple but very accurate information… Appreciate your sharing
    this one. A must read article!

  1. April 9th, 2014 at 21:44 | #1
  2. October 24th, 2015 at 17:46 | #2
  3. March 31st, 2017 at 15:55 | #3
  4. July 16th, 2017 at 14:56 | #4
  5. July 23rd, 2017 at 05:07 | #5
  6. November 18th, 2017 at 16:20 | #6
  7. October 11th, 2019 at 04:09 | #7
  8. October 16th, 2019 at 20:33 | #8
  9. December 18th, 2019 at 21:38 | #9
  10. February 14th, 2020 at 18:03 | #10
  11. March 14th, 2020 at 12:06 | #11