You are here because you had encountered the error in the title. In this article I will explain the error and steps necessary to reproduce and fix it.
The most relevant documentation for inotify is the inotify man page, that you can also read locally with:
$ man inotify
inotify has three parameters, that can be set in /proc interfaces: `max_user_instances`, `max_user_watches` and `max_queued_events`. We will look into each of them separately.
$ cat /proc/sys/fs/inotify/max_user_instances
128
This specifies an upper limit on the number of inotify instances that can be created per real user ID. inotify instance is created by inotify_init (2) call and returns a file descriptor referring to the inotify instance. Usually each process (e.g. tail) creates one inotify instance. There is an example down below.
$ cat /proc/sys/fs/inotify/max_user_watches
8192
This specifies an upper limit on the number of watches that can be created per real user ID. Each “watch” in the watch list specifies the pathname of a file or directory, along with some set of events that the kernel should monitor for the file referred to by that pathname. One inotify instance can have several inotify watches, which points to different directories/files.
$ cat /proc/sys/fs/inotify/max_queued_events
16384
`max_queued_events` is an upper limit on the number of events that can be queued to the corresponding inotify instance. As one inotify instance can have several “watches” (i.e. watching potentially large amount of directories/files), not all of the events (like file change, new file etc.) can be processed immediately. In this case, the events will populate an event queue, which has limit of 16384 events by default. the event queue can overflow. In this case, events are dropped (lost).
Now, I demonstrate how max_user_instances and max_user_watches work together. For the demo I will use the following commands:
$ export ILIMIT=128
$ for((i=0;i<=$ILIMIT;i+=1)); do echo $i >> $i.tmp; done
$ for((i=0;i<=$ILIMIT;i+=1)); do bash -c “tail -f $i.tmp &”; done
$ ps aux | grep -v grep | grep tail | awk ‘{print $2}’ | xargs kill -9
The first one is simply exports ILIMIT variable to your current shell and sets its’ value to 128. The second command creates ($ILIMIT+1) files in your current working directory with names 0.tmp, 1.tmp, 2.tmp etc. The third command starts tail -f ($ILIMIT + 1) times on all the files created by the previous command. Finally the last command look for all the tail processes and kills them.
Now, try to execute the commands with the $ILIMIT set to 128:
cd /tmp
export ILIMIT=128
for((i=0;i<=$ILIMIT;i+=1)); do echo $i >> $i.tmp; done
for((i=0;i<=$ILIMIT;i+=1)); do bash -c “tail -f $i.tmp &”; done
On Ubuntu 18.04 I hit the limit on 86th instance:
tail: inotify cannot be used, reverting to polling: Too many open files
I will now increase max_user_instances to set it higher than max_user_watches:
sudo sysctl -w fs.inotify.max_user_instances=10000
$ cat /proc/sys/fs/inotify/max_user_instances
10000
Clean up your previous tail commands:
ps aux | grep -v grep | grep tail | awk ‘{print $2}’ | xargs kill -9
Now try to create 8500 tail processes, which is lower than default max_user_watches value:
cd /tmp
export ILIMIT=8500
for((i=0;i<=$ILIMIT;i+=1)); do echo $i >> $i.tmp; done
for((i=0;i<=$ILIMIT;i+=1)); do bash -c “tail -f $i.tmp &”; done
I hit the limit on 7996 and have another inotify error message now:
tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling
Now let’s increase the max_user_watches and keep it in the same proportion as in default settings. Each inotify instance should have 64 watches available:
sudo sysctl -w fs.inotify.max_user_watches=640000
$ cat /proc/sys/fs/inotify/max_user_watches
640000
Clean up tail processes spawned previously:
ps aux | grep -v grep | grep tail | awk ‘{print $2}’ | xargs kill -9
Now let’s try it again:
cd /tmp
export ILIMIT=8500
for((i=0;i<=$ILIMIT;i+=1)); do echo $i >> $i.tmp; done
for((i=0;i<=$ILIMIT;i+=1)); do bash -c “tail -f $i.tmp &”; done
And now it has been successfully executed! To make a permanent change to the max_user_instances and max_user_watches variable, simply append these lines to /etc/sysctl.conf:
fs.inotify.max_user_instances=10000
fs.inotify.max_user_watches=640000
This will persist the changes after reboot of the machine. To apply the changes immediately, you need to reload the file:
$ sudo sysctl -p
I hope it clarifies a bit why this error happens and how to fix it.