In the JavaDocs for the InventoryClickEvent, it claims that it is only fired when a player clicks a slot in an inventory, but it is fired when you click outside the inventory as well.

Current Description:

This event is called when a player clicks a slot in an inventory.

A better description of this would be:

This event is called when a player clicks in an inventory.

https://hub.spigotmc.org/javadocs/spigot/org/bukkit/event/inventory/InventoryClickEvent.html