SPIGOT-3846 and execution of inactive ticks even for entities in lazy chunks (in vanilla a lazy entity is defined as an entity with any unloaded chunks within 32 blocks on X/Z) combines to create a situation where a villagers will cause chunk loading for all chunks within 16 blocks of their position. (SPIGOT-3874)

These chunks will never, or rarely, go away since the chunk loading is delayed from when the villager's AI is run (chunks queued for unload are unloaded, the village collection loads the chunks again based on a previous observation, the villagers get loaded again and make a new observation)

 

The exact reason behind this lies in the Entity Activation Range patch, where the vanilla behavior of not ticking whatsoever (including the logic currently in inactiveTick) any entity that has unloaded chunks nearby.

I would suggest changing it so that inactive ticks are ignored based on the criteria CraftBukkit uses (only inactive tick if every chunk within a 2 chunk radius is loaded).