In SpawnerCreature.java:

if (entityinsentient.cM() && entityinsentient.canSpawn()) {
    groupdataentity = entityinsentient.prepare(worldserver.D(new BlockPosition(entityinsentient)), groupdataentity);
    if (entityinsentient.canSpawn()) {
        ++l2;
        worldserver.addEntity(entityinsentient, SpawnReason.NATURAL); // CraftBukkit - Added a reason for spawning this creature
    } else {
        entityinsentient.die();
    }
    // Spigot start
    if ( --moblimit <= 0 ) {
        // If we're past limit, stop spawn
        // Spigot end
        continue label120;
    }
}

The bottom if statement should execute only if worldserver.addEntity returns true. Right now it decreases even when the number of mobs in the server didn't increase.

The moblimit variable seems to keep track of the quota for mob spawns. If the calculated hostile mob cap is 100 and there are 80 hostile mobs loaded in a world, the moblimit will be set to 20 and will decrease each time a mob is spawned (also, incorrectly, each time a mob is not spawned)

Currently if enough spawn events are cancelled, it can prevent other mobs from attempting to spawn.