Debian 7 (64-bit)
Java 8
Spigot 1.8.8

This is still occurring with 1.9.4. Sometimes we do see this in the logs, other times, nothing.
https://gist.github.com/ataranlen/33f494129d8c4689451b7910f300cdaf

I as well am having this issue: http://i.imgur.com/SKLi9c5.png Not sure what the issue is. But I hope it gets fixed soon.

@Aikar
Nope, /save-off is not the problem. At least not the savingDisabled boolean in WorldServer which is what /save-off uses.

After running the server for a while running my plugin to monitor save status, the issue still occurs even when I can guarantee world saving is enabled.

There is some sort of memory issue somewhere. Take a look at these screenshots of VisualVM:
https://gyazo.com/70925ce50fe060fc9f8961986c28600c
https://gyazo.com/8e327c98f1970e539ee61f069e93fc2a

It seems when the server runs out of Eden space, it tries to go a GC on the Eden, which only clears up about 100MB, then moves it all to OldGen. After that the OldGen is at max and does a massive GC that takes ~4+ seconds, which returns OldGen back to the size it was when it started.
I'm using G1GC

You mentioned the FileIOThread having a artificial delay. Would this cause the server to have a massive backlog of work? If so, that might be the problem.

Edit: Went ahead and created my own version of Spigot with your patch that removes the 10ms delay. This seems to have fixed the problem, however it seems odd that hundreds of region files need to be written to almost every minute.

Shevchik that is not correct.

Chunk Loading works like this:

Step 1) Chunk Unload Queue

• Filled By:

1. Chunk GC
2. Player leaving area
3. Plugin calling .unloadRequest() - chunk.unload()/world.unloadChunk(chunk) will force an immediate (unsafe) unload

Step 2) World Tick processes Unload Queue, unloading 100 per tick max

• This process is aborted if /save-off is active.
• Once a chunk is processed off this queue, it will be serialized to NBT, and added to the ChunkRegionLoader queue

Step 3) FileIOThread processes (1) Chunk per 10ms on a separate Thread.

• Pops 1 entry off ChunkRegionLoader Async Queue every 10ms
• This is really slow and a source of high memory usage. I removed this artificial 10ms delay on my server and received massive memory improvements.
• The Chunk accesses the mentioned RegionFileCache to get the reference to the RandomAccessFile, but once that chunk leaves the ChunkRegionLoader queue, the data is then written to an output stream to the file, and the output stream is closed, but the FILE HANDLE is held on to to.

So Alex, please disregard what he said. Your issue is very likely related to save off.

@Daniel
I wrote a little plugin that will monitor the status of world saving. I'll see if that is the problem.

@Shevchik
Yep, figured that one out what can I do about it? Why don't the chunks write to disk when auto-saved? Is this something made by design?

That's because server has a thing called RegionFileCache. All regionfiles are actually stored in memory until full level save (Not performed by default by anything and it is not accessible from BukkitAPi (WorldServer.saveLevel())). Level is saved only on shutdown, so... do not allow your server to crash with OOM, it will kill the chunks for sure.

I'll try to find something that is doing /save-off, but I can't imagine what it might be. I know all about saving parameters and World.save() now... I've spent the better part of an hour trawling through spigot source to find anything that might cause this.

/gc reports about 12-18k usually, which is normal given 70 players * (render distance 9 <19^2>). It's a plot server.

shutdown ignores save settings I believe.

This really sounds like /save-off, this is exactly what happens when saving is off.

Maybe something is repeatedly turning it off?

Calling save will save, but will not trigger an unload. Chunks will not unload if /save-off has been called.

Also, appears there is a bug that the forceSave parameter is ignored for World.save(). If /save-off has been called, it absolutely will not save.

I'm not sure how /gc reports chunks but it could be inaccurate. How many does it report? More than 5k would be a lot.

Your build is not the latest and therefore may be the reason you are having this issue. Spigot is 1 version(s) behind. CraftBukkit is 1 version(s) behind. This message was automatically generated and is not guaranteed to be a solution to your issue.