[SPIGOT-2385] FileIOThread will cause chunk overwrites/data loss on bigger servers Created: 11/Jun/16  Updated: 20/Jul/16  Resolved: 12/Jun/16

Status: Resolved
Project: Spigot
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Kademlia Assignee: Unassigned
Resolution: Fixed Votes: 1
Labels: None

Attachments: PNG File 2016-06-11_12.40.08.png     PNG File 2016-06-11_16.36.53.png    
Issue Links:
Relates
relates to SPIGOT-1153 Chunks not saving correctly (memory l... Resolved

 Description   

Hey, okay this is a bigger one and took a long time to find. I am pretty sure this is a vanilla bug too (but very hard to test). I want to validate my concerns here before reporting to mojang.

Groundwork:

  • FileIOThread is the BackgroundService to save already unloaded Chunks to the specific RegionFile. This service is async
  • RegionFileCache holds a bunch of cached RegionFiles for loading/unloading data
  • ChunkRegionLoader uses RegionFileCache for different lookups/loading/unloading

Problem:
If the maximum loaded RegionFiles in RegionFileCache is reached. A clear-method is called. This call is unsafe and will result in chunk-regenerations in this specific situation.

Reproduction:
Change the RegionFileCache limit from 256 to 2. This should be enough to spam the console with saving-errors as the syncronization-code in RegionFileCache is faulty implemented.

            if (RegionFileCache.a.size() >= 2) {
                a();
            }

Reprodution of chunk-regenerations:
Best way is to change the limit to 2 as seen above

  1. Create a flat world and generate enough chunks in an area
  2. Create a normal world
  3. Copy all the regionfiles from the flat world to the normal world (dont overwrite the level.dat)
  4. Start up the server and fly around in gamemode 3.
  5. The console will be full of errors. About every 1-2 minutes there will be a newly generated chunk will appear in the previously flat area.

One way to fixing this:
As the general implementation of RegionFileCache is faulty the method "c" and "d" need to be rewritten. They are the problem as they hand out *references *to region files. Instead we can change them to hand out the NBTData directly and mark them syncronized. With this setup the syncronization actually works:

    // Kade possibly broken by FileIOThread! too
    @Deprecated
    public static DataInputStream c(File file, int i, int j) {
        RegionFile regionfile = a(file, i, j);

        return regionfile.a(i & 31, j & 31);
    }

    // Kade is broken by FileIOThread! This will return a reference to a file that may be removed before it is used!
    @Deprecated
    public static DataOutputStream d(File file, int i, int j) {
        RegionFile regionfile = a(file, i, j);

        return regionfile.b(i & 31, j & 31);
    }

    public static synchronized NBTTagCompound fixedc(File file, int i, int j) throws IOException {
        RegionFile regionfile = a(file, i, j);
		DataInputStream datainputstream = regionfile.a(i & 31, j & 31);
		if (datainputstream == null) return null; // ChunkRegionLoader loadChunk
		return NBTCompressedStreamTools.a(datainputstream);
    }

    
    public static synchronized void fixedd(File file, int i, int j, NBTTagCompound nbttagcompound) throws IOException {
		RegionFile regionfile = a(file, i, j);
		DataOutputStream dataoutputstream = regionfile.b(i & 31, j & 31);
		NBTCompressedStreamTools.a(nbttagcompound, (DataOutput) dataoutputstream); // from ChunkRegionLoader b(...)
		dataoutputstream.close();
	}


 Comments   
Comment by md_5 [ 12/Jun/16 ]

BTW this is already upstream: MC-10976

Comment by Kademlia [ 12/Jun/16 ]

Hey,
you actually do not need to load all the chunks or load them at once.
By hitting a single chunk of a new chunkarea the area will be loaded into the file-cache. We could theoretically load 256 regions by only loading 256 blocks at the correct coordinates (one block every 512x512 area).

Apart from that on a server with ~100 users the limit should be hit several times a day under normal usage. This is caused by the FileIOThread functionality.
ChunkRegionLoader has two lists (b and c). These lists contain unloaded chunks that are not yet saved to disk. FileIOThread will gradually write the data in those lists to the regionfiles. This means a single flying/teleporting user can cause several RegionFiles to be loaded. Thats why you can even hit this limit by flying alone in gamemode 3 as Loki mentioned.

Comment by md_5 [ 12/Jun/16 ]

Please refrain from commenting on tickets unless they are clearly related. It just confuses the matter.

Comment by Lokio Case [ 12/Jun/16 ]

Yes, I've seen this by flying around in spectator. No, I don't think it has to do with this specific "limit" but does have something to do with saving and loading of chunk data.

Comment by md_5 [ 12/Jun/16 ]

You can load 262,144 chunks just at once just by flying around on spectator mode?

Kademlia if we 'fix' this, it will just be bumping up the limit and sending the bug upstream.
I am curious as to what circumstances even a large server can have 262,144 chunks loaded and still be playable however?
That's 2500 players, each in their own area, with a view distance of 10.

Have you actually seen this happen?
I am failing to see a reality where this is the case.
Can you please elaborate more?

Comment by Lokio Case [ 11/Jun/16 ]

I'm running into this just flying around too fast in spectator mode.

Comment by Kademlia [ 11/Jun/16 ]

Forgot to write why this is mostly a problem on bigger servers:
In order to reach the Limit of 256 Files you need at least an area of 8192x8192 actively used

Generated at Tue Apr 08 01:38:18 UTC 2025 using Jira 10.3.3#10030003-sha1:d220e3fefc8dfc6d47f522d3b9a20c1455e12b7b.