1.16.4 java 13

Will be resolved with 1.16.5 tomorrow

One problem may be actually that BaseComponent api doesn't seem to be able to distinguish between the broken and working components?

wrong order:

TextComponent{text=, BaseComponent(color=, font=null, bold=null, italic=null, underlined=null, strikethrough=null, obfuscated=null, insertion=null, extra=[TextComponent{text=Some line in weapon lore, BaseComponent(color=, font=null, bold=null, italic=null, underlined=null, strikethrough=null, obfuscated=null, insertion=null, extra=null, clickEvent=null, hoverEvent=null)}], clickEvent=null, hoverEvent=null)}

correct order:

TextComponent{text=, BaseComponent(color=, font=null, bold=null, italic=null, underlined=null, strikethrough=null, obfuscated=null, insertion=null, extra=[TextComponent{text=Some line in weapon lore, BaseComponent(color=, font=null, bold=null, italic=null, underlined=null, strikethrough=null, obfuscated=null, insertion=null, extra=null, clickEvent=null, hoverEvent=null)}], clickEvent=null, hoverEvent=null)}

So im not sure how I would be able to detect items that have an incorrect order.

Agreed that there's no need to 'fix' existing items

This is probably something to fix inside the Bungee Chat API. From looking through the code, Minecraft seems to serialize component data in this order:

bold, italic, underlined, strikethrough, obfuscated, color, insertion, clickEvent (action, value), hoverEvent (action, contents), font, extra, text / <translate, with> / score (name, objective) / selector / keybind / <nbt, interpret, block / entity / storage>

Whereas Bungee Chat / Spigot components serialize their data in this order:

<text>, color, font, bold, italic, underlined, strikethrough, obfuscated, insertion, extra, clickEvent (action, value), hoverEvent (action, contents / value (legacy)), <translate, with> / keybind / score (name, objective, value) / selector

So this affects: text, color, font, extra
Bungee Chat API PR: https://github.com/SpigotMC/BungeeCord/pull/3018/

This affects any Spigot-specific API that allows setting display name, lore, or book pages as components. Only items modified via these APIs since the recent text related changes in CraftBukkit are affected (since previously, the Spigot components would be converted into Minecraft components, which would then be serialized via the Minecraft serializer when being applied to an item).

Another alternative would be to first convert the Spigot components into Minecraft components (by parsing the Json), and then using Minecraft's serializer to produce the Json that is being used for the item. But that would require additional back and forth conversions, which ideally should not be required.

Regarding a data fixer for existing items using this alternative ordering:
There is no strict ordering of these elements (either order is valid, they are just not equal). For Spigot's component API it makes sense to produce an order that matches Minecraft's order to avoid this sort of compatibility issue. However, players can in principal also specify the elements in a different order when they create items via vanilla commands (the Minecraft wiki, which is probably the reference for most of these players, also mentions no explicit ordering of these elements, nor does the order implicitly mentioned there match the order used by Minecraft). So a datafixer based on the order of these elements alone might break any existing items that are explicitly created with a different order and commands that rely on a specific order of this data (eg. via Minecraft's nbt target selector).

If we disregard that aspect, an option could be to add a conversion to Minecraft whenever an ItemStack is loaded (nms.ItemStack.load(NBTTagCompound)), and when an ItemStack is deserialized from a config. The conversion would essentially need to parse any potentially affected text data (display name, lore, book pages) into a component, and then serialize that component again. This would be invoked for every item, every time it is loaded again. Ideally this would be toggleable via a Spigot setting such as 'normalize-items' or similar (default disabled).

Once the serialization of Spigot components to Json is 'fixed', a plugin could also manually implement the conversion for specific items by getting the item's display name/lore/book pages as components (via the Spigot API) and then setting the display name/lore/book pages to these retrieved components.

blablubbabc