Forum:');DROP TABLE rework;
Hi guys! This thread is to discuss a large-scale effort to improve the drop tables on the wiki. A little background:
In the last year or so, we've put a ton of work into the monster drop tables on OSRS. By leveraging nearly a billion monster kills from the third-party client RuneLite, and from substantial access to the underlying game code, we've gotten to a point where about 95% of the monster articles have high-confidence exact drop rates for their entire table. These are widely used across the community, and add significant value to the wiki and the game.
Some of you may recall that the original effort to get exact drop rates actually started here, rather than on OSRS Wiki. We had a Reddit thread to get players to submit their RuneMetrics drop data (the #drop-logs channel even used to be called #runemetrics), and this gave us about 50 million kills worth of (aggregated) data that, in theory, we could use to get drop rates for a lot of stuff. However, the effort was deprioritized because of the impending fork, and minimal progress was made. When it came time to try again, it was clear that OSRS was going to be a better place to start, due to the more widely available drop data, and Jagex folks (particularly Ash) who were more open to sharing.
With the OSRS part now substantially complete, I want to return to the original task, and take a critical look at what we can do to improve the drop tables here. Taking a look at a random monster, there are three main areas of improvement I can see:
The members column[edit source]
We have a column on the drops line for whether or not the dropped item is a members item or F2P. I have a few problems with this:
- When you're on a members-only monster, as far as I can tell, knowing whether an item is members/F2P is not useful. This is the case for about 85% of the drop tables on the wiki – it takes up a lot of horizontal space but doesn't have any value.
- To get the members information, we do an SMW call in Module:DropsLine for basically every row on the drop table. I'm pretty sure these are blocking calls, and make the page *significantly* slower to load.
- In the few cases where this type of indicator might be useful (F2P monsters with members-only items in their drop table), whether or not the item itself is F2P does not always correspond to whether the monster drops it in F2P (for example, you need to be on a members world for Catablepon to drop the adamant salvage and better runes). While it's technically possible to override this behavior with a parameter to the template, I can't find any places on the entire wiki where we do it appropriately.
- Additionally, in cases where the drop tables are dependent on whether or not you're in a members world, the important thing isn't just whether the item can be dropped on F2P, but also how that weird behavior impacts the rates of other items (especially ones that you may only be able to get in F2P). This is not something you can really accomplish with the members parameter/column.
In short, I think this column makes the pages way slower to load from the server, cannot possibly be useful about 95% of the time, and even when it could conceivably be useful, we're generally not doing it right.
My proposal: do away with the column, and have a manual override parameter as seen on osrsw:Hill Giant, for cases when F2P monsters either conditionally drop things only in members worlds or only in F2P worlds. If we want, we could use miniature versions of the members/F2P icons instead of (m) and (f).
Main table sub-sections[edit source]
I actually didn't realize we did this, but in 2015 there was a thread from Mol that resulted in us combining everything from the main drop subroutine into a single table. I think the intent here was to make sure we separated out tertiary drops from main-table drops (which is good), but the end result is that we got rid of even basic organization of the main drops. A page like Hill Giant went from having having the drops organized by seeds, herbs, runes, weapons/armour, to being a wall of like 50 different items in a single table that doesn't fit on a single screen. This looks atrocious to me, and I don't understand why anyone thought it was a good idea – we lose the ability to say anything about sub-tables, make it harder to find the relevant items, and end up with similar items often being in very different places on the table. It also leads to extreme cases where we have the table on Chaos Elemental with literally 150 things in it. Does anyone think this is acceptable?
It doesn't seem to me that anyone who commented on the thread realized there was a really straightforward alternative: split the tertiary drops into a separate section, but keep the sections in place for the main drops, on the same level. See osrsw:Brine rat (or really any other article on OSRS) for an example of this. To me this is heaps better than the current appearance on RSW, and gives way more flexibility. The only argument I can see against this is that it makes it harder to sort the drop table, but I can find very little evidence this is something people do – there's not really anything useful to sort it by right now, especially because we don't have accurate rarities for most things.
My proposal: split the main drops back into separate sections for the types of drops. We don't need to do this all at once, but when we're passing over the drop tables as we add more accurate rates, we can do it as part of that process.
Accurate rates[edit source]
Here's the fun one. Let's make a real effort to get accurate drop rates for as many monsters as we can. There was some effort to do this in 2018, but it was not particularly well done – most of the "exact rates" on the wiki right now are blatantly wrong due to one person's over-reliance on powers of 2 and misunderstanding of conditional drops. We can do better.
We still have the RuneMetrics data, but I think we can do another Reddit thread and get another 100 or 200 million kills, with all the time that's passed. We're also in a much better position than two years ago, for two reasons:
One, we have a WAY better relationship with Jagex than we (publicly) did two years ago, which opens up a couple interesting possibilities. For starters, we might be able to just directly get a lot of the drop RuneScript from Jagex like we did for OSRS, but even if that doesn't pan out, I think it's probable that we could take a different route and get access to the internal RuneMetrics loot data that Jagex has collected. Both of these options are more likely if we already have something good to show for it, which we don't yet.
Two, we've already done the bulk of this same project for OSRS. This gives us a lot of awareness into how drop code is structured, what types of subroutines/subtables exist, and what types of conventions they use. But there's something even simpler than that: a ton of the drop tables actually haven't appreciably changed (besides metal salvage replacements) between OSRS and RS3. Take a look at Soldier (Yanille) and compare the drop table to osrsw:Soldier (Yanille): we can say with pretty high confidence that even though we only have about 400 kills on RuneMetrics, the data we DO have is consistent with the confirmed rates from OSRS, so it makes it a hell of a lot easier to assume it's still out of 128 and the rates are the same as before.
Drop table reworks, big or small, are pretty rare – I was able to almost perfectly reconstruct the drop rates for everything in RuneScape Classic (which doesn't even fucking exist anymore) with a few hundred kills logged, because of how minimally things had changed between games. I'm estimating that about 70% of the monsters that existed in 2007 (meaning about 45% of total monsters on the wiki now) haven't had their main drop tables changed, which gives us a tremendous starting point for looking at the RuneMetrics data.
This would be a significant effort, but I think it might be the single biggest thing we can do to improve the wiki. Are people willing to put in the time? I'm super excited to get this going. ʞooɔ 00:46, 22 May 2020 (UTC)
Support - I definately agree with the fact that drop tables (especially larger ones) would benefit from being split into subsections. When I had a smaller monitor I had to do a lot of scrolling to find what I wanted. I also strongly agree with your proposal for more accurate rates. It seems to have become practice to use arbitrary figures not backed up by any evidence.09:11, 22 May 2020 (UTC)
Support - I would prefer having the mini members icons rather than (m) and (f)22:53, 22 May 2020 (UTC)
Support All - As mainly a user of rsw rather than an editor as I am on osw, I like how we do drop table sections on osw visually, I like more >DATA<, and I like accuracy on all wikis. I don't think I'd be able to help with this project though.23:04, 22 May 2020 (UTC)
Support - When I saw that all the drop tables got merged into one table, I was taken aback and thought why we did that, I too did not know why we went with that. Having RSW's drop tables and general structures to match OSRS Wiki's would be a great thing for RSW. Now, if only the same thing could be said for Infoboxes... -- SpineTalk 23:26, 22 May 2020 (UTC)
Support + Question Definitely agree with the idea to simplify the membership situation and reorganize the drops by category, but is there a way that the entire drop list could be viewed if this is done, if someone wanted to for whatever reason? 10:09, 23 May 2020 (UTC)
- At least on OSW the split is done manually for the sections of drops so there is no way to actually join the tables back. So unless something is made to split them up automatically (which will NOT be easy) there won't be following OSW design. Jakesterwars (talk) 14:29, 23 May 2020 (UTC)
Support removing members column, support accurate rates, comment on main table sub-section, clarification questions - Overarching clarification question:
Unlike OSRSW, RSW uses the DropsLine module for several different variants, such as Template:DropsLineArch, Template:DropsLineThiev, Template:DropsLineHunt, Template:DropsLineRW, etc. I would assume this proposal is for all variants, but can you clarify if this is not the case? 14:11, 23 May 2020 (UTC)
- OSW uses variants like that too (DropsLineThieving, DropsLineHunter, Template:DropsLineReward, and DropsLine itself). Nothing changes really in their usage to my understanding on RSW, since all those do is change how they are compiled using Template:Dropping monsters list. Jakesterwars (talk) 14:29, 23 May 2020 (UTC)
- Thanks for the factual correction Jake! I've added a strikethrough for where I was incorrect on OSRSW usage. Given the specific examples in the proposal were for monsters, I wanted to confirm if the roll out of groupings and removal of F2P icons would be applied for the skilling and reward drop tables as a result of this proposal. I'm assuming that its implied in the proposal, per your comment, but I think its helpful to clarify it explicitly for anyone considering the proposal. 14:35, 23 May 2020 (UTC)
- I want to assume that they are included since that is our scope for the OSW as well in terms of splits (where possible or where it makes sense). Jakesterwars (talk) 15:11, 23 May 2020 (UTC)
1. For the members column, I agree that the pivot point for the data should be on how drops differ in F2P vs P2P, not on whether a drop is a F2P or P2P item by itself. In terms of implementation, I would prefer the use of icons for members/F2P, but another possibility would be to segment out P2P vs F2P drop tables into their own subsections (i.e. shared drop table, then subsection of P2P drop table, then subsection of F2P drop table).
- So as a note for this, I think the way we handle it on OSW is also nice for this instance. Hobgoblin shows the (f) and (m) drops very clearly without the icon with a note stating what is going on. In this case, it's because the seed drop table and herb drop table are replaced by those coins. Jakesterwars (talk) 17:27, 23 May 2020 (UTC)
2. For main table subsections - I like the current split out for tertiary drops. I recognize that having the drops organized would improve legibility compared to having information overload in a homogeneous table, but as the proposal is currently worded, existing functionality around sorting for the full table would be removed to provide this organization. A few segmented comments/questions:
- 2a. Is there a way to retain the current capability to sort by all drops for rarity and for GE Value? You mention you have very little evidence that people sort the drop table, and maybe I'm the exception, but it's something that I frequently do when looking at a monster's drops. Additionally, one of your arguments is that it's not very useful right now because of a lack of accurate rarities, but to me that means we should get accurate rarities, not remove the capability to sort. 14:11, 23 May 2020 (UTC)
- In terms of the sortability after split, the sorting will still exist as a whole for each section just not combined. If there was a way to force all headers to sort that have a certain class, that would be fun but there isn't that capability that I know of. Jakesterwars (talk) 17:27, 23 May 2020 (UTC)
- The current proposal would remove the ability to sort the main-roll drop table, but this is already a trade-off we're making by sectioning the drops whatsoever, even into Always/Main/Tertiary sections. I don't see a way to support full sorting with sections unless we do something weird like combining it into one table whenever the user tries to sort (which I don't think would match the user's expectations). ʞooɔ 19:12, 23 May 2020 (UTC)
- 2b. What are the definitions for categories? Would we be doing this based on the Exchange index? Disassembly categories? Individual common sense? I think there needs to be a policy in terms of what the standard groupings of items are. Examples: Some groups have have arrows in Equipment/Weapons, some have it in Runes & Ammunition. Also, what's the level of granularity we'd be going into on this? I wouldn't want to see a drop table with 50 different groups of 2 in it because its split on the specific type of weapon and armour dropped. TLDR: in theory this makes sense but I'd want to have a fully fleshed out policy for consistency across pages if its rolled out. This would also be dependent on the response to 2a. 14:11, 23 May 2020 (UTC)
- To be totally honest, I don't think it's a good idea to have a policy for this. While it would be easy enough to lay out a few obvious sections (runes/ammo, weapons/armour, salvage, herbs, seeds...), there are many more monster-specific cases (e.g. Brawling gloves, statuettes, Ancient Warrior armour on Chaos Elemental) that we won't possibly be able to fully encompass in a pre-defined policy. It's also not always a good idea to put something in a separate section if it's the only item of that type in the table. I don't think it's in our interest to have a rigid, restrictive policy – when we did this on OSRS, there was never really contention or significant disagreement about what the common-sense approach looked like. ʞooɔ 19:12, 23 May 2020 (UTC)
- 2c. Is there a way to implement this grouping functionality on the back end, as opposed to the front end? The way it's currently structured, we're going to have to manually change all the drop tables, which is fine. Is there an ability to tinker with the DropsLine code so that the grouping is done from the back end instead, then presented in the front end? If this were accomplished, then 2b would essentially be pre-defined on the back-end (although there should probably be a capacity for manual override). Additionally, if there were future changes then hypothetically changes in the back-end would automatically apply to all instances where it was rolled out. TLDR: Make the code sort and group for you on the back-end and allow manual override, rather than manually sort on the front end. 14:11, 23 May 2020 (UTC)
- The issue with this idea is we would need to flag all the items with a category for this to happen. Even then, each item is a separate instance of a droplines so there is no "grouping" really that they do. The top and bottom templates are what wraps them up currently. Jakesterwars (talk) 17:27, 23 May 2020 (UTC)
- I think it's better and simpler for future editors if these are actual sections, instead of some complex Lua/JS grouping thing we cook up. ʞooɔ 19:12, 23 May 2020 (UTC)
3. This portion for accurate rates boils down to - should we get accurate rates for drops? I think this answer is a pretty straightforward yes and is something that is pretty apparent that it should be done. Honestly I don't think it belongs with the other two proposals in the same forum, as it seems to be grouping a fairly certain "Yes" with two other topics that would may have more discussion. Are you expecting people to oppose it, or are you asking for a referendum on whether or not a Reddit thread should be created to crowd-source additional RuneMetrics data? Or are you just stating an intention to start this project?14:11, 23 May 2020 (UTC)
- There hasn't really been any discussion on the last point yet. There's a lot more to getting accurate drops than just saying we want them and getting agreement on it. It's not an up-or-down proposal in the classic sense, but I'm hoping to spark a discussion about the various ways we might be able to accomplish this, the feasibility of getting bulk RM data from Jagex, and how people feel about at times using OSRS rates as a beacon when we're figuring out how to fit the RM data to a simple model. It's also essentially an advertisement for what will eventually be a very large project, which needs a lot of smart people helping out. ʞooɔ 19:12, 23 May 2020 (UTC)