Forum:');DROP TABLE rework;
Hi guys! This thread is to discuss a large-scale effort to improve the drop tables on the wiki. A little background:
In the last year or so, we've put a ton of work into the monster drop tables on OSRS. By leveraging nearly a billion monster kills from the third-party client RuneLite, and from substantial access to the underlying game code, we've gotten to a point where about 95% of the monster articles have high-confidence exact drop rates for their entire table. These are widely used across the community, and add significant value to the wiki and the game.
Some of you may recall that the original effort to get exact drop rates actually started here, rather than on OSRS Wiki. We had a Reddit thread to get players to submit their RuneMetrics drop data (the #drop-logs channel even used to be called #runemetrics), and this gave us about 50 million kills worth of (aggregated) data that, in theory, we could use to get drop rates for a lot of stuff. However, the effort was deprioritized because of the impending fork, and minimal progress was made. When it came time to try again, it was clear that OSRS was going to be a better place to start, due to the more widely available drop data, and Jagex folks (particularly Ash) who were more open to sharing.
With the OSRS part now substantially complete, I want to return to the original task, and take a critical look at what we can do to improve the drop tables here. Taking a look at a random monster, there are three main areas of improvement I can see:
The members column
We have a column on the drops line for whether or not the dropped item is a members item or F2P. I have a few problems with this:
- When you're on a members-only monster, as far as I can tell, knowing whether an item is members/F2P is not useful. This is the case for about 85% of the drop tables on the wiki – it takes up a lot of horizontal space but doesn't have any value.
- To get the members information, we do an SMW call in Module:DropsLine for basically every row on the drop table. I'm pretty sure these are blocking calls, and make the page *significantly* slower to load.
- In the few cases where this type of indicator might be useful (F2P monsters with members-only items in their drop table), whether or not the item itself is F2P does not always correspond to whether the monster drops it in F2P (for example, you need to be on a members world for Catablepon to drop the adamant salvage and better runes). While it's technically possible to override this behavior with a parameter to the template, I can't find any places on the entire wiki where we do it appropriately.
- Additionally, in cases where the drop tables are dependent on whether or not you're in a members world, the important thing isn't just whether the item can be dropped on F2P, but also how that weird behavior impacts the rates of other items (especially ones that you may only be able to get in F2P). This is not something you can really accomplish with the members parameter/column.
In short, I think this column makes the pages way slower to load from the server, cannot possibly be useful about 95% of the time, and even when it could conceivably be useful, we're generally not doing it right.
My proposal: do away with the column, and have a manual override parameter as seen on osrsw:Hill Giant, for cases when F2P monsters either conditionally drop things only in members worlds or only in F2P worlds. If we want, we could use miniature versions of the members/F2P icons instead of (m) and (f).
Main table sub-sections
I actually didn't realize we did this, but in 2015 there was a thread from Mol that resulted in us combining everything from the main drop subroutine into a single table. I think the intent here was to make sure we separated out tertiary drops from main-table drops (which is good), but the end result is that we got rid of even basic organization of the main drops. A page like Hill Giant went from having having the drops organized by seeds, herbs, runes, weapons/armour, to being a wall of like 50 different items in a single table that doesn't fit on a single screen. This looks atrocious to me, and I don't understand why anyone thought it was a good idea – we lose the ability to say anything about sub-tables, make it harder to find the relevant items, and end up with similar items often being in very different places on the table. It also leads to extreme cases where we have the table on Chaos Elemental with literally 150 things in it. Does anyone think this is acceptable?
It doesn't seem to me that anyone who commented on the thread realized there was a really straightforward alternative: split the tertiary drops into a separate section, but keep the sections in place for the main drops, on the same level. See osrsw:Brine rat (or really any other article on OSRS) for an example of this. To me this is heaps better than the current appearance on RSW, and gives way more flexibility. The only argument I can see against this is that it makes it harder to sort the drop table, but I can find very little evidence this is something people do – there's not really anything useful to sort it by right now, especially because we don't have accurate rarities for most things.
My proposal: split the main drops back into separate sections for the types of drops. We don't need to do this all at once, but when we're passing over the drop tables as we add more accurate rates, we can do it as part of that process.
Here's the fun one. Let's make a real effort to get accurate drop rates for as many monsters as we can. There was some effort to do this in 2018, but it was not particularly well done – most of the "exact rates" on the wiki right now are blatantly wrong due to one person's over-reliance on powers of 2 and misunderstanding of conditional drops. We can do better.
We still have the RuneMetrics data, but I think we can do another Reddit thread and get another 100 or 200 million kills, with all the time that's passed. We're also in a much better position than two years ago, for two reasons:
One, we have a WAY better relationship with Jagex than we (publicly) did two years ago, which opens up a couple interesting possibilities. For starters, we might be able to just directly get a lot of the drop RuneScript from Jagex like we did for OSRS, but even if that doesn't pan out, I think it's probable that we could take a different route and get access to the internal RuneMetrics loot data that Jagex has collected. Both of these options are more likely if we already have something good to show for it, which we don't yet.
Two, we've already done the bulk of this same project for OSRS. This gives us a lot of awareness into how drop code is structured, what types of subroutines/subtables exist, and what types of conventions they use. But there's something even simpler than that: a ton of the drop tables actually haven't appreciably changed (besides metal salvage replacements) between OSRS and RS3. Take a look at Soldier (Yanille) and compare the drop table to osrsw:Soldier (Yanille): we can say with pretty high confidence that even though we only have about 400 kills on RuneMetrics, the data we DO have is consistent with the confirmed rates from OSRS, so it makes it a hell of a lot easier to assume it's still out of 128 and the rates are the same as before.
Drop table reworks, big or small, are pretty rare – I was able to almost perfectly reconstruct the drop rates for everything in RuneScape Classic (which doesn't even fucking exist anymore) with a few hundred kills logged, because of how minimally things had changed between games. I'm estimating that about 70% of the monsters that existed in 2007 (meaning about 45% of total monsters on the wiki now) haven't had their main drop tables changed, which gives us a tremendous starting point for looking at the RuneMetrics data.
This would be a significant effort, but I think it might be the single biggest thing we can do to improve the wiki. Are people willing to put in the time? I'm super excited to get this going. ʞooɔ 00:46, 22 May 2020 (UTC)
Support - I definately agree with the fact that drop tables (especially larger ones) would benefit from being split into subsections. When I had a smaller monitor I had to do a lot of scrolling to find what I wanted. I also strongly agree with your proposal for more accurate rates. It seems to have become practice to use arbitrary figures not backed up by any evidence.09:11, 22 May 2020 (UTC)
Support - I would prefer having the mini members icons rather than (m) and (f)22:53, 22 May 2020 (UTC)
Support All - As mainly a user of rsw rather than an editor as I am on osw, I like how we do drop table sections on osw visually, I like more >DATA<, and I like accuracy on all wikis. I don't think I'd be able to help with this project though.23:04, 22 May 2020 (UTC)
Support - When I saw that all the drop tables got merged into one table, I was taken aback and thought why we did that, I too did not know why we went with that. Having RSW's drop tables and general structures to match OSRS Wiki's would be a great thing for RSW. Now, if only the same thing could be said for Infoboxes... -- SpineTalk 23:26, 22 May 2020 (UTC)
Support + Question Definitely agree with the idea to simplify the membership situation and reorganize the drops by category, but is there a way that the entire drop list could be viewed if this is done, if someone wanted to for whatever reason? 10:09, 23 May 2020 (UTC)
- At least on OSW the split is done manually for the sections of drops so there is no way to actually join the tables back. So unless something is made to split them up automatically (which will NOT be easy) there won't be following OSW design. Jakesterwars (talk) 14:29, 23 May 2020 (UTC)
Support removing members column, support accurate rates, comment on main table sub-section, clarification questions - Overarching clarification question:
Unlike OSRSW, RSW uses the DropsLine module for several different variants, such as Template:DropsLineArch, Template:DropsLineThiev, Template:DropsLineHunt, Template:DropsLineRW, etc. I would assume this proposal is for all variants, but can you clarify if this is not the case? 14:11, 23 May 2020 (UTC)
- OSW uses variants like that too (DropsLineThieving, DropsLineHunter, Template:DropsLineReward, and DropsLine itself). Nothing changes really in their usage to my understanding on RSW, since all those do is change how they are compiled using Template:Dropping monsters list. Jakesterwars (talk) 14:29, 23 May 2020 (UTC)
- Thanks for the factual correction Jake! I've added a strikethrough for where I was incorrect on OSRSW usage. Given the specific examples in the proposal were for monsters, I wanted to confirm if the roll out of groupings and removal of F2P icons would be applied for the skilling and reward drop tables as a result of this proposal. I'm assuming that its implied in the proposal, per your comment, but I think its helpful to clarify it explicitly for anyone considering the proposal. 14:35, 23 May 2020 (UTC)
- I want to assume that they are included since that is our scope for the OSW as well in terms of splits (where possible or where it makes sense). Jakesterwars (talk) 15:11, 23 May 2020 (UTC)
1. For the members column, I agree that the pivot point for the data should be on how drops differ in F2P vs P2P, not on whether a drop is a F2P or P2P item by itself. In terms of implementation, I would prefer the use of icons for members/F2P, but another possibility would be to segment out P2P vs F2P drop tables into their own subsections (i.e. shared drop table, then subsection of P2P drop table, then subsection of F2P drop table).
- So as a note for this, I think the way we handle it on OSW is also nice for this instance. Hobgoblin shows the (f) and (m) drops very clearly without the icon with a note stating what is going on. In this case, it's because the seed drop table and herb drop table are replaced by those coins. Jakesterwars (talk) 17:27, 23 May 2020 (UTC)
2. For main table subsections - I like the current split out for tertiary drops. I recognize that having the drops organized would improve legibility compared to having information overload in a homogeneous table, but as the proposal is currently worded, existing functionality around sorting for the full table would be removed to provide this organization. A few segmented comments/questions:
- 2a. Is there a way to retain the current capability to sort by all drops for rarity and for GE Value? You mention you have very little evidence that people sort the drop table, and maybe I'm the exception, but it's something that I frequently do when looking at a monster's drops. Additionally, one of your arguments is that it's not very useful right now because of a lack of accurate rarities, but to me that means we should get accurate rarities, not remove the capability to sort. 14:11, 23 May 2020 (UTC)
- In terms of the sortability after split, the sorting will still exist as a whole for each section just not combined. If there was a way to force all headers to sort that have a certain class, that would be fun but there isn't that capability that I know of. Jakesterwars (talk) 17:27, 23 May 2020 (UTC)
- The current proposal would remove the ability to sort the main-roll drop table, but this is already a trade-off we're making by sectioning the drops whatsoever, even into Always/Main/Tertiary sections. I don't see a way to support full sorting with sections unless we do something weird like combining it into one table whenever the user tries to sort (which I don't think would match the user's expectations). ʞooɔ 19:12, 23 May 2020 (UTC)
- 2b. What are the definitions for categories? Would we be doing this based on the Exchange index? Disassembly categories? Individual common sense? I think there needs to be a policy in terms of what the standard groupings of items are. Examples: Some groups have have arrows in Equipment/Weapons, some have it in Runes & Ammunition. Also, what's the level of granularity we'd be going into on this? I wouldn't want to see a drop table with 50 different groups of 2 in it because its split on the specific type of weapon and armour dropped. TLDR: in theory this makes sense but I'd want to have a fully fleshed out policy for consistency across pages if its rolled out. This would also be dependent on the response to 2a. 14:11, 23 May 2020 (UTC)
- To be totally honest, I don't think it's a good idea to have a policy for this. While it would be easy enough to lay out a few obvious sections (runes/ammo, weapons/armour, salvage, herbs, seeds...), there are many more monster-specific cases (e.g. Brawling gloves, statuettes, Ancient Warrior armour on Chaos Elemental) that we won't possibly be able to fully encompass in a pre-defined policy. It's also not always a good idea to put something in a separate section if it's the only item of that type in the table. I don't think it's in our interest to have a rigid, restrictive policy – when we did this on OSRS, there was never really contention or significant disagreement about what the common-sense approach looked like. ʞooɔ 19:12, 23 May 2020 (UTC)
- In that case, I would say lay out the obvious groupings and make it clear that its UCS for the remainder. Similar to how we have in RuneScape:Style_guide/Layout#Order_of_sections. 13:07, 30 May 2020 (UTC)
- 2c. Is there a way to implement this grouping functionality on the back end, as opposed to the front end? The way it's currently structured, we're going to have to manually change all the drop tables, which is fine. Is there an ability to tinker with the DropsLine code so that the grouping is done from the back end instead, then presented in the front end? If this were accomplished, then 2b would essentially be pre-defined on the back-end (although there should probably be a capacity for manual override). Additionally, if there were future changes then hypothetically changes in the back-end would automatically apply to all instances where it was rolled out. TLDR: Make the code sort and group for you on the back-end and allow manual override, rather than manually sort on the front end. 14:11, 23 May 2020 (UTC)
- The issue with this idea is we would need to flag all the items with a category for this to happen. Even then, each item is a separate instance of a droplines so there is no "grouping" really that they do. The top and bottom templates are what wraps them up currently. Jakesterwars (talk) 17:27, 23 May 2020 (UTC)
- I think it's better and simpler for future editors if these are actual sections, instead of some complex Lua/JS grouping thing we cook up. ʞooɔ 19:12, 23 May 2020 (UTC)
Neutral for 2 - I think we're taking one unified system with flaws and are replacing it with a different unified system with different flaws. While I think there should be a system that addresses both flaws, as long as we are consistent across all pages with whatever flawed system is in place. Thus neutral.13:07, 30 May 2020 (UTC)
3. This portion for accurate rates boils down to - should we get accurate rates for drops? I think this answer is a pretty straightforward yes and is something that is pretty apparent that it should be done. Honestly I don't think it belongs with the other two proposals in the same forum, as it seems to be grouping a fairly certain "Yes" with two other topics that would may have more discussion. Are you expecting people to oppose it, or are you asking for a referendum on whether or not a Reddit thread should be created to crowd-source additional RuneMetrics data? Or are you just stating an intention to start this project?14:11, 23 May 2020 (UTC)
- There hasn't really been any discussion on the last point yet. There's a lot more to getting accurate drops than just saying we want them and getting agreement on it. It's not an up-or-down proposal in the classic sense, but I'm hoping to spark a discussion about the various ways we might be able to accomplish this, the feasibility of getting bulk RM data from Jagex, and how people feel about at times using OSRS rates as a beacon when we're figuring out how to fit the RM data to a simple model. It's also essentially an advertisement for what will eventually be a very large project, which needs a lot of smart people helping out. ʞooɔ 19:12, 23 May 2020 (UTC)
Support all - I had concerns about dividing the main drop table and keeping the full table rarity sort functionality but it doesn't look like there's a simple way to achieve this. Habblet (talk) 08:13, 25 May 2020 (UTC)
Oppose 2 - no comment on 1/300:18, 28 May 2020 (UTC)
Oppose 2, Neutral 1&3 - Splitting drops into a bunch of subsections rather than 1 main table and 1 tertiary table makes it significiantly more difficult to sort the drops list. As-is, if you wanted to sort the drops by price, you only have to sort the main and tertiary tables which is far nicer than having to manually look through all the subtables to try and piece together the list in the order you want. As for 3, some of the rarity amounts on osrsw:Hill Giant just look like a mess (the gem table especially), mainly because of the decimal point, comma/thousands separator, and multiple rarity amounts in a single line; I'd oppose making the rarity that accurate, sacrificing quick and easy readability unless you are able to change that in the dropstable settings to get rid of the thousands separator and round off the deminals (which you can't currently in the osrsw drops tables). Californ1a (talk) 12:01, 31 May 2020 (UTC)
- Why are you wanting to sort the entire table in the first place? The only reason I can think of is finding the rarest or most expensive item in 1 sort, which you can't do anyways since tertiary and rdt has to be split. That's also just not a common use, I don't think I have ever looked at a drop table with the intention of finding the most expensive or the rarest item, that information is practically useless, I've almost always been looking for the rarity of a specific item I just got, which is easiest with Ctrl+F, and if not using Ctrl+F, is easiest by having different categorized sections such as Weapons & Armour, Runes & Ammo, etc. 23:09, 31 May 2020 (UTC)
- Two comments on your response.
1. While the current system doesn't allow users to do it in "one sort" now, it does allow it with two to three sorts, which is significantly less than what the result would be if 2 were implemented, where there might be a dozen different categories. I don't think the argument is that the current system is perfect, but that the new proposed system would greatly exacerbates an existing flaw. The relative importance of the flaw is subjective, but I think the existence of it and how it would be affected by the proposal is pretty clear cut. 2. I think a lot of the arguments for 2 are centered on the assumption that RSW users don't utilize the functional portion that is flawed in this proposal, which is why it's ok to proceed with it. We've got some anecdotal evidence that some RSW users do in fact use this functionality, based on testimonials in this thread. Do you have RSW statistics showing that RSW users do not use this functionality? Otherwise I don't think your assumption implied on usage in here stands.00:32, 1 June 2020 (UTC)
- In my opinion, the flaw is that you can't sort the entire drop table at once. That isn't a flaw with the proposed system, it's an existing flaw that'd be exacerbated and has to exist otherwise we lose both accuracy and information. However, that flaw is only a "flaw" for 1 use case that I can imagine. Based on my own experiences, I don't think that use case is common and nobody here has said that they use sorting for that use case, they've just said that they use sorting. If people don't use it for that use case, then I can't imagine why people would want to sort by rarity or value in a single table, hence why I asked why Californ1a would want to use it. I've asked for data in discord but Cook only has data for OSW around a year ago where "the usage was virtually nonexistent". Although I believe his data was for any sorting and not solely the rarity/value columns as the use case I described was about. 01:51, 1 June 2020 (UTC)
- As a few others have mentioned below before I had time to reply, sorting the table by price is useful for putting all the non-trash drops right at the top of the list, so you can very easily see the top 10 or or drops that you'll want to pick up from that mob, especially if you've never been to it before. It's not necessarly just looking for the #1 most expensive item in the sort, but rather the top x amount of items using that sort so you can easily weed out the trash drops. I'd be fine with Oshtur's middground solution though, only splitting the main drops table when it's overly large, on a case-per-case basis, rather than splitting into every minimal category. Californ1a (talk) 07:55, 3 June 2020 (UTC)
Neutral 1, Oppose 2, Support 3
- This information is potentially valuable, because players may not always have membership, or may sell to F2p players. But, this isn't the only place this information can go, and may not be necessary for the drop table itself. I disagree with the reasoning but I'm fine with the end result.
- See User talk:Californ1a's comment above. Overall I don't like the Tertiary and Universal drops being separate tables or subtables is a great way to do it, except for drops that remove themselves from the drop table in certain conditions (slayer-only, lore books, etc.) I think we can do something with this (such as colorizing or moving it into its own template like Template:DropsLineSpiritGems) but I don't think subsections or multiple tables is the best way forward. To your question about large tables, adding a collapsable class is the solution we used for recommended perks. I think it would be apropriate here too.
- More accurate drop rates are a good idea, and taking the lessons learned from last time to do it even better this time is awesome. If we could get all drop tables to be actual fractions or percentages instead of just "Rare" I would consider that an excellent and useful table, regardless of how it's structured.
- Collapsing tables is rarely (if ever) the right solution to a problem. We reverted the change to recommended perks precisely for this reason -- people very frequently can't tell the difference between something missing and something being collapsed. ʞooɔ 22:57, 31 May 2020 (UTC)
- And to be clear, you're proposing that we actually merge the existing drop tables further, combining tertiaries and 100%s into the main table? While this is the logical conclusion that the opposing arguments to #2 eventually come to, this cannot possibly be a good idea. It's literally just a reduction of information. ʞooɔ 23:01, 31 May 2020 (UTC)
- Oppose collapsed drop tables per Cook. Oppose removal of the tertiary/main/universal drop information that's currently communicated based on the current heading grouping, however it doesn't necessarily need to be in the same format it currently is. Hypothetically, you could put this information into a separate column into a single combined table and retain the information while still achieving the combined table. I'm not saying that's the right way to communicate that information, but merely pointing out that a combining tables doesn't necessarily have to lead to a reduction of information. 00:32, 1 June 2020 (UTC)
Support 1, Oppose 2, Support 3
- I agree that this is not necessary in most cases as if you are looking at the drops of a members mob, the member's status is not an important aspect especially if it affects the load time. I also agree that for the cases where they overlap, F2P and Members drops should be stated as this is relevant
- 'I can find very little evidence this is something people do' seems subjective and not logic based like your other points; just because you don't use this doesn't mean that it isn't extremely useful to those that do. It is hard to quantify what is used based off of a page load from someone loading a page to read it - for example to find the location- or someone actively looking for the drop rates. And as stated previously this is based off of old data from OSW this makes this less justifiable. From a logic point of view, if the game is using the drop table for a mob that any of the following drops could drop randomly on this drop table, further separating them adds more clutter to the page and makes it harder for someone to find what they are looking for based off of most expensive drops or finding the drop by name. Perhaps there is a middle ground that allows for sorting all drops and also sorting them by category if that is what the user wants.
- This is rather easy; strong support. From adding drops myself to seeing previous drop tables, it is extremely hard to know whether the person making the table has good/bad rates for that item or whether it is truly accurate - however either of these seem more user friendly than leaving all values as Unknown. With the goal to make the wiki the best and most accurate place for game information, the drops from mobs seem to have been eyeballed for too long. Regardless of the time it takes to gain all of the required information, if the framework is in place to allow all drops to be accurately managed this would be a huge step. --
Sevto 14:44, 2 June 2020 (UTC)
- I have started tracking usage of sortable tables on the wiki, and will report back in about a day. Early returns for drop tables are exceedingly underwhelming. ʞooɔ 19:15, 2 June 2020 (UTC)
Support 1 and 3, Oppose 2
1. I'd definitely like to see separate drop tables for p2p and f2p, though I'm wondering whether this could be accomplished simply by using tabbed drop tables. The p2p tab would be main by default; if you want to see what drops on f2p worlds and adjusted drop rates, you select that tab. Similar functionality already exists, so implementation from a coding standpoint should not be an issue. Filling in all the new tables will take a bit longer.
2. I feel there are relatively few cases where the main drop table is so large it needs to be divided. In those cases where it is very large (the example of the Chaos Elemental was brought up), that can be handled in a one-off fashion as appropriate. In particular, since the Ancient Warriors' drop table is already a sub-table, it should certainly be divided out. But the main drop table has two types of missile weapons, one salvage item, and crafting item - these four items don't need to take up three different tables.
2a. Sorting - I don't use it often, but there are times (mainly to separate member and non-member items). And I may not want to know the most valuable item, but being able to identify the 10-12 most valuable noted drops from aberrant specters with one click is nice. In addition, I think sorting could be used in place of separate drop tables - add an "item type" column and you can see all the herbs in one spot on the main table without needing a subtable for herbs.
Support - For proposal 1, I like the idea of a mini-members icon. For proposal 2, I think I always preferred the tables split up into categories since it's easier to see what types of drops you can actually get, and it's easier to look through those categiries. While it is nice to see what the most valuable drop from a given mob is (and that's generally what I use drop tables for, not to see the rarity of any drop I've already received), it's not terribly hard to do so without needing to sort the table. In addition, so far it seems that only an incredibly small portion of visitors actually do sort tables and, if that's the case, we shouldn't cater specifically to them if doing so stops us from more efficiently organizing the pages. Too many toggles can get annoying, but we could probably do that if some users still want a combined table. ɳex undique 17:33, 3 June 2020 (UTC)
Data - I started tracking table sorting on the wiki via MediaWiki:Gadget-trackSortable.js, about 24 hours ago. In that time, we had approximately 120,000 visits to monster articles with drop tables. To the best of my knowledge, we had a grand total of 71 visits that sorted one of the drop tables -- about 1 in every 2,000. I'm not finding any evidence that this is a common use case.
I would be willing to compromise to have an opt-in toggle that can combine the tables into one (and note that this is technically much simpler to do in this direction, rather than the other way). But I think it's quite silly to let such an uncommon action determine the default behavior for the other 99.95% of visits. ʞooɔ 18:02, 3 June 2020 (UTC)
Support opt-in toggle - Default state of uncombined seems appropriate given the relative frequency of use and technical feasibility, and allows those who do use the full sort to still have that functionality. Thanks for reporting the data on this!18:55, 3 June 2020 (UTC)
Support All, especially Accurate Rates! - Having drop rate groups of common, uncommon, rare, etc. are good enough for the casual player. However, with the maturing player base, I would image that many players that are looking at the drop rates would be able to utilize accurate rates - especially for rarer/unique items.
Closed - There is consensus to implement removing the members parameter as well as displaying accurate drop rates. There is not a consensus to separate the drop tables into sub-categories. The opt-in sections can be implemented if it's critically important to the project as a whole. --LiquidTalk 02:18, 14 June 2020 (UTC)