User:Gaz Lloyd/using gemw

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to navigation Jump to search

This guide applies to both the RSW (including pt-br RSW) and OSRSW.

Grand Exchange prices are a very common thing to want to use, both within articles and externally. This is a guide to using them in many contexts.

This guide covers the four main areas of using GE prices: on wiki articles via templates, within a lua module, within wiki javascript, and externally. There is also a section on understanding the data structure of the price data on the wiki.

Using GEMW in wiki templates[edit | edit source]

Using these templates doesn't necessarily require any understanding of the data structure, which is why that is the next section.

Basic[edit | edit source]

The most common usage on the wiki is in an article directly, which is easily done via templates. We have two templates for simply using a GE price:

Template:GEP
{{GEP|item}}

Outputs the price of item.

{{GEP|item|number}}

Outputs the price of item times number - handy syntactic sugar to skip a normal multiplication. Number can be fractional or negative!

Template:GEPrice
{{GEPrice|item}}

Outputs the price of item with thousands separators if necessary (which will be commas as this is an English wiki).

Additional[edit | edit source]

We have a few extra templates to simplify some operations with GE prices, us them if you wish.

Template:GETotal
{{GETotal|item1|item2|item3|...}}

Outputs the sum of the prices of one of each of item1, item2, .... No limit on number of items.

Template:GETotalqty
{{GETotalqty|name1=item1|qty1=#1|name2=item2|qty2=#2|...|name20=item20|qty20=#20}}

Outputs the sum of #X times itemX for X from 1 to 20. i.e., its GETotal but you can specify the quantity of each item individually.

Supporting[edit | edit source]

These are not strictly GE price fetchers, but are useful for working with prices.

#expr
{{#expr: expression}}

The basic parser function that will calculate a mathematical expression. Will throw an error if something is not right with the expression.

eg {{#expr:{{GEP|Shortbow}} - {{GEP|Logs}} - {{GEP|Flax}}}}

formatnum
{{formatnum: number}}

Formats the number with thousands separators.

Template:fe
{{fe|expression}}

Syntactic sugar to combine #expr with formatnum to calculate and expression then format the result with thousands separators.

Template:Coins
{{Coins|expression}}

Calculates expression (with #expr) then formats the result with an image of coins and a colour representing profit/loss. Generally should be used in tables and not prose.

Template:NoCoins
{{NoCoins|expression}}

Like Template:Coins but without the image. As the image can distrupt the layout of prose, this template is preferred to Coins.

{{NoCoins|expression|c}}

Adds the word 'coins' after the result, which is also coloured.n

Understanding GEMW[edit | edit source]

Understanding the data structure is important for more technical uses of the data. There are three main pages types involved with GE prices. Every item has one of each.

It is worth noting that GEMW names are usually strictly at the item's actual name - unless it needed to disambiguated - and each GEMW name maps to only one item ID (and usually vice versa, but sometimes not as strictly as it should be).

Some examples:

Exchange pages[edit | edit source]

Exchange:Bones - osrsw:Exchange:Bones

Mostly a front end view of the data now. Historically the home of the data before the move to lua.

Exchange pages also have Semantic MediaWiki (SMW) properties set for much of the information (though intentionally nothing that actually changes often, like price). For example, Special:Browse/:Exchange:Bones (osrsw:Special:Browse/:Exchange:Bones). It is generally easier to reach browse by going to the normal Exchange page, then finding the "Browse properties" link toward the bottom of the left sidebar, under toolbox. To query this data, you can use Special:Ask; see Help:Semantic MediaWiki for information on using SMW.

Information module[edit | edit source]

Module:Exchange/Bones - osrsw:Module:Exchange/Bones

Contains all the 'static' information about the item, like value, buy limit, ID, etc. This is used by the GE bot to know what item name matches what item ID.

This is a lua module which returns a table, a data structure fairly similar to JSON (and easily parsed into it).

Prices are no longer part of this module, except for items marked with historical = true, which contain the last recorded price.

Bulk data pages[edit | edit source]

The bulk pages are several pages that contain information for every item in GEMW. The pages are what are actually loaded when you use Template:GEP or similar, with the exception of historical items which load the information module.

RSW OSRSW Purpose
Module:GEPrices/data osrsw:Module:GEPrices/data Maps GEMW name to current GE price
Module:GEPricesByIDs/data osrsw:Module:GEPricesByIDs/data Maps item ID to current item price, as an additional shorthand
Module:LastPrices/data osrsw:Module:LastPrices/data Maps GEMW name to the previous GE price
Module:GELimits/data osrsw:Module:GELimits/data Maps GEMW name to GE buying limit
Module:GEValues/data osrsw:Module:GEValues/data Maps GEMW name to item value (which determines alchemy price - every item in the Grand Exchange only)
Module:GEHighAlchs/data osrsw:Module:GEHighAlchs/data Maps GEMW name to item high-level alchemy (GE items only)
Module:GELowAlchs/data osrsw:Module:GELowAlchs/data Maps GEMW name to item low-level alchemy (GE items only)
Module:GEIDs/data osrsw:Module:GEIDs/data Maps GEMW name to item ID (GE items only)
Module:GEVolumes/data osrsw:Module:GEVolumes/data Maps GEMW name to item volume (only top 100 for RSW)

These are also lua tables. For raw text processing, they're in the format ["item"] = value,, with any " in the item name escaped as \" (I don't think there are any examples of this at time of writing, but just in case).

To parse these you can use the regular expression: \["(.*?)"\] = (\d+), where group 1 is the item name and group 2 is the item value or price. You should be able to just iterate the matches. Some of the modules will have the value being 'nil', which is the lua equivalent to null or undefined. This regex will ignore those, but you may need to account for that if you use a split-by-linebreak-and-parse method.

Exchange API[edit | edit source]

The historical data - used to generate the chart - is stored in a separate database under api.weirdgloop.org. This is stored by game and then by item ID - you will need to know the item ID of what you are looking up to get the historical prices. The easiest way is to parse the GEIDs bulk page above.

All of the APIs use the same query parameters:

Query parameters
Parameter Description Examples
id separated list of item IDs to fetch. Only the /latest endpoint supports more than one item at a time.
?id=4151
?id=4151|49430
name separated list of item names to fetch. These use the exact names as the Exchange pages on the wiki (case sensitive). If both name and id specified, only id is used. Only the /latest supports more than one item at a time.
?name=Abyssal%20whip
?name=Abyssap%20whip|Chronotes
lang A language code for use with name. Currently supported are: pt (for the Brazillian-Portuguese RuneScape Wiki) for the RS endpoint, and the default en for both endpoints (lang=en can be omitted).
?name=Chicote%20abissal&lang=pt
?name=Chicote%20abissal|Cronotas&lang=pt
Endpoints
RS URL OSRS URL Description
https://api.weirdgloop.org/exchange/history/rs/all
Examples: https://api.weirdgloop.org/exchange/history/rs/all?id=4151
https://api.weirdgloop.org/exchange/history/rs/all?name=Chronotes
https://api.weirdgloop.org/exchange/history/osrs/all
Examples: https://api.weirdgloop.org/exchange/history/osrs/all?id=4151
https://api.weirdgloop.org/exchange/history/osrs/all?name=Nightmare%20staff
Returns all prices for this item. Used to generate the full charts. Limited to 1 item per request.
https://api.weirdgloop.org/exchange/history/rs/sample
Examples: https://api.weirdgloop.org/exchange/history/rs/sample?id=4151
https://api.weirdgloop.org/exchange/history/rs/sample?name=Chronotes
https://api.weirdgloop.org/exchange/history/osrs/sample
Examples: https://api.weirdgloop.org/exchange/history/osrs/sample?id=4151
https://api.weirdgloop.org/exchange/history/osrs/sample?name=Nightmare%20staff
Returns a sample of prices - 150 prices spread approximately evenly over the entire dataset of the item. Used to generate the small charts in infobox item, where only a relatively small amount of data points are needed to get the general shape of the price graph. Limited to 1 item per request.
https://api.weirdgloop.org/exchange/history/rs/last90d
Examples: https://api.weirdgloop.org/exchange/history/rs/last90d?id=4151
https://api.weirdgloop.org/exchange/history/rs/last90d?name=Chronotes
https://api.weirdgloop.org/exchange/history/osrs/last90d
Examples: https://api.weirdgloop.org/exchange/history/osrs/last90d?id=4151
https://api.weirdgloop.org/exchange/history/osrs/last90d?name=Nightmare%20staff
Returns the last 90 days of prices, as an array of arrays, for a smaller but accurate chart of recent prices. Limited to 1 item per request.
https://api.weirdgloop.org/exchange/history/rs/latest
Examples: https://api.weirdgloop.org/exchange/history/rs/latest?id=4151
https://api.weirdgloop.org/exchange/history/rs/latest?name=Chronotes
https://api.weirdgloop.org/exchange/history/osrs/latest
Examples: https://api.weirdgloop.org/exchange/history/osrs/latest?id=4151
https://api.weirdgloop.org/exchange/history/osrs/latest?name=Nightmare%20staff
Returns the most recent price. Limited to 100 items per request.
Return types

All return types are a JSON object where the keys are the requested IDs/names, and the values are the requested data for the key.

For the former three entry points (/all /sample /last90d) the data is formatted as an array of arrays. Each inner array is either [ UNIXtimestamp, price ], or if volume is available for that timestamp, [ UNIXtimestamp, price, volume ], where all values are numbers.

For the latter entry point (/latest) the data is formatted as a JSON object, with keys id (string), timestamp (string, as an ISO 8601 format datetime), price (number), and volume (number). Volume will be null if not present. Note that calling the latest entry point for a historical item will not return a price (e.g. ID 1119 - Steel platebody (historical).

If all items you request (ID or name) are not present in the database, a JSON object with with success=false and an error message is returned, e.g. {"success":false,"error":"Item(s) not found in the database"}. If at least one of the specified items exists in the database, you will get a return of the data that exists, and the items that do not exist will not be present in the response. (Example: https://api.weirdgloop.org/exchange/history/rs/latest?id=0%7C4151 - requests ID 0 (dwarf remains, not present in the database since it is untradeable) and ID 4151 (abyssal whip, present). A success=false object will also be returned if you request too many items for an endpoint (100 for latest, 1 for the others).

Exchange API for indices[edit | edit source]

All of these entry points also support accessing the RuneScape:Grand Exchange Market Watch indices too. Simply use the ID as the name of the generator template for the index (obviously, with spaces encoded as %20, though usually your requesting library will do that automatically):

Index RS entry points OS entry points
Template:GE Common Trade Index
osrsw:Template:GE Common Trade Index
Template:GE Food Index
osrsw:Template:GE Food Index
Template:GE Herb Index
osrsw:Template:GE Herb Index
Template:GE Log Index
osrsw:Template:GE Log Index
Template:GE Metal Index
osrsw:Template:GE Metal Index
Template:GE Rune Index
osrsw:Template:GE Rune Index
Template:GE Discontinued Rare Index OSRS doesn't have a discontinued rare index

Be aware that with indices, the ID returned by latest is a string, and the majority of prices are floating points. The timestamp may not line up with the GE update timestamp, since the indices are updated right after a full GE update, using the current time, whereas GEMW items are updated with the timestamp Jagex returns.

A note on volumes[edit | edit source]

Volumes are only available in RSW for the top 100 most traded items. The volumes stored in the historical data are in millions (so 15.4 means 15,400,000 items).

In OSRSW, volumes are available for almost every item, and are not truncated (so 15 is 15).

All volumes became unavailable on around 20 November 2020, and currently are still not yet available again - though historical volumes are still in the historical data.

Using GEMW in wiki modules[edit | edit source]

Modules are lua code that runs as a layer above wikitext. Modules are generally easier to program complex things in, as lua is an actual programming language with proper variables, datatypes, loops, etc, instead of wikitext.

This guide assumes basic understanding of lua and how to use that on the wiki - see Help:Editing/Lua for more information.

Loading prices[edit | edit source]

As prices are stored in lua, there are a few ways to load them in to a module.

Recommended - Module:ExchangeLite

ExchangeLite is the recommended way to obtain GE prices. Usage is simple.

  1. First load in the module local exg = require('Module:ExchangeLite')
  2. When a price is required, it can be loaded using the exg.price(item) function, e.g. exg.price('Bones'). (This is syntactic sugar for #3).
  3. For any other field from the information module, use exg.load({args = { item, field }}), e.g. exg.load({args = { 'Bones', 'limit' }})

The primary downsides of the module are that there are no error checks and no title redirects, so you will have to implement those yourself if that is necessary. See example for error checking.

Recommended - Module:Exchange

Exchange is the full module for loaded GE prices. As it is larger, it is not recommended for larger, more complex modules that use a lot of GE prices as it can cause additional computation time and memory usage over ExchangeLite.

The important functions are (given local exg = require('Module:Exchange')):

  • exg._price(item, multiplier, format, round) - item to get the price of, multiplier for the price, format the result with commas, round the result to X decimal places
  • exg._limit(item) - GE limit for item
  • exg._value(item) - internal value for item
  • exg._diff(item) - price difference for item
  • exg._exists(item) - does the exchange module for the page exist

The Exchange module does have error protection and title redirects, unlike ExchangeLite.

Situational - loading the bulk modules directly

Loading the bulk modules directly can save a little overhead of using one of the middleman exchange modules, but you will have to make sure you handle missing prices yourself.

local prices = mw.loadData('Module:GEPrices/data')

function gep(item)
    local pr = prices[item]
    if pr then
        return pr
    end
    return 0
end

function example()
    return gep('Decorated farming urn (nr)') - gep('Soft clay') * 2
end

Wiki javascript[edit | edit source]

Wiki javascript usage of prices is essentially a subset of external usage, but with some additional advantages of being directly on the wiki. This covers methods that can only be used by being on the wiki.

Making the wiki do the loading[edit | edit source]

The easiest way to load prices with wiki javascript is to not load them with wiki javascript at all! It is simpler to, on the page the script is running on, have the wiki load the price and put it in a containing tag that can be found easily.

e.g. one could use <span class="example-calc-info" data-name="Bones" data-price="{{GEP|Bones}}">{{Coins|{{GEP|Bones}}}}</span>. This can easily be selected via $('span.example-calc-info').attr('data-price').

This can apply for both scripts that run on one page and scripts that run on large numbers of pages.

An example of this is Module:Archaeology calculator materials data (used on Calculator:Archaeology/Restoring artefacts), which loads a collection of prices and dumps them on the page for the calculator to load.

Javascript loading[edit | edit source]

If you need the javascript to load the information and can't put it into a wiki page, then there's a few ways to load the data. All the methods from #External usage also apply.

Two example usages, to get and parse current data:

// jQuery - though any standard GET request with JSON.parse will also be fine
$.getJSON('https://api.weirdgloop.org/exchange/history/rs/latest?id=4151').done(function(res, jqxhr){
    console.log(res);
});

External usage[edit | edit source]

Again, there are several ways to fetch a price, many of which are shared by the on-wiki javascript section. This should work in any language (you'll need some sort of requests library, as well as regular expressions or JSON/XML parsing), but I'll be giving examples in javascript.

Using the API[edit | edit source]

The API is the simplest and most recommended way to get prices. Just send a GET request to the endpoints described in #Exchange API above. This returns a JSON object containing prices.

The only thing we ask you to do is set a descriptive user-agent when querying our APIs. See RuneScape:APIs for more information.

Google Sheets[edit | edit source]

We have written a script for use with Google Sheets to get exchange prices. The script is here: User:Gaz Lloyd/RSW Exchange API for Google Sheets.js

  1. Open a Google Sheet
  2. Tools -> Script editor...
  3. Replace the default empty function with this script (if you already have some other custom functions, probably best to make a new file)
  4. Save the script (top bar icon, or ctrl-s)
  5. Name the project (name it whatever, it doesn't matter)
  6. The functions are now working in the spreadsheet
  7. If you reload the spreadsheet, the 'Exchange' menu will appear at the top of the page (next to Help).
    You can find some basic usage instructions and a function that generates a demo there
    Using either of these will trigger an authorisation request, as it involves modifying the spreadsheet
    The functions should work fine without it

We are making this a full Google Sheets add-on, so stay tuned!

Querying the wiki[edit | edit source]

Bulk pages[edit | edit source]

If you want the current prices, or other bulk data for a lot of items, you can request one of the bulk pages (see #Bulk data pages above) from the wiki by using action=raw, e.g. https://runescape.wiki/w/Module:GEPrices/data?action=raw. This will return the raw wikitext of the page, which you can change from the lua-format tables into your choice of data structure.

You should probably only use the bulk prices module (GEPrices) if you are looking for prices of every item. The API is likely a better fit for most uses.

Other queries[edit | edit source]

You should avoid querying the wiki for Exchange prices or volumes. The API will always be a better thing to use.

If you are looking for non-price/non-volume information, you can fetch the data using the MediaWiki API: https://runescape.wiki/api.php / https://oldschool.runescape.wiki/api.php. You are likely looking for action=query or action=parse.

Generally, if you can't find what you're looking for, come contact us and we're likely to know how to get what you're looking for. We're always happy to help!