Forum:Bot specification: PNGOptimisationBot

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
Forums: Yew Grove > Bot specification: PNGOptimisationBot
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 15 November 2011 by Liquidhelium.

I am planning to make a periodically-running bot to optimise PNG images on the wiki, triggered by additions of File: links to a page under the bot user's page hierarchy. I would like some input from the community.

Here are the specifications:

Proposal 1

  1. The bot would be written in Java.
  2. Using the relevant MediaWiki API, Get Page History, the bot would periodically (e.g. every 1 hour, discussion is needed on this time interval) read its own userpage or a page under its page hierarchy (for example, User:PNGOptimisationBot/Requests) and note the edits that have been made to it after the last time it was read.
  3. A configuration file at $HOME/.pngoptbot.conf would contain the username and password to be used by the bot to authenticate to the wiki, as well as the revision ID of the last request seen on the bot's request page.
  4. For each image: Using the relevant MediaWiki API, Get Image History, the bot would read the name and revision number of the latest version of the image file on the Wikia server, then, if it has not seen that revision, it will download it and cache it, verify that it's a PNG file using its headers, and proceed to optimisation. If the file does not exist on the wiki (i.e. it would be a red link), a notice is posted on the requester's talk page. This notice may be a semi-protected template for ease of change in the future. If the revision number is the one last seen by the bot, nothing is done.
  5. Optimisation would be carried out by 2 tools, OptiPNG and AdvanceCOMP, in the user's program search path (the $PATH environment variable). Both are open-source tools, and using AdvanceCOMP's AdvPNG tool after OptiPNG can reduce file sizes further by 10%.
  6. If an error occurs during optimisation, a notice is posted on the requester's talk page. (Can be a template, etc.) Otherwise, a success notice is posted instead. (Discussion on whether the notice is needed in this case is welcome.)
  7. The resulting images are re-uploaded on Wikia using the relevant MediaWiki API, Upload to File namespace, with an edit summary of "(Semi-automated) PNG recompression requested by [[User:Requester|Requester]] ([[User talk:Requester|talk]] | [[Special:Contributions/Requester|contribs]])" if the optimisation removed at least X% (configurable in .pngoptbot.conf) of the original file's size. Otherwise, nothing is done, not even posting notices to the requester's talk page, but the latest version is kept in cache for the next run of the bot. This is to prevent the bot from spamming talk pages with endless notices in case its request page is vandalised. Duplicate File: links in a single user's edit may also trigger a spam filter which would ignore the entire request or deduplicate it.
  8. Source code will be posted.

Has such a bot been made already with the ability to read its user page for requests? If not, any thoughts?

 a proofreader ▸ 

11:23, November 11, 2011 (UTC)

Proposal 2

As discussed with Cook on IRC, perhaps the bot could be made to do this for all files on the wiki once when it first starts, then do the optimisation on all of the wiki's new files, as determined by the relevant MediaWiki API call, List All Pages, and new revisions of images every week. The request system would still exist in case someone wants to speedily have some images recompressed if they modify them (e.g. antialiased upload or upload with transparency added) and it appears on often-visited pages, like for new content in RuneScape.

The edit summary for weekly optimisation would be "(Automated) PNG recompression". It would make heavy use of the revision number cache to make fewer downloads.

In summary, Proposal 2 is for a request-page bot that also does the whole wiki once a week (and uploads images back only if they're X% smaller).

 a proofreader ▸ 

11:23, November 11, 2011 (UTC)


Comment - There are some bots that compress & crop images, but I don't think they're done by reading page requests. In any case, this seems useless to me. Compressing images is not a priority unless they're huge scenery shots (which are usually compressed shortly after being uploaded). Just seems like extra work for no profit on either end. sssSp7p.pngIjLCqFF.png 16:42, November 3, 2011 (UTC)

Comment - Cowgurl has a point. I wonder, though, an automated bot that only uploads if the filesize is over, say, 200kb and the new optimized image is at least 10% smaller... Not so sure about the whole request system though. If the viewer cares enough to note the image is large, they may as well download and compress it themselves. It's marginally more effort than posting on a page, and a fair bit of the time, the change in filesize would be too small to be worth it, and there is an IRC bot or something that does semi-automatic changes on request. Hofmic Talk 21:39, November 3, 2011 (UTC)

I knew a bit about TLULbot which could do this over IRC. I thought that having a request-page bot would be good for wiki users who don't want to get on IRC or can't for some reason. I have made a second proposal now; see it above.  a proofreader ▸  01:26, November 4, 2011 (UTC)
Well, the proposal is a bit better, I would think, though do you really think that going over about 44,000 images every week, most of which will not be improved, is practical? Going over all the images the first time is a good idea, lots of images aren't compressed (ideally, the bot should read the filesize listed on the page and not bother downloading the small files). If it's request as well, I don't think it would have to go over all the images on the wiki every week, just the new ones. Though a point to ponder: wouldn't most people who would bother to (or even know about) compressing images also be proficient with the IRC (or the ol' fashion method of downloading the file and doing it manually)? And why would those who don't even know about a bot like this?
With that being said, I definitely support an automatic compression bot, but don't think it's at all feasible (if possible) to run it every week on all the images. On a side note, as well, consensus does not mean voting. Hofmic Talk 03:05, November 4, 2011 (UTC)
Where possible, there would be caching so that the bot doesn't request images that it has seen or optimised already.
I think most people on the IRC would be technically savvy, yes, but the point about people not knowing about PNGOptimisationBot also applies: I didn't know about TLULbot doing such a thing until I saw TLUL and Evil talking about it on IRC; the wiki bot would be available for people there to see, but not until people looked for it or saw a mention of it in images' edit histories. It's about the same, I think.
Regarding your side note, sorry about that, I'm still a newbie at this bureaucracy thing. However, the votes could be for either proposal over the other, not just building consensus, if that makes sense.  a proofreader ▸  04:53, November 4, 2011 (UTC)

Support 2 - Compress all of the images, and upload them if the size goes down at least 10 or 15%. Images are a huge part of the load time on pages, and we should do everything we can to make sure it's as quick as possible. ʞooɔ 21:43, November 8, 2011 (UTC)

Support 2 - Per Cook. Matt (t) 05:27, November 9, 2011 (UTC)

Support 2 - Sounds like a solid idea. 222 talk 05:56, November 9, 2011 (UTC)

Support 2, sounds good. Nice to know that we have a competent bot programmer in our midst :) ajr 06:03, November 9, 2011 (UTC)

^ Matt (t) 06:06, November 9, 2011 (UTC)

Support 2 - There are still quite some images that are not compressed, so it would be nice to get this bot to compress those for us. </laziness> :3 JOEYTJE50TALKpull my finger 14:52, November 9, 2011 (UTC)

Support 2 - Do I need a reason anymore? Hofmic Talk 18:56, November 9, 2011 (UTC)

Support 2 - Hasten ALL the loading times! --クールネシトーク 23:22, November 9, 2011 (UTC)

Support 2 - I'm fairly useless in this area, but it sounds good to me. Ronan Talk 23:29, November 9, 2011 (UTC)

Comment - After working on an image renamer bot for Joey, I know more about the different MediaWiki API calls, so I've added them to the proposals. The bot would be more feasible given the existence of a Java wiki bot framework. I've also decided on a style of edit summary that would be good for distinguishing requested edits from weekly maintenance.

 a proofreader ▸ 

11:23, November 11, 2011 (UTC)

Closed - The PNG Optimization Bot's proposal 2 will be approved and implemented. The bot will be granted the bot flag, and as such will be featured on RS:BOTLIST. --LiquidTalk 16:49, November 15, 2011 (UTC)