Welcome!

Welcome to Kirsle.net! This is the personal website of Noah Petherbridge, and it's where my various software projects and web blog lives.

RiveScript.com Makeover

Noah Petherbridge
kirsle
Posted by Noah Petherbridge on Sunday, January 25 2015 @ 09:11:03 PM

I've just spent pretty much the whole day redoing the website for RiveScript.com, and I think it looks pretty nice.

Screenshot

RiveScript.com was the final website on my server that was still running on my legacy PerlSiikir CMS, and it's been on my to-do list for a while to get it migrated over to my new Python CMS, Rophako like what Kirsle.net is currently running on. The old Perl code was clunky and ugly and memory-leaky, and now I'll be at ease if I ever need to migrate to a new web server, as my Python web apps are extremely quick to get up-and-running, whereas it was an hours-long ordeal to get PerlSiikir to run.

So, the bulk of the work actually needed for RiveScript.com was purely front-end. I revamped the whole web design to use Twitter Bootstrap and make it look all hip and edgy like how all the other small software project websites are these days.

Besides the programming language on the back-end, I had other reasons for why I wanted to simplify RiveScript.com: I don't have the motivation or energy to do as much with that site as I did previously.

I used to run a YaBB Forum on RiveScript.com, but it wasn't extremely active and it was getting hit by too many spam bots, so several months ago I shut that down and linked to the RiveScript forum at Chatbots.org.

More recently I had programmed a chatbot hosting service that was on RiveScript.com, but that wasn't very popular either. I know nobody was using it because it had been broken for months and I hadn't heard any complaints. ;) A couple months ago I sunsetted that feature by turning off new site registrations and removing some references to the feature. And now that's officially gone! If you actually had a bot hosted there, contact me and I can get you your bot's reply files back.

So, the new site is simple and minimalistic and is just about the RiveScript language itself. It was an ordeal rewriting all of the pages from scratch (well, most of them) but now that it's done, the site should be very low-maintenance for me.

One of the most fun parts of it was that I ported over my "Try RiveScript Online" page to use JavaScript and run in the browser, whereas the old version had a Perl back-end (because a JavaScript version of RiveScript didn't exist at the time), so that's even one less thing for me to maintain and make sure it doesn't break. :)

And, the front-end pages for the new site are also open source, FWIW.

Brief GNOME 3.14 Review

Noah Petherbridge
kirsle
Posted by Noah Petherbridge on Thursday, December 18 2014 @ 12:39:53 AM

I jumped ship from GNOME 2 to XFCE when GNOME 3 was announced and have ranted about it endlessly, but then I decided to give GNOME 3.14 (Fedora 21) a try.

I still installed Fedora XFCE on all the PCs I care about, and decided my personal laptop was the perfect guinea pig for GNOME because I never do anything with that laptop and wouldn't mind re-formatting it again for XFCE if I turn out not to like Gnome.

After scouring the GNOME Shell extensions I installed a handful that made my desktop somewhat tolerable:

Screenshot
(Click for bigger screenshot)

And then I found way too many little papercuts, some worse than others. My brief list:

Settings weren't always respected very well, and some apps would need to be "coerced" into actually looking at their settings. For example, I configured the GNOME Terminal to use a transparent background. It worked when I first set it up, but then it would rarely work after that. If I opened a new terminal, the background would be solid black. Adjusting the transparency setting now had no effect. Sometimes, opening and closing a tab would get GNOME Terminal to actually read its settings and turn transparent. Most of the time though, it didn't, and nothing I could do would get the transparency to come back on. It all depended on the alignment of the stars and when GNOME Terminal damn well feels like it.

Also, I use a left handed mouse, and GNOME Shell completely got confused after a reboot. The task bar and window buttons (maximize, close, etc.) and other Shell components would be right handed, while the actual apps I use would be left handed. So, clicking the scrollbar and links in Firefox would be left-handed (right mouse button is your "left click"), and when I wanted to close out of Firefox, I'd instead get a context menu popup when clicking the "X" button. Ugh!

I wanted to write this blog post from within GNOME but it just wasn't possible. With different parts of my GUI using right-handed buttons and other parts using left-handed ones, I had context menus popping up when I didn't want them and none popping up when I did. After a while I thought to go into the Mouse settings and switch it back; this didn't help, instead, the parts that used to be right-handed switched to left-handed, and vice versa. It was impossible to use. I just had to painstakingly get a screenshot off the laptop and to my desktop and deal with it over there instead.

These things just lead me to believe the GNOME developers only develop for their particular workflows and don't bother testing any features that other mere mortals might like to use. All the GNOME developers are probably right-handed, and they have no idea about the left-handed bugs. All of the GNOME developers don't use transparency in their terminals, evidenced by the fact that the transparency option disappeared from GNOME 3.0 and only just recently has made a comeback (in GNOME 3.12/Fedora 20).

XFCE is going back on this laptop.

Skype switched to the MSN Messenger Protocol

Noah Petherbridge
kirsle
Posted by Noah Petherbridge on Tuesday, December 09 2014 @ 06:21:54 PM

Today, Fedora 21 was released and I upgraded to it immediately, and decided not to install Skype this time.

Skype for Linux has been poorly maintained, going years between updates sometimes, and who knows what kind of unknown zero-day vulnerabilities are in there. On my previous installation of Fedora, Skype twice showed a weird issue where it replaced some of its icons with Chinese (or Korean, or something) symbols. I posted about it on Reddit and the Skype forum with no responses, as if I'm the only one who's ever seen this. Was I hacked? Maybe.

I took the latest Windows version of Skype and dumped all its icons trying to find these weird symbols but came up empty-handed. And I don't know of any way to pry icons out of the Linux binary of Skype. So... for now I just don't trust it.


In other news, I decided to Google the Skype protocol, and see what the progress is on people attempting to reverse engineer it, to be able to build an open source third-party Skype client (e.g. to have support in Pidgin). The Wikipedia article said something interesting:

On June 20, 2014, Microsoft announced the deprecation of the old Skype protocol. [...] The new Skype protocol - Microsoft Notification Protocol 24 - promises better offline messaging and better messages synchronization across Skype devices.

I wrote previously that the MSN Messenger service was still alive but it looks like it's the future of Skype as well.

The Microsoft Notification Protocol (MSNP) is the protocol used by MSN Messenger/Windows Live Messenger. I'm reasonably familiar with it from back in the day when I used to work on chatbots that signed in to MSN Messenger to accept their add requests and carry on conversations with humans.

MSNP is a plain text, line-delimited protocol similar to SMTP. There is some outdated documentation up through MSNP10 that we referenced in developing an MSN module in Perl. As an example, this is what you'd see going over the network if somebody sent a chat message to a friend:

MSG 4 N 133
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
X-MMS-IM-Format: FN=Arial; EF=I; CO=0; CS=0; PF=22

Hello! How are you?

The protocol consisted of command lines, typically three letters long (some I remember offhand: NLN--go online, FLN--go offline, BRB--set status to "be right back", RNG--request a conversation with a contact ("ring"), ANS--accept a conversation request ("answer"), and MSG--send a message). The RNG and ANS commands were invisible to the user in the official client, but allowed for some interesting behaviors in our chatbots (like immediately sending you a message the moment you open their chat window, before you even begin to type anything!)

It's interesting that Skype "downgraded" to the MSNP though, given that Skype's old protocol was an impenetrable fortress of obfuscation and encryption that nobody's ever managed to reverse engineer. Even third party clients like Trillian that had Skype support were technically using SkypeKit, a developer tool that allowed Skype to be remote controlled, but which kept all the proprietary bits a secret still. On the other hand, it makes sense for them to make Skype conform to Outlook.com and their other services that used MSNP, rather than upgrade all their other services to use the Skype protocol.

The last version of Windows Live Messenger used MSNP19 (or MSNP21 depending who you ask), so the new Skype protocol is just the next version of MSNP.

In Googling MSNP24 I found this site where efforts are already underway at reverse engineering it. This other site has a lot more details on the current status of the Messenger protocol.

It's only a matter of time now until Pidgin can natively support Skype accounts. It will also be fun to program chatbots for Skype. :)

Dusting Off Alicebot Program V

Noah Petherbridge
kirsle
Posted by Noah Petherbridge on Tuesday, December 02 2014 @ 08:16:34 PM

In an older blog post I mentioned getting Alicebot Program V, a Perl implementation of an Alice AIML bot, up and running on a modern version of Perl, but I didn't go into any details on what I actually did to get this to work (which wasn't very nice of me ;) ).

Unfortunately for me I also didn't even write down any notes for myself, so I had to figure it out again, myself, from scratch. Which I decided to do, only this time I wrote down some notes, and published a new, updated version of Program V which you can check out on GitHub -- or for the lazy, get a zipped release of Program V 0.09.

This blog post is about what I needed to do to get Program V up and running, some of which is also outlined in UpdateNotes.md on the GitHub repo.

So, first of all, Program V was programmed for Perl version 5.6, which is ancient; by the time I had discovered the existence of Program V, the current version of Perl was 5.8, and already, Program V would not run out-of-the-box on Perl 5.8; and I was a newbie back then and had no experience to draw from to figure out how to get Program V to work for me. I wanted to use Perl to make chatbots, and Program V was the only AIML implementation around, but I couldn't use it, and it eventually led me to just create my own chatbot language to replace AIML for me -- RiveScript.

The differences between Perl 5.6 and all newer versions are pretty significant and some things have changed, and that's the basic background on why Program V was hard to get to work.

So the first thing I needed was to install the Perl modules Unicode::String and Unicode::Map8 - I learned this the hard way by trying to run build.pl and getting missing module errors, but the README also mentioned I needed these things. And then I got an error message that probably looks familiar to a lot of people who've tried (and failed) with Program V:

[~/g/programv]$ perl build.pl
Constant subroutine AIML::Common::LOCK_SH redefined at lib/AIML/Common.pm line 52.
Prototype mismatch: sub AIML::Common::LOCK_SH () vs none at lib/AIML/Common.pm line 52.
Constant subroutine AIML::Common::LOCK_EX redefined at lib/AIML/Common.pm line 53.
Prototype mismatch: sub AIML::Common::LOCK_EX () vs none at lib/AIML/Common.pm line 53.
Constant subroutine AIML::Common::LOCK_NB redefined at lib/AIML/Common.pm line 54.
Prototype mismatch: sub AIML::Common::LOCK_NB () vs none at lib/AIML/Common.pm line 54.
Constant subroutine AIML::Common::LOCK_UN redefined at lib/AIML/Common.pm line 55.
Prototype mismatch: sub AIML::Common::LOCK_UN () vs none at lib/AIML/Common.pm line 55.
Died at lib/AIML/Unicode.pm line 102.
Compilation failed in require at lib/AIML/Common.pm line 109.
BEGIN failed--compilation aborted at lib/AIML/Common.pm line 109.
Compilation failed in require at build.pl line 32.
BEGIN failed--compilation aborted at build.pl line 32.

The most interesting line to me was when it said Died at lib/AIML/Unicode.pm line 102. without any error message (looks like a die; statement in Perl with no message), so I looked at this line of this file, and it was doing this:

my $changer = do ( $pkg ) or die;

So I debugged it (read: added a print "$pkg\n"; above that line) and found it was trying to import unicode/To/Upper.pl which was nowhere to be found. On my Linux system I ran a locate Upper.pl command, and found that a similarly named file existed at /usr/share/perl5/unicore/To/Upper.pl - the key difference is that "unicode" is spelled "unicore".

There were 3 lines of code in AIML::Unicode that reference files like this and I changed all three to point to the "unicore" versions - fixed it! build.pl would now make some progress.

It then complained about a couple of the AIML files containing illegal characters in their <pattern> tags. No big deal, went in and fixed them. build.pl runs successfully now! And then I ran shell.pl and was dropped into an interactive chat session with an Alice bot.

As for those AIML::Common::LOCK_SH warnings: I have to assume that in Perl 5.6, file locking (flock) was either a new feature or not widely implemented, and the code was trying to do a few error checks (in a Perl 5.6 style way) and if flock isn't found to be available, it would shim those constants itself. But, flock is available, but the error checks were still going off, so AIML::Common was overriding the constants. So, I just fixed the error checks and problem solved. ;)

So once I got it working, I published the updated version of Program V on my GitHub account for others to check it out.

This doesn't mean Program V is actively maintained again. It was abandoned by its author in 2002 and hasn't been able to run on modern Perl ever since; I simply updated it so it works again. If that inspires you, feel free to fork it and maintain it on your own - it was released by its author under the same terms as Perl itself.

Minecraft Clones

Noah Petherbridge
kirsle
Posted by Noah Petherbridge on Wednesday, October 22 2014 @ 06:10:28 PM

These are just some of my thoughts on things that a Minecraft clone should probably do differently to Mojang's (well, Microsoft's) Minecraft, since a clone would be starting from scratch anyway and would be in a good position to fix some of the fundamental problems in the original game that are impossible/unlikely to be fixed by Microsoft.

Some of the Minecraft clones I'm familiar with so far are Terasology, Minetest, and Michael Fogleman's Craft. So far, Craft seems to do the most things "right" (as far as my list of fundamental improvements on Minecraft go) such as only storing changes to the blocks to disk rather than whole entire chunks, but more on that in a minute.

Disclaimer: I'm not a game developer. I am a software developer and have a particular interest in the technology used in games, i.e. I have the knowledge to write code for most given game mechanics and spend countless hours thinking about these sorts of things. I've read a lot about the technical internals of Minecraft and have read some of the source code to these clones, so this blog post is full of my own educated opinions on things. I may, at some point, take on the challenge of creating a Minecraft clone of my own, but until then these are just the things I've been thinking about over the last several months.

Fundamental Game Mechanics

Minecraft has a few fundamental design flaws that aren't likely to ever be fixed and are in all likelihood impossible to fix without horribly breaking the game (or at least without completely throwing away backwards compatibility and not working anymore with maps created by older versions of Minecraft).

Just to get this out of the way first: just because Mojang did it in Minecraft, doesn't mean it was the optimal way of doing it and that clones of the game should do it in exactly the same way. Mojang had the disadvantage of not being able to predict where Minecraft would go in the future (hindsight and all that), so some of the design decisions might've made sense at the time, but nonetheless they prevent certain features from being possible in the game as it exists today.

A Minecraft clone starting from the ground up is in a better position to learn from Minecraft's shortcomings and solve them from the beginning and produce a better game in the end.

Cubic Chunks

Minecraft divides the world into chunks, which are 16x16 blocks horizontal and 256 blocks vertical. This chunk system brings performance and world height implications. The world height in Minecraft is limited to 256 blocks, and they can't really remove that height restriction in the current chunk model. If they wanted to raise the height to, say, 512 blocks, they'd have to redefine a chunk to now be 16x16x512, meaning more data needs to be stored per chunk. That solution isn't scalable.

Instead, a Minecraft clone could be better served by making the chunk system cube-shaped, so a chunk would be 16x16x16 blocks. This would make worlds not only "practically infinitely" large horizontally (as Minecraft worlds are), but also infinitely large vertically. There would be no technical reason why the world height should be limited to 256, 512, or any other arbitrary number of blocks. For sanity's sake, servers might have a configuration option to set a maximum world height, but on a technical level you could build a tower that's 30 million blocks tall if you really wanted to.

Cube shaped chunks could also bring new biome-related features. In Minecraft, the biomes are only assigned on the horizontal level, i.e. a single block column from bedrock to the world height limit ALL have to be the same biome, i.e. Desert or Forest. With cubic chunks it would add more flexibility, and you could have height-dependent biomes, such as an underground Mushroom Cavern biome, or a floating Sky Island biome like in Terraria.

Nether below the world?

This idea may be controversial because the Nether dimension in Minecraft has an 8:1 scale ratio in relation to the Overworld, making Nether Highways useful for fast travel between locations on your Overworld.

However, with a cubic chunk system which opens the way to height-dependent biomes, the Nether could physically reside underneath the overworld, like how it is in Terraria. Just keep digging deep enough and you'll naturally find yourself in the underworld, where different types of monsters spawn and there's fire and brimstone and lava everywhere.

More Optimal Chunk Saving

In Minecraft, when a player wanders into unexplored territory, the game has to generate all of the chunks for the new area (16x16x256 block cuboids), and then the game saves the whole entire chunk to your hard drive. All 65,536 blocks in that chunk.

A better solution would be to do what Fogleman's Craft does. It doesn't save the whole entire chunk's data to disk, it only saves changes made to the chunk after its initial creation.

When you leave the game and come back later, the game would re-generate the "default" chunks, and then apply your saved changes as a "mask layer" over top of the defaults. So any tunnels you dug or castles you built, only the data for those modified blocks needs to be saved to disk.

This would bring so many advantages over the way Minecraft does it, and not just for disk usage reasons. One of my biggest complaints with Minecraft is that each new version tends to add new features to terrain generation (for example, new types of ores or new stone blocks that spawn randomly in generated terrain). If you already have an old Minecraft world and upgrade to the next version, your already-generated chunks do not update to include the new naturally occurring terrain features. You have to go out and explore new chunks to get the new features. Or otherwise, use a third party tool like MCEdit to delete chunks from your world, so that they can be regenerated by the new Minecraft version.

If the game only saved blocks that were changed, any new ores or stone blocks that would've generated in blocks that you've never touched, would appear there now. No needing to mess with MCEdit, and yet all your tunnels and houses and Creeper damage would be left intact between version upgrades.

And what if a new game version introduces massive terrain generation changes? i.e. so that a chunk that used to generate a mountain instead will generate an ocean?

Versioned Terrain Generation

This.

Each chunk generated and saved to disk could store the "generator version" of the terrain generation algorithm that was used to generate that chunk.

If a game update has to break backwards compatibility and radically change terrain generation (as Minecraft had to with the release of v1.7), having chunks remember their generator version would help here.

Instead of deleting all the old source code when you replace the generator, just add a version check.

if algorithm_version == 2:
    do_new_stuff()
elif algorithm_version == 1:
    do_old_stuff()

Then, an old world that got upgraded to a newer version of the game would have their old chunks still work as they expect, but have new chunks use a newer terrain algorithm. Or, they would have the option to say, "keep the ENTIRE world using algorithm version 1" and a world-breaking game update won't change the way their old world used to be generated.

Do Lighting Better

This one is debatable, and I'm not extremely familiar with OpenGL, so take this with a grain of salt.

One of Minecraft's big problems is that light sources can't be dynamic. So you can't be holding a torch and walk through a cave and see the lighting dynamically update as you walk. You'd think that dynamic lighting is a very basic feature in OpenGL and even the most amateur 3D game can do it, so why doesn't Minecraft?

In their case it has to do with monster spawning. Each block has to store its "light level", a value from 0 to 15. If the light level is low enough, hostile monsters can start spawning. Placing a torch, or breaking a torch, or turning a redstone lamp on or off, these events need to touch every block in their radius and recalculate that block's light level. This is an expensive computation, and you can see the game lag if you turn on or off a lot of light sources at once, i.e. by using a redstone circuit involving a large amount of lamps.

I would think a better solution would be to query the pixels on blocks for their brightness levels, and let OpenGL handle the dynamic lighting the "correct" way. Like, the mob spawning rules could say, "find a block that's at least 20 blocks away from a player. Okay, check this block's pixel brightness. Dark enough? Spawn a hostile mob. Otherwise, find a different block and repeat". But again, I'm not super familiar with OpenGL and don't know how possible this is.

An alternative would be to just change the mob spawning rules, and make them more like Terraria. In Terraria monsters can spawn pretty much anywhere, as long as the background tiles aren't ones placed by the player and as long as there aren't a large number of friendly NPCs nearby (so building a town creates a natural radius of anti-monster-spawning). The background tile part wouldn't make sense for a 3D game like Minecraft, but something similar could be worked out that doesn't rely so heavily on light levels.

Let's Talk Blocks

Another bad design decision IMHO in Minecraft is the block ID system. Especially earlier on, every little variation on a block required a whole new, distinct, block ID. Oak Wood Stairs are an entirely separate block than Jungle Wood Stairs. Birch Wood Slabs are different from Spruce Wood Slabs. And so-on.

In Minecraft, a block in the world had a block ID, and one byte worth of meta-data (actually, a half-byte, with value options between 0 and 15). Some blocks used the metadata to hold color information (the Wool blocks all share one Block ID, and they come in 16 predetermined colors, which coincidentally is the maximum number of variations you can squeeze out of a half-byte).

Wood Stairs can't do this because their half-byte metadata has to hold things like which direction the stairs are facing, and whether they're connected to other stairs (to form corner stairs, for example), so each type of wood has to have a completely new Block ID.

Minecraft duct-taped this to allow for more data to be attached to a block by introducing the Tile Entity system. Examples of Tile Entities are Chests, Furnaces, Enchantment Tables, and Cauldrons. They're basically blocks (they have block IDs), but they have arbitrary, unstructured metadata attached to them, to hold things such as their inventories (like with Chests) or their available enchantment level, etc.

The Tile Entity system isn't great though and it brings a lot of overhead to it, so if you have an extremely large number of Tile Entities it will lag out the game. So, in Minecraft's case, it wouldn't be possible to make every single block a Tile Entity, or to even make half the blocks Tile Entities, as it would probably harm the game's performance.

From poking around at the source code of clones such as Terasology and even Craft, it looks like they followed Mojang's flawed block system here, i.e. they'll have 16 colors of Wool and each Wool Block is a different distinct block ID.

A better system would be to think of this problem from the ground up and implement a proper, performant block system. For example, why should Wool Blocks be limited to 16 pre-defined colors? In Minecraft you can dye Leather Armor to pretty much any color imaginable, because Leather Armor has enough data to hold a full RGB color code. Wool Blocks don't, so you can only use a limited color set.

If the system were better designed, Wool Blocks could be any color, and you'd have the ability to recolor a lot of blocks, like in Terraria. In Terraria you can paint pretty much everything, stone, wood, all kinds of colors. In Minecraft this would require a metric ton of new Block IDs for all the types and variations of blocks in the game.

And About Plugins...

It is my opinion that a Minecraft style game should have a well defined vanilla version, with a default set of "plugins."

A plugin-based architecture is great, it's what Minetest uses. It allows for easily adding new types of blocks, crafting recipes, and features to the game. The base game itself can be composed of various "default plugins."

But, making too big a deal of the customizable, pluggable nature of the game causes users to be overwhelmed by choice. In Minecraft, a lot of people play the vanilla game, the one released directly by Mojang/Microsoft and they don't mess around with plugins.

But take for example the Feed the Beast mod pack for Minecraft. Did I say mod pack? I meant mod packs, because there are a ton of them. If I wanted to run my own Feed the Beast server and have my friends play it, we all have to pick one of these and agree on which one we picked. Can somebody use the Monster client mod pack and log in to a Magic 2 World server? I have no idea. Probably not. And this would turn away a lot of casual users. It turned me away and I consider myself to be more tech savvy than most.

This is a problem with Minetest, imho. Minetest wants you to mix and match your mods and create a version of the game you wanna play. But that will just lead to confusion when you want others to play with you.

A Minecraft clone should have a well defined "default" vanilla game that is highly recommended for all players to play. More tech savvy players can play with mods if they want, just like in Minecraft.

A good solution, I think, would be for most mods to be server-side. Mods that introduce new blocks, crafting recipes, and some game mechanics, should be able to work as being purely server-side mods. Clients connecting could download graphics and other assets needed by those mods, but the end user playing the game shouldn't need to worry about it. This isn't the case with Minecraft mod packs like Feed the Beast, where the client has to be playing with the matching mod pack that the server uses or else things will in all likelihood break in a very bad way.

tl;dr.

Minecraft clones should take their opportunity, as a ground-up rewrite, to do everything better than Minecraft did. Just because Mojang did something a certain way, doesn't mean it was the right way.

Kirsle
Channels
Creativity
Software
Web Tools
Subdomains
Miscellany
Links


Fan Club