Welcome to Kirsle.net! This is the personal website of Noah Petherbridge, and it's where my various software projects and web blog lives.

Event Loops

Noah Petherbridge
Posted by Noah Petherbridge on Thursday, November 26 2015 @ 04:12:20 PM

This is something I wanted to rant about for a while: event loops in programming.

Story time!

I like to write chatbots. Back when I got started writing bots, I wrote them in Perl and there were modules available for all the most common chat platforms of the time: Net::OSCAR for AOL IM, Net::YMSG for Yahoo! Messenger, Net::IRC and we also had an MSN Messenger module that was simply named MSN.

All of these modules were written to use their respective protocols in a synchronous manner. That is, they didn't use any event loops, but had functions with names like start() and do_one_loop().

For the simplest bots, where you only wanted to sign on one instance of an AIM bot per Perl script, you'd use the start() method which would enter the main loop and block forever. But if you wanted to run multiple bots from the same Perl script (say, sign on three different AIM bots, two MSN bots and one for IRC all at the same time), you could write your own main loop and manually drive the loops of the respective modules.

Since they all somehow agreed on a common API, the code could look like this:

# Where @connections is an array of bot objects, like Net::OSCAR,
# Net::YMSG, Net::IRC, MSN, etc...
while ($running) {
    # main loop
    foreach my $conn (@connections) {

        # This also frees up the bot to do other background tasks
        # on each loop here.

But, more recently I tried writing a new Perl chatbot which I called Aires, and this bot connected to AIM and Yahoo! Messenger (using a newer Net::IM::YMSG module that I and Matt Austin wrote for the new YMSG protocol, since Yahoo shut down their older protocols around the same time Microsoft announced the same plans to shut down old MSNP protocols).

Net::OSCAR and the new Net::IM::YMSG both used the same old synchronous APIs and all was well. But, I wanted Aires to also use XMPP. But the only Perl interface for an XMPP client was AnyEvent::XMPP, and it required the AnyEvent event loop framework.

To make a long story short, if you already have an existing codebase that does a bunch of stuff and doesn't use an event loop framework, and then you want to utilize some module that does use an event loop framework, things don't go very well. It's pretty difficult to mix and match modules when some of them use a certain event framework and the rest of your code doesn't use the same framework.

To support the XMPP module correctly, my entire codebase would have had to be refactored to work in the AnyEvent framework and I'd have to coerce the other two instant messenger modules to work under those conditions.

Event Loops in Procedural Languages

The root problem, I think, is when people try to build event loop architectures on top of otherwise procedural languages, such as Perl and Python. Procedural languages like these aren't event-oriented and just execute their instructions one at a time in the order they were written, and the languages have no built-in "official" way of writing an event-based program.

So, people write their own event loop systems, such as POE or AnyEvent for Perl, or things like twisted and asyncio for Python. Oftentimes the competing event loops are not compatible with each other. Most event loops want to be THE event loop, and don't support some other loop "manually driving" theirs (like calling a do_one_loop() type of function).

A lot of graphical user interface modules (e.g. Tk and GTK+) have their own event loops, which makes sense for them because it makes more sense to write a program that responds to buttons being clicked and scrollbars being scrolled if the entire front-end of your program revolves around a graphical interface. But these have the same problems as any other event loop and it's hard to mix-and-match them, or to use them from an existing codebase that wasn't architected to use a compatible event loop. In the Perl version of Tk there's a way to manually drive Tk's loop (call $mainwindow->update() instead of MainLoop;) but that isn't universally true of all GUI frameworks.

Mixing an arbitrary event loop framework with a GUI event loop isn't always straightforward and is sometimes impossible, unless you want to partition your program into different threads to completely separate two competing event loops from each other.

Another Story!

After trying to bolt on XMPP support via AnyEvent::XMPP in a way that only works some of the time, I abandoned the Perl version of the Aires bot and rewrote it in Python and gave it the same name. I ran into a similar conundrum when trying to mix interfaces that required the Twisted framework (AIM and XMPP) with ones that didn't (local command line interface). I stopped working on this version of Aires as well.

Fast forward quite a bit.

Yesterday I wrote two chatbots from scratch. One was written in Python and I named it Admiral. The goals for this bot, initially, were to connect to both the Slack chat platform and Google Hangouts, using the hangups module.

I wrote the Slack part first because it was easier and I had some prior experience with it. The official SlackHQ implementation for Python is what I would call synchronous. I had to write my own do_one_loop() function that polled the Slack API to check for any incoming messages over their RealTime Messaging protocol.

When I wanted to attach the Google Hangouts integration, I ran into the same problem with trying to use an event loop on a program that doesn't already use it: Python 3's asyncio. There are several Google Hangouts bots already written for Python, and most of them only do one bot, which I didn't want (I want my one Python script to connect multiple bots to multiple platforms simultaneously). To use the Hangouts module I have to let asyncio take over control of the script and handle the main loop on its own, which would result in the Slack bot being starved out, as the Slack bot's loop to poll for events would never get a chance to run anymore.

Another strike against event loop frameworks being bolted on top of languages that didn't have them built in.

So, the second chatbot I wrote yesterday, I wrote in Go. I called this one Scarecrow, and this one is also a Slack bot for now but I'm more optimistic about this one due to the Go language itself. In addition to Slack it has a local command line interface that can chat locally with you simultaneously to having one or more Slack bots running.

Optimism for Golang

Something I like so far about the Go language is that it has a system for running concurrent tasks built in. They're called Goroutines and they're a first-class citizen in the language, and they should render ALL event loop frameworks obsolete.

On my near-term TODO list for the Scarecrow chatbot is to add XMPP support. I think it won't matter how the XMPP module is written; in the worst case scenario I can go run the XMPP code and isolate it from the entire rest of my program without any issue.

Opinion: Libraries should NOT use event frameworks

This is basically the point of this rant. If you're writing a library, such as an XMPP client or a Google Hangouts client, you absolutely should not use an event-loop framework in your code. Hear me out.

If you wrote a Hangouts module and you wrote it synchronously as mentioned above, with something like a start() and/or do_one_loop() function, you give the end developer of your module the best of all worlds. If I just want to write the simplest chatbot I can, one that uses only your module and nobody else's, I can do that very easily. And if I have a more complex program to write, I can choose an event framework that I personally like and I can wrap your module inside one of my coroutines on my own.

Like, suppose you have a synchronous module and this is my pseudocode written in Go:

func Example() {
    // I want to use a synchronous module... I can just make my own goroutine to wrap it.
    go YourModuleWrapper()

func YourModuleWrapper() {
    for {

This way, if I already have an existing codebase and I find your module and I want to use it, I can. I don't have to see what crazy event loop framework it uses, I don't have to rearchitect the very core of my application to fit the use case of that framework, I don't have to deal with trying to get two competing frameworks to coexist together... I can just use it. If my use case requires a high level of concurrency, I can choose whatever framework I want and I can wrap your code to suit my use case. If I'm using a graphical framework like GTK+ I can wrap your module in that framework's event loop on my own, without having to deal with the headaches of getting different event loops to get along with each other.

Don't use event loop frameworks.

Event loops, if needed at all, should be at the sole discretion of application writers, people who are writing the actual code that runs somewhere and serves a purpose. NOT the library writers. Libraries should not build themselves around a particular event framework.


Google knows when you wake up?

Noah Petherbridge
Posted by Noah Petherbridge on Saturday, September 26 2015 @ 04:25:11 AM

Something pretty odd that's happened to me on more than a couple of occasions:

I'll wake up in the morning, and then a couple minutes later people start sending me messages on Google Hangouts, so my phone's making a bunch of noise. If it's early enough in the morning, some of those messages will even say things like, "you're up early."

The thing is though, I didn't even touch my phone yet. I didn't even look at my phone.

Messages like this don't tend to wake me up, but instead I don't start getting messages until just after I'm awake.

My theory is that Google Hangouts uses the accelerometer on my phone to detect when I start moving in the morning so it knows I woke up, and then it sets my status to Online. I saw similar stuff happen with Facebook Messenger a long time ago too when I used that app... but that's Facebook and it's safe to assume they're always doing creepy stuff like that.

Googling this kind of stuff doesn't turn up many results. But now there's this blog post for anyone else who looks into this. It's happened too many times to be a coincidence; why would my status change to Online at just the same time that I happened to wake up and roll over in bed?


RiveScript in Go

Noah Petherbridge
Posted by Noah Petherbridge on Tuesday, September 22 2015 @ 02:11:59 PM

Over the last couple months I've been slowly working on rewriting RiveScript in yet another programming language: Google's Go language! It's also my first project in Go, which is what tends to be the case (rewriting RiveScript in Java and Python were my first projects in those languages, too).

But this is the last time I'm going to be rewriting RiveScript from scratch.

With the recent release of Golang 1.5, a Go package can be compiled down to a C-compatible shared object file (like a .so or .dll, though I think they're still working on the DLL part). It means that modules written in Go can be linked from other programming languages, especially C or C++, but also to any programming language that can be extended with C. Which, as it turns out, is most of them.

For example, if I want to make RiveScript available to Ruby programmers, I can theoretically create a binding in Ruby for the Go version of RiveScript rather than rewrite RiveScript in Ruby.

The first language I'll probably do this to is Python, because I found a blog about writing Python modules in Go which will be pretty helpful. There's already a native RiveScript module for Python, but it will be a nice learning experience anyway. And who knows, it might even outperform the native Python version. ;)

Back to RiveScript-Go, a feature I think is interesting is that it natively supports JavaScript object macros via a library called otto, which implements a JavaScript interpreter natively in Go. The default rivescript.exe with my Go module already builds in the support for JavaScript objects. :)

Example implementation of the Go module:

package main

import (
    rivescript "github.com/aichaos/rivescript-go"

func main() {
    // Initialize the bot.
    bot := rivescript.New()

    // JavaScript object macro handler.
    jsHandler := rivescript_js.New(bot)
    bot.SetHandler("javascript", jsHandler)

    // Golang functions work too, but must be compiled in.
    bot.SetSubroutine("gotest", func(rs *rivescript.RiveScript, args []string) string {
        return "Hello from Go!"

    // Load a directory of RiveScript files.


    // Drop into the interactive command shell.
    reader := bufio.NewReader(os.Stdin)
    for {
        fmt.Print("You> ")
        text, _ := reader.ReadString('\n')
        text = strings.TrimSpace(text)
        if len(text) == 0 {

        reply := bot.Reply("localuser", text)
        fmt.Printf("Bot> %s\n", reply)

The JavaScript objects also get full access to the RiveScript object, so they can call methods like CurrentUser() and SetUservar().

Check it out on GitHub: https://github.com/aichaos/rivescript-go

Also, if I ever write a successor to RiveScript, I'll probably only implement it in Go or some other portable language rather than rewrite it five different times in five different languages. Rewriting something is a good way to learn a new language, but then it becomes pretty tedious maintaining them all. ;)

Internet Protocol version 6

Noah Petherbridge
Posted by Noah Petherbridge on Monday, July 20 2015 @ 01:53:26 PM

Last weekend I was pleasantly surprised to discover that Time Warner Cable already supports IPv6 at my apartment.

They shipped me a newer cable modem/WiFi router combo device earlier this year as part of their plan to upgrade everyone's Internet speeds in Los Angeles. I didn't realize that this modem also supported IPv6, and it wasn't enabled by default.

I was playing Splatoon on my WiiU and it was having Internet connectivity problems, making multiplayer impossible. I had this issue before with Smash Bros. 4, and I couldn't figure out a work-around short of putting my WiiU into the DMZ of my router (which I generally didn't like the idea of). This meant ALL inbound connections from the Internet would bypass the router and go straight to the WiiU. It did solve the multiplayer problem, but I didn't like it and it also meant I couldn't port-forward anything to my desktop PC.

Since Splatoon basically requires online access (the single player mode is pretty limited, and you can't access any of the shops without leveling up, and you can only level up by playing online...) I was more motivated to figure it out. (Long story short, I had to disable UPnP support -- the opposite of what you'd expect; normally enabling UPnP is the fix. Weird)

I stumbled upon some IPv6 related options though, like to enable the DHCP server for IPv6. It was disabled by default so I turned it on to see what would happen. My PC was then given a publicly routable IPv6 address from my router. :) I went to a Test your IPv6 site and confirmed that all of a sudden, my Internet is IPv6 ready!

Also, I went to the What Is My IP site and instead of saying my usual IPv4 address it told me my IP was a big long string of numbers.

I was then curious about inbound routing to my IP, because one of the big features of IPv6 is that NAT is no longer required, and that every device is able to have its own publicly routeable address. I didn't have immediate access to any off-site IPv6 devices though to test it which made things more difficult.

But then I found out I could enable IPv6 on my DigitalOcean VPS. I'd heard of them rolling out IPv6 support a while back, but I thought it was limited to certain data centers or available only to newly created VPS's. It turns out I just had to click a big "Enable" button and then configure the addresses on my server (to set them to the addresses that DigitalOcean says belong to me).

So... Kirsle.net now has a DNS AAAA record pointing to my IPv6 address on my VPS.

kirsle.net.    819    IN    A
kirsle.net.    1799   IN    AAAA    2604:a880:1:20::46:5001

Inbound Routing

Inbound routing for my personal PC still wasn't working, though. I set up a basic nginx server and tried hitting it from my web server but it didn't work. I could ping my home IPv6 address, but couldn't do much else.

I booted up my laptop and confirmed that it got its own IPv6 address distinct from my desktop PC, and that my laptop could connect to my desktop over its IPv6 address, so that ruled out a software firewall as being the problem.

I logged into my router again to poke around (the router is an ARRIS DOCSIS 3.0 Touchstone Residential Gateway, model DG1670A). I tried turning off its firewall. This worked sort of -- IPv6 addresses were now inbound routable, but IPv4 port forwarding no longer worked. Not an option for me. I turned the firewall back on and poked around some more.

There was a section called "Client IPv6 Filters" and the description on its page said "The Router can be configured to restrict access to the Internet, e-mail or other network services."

This sounded like it was designed for blocking outbound connections, like if I wanted to blacklist a website or something from being connected to from my network. But I clicked the Add button and saw this modal:

Add Client IP Filter screenshot

The Action/Direction options were "Allow+Incoming" or "Deny+Outgoing". I left it on the Allow option, copy/pasted my desktop's IPv6 address into both address boxes and clicked the confirm button. Now my web server was able to curl my desktop PC and get the nginx default web page. Success!

nginx on IPv6

When getting the IPv6 support set up in nginx on my web server I ran into some interesting problems.

My server hosts a ton of different sites, and normally every single server directive would have its own listen 80; indicating that the server is a candidate for handling requests to IPv4 port 80 on the server.

This doesn't work the same on IPv6.

If I had more than one server set to listen [::]:80 ipv6only=true (not sure if that last part was necessary), nginx would complain about it and fail to start.

I think what needs to happen is that if I want IPv6 support enabled for multiple domains at the same time, I have to individually assign an IPv6 address to each one. With IPv6, you basically have free reign over 64 bits of your address space to assign however you want. For example, DigitalOcean says my IPv6 prefix is 2604:a880:1:20 and the next 64 bits after that are all mine. On my home network with Time Warner Cable, I'm similarly given a fixed 64-bit prefix and the rest can be assigned by me (either via DHCP or statically).

So it's probably possible to tack on a whole bunch of static IPs of my choosing onto the ethernet device of my server, and then bind to those specific addresses in nginx, one for each site. I'll cross that bridge when I get to it.

Fedora 21 on the 2015 Macbook Air

Noah Petherbridge
Posted by Noah Petherbridge on Saturday, May 02 2015 @ 01:06:37 PM

Today I picked up a Macbook Air (13", early 2015 model) because I wanted a new laptop, as my old laptop (the Samsung Series 5) has a horrible battery life, where it barely lasts over an hour and gives up early (powering down at 40% and not coming back up until I plug it in). This is also my first Apple computer. I'm the furthest thing from an Apple fanboy, but the choices I was throwing around in my head were between an Apple computer and a Lenovo Thinkpad.

I was given a Thinkpad as my work laptop, and it's by far the most impressive PC laptop I've ever used; it can drive three displays and run lots of concurrent tasks and has an insane battery life. Every PC laptop I've owned in the past have sucked in comparison. I hear people compare Apple computers to Thinkpads, so that's why the choice came down to one of these, and I didn't want another Thinkpad sitting around the house. ;)

Months before getting a Macbook I was looking into what kind of effort it takes to install Linux on a Macbook. There's a lot of information out there, and most of it suggests that the best way to go is to install a boot manager like rEFIt (or rEFInd, since rEFIt isn't maintained anymore). I saw some pages about not using rEFIt and installing Grub directly, which were from a Debian and Arch Linux perspective, and it sounded really complicated.

It seems that nowadays, with a user friendly Linux distribution like Fedora, a lot of this works much more flawlessly than the dozens of tutorials online would suggest. I just made a Fedora LiveUSB in the usual way (as if installing on a normal PC), rebooted the Macbook while holding the Option key, so that I was able to select the USB to boot from.

When installing Fedora to disk, the process was very much the same as doing it on a normal PC. I let Fedora automatically create the partition layout, and it created partitions and mount points for /, /boot and /home like usual, but it also created a partition and mount point for /boot/efi (for installing itself as the default bootloader in the EFI firmware on the Macbook). After installation was completed, I rebooted and the grub boot screen comes up immediately, with options to boot into Fedora.

One weird thing is, the grub screen apparently sees something related to Mac OS X (there were two entries, like "Mac OS X 32-bit" and "Mac OS X 64-bit", but both options would give error messages when picked).

If I want to boot into OS X, I hold down the Option key on boot and pick the Macintosh HD from the EFI boot menu. Otherwise, if the Macbook boots normally it goes into the grub menu and then Fedora. So, the whole thing is very similar to a typical PC dual-boot setup (with Windows and Linux), just with one extra step to get into OS X.

Update: I'm keeping a wiki page with miscellaneous setup notes and tips here: Fedora on Macbook

Web Tools

Fan Club