Kirsle.net logo Kirsle.net

Welcome to Kirsle.net!

This is the personal homepage of Noah Petherbridge, and it's where I keep my web blog and various creative projects.

I blog about anything I find interesting, and since I have a lot of varied interests, my blog entries are kind of all over the place. You can browse my tags to sort them by topic and see which ones I frequently write about, or the archive has a complete history of my posts, dating back to 2008!

Besides my blog, I have pages for my creative projects, which are linked to on the navigation bar.

I write a lot about Linux and Android, Minecraft, and I like to rant about stuff. Generally anything that makes me curious. Also check out my Bookmarks for all sorts of cool websites about various topics I'm interested in.

For the geeks: this website respects your privacy and doesn't run any third party ads or analytics. This site speaks HTTP and doesn't require any JavaScript to work.

Journey to get WebRTC working well in Safari
May 9, 2024 by Noah

A while back (February 2023) I built an open source webcam chat room for one of my side project websites.

It was very much designed to resemble those classic, old school Adobe Flash based webcam chat rooms: where it is first and foremost a text based chat (with public channels and private messages), and where some people could go on webcam and they could be watched by other people in the chat room, in an asynchronous manner.

It didn't take terribly long to get the chat room basically up and running, and working well, for browsers such as Chrome and Firefox - but it was a whole other story to get it to work as well for Apple's web browsers: Safari, and iPads and iPhones.

This blog post will recount the last ~year and change of efforts to get Safari to behave nicely with my chat room, and the challenges faced and lessons learned along the way.

Screenshot of BareRTC

Description of the chat room's features

First, it will be helpful to describe the basic features of the chat room and how I designed it to work, to set some context to the Safari specific challenges I faced along the way.

  • Users can choose to turn on their webcam and microphone and allow others on the chat room to watch them.
  • Users who are not on camera themselves are able to passively watch those who are (receive-only video streaming).
  • A pair of users who are both on camera are able to watch each other's videos if they want (two-way video streaming).
  • Each user can decide which cameras they want to watch - it is 'asynchronous' meaning the calls don't need to be two-way, and users can watch each other in any combination available.

There are a few other features on top of these, but the above are the basic fundamentals that are relevant to this story about getting this all to work on Safari.

The additional features include:

  • If a user isn't comfortable with their camera being watched by somebody who isn't sharing their own camera, they can restrict it and make their cam only available to people sharing their own too (so that they could have a chance to open your camera in return and see what you look like as well).
  • And there's an option for "when somebody opens my camera, I'll also open their camera automatically" - so if somebody who is on webcam clicks to see yours, their camera too will appear on your screen, making it an instant two-way call without you needing to separately open their camera back.

WebRTC Crash Course

The underlying web browser standard that allows videos to be shared at all is called WebRTC, which stands for "Web Real Time Communication." It is supported in all major web browsers, including Safari, but the devil is in the details.

WebRTC basically enables two web browsers to connect to each other, directly peer-to-peer, and exchange data (usually, video and audio data but any kind of data is possible). It can get two browsers to connect even when both sides of the connection are behind firewalls or behind a NAT (as 99% of regular home Internet users are).

For my chat room, it means that webcam data is sent directly between the chat users and none of it needs to pass through my server (which could be expensive for me to pay for all that bandwidth!).

It's a rather complex, and poorly documented, system but for the sake of this blog post, I will try and distill it down to its bare essence. The following is massively simplified, but if curious to dive in to the weeds on it, the best resource I found online is this free e-book: WebRTC for the Curious.

How WebRTC Basically Works

When two users are logged on to my chat room, and one wants to open the other's camera, the basic ingredients that make the WebRTC magic work includes:

  1. You need a signaling server which is just any server that's able to pass messages back and forth between the two parties, so that they can communicate and negotiate how they'll directly connect to each other.
  2. With the signaling server available, the two clients will pass messages back and forth to negotiate their connection.
    • You don't need to worry too much about this part: the web browsers know what they're talking about and they speak their own language, all your signaling server needs to do is pass these messages along.
    • The two important message types are Session Description Protocol (SDP) where the clients negotiate the features they want (video, audio, codecs support, etc.), and ICE Candidate messages where they negotiate how they'll connect to each other.
  3. The two browsers establish a direct connection between themselves, with possibly video and audio channels enabled through which they can transmit data: the video call can be established.

Signaling Server

The signaling server in WebRTC is much simpler than it sounds: it is really just any server you write which is capable of passing messages along, back and forth between the two parties who want to connect. It could be a WebSocket server, it could be based on AJAX requests to a PHP script, it could even be printed out on a post card and delivered by snail mail (though that way would take the longest).

For my chat room's use case, I already had a signaling server to use: my WebSockets server that drives the rest of the chat room.

The server side of the chat room was a WebSockets server, where users would post their chat messages and the server would broadcast those back out to everybody else, and the server would push "Who's Online" list updates, etc. - so I just added support for this same WebSockets server to allow forwarding WebRTC negotiation messages between the two users.

Terminology: Offerer and Answerer

There are a couple of important terms used in WebRTC that are not super intuitive at first glance.

The two parties of a WebRTC connection are named the Offerer and the Answerer.

  • The Offerer is the one who first decides to initiate the connection. They are "offering to connect" to the other user.
    • On my chat room: this is the person who clicked the button to open your webcam.
  • The Answerer is the other person: they see your offer to connect and they answer it.
    • On my chat room: the answerer is the one whose webcam is active.

Both the Offerer and the Answerer are able to attach data channels to their side of the connection. Most obviously, the Answerer will attach their active webcam feed to the connection, so that the Offerer (who wanted to watch it) is able to receive it and show it on their screen.

The Offerer is also able to attach their own camera to that opening connection, as well, and their video data will be received automatically on the Answerer's side once the connection is established. But, more on that below.

Things learned during the earliest prototype

So, going back to the original design goals of my chat room above, I wanted video sharing to be "asynchronous": it must be possible for Alice, who is not sharing her video, to be able to watch Bob's video in a one-directional manner.

The first interesting thing I learned about WebRTC was that this initially was not working!

  • When Alice created her offer to connect to Bob, she didn't request video or audio channels to be opened, because she was not sharing a video stream of her own on that connection.
    • So, even though Bob did add his video stream to his answer, Alice did not receive it because she didn't negotiate for those channels to be available.
  • However, if Alice turned her webcam on and she attached her video feed to the offer, then those channels were opened (because she would be using them herself), and she did receive Bob's video correctly then.
    • As a very interesting quirk: when this happened, Alice's video automatically opened on Bob's screen as well! Bob did not click to see Alice's video, and yet her video opened itself anyway, because Alice sent her video during the initial offer!

So the conundrum at first, was this: I wanted Alice to be able to receive video, without sharing her own video.

I found that I could do this by setting these parameters on the initial offer that she creates:

pc.createOffer({
    offerToReceiveVideo: true,
    offerToReceiveAudio: true,
});

Then Alice will offer to receive video/audio channels despite not sharing any herself, and this worked OK.

But, I came to find out that this did not work with Safari, but only for Chrome and Firefox!

I learned that there were actually two major iterations of the WebRTC API, and the above hack was only supported by the old, legacy version. Chrome and Firefox were there for that version, so they still support the legacy option, but Safari came later to the game and Safari only implemented the modern WebRTC API, which caused me some problems that I'll get into below.

Safari Problems

So, in February 2023 I officially launched my chat room and it worked perfectly on Firefox, Google Chrome, and every other Chromium based browser in the world (such as MS Edge, Opera, Brave, etc.) - asynchronous webcam connections were working fine, people were able to watch a webcam without needing to share a webcam, because Firefox and Chromium supported the legacy WebRTC API where the above findings were all supported and working well.

But then, there was Safari.

Safari showed a handful of weird quirks, differences and limitations compared to Chrome and Firefox, and the worst part about trying to debug any of this, was that I did not own any Apple device on which I could test Safari and see about getting it to work. All I could do was read online (WebRTC stuff is poorly documented, and there's a lot of inaccurate and outdated information online), blindly try a couple of things, and ask some of my Apple-using friends to test once in a while to see if anything worked.

Slowly, I made some progress here and there and I'll describe what I found.

First, Safari couldn't log into my chat room AT ALL

The first problem with Safari wasn't even about WebRTC yet! Safari did not like my WebSockets server for my chat room.

What I saw when a Safari user tried to connect was: they would connect to the WebSockets server, send their "has entered the room" message, and the chat server would send Safari all the welcome messages (listing the rules of the chat room, etc.), and it would send Safari the "Who's Online" list of current chatters, and... Safari would immediately close the connection and disconnect.

Only to try and reconnect a few seconds later (since the chat web page was programmed to retry the connection a few times). The rest of the chatters online would see the Safari user join/leave, join/leave, join/leave before their chat page gave up trying to connect.

The resolution to this problem turned out to be: Safari did not support compression for WebSockets. The WebSockets library I was using had compression enabled by default. Through some experimentation, I found that if I removed all the server welcome messages and needless "spam", that Safari was able to connect and stay logged on -- however, if I sent a 'long' chat message (of only 500 characters or so), it would cause Safari to disconnect.

The root cause came down to: Safari didn't support WebSocket compression, so I needed to disable compression and then Safari could log on and hang out fine.

So, finally on to the WebRTC parts.

Safari supports only the New WebRTC API

Safari browsers were able to log on to chat now, but the WebRTC stuff simply was not working at all. The Safari user was able to activate their webcam, and they could see their own local video feed on their page, but this part didn't involve WebRTC yet (it was just the Web Media API, accessing their webcam and displaying it in a <video> element on the page). But in my chat room, the Safari user was able to tell the server: "my webcam is on!", and other users would see a clickable video button on the Who List, but when they tried to connect to watch it, nothing happened.

So, as touched on above, WebRTC is an old standard and it had actually gone through two major revisions. Chrome and Firefox were there for both, and they continue to support both versions, but Safari was newer to the game and they only implemented the modern version.

The biggest difference between the old and new API is that functions changed from "callback based" into "promise based", e.g.:

// Old API would have callback functions sent as parameters
pc.setLocalDescription(description, onSuccess, onFailure);

// New API moved to use Promises (".then functions") instead of callback functions
pc.setLocalDescription(description).then(onSuccess).catch(onFailure);

The WebRTC stuff for Safari wasn't working because I needed to change these function calls to be Promise-based instead of the legacy callback function style.

Then, cameras could "sometimes" connect in Safari

By updating to the modern WebRTC API, Safari browsers could sometimes get cameras to connect, but only under some very precise circumstances:

  1. First, the Safari browser needed to turn its own local webcam on.
  2. Then, the Safari browser could sometimes connect to somebody else's camera, but only if that person had the option enabled "When somebody opens my camera, I also open their camera automatically."

This was rather inconvenient and confusing to users, though: the Safari user was never able to passively watch somebody else's camera without their own camera being on, but even when they turned their camera on first, they could only open about half of the other cameras on chat (only the users who wanted to auto-open Safari's camera in return).

This was due to a couple of fundamental issues:

  1. The option to set up a receive-only video offer was only available in the legacy WebRTC API (that offerToReceiveVideo: true option), which Safari did not support.
    • So the only way for Safari, as the offerer, could get a video channel open was by offering its own local video on that connection as well.
  2. However, by offering Safari's local video, it would force that video to open on the other person's screen. This is why I needed to limit it to only people who wanted to automatically open Safari's video, so that it appearing on their screen is expected behavior for them.

For a while, this was the status quo. Users on an iPad or iPhone were encouraged to try switching to a laptop or desktop PC and to use a browser other than Safari if they could.

And only if Safari initiated the connection

There was another bug on my chat room at this point, too: the Safari browser had to be the one to initiate the WebRTC connection for anything to work at all. If somebody else were to click to view Safari's camera, nothing would happen and the connection attempt would time out and show an error.

This one, I found out later, was due to the same "callback-based vs. promise-based" API for WebRTC: I had missed a spot before! The code path where Safari is the answerer and it tries to respond with its SDP message was using the legacy API and so wasn't doing anything, and not giving any error messages to the console either!

Safari only supported two-way video calls?

At this stage, I still had no access to an Apple device to actually test on, so the best I could do was read outdated and inaccurate information online. It seems the subset of software developers who actually work with WebRTC at this low of a level are exceedingly rare (and are all employed by large shops like Zoom who make heavy use of this stuff).

I had found this amazing resource called Guide to WebRTC with Safari in the Wild which documented a lot of Safari's unique quirks regarding WebRTC.

A point I read there was that Safari only supported two-way video calls, where both sides of the connection are needing to exchange video. I thought this would be a hard blocker for me, at the end of the day, and would fly in the face of my "asynchronous webcam support" I wanted of my chat room.

So the above quirky limitations: where Safari needed to have its own camera running, and it needed to attach it on the outgoing WebRTC offer, seemed to be unmoveable truths that I would have to just live with.

And indeed: since Safari didn't support offerToReceiveVideo: true to set up a receive-only video channel, and there was no documentation on what the modern alternative to that option should be, this was seeming to be the case.

But, it turned out even that was outdated misinformation!

A hack to allow Safari to "receive-only" videos from others

Seeing what Safari's limitations appeared to be, in my chat room I attempted a sort of hack, that I called "Apple compatibility mode".

It seemed that the only way Safari could receive video, was to offer its own video on the WebRTC connection. But I wanted Safari to at least, be able to passively watch somebody's camera without needing to send its own video to them too. But if Safari pushed its video on the connection, it would auto-open on the other person's screen!

My hacky solution was to do this:

  • If you (on Firefox) are on webcam, and you do not want to auto-open your viewer's videos, but your viewer gave you a video stream anyway: your page would just ignore their video, and not show it on your screen.
  • For Safari users, then: when they click to watch your video, they would always offer their local video too, so that from Safari's perspective, it was a "two-way video call:" Safari is sending video (which you ignore), and it receives your video in exchange.

But, this is obviously wasteful of everyone's bandwidth, to have Safari stream video out that is just being ignored. So the chat room would only enable this behavior if it detected you were using a Safari browser, or were on an iPad or iPhone, so at least not everybody was sending video wastefully all the time.

And then I bought a Macbook Air

Recently, I broke my old laptop on accident when I spilled a full cup of coffee over its keyboard, and when weighing my options for a replacement PC, I decided to go with a modern Macbook Air with the Apple Silicon M3 chip.

It's my first Apple device in a very long time, and I figured I would have some valid use cases for it now:

  • To be able to actually test my chat room in Safari first hand and debug this nightmare properly.
  • As an aside, to be able to release proper Mac ARM ports of my videogame side project, Sketchy Maze.

The first bug that I root caused and fixed was the one I mentioned just above: when somebody else was trying to connect in to Safari, it wasn't responding. With that bug resolved, I was getting 99% to where I wanted to be with Safari support on my chat room:

  • A user (on e.g. Firefox), who is not on webcam himself, is able to click and open a Safari user's camera and watch it (one-way video).
  • The Safari user (who is on camera themself) was able to open anybody else's video, even those who didn't want to auto-open Safari's back, by always giving them their video anyway and having them ignore it.
  • If the Safari user did open someone's video who wanted to auto-open theirs back, it would work as expected.

The only remaining, unfortunate limitation was: the Safari user always had to have its local webcam shared before it could connect in any direction, because I still didn't know how to set up a receive-only video connection without offering up a video to begin with. This was the last unique quirk that didn't apply to Firefox or Chrome users on chat.

How to actually set up a receive-only video channel in Safari

So, the other day I sat down to properly debug this and get it all working.

I had to find this out from a thorough Google search and landing on a Reddit comment thread where somebody was asking about this question: since the offerToReceiveVideo option was removed from the legacy API and no alternative is documented in the new API, how do you get the WebRTC offerer to request video channels be opened without attaching a video itself?

It turns out the solution is to add what are called "receive-only transceiver" channels to your WebRTC offer.

// So instead of calling addTrack() and attaching a local video:
stream.getTracks().forEach(track => {
    pc.addTrack(track);
});

// You instead add receive-only transceivers:
pc.addTransceiver('video', { direction: 'recvonly' });
pc.addTransceiver('audio', { direction: 'recvonly' });

And now: Safari, while not sharing its own video, is able to open somebody else's camera and receive video in a receive-only fashion!

No more hacks or workarounds needed!

At this point, Safari browsers were behaving perfectly well like Chrome and Firefox were. I also no longer needed that "Apple compatibility mode" hack I mentioned earlier: Safari doesn't need to superfluously force its own video to be sent on the offer, since it can attach a receive-only transciever instead and receive your video normally.

In retrospect, what actually were the issues?

There were really only two quirks about Safari at the end of the day:

  1. Safari only implemented the modern (Promise-based) WebRTC API.
  2. To set up a receive-only video channel for Safari, you needed to add transceivers to the WebRTC offer when the connection begins.

And that second bit ties into the first: the only way I knew initially to get a receive-only video connection was to use the legacy offerToReceiveVideo option which isn't supported in the new API.

And even in Mozilla's MDN docs about createOffer, they point out that offerToReceiveVideo is deprecated but they don't tell you what the new solution is!

Honorable mentions (related rants)

One of the more annoying aspects of this Safari problem had been, that iPad and iPhone users have no choice in their web browser engine.

For every other device, I can tell people: switch to Chrome or Firefox, and the chat works perfectly and webcams connect fine! But this advice doesn't apply to iPads and iPhones, because on iOS, Apple requires that every mobile web browser is actually just Safari under the hood. Chrome and Firefox for iPad are just custom skins around Safari, and they share all its same quirks.

And this is fundamentally because Apple is scared shitless about Progressive Web Apps and how they might compete with their native App Store. Apple makes sure that Safari has limited support for PWAs, and they do not want Google or Mozilla to come along and do it better than them, either. So they enforce that every web browser for iPad or iPhone must use the Safari engine under the hood.

Recently, the EU is putting pressure on Apple about this, and will be forcing them to allow competing web browser engines on their platform (as well as allowing for third-party app stores, and sideloading of apps). I was hopeful that this meant I could just wait this problem out: eventually, Chrome and Firefox can bring their proper engines to iPad and I can tell my users to just switch browsers.

But, Apple isn't going peacefully with this and they'll be playing games with the EU, like: third-party app stores and sideloading will be available only to EU citizens but not the rest of the world. And, if Apple will be forced to allow Chrome and Firefox on, Apple is more keen to take away Progressive Web App support entirely from their platform: they don't want a better browser to out-compete them, so they'd rather cripple their own PWA support and make sure nobody can do so. It seems they may have walked back that decision, but this story is still unfolding so we'll see how it goes.

At any rate: since I figured out Safari's flavor of WebRTC and got it all working anyway, this part of it is a moot point, but I include this section of the post because it was very relevant to my ordeal of the past year or so working on this problem.

Safari wasn't actually quirky after all

Early on with this ordeal, I was thinking that Safari's implementation of WebRTC was quirky and contrarian just because they had different goals or ideas about WebRTC. For example, the seeming "two-way video calls only" requirement appeared to me like a lack of imagination on Apple's part: like they only envisioned FaceTime style, one-on-one video calls (or maybe group calls, Zoom style, where every camera is enabled), and that use cases such as receive-only or send-only video channels were just not supported for unknowable reasons.

But, having gotten to the bottom of it, it turns out that actually Safari was following the upstream WebRTC standard to a tee. They weren't there for the legacy WebRTC API like Firefox and Chrome were, so they had never implemented the legacy API; by the time Safari got on board, the modern API was out and that's what they went with.

The rest of it came down to my own lack of understanding combined with loads of outdated misinformation online about this stuff!

Safari's lack of compression support for WebSockets, however, I still hold against them for now. 😉

Further Reading

If you ever have the misfortune one day to work with WebRTC at a low level like I have, here are a couple of the best resources I had found along the way:

Tags: 1 comment | Permalink
Dream Journal: Healthcare Robot
March 17, 2024 by Noah

I frequently have pretty vivid, detailed and crazy dreams and when I have a particularly interesting one, I write it down in my dream journal. The one I awoke from this morning left such a strong impression on me that I also feel like sharing it here on my blog!

It involves artificial intelligence, humanoid robots and morally bankrupt billionaire corporations. I like to think I can tell a good story, so I hope you'll check this one out!


So at the start of this dream, I had received a mysterious package in the mail. It was a medium sized box, maybe three feet tall, and had no return address but it was addressed to me, and it wasn't anything that I ordered myself.

Inside the box was a humanoid robot, covered in sleek white panels and looking like something that Apple might design. I powered it on, and it was able to speak with me in plain English and seemed to have a level of intelligence on par with ChatGPT. It had arms and legs and a robot face and everything. Presumably it could walk and move around as well. When it would enter an idle mode (maybe to recharge), it would "turtle up" and retreat into its shell, taking up less space in the room, a few feet tall. When standing, it was the size of a small adult human, maybe 5 feet tall or so.

An A.I. generated image of what this sleek humanoid robot may have looked like. The photo depicts a robot covered with shiny white plastic, a blank white face, and arms and legs standing in front of a cyberpunk green circuit board background texture.

An A.I. generated image of what this robot may have looked like, created on deepai.org.

On a different day in this dream, I was hanging out in the living room and I had a couple of friends over, and the robot was in its idle pose nearby. Suddenly, the robot whirred to life, and panels on its body opened up, and then I can only describe that it 'attacked' me. Out of these panels, a bunch of wires shot out quickly, some with needles on the ends, which all found their way to my body. One such needle went into my arm, into a vein, reminiscient of when you are put on an IV drip at a hospital. I don't remember where the other wires landed.

The robot then injected me with something: tiny little machines. Some of them assembled themselves together inside my body, forming a larger disc shaped structure a couple of inches across, and I could see it move around under my skin and could feel it if I touched it. Most of the other tiny machines navigated their way through my veins, arteries and other ducts in my body.

I could feel all of these things moving around, and especially around my lower abdomen. I've never experienced having kidney stones before, but I imagine it could be similar, though instead of painful, the movement felt ticklish and highly sensitive, similar to post-orgasm sensitivity, as they worked their way around my body.

Naturally, I was freaking the fuck out, but I was also afraid to respond too violently to all of this in case I broke the robot and left its work in a half-complete state, which could result in a much worse outcome for me. My friends in the room with me were watching all this happen and they were shocked and didn't know what to do either.

During this time, the robot was unresponsive and didn't respond to any voice commands. It was in its idle pose, but clearly busy computing something, with lights on its chassis flashing and everything.

After no more than 10 minutes or so, the robot was done and it disengaged from my body, retracting all the wires and tools back into itself, closing its panels and then transforming into its humanoid social form to talk to me.

Of course, I asked it what the fuck just happened. It explained to me that it is a healthcare robot and it had just finished giving me an examination. It told me I had some plaque buildup in my arteries that it cleaned out for me, and that while my kidneys were in the range of normal healthy function, they were a bit on the lower end of that range (whatever that meant).

Naturally, I had a whole lot of questions. This robot was made from some highly sophisticated technology that I had never seen the likes of before.

I asked it who built it. The robot replied that it was created by a major multi-billion dollar pharmaceutical company. It told me the name of the company, though I don't remember it now.

I asked it why it was delivered to my house. It explained that the company who created it was facing some problems: they weren't cleared yet to begin human trials on the new technology, and they were facing fines or regulations and were at risk of going bankrupt or forced to shut down. They mailed these units out for free to random people as a way to circumvent the law, get some human test subjects, and (hopefully) prove that the machine is safe and effective. I was just one of the "lucky" ones who received one of these units.

I asked the robot whether I can opt-out of their test program, and the robot said I could not. In the room with us was a small figurine of Baymax from Big Hero 6. I picked it up and showed it to the robot and said: "in the movie Big Hero 6, they had healthcare robots and you could say to the robot 'I am satisfied with my care' and they will power down and stay that way" and the robot told me that that wouldn't work on it, and that it was designed for long-term active care.

I asked the robot whether I could tell people about all of this - I had a good mind to, at the very least, write about this experience on my blog. It strongly advised against me telling anybody about it. It didn't threaten me or tell me what would happen if I disobeyed, and I know I never signed a non-disclosure agreement about this, but given the highly advanced technology it had just displayed for me, I decided I would comply with it - at least for now. But I had found the whole thing highly sketchy and unethical and was beginning to contemplate what my next steps should be, before finally waking up from this dream.

Tags: 0 comments | Permalink
What is your favorite memory of Bot-Depot?
December 4, 2023 by Noah

Anonymous asks:

What is your favorite memory of Bot-Depot?

Oh, that's a name I haven't heard in a long time! I have many fond memories of Bot-Depot but one immediately jumped to mind which I'll write about below.

For some context for others: Bot-Depot was a forum site about chatbots that was most active somewhere in the range of the year 2002-08 or so, during what (I call) the "first wave of chatbots" (it was the first wave I lived through, anyway) - there was an active community of botmasters writing chatbots for the likes of AOL Instant Messenger, MSN Messenger, Yahoo! Messenger, and similar apps around that time (also ICQ, IRC, and anything else we could connect a bot to). The "second wave" was when Facebook Messenger, Microsoft Bot Platform, and so on came and left around the year 2016 or so.

My favorite memory about Bot-Depot was in how collaborative and innovative everyone was there: several of us had our own open source chatbot projects, which we'd release on the forum for others to download and use, and we'd learn from each other's code and make better and bigger chatbot programs. Some of my old chatbots have their code available at https://github.com/aichaos/graveyard, with the Juggernaut and Leviathan bots being some of my biggest that were written at the height of the Bot-Depot craze. Many of those programs aren't very useful anymore, since all the instant messengers they connected to no longer exist, and hence I put them up on a git repo named "graveyard" for "where chatbots from 2004 go to die" to archive their code but not forget these projects.

Most of the bots on Bot-Depot were written in Perl, and one particular chatbot I found interesting (and learned a "mind blowing" trick I could apply to my own bots) was a program called Andromeda written by Eric256, because it laid down a really cool pattern for how we could better collaborate on "plugins" or "commands" for our bots.

Many of the Bot-Depot bots would use some kind of general reply engine (like my RiveScript), and they'd also have "commands" like you could type /weather or /jokes in a message and it would run some custom bit of Perl code to do something useful separately from the regular reply engine. Before Andromeda gave us a better idea how to manage these, commands were a little tedious to manage: we'd often put a /help or /menu command in our bots, where we'd manually write a list of commands to let the users know what's available, and if we added a new command we'd have to update our /help command to mention it there.

Perl is a dynamic language that can import new Perl code at runtime, so we'd usually have a "commands" folder on disk, and the bot would look in that folder and require() everything in there when it starts up, so adding a new command was as easy as dropping a new .pl file in that folder; but if we forgot to update the /help command, users wouldn't know about the new command. Most of the time, when you write a Perl module that you expect to be imported, you would end the module with a line of code like this:

1;

And that's because: in Perl when you write a statement like require "./commands/help.pl"; Perl would load that code and expect the final statement of that code to be something truthy; if you forgot the "1;" at the end, Perl would throw an error saying it couldn't import the module because it didn't end in a truthy statement. So me and many others thought of the "1;" as just standard required boilerplate that you always needed when you want to import Perl modules into your program.

What the Andromeda bot showed us, though, is that you can use other 'truthy' objects in place of the "1;" and the calling program can get the value out of require(). So, Andromeda set down a pattern of having "self-documenting commands" where your command script might look something like:

# The command function itself, e.g. for a "/random 100" command that would
# generate a random number between 0 and 100.
sub random {
    my ($bot, $username, $message) = @_;
    my $result = int(rand($message));
    return "Your random number is: $result";
}

# Normally, the "1;" would go here so the script can be imported, but instead
# of the "1;" you could return a hash map that describes this command:
{
    command => "/random",
    usage => "/random [number]",
    example => "/random 100",
    description => "Pick a random number between 0 and the number you provide.",
    author => "Kirsle",
};

The chatbot program, then, when it imports your folder full of commands, it would collect these self-documenting objects from the require statements, like

# A hash map of commands to their descriptions
my %commands = ();

# Load all the command scripts from disk
foreach my $filename (<./commands/*.pl>) {
    my $info = require $filename;

    # Store their descriptions related to the command itself
    $commands{ $info->{'command'} } = $info;
}

And: now your /help or /menu command could be written to be dynamic, having it loop over all the loaded commands and automatically come up with the list of commands (with examples and descriptions) for the user. Then: to add a new command to your bot, all you do is drop the .pl file into the right folder and restart your bot and your /help command automatically tells a user about the new command!

For an example: in my Leviathan bot I had a "guess my number" game in the commands folder: https://github.com/aichaos/graveyard/blob/master/Leviathan/commands/games/guess.pl

Or a fortune cookie command: https://github.com/aichaos/graveyard/blob/master/Leviathan/commands/funstuff/fortune.pl

After we saw Andromeda set the pattern for self-documenting commands like this, I applied it to my own bots; members on Bot-Depot would pick one bot program or another that they liked, and then the community around that program would write their own commands and put them up for download and users could easily download and drop the .pl file into the right folder and easily add the command to their own bots!

I think there was some effort to make a common interface for commands so they could be shared between types of chatbot programs, too; but this level of collaboration and innovation on Bot-Depot is something I've rarely seen anywhere else since then.

We had also gone on to apply that pattern to instant messenger interfaces and chatbot brains, as well - so, Leviathan had a "brains" folder which followed a similar pattern: it came with a lot of options for A.I. engine to power your bot's general responses with, including Chatbot::Alpha (the precursor to my RiveScript), Chatbot::Eliza (an implementation of the classic 1970s ELIZA bot), and a handful of other odds and ends - designed in a "pluggable", self-documenting way where somebody could contribute a new type of brain for Leviathan and users could just drop a .pl file into the right folder and use it immediately. Some of our bots had similar interfaces for the instant messengers (AIM, MSN, YMSG, etc.) - so if somebody wanted to add something new and esoteric, like a CyanChat client, they could do so in a way that it was easily shareable with other botmasters.

For more nostalgic reading, a long time ago I wrote a blog post about SmarterChild and other AOL chatbots from around this time. I was meaning to follow up with an article about MSN Messenger but had never gotten around to it!

Tags: 2 comments | Permalink
What we once had (at the height of the XMPP era of the Internet)
August 4, 2023 by Noah

A discussion thread I got pulled into on Mastodon had me suddenly nostalgic for something we once had on the Internet, and which was really nice while it lasted: the Extensible Messaging and Presence Protocol, or XMPP, or sometimes referred to by its original name, Jabber.

What kicked off the discussion was somebody asking about a more modern "decentralized chat platform" known as Matrix (not the movie). A lot of people online talk about how they like Matrix, but I one time had a go at self-hosting my own Matrix node (which I feel I should rant about briefly, below), and the discussion turned back towards something that we used to have which was (in my opinion) loads better, XMPP.

Read more...

Tags: 5 comments | Permalink
Lucid dreams and exploring my consciousness
July 8, 2023 by Noah

One of the many things I get really interested in is the nature of "reality" or at least the conscious parts of it (e.g. my perception of reality). The mind is such a curious and powerful thing -- like how none of us has ever actually 'seen' Objective Reality, because our brains are trapped in our dark skulls and all we really get are electrical signals sent by our eyes and other sensors. The brain then makes up all the stuff you see and call reality: a multimedia simulation based on what your brain thinks is going on out there in the world. We've all heard questions like "is my green, your red?" and how you'll never know for sure what somebody else sees in their reality.

In recent years I have been getting into meditation and really paying attention, really studying the way my mind works and I have found a few interesting things that I haven't seen talked about much online. I am currently suspicious that there is a connection between lucid dreaming, closed-eye visuals during wakeful consciousness, and the place in between the two: psychedelic visuals, which you can apparently just meditate your way into seeing, no entheogens required.

Closed eye visuals

Let's start with the easy one. When I close my eyes and look into the darkness of my eyelids, I frequently will see some shapes and colors. From what I've heard from other people, this seems fairly common - you might see these too.

My regular closed-eye visuals tend to just be vague, formless blobs of color, maybe a blue blob here and a green one there. All kind of meandering around on their own, but very low resolution. Maybe they might vaguely form a face or some recognizable shape, but just loosely -- like seeing shapes in the clouds -- nothing very exciting.

But spoiler alert: they can get much more exciting.

Closed-eye visuals when I wake up from dreaming are INTENSE

One night, I woke up from dreaming at maybe 3:00 in the morning (who knows), to get up and use the bathroom as one does. When I got back in bed I closed my eyes and I decided to look for my closed eye visuals (often if I'm not deliberately trying to see them, they go unnoticed by me and I think I just "see" blackness with my eyes closed).

I was expecting to see the usual amorphous blobs, but what I instead saw blew me away: my closed eye visuals (henceforth called CEVs) were breathtakingly sharp, vivid, colorful and crisp. I was seeing a colleidoscope of random images flashing before my eyes. It reminded me a lot of the Marvel movie opening logo. But every image I saw was just so crystal clear, with sharp lines and edges and vivid colors -- a far cry from the usual vague shapes my CEVs take on during the daytime.

I attribute this to the fact I had woken up from dreaming (REM sleep) and my brain was still close to that dream state. I have a suspicion that CEVs have a whole spectrum of forms they take: the vague blobbies is on one end of the spectrum, and full-color life-like hallucinations that we call "dreams" are at the opposite end.

And in between those two poles are other states that some people are familiar with.

I learned how to meditate my way into seeing psychedelic visuals

A couple of years ago, quite by accident, I managed to meditate my way into seeing psychedelic visuals over top of reality. And you can probably learn to do this, too. I will save you the long story (I had written about it before if curious) and cut straight to the chase -- here is how I've been able to consistently pull this off on command.

The trick is to just lock your eyes onto as fine a point as you can manage, and resist the urge to twitch your eyes even as your vision begins to play tricks on you. Many of you have probably experienced this much already: you know if you stare at a thing for too long, your vision plays tricks and things start disappearing or acting weird because of the sensory satiation, and darting your eyes will "reset" your vision to normal. But try and resist the urge to dart your eyes and just let your visuals do what they shall.

What happens next for me is: I will begin to see my closed eye visuals project themselves over top of reality, with my eyes open. The blobby, amorphous shapes I see with my eyes closed appear while they're open. But then my CEVs will begin to "play off of" or interact with my reality: where the border of one of the blobs touched a ridge on my ceiling that I was staring at, the ridge would wiggle or bounce back, as if my CEVs are interacting with it.

These visuals slowly ramp up and I'll begin seeing geometric patterns, fractals, lots of colors, and as these fractals interact with the scene I am staring at, they get everything to start wiggling and dancing. It can take me about 30 minutes into my meditation before I am seeing fully, 100% authentic, height of the strongest acid or mushroom trip I'd ever been on, visuals projected out onto whatever I was staring at during my meditation.

And when I've decided I had enough: I just look somewhere else and snap! everything is back in its place, in much the same way that darting your eyes during the earliest part of all of that "resets" your vision to normal.

I have recently learned how to continue my dreams after I wake up

Tell me if this sounds familiar: when I was a kid I would sometimes be woken up from a really good dream and wished I could have stayed in my dream a little bit longer to see how it ended. I could try and "make up" the ending myself, daydream how I think it might have gone, but it never felt right -- when you're in a dream, it's like your subconscious is telling you a story, and you can't quite predict what's going to come next. There's something magical about dreams that when I try and make up the ending myself, it doesn't come out right.

Well -- and I don't know exactly when this started, but it was in recent months at least -- I have somehow gained the ability to let my imagination wander freely and I can actually let my dreams finish themselves out, autonomously, after I wake up and I'll be fully aware I'm awake (and feel my body in bed, be able to hear my surroundings) but still visually see and hear the dream at the same time.

I attribute this to my practice with meditation -- I don't know of concrete steps that someone could try (not like my previous section) but I suspect the ability to "let go" of your mind, or to just watch it and not mess with it, was a feature here. I'll give you an example of what I mean.

A couple years ago I was reading some of Carl Jung and he talked about "active imagination" where he was able to visualize a scene in his head with characters in it, and then set his imagination free, and he could interrogate the characters and ask them questions and he would get responses from them that he couldn't predict (in very much the same way as dream characters). It's different to regular conscious daydreaming where you are controlling the dialogue of all the characters. It's more like when you are dreaming, and your dream characters talk to you and you can't quite predict what they're going to say because it's a different part of your mind that controls them: your subconscious.

After reading this I was trying, and failing, to do active imagination myself. I could set up the scene, and ask my characters a question, but I always felt like it was I who was dictating their responses -- I was just daydreaming consciously with myself. I haven't yet tried active imagination again, but this ability to let my subconscious wander and finish my dreams out seems to be very close to what I was thinking here.

Lucid dreaming

There's already a lot of literature out there about lucid dreaming so I won't go too deep here.

Lucid dreaming is basically when you become aware that you're dreaming. You may be having some crazy dream about something nonsensical happening and have the epiphany: "oh shit, this is a dream!" and you can have a lot of fun with it. You can decide to change anything you want about the dream, or fly through the air like Superman, or summon specific dream characters that you want to talk to. I've had a good handful of lucid dreams in my life, but never very consistently. In many cases I'll realize I'm dreaming, and then just let my dream tell its story without meddling with it too much. I've found that if I meddle too much, my dream "gives up" on its story and just gives me all the controls, and it becomes a little bit less fun for me. I have "god mode" control but much less of the dream is unpredictable or surprising then (though my dream characters are still unpredictable in what they say).

From talking to my dream characters and asking them deep and personal questions about myself, and getting surprising answers back from them, I am suspicious that talking to a dream character is the same as talking directly to my inner subconscious, in plain English.

That's all for now!

Just thought I'd jot down some of my latest findings. I have told some friends about these and some of them have experienced similar things, if not to the same extent I have yet. I hope this may have sparked your curiosity and if others may be able to reproduce some of my findings themselves.

I have more posts like this on my #42 blog tag where I write about life, the universe, and everything. It's a lot of stuff about consciousness and occasionally spiritual stuff (the two have many likes alike, and all the characters and archetypes that Jung says are in our consciousness have mirrors in religious figures from all cultures which I also find very fascinating!)

Tags: 0 comments | Permalink
Is Safari the new Internet Explorer - or worse?
March 30, 2023 by Noah

So I haven't posted a good rant on my blog in quite some time - I had chilled out a lot in my older years, but I just have to tell this story about Safari and my struggles in getting it to work with my chat room I had built recently.

My chat room is a fairly basic app - it's open source and written "from scratch" using WebSockets to handle the client/server communication and they pass basic JSON messages around. All fairly standard stuff and shouldn't be a big ask for any modern web browser that supports modern web standards. It works flawlessly in Google Chrome as well as all other Chromium derivatives (including Edge, Brave, etc.), and it works flawlessly on Firefox, and when you run either browser on your Windows, Linux or Mac OS computer. It even works great on all Androids, too - using Chromium or Firefox browser engines.

But then there's Safari. Safari "supports" WebSockets but it doesn't do so very well and I've been fighting this for weeks now trying to chip away at this problem. Both when you run Safari on your Mac OS Ventura desktop or on your iPhone or iPad, Safari is very easy to overwhelm and it will disconnect from the chat room at the slightest hiccup of an issue.

My rant here actually is about three different problems that have made my life difficult trying to get Safari to work:

  1. The way that Safari just has to "do things differently" and have kneecapped limitations and quirky bugs that Chrome and Firefox don't experience at all.
  2. The way that all web browsers on iOS are really just Safari and so iPhones and iPads simply can not use my chat room regardless of browser.
  3. The way Apple designed their closed ecosystem where, if I want to debug this on iOS and see what kind of error message Safari is even throwing, I am required to purchase an iPad + a Macbook because only Apple's hardware can do iOS development and pair with one another (where by comparison, I can debug an Android browser using any kind of PC I want).

Safari and WebSockets: "Protocol errors"

I don't own a Macbook or an iOS device and so would have no way to even debug or look into this problem, but at least there is an option to run Mac OS inside of a virtual machine (using something like OSX-KVM) so at least I can look into the desktop Safari browser and see what its deal is.

First - here is how my chat basically works: you connect, send your username to log in, the chat tells everyone you joined, sends everyone the new Who's Online roster, and sends some "welcome messages" to the one who joined where I can send my rules or new feature announcements to you.

What I would see when a Safari user logged in was: they'd connect, log in, receive all those welcome messages and then they would immediately hangup the connection and log off. On their side, the web browser gives a very unhelpful error message regarding the WebSocket:

Protocol error.

That's it - it doesn't say what the error was. Even when I have a Safari browser in front of me, they give me no help at all to find out what's wrong!

Through trial and error, I found out:

  • If I remove all the welcome messages (to reduce the noise I send to Safari on join), it was able to log in to chat OK and send and receive messages!
  • From more poking, I found that there seems to be a length limitation!? And I'm not even talking about huge messages either - if I just type about 500 characters of text and send it, Safari will immediately disconnect with "Protocol error"!
  • So I also root caused: in those welcome messages, where I listed some rules or new features, those messages were "too long" for Safari and scared it away!

...and that kind of thoroughly sucks. I can remove all the welcome messages so as to allow Safari to at least log on, but then just one user posting a paragraph of text will kick all the Safari users out of the chat room!

Chrome and Firefox don't experience this issue. A while ago I added picture sharing in chat - you send your picture over WebSocket and it echoes it as a data: URL to all the chatters, passing those bytes directly back out; Firefox and Chrome can cope with this perfectly, but that would for sure kick Safari users off for too long of a message!

All web browsers on iOS are Safari

So, when it comes to Mac OS users I can tell them: Chrome and Firefox work better. But this advice does not fly on iPads or iPhones because, per Apple's rules, web browsers on iOS are not allowed to bring their own browser engines - they all are just custom wrappers around Mobile Safari.

And you guessed it: Mobile Safari doesn't like my WebSockets either!

I am hoping that with EU pressure placed on Apple to where they will allow competing browser engines to join their platform, that at least some of my iOS users will have a way to use my chat room. But how it stands currently is: iPads and iPhones simply can't use my chat room at all, or if they can get on (maybe it's the "too long of message" issue with them as well), they will be fragile and easy to boot offline just by writing too long of a message.

Apple will not innovate on Safari and make it support modern web standards, and they've made sure that nobody else can innovate either. The Blink engine from Chromium and Gecko from Firefox both work great with WebSockets and if only they were allowed to exist in their true forms on Apple mobiles, I wouldn't be ranting about this right now, I could just say "don't use Safari."

And side rant: the reason Apple won't allow Chrome or Firefox to compete with them is because they are scared shitless about Progressive Web Apps, which could compete with their native app store. They won't innovate on Safari at all until their feet are held to the fire (thanks for that as well, EU!), their web browser sucks (as this WebSockets thing illustrates), they're holding everybody back - the new Internet Explorer 6.0!

Vendor lock-in means I can't even debug this properly

Even if I owned an iPad, it wouldn't help - you can't get to the browser logs on an iPad browser to even see what kind of error message it's throwing. Though I imagine the error message would be just as helpful as desktop Safari, anyway: "Protocol error."

In order to get logs off an iOS web browser, you need to pair it with a Macbook computer -- the only kind of device that is allowed to run the iOS development kits and is the only way to debug anything that happens on your iPad.

With Mac OS, at least there is a way I can spin up a virtual machine and get my hands on Safari. There is no way to do this for iOS. There's no virtual machine where I can run the genuine Mobile Safari app and have a look at this issue myself. I wish Apple weren't so closed in their ecosystem - comparing it to Android for example, you can debug an Android app using any kind of computer: Windows, Linux, Mac OS, even another Android, even on-device on the same Android.

I am not an iOS developer, I don't care to be an iOS developer, and I don't own any Apple hardware, and it really sucks when I run into a web app issue that should have nothing to do with Apple specific software, and I simply can not even get in and look at the error messages.

I'd been chipping away at this for weeks, basically blindly trying things and throwing shit at the walls and hoping one of my Apple using friends tries my chat room once in a while to see if any of my efforts have worked (spoiler: they haven't worked).

I've also tried reaching out to developers on Mastodon and other social media: my chat room is open source, could somebody with Apple hardware help me debug and see if they can find out what I can do better to get Safari to like my chat room. Maybe I didn't reach the right ears because I've gotten only crickets in response. Meanwhile about a third of my users still can not get onto my chat room at all.

Where I've landed so far is: it seems Safari can connect, but that < 500 character limit issue seems horribly broken and I don't want Safari users getting kicked off left and right by surprise, it's a bad user experience.

Maybe I'll just wait until Chrome and Firefox can come to iOS properly

If I wait it out long enough (and if the EU is successful), Apple may permit actually good web browser engines on their platform and then my chat will work perfectly on them. It may just take a couple of years though - first Apple would have to be successfully sued into submission, and then Google/Mozilla would have to port their browser engines over which could be its own long tail of development work.

Maybe as a consequence of that regulation, Apple will actually put some new work into Safari instead of just neglecting it and they'll fix their shit. Currently it seems like my WebSocket messages need to be smaller than a tweet on Twitter, and it honestly won't be worth all the effort it would take me to reimagine my whole chat room and come up with a clever protocol to break messages apart into tiny bite sized chunks when it's only one, non-standards compliant web browser, which drags its stick thru the mud as badly as Internet Explorer ever did, that has the issue here.

So are they the new Internet Explorer, or somehow even worse?


I have sometimes had people visit my blog because they were Googling around in general about Apple and they like to fight me in the comments because I dissed their favorite brand. If this is you, at least say something constructive in the comments: have you ever built a WebSockets app and do you have experience to share in how to get Safari to behave? If you're simply an end user fanboy and never installed Xcode in your life, save your comments, please.

Tags: 1 comment | Permalink
BotW Save Hacking Question
February 13, 2023 by Noah

Cardell asks:

So this is related to The Breath of the Wild save hacking. I play on the Yuzu emulator now. I just want the Mster Cycle earlyish and fresh vanilla game. Have found a save file editor but it doesn't let you add runes so I can't get the cycle. How did you add it to your save files on the blog? I'm sorry to bother you with something so insignificant but what I'm trying to do is so niche I can't find anything lol

I'm not sure about the Switch version but may have some ideas to get you started.

The savegame editor I used may be for the WiiU only and it provided a simple "Motorcycle" checkbox that adds the rune. If the Switch save editor doesn't do that, but does give you 'advanced' access to the various game flags, maybe setting the flag IsGet_Obj_Motorcycle may do it - the flag is set in my WiiU save files. Or otherwise browse and search around in the flags - there are a lot of them that control all sorts of features in the game, and sometimes they have uintuitive names (IsGet_Obj_RemoteBomb does the bomb rune, IsGet_Obj_StopTimer is the stasis rune, IsGet_Obj_Magnetglove is the magnesis rune, IsGet_Obj_IceMaker for cryonis).

There is also a caveat around runes at all when you're talking very early game: you can hack all these runes in to your profile, but you can't use them in-game until you have picked up a rune yourself. The D-pad shortcut doesn't work otherwise. What I did for my WiiU save files was to:

  1. Mod my game to add all runes + motorcycle + paraglider so I could leave the Plateau immediately.
  2. Do the main quests in Kakariko and Hateno Village until I get the Camera rune from Pura.

With the Camera rune added, the D-pad shortcut unlocks and I can now use all of the runes, including the motorcycle, despite never entering one of the Plateau tutorial shrines. (Doing those tutorial shrines would also unlock your rune ability - you just need a rune added "normally" for the tutorial to advance enough to let you use them!)

Good luck!

Tags: 1 comment | Permalink
An open source video chat room
February 7, 2023 (updated May 2, 2024) by Noah

I'm announcing my latest open source project: a WebRTC chat room with asynchronous video support that I'm calling BareRTC. It is very much styled after the classic old-school Flash-based chat rooms that were popular in the early 2000's.

I have a demo of it available here: https://chat.kirsle.net/. I can't guarantee I'll be lurking in that room at any given time but you can test it out yourself on a couple of devices or send the link to some friends and see how it works.

Its primary features are:

  • Classic text chat room with public channels and Direct Messages.
  • Users may broadcast their webcam and microphone, and other users may watch any number of such broadcasts from other users that they are interested in, in an asynchronous manner.
  • It plugs in to your existing userbase easily via signed JSON Web Tokens, or (by default) guests can pick their own username and join in.

The back-end is written in Go and it should easily install onto any server. With WebRTC, the webcam streams are peer-to-peer so you don't need a server in the middle to relay all that video bandwidth (which could be expensive!). Here is a screenshot from the source repo:

Screenshot of BareRTC

In this blog post I'll talk a bit about the technicals and difficulties I ran into getting this app to work.

Why build my own chat room?

One of my side projects is nonshy, a social networking site for nudists and exhibitionists. I wanted a chat room for it and I wanted it to work as described above: where some users can be on video but the room isn't all about video and is primarily a text chat room, as not everybody has a camera or the will to be broadcasting on it at all times.

I really expected that a chat room like this should have already existed out there as a free and open source project that I could plug in to my site. There were so many Flash-based rooms just like this in the early 2000's, and WebRTC is a standard web technology now, there are lots of open source apps and libraries that use WebRTC, but none of them worked like these classic old chat rooms did. Most WebRTC apps are in the style of Zoom or Jitsi, where it's expected that all users will be on video.

I didn't want to build my own chat room, but as there was nothing suitable for me to use I had to do it myself. It was a fun learning process to play with WebRTC for the first time.

My chat app now fills the void of such a thing not existing. I didn't want to build it specifically for my social networking site, so it's a stand-alone app that plugs in to any existing userbase by letting your existing site sign a JWT token to authenticate users in. With the JWT you can also convey a profile picture image, profile URL and admin/operator status to the user as they enter the chat.

I named it "BareRTC" for the punny word play: I was "grumpy like a bear" that I had to program this damn thing myself and for the play on words that the itch I was scratching was for a nudist website in particular.

What is WebRTC?

WebRTC is a web standard that allows two browsers to connect to each other (peer-to-peer, ideally) and transmit arbitrary data between them -- usually, video and audio data. Most modern video apps including Zoom, Jitsi Meet, Discord and others are using WebRTC to get video chat to work in a web browser.

The great thing about WebRTC is that it's peer-to-peer so you don't need a server to relay video frames back and forth between users -- saving you a lot of money in bandwidth costs. WebRTC usually works even if both sides of the connection are behind firewalls (such as most home users with a WiFi router that uses NAT). In case neither side can connect to the other, "enterprisey" WebRTC applications will fall back on using a server in the middle to relay video frames back and forth (called a TURN server). My app does not support TURN servers yet, but peer-to-peer video usually works in "most" cases.

How does WebRTC work?

These are the things I learned along the way, as this is my very first WebRTC project.

First, the two web browsers need a way to negotiate how they'll connect to one another and what features they will support (data channels, video or audio streams, and which codecs to use with those, and so on). To handle this initial negotiation process, you need a signaling server which is just any server that can echo messages back and forth between the two clients.

My chat room (before adding video support) already had a signaling server: I am using WebSockets to handle the server side of the chat protocol. The web page front-end connects to the WebSockets server to log in with a username and send and receive chat messages.

So I have two users "alice" and "bob" who want to connect peer-to-peer and share video. I programmed my WebSocket server to also be the signaling server for WebRTC: Alice sends a message to my server meant for Bob and my server forwards it along, and the two of them share "ICE candidates" and "SDP messages" back and forth which is how they negotiate how they'll connect to each other (ICE) and what features they will support (SDP). This is the easy part of WebRTC: your back-end signaling server just gives them a method to relay these to each other and then the two browsers have everything they need to (hopefully) establish a peer-to-peer connection directly between themselves.

Asynchronous video

The part I was getting stuck on the most was that my chat room's video feature is asynchronous: one user is broadcasting a video, and the one who wants to tune in to that might not be broadcasting their own. I could see the ICE and SDP messages being sent between the two users but then nothing would happen: the receiving party was not getting the video streams sent by the caster.

But if the receiver was also casting their own video, the two could connect find and see each other's video!

I found out this is because in the SDP negotiation process, the one connecting (the "offerer") negotiates what features they expect to share, and if they are not sharing their video, they don't request video support; and so the "answerer", even though they added their video streams to the connection, their video is not actually delivered to the offerer.

The fix was that when you called createOffer() you had to say {offerToReceiveVideo: true, offerToReceiveAudio: true} and then all is well.

A good and simple barebones WebRTC example

A crucial part to my success building this app was this barebones WebRTC video chat demo.

It's only 119 lines of basic JavaScript code that quickly gets two browsers into a video call with each other. It uses a service called ScaleDrone for the signaling server (simple channel to allow the ICE/SDP messages to relay between the two browsers). I do not use ScaleDrone in my app but I could see how the JavaScript used it and have my own WebSocket server do the same for my needs.

Some of the quirks that I had to deal with comparing my app to this example were:

  • This example, on page load, immediately requests the webcam and mic of the user and establishes the WebRTC connection immediately.
  • My chat room however, only establishes the WebRTC connection when a user clicks to see another's camera. (Opening your own camera and awaiting viewers, no WebRTC is done yet, just simple HTML Video APIs).
  • In the example, both sides are transmitting video so they both added their streams. I already described the issue above with my chat room's async video feature.

Open source

My chat room app is released under the GNU General Public License v3 and if you wanted a chat room like this for your website, you're free to check my project out! At some point I may put a mirror to my code on GitHub to allow social development and pull requests for others if you'd like to help improve the feature set.

Check out the source at https://git.kirsle.net/apps/BareRTC

Tags: 0 comments | Permalink
Error Message Generator 2.0
January 22, 2023 by Noah

I have just released a new toy program on my website: Error Message Generator 2.0, or ErrorGen for short.

ErrorGen is a simple program that lets a user configure customized error dialog pop-ups, with a custom icon, message and buttons, to prank their friends with or to put to good use by shell scripts if you want to ask the user a quick question from your external program.

My original ErrorGen was inspired by a web tool called "Atom Smasher's Error Message Generator" which would produce images of error dialogs that you could save to disk. My program, however, created "real" dialogs on your desktop PC that you could drag around the screen and interact with. The original version was written using Perl/Tk in 2006 and hasn't been updated a lot since - with the latest release built in 2008 for Windows XP and it hasn't aged well and doesn't run as easily on modern Windows anymore.

In 2022, Atom Smasher's page went offline and I have seen an uptick of interest in my old ErrorGen program ever since: it is recently the #1 most requested page on my website!

So, on January 21, 2023 I decided to reinvent my ErrorGen program from scratch, this time programming it in Go and to explore the Fyne UI toolkit which I had seen around but hadn't played with before. ErrorGen 2.0 has equivalent features to what my original Perl version had, but with a fresh and modern look based on Material Design that comes with Fyne and built for the modern era. I also have some plans to extend ErrorGen 2.0 with new features and especially make it more useful for command line interfaces, to make something on par with GNOME's Zenity tool.

Screenshot of ErrorGen 2.0

You can check out the new ErrorGen on my Error Message Generator page. The classic Perl version from 2006 is still available here if you want it.

Tags: 4 comments | Permalink
Is ZenMsg a virus?
January 2, 2023 (updated January 2, 2023) by Noah

Is Zenmsg a virus? asks:

It only opened a command box, and nothing else happened. Is this a virus?

No, it's not. 😊 ZenMsg is a command-line program (so it opens your DOS prompt if double-clicked on), but it requires command-line options to tell it what to do. When run without any options, it prints its usage information to the terminal and then exits; so when double-clicked on, your DOS prompt appeared and then closed because ZenMsg exited.

You'll want to run it from a PowerShell or Command Prompt window first (so that the console sticks around after the program exits), and you can see it just prints its usage information:

C:\Users\Noah\Downloads> ZenMsg.exe
Usage:
      ZenMsg [--error --alert --info --question]
             [--title] [--text] [--button label]
             [--icon name_or_file]
             [--default label] [--cancel label]
             [--disabled n]
             [--version] [--help]

    Use the "--help" option for more help.

(and it goes into detail on all the options)

If you call it with ZenMsg --help it goes into full detail (the same documentation that's in the ZenMsg.html page the program ships with), including all the names of built-in icons. Every icon available on the Error Message Generator is built in to ZenMsg, and you can point it to a custom image on disk to use your own icon:

BUILT-IN ICONS

Here is a list of all the built-in icons that you can use by name:

  aim_guy         - Blue AIM guy icon
  aol_icon        - Blue AOL icon
  attention       - Yellow triangle around an exclamation mark
  bomb            - Round black bomb icon
  bomb_dynamite   - Icon of a bundle of dynamite and a trigger
  bomb_grenade    - Icon of a grenade
  bulb            - White light bulb
  butterfly       - MSN Butterfly icon
  cake            - Slice of pink cake on a blue plate
  circularsaw     - Icon of a handheld circular saw
  control_panel   - Generic control panel icon
  cow             - Icon of a cow and a computer tower
  defrag          - Disk Defragmenter icon
  disk_blue       - Generic blue floppy disk icon
  disk_blue_label - Blue floppy disk with a label
  disk_orange     - Generic orange floppy disk
  disk_red        - Generic red floppy disk
  disk_red_label  - Red floppy disk with a label
  disk_skull      - Gray floppy disk with skull and crossbones
  disk_yellow     - Generic yellow floppy disk
  error           - Old-school X in a red circle error dialog icon
  error2          - Modern, shiny incarnation of an error dialog icon
  error3          - Beveled error dialog icon (like Windows XP)
  error4          - A red X icon
  file_cabinet    - File cabinet icon
  find            - Find Files icon
  floppy_drive    - Generic floppy drive icon
  fortunecookie   - Icon of a fortune cookie
  garbage_empty   - Empty garbage can
  garbage_full    - Bloated garabage can
  gun             - Icon of a revolver pistol
  hammer          - Icon of a hammer
  heart           - Icon of a shiny red heart
  help            - Old-school Windows Help icon
  hub             - Icon of a hardware hub of sorts (networking?)
  hwinfo          - Icon of a PCI device with blue "i" bubble above it
  ie5             - Icon of old-school Internet Explorer
  info            - Speech bubble with an "i" inside
  keys            - Generic icon of keys
  keys2           - Old Windows key icon
  keys3           - Generic key and padlock icon
  labtec          - Icon of a server or something?
  mac             - Striped colorful Apple logo
  mail            - Generic icon of an envelope
  mail_deleted    - Same envelope with a red X emblem in the corner.
  mailbox         - Mailbox with the flag down
  mouth           - Smiling mouth icon
  msdos           - MS-DOS icon
  mycomputer      - A "My Computer" icon
  mycomputer2     - A "My Computer" icon
  mycomputer3     - A "My Computer" icon
  newspaper       - Generic newspaper icon
  peripheral      - Generic computer peripheral icon
  plant_leaf      - A certain green leafy plant
  pocketknife     - A swiss army pocket knife
  question        - Icon of a speech bubble with a "?" inside
  radiation       - Yellow and black radiation symbol
  ram             - Icon of a couple sticks of RAM
  recycle         - Green recycle arrows logo
  recycle2        - Recycle arrows enveloping a globe of Earth
  scanner         - Generic scanner icon
  screw           - Golden screw icon
  screw2          - Gray screw icon
  setup           - Generic icon for "setup.exe" type programs
  skull           - Black skull and crossbones
  skull2          - Picture of a skull
  skull3          - White skull and crossbones
  tux             - Icon of our favorite Linux mascot
  tux_config      - Tux dressed up like a repairman
  ups             - Icon of an uninterruptible power supply
  zipdisk         - Icon of a single zip disk
  zipdisks        - Icon of numerous zipdisks

You can call ZenMsg from a batch file or any other program (e.g. a Python or Perl script could call ZenMsg.exe and send it parameters). For example, open Notepad and save the following as "example.bat" (with quotes, ensuring that it gets a .bat extension and not .bat.txt) and place it in the same folder next to ZenMsg.exe:

@echo off
ZenMsg --alert --title "Critical Error" --text "Now you've done it." ^
	--button "Ok" --button "Cancel" --button "Accept blame" ^
	--disabled 1 --disabled 2 > zenmsg-answer.txt

echo The user had selected:
type zenmsg-answer.txt
del zenmsg-answer.txt

Double-clicking your example.bat file would then pop up that alert box. ZenMsg prints the user's selected button to its standard output, which we captured above by piping it into zenmsg-answer.txt (it's possible to get output from commands in e.g. Perl scripts too, so your program can ask the user a question and then have branching behavior depending on which button the user clicked on).

Tags: 0 comments | Permalink