How to trace down a memory leak in APE with valgrind

How to trace down a memory leak in APE with valgrind

Given:

You already have a working/compiled APE server running and you still have the sources.

Rebuild (don’t use build.sh) with the following steps

Enter your APE source code directory and then do

cd deps/js/src/
./configure --enable-debug --enable-valgrind
make
cd ../../../
make

Now copy your rebuilt libape and aped to it’s default locations (see http://www.xosofox.de/2010/11/get-your-ape-up-to-date/)

Now run your ape with valgrind as a wrapper (I prefer to cd to /etc/ape so I don’t have to specify the ape.conf)

valgrind --leak-check=full --log-file=/path/to/mem.log /usr/bin/aped

pkill aped

All leaks will be written to the log and can be analyzed

Note: valgrind is slowing down the application dramatically in terms of CPU

(Thanks to Anthony @paraboul, all the information is coming from a chat with him)

APE User Handling – reduce the WTF (and why there is no getUserByName)

Having been struggeling with APE’s user management, sessions and related errors like Error 007 “NICK_USED”, I finally had to take the time to understand more in details, what is going on internally inside the ape-server regarding user management.

Because it is literally: Nothing.

WHAT? Yes, well, kind of…

Documentation again is not very good regarding this topic, so I tried to put up some stuff on my own – hope it helps

First of all, we need to understand what “Users” are.

What are APE users?

Questions like “Why is there no Ape.getUserByName?” are a natural question that I asked myself as well – but it only showed to me in the end that I did not fully understand the concept.

So the simple reason in a nutshell:

Web User != Ape User

In a setup with a client, a web server and an ape server, there is one most important thing:

The user (in terms of person using the web application) is not the user an ape server knows.

Web User: If your web appliction is referring to a “user”, it mostly is ONE person, having a nickname, a password, a name, an email… (called “web user” from now on)

Ape User: On an APE server, a user is “an instance of an object with a pubid and pipe that established a connection to the ape server and keeps polling”. (called “ape user [object]” now)

And with this said, the problem becomes obvious: APE does NOT AT ALL know or care about your web app’s user database and cannot relate to it.

Sure, in a default installation and configuration the APE server has some handling of the username (you will have experienced that when you tried to get your first ape project running). And you will have wondered, what those errors “006 BAD_NICK” or “007 NICK_USED” mean and where they come from.
But actually, it is not “APE” that is doing that, because the server does not care about nicknames or usernames. In fact, this comes from a file/module on the server called “nickname.js” – and it is in the “example” directory. And actually, this is what it’s meant to be: an example only!!!

Because in some cases, this can become a nightmare (especially, if the guy in front of your browser hits F5/Rrefresh) and might/does not fit your needs.

So, what then?

Good news: You have the total control of what you want to have
Bad news: You have to implement it.

Defining what you want, need and can do, requires some understanding of the interaction between the APE_JSF client framework and the APE server.

First of all and most important: consider the APE server to be DUMB! It’s just a stupid machine that waits for incoming connections and if told so, passes it on. It does not know about usernames of the web application and has NO CLUE AT ALL how two web users can exchange messages.

However, if there is a message incoming from an ape user’s pipe and it includes the info “pass this on to the pipe with pubid ‘0123456789abcdef'” it will happily do so – maybe resulting in a message to another web user, if your app works like this (basic chat).

So, what this means is:

  1. ape user objects have a pubid
  2. Your CLIENT (APE_JSF, JS on web browsers side) knows it’s own pubid. And in a public channel where there is an event when another ape user objects joins a channel, your client can TRACK other ape user objects (with pubid).
  3. Your client cannot send a message to a nickname, but to an ape user object (if it knows the pubid).
  4. How can your client know, which ape user object relates to which web user?

Enter: “nickname.js”!

The nickname.js default example requires the CLIENT to pass a “name” when establishing the connection to the ape server. And this “name” is stored on the server side in a hash (key-value-pair). Additionally, it is added to the ape user object as a property.
By this, it is possible on the client side to track the web-user <-> ape-user relation. If you listen to the userJoin event, you get the ape user object AND a property, telling you the web user name.

Now, on the server side, all the ape server does in this nice (and at the same time dreadful) “nickname.js”, is creating a hash with the web user nicknames.
So, whenever a new user (ape user) tries to register with a nickname (web user nickname), it checks the name against this hash. And if the nickname already is present, throws a well known error.

The main issue with this is, that whenever your client hits refresh, he will reconnect to the ape server and try to initialize a new ape user object and forget about the old object. This means, that the previous ape user object will remain on the server for the next 45 seconds (default time out) and you cannot connect a new users with the same nickname for this timeframe.

And all of this is handeled by nickname.js, allowing only “one web user at a time”.

Enter: “apeCoreSession.js”

or apeCore.js vs apeCoreSession.js – which one to use?

In your configuration you have to define which of the two files your client should use. This also implies some effects on you user handling.

Because with apeCoreSession, APE_JSF tries to re-attach to the previous session on every reload. This means, that when you hit refresh and the APE JS Framework initializes, it will try to “recover” your previous session in terms of the ape users pubid as well as joined channels.

Even when you are having multiple browser windows open, all sessions will be reachable under one common ape user object = pubib and if you have a name property set, you will have a chance to find a pubid for a give username.

But what if the user opens another BROWSER (not tab) or logs in from another machine/location? Baam, problem again…

Now, depending on the purpose of your apllication, you can decide what you need to do – like not allowing additional users with the same name or handle it.

For test purpose, I rewrote the nickname.js to test, understand and show, what is going on internally.

The first idea was to create a “flat” hash with username (key) and pubid (value). However, this means that whenever a web user connects again, the old pubid is forgotten and only the new one saved  – it could do the job in combination with apeCoreSession.

//global userlist - a hash of key (=username) and values (= hash of pubids)
var userlist = new $H;

Ape.registerHookCmd("connect", function(params, cmd) {
 if (!$defined(params.name)) return 0;
 cmd.user.setProperty('name', params.name);
 return 1;
});

function listUserNames()
{
 Ape.log("--------Current Userlist-------------------------");
 userlist.each(function(v,k){
 Ape.log(k + ": ");
 v.each(function(pubid){
 Ape.log("    "+ pubid);
 });
 });
 Ape.log("--------Current Userlist End---------------------");
}

Ape.addEvent('adduser', function(user) {

 var name=user.getProperty('name');
 var pubid=user.getProperty('pubid');

 //Ape.log("NEW: " + name + " as " + pubid);

 var uis=userlist.get(name);

 //if no user instance hash, create one
 if (!$defined(uis))
 {
 uis= new $H;
 userlist.set(name, uis);
 }

 uis.set(pubid,pubid);
 listUserNames();
});

Ape.addEvent('deluser', function(user) {
 var name=user.getProperty('name');
 var pubid=user.getProperty('pubid');
 Ape.log("Timout: " + name + " " + pubid);
 var uis=userlist.get(name);
 uis.erase(pubid);
 //if last instance left, erase hash
 if (uis.getLength()==0) userlist.erase(name);
 listUserNames();
});

This example however creates a hash PER USER, tracking ALL the current pubids. So, in theory, you could send a message from the server to all pubids (= user instances) of a given username.

So, just give it a try in a test setup, replacing the original nickname.js and while watching the ape log file and hitting F5 in several different browsers, you will get a better understanding of web users, ape users and pubids.

Feedback or Flattr clicks much appreciated 😉

Playing with my new pet: APE

My Questions to APE – answered

I’m planning to use APE (www.ape-project.org) for my next big project. It provides “Real-Time-Push” functionality based on Java Script, making real-time web applications possible.
After playing with it for some time now, getting the demos to run and trying out some stuff on my own, I wanted to share some of the things I learned.
Before I started playing with APE, I put down some basic questions occuring in my head. I would have liked to get them clarified directly to get a better overview of what I’m dealing with. Unfortunately, the documentation is a little bit thin on giving you a “basic big picture” before it goes into API calls and Event details.
So, a couple of days ago, I just put down the basic questions I had at the beginning – and hopefully, as I got a better understanding now, I found answers – and maybe you can use my notes to get the big picture easier than I did…

My initial Questions and their answers after playing with it

Ape works with pipes – what are pipes?

A kind of virtual “handles” in the framework to communicate with the server or another users. If you joined a channel, the server will tell you which users are connected to it – along with their “pipe”. And you can send a message to that pipe – the server will forward it to the user.

How does the server know if a user is allowed to join a channel?

If the server does not have any additional intelligence, it does not care. Its up to the server, to decide what to do.
A client JavaScript can simply call a .join(“myChannel”) and if it exists, he’ll join. If not, the channel will be created.

If I’m implementing a chat with privat chatrooms, how can I prevent a “bad” user from joining a private channel if he guesses the name or finds out about the user?

A client sends a message to another user “on a pipe” to the server, the server will pass on the message – so there is no public/private channel, but a direct user pipe that cannot be intercepted

How do channels get their name at all?

You need to know the name to connect to it – or you just create one with a given name

If a user knows a channel’s name, can he join it?

If his JavaScript lets him…

What’s the difference between a MultiPipe and a UniPipe

Uni-Pipe is a direct pipe from user to user. A multi pipe is what we would call a public chat room.

Is everything I send to the channel sent to everybody?

Well, yes and no…
In the end, the server INITIALLY is quite dumb… but it provides a lot of opportunities.
Clients can send commands to the server, the server receives data and decides what to do with it.

  • If Alice sends a “say”-command (literally speaking) on a channel she joined, the server side module will decide to forward the message to all users in the channel.
  • If Alice sends a “leave”, the server would notify all others of her disappearing.
  • If Alice sends “Say ‘Hi’ to my friend Bob” (talking to Bob’s pipe), the server would forward the message to Bob, notifying him that it was Alice

Well, finally, and how does it really work? Does “Push” create incoming connections?

Simple answer: Its not “really” push, so no incoming connection.
Actually, the idea behind the client-server communication is quite simple:
Your client side JavaScript connects via AJAX to the server. This connetion will be kept open by the client and the server. Now, as soon as there is data to be sent, either from client to server or back, this open connection will be used.
So, the CLIENT always opens the connection and the server just has the option to either keep the user waiting or send him something, immediately.
And then it is up to the client side JS framework, to deal with the data.
In a default config, this “long request” will be refreshed every 25 seconds, as a keep-alive to the server. After 40 seconds of no answer, the server will consider a user as timed out.

Wrap up:

APE is cool! APE is fast! APE is fun! And there is some documentation…
At the time of this writing (March 2010), getting into it is a little bit tideous because of the fact that some documentation between the APE comics and the API description is missing. Thats why I provided my notes, hoping it will help some-one.
From what I learned so far, APE is providing a great base for getting your stuff done.
All the functionality that you can find in the demos MAINLY relies on heavy clients, while the server is just passing on messages. However, you can put additional logic into your server, it can talk to a MySQL DB, provide authentication, you name it. You just have to do it.
So, APE provides everything you need to get your real time application up and running, you just have to define the “protocol”, being the commands the client can send to the server and what the server will do with it.

And, hey, come on, seeing stuff being updated directly in your browser as it happens, WITHOUT FLASH, is just awesome 😉