Symfony2 and me – let’s be friends – Part 1

My journey, starting a rewrite of an exisitng project from scratch with Symfony2

Step 1: Get the framework:

composer.phar create-project symfony/framework-standard-edition path
After giving my VM some more RAM (512MB wasn’t enough) it succeeded.

Step 2: Get rid of Doctrine and grab Propel

So I edit the composer.json file and delete the two doctrine lines (if someone other lib needs doctrine/common for annotations, they will solve the dependency, right?
Then add "propel/propel-bundle": "1.1.*" to the file instead, as stated on http://www.propelorm.org/cookbook/symfony2/working-with-symfony2.html
Then I run “composer.phar update” and it updates my monolog, removes doctrine and installs propel, its bundle and phing.
After that, however, the symfony cache update fails – if course, I removed Doctrine which is configured somewhere in the the Kernel.

So I remove the Doctrine line from app/AppKernel.php and all doctrine stuff from app/config/config.yml – and HOORAY, my app/console is green again

Step 3: Does it work?

My webserver is already configured to check for an app.php in the web subfolder, so it works – but first I’ll add my IP to app_dev.php so I can see all the debug beauty of Symfony2
Checking the Debug toolbar I notice Propel is missing – of course, I need to add it to the kernel where I kicked out Doctrine
So I add new Propel\PropelBundle\PropelBundle(), to app/AppKernel.php where I commented Doctrine befor, and app/console knows some good old Propel commands now.

Step 4: Freeze this into git

This is a good start, so I want to get this status into git. I’m surprised to see that there IS already a lot of git-information in the directory, being the symfony2 history.
Quick fix, rm -rf .git and git init again, here I come. On the plus side, I already have a nice .gitignore 😀
git add .; git commit -m “Fresh start”

How to trace down a memory leak in APE with valgrind

How to trace down a memory leak in APE with valgrind

Given:

You already have a working/compiled APE server running and you still have the sources.

Rebuild (don’t use build.sh) with the following steps

Enter your APE source code directory and then do

cd deps/js/src/
./configure --enable-debug --enable-valgrind
make
cd ../../../
make

Now copy your rebuilt libape and aped to it’s default locations (see http://www.xosofox.de/2010/11/get-your-ape-up-to-date/)

Now run your ape with valgrind as a wrapper (I prefer to cd to /etc/ape so I don’t have to specify the ape.conf)

valgrind --leak-check=full --log-file=/path/to/mem.log /usr/bin/aped

pkill aped

All leaks will be written to the log and can be analyzed

Note: valgrind is slowing down the application dramatically in terms of CPU

(Thanks to Anthony @paraboul, all the information is coming from a chat with him)

Symfony Day 2011 in Cologne

Symfony Day was amazing!!!

I love the community and it was great fun to see all the guys I already met during the last years and this year’s Symfony Live in Paris.

Again, a big THANK YOU to all organisers and of course all the people that make Symfony2 happen!

I just wanted to quickly paste my (sparse) notes that I took for myself – in case they help out someone else as well.

Basically, they are just some reminders for me of stuff that I wanted to check out.

Silex

  • Silex routes can use value() and assert() to be enhanced
  • Silex needs cookbook contributions
  • Silex can easily be functional tested (and require “” can have a return value!)

SPL

A shame, I never really looked at it before!? So take a closer look at http://php.net/manual/de/spl.exceptions.php and SPL in general

Links to follow/read

Pragmatic Programmer: http://pragprog.com/the-pragmatic-programmer/extracts/tips

Sonata

Check out www.Sonata-project.org – adminbundle

See, if it’s really that complicated as the talk made it seem

I did not note so much else, either because I already knew it, wanted to take a look at the presentations later, or player #sfdayjeopardy (thanks to Jeremy “Rick Astley” Mikola for that)

There’s one more thing

Actually, there were two things: Fabpot is a visionary! And his vision, of looking at Symfony2 not (only) as a framework but as a collection of several, individual and powerful components is great. MAYBE not for the developer looking for “a thing that does all the magik™” but for the more experienced one that can pick the best of a wide variety of components. Like some other PHP projects do now…

AND: Me, being a big fan of Propel, was happy to see all the momentum propulsion it got recently, including development support from inside sensio.

Hope to see you all again in June(?) in Paris and in November(?) in Berlin?

Fluchstaben

Quick post: I finished a bookmarklet, based on this years April Fool’s Day fun of www.karopapier.de:
FLUCHSTABEN

Klick this, it’s a JS call that will make some random letters of this website move.

Why I did it? Because it works!

You can save it as a bookmark(let) and use it on every webpage out there

1) Press the bookmark
2) Press multiple times for additional fun
3) Click one of the flying letters to make them go back where they belong.

Like it?

PS: “Fluchstaben” is a combination of the German words for “flying” and “letters” (“Flug” & “Buchstaben”)

A TURTED based real-time thing that I’m proud of…

I recently finished a nice little example of how to use real-time data for an exciting and equally relaxing view:

Blocker Champ Real Time


http://reloaded.karopapier.de/blocker

What’s that?
It shows a list of players and the number of games where it is their turn.
As soon as they make a move, they are moved to the new position with a nice animation.

Interestingly enough, the hardest part was the JQuery/UI animation stuff.
Since it already is based on the components that are soon to become the TURTED project the real time events were already there to be intercepted and used.

So, if you feel stressed, just sit back, relax and watch the squares move – the ZEN of Karopapier

JS Performance: Numbered Array vs Associative Array

In JS, you sometimtes have to keep track of lists. Usernames, sessions, available Pictures ,…
And often, you have to add and delete some of them.

So, how do you handle these lists? There are basically two options:

a) Good old numbered Array
b) Associative Array, key/value store

Examples?

var users=[];
users[0]=”Adam”;
users[1]=”Bob”;
users[2]=”Claire”;

users={};
users[‘Adam’]=”Adam”;
users[‘Bob’]=”Bob”;
users[‘Claire’]=”Claire”;

Both have their advantages and disadvantages.
An array keeps the order, you have numbered keys. However, deleting or finding an element requires you to walk through the array.

So, the question to me was: How about performance? How big is the difference?

Long story short:
In case you do not care about the order and will only have unique keys, use an associative array!
To add a value:
assoArray[key]=value;
To get a value:
var myVal=assoArray[key];
To delete a key/value:
delete(assoArray[key]);

This is easy and performs much better. At least in my quickly hacked, unoptimised test you can find here: Simple test html
If you think I did something wrong, please enlighten me.

Get your APE up-to-date

After developing an APE application for quite some time now, it was time for me to install an APE server on the future production environment.

Since it created some hassle for me, not doing THIS every day, here’s a small log that might help you (or me some time later) to do the same, Upadate your APE server to the latest git version

I’m doing this on a debian 32 bit.

Since I am lazy, I will first install the deb package so I get all the nice entries in /etc/init.d, have the lib files placed in the correct directories…
And in a second step replace the ape version with the current one from github.

1: Install the “old” 1.0 package from the website

Download your package (like APE_Server-1.0.i386.deb) from http://www.ape-project.org/download/APE_Server.html to the machine.

Run dpkg -i APE_Server-1.0.i386.deb

In case this already fails, you might need to check if there are any libraries missing (see below)

2: Get the latest version

To get the latest version from github, I usually do

svn co http://svn.github.com/APE-Project/APE_Server.git
or
git clone http://github.com/APE-Project/APE_Server.git

This will create a APE_Server.git directory in your current location. Now compile it:

3: Compile the new version

cd APE_Server.git
sh build.sh

You can have different types of errors here, most of them related to the fact that you might not have all compilers or libraries installed. If this happens, try:

aptitude install make
aptitude install gcc
aptitude install g++
aptitude install python
aptitude install zip
aptitude install mysql

To be honest, I am not sure which libraries are required exactly – but this did the job for me. And if I could have installed the specific required libraries only I don’t care… as long as I get what I need.

After a finished compile (it can take a couple of seconds/minutes, warnings flying by…) you are ready to replace the existing ones.

4: Replace old files

Basically have to replace two files:

aped and libmod_spidermonkey.so

If running, stop your ape

/etc/init.d/ape-server stop

Copy the new files from your compile dir to their designated locations:

cp bin/aped /usr/bin/aped
cp modules/lib/libmod_spidermonkey.so /usr/lib/ape/libmod_spidermonkey.so

(Paths may vary depending on system and linux distribution)

Restart your ape

/etc/init.d/ape-server start

If you connect to your ape server (e. g. http://127.0.0.1:6969/) with your browser, it should show you the version info with current build time

Corrections, additions, suggestions and comments welcome

APE User Handling – reduce the WTF (and why there is no getUserByName)

Having been struggeling with APE’s user management, sessions and related errors like Error 007 “NICK_USED”, I finally had to take the time to understand more in details, what is going on internally inside the ape-server regarding user management.

Because it is literally: Nothing.

WHAT? Yes, well, kind of…

Documentation again is not very good regarding this topic, so I tried to put up some stuff on my own – hope it helps

First of all, we need to understand what “Users” are.

What are APE users?

Questions like “Why is there no Ape.getUserByName?” are a natural question that I asked myself as well – but it only showed to me in the end that I did not fully understand the concept.

So the simple reason in a nutshell:

Web User != Ape User

In a setup with a client, a web server and an ape server, there is one most important thing:

The user (in terms of person using the web application) is not the user an ape server knows.

Web User: If your web appliction is referring to a “user”, it mostly is ONE person, having a nickname, a password, a name, an email… (called “web user” from now on)

Ape User: On an APE server, a user is “an instance of an object with a pubid and pipe that established a connection to the ape server and keeps polling”. (called “ape user [object]” now)

And with this said, the problem becomes obvious: APE does NOT AT ALL know or care about your web app’s user database and cannot relate to it.

Sure, in a default installation and configuration the APE server has some handling of the username (you will have experienced that when you tried to get your first ape project running). And you will have wondered, what those errors “006 BAD_NICK” or “007 NICK_USED” mean and where they come from.
But actually, it is not “APE” that is doing that, because the server does not care about nicknames or usernames. In fact, this comes from a file/module on the server called “nickname.js” – and it is in the “example” directory. And actually, this is what it’s meant to be: an example only!!!

Because in some cases, this can become a nightmare (especially, if the guy in front of your browser hits F5/Rrefresh) and might/does not fit your needs.

So, what then?

Good news: You have the total control of what you want to have
Bad news: You have to implement it.

Defining what you want, need and can do, requires some understanding of the interaction between the APE_JSF client framework and the APE server.

First of all and most important: consider the APE server to be DUMB! It’s just a stupid machine that waits for incoming connections and if told so, passes it on. It does not know about usernames of the web application and has NO CLUE AT ALL how two web users can exchange messages.

However, if there is a message incoming from an ape user’s pipe and it includes the info “pass this on to the pipe with pubid ‘0123456789abcdef'” it will happily do so – maybe resulting in a message to another web user, if your app works like this (basic chat).

So, what this means is:

  1. ape user objects have a pubid
  2. Your CLIENT (APE_JSF, JS on web browsers side) knows it’s own pubid. And in a public channel where there is an event when another ape user objects joins a channel, your client can TRACK other ape user objects (with pubid).
  3. Your client cannot send a message to a nickname, but to an ape user object (if it knows the pubid).
  4. How can your client know, which ape user object relates to which web user?

Enter: “nickname.js”!

The nickname.js default example requires the CLIENT to pass a “name” when establishing the connection to the ape server. And this “name” is stored on the server side in a hash (key-value-pair). Additionally, it is added to the ape user object as a property.
By this, it is possible on the client side to track the web-user <-> ape-user relation. If you listen to the userJoin event, you get the ape user object AND a property, telling you the web user name.

Now, on the server side, all the ape server does in this nice (and at the same time dreadful) “nickname.js”, is creating a hash with the web user nicknames.
So, whenever a new user (ape user) tries to register with a nickname (web user nickname), it checks the name against this hash. And if the nickname already is present, throws a well known error.

The main issue with this is, that whenever your client hits refresh, he will reconnect to the ape server and try to initialize a new ape user object and forget about the old object. This means, that the previous ape user object will remain on the server for the next 45 seconds (default time out) and you cannot connect a new users with the same nickname for this timeframe.

And all of this is handeled by nickname.js, allowing only “one web user at a time”.

Enter: “apeCoreSession.js”

or apeCore.js vs apeCoreSession.js – which one to use?

In your configuration you have to define which of the two files your client should use. This also implies some effects on you user handling.

Because with apeCoreSession, APE_JSF tries to re-attach to the previous session on every reload. This means, that when you hit refresh and the APE JS Framework initializes, it will try to “recover” your previous session in terms of the ape users pubid as well as joined channels.

Even when you are having multiple browser windows open, all sessions will be reachable under one common ape user object = pubib and if you have a name property set, you will have a chance to find a pubid for a give username.

But what if the user opens another BROWSER (not tab) or logs in from another machine/location? Baam, problem again…

Now, depending on the purpose of your apllication, you can decide what you need to do – like not allowing additional users with the same name or handle it.

For test purpose, I rewrote the nickname.js to test, understand and show, what is going on internally.

The first idea was to create a “flat” hash with username (key) and pubid (value). However, this means that whenever a web user connects again, the old pubid is forgotten and only the new one saved  – it could do the job in combination with apeCoreSession.

//global userlist - a hash of key (=username) and values (= hash of pubids)
var userlist = new $H;

Ape.registerHookCmd("connect", function(params, cmd) {
 if (!$defined(params.name)) return 0;
 cmd.user.setProperty('name', params.name);
 return 1;
});

function listUserNames()
{
 Ape.log("--------Current Userlist-------------------------");
 userlist.each(function(v,k){
 Ape.log(k + ": ");
 v.each(function(pubid){
 Ape.log("    "+ pubid);
 });
 });
 Ape.log("--------Current Userlist End---------------------");
}

Ape.addEvent('adduser', function(user) {

 var name=user.getProperty('name');
 var pubid=user.getProperty('pubid');

 //Ape.log("NEW: " + name + " as " + pubid);

 var uis=userlist.get(name);

 //if no user instance hash, create one
 if (!$defined(uis))
 {
 uis= new $H;
 userlist.set(name, uis);
 }

 uis.set(pubid,pubid);
 listUserNames();
});

Ape.addEvent('deluser', function(user) {
 var name=user.getProperty('name');
 var pubid=user.getProperty('pubid');
 Ape.log("Timout: " + name + " " + pubid);
 var uis=userlist.get(name);
 uis.erase(pubid);
 //if last instance left, erase hash
 if (uis.getLength()==0) userlist.erase(name);
 listUserNames();
});

This example however creates a hash PER USER, tracking ALL the current pubids. So, in theory, you could send a message from the server to all pubids (= user instances) of a given username.

So, just give it a try in a test setup, replacing the original nickname.js and while watching the ape log file and hitting F5 in several different browsers, you will get a better understanding of web users, ape users and pubids.

Feedback or Flattr clicks much appreciated 😉

My biased view on Doctrine vs Propel

I’m getting old and I’m old fashioned.

I’m still using mysql, not mongoDB.

I’m using php and apache, not node.js, ruby, python or lighttpd, nginx or ….

And I’m using propel again (1.5), not doctrine. AND I’M LOVING IT!!

After #sfdaycng 2009, the presentations and talks there, I forced myself to giving Doctrine a try and really liked it then. A lot of reading through documentation in the beginning if you are new to it, but hey, that’s normal. However, today, even when writing a simple query, I still have to consult the docs. Why? The syntax seems not to be “my style” somehow, it won’t fit into my type of writing code.

But there is such a HUGE difference between Doctrine and Propel and Doctrine is SO MUCH BETTER

Is it? Do I care what’s inside if I’m only querying a database without using all possibilities? Remember sfLive 2010?

“Is Doctrine re-inventing propel” Hahaha LOL ROFL LMAO

At this years big symfony conference #sflive2010, in a Q&A session about Doctrine2, a poor guy obviously asked the question, that made it into a big running gag: “Are you trying to re-invent propel?”

I have to admit and regret that I was not there, but if you followed the #sflive2010 hashtag on twitter, it was not very long until this got tweeted over and over again. And everybody was making fun of it…

Sure, it is kind of funny to ask @jwage, (lead) developer of Doctrine that question… But hey? What’s really so funny about it?

While everybody in the audience laughed, it really would have been interesting to

a) see how many were trying to smirk, wondering whats so funny about that

b) know how many would have been able to give a good reply on that?

And with “good reply” I mean, explain the difference.

And with “difference” I mean, explain it beyond

  • “doctrine has the schema.yml and the model classes in a subfolder”
  • “propel uses two classes User and UserPeer, doctrine has User and UserTable”
  • “propel has a criteria object, doctrine a query object”

If you started with symfony 1.0 as a developer that did not work with frameworks like symfony before (like I did), everthing is “symfony” – may it be propel, phing, creole, PDO, … and getting into these differences was not really worth it in the first place if you wanted to get started with writing your first applications.

What I’m trying to say: I guess that a lot of symfony developers out there don’t know or care about “Doctrine” vs “Propel” – they use the default. And they call it “symfony”, thinking that it belongs to it. If they started with symfony <1.2, they’ll use propel, now they use doctrine – because it’s the default. You learn it, you use it, you stick to it.

And I sticked to propel throughout 1.2. And it was quite save for me getting along ignoring Doctrine. Just then, when I started using Doctrine after #sfdaycgn I found new things, new approaches, things I liked, things I had to learn and a lot of stuff I could not get the hang of.

So now, with propel 1.5 and the new Query objects, I gave it a try and was so excited, I decided to come back to Propel. Just because it “feels like home” and fits my personal style of thinking/coding.

And of course both ORMs have their features and behaviours and at #sfdaycng Doctrine had more of them and so on… But I also assume that a lot of those features are beyond most everyday developer’s needs

But Propel is dead

Oh no, it’s not!!!

With the new 1.5 it REALLY got amazingly easy to write your code, extend your Query Model and easy query creation. 1.6 is on it’s way and 2.0 was just recently announced to be based on Doctrine2, as a layer on top, providing the usability features… that’s something we will have to wait for, but I does sound very good.

But Benchmarks show that…

So what? I couldn’t care less. Why should I? I am not running sites that have gazillions of users with quadrubillions of tables and hexafoobarillions of rows – I don’t care if grabbing all posts with a certain topic and tag for a given user takes 111 ms or 132 ms. But I do care if I have to consult the docs for writing that query because I couldn’t remember the syntax or – on the other hand – my IDE (yeah, right, eclipse, I’m REALLY old fashioned) nearly automagically creates the query for me.

But what are you actually trying to say?

  • Most developers don’t need benchmarks to find their ORM – it’s about own style and preference (or customer specs, but thats another story)
  • Some symfony developers might not even have thought about giving “the other one” a try.
  • Personally, I get along with propel 1.5 better
  • I’m getting old

So, if you know Doctrine, give Propel 1.5 a try.

If you only use Propel, give Doctrine a try.

And if you still use Propel 1.4 – MAN!!! Get the 1.5!!!

Any comments appreciated, I really like to see what I’m missing

PS: I’m even more old fashioned: I’m using vi – not vim, but eclipse with the vi plugin. Why? REAL code completion! A wonderful thing with propel 1.5!

Automated tweets now there is OAuth? Sending Tweets from PHP Command line

For quite some time I’ve been using a scheduled php script to automatically send twitter updates (tweets) to my own account.

Since it was very easy and convenient to do this with the so called basic authentication, I used/modified the simple script provided by Fabien Potencier.

NOW, BASIC AUTH IS GONE!! (Correction: See Update below)

It was long announced, but since September 1st, they really switched it off and this method does not work anymore because now there is only OAuth.

The “strange” (and more secure) thing about OAuth is, that you allow an application to do something with your twitter account, but you do not give out your credentials to the application, you just grant permissions. In most cases, when you are browsing the web, this makes sense if a third party application wants to do something with your account. But in my case, there is just ME, MY TWITTER ACCOUNT and MY PHP SCRIPT that want to communicate, not a bunch of different users.

So: What do I have to do to make it work again?

Step 1: Officially set up your application in Twitter Apps

On http://dev.twitter.com/apps you can and have to define your application with a name, some details and a callback URL. Details on this one later.

Step 2: Get and configure the TwitterOAuth PHP Package

On Github, one of the Twitter developers maintains a package for PHP. http://github.com/abraham/twitteroauth. Download/git-clone it to your web server. You need to adjust the config.php file in this package with the Consumer Key and Consumer Secret of your newly created twitter application (found int the app details section)

Step 3: Allow authentication from your twitter application to your twitter account

Now that you have the twitteroauth pack on your webserver, you should be able to access it and see the default page with a “Sign in with Twitter” button. If you do this, you will be redirected to a screen you might have seen with other third party applicatios before. Only, this time, it’s yours:

In this case, I want to connect with my @managerator to the application “managerator”, might be a bit confusing. (And please excuse me for using a German UI 😉 )

If you grant access successfully, you can check this in you Twitter account’s Connection details on http://twitter.com/settings/connections

Step 4: Use the connection

Now that we have connected application and account, how can we send a tweet from the application to the account?

Check the “index.php” file in the twitteroauth package and you will see, that it loads/requires the necessary php files and afterwards does some API calls on the $connection. And, using a browser, this works. Of course, only if your browser is athenticated on twitter already (and of course with the account that granted access to the application).

Now, in my case I want to use the command line php (like a cron-job) to send messages. And of course, PHP is not authenticated on twitter. Thus, a simple call to “php index.php” on the command line will fail.

So, how do we get around this??

Step 5: Store the access_token

When you look again at the index.php, you will see

/* Get user access tokens out of the session. */
$access_token = $_SESSION[‘access_token’];

This access token is an associative array that contains all the things you need for further authentication – coming from your current browser session

So, simply var_dumping the access_token from your browser session and pasting the details into your code like this

/* Get user access tokens out of the session. */
//$access_token = $_SESSION[‘access_token’];
$access_token=array(
“oauth_token”=>”**************************”,
“oauth_token_secret”=> “***************************************”,
“user_id”=> “************”,
“screen_name”=> “managerator”,
);

Will make your code callable from the command line.

Note: user_id and screen_name is not used, it’s just part of the token

Summary

This was a quick hack of my way to get the auto-updated management bullshit tweets working again for @managerator. It’s not very nice, but it works for the time being.

And I hope it helps you, too!

Comments appreciated!!

Update Sept 14th:

Basic Auth is only gone for everbody else but own twitter apps.

As I read on this article http://blog.nelhage.com/2010/09/dear-twitter/ you can circumvent OAuth and still use basic auth by adding “?source=twitterandroid” to your API URLs… That’s lame, Twitter!!!

Just tested and verified it, as of today, Sept 14th.