Tuesday, November 21, 2006

multimedia and firm RealTime

Al ritgh....from now one I promisse (again) to post more often....

Last week I was with Bilboed (Edward from Fluendo) a very smart french "Gringo" guy that I could take some beer and talk regarding GStreamer and Multimedia.

It was a very amazing week and weekend and I promisse to post some pictures here (I ll ask for the other guys for the pictures. I'm a guy that really like works with multimedia and high-end tecnologies but I'm not a user of them :-P)

....ok... lets talk, just a litle, regarding multimedia schedulling and GStreamer....it is what I'm going to go in deep during my vacation so I can finish my Master...

I have already read some state-of-art articles regarding multimedia scheduling, and the ones that I liked most was the ones that sorts multimedia as a firmware realtime task.

It really does make sense.

... Let's suppose you have a task running at you mobile device, and decoding a OGG video clip file (if you were me you were watching to Bob Marley video clips). Now, lets's suppose you are in front of a shop store and want to buy a new Bob Marley album. Well, the wireless and strong crypto (you are buying using your credit card) is makes the cpu usage to go full for a while. Your audio task (decoding vorbis) is fine, it reaches the "deadline" and play the song in realtime, but the video task (decoding theora) was too lazy.

Well, well, well...What do you want?
1- your video decodes everything and becomes out-of-sync with audio?
No, of course not.
2- your video will just discards a few frames and them when the cpu gets lower again it will be syncronized with audio and deconding at expected quality.

Because of this, multimedia task fits well as a firm realtime task, i.e. if the task is late, you can discard the task, doesn't make any sense to decode video frames that are, lets say, 2 seconds late related to audio.

That is the concept of firm realtime tasks. Try to run the task before its deadline, but if too late just discard it (it is not considered as fail)

How is GStreamer related to it?

It is so amazing how GStreamer does all of this in a so simple, easy to implement and flexible way.

a: simple - it is has (lets simplify here) 2 threads, one for audio and another one for video. The threads are schedulled by the operating system as any other one. Then, lets suppose the video has just been decoded and is read to be displayed, but is still not time to render, lets wait and block a few miliseconds because we are working with 24 frames per second. So the audio thread can finish its job "without" worry be preempted. If audio or video frames are delayed (out of jitter), we discard it.
YES, I KNOW, it is not the Best One, but it is like a Best-effort one and works so good and so simple.
2- flexible - We don't have to worry about dynamically created tasks. If we want to decode it as fast as we can, fine its easy with GStreamer frame work.

Ok I have just started talking about it without introduce the basics of RealTime concepts and Multimedia....may be I will do so.

What Im doing is to make some measurements of state-of-art realtime multimedia schedullers for Nokia 770 like devices (running in a dual core OMAP chip with a DSP and ARM)

As Im going on my researches I will post here (I hope someone else give some more ideas here)

Friday, September 09, 2005

starting in the blog's world

no so much by now...I ve just created it. I promisse put some interesting info soon!