( Pollux's blog )

I'm currently trying to generated interactive (and animated) charts in Python + Qt. The wanted library would be:

• portable: this is one of the reasons of the choice of PyQt
• simple: same reason
• interactive: I want to be able to select, for example, the slices of a pie chart. A signal of events like Qt's would be perfect
• animated: this is useless, but looking at things like AnyChart or FusionCharts, the result is really nice !
• light on dependencies: relying on tons of libs makes the project hard to maintain and not portable, especially for windows where there is not packaging and dependency system.
• free software

A quick search gave me the following products:

• matplotlib: mostly for scientific plots, but there is a nice number of options, a well-documented API.
• pyQwt: Python bindings for Qwt. Again, it's more scientific plot than charts
• cairoplot: projects looks dead (or in the "yeah, the project's not finished, but we're recoding it in $LANG to be faster" syndrome, which is more or less the same). It generates images, though item maps can be extracted. The name tells it, it uses Cairo.
• pyCha: some nice charts, uses Cairo. Very simple API (not much options).
• reportlab: not really for interactive applications, but it can generate charts (and images).
• KD Charts: not tested. It looks nice, but the license is not free.
• ChartDirector: not free

Globally, the result is quite a deception:

• There is no immediate reply (yes, I'm lazy).
• Cairo is nice for Linux projects, but is a pain to use on other platforms
• There is nothing using Qt only, except PyQwt which cannot be used directly :/ This is surprising, especially given that Qt's API is really nice and almost offers the solution natively.
• Charts libraries for web apps are easy to use, there seems to be nothing easy on desktop apps.

Custom version

This was pretty simple (at the beginning, at least): Qt provides two classes, QGraphicsView and QGraphicsScene, to draw objects. These classes are pretty good, and adding a PieChart, for ex, was only a matter of calculating the slices:

for d in data:
   (l,v) = d
   start_angle = sum
   span = (v*360 / self.current_sum)
   el = QGraphicsEllipseItem(0, - ELLIPSE_RADIUS, ELLIPSE_RADIUS, ELLIPSE_RADIUS)
   el.setStartAngle(start_angle*16)
   el.setSpanAngle(span*16)

To do the animation, just create a QTimer, and redraw the scene in the event handler. The setSpanAngle function can be use, for example, to display the slices regularly in 20 steps, drawing (360/20) degrees of the total Pie at each step:

current_step = (self.i+1) * (360/20)
if (current_step >= o._start_angle):
   o.setVisible(True)
   if current_step > o._start_angle + o._span_angle:
      o.setSpanAngle( o._span_angle * 16 )
   else:
      o.setSpanAngle( 16 * (current_step - o._start_angle) )

Using all features of Qt is pretty simple, like brushes to create a gradient etc. data are stored in a model, and catching events allows to redraw the scene smoothly when something is changed.

This solution is implemented in PyQt only (and could easily be in Qt only), has no extra dependency, and is reasonably fast. Using Qt objects natively allows to use events like hover, click in a very trivial way.

Code is stored in a git repository. I've named it Cutie Chart, which is a pretty bad pun (I'll say I'm almost sorry):

git clone http://git.wzdftpd.net/cutie-chart.git

The repository also contains animation on Bar Charts, scaling each bar progressively to its normal size.

Example of result:

Matplotlib version

Using matplotlib would simplify things a lot, and bring tons of existing classes to the projects. However, things may not be so easy.

matplotlib provides a class FigureCanvasQTAgg. This class is a Qt canvas, so it can be used as a widget.

You have to create a Figure, a subplot and draw the pie:

fig = Figure(figsize=(width, height), dpi=dpi)
self.axes = fig.add_subplot(111)
self.axes.hold(False)
self.axes.plot([0.1, 0.1, 0.8, 0.8])
(patches, texts, autotexts) = self.axes.pie(self.data, explode=self.explode, labels=self.labels, colors=self.colors, autopct='%1.1f%%', shadow=True)

The animation is a bit trickier: changing the span angle is not possible. I've tried several options with no success (like using Transform etc.), until I decided to create a different animation, better suited for matplotlib. In this version, we add a dummy slice, to create the effect that each slice starts at 0% and grow to its proportion regularly. As previously, we use a QTimer:

s = sum(self.data)
i = self.i
self.anim_fracs[-1] = (s*(20-i)/i)

self.axes.pie(self.anim_fracs, colors=self.colors, labels=self.anim_labels, autopct='%1.1f%%')
self.draw()

Code will be committed in the same repository soon. I've uploaded the current file here

It works fine, and using matplotlib is nice, with many features and support classes (for ex the ability to create SVG or PDF, to have many other classes for charts etc). Matplotlib also handle the automatic placement of the legend, 3d effects, z-shape, which would take some time to add manually ;). However, the drawback is that the integration with Qt is limited, like the support of signals or interaction on objects.

Example of output:

The code for nfqueue-bindings is now almost ready, I have made some progress since last week:

* you can now modify packets in live, and send the new packet with the verdict
* new functions are wrapped, and the creation of the queue can be done in one function
* more examples

I have presented a special script for SSTIC, using the weather to decide if a packet should be accepted or dropped :) While the utility of the module still has to be proven, it is a good example of how easy it is to use the new bindings.

The slides can be found online here, and contains some code examples (with some funny things ;). They are in french, but they should be quite easy to understand.

Random ideas:

* The Netfilter workshop will be held in Paris from 30 September to 3 October 2008.
* Eric has presented nf3d, a nice tool to view netfilter logs (from ulogd2) in 3D.

Gamers will recognize a nice try to convert network logs into Guitar Hero tracks ;)

* Some people have weird habits at SSTIC !

I am currently working (amongst other projects ..) on nfqueue-bindings, set of high-level modules for several languages (Python and Perl, for the moment), for libnetfilter_queue.

The goal is to provide a library to gain access to packets queued by the kernel packet filter. For more details, see nfqueue-bindings project site.

Current state

Actually, you can

* access the module in Perl or Python
* create a queue connected to netfilter
* register a callback
* access the contents of the packet. As I do not want to do what was already done many times, I use some other libraries to decode the packet: 
 * NetPacket::IP for Perl
 * dpkg for Python.
 * If you know some other libraries, please let me know.
* set the verdict (decision) to ACCEPT or DROP for the packet

I have written some scripts to show what can be done in a few lines of code. The current examples are:

* example Perl script
* example Python script
* Packet dumper, in pcap format (use scapy)
* HTTP request checker
* A surprise I will present in a lightning talk at SSTIC :)

I will make a release as soon as the code is stable (and can be installed).

Examples

Create and bind the queue (Perl)

use nfqueue;

use NetPacket::IP qw(IP_PROTO_TCP);
use NetPacket::TCP;

my $q;

sub cb
{
...
}

$q = new nfqueue::queue();
$q->open();
$q->bind();
$q->set_callback(\&cb);

$q->try_run();

Create and bind the queue (Python)

import nfqueue
from dpkt import ip

q = None

def cb(dummy, payload):
  ...

q = nfqueue.queue()
q.open()
q.bind()
q.set_callback(cb)
q.create_queue(0)

q.try_run()

Decode packet (Perl)

Now all we have to do is to use the callback !

my $ip_obj = NetPacket::IP->decode($payload->get_data());
print("$ip_obj->{src_ip} => $ip_obj->{dest_ip} $ip_obj->{proto}\n");

if($ip_obj->{proto} == IP_PROTO_TCP) {
  my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
  	print "TCP src_port: $tcp_obj->{src_port}\n";

Decode packet (Python)

data = payload.get_data()
pkt = ip.IP(data)
print "proto:", pkt.p
print "source: %s" % inet_ntoa(pkt.src)
print "dest: %s" % inet_ntoa(pkt.dst)
if pkt.p == ip.IP_PROTO_TCP:
  print "  sport: %s" % pkt.tcp.sport
  print "  dport: %s" % pkt.tcp.dport

Set verdict

Perl:

$payload->set_verdict($nfqueue::NF_DROP);

Python:

payload.set_verdict(nfqueue.NF_DROP)

Links

* libnetfilter_queue
* nfqueue-bindings project site
* nfqueue-bindings source (git browser)
* Get source: git clone http://git.inl.fr/git/nfqueue-bindings.git

Default init

ANSI C requires all uninitialized static and global variables to be initialized with 0 (§6.7.8 of the C99 definition). This means you can rely on the following behavior:

int global;
void function() {
  printf("%d\n",global);
}

This will print 0, and it is guaranteed by the standard.

However, this is not handled by the compiler. All you will be able to see is that the variable is put in the bss section:

08049560 l     O .bss   00000004              static_var.1279
08049564 g     O .bss   00000004              global_var

It is the startup code of the linker which initializes the variables.

The C compiler usually puts variables that are supposed to be initialized with 0 in the .bss section instead of the .data section. Opposed to the .data section, the .bss section does not contain actual data, it just specifies the size of all elements it contains. The C compiler just *assumes* that the linker, loader, or the startup code of the C library initializes this block of memory with 0. This is an optimization; .data elements occupy space in the image (or ROM or flash memory) and in RAM whereas .bss elements need to occupy RAM space only if they are initialized at run-time.

(Gcc provides even an option (-fno-zero-initialized-in-bss) to do not rely on this optimization, that is, to put all 0-initialized elements into the .data section as well.

You can use

__attribute__((section, ".mysection))

on every uninitialized variable to instruct the compiler to put it into your own section.

I have a strange bug with bash shebang: when I try to give twistd as interpreter, bash tries to execute the script as a shell script !

Here is a simple, not working, twisted script with a shebang:

#! /usr/bin/twistd -y
from twisted.application import internet, service

Bash execution:

$ bash -c ./test.tac
from: can't read /var/mail/twisted.application

Bash is trying to execute the script as a shell script ! (from is a shell command).

Zsh execution:

$ zsh -c ./test.tac
Failed to load application: 'application'

The error is correct (there is no application defined in the twisted script). It really looks like a bug in bash ..

Quilt is a nice tool to manage series of patches, and is particularly adapted to subversion (not very useful for git, the concept of patch series is integrated). It can manage dependant patches, edition, updating patches for a code change, etc.

Start by telling quilt where to store patches:

$ export QUILT_PATCHES=debian/patches

Quilt will create the directory automatically when creating the first patch.

Now, suppose we want to create a new patch, called my_nice_patch:

$ quilt new my_nice_patch
Patch my_nice_patch is now on top

"On top" ? quilt manages patches as a stack, so you will have to push patches to apply them, and pop to deapply. Now that we have a patch name, we have to mark the files we will modify in this patch:

$ quilt add reports.py gather.py
File reports.py added to patch my_nice_patch
File gather.py added to patch my_nice_patch

So far so good. Three commands, and we have done nothing :) Files can be modified using your favorite editor (subliminal hint: vim), as usual. At any moment, you can get the diff between your modifications and the unpatched files:

quilt diff

will print a standard diff.

At this point, you have finished your patch. If you look at the debian/patches directory, you'll see .. nothing ! That's because a patch has to be "refreshed" to be written to disk.

$ quilt refresh
Refreshed patch my_nice_patch

The patch has been written to disk:

$ ls debian/patches/
my_nice_patch
series

The "series" files is an index of the patches, the other files are the patches. quilt uses the standard "diff" format, so patch can be used as usual (using diff -p1).

To apply all patches:

quilt push -a

To deapply all patches:

quilt pop -a

So what is nice with quilt ? It handles gracefully merges, updates, etc. Suppose your patch was done for version x of the sources, and you update the sources. You can reapply patches using the same commands:

$ quilt push my_nice_patch
Applying patch my_nice_patch
patching file gather.py
patching file reports.py
Hunk #1 succeeded at 145 with fuzz 1 (offset 38 lines).

Now at patch my_nice_patch

Ok, the patch applied with some fuzz, but correctly. How do I update my patch for the new sources ? Well, as usual:

$ quilt refresh 
Refreshed patch my_nice_patch

It wasn't hard ?

This is the end of this introduction to quilt, which can do much more advanced things than this simple example ! See the documentation for more.

Links:

• Introduction to quilt (pdf)
• Installation: apt-get install quilt