I’m trying to check for a certain chain of events in an LTTNG event log using Babeltrace 1. The LTTNG log is loaded using a Babeltrace collection:
import babeltrace my_collection = babeltrace.TraceCollection() my_collection.add_traces_recursive(trace_path, 'ctf')
The special events I’m looking for are almost indistinguishable from the normal events happening, except there is a few extra events once the chain have already started. So I need to look for these special events, and then search backward for the actual start.
The problem is that Babeltrace only lets me go forward in the event list. The simple solution seemed to be to create a clone of the events in my own list:
events = [e for e in my_collection.events]
The problem with this is that all events in the list now reference the last event. Which indicates that Babeltrace reuses the same object over and over and the generator only returns a reference to this single object.
I have tried to use copy.copy:
events = [copy.copy(e) for e in my_collection.events]
This didn’t help, and copy.deepcopy doesn’t work at all. I’ve also tried itertools.tee:
events = list(itertools.tee(my_collection.events))
But this returns a list of _tee object which can’t be used as proper event objects.
Is there a way to search backward using the Babeltrace event collection generator? Or is there a way to clone the event object properly to create my own list?
Advertisement
Answer
Babeltrace co-maintainer here.
Indeed, Babeltrace 1 reuses the same event record object for each iteration step. This means you cannot keep an “old” event record alive as its data changes behind the scenes.
The Python bindings of Babeltrace 1 are rudimental wrappers of the library objects. This means the same constraints apply. Also, Babeltrace 1 doesn’t offer any event record object copying function, so anything like copy.copy() will only copy internal pointers which will then exhibit the same issue.
Babeltrace (1 and 2) iterators cannot go backwards for performance reasons (more about this below).
The only solution I see is making your own event record copying function, keeping what’s necessary in another instance of your own class. After all, you probably only need the name, timestamp, and some first-level fields of the event record.
But Babeltrace 2 is what you’re looking for, especially since we don’t maintain Babeltrace 1 anymore (except for critical/security bug fixes).
Babeltrace 2 offers a rich and consistent C API where many objects have a reference count and therefore can live as long as you like. The Babeltrace 2 Python bindings wrap this C API so that you can benefit from the same features.
While the C API documentation is complete, unfortunately the Python bindings one is not. However, we have this, which at least shows some examples to get you started.
About your comment:
since it seems the events are a kind of linked list where one could walk backward
No, you cannot. This is to accomodate limitations of some trace formats, in particular CTF (the format which LTTng uses). A CTF packet is a sequence of serialized binary event records: to decode event record N, you need to decode event record N – 1 first, and so on. A CTF packet can contain thousands of contiguous event records like this, CTF data streams can contain thousands of packets, and a CTF trace can contain many data streams. Knowing this, there would be no reasonable way to store the offsets of all the encoded CTF event records so that you can iterate backwards without heavy object copies.
What you can do however with Babeltrace 2 is keep the specific event record objects you need, without any copy.
In the future, we’d like a way to copy a message iterator, duplicating all its state and what’s needed to continue behind the scenes. This would make it possible to keep “checkpoint iterators” so that you can go back to previous event records if you can’t perform your analysis in one pass for some reason.
Note that you can also make a message iterator seek a specific timestamp, but “fast” seeking is not implemented as of this date in the ctf plugin (the iterator seeks the beginning of the message sequence and then advances until it reaches the requested timestamp, which is not efficient).