Interactive Data Exploration for Computer Vision Projects with Rerun

instruments for knowledge visualization and exploration have allowed me to iterate method faster on my pc imaginative and prescient tasks, particularly when the issues I’m confronted with are usually not easy and I must make algorithm or design choices primarily based on dynamic time-varying alerts. These alerts can typically be difficult to research by simply taking a look at quickly altering numbers plotted on the display or saved in a desk.

Whereas engaged on a few of these issues, I explored the built-in interactive components from OpenCV, however aside from some sliders the choices there are very restricted, particularly when attempting to combine some animated plots. There’s some hacky methods to get dynamic plots from matplotlib into OpenCV, which I explored on this post.

I additionally explored completely different UI frameworks like tkinter, which I utilized in my last project for a sentiment evaluation visualization. I constructed a customized part that allowed me to show dynamic frames. Nonetheless it nonetheless didn’t actually really feel like the best instrument for the duty, particularly when attempting to work with interactive plots.

After which I stumbled upon Rerun. Each every now and then I uncover a instrument or framework that actually excites me, and this was precisely a kind of instances. Rerun is an open supply instrument for visualizing knowledge sometimes discovered within the robotics area starting from easy time sequence knowledge and static photos to complicated 3D level clouds, video streams or different varieties of multi-modal knowledge. The demos look actually spectacular and the setup and code samples are so easy, I used to be instantly hooked.

So I made a decision to remodel my ball monitoring demo from a earlier venture and plot the info utilizing rerun. Let me present you the way straightforward it’s to make use of and create interactive functions!

Quickstart

You’ll be able to set up rerun in any of your python tasks utilizing pip or uv:

pip set up rerun-sdk
uv add rerun-sdk

You can begin the viewer after putting in the sdk by merely working it from the command line:

rerun

The viewer shall be your essential window the place your experiments shall be proven. You’ll be able to go away it open or shut it between your experiments.

To instantiate a rerun viewer from a python script, it’s essential spawn an occasion with an experiment identify:

import rerun as rr

rr.init("ball_tracking", spawn=True)

Ball Monitoring Demo

Rerun experiment recordings will be saved to and loaded from .rrd recording recordsdata. You’ll be able to obtain the recording file for the ball monitoring demo from here. Press Ctrl + O or choose Open... within the menu on the highest left of the rerun viewer and cargo the downloaded recording file.

You will note the ball monitoring demo playback as soon as after which the video stops. On the backside of the viewer, you have got the timeline. You’ll be able to scrub by means of the timeline by clicking and dragging the deal with.

These recording recordsdata solely comprise the info of the experiment together with the video, its annotations and the time sequence of the monitoring. The structure of the viewer is saved individually in a .rbl blueprint file. Obtain the blueprint file for the demo here and open it on prime of the prevailing knowledge file.

Now we have now a barely higher overview with the place, velocity and acceleration plots separated and the video prominently centered.

Within the video body you’ll be able to click on on the annotations and within the left Blueprint panel you’ll be able to cover or present them individually.

Time Collection Plots

To investigate a particular plot, you’ll be able to click on on the develop view button on the prime proper of any window, for instance the place plot.

It is a TimeSeriesView. This view can be utilized to plot knowledge in a 2D chart with the x-axis representing the time area, in our case the discrete body index of the video. On this ball monitoring demo, we iterate over the video frames in a loop, the place we will set the body index of our timeline explicitly.

frame_index = 0

whereas True:
    ret, body = cap.learn()
    if not ret:
        break

    frame_index += 1
    if frame_index >= num_frames:
        break

    rr.set_time("frame_idx", sequence=frame_index)

To create the plot for the place, it’s essential log a Scalar worth for the place of the tracked ball at each body index. On this case after we calculate the place we will merely log it to rerun:

rr.log("ball/position_y", rr.Scalars(pos))

To configure the model of this plot, we additionally must log one thing to the identical entity path (ball/position_y), however for the reason that model doesn’t change we will log it as soon as earlier than the loop and provide a static argument.

rr.log(
    "ball/position_y",
    rr.SeriesLines(colours=[0, 128, 255], names="pos y"),
    static=True,
)

To outline the axis vary that’s seen per default, we have to specify a structure part.

view_pos = rrb.TimeSeriesView(
    origin="ball/position_y",
    axis_y=rrb.ScalarAxis(vary=(0, 700)),
)
structure = rrb.Blueprint(view_pos)
rr.send_blueprint(structure)

Video Stream

Equally we will create a view for the video frames by logging the picture to rerun. Since rerun expects an RGB photos however OpenCV ues BGR for its shade channel ordering, we have to convert the frames from BGR to RGB earlier than passing them to rerun.

frame_rgb: np.ndarray = cv2.cvtColor(body, cv2.COLOR_BGR2RGB)
rr.log("body", rr.Picture(frame_rgb))

So as to add annotations to the picture view we have to log spatial components to a sub path of the desired entity path. For instance, we will draw the middle of the tracked ball:

rr.log(
    "body/factors",
    rr.Points2D(
        positions=[center],
        colours=[0, 0, 255],
        radii=4.0,
    ),
)

To incorporate the video body view within the structure, we will use a Spatial2DView node:

view_frame = rrb.Spatial2DView(origin="body")

Then we will stack the plot from earlier than vertically with the body view through the use of a Vertical structure part:

structure = rrb.Blueprint(
    rrb.Vertical(
        view_pos,
        view_frame,
        row_shares=[0.33],
    ),
)
rr.send_blueprint(structure)

The row shares specifies how a lot every of the rows occupies in percentages. We are able to omit the second row share entry for the body view for the reason that shares have so as to add as much as 1.

Limitations

Whereas engaged on this venture, I bumped into some limitations of Rerun. Within the unique venture I visualized a prediction of the trajectory at each timestep, however that is at present not attainable in a time sequence view. Additionally the layouts and configuration of the plotted knowledge is restricted, for instance there’s no built-in method to attract a circle with out fill. However for the reason that venture could be very actively being developed, there’s a superb probability that a few of these may be attainable sooner or later.

Conclusion

The developer expertise with this instrument as a pc imaginative and prescient engineer is extraordinarily good, the consumer interface masses nearly immediately and the timeline scrubbing will be extremely useful for understanding or debugging alerts in time sequence plots or in movies. I’ll undoubtedly maintain utilizing and exploring this venture and may solely advocate you to attempt it out for your self!

For extra particulars and the complete implementation, checkout the supply code of this venture below src/ball_tracking/trajectory_rerun.py.

https://github.com/trflorian/ball-tracking-live-plot

All visualizations on this submit had been created by the creator.