both approaches allow annotation values to have well-specified custom structures (not just strings!).

both approaches allow signals/annotations to be deserialized separately.

Both approaches are better than our current approach from a performance and tooling interop perspective.

In this approach, we have two tables: signals.arrow (each row is a signal) and annotations.arrow (each row is an annotation).

In this approach, we have a single table: recordings.arrow (each row is a single recording) with 3 columns ( uuid, signals, annotations) Okay - I've implemented two primary alternative approaches here that I'd like input on. Theoretically, IIRC Arrow's struct layout technically enables similar field access optimizations either way, but it would be nice to keep things less nested if possible.will keep playing around with it. I wonder if I can just store two tables alongside each other in a single Arrow file.keep going back and forth between separate tables and just continuing with the nested struct option used in my example lol. In many senses, it'd be really, really nice to have annotations in their own table and replace the current 'value' field with 'the dataset author can include whatever additional columns they want alongside the recording UUID and time span columns.' However, I'm not sure if the complexity of having multiple files is really worthwhile. Sample rates and time spans are next after, and I'd say the least likely thing to be individually predicated against is plain encoding information (which will likely just be used when sample data is loaded). in reductions/searches) are signal names, channel names, and annotation values. Internally at Beacon, we canonically associate a UUID with our annotations, which has been globally useful enough that it might be worthwhile to actually bake into Onda itself.ĮDIT: After thinking about the above, I think the fields most likely to be individually predicated against (e.g. The only roadbump that I really encountered was apache/arrow-julia#75, though it's pretty workaroundable could just go back to Vector anyway. I guess it'd be worth breaking up annotations and signals into separate columns, at least.I'll play around with it. Unfortunately, it really is the case that most of the Onda metadata is pretty 'record-like' in that you really do want to read most of a row pretty often.

It'd be nice if Onda's metadata was more columnar in nature - when the data is amenable, migrating from fewer columns/more nesting to more columns/less nesting is generally a win from an accessibility perspective (since it makes it easier to naively express finer-grained accesses). So it seems like switching to Arrow.jl is definitely doable and seems like it'd even yield slight performance improvements. I did a super naive little experiment to make sure that basic Onda metadata was (de)serializable via Arrow.jl can find it here (note that I'm also playing with some of the other v0.5.0 changes there too). To be clear, whenever I refer to 'Arrow files' I mean. Any talk of 'Arrow files' needs to be really explicit about what's going on.

Anytalk emojis