{"version":"https:\/\/jsonfeed.org\/version\/1","title":"Tangram Vision Blog","home_page_url":"https:\/\/www.tangramvision.com","feed_url":"https:\/\/api.feedifyrss.com\/feed\/blog\/feed.json","description":"RSS feed for Blog","items":[{"id":"urn:sha256:80f6eb5e3a9681e2c011f8efcdb412b946fc05213f7b5ff316a9f6bb236bf349","content_html":"<p>MetriCal, Tangram Vision\u2019s world-class multi-modal calibration suite, has moved to <a href=\"\/pricing\">credit-based self-serve pricing<\/a>. Download the software, run it locally, and pay only for successful calibrations. No more sales cycles. No forced demo calls. No more guessing whether it'll work for your sensors. Get results in hours, not weeks.<\/p><p>You can start using MetriCal today. <a href=\"https:\/\/docs.tangramvision.com\/metrical\/configuration\/installation\/\">Download it<\/a>, claim your free <a href=\"https:\/\/hub.tangramvision.com\/#\/billing\/plan-trial\">15-credit trial<\/a>, and run your first calibrations without entering payment information.<\/p><h2>How Credit-Based Pricing Works<\/h2><p>The mechanics are straightforward. Every calibration costs 3 base credits, plus 1 additional credit per sensor modality beyond the first. A camera-only calibration costs 3 credits. Add LiDAR to a run, and it's 4 credits total. Camera, LiDAR, and IMU is 5 credits, etc etc.<\/p><p>Here's the cool part: credits only deduct when calibrations succeed. MetriCal analyzes both your results and data quality, checking for <a href=\"https:\/\/docs.tangramvision.com\/metrical\/results\/report#data-diagnostics\" target=\"_blank\">high-risk diagnostics<\/a> that indicate problems with your data capture. If a calibration fails or throws warning flags, you don't pay. You're not burning budget while you debug bad datasets or refine your collection process.<\/p><p>Credits live at the organization level, not per-user or per-machine. Your entire team can run MetriCal across multiple systems without seat restrictions. Deploy it on your development workstations, your production line, or your field test rigs; the credits pool across all of them. This removes the artificial constraints that come with traditional per-seat licensing and lets you use the software where you actually need it.<\/p><h2>Software That Grows With You<\/h2><p>Once you've validated that MetriCal works for your sensors, choose the pricing path that fits your workflow. Credit bundles (\\$249 for 15 credits, 3-month rollover) work well for occasional needs; subscriptions (starting \\$999\/month for 75 credits, 6-month rollover, automatic overages) make sense for ongoing development. Enterprise removes credit constraints entirely for teams running high-volume production lines or needing offline deployment.<\/p><p>Ultimately, you decide. There's no sales gatekeeping, no negotiation, no waiting for quotes. Pricing is transparent and published. You can evaluate MetriCal on your terms and scale up when it makes sense for your operation, not when a contract renewal comes due.<\/p><h2>Why We Made This Change<\/h2><p>Tangram Vision has spent the last 5 years developing world-class calibration software. The technological task we face every day is daunting: create a calibration suite that is not only precise, but also works across modalities in a seamless, user-friendly way.<\/p><p>The only thing harder than that was figuring out how to license it.<\/p><p>Multi-modality calibration has always been a high-friction problem. You need to know if calibration software works for your specific sensor suite before committing to it, but traditional licensing forces you into a contract sight unseen. You\u2019re then locked into rigid pricing that doesn\u2019t match your actual usage.<\/p><p>We learned this the hard way by <em>being the company that forced the contract<\/em>, by putting users into those licenses. We heard the same feedback from robotics teams time and time again:<\/p><p>\u201cWe need to try this in our software before we\u2019re confident\u201d<\/p><p>\u201cOur calibration volume varies month-to-month; fixed pricing makes no sense\u201d<\/p><p>\u201cWe don\u2019t need an Enterprise contract just to calibrate 10 systems\u201d<\/p><p>Our previous licensing models (and we\u2019ve had many) either leaned too far into the development side or the scale side. We asked users to restrict their calibration modalities, limit their calibration counts, or pay for machines they didn't use. We asked small companies to accept all-or-nothing contracts that were prohibitively expensive. Nothing fit, and it frustrated everyone.<\/p><p>So, over the course of 2025, we faced this problem head-on and figured out a better way. Today, that work makes its public debut.<\/p><h2>Get Started Today<\/h2><p>MetriCal is available now with credit-based pricing. <a href=\"https:\/\/docs.tangramvision.com\/metrical\/configuration\/installation\/\">Download<\/a> the software, start your free <a href=\"https:\/\/hub.tangramvision.com\/#\/billing\/plan-trial\">15-credit trial<\/a>, and run calibrations on your actual hardware. No payment information required. No sales calls. Just download and calibrate.<\/p><p>See the full pricing breakdown and compare plans at <a href=\"\/pricing\">tangramvision.com\/pricing<\/a>. If you're evaluating Enterprise options for high-volume production or specialized deployment needs, visit <a href=\"\/enterprise\">tangramvision.com\/enterprise<\/a> for details on unlimited calibrations, offline licensing, and dedicated support.<\/p><p>Questions about which plan fits your workflow? <a href=\"\/contact\">Contact us directly<\/a> and chat with the experts in calibration systems.<\/p><p>Professional multi-sensor calibration shouldn't require enterprise sales cycles or upfront contracts before you know it works. Credit-based, self-serve pricing puts control back in your hands.<\/p>","url":"https:\/\/www.tangramvision.com\/pay-for-what-you-calibrate-tangram-vision-introduces-credit-based-licensing","title":"Announcing Public Pricing and Credit-Based Licensing for MetriCal","summary":"MetriCal, Tangram Vision\u2019s world-class multi-modal calibration suite, has moved to credit-based self-serve pricing","date_modified":"2025-11-19T00:00:00.000Z","author":{"name":"[\"brandon-minor\"]"},"tags":["company-news"]},{"id":"urn:sha256:668c9bf506b481a0b1c0e721c3e2c8a60c8b812a1ee79fb5605a0b26895f6bcd","content_html":"<p>A lot of great engineers have trouble calibrating narrow field-of-view cameras, which we\u2019ll take here to be around 25\u00b0 FOV or less. Frequently, they will find that even if their calibration results in low reprojection error, the parameters will be wildly inconsistent. Repeated calibrations with many of the same type of camera and lens, or even repeated calibration of the same unit, get different answers. Even worse, the intrinsics may look reasonable, but downstream vision tasks like extrinsics calibration or stereo depth inference between cameras are now completely unreliable.<\/p><p>Let's discuss the theory behind why this happens, and what we can do about it.<\/p><p><em>Try it yourself<\/em>: You can find an example dataset and MetriCal guide for this technique in the <a href=\"https:\/\/docs.tangramvision.com\/metrical\/calibration_guides\/narrow_fov_cal\" target=\"_blank\">MetriCal documentation<\/a>.<\/p><h1>NFOV Calibration is Tough<\/h1><p>In a nutshell, it\u2019s difficult to uniquely observe the focal length (i.e. the \u201cfx, fy\u201d parameters you may be familiar with) of the camera system when using a NFOV camera. To see why, we\u2019ll briefly discuss what focal length is, how it affects image formation, as well as a quick overview of the most common calibration algorithm and the impact a narrow field of view has on it.<\/p><h2>What\u2019s a focal length?<\/h2><p>Key to the process of calibrating camera intrinsics is estimating the camera\u2019s focal length. In intrinsics camera models, focal length is actually two quantities bound up in one value.<\/p><p>In the abstract, we can derive focal length in pixels by taking the expression $\\frac{f_m}{pp}$  where $f_m$ is the focal length in meters and $pp$ is the pixel pitch in meters-per-pixel. Pixel pitch represents density of the sensor.<\/p><p>The metric focal length, however, is a lot more abstract and is only physically meaningful for a real pinhole camera. Practically speaking, the metric focal length captures the \u201cwideness\u201d or \u201cnarrowness\u201d of the field-of-view of a sensor-and-lens combo. Generally a lens is thought to be what determines this quantity, but the sensor\u2019s overall size affects the metric focal length as well. This is why lens focal length is often expressed in \u201c35mm\/full frame equivalent\u201d in order to provide a frame of reference for how pictures taken with that lens look when taken with a sensor of known size.<\/p><p>Taking your sensor as a given, the focal length parameter of your camera\u2019s intrinsics model primarily varies with the \u201cwideness\u201d or \u201cnarrowness\u201d of the field-of-view, which is primarily determined by the lens.<\/p><p>The question for those calibrating a camera is then: \u201chow do I observe the camera\u2019s focal length?\u201d The answer is to observe focal length\u2019s unique effects on the image formation process.<\/p><h2>What does focal length look like?<\/h2><p>When people think of camera focal length, they primarily consider the field of view and the ability to see objects that are far away. A wide field of view lens (i.e. low focal length value) lets you see objects near the camera and potentially very far off the camera\u2019s optical axis. A narrow field of view lens (i.e. high focal length value) lets you see objects far away but only within a small range around the camera\u2019s optical axis.<\/p><p>Foreshortening is another related effect determined by focal length. Foreshortening describes the effect of perspective, which is the difference in apparent size of objects at varying depths. Wide FOV lenses display a lot of foreshortening. An object at a given distance will appear much larger than one even slightly further away. With a narrow FOV, foreshortening is much less pronounced. Objects set apart even a significant distance may appear approximately the same size in a narrow field of view lens.<\/p><p>Consider the below video. In the scene, there\u2019s a calibration board tilted away from the camera and a few equally-sized spheres placed around it. As the video progresses, the focal length sweeps from wide to narrow. Simultaneously, the camera moves backwards to approximately maintain the size of the board in the image.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/5L8gP4MVwIvJSJGMRX6qtoxk.mp4\"><\/video><p>However, nothing in this scene is actually moving. This process is called a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Dolly_zoom\">dolly zoom<\/a>. As the focal length narrows, the scene appears to compress. Even though the closest and furthest spheres are twelve meters apart, they appear similarly sized. Also, the far edge of the board tilted away from the camera appears to pull in, with the board's apparent shape under projection going from a trapezoid to a rectangle.<\/p><p>Foreshortening is very important to observing focal length; we\u2019ll see why in a bit.<\/p><h2>How does calibration work?<\/h2><p>First, we\u2019ve got to take a brief detour to discuss how calibration works. We\u2019ve got a <a href=\"\/blog\/calibration-from-scratch-using-rust-part-1-of-3\">series of blog posts<\/a> about this if you want more detail, but we\u2019ll do a short overview here.<\/p><p>In a typical calibration scenario, we take images of a target of known dimensions. The target has detectable and uniquely-identifiable features which correspond to 3D feature locations in some model coordinate system. We project the 3D features into the image using the current estimate of the camera intrinsics and find the error between the projected values and the detections. This is called reprojection error. The calibration process is an optimization problem whose objective is: \u201cadjust the intrinsics in order to minimize (the sum-squared of) reprojection error\u201d.<\/p><p>In order to project a 3D point into the image, the 3D point has to be in the camera\u2019s coordinate system. The 3D points described by the target\u2019s design are in some relatively-arbitrary model coordinate system. In order for calibration to work, we also have to jointly estimate the camera-from-model transform for each image. That per-image transform is a nuisance parameter: something we have to estimate but isn\u2019t actually of interest to us in the end.<\/p><p>The reason for bringing up details of the calibration algorithm is that it highlights a natural ambiguity that arises when calibration data isn\u2019t captured correctly. When a planar target\u2019s normal is aligned with the camera optical axis, focal length can freely trade off with the Z component (or depth distance) of the camera-from-model transform. Which means that in minimizing reprojection error you can choose any focal length and the camera-from-model depth (i.e. Z component) will simply compensate and vice versa. In this scenario, it is impossible to learn the focal length.<\/p><p>This is why we suggest users capture a wide variety of angles between the board normal and the camera\u2019s axis. When the board is tilted away from the camera, focal length becomes observable through the foreshortening effects (like we saw in the previous video), because the board by virtue of being tilted exists at a variety of depths relative to the camera.<\/p><p>The following video highlights the aforementioned ambiguity. The scene is the same as above, except the board\u2019s normal is aligned to the camera\u2019s optical axis. As before, the camera\u2019s field of view gets gradually narrower as the camera moves backwards. Again\u2014 nothing else is moving. In this scenario, it\u2019s possible to exactly preserve the appearance of the board in the image even as the camera\u2019s position and focal length change wildly.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/fl416GwxjgNyUJbunfqGooDDlk0.mp4\"><\/video><p>It\u2019s important to highlight that this isn\u2019t a binary thing. Calibration doesn\u2019t just all of a sudden work as soon as you go one degree off the optical axis. All sorts of inaccuracies introduce error into the process and reduce the observability of the focal length and thus your margin for error in the calibration data capture process. These include:<\/p><ul><li data-preset-tag=\"p\"><p>static inaccuracy of the board<\/p><\/li><li data-preset-tag=\"p\"><p>frame-to-frame non-rigid warping of the board<\/p><\/li><li data-preset-tag=\"p\"><p>blur (both motion and defocus)<\/p><\/li><li data-preset-tag=\"p\"><p>resolution (both sensor and lens sharpness\/spatial resolution)<\/p><\/li><li data-preset-tag=\"p\"><p>sensor noise and compression artifacts<\/p><\/li><li data-preset-tag=\"p\"><p>detector algorithm limitations<\/p><\/li><\/ul><h2>Back to Focal Length<\/h2><p>So fine, you say, I\u2019ll tilt the board and git gud at angles or whatever. What does this have to do with NFOV cameras? The problem is that NFOV cameras have very little foreshortening effect no matter how much you tilt them.<\/p><p>Here\u2019s the last frame (i.e. at the most narrow FOV) of the first video. That board is tilted 35 degrees off-axis, but you wouldn\u2019t guess that from the image. The far side (left) of the board is about the same size as the near side (right) of the board. There probably is some perspective effect here which, in a totally-ideal, noise-free case, might be enough, but practically when calibrating with images like this, depth and focal length are going to tradeoff massively.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/t9UzHOU0L3qZ1n1aa6eflF4DzY.png\"><p>For marginal cases, there are some simple ways to address this. You can tilt the board or camera more if you haven\u2019t already. Often it\u2019s straightforward to tilt the board, but \u201ctilting\u201d the camera with an extremely narrow focal length while keeping the target in view requires moving the camera over a very large area, which might be logistically impractical. Conversely, tilting the board too much will eventually cause feature detectors to fail\u2014 45\u00b0 off axis is a good rule of thumb.<\/p><p>You could also make a larger board, which when tilted will cover a greater range of depths, helping the observability of foreshortening and thus focal length. This too has limitations as the size of the board will have major impact on cost, availability, accuracy, rigidity, ease-of-manipulation, and space requirements.<\/p><p>Failing these two simple fixes, the next step is to think beyond just planar targets.<\/p><h2>3D Targets (probably) aren\u2019t enough<\/h2><p>A solution one might naturally consider would be to build a target that innately has some variation in depth, i.e. one that\u2019s not a board or plane. The calibration process does not fundamentally require a planar target: the camera-from-model transform can be estimated with an algorithm like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Perspective-n-Point\">PnP<\/a> provided you have a reasonable guess for intrinsics. <\/p><p>A non-planar target is a target that has variation in all spatial dimensions, and thus will hopefully display some foreshortening when imaged, no matter what its orientation relative to the camera is. As an example, consider a pyramid shaped object, with its apex pointed at the camera, and some features on its faces. We can see foreshortening in the spacing of the lines when imaged, and we can do so without having to tilt the object away from the camera.<\/p><p><br><\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/huyLJChM2Jg3h79kWRl1bT01W0.png\"><p><br><\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/p6MjxS1HpMKj395AqXCCKHYTXA.png\"><p>Non-planar targets certainly represent an interesting approach to calibration, and they have their niche uses. However, a lot of the same drawbacks you\u2019d get from building a large board and tilting it still apply. The target still needs to be large (think like several meters long) to get the desired effect, it's sensitive to initial conditions, and it is substantially harder to build and come up with a reliable detection algorithm.<\/p><p>In contract, planar targets are advantageous for a few reasons:<\/p><ul><li data-preset-tag=\"p\"><p>They\u2019re relatively easy to describe with just a few parameters (e.g. width, height, spacing)<\/p><\/li><li data-preset-tag=\"p\"><p>They\u2019re easily made (just print them)<\/p><\/li><li data-preset-tag=\"p\"><p>The planar arrangement imposes some invariants that make writing detectors easier. For example, a planar circle is an ellipse under projection. Grids of features will have some understandable relation under projection, which may make decoding targets easier.<\/p><\/li><li data-preset-tag=\"p\"><p>The camera-from-model transform can be more-easily bootstrapped by estimating a planar homography.<\/p><\/li><\/ul><h1>The MetriCal Approach<\/h1><p>To provide a (patent-pending!) user-friendly approach to narrow field of view camera calibration, we\u2019ve combined the advantages of planar and non-planar targets by using multiple planar targets merged together. Instead of relying on the size and relative angle of a single board to observe foreshortening and thus focal length, we\u2019ll use multiple boards offset at different depths from the camera. Since the calibration target consists of multiple boards, we get the advantages of board targets (easy construction, well-known detector algorithms etc.), and we get to build a composite target with a large depth range simply by spacing the boards apart.<\/p><p>In this video, two boards are set a few meters apart along the camera\u2019s optical axis. We set up a similar dolly-zoom to preserve the appearance of the left board. Notice how the right board appears to grow as the focal length gets narrow. This shows how we cannot trade-off a single camera-depth value and focal length to preserve the appearance of two objects at different depths.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/K4kyc4yoDs73WRGgifIqlmQPEtY.mp4\"><\/video><p>It does not, however, suffice to simply use multiple boards. In a typical calibration scenario, the camera-from-board transform has to be estimated for every board in each image. Estimating that transform is where the focal length ambiguity rears its head, and having two boards with two camera-from-board transforms per image is really no different. The trick is to merge the board targets into one calibration target so that only one transform per image need be estimated. Since the individual targets are at different depths, no single camera-from-board depth value can trade off with focal length in the optimization. This means focal length has become unambiguous.<\/p><p>In order to merge the board targets together such that a single camera-from-target transform is sufficient, we have to know the spatial relationship between the boards. The simplest way to learn those relationships is to survey the board-system with a calibrated camera. Fortunately, this is something that MetriCal can already do today so long as you have at least one non-NFOV camera!<\/p><p>So for narrow-field-of-view cameras, calibration becomes a multi-step process and requires an auxiliary camera. Many modern perception sensor suites contain multiple cameras of different focal lengths. Those cameras need to be calibrated regardless, so often this requirement is no real imposition.<\/p><p>The process generally looks like this:<\/p><ol><li data-preset-tag=\"p\"><p>A target field with targets at varying depths is constructed.<\/p><\/li><li data-preset-tag=\"p\"><p>Data is captured with a more easily calibrated camera (i.e. a wider FOV one). If the target field and camera are amenable to it, one dataset can be used to jointly calibrate this camera and survey the target field in one pass. If not, separate calibration and survey datasets are captured.<\/p><\/li><li data-preset-tag=\"p\"><p>The camera is calibrated and the target field is surveyed with MetriCal either in one pass or separately. This is done via MetriCal\u2019s <code>calibrate<\/code> step.<\/p><\/li><li data-preset-tag=\"p\"><p>The target field survey and calibration are used to create a consolidated object space configuration for the target field. This is done via MetriCal\u2019s <code>consolidate<\/code> step.<\/p><\/li><li data-preset-tag=\"p\"><p>Data is captured with the narrow field of view camera. The camera should observe the target field as it was surveyed. Generally the same sorts of data capture requirements apply as always. The image space should be well-covered.<\/p><\/li><li data-preset-tag=\"p\"><p>The narrow field of view dataset and consolidated object space configuration are used to calibrate the narrow field of view camera.<\/p><\/li><\/ol><h1>Try It Out!<\/h1><p>At the time of writing, we\u2019ve just released version 15.0.0 of MetriCal. This version focused hard on quick start time for new MetriCal users, as well as improvements for production line setups. Some customers are already using the technique above in their production line!<\/p><p>If you want to try it out yourself, follow our Calibration Guides in the <a href=\"https:\/\/docs.tangramvision.com\/metrical\/calibration_guides\/narrow_fov_cal\" target=\"_blank\">MetriCal documentation<\/a>. There, you can find demo datasets that show off what MetriCal can do, along with a ton of other tips and tricks for multi-modal calibration.<\/p>","url":"https:\/\/www.tangramvision.com\/nfov-calibration-made-easier","title":"Narrow FOV Calibration Made Easy(er)","summary":"Calibrating narrow field-of-view cameras is hard. We make it a little easier.","date_modified":"2025-10-24T00:00:00.000Z","author":{"name":"[\"paul-schroeder\"]"},"tags":["calibration-mathematics"]},{"id":"urn:sha256:bfe250fd1e2cf28d52cbf0f81803fb6ba04a437bce7fd48600fd0f39cdfc669e","content_html":"<p>In a few weeks, Tangram Vision will be releasing version 15.0.0 of MetriCal. This release is probably our biggest yet (though we say that every release). It represents a refinement of MetriCal\u2019s processes and workflow, based directly on customer conversations and experience.<\/p><p>As part of this refinement, we are doing something a little radical: MetriCal\u2019s outputs will no longer be JSON. Instead, <strong>all outputs will be formatted into an MCAP<\/strong>.<\/p><p>This choice makes a lot of sense to us, and represents a quality-of-life improvement to our customers. However, we haven\u2019t seen other platforms do anything similar before. In that light, we wanted to use this space to explain our rationale and explore the ramifications of the move.<\/p><p><em>Note<\/em>: This post is about the motivation for the switch. We\u2019ve left the implementation notes on the change to the end; be sure to skip ahead if that's what you're looking for.<\/p><h2>What\u2019s an MCAP?<\/h2><p><a href=\"https:\/\/mcap.dev\/\">MCAP<\/a>, for those unaware, is a relatively young data storage format in the robotics space. Despite this, it has become the default recording format for ROS2 since its debut and supplants any uses of the old ROS1 bag format. As such, MetriCal has <a href=\"https:\/\/docs.tangramvision.com\/metrical\/configuration\/data_formats\">read MCAP datasets<\/a> for quite some time.<\/p><p>Using MCAPs over folders or ROSbags presents a laundry list of advantages:<\/p><ul><li data-preset-tag=\"p\"><p>MCAP has an <a href=\"https:\/\/mcap.dev\/spec\">open-specification<\/a>. Likewise, a number of different client libraries are available for various languages which makes it easy for users to work with regardless of whether they use Rust or Python or otherwise.<\/p><\/li><li data-preset-tag=\"p\"><p>MCAP being the default ROS2 bag format gives it some (empirical) credibility in being both extensible to different use-cases and not subject to itself breaking in backwards incompatible ways.<\/p><\/li><li data-preset-tag=\"p\"><p>Regardless of the encoding picked, MCAP supports compression at the chunk level, which means that even text-based message encodings benefit from compression.<\/p><\/li><li data-preset-tag=\"p\"><p>MCAP channels typically contain full schema definitions of the data they carry, which means that even if a message schema were to change (either in a backwards compatible way or otherwise) the most up-to-date version of the schema can be loaded and verified against what your software is currently expecting.<\/p><\/li><li data-preset-tag=\"p\"><p>Metadata, attachments, and messages inside of the format can be managed independently of the format itself.<\/p><\/li><\/ul><p>It\u2019s for these reasons that we strongly recommend anyone working with MetriCal today use MCAPs to record and play back data. In fact, we plan to deprecate other modes of data intake in the near future (but not for 15.0.0, don\u2019t worry).<\/p><h2>On Backwards Compatibility<\/h2><p>Despite the many ways you can input data into MetriCal today, it only writes results one way: JSON. Unfortunately, when first designing these outputs in 2021, there was no obvious alternative to JSON. It fit our needs and got the job done. It wasn\u2019t until MetriCal started being integrated into real production lines, for real use cases and customers, that we started realizing the repercussions of adopting the format for our outputs.<\/p><p>One of the largest consequences of this decision, and a large driver of this recent work, was a lack of backwards compatibility. In particular, the <code>results.json<\/code> file produced at the end of calibration runs have never been guaranteed backwards compatible. This causes problems when trying to investigate an older <code>results.json<\/code> with a newer version of MetriCal than what generated the file. <\/p><p>We kept it this way for a long time. It was both a product decision as well as a technological limitation.<\/p><h3>As Product Decision: What\u2019s Useful?<\/h3><p>Before MetriCal reached today\u2019s semi-stable state, output metrics changed rapidly and, as a business, we had not fully materialized which metrics were useful. Given that these metrics are often the output of a complex, multi-faceted optimization process, it was not always an easy decision to articulate what every metric meant, or whether it was something that could be interpreted in context of the calibration. Even today, there are metrics that are easy-to-compute but hard-to-interpret. Put all together, we weren\u2019t in a logistic position to make something like <code>results.json<\/code> backwards-compatible in any way.<\/p><h3>As Technical Decision: What\u2019s Feasible?<\/h3><p>Technically, the choice of JSON was done for expediency: It was simple enough to use Rust\u2019s <code>serde<\/code> and <code>serde-json<\/code> crates to serialize any type we wanted to JSON with very little to change other than the type definition itself. However, we knew early on that JSON has its own set of limitations and was not in itself a great choice:<\/p><ul><li data-preset-tag=\"p\"><p>Serializing metrics in a text-based format (which are often floating point values) meant that we were writing very large text files (compared to a binary equivalent representation of the metrics)<\/p><\/li><li data-preset-tag=\"p\"><p>Additionally, JSON does not properly support \u201cfloats\u201d the same way that e.g. Rust does. Completely valid floats like <code>NAN<\/code>, <code>INFINITY<\/code> and such are not representable in the JSON standard<\/p><\/li><li data-preset-tag=\"p\"><p>Schema conformance was routinely an issue. <a href=\"https:\/\/json-schema.org\/\">JSON schema<\/a> is extant to the JSON format itself, and unlike other schema formats (Protobuf, Cap\u2019n Proto, Flatbuffers, etc.) does not have mechanisms or syntax in place to enforce backwards compatible growth of the format<\/p><\/li><li data-preset-tag=\"p\"><p>The use of <code>serde<\/code> meant that we could not reliably uncouple our internal definitions of the type hierarchy from the representation at rest. This couples directly with managing JSON schema, which rather than being thought out and hand-crafted was instead derived downstream from the <code>serde<\/code>-compatible type implementations<\/p><\/li><\/ul><p>We kicked the proverbial can down the road until we couldn\u2019t anymore. MetriCal is now expected to deliver not just accuracy and consistency in its metrics, but also its procedures. It\u2019s relied upon by some of the largest automation companies in the world. We needed a solution to providing backwards-compatible outputs that would serve those customers long-term.<\/p><h2>MCAP To The Rescue<\/h2><p>And that solution, you already know, is MCAP. We knew the advantages clearly from our time working with the format for data ingress; those same advantages made it a clear choice for data output as well. Specifically, the format directly solved a lot of our technical concerns with <code>serde<\/code> and the <code>serde-json<\/code> crate:<\/p><ul><li data-preset-tag=\"p\"><p>The method through which MCAP encodes messages (either as binary or through some text-based <a href=\"https:\/\/mcap.dev\/spec\/registry#message-encodings\">well-known encoding<\/a>) is configurable on a per-channel basis. This allowed us to avoid serializing a lot of floating-point numbers as text.<\/p><\/li><li data-preset-tag=\"p\"><p>Closely related to the last point: that compression factor really made a difference. Having witnessed some very-large results JSON files in the past (\u2265300MiB!), any ability to compress these artefacts is welcome.<\/p><\/li><li data-preset-tag=\"p\"><p>Foxglove (the company that produced MCAP) already provides <a href=\"https:\/\/github.com\/foxglove\/mcap\/tree\/main\/cpp\/examples\/protobuf\/proto\/foxglove\">a number of helpful message definitions<\/a> that are widely accepted and used across the industry. While Tangram still had to define custom messages for our metrics types, it made it easier to not have to reinvent the wheel for e.g. <code>Vector3<\/code> or <code>Pose<\/code> or <code>Quaternion<\/code> messages.<\/p><\/li><li data-preset-tag=\"p\"><p>Our schemas are now self-documenting due to MCAP\u2019s structure.<\/p><\/li><\/ul><p>There were also reasons of convenience. For instance, MetriCal already supports reading MCAP files, and we could preserve backwards compatibility even with our JSON structs via MCAP\u2019s attachments paradigm.<\/p><p>Fundamentally, using MCAP provides a layer of indirection between our internal representation (in our Rust code) of the various metrics versus how they're serialized at rest. What\u2019s more, MCAP makes sense from an operational standpoint due to its heavy use in the ROS ecosystem as well as due to the fact that many high-quality libraries exist in multiple languages.<\/p><h2>Benefits to MetriCal Users Today<\/h2><p>Besides having a backwards compatible output from our calibration process, there are other benefits that MetriCal users will experience today by switching to MCAP.<\/p><p>First off, the compression benefits cannot be understated. In some of the larger IMU and LiDAR datasets we have seen calibrated, this provides on average 3\u00d7 compression (although the full amount of savings will depend largely on the type of calibration being performed).<\/p><p>In addition, Result MCAPs allow reconstruction of the exact inputs to a specific version of MetriCal. Before, it was often difficult to pin down the exact plex and object-space that produced poor results; users often forgot the exact command invocation used to produce a given set of results. By organizing all of that information into a single artefact, we can pull everything from the results MCAP, rather than try to rely on memory alone.<\/p><h2>Get Ready for 15.0.0!<\/h2><p>We realize that this blog post may sound like we\u2019re making the case for MCAPs over JSON, but there\u2019s really no case to be made; the advantages are clear. MCAP makes a ton of sense, it solves a lot of user problems, and makes our lives easier. We\u2019re happy about the results (no pun intended), and we know you will be, too.<\/p><p>This was one of the biggest changes in 15.0.0, but it\u2019s not all. The next edition of MetriCal will fulfill a ton of promises we\u2019ve been making to users for literal years, and we couldn\u2019t be more excited to release it. So don\u2019t touch that dial! Stay tuned to Tangram for more great announcements in the coming weeks.<\/p><p>\u2014<\/p><h1>Implementation Discussion<\/h1><p>If you\u2019re interested in the implementation notes for this change, keep reading!<\/p><h2>Tangram\u2019s Results MCAP<\/h2><p>The use of MetriCal\u2019s results JSON usually serves (one of) several purposes:<\/p><ul><li data-preset-tag=\"p\"><p>The results JSON contains the optimized plex JSON<\/p><\/li><li data-preset-tag=\"p\"><p>The results JSON contains the optimized object-space JSON<\/p><\/li><li data-preset-tag=\"p\"><p>The results JSON could be used to produce various different graphs or visualizations of metrics that Tangram may not itself provide (either via Rerun or our console output)<\/p><\/li><li data-preset-tag=\"p\"><p>Internally, Tangram could use the results JSON as a debugging tool \u2014 the serialized form of this contained information on the version of MetriCal used when the file was generated<\/p><\/li><li data-preset-tag=\"p\"><p>The results JSON could be combined with MetriCal\u2019s <a href=\"https:\/\/docs.tangramvision.com\/metrical\/commands\/report\">report-mode<\/a> to reproduce the calibration report \/ console output that was created during the calibrate-mode run<\/p><ul><li data-preset-tag=\"p\"><p>This in some ways subsumes all other uses, as report-mode will utilize all of the data in a results file (JSON or MCAP) if provided<\/p><\/li><\/ul><\/li><\/ul><p>What this meant was that we needed to organize the data within a given results MCAP such that it corresponded with the appropriate use. To give a brief introduction of the MCAP format, you can typically write data to three different kinds of places:<\/p><ol><li data-preset-tag=\"p\"><p>A metadata record, which holds a key-value map of various string values<\/p><\/li><li data-preset-tag=\"p\"><p>Messages written to channels. Data written to a channel can be in any one of the well-known encodings<\/p><\/li><li data-preset-tag=\"p\"><p>Attachments of separate, single files that are related to the data, but not part of any specific channel stream<\/p><\/li><\/ol><p>Since you might find yourself using this sometime soon, we\u2019ll explain why data was placed in each section below. If you are reading this after getting a results MCAP yourself, we highly recommend installing the <a href=\"https:\/\/mcap.dev\/guides\/cli\">MCAP CLI tool<\/a> to aid in understanding what\u2019s being written under the hood.<\/p><h2>Metadata<\/h2><p>Tangram writes a <code>metrical<\/code> metadata record to the MCAP file. The metadata is primarily meant for debugging purposes, as it contains fairly sparse data about how MetriCal was run when the results MCAP was written. Specifically, we encode:<\/p><ul><li data-preset-tag=\"p\"><p><strong>program<\/strong>: the name of the software (metrical)<\/p><\/li><li data-preset-tag=\"p\"><p><strong>version<\/strong>: the version of the software that produced the results MCAP<\/p><\/li><li data-preset-tag=\"p\"><p><strong>arguments<\/strong>: the arguments passed to the process when it was invoked<\/p><\/li><\/ul><p>If you ever need to grab this information, any MCAP compatible software should work. For example with the MCAP CLI tool:<\/p><pre data-language=\"Shell\"><code>$ mcap get metadata --name metrical results.mcap\n{\n  \"arguments\": \"calibrate --camera-motion-threshold 3.5 -o results.mcap --report-path report.html camera_imu_box.mcap plex.json camera_imu_box_object.json --override-diagnostics\",\n  \"program\": \"metrical\",\n  \"version\": \"15.0.0-rc.0\"\n}<\/code><\/pre><h2>Channels<\/h2><p>Tangram writes all of our metrics data out to channels. The individual metrics can be broken down into three rough categories:<\/p><ol><li data-preset-tag=\"p\"><p>Pre-Calibration data. These metrics are any heuristics or data that we derive purely from the feature detection and data ingestion process.<\/p><\/li><li data-preset-tag=\"p\"><p>Residual metrics, abbreviated as just \u201cmetrics.\u201d These are directly related to the various optimization costs constructed during the calibration, and usually give some sense of the residuals that get minimized during a calibration run.<\/p><\/li><li data-preset-tag=\"p\"><p>Summary metrics. These are typically a summary of other residual metrics that may be useful to reference as a global view of the optimization.<\/p><\/li><\/ol><p>Using the MCAP CLI tool:<\/p><pre data-language=\"Shell\"><code>$ mcap info results.mcap\nlibrary:   mcap-rs-0.23.3\nprofile:\nmessages:  1096\nduration:  18.145069ms\nstart:     2025-09-04T09:15:37.638397288-06:00 (1756998937.638397288)\nend:       2025-09-04T09:15:37.656542357-06:00 (1756998937.656542357)\ncompression:\n\tlz4: [9\/9 chunks] [6.11 MiB\/4.15 MiB (31.99%)] [228.95 MiB\/sec]\nchannels:\n\t(1) \/pre_calibration\/topic_filter_statistics      4 msgs (220.45 Hz)     : tangram.pre_cal.TopicFilterStatistics [protobuf]\n\t(2) \/pre_calibration\/binned_feature_counts        3 msgs (165.33 Hz)     : tangram.pre_cal.BinnedFeatureCount [protobuf]\n\t(3) \/metrics\/image_reprojection                1072 msgs (59079.41 Hz)   : tangram.metrics.ImageReprojection [protobuf]\n\t(4) \/metrics\/composed_relative_extrinsics         6 msgs (330.67 Hz)     : tangram.metrics.ComposedExtrinsics [protobuf]\n\t(5) \/metrics\/imu_preintegration_error             3 msgs (165.33 Hz)     : tangram.metrics.ImuPreintegrationError [protobuf]\n\t(6) \/metrics\/object_inertial_extrinsics_error     3 msgs (165.33 Hz)     : tangram.metrics.ObjectInertialExtrinsicsError [protobuf]\n\t(7) \/summary\/overall                              1 msgs                 : tangram.summary.OverallSummary [protobuf]\n\t(8) \/summary\/camera                               3 msgs (165.33 Hz)     : tangram.summary.CameraSummary [protobuf]\n\t(9) \/summary\/imu                                  1 msgs                 : tangram.summary.ImuSummary [protobuf]\nchannels: 9\nattachments: 5\nmetadata: 1<\/code><\/pre><p>As can be seen, each of the above channels is prefixed with either <code>\/pre_calibration<\/code>, <code>\/metrics<\/code>, or <code>\/summary<\/code>, for each of the three sections above. As can be seen in the output above, we now have a collection of published schemata for each message kind. <a href=\"https:\/\/gitlab.com\/tangram-vision\/oss\/tangram-protobuf-messages#\">These can be found here<\/a>. Tangram has chosen to use Protobuf as the encoding kind for these messages because it provides a binary encoding of each of these messages and because Protobuf messages can be evolved in a backwards-compatible manner.<\/p><h2>Attachments<\/h2><p>Last but certainly not least are the attachments. Attachments are a bit of an odd duck in the MCAP space: while one can attach as many unrelated files to an MCAP as one chooses, attachments differ from messages on channels in that they:<\/p><ul><li data-preset-tag=\"p\"><p>Do not have an associated schema embedded into the MCAP<\/p><\/li><li data-preset-tag=\"p\"><p>Have listed offsets in the MCAP\u2019s summary section so that they can be quickly retrieved<\/p><\/li><\/ul><p>In terms of deciding what should be an attachment versus what should be a message on a channel, we opted to look at the typical data relationship that attachments imply. Notably, while you can have multiple attachments with the same name inside of a given MCAP, attachments are primarily optimized for the use-case where there is either zero or one attachment related to the topics listed in the MCAP. Conversely, channels tend to be optimized for the use-case where there may be 0 to N-many messages associated with a particular schema or topic.<\/p><p>Given that, we opted to attach some of the following entities to the MCAP itself:<\/p><ul><li data-preset-tag=\"p\"><p>The input plex<\/p><\/li><li data-preset-tag=\"p\"><p>The input object-space<\/p><\/li><li data-preset-tag=\"p\"><p>The optimized plex (post-calibration)<\/p><\/li><li data-preset-tag=\"p\"><p>The optimized object-space (post-calibration)<\/p><\/li><\/ul><p>For example:<\/p><pre data-language=\"Shell\"><code>$ mcap list attachments results.mcap\nname                  \tmedia type      \tlog time           \tcreation time\tcontent length\toffset \t\ninput-plex            \tapplication\/json\t1756998937638220277\t0            \t6148          \t291    \t\ninput-object-space    \tapplication\/json\t1756998937638237581\t0            \t931           \t6510   \t\nassociation           \tapplication\/json\t1756998937661339340\t0            \t2038494       \t4382014\t\noptimized-object-space\tapplication\/json\t1756998937663787485\t0            \t28972         \t6420580\t\noptimized-plex        \tapplication\/json\t1756998937663867444\t0            \t16197         \t6449635<\/code><\/pre><p>It was important in particular as part of these efforts to preserve the usage of both plex and object-space JSON files as part of our workflow. Not only do we often hand-edit these internally, these files can serve as (mostly) human-readable configurations for a given calibration run. By adding the plex and object-space as attachments to the MCAP rather than as channels proper, we have the advantage of being able to extract these entities easily without performing a search across chunks:<\/p><pre data-language=\"Shell\"><code>$ mcap get attachment --name optimized-plex results.mcap &gt; optimized-plex.json<\/code><\/pre><p>It is not always often that one needs to explicitly grab the optimized plex out of the results, since we usually advise users towards something like <a href=\"https:\/\/docs.tangramvision.com\/metrical\/commands\/shape\/shape_overview\">MetriCal\u2019s shape-mode<\/a> after calibration has succeeded. However, in many cases a plex or object JSON is conceptualized as a \u201cdistinct\u201d file relative to e.g. the dataset or results. In this way, using attachments connotes a closer connection to this mental model.<\/p><p>Note, however, that this step of extracting the plex is often totally unnecessary! MetriCal is also smart enough to pull the resultant plex or object-space from the results in context. For example, if you were to run <code>metrical-shape<\/code>:<\/p><pre data-language=\"Shell\"><code>$ metrical shape mst results.mcap .\/<\/code><\/pre><p>then, despite the input expecting a Plex, MetriCal will happily interpret the provided MCAP and extract the correct result plex for you.<\/p>","url":"https:\/\/www.tangramvision.com\/moving-metrical-metrics-to-mcaps","title":"Moving MetriCal Metrics to MCAPs","summary":"MCAPs aren't just for inputs anymore","date_modified":"2025-09-16T00:00:00.000Z","author":{"name":"[\"jeremy-steward\"]"},"tags":["company-news"]},{"id":"urn:sha256:559e7f56fcf70e8e4093eec3e01c9d35edccbec5b2455d1cff3a099e865ceb6a","content_html":"<p>I was recently discussing camera manufacturing with a roboticist who was building a stereo vision system. While most of our conversation covered familiar ground (sensor resolution, baseline distance, synchronization), one question caught my attention:<\/p><p>\"Why don't lens manufacturers just tell you which intrinsic model to use during calibration?\"<\/p><p>It's a perfectly reasonable question. Camera manufacturers invest enormous resources in optical design and should want their lenses to work seamlessly with computer vision applications. They certainly know their products better than anyone else. So why do we, as users, have to reverse-engineer distortion models for every lens we use?<\/p><p>The answer lies in the history of lens distortion modeling itself, and in the honest limitations that the originators of this field built into their work. To understand why manufacturers stay silent on calibration models, we need only to turn to one of the most widely used frameworks in computer vision: the Brown-Conrady model.<\/p><h2>But Why Lens Models?<\/h2><p>Everyone who's delved into computer vision has encountered the Brown-Conrady model. Based on publications over 100 years old, it remains the go-to for describing camera lens distortion, largely due to OpenCV\u2019s early adoption of the model as their default camera distortion profile.<\/p><p>The story begins in 1919 with A.E. Conrady's groundbreaking work in his publication \"<a href=\"https:\/\/academic.oup.com\/mnras\/article\/79\/5\/384\/1078771?login=false\">Decentred Lens-Systems<\/a>\". Working at a time when photogrammetry was still a nascent practice, Conrady developed the first rigorous mathematical treatment of lens decentering (aka tangential) distortion. His work, published in the Monthly Notices of the Royal Astronomical Society, laid the theoretical groundwork for understanding how imperfect lens alignment creates systematic errors in photographs. He discussed his analysis in terms of astronomy, but his results were applied to microscopy, photography, and even periscopes.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/V7MCAmxym0K6jUOybaTnQCMRiz8.png\"><p>However, it\u2019s important to note that \u201capplied\u201d here means that his work was used to better understand and improve lens construction techniques, not develop computational correction methods like we use today. Opticians of the era were aware of the aberrations produced by decentered lenses; Conrady\u2019s work was just the first to provide mathematical expressions for those effects.<\/p><p>It was Duane Brown's 1966 publication \"<a href=\"https:\/\/web.archive.org\/web\/20180312205006\/https:\/\/www.asprs.org\/wp-content\/uploads\/pers\/1966journal\/may\/1966_may_444-462.pdf\">Decentering Distortion of Lenses<\/a>\" that transformed Conrady's theoretical work into a practical calibration tool. Brown's contribution was big: he not only refined Conrady's mathematical model, but also demonstrated that decentering distortion could be calibrated analytically rather than requiring perfect physical alignment of lens elements.<\/p><h2>The Brown-Conrady Model Explained<\/h2><p>The Brown-Conrady model describes lens distortion as a combination of radial and tangential components:<\/p><ul><li data-preset-tag=\"p\"><p><strong>Radial<\/strong> distortion occurs when the lens elements aren't perfectly spherical, causing straight lines to appear curved (e.g. \u201dbarrel\" or \"pincushion\" effects)<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Tangential<\/strong> distortion (also called decentering distortion) results from imperfect alignment of lens elements, where the optical centers don't lie on a straight line.<\/p><\/li><\/ul><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/ldpUpUumSWzVUZxqSqfzXR44.png\"><p>The mathematical elegance of the Brown-Conrady model lies in its ability to describe these complex optical phenomena with a relatively simple polynomial expansion. This made it revolutionary for 1960s photogrammetry, where precise measurements from aerial photographs were crucial for mapping applications. MetriCal, Tangram\u2019s calibration suite, has more than one model based on this math directly: <a href=\"https:\/\/docs.tangramvision.com\/metrical\/calibration_models\/cameras?camera-models=opencv-radtan\">OpenCV\u2019s RadTan model<\/a> and our own <a href=\"https:\/\/docs.tangramvision.com\/metrical\/calibration_models\/cameras?camera-models=pinhole_with_brown_conrady\">Pinhole with BC<\/a>.<\/p><p>So that\u2019s it. After that, all camera manufacturers used Brown-Conrady for everything. Case closed, no notes.<\/p><h2>The Fundamental Problem: Models vs. Reality<\/h2><p>Just kidding. It\u2019s still messy.<\/p><p>Brown himself recognized a critical limitation in his work. In the same publication that he presented this unified model, he demonstrated that even the best lens systems exhibit distortion that varies with focus distance, aperture, and even zoom setting. He showed experimentally that distortion isn't constant across a lens's operating range. Instead, it's a complex, multidimensional function that changes with every adjustment you make to your camera.<\/p><p>Consider what this means for a modern zoom lens:<\/p><ul><li data-preset-tag=\"p\"><p>Distortion varies across the zoom range<\/p><\/li><li data-preset-tag=\"p\"><p>Distortion changes with focus distance<\/p><\/li><li data-preset-tag=\"p\"><p>Distortion shifts with aperture settings<\/p><\/li><li data-preset-tag=\"p\"><p>Even manufacturing tolerances mean identical lens models perform differently<\/p><\/li><\/ul><h2>The Impossibility of Perfect Profiles<\/h2><p>This is why camera manufacturers don't ship distortion profiles with their lenses: any fixed mathematical model would be, at best, an approximation valid only under specific conditions. A profile accurate at one focal length might be significantly wrong at another, or show just a part of the whole. A correction perfect for infinity focus could introduce errors when focused at minimum distance.<\/p><p>Brown's experimental work showed that residual errors after applying even carefully calibrated corrections could still amount to several microns, enough to cause noticeable artifacts in precision applications. For consumer photography, this might be fine. For autonomous and high-precision applications, they could become a straight-up safety risk.<\/p><p>Camera manufacturers understand that providing \"official\" distortion profiles would create unrealistic expectations of precision. Instead, they focus on optical design that minimizes distortion mechanically, leaving software corrections to third parties who can iterate and improve their models without being bound to a specific manufacturer's reputation.<\/p><h2>The Lesson<\/h2><p>Brown and Conrady's work remains foundational because it established the mathematical framework for understanding lens distortion. But their greatest insight might be the recognition that these are <em>just models<\/em>. They\u2019re useful approximations of reality, not perfect descriptions of it.<\/p><p>The next time you wonder why your lens doesn't come with a factory distortion profile, remember that the very researchers who developed our distortion models understood their limitations. Sometimes the most honest answer is acknowledging what we can't perfectly predict, rather than pretending our models capture every nuance of optical reality.<\/p><p>In the end, the Brown-Conrady model's enduring value isn't its perfection. Instead, it's the honest recognition that lens distortion is more complex than any single mathematical description can capture. That humility, built into the model from its inception, explains why it remains useful more than a century later.<\/p><h2>Further Reading<\/h2><p>Read Brown's paper for the math; read Conrady's paper for the prose.<\/p><ul><li data-preset-tag=\"p\"><p>A. E. Conrady, Decentred Lens-Systems,&nbsp;<em>Monthly Notices of the Royal Astronomical Society<\/em>, Volume 79, Issue 5, March 1919, Pages 384\u2013390,&nbsp;<a href=\"https:\/\/doi.org\/10.1093\/mnras\/79.5.384\">https:\/\/doi.org\/10.1093\/mnras\/79.5.384<\/a><\/p><\/li><li data-preset-tag=\"p\"><p>Duane C. Brown, Decentering Distortion of Lenses. 1966. <a href=\"https:\/\/web.archive.org\/web\/20180312205006\/https:\/\/www.asprs.org\/wp-content\/uploads\/pers\/1966journal\/may\/1966_may_444-462.pdf\">https:\/\/web.archive.org\/web\/20180312205006\/https:\/\/www.asprs.org\/wp-content\/uploads\/pers\/1966journal\/may\/1966_may_444-462.pdf<\/a><\/p><\/li><\/ul>","url":"https:\/\/www.tangramvision.com\/the-innovative-brown-conrady-model","title":"The Innovative Brown-Conrady Model","summary":"aka Why Camera Manufacturers Don't Profile Your Lens (And Why They're Right)","date_modified":"2025-08-15T15:02:31.229Z","author":{"name":"[\"brandon-minor\"]"},"tags":["sensor-science"]},{"id":"urn:sha256:131c33e5574070e64c46329eb1fa9c044fc3ff5da000311510c202dd01873ff0","content_html":"<h6>We held a weekly webinar that demonstrates LNS in action. Watch that here (<a href=\"https:\/\/youtu.be\/rcpc8IX31KA\" target=\"_blank\">link<\/a>).<\/h6><p>\u2014<\/p><p>In environments where GPS signals fade away, precise positioning becomes a critical challenge. Today, we're excited to announce that MetriCal now supports calibration of Local Navigation Systems (LNS), extending our industry-leading calibration platform to tackle odometry alignment with the rest of MetriCal's sensor modalities. Supporting this modality has been one of the oldest MetriCal requests; we're proud to fulfill it today.<\/p><h2>What Are Local Navigation Systems?<\/h2><p>Unlike GPS, which relies on satellite signals, LNS uses odometry data (precise measurements of position and orientation changes over time) to provide accurate localization indoors and in other GPS-denied scenarios.<\/p><p>Think of LNS as your system's internal compass and measuring tape combined. By tracking incremental movements and rotations, these systems build a detailed understanding of where they are and where they've been; this makes them essential for applications like:<\/p><ul><li data-preset-tag=\"p\"><p><strong>Autonomous warehouse robots -<\/strong> navigating complex facility layouts<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Manufacturing automation -<\/strong> requiring precise positioning on production lines<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Underground mining operations -<\/strong> where GPS is simply unavailable<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Indoor inspection drones -<\/strong> mapping building interiors<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Agricultural machinery -<\/strong> operating in covered facilities<\/p><\/li><\/ul><p>Even the most sophisticated local navigation system is only as good as its calibration. When LNS sensors work alongside cameras, LiDAR, and other perception hardware, precise calibration ensures that all sensors speak the same spatial language. MetriCal's LNS calibration solves these challenges by providing the same sub-millimeter accuracy that has made our platform the trusted choice for thousands of deployed systems.<\/p><h2>How MetriCal Handles LNS Calibration<\/h2><p>Our approach to LNS calibration leverages MetriCal's proven methodology while addressing the unique challenges of odometry-based positioning.<\/p><h3>Camera-Centric Approach<\/h3><p>MetriCal's LNS calibration requires at least one camera in the sensor suite. By comparing the world pose of the camera between frames with the interpolated motion from the local navigation system, we establish precise spatial relationships; this ensures perfect sensor alignment across your entire perception stack.<\/p><h3>Motion-Driven Precision<\/h3><p>LNS calibration is fundamentally data-driven. The system needs to observe motion across as many degrees of freedom as possible: translation along X, Y, and Z axes, plus rotation around each axis. This \"excitation\" allows MetriCal to distinguish true motion signals from sensor noise and bias; the result is calibrations you can trust in the most demanding applications.<\/p><h3>Data Intake<\/h3><p>MetriCal currently supports LNS data primarily through the MCAP format:<\/p><figure><table><tbody><tr><th><p><strong>Message Type<\/strong><\/p><\/th><th><p><strong>ROS1 Encodings<\/strong><\/p><\/th><th><p><strong>ROS2 with CDR Serialization<\/strong><\/p><\/th><\/tr><tr><td><p>Odometry<\/p><\/td><td><p><a href=\"https:\/\/docs.ros.org\/en\/noetic\/api\/nav_msgs\/html\/msg\/Odometry.html\" target=\"_blank\">nav_msgs\/Odometry<\/a><\/p><\/td><td><p><a href=\"https:\/\/github.com\/ros2\/common_interfaces\/blob\/iron\/nav_msgs\/msg\/Odometry.msg\" target=\"_blank\">nav_msgs\/msgs\/Odometry<\/a><\/p><\/td><\/tr><\/tbody><\/table><\/figure><p>If you have odometry data that doesn't fit these message types, let us know what you need by sending a quick note to <a href=\"mailto:info@tangramvison.com\" target=\"_blank\">info@tangramvison.com<\/a><\/p><h2>Getting Started with LNS Calibration<\/h2><p>Adding LNS calibration to your MetriCal workflow is straightforward. The process follows our familiar pattern:<\/p><ol><li data-preset-tag=\"p\"><p><strong>Data Collection<\/strong>: Capture synchronized data from your LNS and camera systems<\/p><\/li><li data-preset-tag=\"p\"><p><strong>System Configuration<\/strong>: Use MetriCal's initialization tools to identify LNS topics<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Calibration Execution<\/strong>: Run the calibration process to generate precise extrinsic parameters<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Results Analysis<\/strong>: Review comprehensive reports and metrics to validate calibration quality<\/p><\/li><\/ol><p>For teams already using MetriCal, integrating LNS calibration requires minimal learning curve; the same intuitive interface and proven workflows you know extend seamlessly to this new modality.<\/p><p>Download demo data along with a full walkthrough by visiting our calibration guide on LNS: <a href=\"https:\/\/docs.tangramvision.com\/metrical\/calibration_guides\/local_navigation_cal\" target=\"_blank\">https:\/\/docs.tangramvision.com\/metrical\/calibration_guides\/local_navigation_cal<\/a> <\/p><p>You can also watch us demonstrate MetriCal processing of an LNS dataset in this webinar: <\/p><iframe src=\"https:\/\/www.youtube.com\/embed\/rcpc8IX31KA?iv_load_policy=3&amp;rel=0&amp;modestbranding=1&amp;playsinline=1&amp;autoplay=0&amp;mute=1\" data-thumbnail=\"Medium Quality\" frameborder=\"0\" allow=\"presentation; fullscreen; accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\"><\/iframe><h2>Calibration Keeps Getting Easier<\/h2><p>As autonomous systems move into increasingly complex environments, the ability to operate precisely without GPS becomes mission-critical. MetriCal's LNS support represents our commitment to providing comprehensive calibration solutions that evolve with the needs of the robotics and autonomous systems industry.<\/p><p>Whether you're deploying warehouse automation, developing next-generation agricultural machinery, or building inspection systems for GPS-denied environments, MetriCal's LNS calibration ensures your sensors work together with the precision your applications demand. <a href=\"https:\/\/docs.tangramvision.com\/metrical\/configuration\/installation\" target=\"_blank\">Try it for yourself<\/a> today!<\/p>","url":"https:\/\/www.tangramvision.com\/introducing-local-navigation-systems-metrical-s-latest-calibration-capability","title":"Introducing Local Navigation Systems: MetriCal's Latest Modality","summary":"Odometry. Sensor-to-chassis. Local Nav. Whatever you call it, MetriCal now has your back.","date_modified":"2025-07-30T00:00:00.000Z","author":{"name":"[\"brandon-minor\"]"},"tags":["company-news"]},{"id":"urn:sha256:f63cc4e0edef3a4fca5eea3f68f9e66b7642d45a7ff820ac8a470bfcbfbd82a5","content_html":"<p><em>We've put together a jupyter notebook for this blog post. It can be found here: <\/em><a href=\"https:\/\/colab.research.google.com\/drive\/144pDQJARtzZ_GotbrLOVJfMM_ZU6QgYJ#scrollTo=a6234ff7-4fba-495b-89d7-2b3cd490188d\" target=\"_blank\"><em>https:\/\/colab.research.google.com\/drive\/144pDQJARtzZ_GotbrLOVJfMM_ZU6QgYJ#scrollTo=a6234ff7-4fba-495b-89d7-2b3cd490188d<\/em><\/a><\/p><p>\u2014<\/p><p>As a 3D computer vision professional, I\u2019ve been asked to fit a plane to a set of 3D points many times in my career. In fact, the take-home question in the interview which led to my first job out of school was on this very topic. The plane fitting problem arises naturally in computer vision: sometimes the plane itself is the salient feature of interest, or we\u2019re looking for the ground in a LiDAR scan so it can be rotated and viewed in a legible way. Either way, it\u2019s a common enough primitive that it\u2019s worth considering how we model and fit them. While this is well-trod territory, our <a href=\"https:\/\/www.tangramvision.com\/blog\/the-deceptively-asymmetric-unit-sphere\">recent blog post<\/a> about optimization, 3D rays, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lie_group\">Lie groups<\/a> has got me thinking about the problem a bit differently.<\/p><p>This article assumes some understanding of 3D computer vision, linear algebra and least-squares optimization. Of particular interest is the use of manifolds to represent quantities like 3D rotations and transformations.<\/p><h2>Fitting Planes the \"Usual\" Way<\/h2><p>First, let\u2019s briefly consider how we parameterize a plane in 3D. A common way is to indicate the direction of the plane\u2019s normal, represented as a 3D unit-vector, and a distance along the normal where we find the plane.<\/p><p>Therefore, a point $x\\in\\mathbf{R}^3$ on the plane satisfies the following equation:<\/p><p>$$ \\hat{n}^Tx+d=0 $$<\/p><p>With this parameterization, there\u2019s an ambiguity in sign. If we flip the sign of the normal, we can also flip the sign of the distance to get the same plane.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/397rWJHQHLxAHoEQSX7de8GJdhs.png\"><p>We can now use this formulation to fit a plane to a point sample. Let's use this one:<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/ccLDvuS3eoIOvsKk34aMblJkaI.png\"><p>The way I\u2019ve always done this was with a technique I learned in school. First, we calculate the centroid of all points, subtract it away from the set giving us a zero-mean set of points, centered about the origin.<\/p><p>$$ x_{centroid} = \\frac{1}{n}\\sum_{i}^{n}x_i \\\\ x_i \\in \\mathbf{R}^3 $$<\/p><p>Next we want to analyze the resulting zero-mean points to identify the plane normal. We can perform what is essentially <a href=\"https:\/\/en.wikipedia.org\/wiki\/Principal_component_analysis\">Principal Components Analysis<\/a> to do so. A 3D plane exists in three-dimensions, but it is fundamentally a 2D object. Therefore, you\u2019d expect most or all of the spread or variance of a 3D planar point-cloud to be \u201cexplained\u201d by a 2D subspace. PCA identifies directions which are weighted by how much of the data's variance is in that direction. Practically, PCA is the process of taking eigendecomposition of the sample covariance matrix. The eigenvectors give us these directions and the associated eigenvalues tell us how much of the data's variance is in that direction.<\/p><p>$$ COV_{sample}= \\sum_{i}^{n}(x_i - x_{centroid})(x_i - x_{centroid})^T \\in \\mathbf{R}^{3\\times3} $$<\/p><p>So for the ideal noiseless plane fitting problem, we'd expect two non-zero eigenvalues and one zero eigenvalue. The eigenvectors associated with the non-zero eigenvalues will span the subspace formed by the plane; the eigenvector associated with the zero eigenvalue, i.e. the one that spans the <em>null-space<\/em>, will correspond to the plane\u2019s normal. In practice, the presence of noise will cause the otherwise zero eigenvalue associated with the plane's normal to have some positive value, but for a mostly planar point cloud it will be dominated by the other two eigenvalues.<\/p><p>You can see this at work in the figure below. The eigenvectors are shown as colored arrows and are scaled by their associated eigenvalues. Our two large-magnitude eigenvalues\/vectors are represented in red and green; they correspond to the dimensions of most \"spread\" in the point distribution. The blue vector, meanwhile, only captures a fraction of the variance in this distribution. That blue vector is our plane normal!<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/IfebnCcm6WJTGSLROkfFUTqh0mA.png\"><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/NbgKDLFCcUb008woR5FZylw2Pck.png\"><p>So that explains how to fit $\\hat{n}$. To fit $d$ you take a point known to be on the plane (like the centroid might be, assuming there isn\u2019t a lot of noise or bias), and plug that into the above plane equation and solve.<\/p><h3>What To Do When the \u201cUsual\u201d Breaks Down<\/h3><p>PCA is a very effective technique. It\u2019s simple to implement, easily-explainable and works in the presence of noise, provided there aren\u2019t outliers\u2014e.g. the noise is zero-mean Gaussian noise. To handle outliers, you can wrap this technique in <a href=\"https:\/\/en.wikipedia.org\/wiki\/Random_sample_consensus\">RANSAC<\/a>. This is effective since fitting planes to small samples and testing for model fitness are fast and simple operations. But RANSAC has its drawbacks. A big one being that the time needed to fit the model is a function of the outlier rate\u2014 meaning it can be slow in the presence of many outliers. There are <a href=\"https:\/\/en.wikipedia.org\/wiki\/Robust_principal_component_analysis\">robust versions<\/a> of PCA, but as far as I understand, they\u2019re not straightforward extensions of the base algorithm.<\/p><p>What I\u2019d really like is a way to turn this into a parametric, unconstrained, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Non-linear_least_squares\">non-linear, least-squares problem<\/a>. Least-squares and non-linear least squares are the foundation of so many classic 3D vision techniques (ICP, Bundle Adjustment, Zhang\u2019s Calibration, KLT Tracking, Pose Graph Optimization\u2014 I could go on).<\/p><p>There are many advantages to using non-linear least-squares:<\/p><ul><li data-preset-tag=\"p\"><p>There\u2019s a good chance that in a given computer vision system, you may already be optimizing iterative, non-linear, least squares problems, meaning that there\u2019s already a system in place to optimize and solve.<\/p><\/li><li data-preset-tag=\"p\"><p>It\u2019s straightforward to jointly estimate multiple parameters at the same time, producing more consistent results. For example, if we decided to make the 3D sample points parameters themselves (because they\u2019re e.g. the result of triangulating between two views) and optimize them alongside the plane parameters, we may produce a more consistent result than estimating the 3D points and separately, later estimating the plane parameters.<\/p><\/li><li data-preset-tag=\"p\"><p>Extending the problem to weighted least squares opens up the possibility of downweighting low-confidence inputs (e.g the 3D sample points) and introducing priors on parameters (e.g. the plane normal)<\/p><\/li><li data-preset-tag=\"p\"><p>It\u2019s relatively easy to use <a href=\"https:\/\/en.wikipedia.org\/wiki\/Robust_statistics#M-estimators\">robust cost functions<\/a> to introduce outlier rejection to a non-linear least squares problem.<\/p><\/li><\/ul><h2>Plane Fitting the Least-Squares Way<\/h2><p>Posing the least squares plane fighting problem is straightforward. We\u2019ll use a point-to-plane cost function, not unlike what you see with some variants of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Iterative_closest_point\">ICP<\/a>.<\/p><p>$$ \\underset{\\hat{n},d}{\\text{argmin}}\\sum_{i}\\left\\lVert \\hat{n}^Tx_i-d \\right\\rVert_{2}^{2} $$<\/p><p>You may notice that this is actually a linear cost function. The trouble is that the normal needs to be a unit vector. So without further modifications, this becomes a constrained optimization problem, and that is a whole can of worms. We\u2019ve run into this problem before when trying to optimize over 3D transformations. Rotation matrices and quaternions have similar constraints that make them unsuitable for use directly in optimization problems.<\/p><h3>Unit Vector As Manifold?<\/h3><p>The solution is to optimize on-manifold. For rotations, <a href=\"https:\/\/en.wikipedia.org\/wiki\/3D_rotation_group\">SO(3)<\/a>\u2019s status as a Lie group makes this relatively easy. We just need an expression for the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Exponential_map_(Lie_theory)\">exponential<\/a>- and logarithm-maps and some corresponding jacobians, and we\u2019re good to go.<\/p><p>So can we do the same with a 3D unit vector? The space of 3D unit-vectors is the \u201cUnit 2-Sphere\u201d, or S2, which corresponds to an ordinary sphere as you or I would conceptualize. However, as a <a href=\"https:\/\/www.tangramvision.com\/blog\/the-deceptively-asymmetric-unit-sphere\">previous blog post<\/a> (see \u201cSymmetry of the Sphere\u201d) points out, <a href=\"http:\/\/en.wikipedia.org\/wiki\/Sphere#Dimensionality\">S2<\/a> is <em>not<\/em> a Lie group. Recall that the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hairy_ball_theorem\">Hairy-Ball Theorem<\/a> states, \u201c\u2026 that there is no nonvanishing continuous tangent vector field on even-dimensional n-spheres\u201d. S2 is an n-sphere of dimension 2, so the theorem unfortunately applies.<\/p><p>However, just because we\u2019re not optimizing over a Lie group doesn\u2019t mean we\u2019re stuck. You absolutely can optimize over a manifold that's not Lie group, but we'll save that for another article. Let\u2019s see if we can leverage rotations.<\/p><h3>Optimizing Over a Subset of Rotations<\/h3><p>Even before learning about S2\u2019s disqualification from being a Lie-group and how to work around that, I had the following geometrical intuition about how to use manifolds to reason about 3D unit vectors: We can parameterize a unit vector by starting with a canonical unit vector, let's say, the Z axis, and rotate it so that it points in a different direction to get a new unit vector. We can then use the rotation itself to parameterize the unit vector.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/zrRZv2kZj33kOVo3T99EA8GDy0.gif\"><h6>This animation shows how a vector (red) in the XY plane (gray) maps to a point on the unit sphere (green) by rotating (purple) the Z axis (blue).<\/h6><p>Our only issue is that a general rotation in 3D has three degrees of freedom, but a unit vector has just two\u2014 making this choice an overparameterization. The trick is to recognize that a rotation <em>about<\/em> the canonical unit vector doesn\u2019t change that vector. So we can think of S2 as mapping to an equivalence class of SO(3) where each member of S2 has a corresponding class of rotations that move the canonical vector to that point on the sphere, allowing for any roll around the canonical vector.<\/p><p>Practically speaking, this means we can fit unit vectors (like plane normals) by optimizing on the SO(3) manifold. When optimizing on SO(3), which is a Lie group, one typically works with the local tangent-space coordinates (this would be good time to review <a href=\"https:\/\/arxiv.org\/pdf\/1812.01537\">\u201cA micro Lie theory\u201d<\/a>) of which there are three parameters. In this case, with the Z-axis as our starting point, we\u2019ll set the third component to be zero since it has no effect on the resulting vector. That third parameter would have the effect of rotating a point that sits on the axis of rotation (which does nothing). This gives us two parameters ( $r_x, r_y$) with which to characterize the unit vector.<\/p><h3>Unit Vector As Manifold (Really)<\/h3><p>The local tangent-space coordinates for manifolds representing spatial rotations have a very useful property of being interpretable as an axis-angle representation of a rotation. The magnitude of the tangent-space coordinate is the angle, and the direction is the angle about which the rotation occurs. Hopefully it\u2019s clear that the tangent space of the Z-axis in 3D is the XY plane. Thus, we can interpret the 2D tangent-space coordinates of our rotation as a vector in the XY plane about which we rotate the Z-axis.<\/p><p>Any vector in the XY plane maps to a corresponding 3D unit vector. The mapping isn\u2019t unique since we can keep wrapping around the sphere, but it is \u201csmooth\u201d. A change in the tangent space coordinates can be related to a change in the resulting unit vector through the jacobian of the exponential map. The big trick to optimizing on-manifold is recognizing that the tangent space is just an unconstrained vector space, so we get to do all the things we\u2019re used to doing: calculus, optimization techniques like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Gauss%E2%80%93Newton_algorithm\">Gauss-Newton<\/a> etc.<\/p><h3>Fitting the Plane<\/h3><p>Let\u2019s see how to put all this into practice. Recall the original optimization problem:<\/p><p>$$ \\underset{\\hat{n},d}{\\text{argmin}}\\sum_{i}\\left\\lVert \\hat{n}^Tx_i-d \\right\\rVert_{2}^{2} $$<\/p><p>Let\u2019s replace $\\hat n$ with the rotation we described earlier<\/p><p>$$ \\underset{R,d}{\\text{argmin}}\\sum_{i}\\left\\lVert (R\\hat{z})^Tx_i-d \\right\\rVert_{2}^{2} $$<\/p><p>We\u2019re not actually going to optimize over a rotation matrix $R$, so let\u2019s introduce $r_x, r_y$ which parameterize our SO(3)\/SO(2) rotation and the exponential map $\\text{exp}(\\cdot)$ which maps those parameters to a rotation matrix or quaternion etc. that we use to apply the rotation.<\/p><p>$$ \\underset{r_x, r_y,d}{\\text{argmin}}\\sum_{i}\\left\\lVert (\\text{exp}([r_x r_y]) \\hat{z})^Tx_i-d \\right\\rVert_{2}^{2} $$<\/p><p>Now finally, our set of parameters $r_x, r_y,d$ form an unconstrained parameter vector and the problem is now a regular, unconstrained, nonlinear, least squares problem. To solve it, we could use something like Gauss Newton. We just need to evaluate the residual vector (the part inside the norm) and the jacobian of the residual with respect to the parameters.<\/p><p>In the jupyter <a href=\"https:\/\/colab.research.google.com\/drive\/144pDQJARtzZ_GotbrLOVJfMM_ZU6QgYJ#scrollTo=a6234ff7-4fba-495b-89d7-2b3cd490188d\" target=\"_blank\">notebook<\/a> for this post, I generated some random planar points and showed how to use the above technique to fit a plane. The optimization problem is solved with SciPy\u2019s <a href=\"https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.optimize.least_squares.html\">least squares solver<\/a> \u2014 simply pass in the residual, jacobian and data points.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/KjnchRVLPezEJyFSUbcjGXVuo.png\"><h6>A plane fit to some points.<\/h6><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/MwByl3UFgyMa0T4YkQHS0JBeYJE.png\"><h6>The same points and fit plane viewed on-edge<\/h6><p>Now we can try adding some gross outliers. As you can see, our fit plane no longer lines up very well with the data:<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/Y5M0sqy0Ez1B0wZmyN1orm1MfF8.png\"><h6>Both the PCA-based approach and the non-robust least-squares approaches struggle to fit the plane well in the presence of outliers<\/h6><p>However, now that we're using least-squares, we can easily make use of robust least squares fitting. Simply by changing one parameter, I can instruct SciPy to use a robust cost function (\u201dloss\u201d in their parlance), giving us robustness to outliers with a minimum of effort or increase in computational cost.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/y1O1QbwZQq9KOsdbxL3wAKYCc.png\"><h6>Same points with original fit (blue disc) and robust fit (reddish disc)<\/h6><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/udvftyE0WEkxqG2wRDr6Xs99qio.png\"><h6>Same points and fits viewed on edge<\/h6><p>Even if you\u2019re working outside a batteries-included library like SciPy, adding a robust cost to an iterative least squares system is fairly straightforward, certainly more simple than turning PCA into Robust PCA.<\/p><h2>On to S2\u2026<\/h2><p>We\u2019ve shown that plane fitting can be an on-manifold least squares problem. Despite S2 (the space of 3D unit vectors) not being a Lie group, we can still use SO(3) (which <em>is<\/em> a Lie group) to, in effect, fit a 3D unit vector and thus a plane normal. <\/p><p>In Part II, we\u2019ll use the same plane fitting problem to show how to optimize over S2 more directly, even though it is not a Lie group.<\/p><p>\u2014<\/p><h2>Further Reading<\/h2><ul><li data-preset-tag=\"p\"><p>For a good primer on Lie groups and algebra, check out \u201c<a href=\"https:\/\/arxiv.org\/pdf\/1812.01537\">A micro Lie theory for state estimation in robotics<\/a>\u201d.<\/p><\/li><\/ul>","url":"https:\/\/www.tangramvision.com\/a-different-way-to-think-about-plane-fitting","title":"Plane Fitting, Part 1: A Different Way To Think about Plane Fitting","summary":"We're using plane normals in not-so-normal ways","date_modified":"2025-07-23T15:52:16.039Z","author":{"name":"[\"paul-schroeder\"]"},"tags":["calibration-mathematics"]},{"id":"urn:sha256:f31696d27c22cfdeeb9b6741eac983b28ffcec6b0f210819a164d349ef6bba1a","content_html":"<p>Tangram Vision recently released <a href=\"https:\/\/docs.tangramvision.com\/metrical\/configuration\/installation#install-metrical-via-apt-repository\">our very own apt-repository<\/a> for downloading and installing MetriCal. This gives us the ability to ship MetriCal without relying on Docker, making it easier for users to install and use our software without first needing to understand containers. The journey to create this repository prompted many questions about how MetriCal would behave in the broader context of the system compared to running inside a container. One thing is true about any local software, container or not: sometimes, you will just have to read a file.<\/p><p>There is some surprising complexity embedded in how MetriCal interacts with the filesystem layer in Linux; we\u2019ve made (and fixed) plenty of mistakes over the years. After reviewing our own progress, we felt that these fixes presented a good introduction to a topic of intermediate difficulty in Rust: how does one build robust (and understandable) filesystem interactions? Moreover, are there any best-practices we can offer as guidance to others building great software for robotics?<\/p><p><em>Note<\/em>: We realize that some of the complexity of this topic is managed entirely differently across operating systems. As MetriCal operates primarily on Linux, try to understand that most of the advice listed here is based around the interactions that can occur on a Linux or POSIX-like system.<\/p><p>For those readers TL;DRing their way through this article, here\u2019s the punchline: Our short and sweet list of filesystem guidelines for any developer.<\/p><h2>Filesystem Guidelines for any Developer<\/h2><ol><li data-preset-tag=\"p\"><p>Paths should be treated like pointers to resources, and not owned resources unto themselves<\/p><\/li><li data-preset-tag=\"p\"><p>File descriptors or objects like <code>File<\/code> should almost exclusively be passed around in your software. Prevent future developers from re-using a path just because it \u201cseems fine at the time\u201d (hint: it is not fine).<\/p><\/li><li data-preset-tag=\"p\"><p>Outsource interactions with the filesystem to battle-tested software, rather than hand-rolling a recursive directory walk yourself.<\/p><\/li><li data-preset-tag=\"p\"><p>Aim to reduce or lint out the use of <code>Path::join<\/code>. While path-like objects are convenient, it is best to build more of our interfaces around the <code>std::io::Read<\/code> or <code>std::io::Write<\/code> traits, or if a <code>File<\/code> is absolutely necessary, the <code>File<\/code> type directly.<\/p><\/li><\/ol><p>Still here? Great! Let\u2019s dig into why these work.<\/p><h2>Filesystem Grievances<\/h2><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/WtrjAxnzNciWQhd7LPkeoyMRdMA.jpg\"><h6>Sometimes, computers hurt.<\/h6><p>\u201cFilesystem issues\u201d covers a broad range of topics: anything from opening a file with incorrect permissions, trying to log to a file such as our <code>report.html<\/code>, or reading ROS1 bag or MCAP data to optimize during the course of the calibration process. To keep this focused and in context, we are going to assume that you, the reader, are familiar with Rust and have used the <code>std::fs<\/code> APIs enough to at least be familiar enough with how they roughly work.<\/p><p>Based on our own interactions with the filesystem, our grievances can be broken down into one of the following categories:<\/p><ol><li data-preset-tag=\"p\"><p>Error communication when things go wrong<\/p><\/li><li data-preset-tag=\"p\"><p>Heavy path use and re-use<\/p><\/li><li data-preset-tag=\"p\"><p>Complex interactions being made more difficult due to not having good default APIs in Rust\u2019s standard library<\/p><\/li><\/ol><h3>Grievance #1: Error communication is difficult<\/h3><p>Consider the following example Rust program:<\/p><pre data-language=\"Rust\"><code>fn main() {\n    let err = std::fs::File::open(\"\/tmp\/not-a-file\");\n\n    let err = miette::Report::from_err(err);\n    println!(\"{err:?}\");\n}<\/code><\/pre><p>Running this program, we get an inscrutable error message:<\/p><pre data-language=\"Markdown\"><code>\u00d7 No such file or directory (os error 2)<\/code><\/pre><p>When this gets printed out to your console during runtime, it is very hard to understand what exactly the issue is, where it originated from, or <em>what the caller is supposed to do about it<\/em>.<\/p><p>This illustrates a fundamental issue: calling a program from the command line often means that you are thinking in and providing paths to the program, but the program\u2019s operation is entirely dependent on the abstractions provided by the host language (e.g. Rust\u2019s <code>std::fs<\/code>).<\/p><p>Users calling the software won\u2019t understand:<\/p><ul><li data-preset-tag=\"p\"><p>Which path did you try to use for this operation?<\/p><\/li><li data-preset-tag=\"p\"><p>What operation did you perform that resulted in the error?<\/p><\/li><li data-preset-tag=\"p\"><p>How can they fix it?<\/p><\/li><\/ul><p>It is not always evident whether all of these questions need answers, or if answers can be provided by the program itself.<\/p><p>Consider a program that fails due to trying to open a file whose path points to a broken symbolic link (symlink). Is the program responsible for understanding that the symlink is broken, or is there something fundamentally more broken about the caller\u2019s environment and machine state? We can\u2019t provide fine-grained support for issues without at least some <em>contextual understanding<\/em> of the system and how the software is being executed.<\/p><p>As MetriCal grows and evolves, a growing challenge is in how we report the error from the system itself (i.e. the error code and the message) and tying the surrounding context back to the error. Regrettably, this issue compounds with every other filesystem issue. When something goes wrong the ability to debug the issue is just as valuable as the actual fix itself.<\/p><h3>Grievance #2: Heavy path use and re-use<\/h3><p>Paths are a great abstraction to point to a specific resource on your system; after all, it is often said that in Unix, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Everything_is_a_file\">everything is a file<\/a>. Unfortunately, heavy usage of <code>Path<\/code>, <code>PathBuf<\/code>, and path-like objects (i.e. anything that is <code>AsRef&lt;Path&gt;<\/code> in Rust) lead to a whole host of different error categories that are very difficult to reproduce at scale. The worst of these are <a href=\"https:\/\/en.wikipedia.org\/wiki\/Time-of-check_to_time-of-use\">time-of-check-time-of-use<\/a> (TOCTOU) errors, which mostly arise from the fact that \u201cfiles\u201d themselves are a kind of global, mutable state on your system. <\/p><p>In most programs, the default \u201cabstraction\u201d over a file or resource is a file descriptor or file object (usually something like <code>std::fs::File<\/code>). MetriCal recently developed an error in how we generate our <code>report.html<\/code> reports from the console logs. The system had two different processes running on separate threads that were both trying to write to the same file simultaneously: a console-to-HTML converter and a logging system. Without proper thread synchronization, these competing writes created a race condition that sometimes corrupted the final report file. However, the <em>fundamental<\/em> reason this occurred is because we misused the abstraction that is paths and path-like objects.<\/p><h3>Grievance #3: Complex filesystem interactions<\/h3><p>Similarly, complex filesystem interactions such as recursively walking a directory can easily result in a number of TOCTOU errors even when paths are not re-used:<\/p><ul><li data-preset-tag=\"p\"><p>Infinitely recursing the directory over and over again because of a failure to check for hard or symbolic links that point to a higher directory. It is imperative to check for e.g. repeated inode \/ vnode information when recursing.<\/p><\/li><li data-preset-tag=\"p\"><p>Opening too many file descriptors at once and exhausting your operating system\u2019s limits on the number of open file descriptors.<\/p><\/li><li data-preset-tag=\"p\"><p>Crossing an unexpected filesystem boundary that may mean any inode \/ vnode tracking must be managed separately for each filesystem being traversed.<\/p><\/li><li data-preset-tag=\"p\"><p>Permissions and ownership of items in the directory may be entirely different than the permissions and ownership of the parent directory itself, resulting in a range of hard-to-debug permissions errors. Coupled with the aforementioned difficulty in error reporting, this can sometimes mean errors that are difficult to pin down and describe effectively to users.<\/p><\/li><\/ul><p>While Rust's <code>std::fs::read_dir<\/code> makes directory traversal look simple, recursively walking through all subdirectories involves many hidden system calls that can fail at any point. Rust's user-friendly API can mislead developers into thinking recursive directory operations are as straightforward as reading a single directory, when they're actually much more complex and error-prone.<\/p><p>This issue becomes particularly problematic when running software inside Docker, as we do with MetriCal. The software only recognizes paths within the Docker container (e.g., you'll see <code>\/volumes\/data\/my\/path\/to\/files<\/code> instead of <code>my\/path\/to\/files<\/code>). Additionally, problems like broken symlinks and missing data from incorrectly mounted filesystems can manifest in new ways, stemming entirely from improper Docker volume configuration for a specific system.<\/p><p>Part of our impetus for providing apt\/deb packages for MetriCal is that for a subset of users, running MetriCal locally without a container vastly simplifies their understanding of what happens when something complex and\/or erroneous does occur. Docker makes a lot of the deployment part of the software easy, but can ramp up complexity in managing how it interacts with local parts of the filesystem.<\/p><h2>Filesystems: Distributed Systems, Not Trees<\/h2><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/DKX8p0xjuNpaMMl4bjKdF4jYNo.jpg\"><h6>Not pictured: A filesystem<\/h6><p>A major contributor to problematic filesystem patterns is a fundamental misunderstanding of what the filesystem is. A common model taught to programmers about how to reason about the filesystem is that it is fundamentally a tree-like structure; however, this is only true of paths themselves, not the actual filesystem-as-an-abstraction.<\/p><p>Despite widespread belief that filesystems function like tree-like structures, they more closely resemble distributed systems. Hard links, symbolic links, and mounted directories (which can cross filesystem boundaries) all demonstrate ways in which filesystems aren't truly tree-like. The simplest way to understand this comparison is that filesystems sit beneath multiple processes and contain shared, mutable state that's accessed through the kernel's <em>virtual filesystem<\/em> (VFS) layer.<\/p><p>Many problems with filesystem patterns stem from the same issue seen in non-Rust programs\u2014specifically, a weak concept of ownership where other processes or the kernel itself might not honor ownership principles. Unlike memory management issues, however, we can't simply \"switch to Rust\" and expect the borrow-checker to magically resolve our filesystem challenges.<\/p><p>These errors primarily stem from developers' incomplete understanding of the distinction between paths and file objects, as well as a tendency to underestimate the complexity of filesystem operations. A good practice is to become familiar with how your operating system handles paths and file descriptors, along with the range of possible errors for each operation. For perspective, <code>std::io::ErrorKind<\/code> <a href=\"https:\/\/doc.rust-lang.org\/std\/io\/enum.ErrorKind.html\">contains over 30 different variants<\/a>. Understanding these errors and developing better ways to present them is essential for building robust software.<\/p><h2>Filesystem Guidelines for any Developer (Again)<\/h2><p>That understanding is how we got these common guidelines that developers should live by when writing software that interacts with the filesystem:<\/p><ol><li data-preset-tag=\"p\"><p>Paths should be treated like pointers to resources, and not owned resources unto themselves<\/p><\/li><li data-preset-tag=\"p\"><p>File descriptors or objects like <code>File<\/code> should almost exclusively be passed around in your software. Prevent future developers from re-using a path just because it \u201cseems fine at the time\u201d (hint: it is not fine).<\/p><\/li><li data-preset-tag=\"p\"><p>Outsource interactions with the filesystem to battle-tested software, rather than hand-rolling a recursive directory walk yourself.<\/p><\/li><li data-preset-tag=\"p\"><p>Aim to reduce or lint out the use of <code>Path::join<\/code>. While path-like objects are convenient, it is best to build more of our interfaces around the <code>std::io::Read<\/code> or <code>std::io::Write<\/code> traits, or if a <code>File<\/code> is absolutely necessary, the <code>File<\/code> type directly.<\/p><\/li><\/ol><h2>How Rust Makes This Better<\/h2><p>Fortunately, the errors listed in the previous sections were something that Tangram could tackle and solve. While we will probably never root out every possible filesystem problem in MetriCal, we have made great strides in doing so, and continue to find ways to make this better every day.<\/p><p>Here are some of the patterns we have leveraged and some of the crates that we found help us write better code by default day to day:<\/p><h3>Lint <code>std::fs<\/code> calls out of your project<\/h3><p>It is a bit heavy-handed, but an easy way to avoid the pitfalls of the <code>std::fs<\/code> APIs in Rust is simply to create a <code>clippy.toml<\/code> file and lint these types and functions out of your code entirely. An example of such a configuration might be:<\/p><pre data-language=\"JavaScript\"><code>disallowed-types = [\n  \"std::fs::DirEntry\",\n  \"std::fs::File\",\n  \"std::fs::OpenOptions\",\n  \"std::fs::ReadDir\",\n]\n\ndisallowed-methods = [\n  \"std::fs::canonicalize\",\n  \"std::fs::copy\",\n  \"std::fs::create_dir\",\n  \"std::fs::create_dir_all\",\n  \"std::fs::hard_link\",\n  \"std::fs::metadata\",\n  \"std::fs::read\",\n  \"std::fs::read_dir\",\n  \"std::fs::read_link\",\n  \"std::fs::read_to_string\",\n  \"std::fs::remove_dir\",\n  \"std::fs::remove_dir_all\",\n  \"std::fs::remove_file\",\n  \"std::fs::rename\",\n  \"std::fs::set_permissions\",\n  \"std::fs::soft_link\",\n  \"std::fs::symlink_metadata\",\n  \"std::fs::write\",\n  \"std::os::unix::fs::symlink\",\n  \"std::os::windows::fs::symlink_dir\",\n  \"std::os::windows::fs::symlink_file\",\n]<\/code><\/pre><p>These APIs are generally feature complete and the docs are definitely transparent about their limitations, but the problem is that they are often just as easy to misuse (and with disastrous error reporting \/ management) when things go wrong. However\u2026<\/p><h3>Use <code>fs-err<\/code> or <code>cap-std<\/code> as a replacement<\/h3><p>Fortunately there are replacements for <code>std::fs<\/code> that make a lot of these issues either:<\/p><ol><li data-preset-tag=\"p\"><p>Disappear entirely; or<\/p><\/li><li data-preset-tag=\"p\"><p>Much more difficult to misinterpret<\/p><\/li><\/ol><p>Two crates we love: <code><a href=\"https:\/\/docs.rs\/fs-err\/latest\/fs_err\/index.html\">fs-err<\/a><\/code> and <code><a href=\"https:\/\/docs.rs\/cap-std\/latest\/cap_std\/fs\/index.html\">cap-std<\/a><\/code>. It is beyond the scope of this article to demonstrate everything in these libraries, but consider just <code>fs-err<\/code> which advertises itself as a drop-in replacement for <code>std::fs<\/code> . Using our previous sample program, errors from this crate look like the following:<\/p><pre data-language=\"Rust\"><code>fn main() {\n    \/\/ Notice how we replaced std::fs::File with fs_err::File\n    let err = fs_err::File::open(\"\/tmp\/not-a-file\").unwrap_err();\n\n    let err = miette::Report::from_err(err);\n    println!(\"{err:?}\");\n}<\/code><\/pre><p>And produce the following output:<\/p><pre data-language=\"Markdown\"><code>\u00d7 failed to open file `\/tmp\/not-a-file`: No such file or directory (os error 2)<\/code><\/pre><p>\u2026which is significantly clearer than our initial pass! This error message includes both the operation (\"failed to open file\") and the specific path (<code>\/tmp\/not-a-file<\/code>) that caused the failure. For this reason alone, replacing <code>std::fs<\/code> with an alternative like <code>fs-err<\/code> represents a clear improvement for end-users, all without requiring any additional code instrumentation.<\/p><h3>Use <code>walkdir<\/code> for iterating over directory contents<\/h3><p>Likewise, we heartily support the <code><a href=\"https:\/\/docs.rs\/walkdir\/2.5.0\/walkdir\/index.html\">walkdir<\/a><\/code> crate. Like <code>fs-err<\/code> in the previous section, <code>walkdir<\/code> handles most of the underlying complexity of recursively walking a directory and iterating over files. This crate is fantastic because it can easily be configured by default to skip recursing over symlinks and separate filesystems.<\/p><p>Overall, the implementation avoids the vast majority of the obvious bugs for recursing through a directory and provides a compelling implementation that\u2019s trusted by many Rust users, including the fantastic <code><a href=\"https:\/\/docs.rs\/ignore\/latest\/ignore\/index.html\">ignore<\/a><\/code> crate and <code><a href=\"https:\/\/github.com\/sharkdp\/fd\">fd-find<\/a><\/code> tool.<\/p><h3>Avoid passing path-like objects around on their own<\/h3><p>The simplest way to avoid path re-use is to <em>eliminate path-like objects entirely<\/em>. Passing around generic path types like <code>P: AsRef&lt;Path&gt;<\/code> or <code>PathBuf<\/code>, especially beyond initial command line argument parsing, often leads to the problematic patterns we've discussed. While we can't immediately open every file that MetriCal might use, we should never clone or pass around paths without carefully considering file descriptor access and ownership. As with all Rust code, proper ownership thinking is the path to victory.<\/p><p>We open file descriptors as late as possible for pragmatic reasons, but aim to discard paths and path-like objects as soon as we can. Crates like <code>cap-std<\/code> enhance this approach by using the <code>openat<\/code> family of calls to provide a capability-first API that only allows relative paths (and lets you control whether to accept symlinks).<\/p><h2>Conclusion<\/h2><p>Filesystems are tricky beasts, and some of the most difficult bugs can arise due to less-than-obvious mismatches in how we think about our abstractions (paths and files-as-individually-owned-resources) with how our system treats our filesystem (closer to a distributed system of global mutable state). While many of the bugs explored here never resulted in extreme data corruption or the deletion of customer data, many did result in annoying error messages or less-than-graceful behaviour when reading or writing calibration data from MetriCal.<\/p><p>While Rust cannot always magically solve all filesystem interactions, the patterns and libraries outlined above continue to improve MetriCal's robustness. By adopting similar approaches in your own code, you can avoid many common pitfalls and build more reliable software. We hope you do!<\/p><h2>Further Reading<\/h2><ul><li data-preset-tag=\"p\"><p>See the justification for something like <code>cap-std<\/code> \/ <code>openat<\/code> by reading the \u201cRationale for openat()\u201d section at <a href=\"https:\/\/www.man7.org\/linux\/man-pages\/man2\/openat.2.html\">https:\/\/www.man7.org\/linux\/man-pages\/man2\/openat.2.html<\/a><\/p><\/li><li data-preset-tag=\"p\"><p>A good primer on how TOCTOU errors can occur in practice is here: <a href=\"https:\/\/val.packett.cool\/blog\/use-openat\/\">path.join Considered Harmful, or openat() All The Things<\/a><\/p><\/li><\/ul>","url":"https:\/\/www.tangramvision.com\/building-robust-filesystem-interactions-in-rust","title":"Building Robust Filesystem Interactions in Rust","summary":"Sometimes, unfortunately, you just have to read a file","date_modified":"2025-07-15T15:23:03.071Z","author":{"name":"[\"jeremy-steward\"]"},"tags":["programming-insights"]},{"id":"urn:sha256:41307162e55a19aab64f1e292f72bc0763a4b2af556892de6f396d23aeadf89e","content_html":"<p>We're proud to officially announce that Tangram Vision has been awarded a NASA Phase II SBIR to solve one of autonomous robotics' biggest challenges: keeping sensor systems calibrated without human intervention. Our work will advance autonomous lunar navigation for NASA's Artemis program and establish the foundation for a long-term autonomous lunar presence.<\/p><p><strong>Design Partners Wanted: <\/strong>We\u2019re looking for corporate design partners for this project. Read more below, then email us at <a href=\"mailto:info@tangramvision.com\">info@tangramvision.com<\/a> with your own use case and needs.<\/p><h1>The Challenge<\/h1><p>NASA's Artemis program aims to establish a permanent lunar base as a stepping stone to Mars. But this won't just be a human outpost - autonomous systems will handle surveying, discovery, and maintenance operations, often working alone for extended periods.<\/p><p>Consider the sensor suites that <a href=\"https:\/\/www.tangramvision.com\/blog\/sensing-breakdown-waymo-jaguar-i-pace-robotaxi\">sophisticated<\/a> <a href=\"https:\/\/www.tangramvision.com\/blog\/sensing-breakdown-locomation-autonomous-relay-convoy\">autonomy<\/a> <a href=\"https:\/\/www.tangramvision.com\/blog\/sensing-breakdown-farmwise-ft35\">companies<\/a> deploy on Earth - complex camera arrays, LiDAR, and IMUs that require constant calibration and maintenance. Even with dedicated field engineers on call, keeping these systems perfectly aligned is challenging. Now imagine that same task on the Moon, where harsh conditions constantly affect sensor performance and no help is available.<\/p><h2>The Unforgiving Lunar Surface<\/h2><p>The lunar environment presents unique sensing challenges:<\/p><ul><li data-preset-tag=\"p\"><p><strong>Radiation bombardment:<\/strong> Without atmosphere or magnetic field protection, space radiation gradually degrades sensor components.<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Lunar dust:<\/strong> Fine particles enhance reflectance, increase contrast, and statically adhere to sensor surfaces, changing their optical properties over time.<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Extreme lighting:<\/strong> Minimal axial tilt creates persistent shadows alongside areas of intense, continuous sunlight - imagine trying to navigate from bright stadium lighting directly into a cave!<\/p><\/li><\/ul><iframe src=\"https:\/\/www.youtube.com\/embed\/XHpBK_ErIy0?iv_load_policy=3&amp;rel=0&amp;modestbranding=1&amp;playsinline=1&amp;autoplay=0&amp;mute=1\" data-thumbnail=\"Medium Quality\" frameborder=\"0\" allow=\"presentation; fullscreen; accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\"><\/iframe><h6>Shadows on the lunar surface. Credit: NASA's Scientific Visualization Studio<\/h6><p>This creates a perfect storm for autonomous systems:<\/p><ol><li data-preset-tag=\"p\"><p><strong>Immediate challenge<\/strong>: Traditional vision systems fail in extreme lunar lighting conditions.<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Long-term challenge<\/strong>: The harsh environment gradually changes sensor behavior, making initial calibrations obsolete.<\/p><\/li><\/ol><p>The solution requires both novel sensor use <em>and<\/em> the ability to recalibrate autonomously.<\/p><h1>Our Approach<\/h1><p>Our approach tackles both challenges simultaneously. We're developing an <a href=\"https:\/\/www.tangramvision.com\/blog\/sensoria-obscura-event-cameras-part-i\">event-based<\/a> stereo vision system combined with an IMU that breaks new ground in how robots perceive in harsh environments.<\/p><h2>Benefit of Event Cameras<\/h2><p>Event cameras offer three critical advantages for lunar operations:<\/p><ul><li data-preset-tag=\"p\"><p><strong>Extreme dynamic range<\/strong> (120dB vs 60dB for traditional cameras) - they can simultaneously see details in deep shadows and bright sunlight.<\/p><\/li><li data-preset-tag=\"p\"><p><strong>Ultra-low power consumption<\/strong> - using just 10mW compared to 5-10W for traditional cameras, crucial for energy-limited space missions.<\/p><\/li><li data-preset-tag=\"p\"><p><strong>High-speed performance<\/strong> - microsecond response times enable navigation during rapid lighting changes (aka moving fast across lunar terrain).<\/p><\/li><\/ul><p>Our Phase I grant work was focused on proving out the use case of such a stereo system for lunar navigation. That work proved successful: when used in an odometry system, our stereo rig just might solve the issue of navigation in lunar lighting\u2026 assuming it remains calibrated.<\/p><h2>AutoCal<\/h2><p>This brings us to the real breakthrough in Phase II: <strong>AutoCal<\/strong> - our operator-free multi-modal calibration system.<\/p><p>During Phase I, we demonstrated calibration of event camera + IMU pairs using <a href=\"https:\/\/www.tangramvision.com\/metrical\">MetriCal<\/a>, our production line calibration software. However, MetriCal relies on rigid <a href=\"https:\/\/docs.tangramvision.com\/metrical\/targets\/target_overview\">targets<\/a> in order to derive calibration figures and runs best in a controlled environment (think turntables or robot arms). MetriCal is best in its class on the factory floor, but autonomous vehicles on the Moon won\u2019t be in a factory; they\u2019ll be jet-setting along the lunar surface. They\u2019ll need constant calibration verification and refinement during deployment to successfully operate.<\/p><p>That\u2019s AutoCal: <strong>calibration that happens during normal operation<\/strong>, without targets, turntables, or human intervention. Starting with a MetriCal-verified calibration, AutoCal will monitor and tweak those calibration parameters in response to the changing operating conditions of the lunar surface. It uses environmental features and vehicle motion to continuously verify and adjust sensor alignment \u2014 essentially turning every drive into a calibration session.<\/p><p>Developing AutoCal is the main technical focus at Tangram Vision for the next 18 months (and maybe forever).<\/p><h1>Become A Design Partner<\/h1><p><strong>The terrestrial opportunity is immediate.<\/strong> By solving calibration for the Moon's extreme conditions, we're creating technology that will transform Earth-based robotics. Think of AutoCal as \"space-grade reliability\" for any environment - from wind farms to mining operations to autonomous agriculture.<\/p><p><strong>We're seeking design partners<\/strong> to help us develop AutoCal in any and all terrestrial environments. If your autonomous systems require continuous operation in interesting or challenging conditions, we want to work with you. Contact us at <a href=\"mailto:info@tangramvision.com\">info@tangramvision.com<\/a> with your company details and use case. Space is limited; in order to best focus our efforts, we plan to restrict the number of partners we work with for the foreseeable future.<\/p><p>\u2014<\/p><p>We couldn\u2019t be more proud to support NASA and the organization\u2019s lunar exploration goals. This project represents everything Tangram Vision stands for: solving the hardest perception problems to make autonomous systems reliable everywhere they're needed. From the Moon to your backyard, we're building the future of autonomous perception.<\/p>","url":"https:\/\/www.tangramvision.com\/tangram-vision-awarded-nasa-grant-for-operator-free-calibration-in-space","title":"Tangram Vision Awarded NASA Grant for Operator-free Calibration in Space","summary":"Tangram Vision has been awarded a NASA Phase II SBIR for continuous perception calibration without human intervention.","date_modified":"2025-07-08T16:12:50.175Z","author":{"name":"[\"brandon-minor\"]"},"tags":["company-news"]},{"id":"urn:sha256:d6c7bcf2437afce7d5056a28d41b0ed77693633f955333de7087d1e25e0b5603","content_html":"<p>Hello, all. We\u2019re Tangram Vision, the Reliable Perception company. We do calibration.<\/p><p>If you've ever worked on calibration, you know it touches nearly every system in robotics, automation, and physical AI. Everything using perception data eventually needs calibration, which means calibration engineers end up becoming experts in... well, everything.<\/p><p>This creates an interesting problem. Calibration experts can spot issues everywhere: navigation algorithms that will drift off course, deep learning pipelines making systematic errors, timing problems between sensors. With this broad knowledge comes an irresistible yearning to fix every problem they encounter.<\/p><p>For the past four years, we at Tangram have yearned. We always knew that calibration was a pain point for robotics, but what about everything calibration touches? Could we fix those on the way? We tinkered and toyed with big ideas to make the robotics world better, but ultimately they never found purchase:<\/p><ul><li data-preset-tag=\"p\"><p>What if there was a universal data format for sensors? That was .tang, an idea we ditched in our first year (thanks, MCAP!).<\/p><\/li><li data-preset-tag=\"p\"><p>What if we could build a system that streamed and fused data from any sensor type in real-time? That\u2019s <a href=\"https:\/\/www.youtube.com\/watch?v=jwJ6XAS8WM4\" target=\"_blank\">TVMux<\/a>, an API which we ultimately never deployed. It\u2019s kept in-house as a streaming solution for our own future needs.<\/p><\/li><li data-preset-tag=\"p\"><p>What if we built a self-calibrating sensor? That was <a href=\"https:\/\/www.youtube.com\/watch?v=KJNyf0906ow\" target=\"_blank\">HiFi<\/a>, a Kickstarter project that we shelved early this year. I\u2019ve only recently finished refunding all of our pre-orders and backers for that effort (thanks to all who supported us, by the way!)<\/p><\/li><\/ul><p>It\u2019s not that these are bad ideas. Far from it; I think they\u2019re all fantastic ideas, and I\u2019d love to revisit these when we\u2019re in a place to do so. However, time is a zero-sum commodity. By using time here, we were diverting time from the problem that we were uniquely capable of solving: calibration. Our core mission took a hit, along with our business.<\/p><p>So starting in 2025, we pulled plugs, accepted sunk costs, and focused up. All engineering time now goes directly to making our calibration software better: better user experiences, better mathematics, better integration. <a href=\"https:\/\/www.tangramvision.com\/metrical\" target=\"_blank\">MetriCal<\/a> is getting more modalities and faster runtimes. We\u2019re iterating on sales and pricing to make our products affordable to everyone. NASA just awarded us a Phase II SBIR to build extraterrestrial calibration (which we\u2019ll post about soon). Even our branding and messaging has been honed; you can <a href=\"https:\/\/www.tangramvision.com\/\">check out the website<\/a> for the new look. We\u2019re laser-focused, and it feels amazing.<\/p><p>It\u2019s true that we\u2019ve been a little quiet. But we\u2019re still here, still busy, and doing great. We're excited to share more about what we've been building - both in upcoming blog posts and in the product updates you'll start seeing again. We hope to make your life just a little bit better by making calibration suck a little bit less. Thanks for sticking with us.<\/p>","url":"https:\/\/www.tangramvision.com\/re-introducing-tangram-vision","title":"Re-introducing Tangram Vision","summary":"Unbothered. Happy. In Our Lane. Focused. Flourishing.","date_modified":"2025-07-02T00:00:00.000Z","author":{"name":"[\"brandon-minor\"]"},"tags":["company-news"]},{"id":"urn:sha256:a36c20fdd14bc8a9870cd36325d3f85da3e171dbd7837e710584739ade13852c","content_html":"<p>Previously, we talked about <a href=\"https:\/\/www.tangramvision.com\/blog\/camera-modeling-pinhole-obsession\">Pinhole Obsession<\/a> and the downsides of assuming a default pinhole model. We presented an alternative formulation where rays are modeled on the unit sphere \\(\\mathcal{S}^2\\) instead of the normalized image plane \\(\\mathbb{T}^2\\).<\/p><p>We\u2019ve also previously written posts about how many problems in robotics can be formulated as <a href=\"https:\/\/www.tangramvision.com\/blog\/introduction-to-optimization-theory\">an optimization<\/a>. At a high level, many continuous optimization algorithms perform the following steps<\/p><ul><li data-preset-tag=\"p\"><p>Compute a quantity to minimize \u2014 error, misalignment, risk, loss, etc.<\/p><\/li><li data-preset-tag=\"p\"><p>Compute a direction along which the quantity is locally reduced<\/p><\/li><li data-preset-tag=\"p\"><p>Move the parameters in the quantity-reducing direction<\/p><\/li><li data-preset-tag=\"p\"><p>Repeat until the problem converges<\/p><\/li><\/ul><p>Many continuous optimization problems are modeled on <a href=\"https:\/\/en.wikipedia.org\/wiki\/Vector_space\">vector spaces<\/a> and \u201cmoving\u201d the parameters is simply vector addition. However, our prescription of using the unit sphere to model rays is obviously <strong>not<\/strong> a linear space! Therefore, we must start our discussion by describing how to travel on the sphere.<\/p><h2>Traveling on the Sphere<\/h2><p>Generally, when we <em>travel<\/em> we want to do so in the most efficient manner. Undoubtedly, you\u2019ve heard the expression \u201cget from point a to b,\u201d perhaps also with the implication \u201cas fast as possible.\u201d Therefore, we\u2019ll want to minimize the distance that we travel or equivalently travel along the shortest path. In Differential Geometry, we call these \u201cshortest paths\u201d <em>geodesics.<\/em><\/p><p>An alternate way to view a <em>geodesic<\/em> is by starting at a point \\(p\\) on the manifold and traveling in the direction of some vector \\(v\\), carefully minimizing the distance at each increment. However, minimizing the distance at each step may mean we have to <em>change direction<\/em> along our shortest path.<\/p><p>\u201cChanging direction\u201d and \u201ctraveling in the shortest path\u201d may seem to counter one another. After all, we all know that the shortest path between two points is a line. However, imagine we are flying in a plane starting at the equator and a heading of 45\u00b0. If our goal is to travel <em>as far as possible<\/em> from our starting point, how should we chart our course?<\/p><blockquote><p>\ud83d\udca1 As usual, ignore air resistance, the jet stream and assume a spherical earth. This is a thought exercise, not pilot\u2019s school.<\/p><\/blockquote><p>A simple suggestion would be to maintain a heading of 45\u00b0 during our travel, like we would locally: travel in a constant direction. However, if we maintain a fixed heading indefinitely, we will end up close to the north pole! The line traced by a path of constant heading is known as a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Rhumb_line\">Rhumb Line<\/a> and, as pictured, is certainly not the shortest distance between two points on a sphere.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/MFfDPY5puAGQMUAiEMHQSwM54Xo.png\"><p><em>PC: <\/em><a href=\"https:\/\/en.wikipedia.org\/wiki\/Rhumb_line#\/media\/File:Loxodrome.png\"><em>https:\/\/en.wikipedia.org\/wiki\/Rhumb_line#\/media\/File:Loxodrome.png<\/em><\/a><\/p><p>Alternatively, to truly travel the <em>farthest distance,<\/em> we travel along the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Great-circle_distance\">great circle<\/a> until the aircraft runs out of fuel. Here it\u2019s more obvious that the geodesic\u2019s heading is continuously changing. One implication of this is that if we start at one point along the geodesic with a vector \\(v\\), we may not end up in the same location as if we start at a different point along the geodesic with the same vector \\(v\\).<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/dqdZV1eW7wAMdiv4IYdJw2w2dsA.png\"><p><em>PC: <\/em><a href=\"https:\/\/en.wikipedia.org\/wiki\/Great-circle_distance#\/media\/File:Illustration_of_great-circle_distance.svg\"><em>https:\/\/en.wikipedia.org\/wiki\/Great-circle_distance#\/media\/File:Illustration_of_great-circle_distance.svg<\/em><\/a><\/p><p>In Differential Geometry, the operator that traces the geodesic is known as \u201cthe exponential map.\u201d For the reasons listed above, it\u2019s defined as a local operator originating at a point \\(p\\) and traveling in a direction \\(v\\) i.e.<\/p><p>$$<br>\\text{Exp}_p(v)<br>$$<\/p><p>Similarly, the operator that computes the direction and distance between two points on the manifold (the manifold\u2019s logarithm) is defined as a local operator \\(\\text{Log}_p(q)\\).<\/p><p>So to perform optimization on the unit sphere, we<\/p><ul><li data-preset-tag=\"p\"><p>Compute the quantity to minimize \u2014 error, misalignment, risk, loss, etc.<\/p><\/li><li data-preset-tag=\"p\"><p>Compute a direction \\(v\\) along which the quantity can be locally reduced<\/p><\/li><li data-preset-tag=\"p\"><p>Move the parameters along the great circle using the exponential map \\(\\text{Exp}_p(v)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Repeat until the problem converges<\/p><\/li><\/ul><p>This begs the question: how do we compute this direction \\(v\\)? What does it really mean?<\/p><h2>The Briefest Introduction to Differential Geometry<\/h2><blockquote><p>\u26a0\ufe0f Many maths below! But they\u2019re cool maths, we promise.<\/p><\/blockquote><p>To explain what a direction \\(v\\) means on the sphere and to explain why working with \\(\\mathcal{S}^2\\) is difficult, we have to briefly touch on one of my favorite subjects: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Differential_geometry\">Differential Geometry<\/a>. This post will dive deep into the motivations behind why we use differential geometry in computer vision, and what advantages it brings. Much of this may seem like a tangent (no pun intended), but don\u2019t worry: we\u2019ll bring it all back to our Pinhole Obsession and optimization at the end!<\/p><blockquote><p>\ud83d\udca1 Obviously, we cannot cover the entirety of Differential Geometry in one blog post. There are many great resources to learn Differential Geometry. Some of our favorites are:<\/p><ul><li data-preset-tag=\"p\"><p><a href=\"https:\/\/link.springer.com\/book\/10.1007\/978-1-4419-9982-5\">Introduction to Smooth Manifolds<\/a><\/p><\/li><li data-preset-tag=\"p\"><p><a href=\"https:\/\/arxiv.org\/pdf\/1812.01537\">A Micro Lie Theory for State Estimation in Robotics<\/a><\/p><\/li><li data-preset-tag=\"p\"><p><a href=\"https:\/\/link.springer.com\/book\/10.1007\/978-3-030-46040-2\">Differential Geometry and Lie Groups<\/a><\/p><\/li><\/ul><\/blockquote><p>Historically, differential geometry arose from the study of \u201ccurves and surfaces.\u201d As traditionally taught in multi-variable calculus, a curve is a mapping \\(\\gamma: \\mathbb{R} \\to \\mathbb{R}^3\\) and likewise, a surface is a mapping \\(\\sigma: \\mathbb{R}^2 \\to \\mathbb{R}^3\\).<\/p><p>Differential geometry generalizes these \u201ccurves and surfaces\u201d to arbitrary dimensions. These generalized \u201ccurves and surfaces\u201d are known as <em>smooth manifolds<\/em>. In this post, we\u2019ll skip the rigorous definition of a smooth manifold and simply define it as<\/p><blockquote><p><em>Smooth Manifold:<\/em> An N-dimensional space without corners, edges or self-intersections.<\/p><\/blockquote><p>This smoothness (called continuity) allows us to perform optimization \u201con the manifold\u201d using our standard calculus techniques. We can compute errors and also compute the directions along which we can reduce those errors.<\/p><h2>Intrinsic vs Extrinsic Manifolds<\/h2><p>To assist intuition, we\u2019ve leaned on prior knowledge of multi-variable calculus of curves and surfaces. At the undergraduate level, curves and surfaces are presented as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Nash_embedding_theorems\">embedded entities<\/a> living in a higher-dimensional ambient space e.g. 3D Euclidean Space. However, this ambient space isn\u2019t <em>strictly<\/em> needed for the development of differential geometry. In their search for a minimal set of axioms, differential geometers have taken great care to remove this dependence on an ambient space; they describe <em>smooth manifolds<\/em> as objects existing in their own right. Thus, there are two ways of thinking about differential geometry:<\/p><ul><li data-preset-tag=\"p\"><p><em>Extrinsic<\/em>: manifolds as embedded objects in space<\/p><\/li><li data-preset-tag=\"p\"><p><em>Intrinsic<\/em>: manifolds are entities in their own right<\/p><\/li><\/ul><p>In this manner, the error-minimizing direction we are searching for can either be viewed as <em>extrinsic<\/em> (embedded in ambient space) or it can be viewed as <em>intrinsic<\/em> (living alongside the manifold).<\/p><p>As engineers, the decision to use the intrinsic view of a manifold vs. the extrinsic view mostly impacts representation and computation. Thus, for a specific manifold, we choose the representation that\u2019s computationally easiest to work with. Although the remainder of this post only briefly touches on the representation of manifolds, it\u2019s <strong>always<\/strong> necessary to determine what representation is being used. The decision of an intrinsic or extrinsic representation will change how computation is performed on the manifold. Regardless of the choice of <em>intrinsic<\/em> or <em>extrinsic<\/em> representation, we often \u201cbundle\u201d the representation of the manifold with the representation directions on that manifold.<\/p><blockquote><p>\u26a0\ufe0f The concepts of <em>intrinsic<\/em> and <em>extrinsic<\/em> used in differential geometry are unrelated to intrinsics (interior orientation) and extrinsics (exterior orientation) of cameras. This is simply an unfortunate naming collision at the intersection of two fields.<\/p><\/blockquote><h2>A Brief Tangent for Tangents<\/h2><p>\u2026which brings us to <strong>the<\/strong> key concept in differential geometry: the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tangent_bundle\">Tangent Bundle<\/a>. The Tangent Bundle is the underlying manifold stitched together with the vector spaces that are tangent to the manifold at each point.<\/p><p>To illustrate this, consider the unit circle \\(\\mathcal{S}^1\\) below. The blue circle represents the underlying manifold and the red lines represent the one dimensional tangent space at each point on the circle. Together, the points and tangent spaces form the tangent bundle. It is important to note that that vectors in one tangent space should be regarded as distinct and separate from vectors in another tangent space.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/D5bhd7uqiDV9RPn7MQaGuPiR9U.png\"><p><em>PC: <\/em><a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Tangent_bundle.svg\"><em>https:\/\/commons.wikimedia.org\/wiki\/File:Tangent_bundle.svg<\/em><\/a><\/p><p>Now that we have the correct picture in our heads, let\u2019s visualize tangent vectors and the tangent bundle in more detail.<\/p><p>Consider an arbitrary manifold \\(M\\), and furthermore consider a curve on that manifold \\(\\gamma: \\mathbb{R} \\to M\\). This curve \\(\\gamma(t)\\) can be thought of \u201cthe location on the manifold at time \\(t\\).\u201d Furthermore, let \\(\\gamma(0)\\) be some point of interest \\(p \\in M\\) on the manifold.<\/p><p>For visualization, consider this wavy circle manifold:<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/DssFuIjaEGBc8P2bMtLOJHCymA.png\"><p>Now, consider the curve \\(\\gamma(t)\\) living on this manifold.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/nz31mp82xL8mJDw9qNabiwfgc.mp4\"><\/video><p>If we take the derivative of this curve, we obtain a pair \\((\\gamma(t), \\gamma'(t))\\), where the first entry describes the point along the curve and the second entry describes how the curve is changing in time. This can be visualized as a vector \u201criding along\u201d the curve.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/oGzVS7TeH4kOCLSdHEoEEuA93mo.mp4\"><\/video><p>The magnitude and sign of this vector can be changed by re-parametrizing the curve as<\/p><p>$$\\gamma_{\\alpha}(t) = \\gamma(\\alpha t)$$<\/p><p>Specifically, note that<\/p><p>$$\\gamma'_\\alpha(0) = \\alpha \\gamma'(0)$$<\/p><p>In this manner, we can \u201cscale\u201d curves in the same way that we would scale vectors. Similarly, we can derive addition, inverse and zero curves and form a \u201cvector space of curves.\u201d<\/p><p>Thus, we define the tangent space \\(T_pM\\) as the set of all curves that pass through a point of interest \\(p \\in M\\). The tangent bundle of a manifold \\(TM\\) is the set of all tangent spaces combined under a <em>disjoint<\/em> union. In this context, a disjoint union simply means we cannot combine vectors in distinct tangent spaces. This is perhaps the most important takeaway: tangent vectors living in distinct tangent spaces cannot be added, combined or related without using some additional manifold structure.<\/p><blockquote><p>\u26a0\ufe0f Here, we have <em>de facto<\/em> assumed that we can take the derivative of a curve on the manifold. This is made more rigorous in most differential geometry books by either using charts on the manifold or an extrinsic view of a manifold facilitated by an embedding. Furthermore, we didn\u2019t really show how to \u201cadd curves.\u201d We haven\u2019t done anything <em>improper<\/em> with this explanation, but it certainly is lacking detail! You can refer to the previously mentioned resources to learn more about the mathematically precise definitions of the Tangent Bundle.<\/p><\/blockquote><p>If this is your first time studying Differential Geometry, you may be very uncomfortable with the notion of a vector being <em>attached<\/em> in any way to a specific point. After all, vectors are commonly taught as being geometric entities that can be moved anywhere in space, e.g. vector addition is taught using the \u201chead to tail\u201d method. It\u2019s unclear when presenting these geometric explanations if the vectors are \u201csitting on top of\u201d the underlying manifold or if they are in their own \u201cvector space.\u201d<\/p><p>To see why this \u201chead to tail\u201d method breaks down on manifolds, imagine traveling 1000 miles north and then 1000 miles west. This will take you to a very different location than if you had traveled 1000 miles west and then 1000 miles north. In other words, vectors may point in different directions at different points on the manifold, so we can only really discuss what a vector means <em>locally<\/em>.<\/p><p>The representation we have presented here of \\((\\gamma(0), \\gamma'(0))\\) , is useful in describing the construction tangent vectors on a manifold and also useful in derivations of certain operators, but it is typically not used in computation. In \u201cthe real world,\u201d we typically choose some canonical coordinates for the manifold and a basis for the associated tangent space and think of tangent vectors as \\((p, v) \\in M \\times \\mathbb{R}^n\\).<\/p><p>With this understanding of tangent spaces, the exponential map can be understood more formally as an operator mapping vectors in a tangent space onto manifold.<\/p><p>$$<br>\\text{Exp}_p(v): T_pM \\to M<br>$$<\/p><p>Likewise, the logarithmic map can be understood as an operator mapping points on the manifold back into a tangent space<\/p><p>$$<br>\\text{Log}_p(q): M \\times M \\to T_pM<br>$$<\/p><blockquote><p>\ud83d\udea7 Ok, <em>technically<\/em>, more structure is needed for the existence of an exponential map. Since the exponential map minimizes distance along a path, we must have some notion of <em>length.<\/em> The subfield of Differential Geometry with <em>lengths<\/em> and <em>angles<\/em> is called <a href=\"https:\/\/en.wikipedia.org\/wiki\/Riemannian_geometry\">Riemannian Geometry<\/a> and is a fascinating subject in its own right. The definition of <em>lengths<\/em> and <em>angles<\/em> for a specific manifold is called the Riemannian Metric. We don\u2019t have the space in this post to develop these ideas fully but we encourage the motivated reader to learn more!<\/p><\/blockquote><p>With this understanding, we have the following prescription for optimization on a generic manifold \\(M\\)<\/p><ul><li data-preset-tag=\"p\"><p>Compute the quantity to minimize \u2014 perhaps using our logarithmic map \\(\\text{Log}_p(q)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Compute a direction \\(v\\) in \\(T_pM\\) along which the quantity is locally reduced<\/p><\/li><li data-preset-tag=\"p\"><p>Move the parameters using the exponential map \\(\\text{Exp}_p(v)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Repeat until the problem converges<\/p><\/li><\/ul><p>At this point, you are probably saying<\/p><blockquote><p>\u201cWhy isn\u2019t multi-variable optimization taught like this?\u201d<\/p><\/blockquote><blockquote><p>\u201cI haven\u2019t had to worry about <em>tangent spaces<\/em> and <em>exponential maps<\/em> before.\u201d<\/p><\/blockquote><blockquote><p>\u201cAdding vectors has always <em>just worked<\/em> for me in the past.\u201d<\/p><\/blockquote><p>Yes, BUT. The secret here is that you\u2019ve been exploiting a very special property of euclidean space: <em>continuous symmetry<\/em> of its tangent bundle.<\/p><h2>Continuous Symmetry<\/h2><p>Some smooth manifolds exhibit a natural symmetry. By symmetry, we mean a concept that implies more than reflections or rotations of polygons taught in grade school. For instance, reflections and rotations of polygons are examples of <em>discrete symmetry.<\/em> However, in the study of smooth manifolds, we are interested in <em>continuous<\/em> <em>symmetry<\/em> of the Tangent Bundle. This is what makes optimization so simple in euclidean space.<\/p><p>To illustrate, consider the smooth manifold of real numbers of dimension \\(n\\): \\(\\mathbb{R}^n\\) or Euclidean Space. We will visualize the plane \\(\\mathbb{R}^2\\), but realize that these properties apply in higher dimensions, too.<\/p><p>Let\u2019s place a uniform vector field on \\(\\mathbb{R}^n\\), where \u201cthe same\u201d vector \\(v\\) is attached to each point in the space \\(p\\). Imagine those points <em>flowing<\/em> along the vectors, as if the vectors describe the current of a river and the points are rafts in that river.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/YIX6oQlRgl5BjVzR6cpQ2f0nZE.mp4\"><\/video><p>You\u2019ll notice that the points before and after the flow are effectively the same! Since this flow doesn\u2019t <em>fundamentally<\/em> change the total set of points, it\u2019s called an <em>invariant flow.<\/em><\/p><p>Now let\u2019s flip it: instead of moving each point along the flow of individual vectors, let\u2019s recenter the points on a new origin.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/gXcZc9TdEzxmUHRXpsByrdAwoY.mp4\"><\/video><p>Notice that this recentering effectively produces the same transformation as the vector field above. This is one important property of continuous symmetry of the manifold: the equivalence of local transformations under <em>constant<\/em> vector fields and global transformations of the entire space.<\/p><p>We can also recenter both the vector field and its associated points. Doing so, we find that the points and vector field are fundamentally <em>unchanged.<\/em><\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/ljH8ZV5CVWlxTtYdvC7YAtixyJ4.mp4\"><\/video><p>Furthermore, the points and vector field don\u2019t have to be recentered in the same direction as the vector field. Under any arbitrary translation, the points and the <em>constant<\/em> vector field remain unchanged*.* In this way, both the manifold and the Tangent Bundle exhibit continuous symmetry.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/yT2XeaVDRmgycItSdrXcV9MHpWw.mp4\"><\/video><p>So to recap, when we say <em>continuous symmetry<\/em> of the tangent bundle of \\(\\mathbb{R}^n\\) we mean three things:<\/p><ul><li data-preset-tag=\"p\"><p>Local flows along an invariant vector field are the same as a global translation of all points<\/p><\/li><li data-preset-tag=\"p\"><p>Arbitrary translations of points yields the same set of points<\/p><\/li><li data-preset-tag=\"p\"><p>Arbitrary translations of <em>constant<\/em> vector fields yields the same <em>constant<\/em> vector field<\/p><\/li><\/ul><p>One consequence of this continuous symmetry is that we can effectively treat <em>any<\/em> point in \\(\\mathbb{R}^n\\) as the origin. To illustrate this, we will \u201csubtract\u201d two points \\(p\\) and \\(q\\) and get a vector pointing from \\(p\\) to \\(q\\), and then flow \\(p\\) along the vector to recover \\(q\\).<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/xMN5V0EKeGLJ1gIyTroRmknr5Y.mp4\"><\/video><p>If we follow this animation <em>carefully,<\/em> we have to remember that this red vector lives in \\(T_p\\mathbb{R}^n\\) and then <em>flow<\/em> the point starting at \\(p\\) until it reaches \\(q\\). Now, let\u2019s first recenter the origin at \\(p\\) and then perform the subtraction.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/dLppnverVhFjILyVe9CnQ8SLrg.mp4\"><\/video><p>We see that this produces the <em>same<\/em> result, and this time the vector was \u201cattached to the origin.\u201d In effect, we can do the subtraction <em>as if<\/em> the space was centered at \\(p\\) and then only worry about vectors living in \\(T_0 \\mathbb{R}^n\\).<\/p><p>You probably didn\u2019t realize it, but we\u2019ve set ourselves up for easy computation! If we use the standard basis of \\(\\mathbb{R}^n\\), we simply subtract the coordinates of each point element-wise to get the vector. If we want to translate a point by a vector, we again can simply perform element-wise addition of the coordinates. The annoying book-keeping that comes with this vector distribution is effectively non-existent, because we just treat all vectors <em>as if<\/em> they lived in the tangent space at the origin.<\/p><p>In fact, since \\(T_0\\mathbb{R}^n\\) is essentially just a copy of \\(\\mathbb{R}^n\\), we can confuse points and vectors and likely <em>get away with it<\/em>. The distinction between a point and the vector that points from the origin to a point is so subtle that in practice it\u2019s often not worth mentioning.<\/p><p>Using the continuous symmetry of \\(\\mathbb{R}^n\\) we formulate our optimization steps as<\/p><ul><li data-preset-tag=\"p\"><p>Compute the quantity to minimize <em>as if<\/em> our parameters represent the origin<\/p><\/li><li data-preset-tag=\"p\"><p>Compute a direction along which the quantity is locally reduced<\/p><\/li><li data-preset-tag=\"p\"><p>Add the minimizing direction to our parameters <em>as if<\/em> our parameters represent the origin<\/p><\/li><li data-preset-tag=\"p\"><p>Repeat until the problem converges<\/p><\/li><\/ul><p>With continuous symmetry in euclidean space on our side, we can use the \\(\\mathbb{R}^n\\) hammer on <em>everything<\/em>. Recalling our <a href=\"https:\/\/www.tangramvision.com\/blog\/camera-modeling-pinhole-obsession\">previous post<\/a>: consider \\(\\mathbb{T}^2\\). You\u2019ll notice now that it is essentially \\(\\mathbb{R}^2\\) with a couple of \u201cextra points at infinity.\u201d This makes computer vision with camera rays and the pinhole camera model feel easy! We just treat \\(\\mathbb{T}^2\\) as \\(\\mathbb{R}^2\\) and compute reprojection error by subtracting points and minimizing that error.<\/p><h2>Lie Groups: <em>Continuing<\/em> the Symmetry Party<\/h2><p>But, I hear you mutter to yourself,<\/p><blockquote><p>\u201cAre there manifolds that have continuous symmetry of their Tangent Bundle, but are not flat like \\(\\mathbb{R}^n\\)?\u201d<\/p><\/blockquote><p>Great question! The answer is yes! An example of manifolds that fit these criteria are <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lie_group\">Lie Groups<\/a>. A Lie Group (we\u2019ll call it \\(\\mathcal{G})\\) is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Group_(mathematics)\">mathematical group<\/a> that is also a smooth manifold.<\/p><p>Many commonly-used manifolds in robotics are Lie Groups:<\/p><ul><li data-preset-tag=\"p\"><p>Special Orthogonal Group \\(SO(3)\\) - used for describing rigid-body rotations<\/p><\/li><li data-preset-tag=\"p\"><p>Special Euclidean Group \\(SE(3)\\) - used for describing rigid-body motion<\/p><\/li><li data-preset-tag=\"p\"><p>Special Linear Group \\(SL(3)\\) - used for describing continuous-time homographies<\/p><\/li><li data-preset-tag=\"p\"><p>Similarity Group \\(Sim(3)\\) - used for describing similarity transforms (rigid-body + scale)<\/p><\/li><\/ul><p>Given their wide usage, it\u2019s worthwhile to explore what makes them so useful.<\/p><p>A Lie Group is a mathematical group; it\u2019s in the name. This means it has a way to compose elements, typically denoted as \\(g \\circ h\\) for group elements \\(g,h \\in \\mathcal{G}\\). Additionally, the group has an identity element (typically called \\(e\\)). In some fashion, this identity can be considered the \u201corigin\u201d. Since a Lie group is <em>also<\/em> a smooth manifold, it has a tangent space at this \u201corigin.\u201d The tangent space at the identity element is called the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lie_algebra\">Lie Algebra<\/a> and is denoted by \\(T_e \\mathcal{G} = \\mathfrak{g}.\\)<\/p><p>To help us develop some intuition of Lie Groups, we will consider the simplest non-euclidean Lie Group, the unit circle \\(\\mathcal{S}^1\\) and its associated Lie Algebra. Note that unlike our visualization of \\(\\mathbb{R}^n,\\) we have chosen to explicitly display a tangent space of unit circle, since it takes a different shape than the underlying manifold.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/2zn2ztK3Nsc68oHVArNCzzZejjA.png\"><p>Now, we proceed to choose some vector in this tangent space at the identity.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/nyNiroWtnweWStykG20oIEnIE.png\"><p>Now, let\u2019s try to move a point along our chosen vector.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/NOITlJ8rfeHrHLl7RkqPtvFT1WQ.mp4\"><\/video><p>If we simply move the point along the vector, the point leaves the manifold! This is not a valid flow, because the points must <em>by definition<\/em> live on the underlying manifold. To fix this, we will define the <em>flow<\/em> as the curve \\(\\varphi(t)\\) such that \\(\\varphi'(t)\\) matches the tangent vector. For a vector field \\(X\\) defined on the entire manifold, we denote this flow as \\(\\varphi_X(t)\\).<\/p><p>However, this definition makes solving for \\(\\varphi_X(t)\\) a bit tough since we have to solve an ordinary differential equation on a manifold. If we instead think of Euler integration, we can <em>approximate<\/em> a solution by \u201cmoving along\u201d the manifold in the direction of the vector field.<\/p><p>In \\(\\mathbb{R}^n\\), for a given vector field \\(X: \\mathbb{R}^n \\to T\\mathbb{R}^n\\) euler integration gives us the recurrence of<\/p><p>$$<br>p_k = p_{k-1} + X(p_{k-1}) \\Delta t<br>$$<\/p><p>On the manifold we can use the exponential map to similar effect. For a given vector field \\(X: M \\to TM\\) \u201ceuler integration\u201d takes the form of<\/p><p>$$p_k = \\text{Exp}_{p_{k-1}}(X(p_{k-1})\\Delta t)$$<\/p><p>We can imagine this exponential map as a \u201cpushed down\u201d vector on the manifold.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/iVv0DymASM7hFSxhbC2SR5NNko.mp4\"><\/video><blockquote><p>\u26a0\ufe0f In actuality, there is no such thing as a \u201cpushed down vector,\u201d however it\u2019s a useful concept for visualization. If we use the definition of the exponential map, we can imagine this \u201cpushed down vector\u201d as the line drawn by the equation \\(\\text{Exp}(\\alpha v)\\) for a fixed vector \\(v\\) and a scalar \\(\\alpha \\in [0,1]\\).<\/p><\/blockquote><p>Now following our example with \\(\\mathbb{R}^2\\), let\u2019s try to create a <em>constant<\/em> vector field by copying a vector into all points on the manifold. If we naively copy this vector in the same way that we did in the euclidean case (remember that big grid of arrows?), we get the following:<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/KPfJc8rEhqavhNtgyqtGLFq96z0.png\"><p>This cannot be correct; our vectors wouldn\u2019t live in their own respective tangent spaces. These \u201ctangent vectors\u201d have some component pointing normal to the manifold, which implies they are not tangent vectors at all! How do we address this mathematically?<\/p><blockquote><p>\u26a0\ufe0f Here we are claiming that this visualization is incorrect because we are considering the unit circle and its tangent bundle as entities <em>embedded<\/em> in ambient euclidean space \\(\\mathbb{R}^2\\). In reality, we only care about the geometric and algebraic structure of the unit circle and our visualization is <em>somewhat independent<\/em> of that structure. It it not uncommon to visualize a manifold and an associated fibre bundle (e.g. the tangent bundle) as living in orthogonal spaces but connected at a single point. This explains why you sometimes may see the tangent bundle visualized as a manifold with non-overlapping tangent spaces.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/I0nRJeNc9lyzWMo93Bu5C7HAWk.png\"><p><em>PC: <\/em><a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Tangent_bundle.svg\"><em>https:\/\/commons.wikimedia.org\/wiki\/File:Tangent_bundle.svg<\/em><\/a><\/p><p>In this post, we\u2019ll continue with the embedded and overlapping view to aid intuition, but don\u2019t be surprised if you see the tangent bundle drawn differently in other resources.<\/p><\/blockquote><p>Instead of naively copying the vector at identity to every point on the lie group, let\u2019s consider an arbitrary point \\(g \\in \\mathcal{G}\\). We can consider this point on the manifold as simply a point or, using the group structure, we can consider it as an operator since \\(g \\circ e = g\\). In other words, \\(g\\) can be regarded as an operator that takes the identity into \\(g\\). This \u201coperatorness\u201d of \\(g\\) is typically denoted as the map<\/p><p>$$<br>\\begin{aligned}<br>L_g: \\mathcal{G} &amp;\\to \\mathcal{G} \\\\<br>h &amp;\\mapsto g \\circ h<br>\\end{aligned}<br>$$<\/p><p>where the \\(L\\) denotes that we are composing with \\(g\\) on the left. You can form a similar map by composing on the right; we\u2019ll call it \\(R_g\\).<\/p><p>We can use this operator concept to \u201cpush\u201d vectors around on the manifold. Instead of \\(L_g\\) operating on a fixed value \\(h\\), we can imagine it operating on a curve \\(\\gamma(t)\\) with \\(\\gamma(0) = h\\).<\/p><p>Now we have a new <em>shifted<\/em> curve<\/p><p>$$<br>L_g(\\gamma(t)) = g \\circ \\gamma(t)<br>$$<\/p><p>We can now consider the derivative of this new <em>shifted vector<\/em> as a vector living in the tangent space \\(T_{g \\circ h} \\mathcal{G}\\):<\/p><p>$$<br>\\frac{d}{dt}L_g(\\gamma(t))\\vert_{t=0}<br>$$<\/p><p>\u2026just like we did in our original vector field in \\(\\mathbb{R}^n\\). For convenience of notation, we\u2019ll drop the explicit parametrization of the curves and denote this vector shift as \\(dL_g(h,v)\\) for some \\(v \\in T_h \\mathcal{G}.\\)<\/p><blockquote><p>\ud83d\udca1The argument made here to derive this <em>vector shift<\/em> function actually extends to arbitrary manifolds. If we have an map between manifolds<\/p><p>$$<br>\\begin{aligned}<br>f: M \\to N<br>\\end{aligned}<br>$$<\/p><p>that obeys the continuity properties of the manifold, we can create an induced map on the tangent bundle<\/p><p>$$<br>df: TM \\to TN<br>$$<\/p><p>This induced map is often referred to as the differential or \u201cpush forward\u201d map because it \u201cpushes\u201d vectors in tangent spaces of one manifold onto tangent spaces of another manifold.<\/p><\/blockquote><p>Now, we will state a few facts without proof about \\(dL_g(h,v)\\)<\/p><ol><li data-preset-tag=\"p\"><p>\\(dL_g(h,v)\\) is linear in \\(v\\)<\/p><\/li><li data-preset-tag=\"p\"><p>\\(dL_g(h, v)\\) actually does not depend on \\(h\\), so we can write it as \\(dL_g(v)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>\\(dL_g(dL_h(v)) = dL_{g \\circ h}(v)\\) or in other words group composition is compatible with vector shifting.<\/p><\/li><\/ol><p>Got all that? Good! Now that we have this structure in place, let\u2019s return to our tangent vector at the origin.<\/p><img alt=\"\" src=\"https:\/\/framerusercontent.com\/images\/nyNiroWtnweWStykG20oIEnIE.png\"><p>This time we will copy the vector to the other points on the manifold by <em>pushing<\/em> the vector with \\(dL_g\\) to each \\(g\\) on the manifold.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/s5NbbhRpZDVsUAdTgN8bAiblk0.mp4\"><\/video><p>If we flow all the points on the manifold along these vectors we\u2019ll notice that all the points change in a consistent way! In the case for \\(\\mathcal{S}^1\\) they are all rotating around the circle.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/RdwRRbbRvtb3BYwYzksDhJYXptA.mp4\"><\/video><p>Here, we\u2019ve found another invariant flow field. Let\u2019s determine if the vector field is <em>unchanged<\/em> under the combined transformation<\/p><p>$$<br>(h, v) \\mapsto (L_g(h), dL_g(v))<br>$$<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/7Rff03MKbRxxBVGZrbwZW8k4.mp4\"><\/video><p>Just like the case for \\(\\mathbb{R}^n\\), the points and vector field remain <em>unchanged<\/em>. Furthermore, like \\(\\mathbb{R}^n\\) the transformation does not have to be <em>in the same direction<\/em> as the vector field.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/QkkbwBX7Kb06Anu16E0T4RY8rc.mp4\"><\/video><p>Since these vectors don\u2019t all point in the same direction (at least from an extrinsic point of view), we cannot call this a <em>constant<\/em> vector field. However, the vector field is invariant to group transformations, so we will call it an <em>invariant<\/em> vector field.<\/p><p>With these observations, we can state that all Lie Groups have a <em>continuously symmetric<\/em> Tangent Bundle*.* The difference from the euclidean case is that points and (certain) vector fields are invariant under group <em>transformations<\/em> instead of <em>translations<\/em>.<\/p><blockquote><p>\ud83d\udca1Although this difference may seem significant at first, it turns that \\(\\mathbb{R}^n\\) is actually Lie Group with translation as the group operator! Would you look at that.<\/p><\/blockquote><p>As an added bonus, for these <em>invariant<\/em> vector fields, the Euler integration using the exponential map is <strong>exact.<\/strong> Specifically, for an <em>invariant<\/em> vector field the following closed-form integration holds:<\/p><p>$$<br>\\varphi_X(t) = \\text{Exp}_{p}(X(p)t)<br>$$<\/p><blockquote><p>\u26a0\ufe0f Here in this section, we\u2019ve implied a parallel between <em>geodesics<\/em> on manifolds with a Riemannian Metric and the exponential map on Lie Groups. Technically, we have not defined a notion of a Riemannian Metric for Lie Groups. In this manner, it is possible to define a Riemannian Metric on Lie Groups such that the <em>geodesic exponential<\/em> and the <em>group exponential<\/em> are distinct. Regardless, broadly speaking, in both cases the exponential map helps us move along the manifold in a coherent manner.<\/p><\/blockquote><p>As with Euclidean space, we can \u201crecenter\u201d the manifold by using the group operator to move \\(g\\) to the identity \\(e\\) while maintaining the distance to all other points. To do this, we simply apply the operator \\(L_{g^{-1}}\\) to all points on the manifold.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/QU3QkvBNsPacKvwdR71jLYRjbh4.mp4\"><\/video><p>Likewise, we can \u201csubtract\u201d any two points \\(h\\) and \\(g\\) on the manifold by shifting the point we are subtracting to the origin and then using our aforementioned logarithm \\(\\text{Log}_e(g^{-1} \\circ h)\\).<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/1JvYlae1VX8DrpHZtveOegzLFAI.mp4\"><\/video><p>Leveraging this <em>continuous symmetry<\/em> of Lie Groups, our optimization steps become<\/p><ul><li data-preset-tag=\"p\"><p>Compute the quantity to minimize <em>as if<\/em> our parameters represent the origin i.e. using \\(\\text{Log}_e(g^{-1} \\circ h)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Compute a direction along which the quantity is locally reduced<\/p><\/li><li data-preset-tag=\"p\"><p>Add the minimizing direction to our parameters <em>as if<\/em> our parameters represent the origin i.e. using \\(g \\circ \\text{Exp}_e(v)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Repeat until the problem converges<\/p><\/li><\/ul><p>Which makes for a pretty simple algorithm! We really only have to worry about two extra operators \\(\\text{Log}_e\\) and \\(\\text{Exp}_e\\). Maybe optimizing \u201con the manifold\u201d isn\u2019t so difficult after all? Let\u2019s apply this <em>continuous symmetry<\/em> concept to the unit sphere to finally cure our Pinhole Obsession.<\/p><h2>Symmetry of the Sphere<\/h2><p>Back to our unit sphere! The whole reason we did this in the first place.<\/p><p>Ideally, we\u2019d want the unit sphere to exhibit <em>continuous symmetry<\/em> of the Tangent Bundle so we can take advantage of its benefits:<\/p><ul><li data-preset-tag=\"p\"><p>Local flow along <em>invariant<\/em> vector fields is the same as a global transformation<\/p><\/li><li data-preset-tag=\"p\"><p>Some arbitrary concept of \u201corigin\u201d or \u201cidentity\u201d<\/p><\/li><li data-preset-tag=\"p\"><p>Invariance of points and certain vector fields under global transformations<\/p><\/li><\/ul><p>At first glance, the sphere <em>looks<\/em> fairly symmetric, so constructing an <em>invariant<\/em> vector field should be straightforward. Let\u2019s try to find such a vector field. To start, let\u2019s choose a vector field where all the vectors point at the north pole.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/CrheGsVqyuk58xYGKSGqRJKN4.mp4\"><\/video><p>Here we notice that the points on the manifold are generally not all moving in the \u201csame direction.\u201d The points close to the north pole are converging and the points close to the south pole are diverging. This is not an invariant flow field.<\/p><p>Let\u2019s again try to find an invariant flow field but this time choose the vector field where all the vectors are pointing east.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/8zDULSYAl8v8G18XB1zFZjDyRZY.mp4\"><\/video><p>Following the flow, the total set of points remains unchanged*.* This flow field yields a rotation of the sphere. So, the manifold itself exhibits some <em>continuous symmetry<\/em> because rotation leaves the set of points on the manifold unchanged.<\/p><blockquote><p>\u2b50 If you are wondering why these points flowing east are not moving along great circles, then you are a very astute reader! Remember, that in general using the exponential map \\(\\text{Exp}_p(v)\\) to solve the flow \\(\\varphi_X(t)\\) of a vector field \\(X\\) is an <em>approximation.<\/em> The true definition is that derivative of the flow \\(\\varphi_X'(t)\\) must match the vector field \\(X\\) at every point along the curve.<\/p><\/blockquote><p>Does this symmetry extend to the tangent bundle \\(T\\mathcal{S}^2\\)? Well, if we swap the north pole and south pole, the vector field is now pointing west! So, the vector field that produces rotations is <em>not<\/em> <em>invariant<\/em> under rotations.<\/p><video loop=\"\" playsinline=\"\" src=\"https:\/\/framerusercontent.com\/images\/K1OqgYxkiVg1MeoYFaFmldMxts.mp4\"><\/video><p>These visualizations, and the aircraft example from beginning of this post point to \\(\\mathcal{S}^2\\) not having <em>continuous symmetry<\/em> of its Tangent Bundle. This lack of Tangent Bundle symmetry can be proven rigorously by using the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hairy_ball_theorem\">Hairy Ball theorem<\/a>. Unfortunately, this means that we cannot take advantage of <strong>all<\/strong> the benefits of symmetry when optimizing over \\(\\mathcal{S}^2\\). Specifically:<\/p><ul><li data-preset-tag=\"p\"><p>There is no invariant vector field under a set of global transformations<\/p><\/li><li data-preset-tag=\"p\"><p>We can\u2019t <em>recenter<\/em> a point of interest to do tangent-space computations in a more convenient location<\/p><\/li><li data-preset-tag=\"p\"><p>Each tangent space must be regarded as distinct and not simply <em>shifted copies<\/em> of an origin tangent space<\/p><\/li><\/ul><p>Ultimately, optimizing over a generic asymmetric manifold is a large increase in complexity as opposed optimizing over Euclidean Space or Lie Groups. For each computation, we must perform careful book-keeping to ensure the coherence between:<\/p><ul><li data-preset-tag=\"p\"><p>Points on the manifold: \\(p \\in \\mathcal{S}^2\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Vectors in the tangent spaces at each point: \\(v \\in T\\mathcal{S}^2\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Exponential maps to \u201cmove along the manifold\u201d: \\(\\text{Exp}_p(v)\\)<\/p><\/li><li data-preset-tag=\"p\"><p>Logarithmic maps to compute the difference between points: \\(\\text{Log}_p(q)\\)<\/p><\/li><\/ul><p>If this coherence is not maintained and properly modeled, the optimization is likely to fail. After really digging into the full complexity of optimization over a generic asymmetric manifold, Lie Groups and Euclidean Space seem even more pleasant to work with.<\/p><h2>\u201cGrouping\u201d it All Up<\/h2><p>As we can see, optimization over Lie Groups (including Euclidean space) is actually a special case of optimization over a manifold. With <em>continuous symmetry,<\/em> we can actually ignore the effects of the starting point on the exponential operator*. Continuous symmetry* allows us to recenter the manifold on our starting point \\(g\\) (the Lie algebra) and <em>pretend<\/em> that we started at the origin. This is <em>incredibly handy<\/em> and in robotics we use this property a lot!<\/p><p>You\u2019ll notice that in the optimization steps For Lie Groups, we only used the exponential and logarithmic maps at one point: the identity \\(e\\). From a mathematical perspective, this is <strong>the<\/strong> defining characteristic of Lie Groups. This defining characteristic can be described as an isomorphism between<\/p><ul><li data-preset-tag=\"p\"><p>The group itself \\(\\mathcal{G}\\)<\/p><\/li><li data-preset-tag=\"p\"><p>The Lie Algebra \\(\\mathfrak{g} = T_e\\mathcal{G}\\)<\/p><\/li><li data-preset-tag=\"p\"><p>The invariant vector fields on \\(\\mathcal{G}\\)<\/p><\/li><\/ul><p>There is a number of deep mathematical connections that can be made here, and we encourage the interested reader to learn more. However, for our use-case, it means we can rewrite our exponential map as<\/p><p>$$<br>\\text{Exp}_g(v) = g \\circ \\text{Exp}_e(v)<br>$$<\/p><p>If you\u2019ve read the previously mentioned \u201cMicro Lie Theory\u201d paper, you\u2019ll recognize this as the \u201ccircle plus\u201d operator i.e. \\(g \\oplus v\\). With Lie Groups, there <strong>really<\/strong> is only one exponential map of concern \u2014 the one defined at the identity. This does not hold true for general manifolds. This is why when talking about Lie Groups <strong>the<\/strong> exponential \\(\\text{Exp}\\) map will be described instead of mentioning exponential maps at each point.<\/p><p>So, when using Lie Groups for optimization, we can ignore the specifics about \u201cwhich tangent space\u201d we\u2019re in. We operate either on the group \\(\\mathcal{G}\\), or in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lie_algebra\">Lie Algebra<\/a> \\(\\mathfrak{g}\\). To recall our tools analogy, In the land of Lie Groups, we have two tools to master: a hammer \\(\\mathcal{G}\\) and\u2026 screwdriver \\(\\mathfrak{g}\\).<\/p><p>Did you have fun? We sure did. Thanks for going on this deep dive into Differential Geometry and its implications for optimization problems in robotics. Everyone loves Lie groups, no doubt about it.<\/p><p>Of course, if you\u2019re experiencing these concerns in your day-to-day development, it\u2019s probably time you considered working with the Tangram team. We\u2019ve solved these problems so many different ways, on twice as many platforms. It\u2019s our specialty! All this cool math in an easy-to-use package that can shorten your time to market; what\u2019s not to love?<\/p><p>\u200d<\/p>","url":"https:\/\/www.tangramvision.com\/the-deceptively-asymmetric-unit-sphere","title":"Camera Modeling, Part 2: The Deceptively Asymmetric Unit Sphere","summary":"The deficiencies of the Unit Sphere and an exploration of Lie groups.","date_modified":"2024-11-21T00:00:00.000Z","author":{"name":"[\"devon-morris\"]"},"tags":["calibration-mathematics"]}]}