Abstract: Physical sensing and control often involves switching in the governing equations. For instance, skid-steered vehicles must violate nonholonomic constraints in order to maneuver. This sliding of the wheel against the ground causes the vehicle to behave discontinuously during a maneuver as well as making the vehicle’s state difficult to estimate. Estimation of contact state is most naturally framed in the context of switched systems, where the vehicle’s ground interaction is modeled by partitioning the system dynamics into distinct modes of behavior. Thus, as the vehicle maneuvers, the system evolves over some mode sequence, transitioning between modes over some set of switching times. Similarly, visual and tactile sensing is typically described in terms of features–edges, corners, and other abstractions of continuous phenomena as sensed through a physical, dynamic system. A hybrid form of the maximum principle can be used to pose the estimation process as a hybrid optimization over the space of all possible modal behaviors. Somewhat surprisingly, this combinatoric estimation problem can be represented as a projection of an infinite-dimensional optimization problem that can be approximated by relaxing the projection. Using this approach, both first-order and second-order optimization techniques can be employed, even in the presence of significant noise. Moreover, because of the quadratic convergence associated with second-order methods, one can implement these techniques in real-time settings.