Ocean scientists have dreamed and recently started to realize an ocean observing revolution with autonomous observing platforms and sensors. Critical questions to be answered by such autonomous systems are where, when, and what to sample for optimal information, and how to optimally reach the sampling locations. Definitions, concepts, and progress towards answering these questions using quantitative predictions and fundamental principles are presented. Results in reachability and path planning, adaptive sampling, machine learning, and teaming machines with scientists are overviewed. The integrated use of differential equations and theory from varied disciplines is emphasized. The results provide an inference engine and knowledge base for expert autonomous observing systems. They are showcased using a set of recent at-sea campaigns and realistic simulations. Real-time experiments with identical AUVs in the Buzzards Bay and Vineyard Sound region first show that our predicted time-optimal paths were faster than shortest distance paths. Deterministic and probabilistic reachability and path forecasts issued and validated for gliders and floats in the northern Arabian Sea are then presented. Novel Bayesian adaptive sampling for hypothesis testing and optimal learning are finally shown to forecast the observations most informative to estimate the accuracy of model formulations, the values of ecosystem parameters and dynamic fields, and the presence of Lagrangian Coherent Structures.