Machine learning techniques are applied to Lagrangian trajectory prediction, which is important in oceanography for providing guidance to search and rescue efforts, forecasting the spread of harmful algal blooms, and tracking pollutants and marine debris. This study evaluates the ability of two types of neural networks for learning ocean trajectories from nearly 250 surface drifters released during the Grand Lagrangian Deployment in the Gulf of Mexico from Jul-Oct 2012. First, simple fully connected neural networks were trained to predict an individual drifter’s trajectory over 24 h and 5 d time windows using only that drifter’s previous velocity time series. These networks, despite having successfully learned modeled trajectories in a previous study, failed to outperform common autoregressive models in any of the tests conducted. This was true even when drifters were pre-sorted into geospatial groups based on past trajectories and different networks were trained on each group to reduce the variability that each network had to learn. In contrast, a more sophisticated social spatio-temporal graph convolutional neural network (SST-GCNN), originally developed for learning pedestrian trajectories, demonstrated greater potential due to two important features: learning spatial and temporal patterns simultaneously, and sharing information between similarly-behaving drifters to facilitate the prediction of any particular drifter. Position forecast errors averaged around 60km at day 5, roughly 20km lower than autoregression, and even better for certain subsets of drifters. The passage of Tropical Cyclone Isaac over the drifter array as a tropical storm and category 1 hurricane provided a unique opportunity to also explore whether these models would benefit from adding wind as a predictor when making short 24 h forecasts. The SST-GCNNs were found to not benefit from wind on average, though certain subsets of drifters (based on deployment) exhibited slightly lower forecast errors at hour 24 with the addition of wind.