In this paper, we propose a trajectory prediction method that takes into account pedestrian behavior. To realize safe automated driving in urban areas, it is necessary to predict the future movements of road users. Pedestrians entering the vehicle’s direction are a target of interest, but they are difficult to predict because they change their behavior through significant interactions with the vehicle. In this study, we first predict whether pedestrians will yield the way to a vehicle. Next, the predictions are then used to predict the future trajectories of all road users in the scene. The proposed model consists of two neural network structures: the Yielding judgment module to predict pedestrian behavior and the Trajectory prediction module. The yielding judgment module provides intuitive and easily understandable indicators, which also helps to increase the interpretability of the overall model. We evaluated the usefulness of the proposed model using a publicly available dataset. The proposed method was found to reduce the average displacement error by 2.79% and the final displacement error by 2.70% compared to the case where the target pedestrian’s behavior is not considered.