Docking simulation is often performed for the activity prediction instead of ligand-based methods based on machine learning approaches, when we have no known drug information about a target protein. Because it calculates the binding energy by virtually docking a drug candidate compound with a binding pocket of a target protein, and does not require any other experimental information. However, the conformation search of a compound and evaluation of binding energy in a docking simulation are computationally heavy tasks, and thus it requires huge computation resources. Therefore, a machine learning-based method to predict the activity of a drug candidate compound against a novel target protein is highly required. Recently, Tsubaki et al. proposed an end-to-end learning method to predict the activity of compounds for novel target proteins. However, the prediction accuracy was insufficient because they used only amino acid sequence information for introducing protein information to a network. In this research, we proposed an end-to-end learning based compound activity prediction using binding pocket information of a target protein, which is more directly important to the activity. The proposed method predicts the activity by end-to-end learning using graph neural network for both compound structure and protein binding pocket structure. As a result of experiments on MUV dataset, the proposed method showed higher accuracy than existing method using only amino acid sequence information. In addition, proposed method achieved equivalent accuracy to docking simulation using AutoDock Vina with much shorter computing time.