Efforts to improve early identification of aggressive high grade breast cancers, which pose the greatest risk to patient health if not detected early, are hindered by the rarity of these events. To address this problem, we proposed an accurate and efficient deep transfer learning method to handle the imbalanced data problem that is prominent in breast cancer data. In contrast to existing approaches based primarily on large image databases, we focused on structured data, which has not been commonly used for deep transfer learning. We used a number of publicly available breast cancer data sets to generate a 'pre-trained' model and transfer learned concepts to predict high grade malignant tumors in patients diagnosed with breast cancer at Mayo Clinic. We compared our results with state-of-the-art techniques for addressing the problem of imbalanced learning and confirmed the superiority of the proposed method. To further demonstrate the ability of the proposed method to handle different degrees of class imbalance, a series of experiments were performed on publicly available breast cancer data under simulated class imbalanced settings. Based on the experimental results, we concluded that the proposed deep transfer learning on structured data can be used as an efficient method to handle imbalanced class problems in clinical research.