Ulcerative Colitis (UC) is a chronic inflammatory disease characterized by periods of relapses and remissions affecting more than 500,000 people in the United States. The therapeutic goals of UC are to first induce and then maintain disease remission. However, it is very difficult to evaluate the severity of UC objectively because of non-uniform nature of symptoms and large variations in their patterns. To address this, we already developed an approach using the image textures in our previous work. But, we found that it could not handle larger number of variations in their patterns. In this paper, we propose a different approach using CNN (Convolutional Neural Network) to measure and classify objectively the severity of UC presented in optical colonoscopy video frames. We call the proposed approach using CNN as Ulcerative Colitis Severity CNN (UCS-CNN) which utilizes endoscopic domain knowledge and convolutional neural network to classify different UC severity of colonoscopy images. The experimental results show that the proposed UCS-CNN can evaluate the severity of UC reasonably.