Advances in medical field have introduced new and progressive ways to intensify surgical resident training and surgical skills learning by developing systematic simulator training programs alongside traditional training. Both training methods need constant presence of a competent surgeon to subjectively assess the surgical dexterity of the trainee. Several studies have been done to measure user's skill objectively and quantitatively, but all use sensors which could interfere with skill execution. Also the sterilization process in an actual surgery makes the use of sensors impossible. This paper proposes a novel video-based approach for observing surgeon's hand and surgical tool movements in both surgical operation and training. Data is captured by video camera and then explored using computer vision algorithm. Finally by analyzing basic statistical parameters, observer-independent model has been developed through objective and quantitative measurement of surgical skills.