Researches have been conducted to overcome Distributed Denial of Service (DDoS) flooding attack. Beside the use of signature based detection, anomaly based detection is also used to detect the attack. Several methods such as statistic, information theory, data mining and forecasting have been proposed. In several researches, they just focused to detect the traffic anomaly, but not to recognize the types of anomaly that were detected such as flashcrowd, types of botnet, types of DDoS, and prevention action. In this paper we categorize anomaly traffic detection system based on process and capability focus. Anomaly detection system process including traffic features, preprocessing, and detection process. Capability focus based on each main research problem to be solved, there are detectingonly anomaly, types of anomaly, and prevention system that include process to overcome the attack. At the end of paper, we provide overview of research direction and opportunities that may be done in future research.