Censys enables fast searching of actionable internet data

How does one secure the Internet? With actionable, real-time data, which is now available through the use of Censys, a tool developed by researchers at the University of Michigan and the University of Illinois at Urbana-Champaign. Censys is a search engine that enables researchers to ask questions about the hosts and networks that compose the Internet and get an immediate reply.

Since its widespread adoption beginning in the late 1980s and 1990s, the Internet has grown and matured explosively. That growth has resulted today in a global community of over three billion Internet users and about one billion public websites, a vast and unruly landscape that has enabled innovation but which has experienced its share of security vulnerabilities and exploits. The enormous and roiling nature of this environment has presented challenges for researchers seeking to monitor specific aspects of the health and status of the world’s most complicated network.

The introduction of ZMap by Michigan researchers in 2013 represented a breakthrough in terms of the ability to scan and collect data from the entire Internet. Previous techniques had required a month’s time and/or the use of high-powered clusters of computers; ZMap changed the equation for Internet scanning by enabling a scan of the entire IPv4 space in minutes while running on a single computer.

The fast Internet-wide scanning made possible by ZMap opened new avenues for security research, ranging from uncovering widespread vulnerabilities in random number generators to tracking the evolving impact of Heartbleed. However, significant effort was still required to query the collected data; even simple questions such as “What models of embedded devices prefer CBC ciphers?” required developing an application scanner, manually identifying and tagging devices, negotiating with network administrators, and responding to abuse complaints.

Censys builds on ZMap by providing a search engine for ZMap data. Every day, Censys is updated with a fresh set of data collected after ZMap pings about four billion IP addresses allocated to devices connected to the Internet. Researchers can then interact with this data through Censys’ search interface, report builder, and SQL engine.

“We’re trying to maintain a complete, searchable database of everything on the Internet,” says Zakir Durumeric, the Michigan graduate student researcher who leads the open-source project.

Censys supports full-text searches on protocol banners and querying of a wide range of derived fields. It can identify specific vulnerable devices and networks and generate statistical reports on broad usage patterns and trends. Censys returns these results in sub-second time, dramatically reducing the effort of understanding the hosts that comprise the Internet. Searching on Censys can reveal how widespread a flaw is, what devices suffer from it, who they are operated by, and even their approximate location.

Censys was developed by Durumeric, graduate student David Adrian, fourth-year undergraduate Ariana Mirian, and Prof. J. Alex Halderman, all of Michigan, along with Prof. Michael Bailey of UIUC. Their research paper on Censys, entitled “A Search Engine Backed by Internet-Wide Scanning,” appeared at the 22nd ACM Conference on Computer and Communications Security (CCS) in October 2015. The paper contains a full description of Censys’s architecture and several use cases.