Throughout the past decade, the popularity of Darknet usage has been skyrocketing secondary to the public’s concern regarding their privacy and anonymity on the internet. Most research studies delving into the structural features of darknet communities during the past few years, concluded that most activities taking place in various darknet communities are more or less illegal. However, more recently, many have started to believe that this is not entirely true.
Researchers examining darknet communities utilize information derived from crawling by means of the Breadth First Search (BFS) crawling algorithm. Even though crawling represents an ideal method to study the semantic and topological structure of various darknet communities, the specific algorithm used to crawl the darknets greatly determines how accurate the knowledge obtained from these parts of the deep web is.
Aiming at evaluating the results of research studies that examined the structure of current darknet communities, a recently published thesis analyzed how various crawling algorithms analyze darknet communities and how this influences the information we know about them. The thesis illustrated the limitations of the BFS; the main crawling algorithm used for examining darknet communities, and presented a detailed view of the network via utilizing two additional crawling algorithms to BFS; Random First Search (RFS) and Depth First Search (DFS). As such, the topological structure of darknet communities was obtained via implementation of three crawling algorithms; DFS, BFS and RFS, while the semantic structure of these communities was examined via a special probabilistic algorithm known as “Latent Dirichlet Allocation” (LDA).
Interesting Results that Can Change How the World Views Darknet Communities:
The thesis included a comparative analysis of the crawling algorithms that are used when studying the darknet. The analysis concluded that any future studies have to consider using a combination of multiple crawling algorithms such as DFS, BFS and RFS to boost the accuracy of their results. Via utilization of topic models, the thesis concluded that BFS provides a relatively smaller topic variety, while DFS presents a broader topic variety, due to the difference between the method of functionality of each algorithm.
The thesis illustrated how the structural view of various darknet communities can vary depending on the starting point used to conduct the crawling procedures. As such, to yield more accurate views of the structure of various darknets, researchers should analyze the network via using multiple starting points for crawling and then combining the results of all successful crawls for further analysis.
The thesis also proposed a method to obtain an accurate view of the structural features of the darknet communities. Information regarding topic distribution per webpage was utilized as a metric to achieve this goal. Fifty randomly chosen starting points were used to conduct crawling for every 300 links and the average of topics distribution per each used crawling algorithm was observed. As such, the analysis concluded that RFS exhibits a relatively stable convergence to the network’s real value, when compared to BFS and DFS. DFS produced higher values for topics distribution averages, while BFS produced lower values. According to the crawling algorithms’ behavior, it was proven that for DFS to yield accurate results, 82% of darknet webpages have to be crawled, while 100% of darknet webpages have to be crawled by BFS to obtain accurate, meaningful results. Consequently, we can clearly state that any results, regarding the structural view of darknet communities, derived from research studies that ignored these findings are mostly inaccurate.
Despite the fact that solid conclusions regarding the structure of today’s darknet communities via means of the results of this thesis cannot be obtained, it represents a roadmap that can guide future studies and totally change how the world views the darknet and the deep web in general. The study opens the door to a myriad of research topics that can formulate a better understanding of various darknet communities. So, if most previous research studies have shown that most activities on darknets are illegal, based on the results of this thesis, this has been proven to be an inaccurate conclusion by analysis of the crawling approaches used to draw this conclusion.