The Truffle Pig's Guide to Source Discovery in OSINT

Open Source Intelligence is about turning data and information from public sources into intelligence that gives you actionable answers to your research questions. And while people new to the field of often need to be reminded that OSINT is not just data collection, without sufficient data, your research questions will be answered wrongly or not at all. OSINT As I said: “data and information for OSINT come from public SOURCES”, this statement obviously leads to a couple of questions: What are these sources? Where do I find sources? How can I find new sources? What are the right sources for my investigation? Let’s start with answering the first one: Existing collections of sources Luckily the OSINT community is amazing and there are so many people out there sharing OSINT sources. So probably you will find the sources you need for your investigation in the following list of links: https://github.com/jivoi/awesome-osint https://github.com/sinwindie/OSINT https://github.com/lockfale/OSINT-Framework http://osintframework.com https://start.me/p/DPYPMz/the-ultimate-osint-collection https://inteltechniques.com/tools/index.html https://github.com/cipher387/osint_stuff_tool_collection And the list could go on and on and on. If you think I should add some more, just shoot me a DM on Twitter @secbyaccident and I will add them. If you want to search for more of these repositories, good places to look are GitHub and start.me. Pivoting from existing information Pivoting is essential during every step of OSINT (and probably in the entire field of cybersecurity). Therefore, it should also be applied when searching for sources. For instance, if you know that a account is related to a group you're investigating, their tweets, followers, hashtags, and links could lead to other, less-known sources. Look for patterns or recurring themes, and follow those trails. A website they often link to could lead to another platform they use for communication. Likewise, a hashtag could lead you to other individuals or groups related to your investigation. But Pivoting can also help you discover new sources. Twitter Discovering new sources When conventional methods don't yield enough results, this is where creative problem-solving and critical thinking come into play. With the information you have in hand, ask yourself questions like: "Could my target be active on niche forums related to their hobbies or interests?”, “Are there certain certificates or licenses they possess which might be listed in a specific database?" For example, if your subject is interested in model trains, perhaps they frequent forums such as modeltrainforum.com, sharing insights or asking questions. Similarly, if you know that an individual has an airmen license, you could potentially access more information from the FAA's database https://amsrvs.registry.faa.gov/airmeninquiry/. Remember, people often feel more secure in niche forums and may reveal information they wouldn't elsewhere. In conclusion, when it feels like you've exhausted all your sources, take a step back and go through this thought process. It might help you discover new sources you can add to your OSINT repository and possibly share with the community. In fact, you might discover a whole bunch of new information that restarts the iterative cycle of information gathering. Selecting the right source By now you have plenty of sources you can use in your investigation and the problem is not how you get enough data, but rather getting the right data. And where does the right data come from? From the right sources. But how do we find them? During the initial stage of your information-gathering process, it is generally a good idea to look at as many sources as possible, to see what information they can give you. But most of the time you do not do OSINT like a Pokemon trainer, it is almost never “gotta catch ’em all”, but more “gotta catch the ones that answer the specific question that led me to start my OSINT process” (not great for a theme song, I know). Since my background is in Offensive Security, let’s look at a research question from this field. Imagine you are tasked to craft a phishing email for Chad the Domain Administrator. The investigation could look like this: We know that Chad uses the username ChAdmin. This tells which social media profiles he owns and that he has a great sense of humor. We find ChAdmin on Twitter. He is very active there, but sadly he interacts a lot with members of an extremist group. So we can collect the hashtags he uses, map out the people he frequently interacts with, and do some graph-based analysis to see, if there is one central account, they all interact with. I remember I saw a great tool on GitHub to do this. …WAIT… This does not help my research question at all. Oh, he has an Instagram as well. Look at that, his cute cousin and him playing with model trains. And here he is at a model train convention. Oh, look ChAdmin is also on modeltrainforum.com and he seems to be frequently shopping at themodeltrainshop.co.uk. And his company frequently sends out discount offers from companies they partner with. So now we send out a look-alike email with an offer for his favorite shop aaaaaand we are in. As you can see critical thinking is important while doing OSINT and what really separates advanced practitioners from beginners, is not only having all the amazing tools and sources but knowing when to use which one. (If you are a doctor, please do not remove an appendix with a lawn mower, even tho it might be a new and shiny one) Conclusion Nowadays the amazing OSINT community has mapped out a big part of the internet and has stored all the interesting stuff in nice collections. This presents us with the new problem of selecting the sources valuable to our investigation. And from time to time the right source, might not be known to the OSINT community yet and we need to have the skills to find it on our own. I hope this article has given you some ideas on how to do this. Also published . here If you enjoyed this article, you can follow me on Twitter @secbyaccident, I will post stuff about OSINT and cybersecurity.