Installing and Testing the Critical Stack Intel Client Richard Bejtlich January 21, 2015 01:14 Follow To help users validate that the Critical Stack Intel Client is working as expected, I wrote a document showing how to identify traffic from Tor exit nodes. Check out the article here. Related articles Welcome to the Critical Stack Intel Client Install .deb: Debian & Ubuntu Full Documentation (all the things) Intel Client After Updating Bro on Security Onion Adding your API key Comments 27 comments Sort by Date Votes orhiee February 20, 2017 21:33 hello, i did a install on a security onion with apt-get and dont really have any of the issues you guys are having but i have another issue, cant get all the feeds :( i am having errors as follows: critical-stack 21:23:13 [DEBUG] %!(EXTRA string=%s - %s, string=ET: Botnet Command and Control, *errors.errorString=Feed missing: critical-stack-intel-59-ET--Botnet-Command-and-Control.bro.dat, string=)critical-stack 21:23:13 [DEBUG] %!(EXTRA string=%s - %s, string=OpenPhish.com (Verified), *errors.errorString=Feed missing: critical-stack-intel-58-OpenPhish.com-(Verified).bro.dat, string=)critical-stack 21:23:14 [DEBUG] %!(EXTRA string=%s - %s, string=Ponmocup: Botnet IPs, *errors.errorString=Feed missing: critical-stack-intel-32-Ponmocup--Botnet-IPs.bro.dat, string=)critical-stack 21:23:15 [DEBUG] %!(EXTRA string=%s - %s, string=Ponmocup: Malware IPs, *errors.errorString=Feed missing: critical-stack-intel-31-Ponmocup--Malware-IPs.bro.dat, string=)critical-stack 21:23:15 [DEBUG] %!(EXTRA string=%s - %s, string=Ponmocup: Botnet Domains, *errors.errorString=Feed missing: critical-stack-intel-30-Ponmocup--Botnet-Domains.bro.dat, string=) created a different sensor with 3 of the non working feeds (thought it might be not working due to size) but still cant get them did a re-install from the web page - no go any ideas ? thanks 1 Permalink Shaun Baker March 29, 2018 06:23 Did critical stack drop ARM support? The wget points to their client page that now only carries deb and rpm packages. The repo doesn't seem to work either. 1 Permalink Bo Nilsson December 13, 2016 08:48 Hi I am building a bro IDS on raspbian. I am running: critical-stack-intel api < MY KEY > and gets this in response: critical-stack 08:12:45 [ERROR] Error: operation not supportedcritical-stack 08:12:45 [INFO] Root privileges are required to run critical-stack-intel.critical-stack 08:12:45 [INFO] Permissions are dropped when not needed for security. I installed and got this: root@intrution:/home/pi# dpkg -i critical-stack-intel-arm.deb(Reading database ... 38356 files and directories currently installed.)Preparing to unpack critical-stack-intel-arm.deb ...Unpacking critical-stack-intel (0.5.3) over (0.5.3) ...Setting up critical-stack-intel (0.5.3) ...Starting Critical Stack Intel ...Critical Stack Intel started.Add your API key: sudo critical-stack-intel api <key-here>Processing triggers for systemd (215-17+deb8u5) ... I am bewildered as to why? As you can see I have even been using the root account to install critical-stack after installing bro from source using sudo. Hope full for assistance, bo 1 Permalink Info-france December 13, 2016 21:41 @Bo Nilson: As you can see after a "cat /etc/passwd" critical-stack is in /bin/false. from root => su critical-stack -s /bin/bash :) i discovered this today ^^ ! 0 Permalink Corey January 26, 2015 21:42 Thanks Richard. I'll submit a ticket now. 0 Permalink Matt December 20, 2016 17:51 @Wes @Info-france Hi, I'm also working on the same setup as you both. I've got Critical Stack integrated with Bro but the issue I'm finding is that not all of the feeds I've subscribed to are downloaded when using 'critical-stack-intel pull'. At the moment only 2 out of 5 test feeds that I've subscribed to are being downloaded to my Pi - are either of you running into the same issue? Cheers, Matt 0 Permalink MrPsychoinc4587 November 28, 2016 21:58 sudo dpkg -i critical-stack-intel-arm.deb dpkg: error processing archive crtitical-stack-intel-arm.deb (--install): cannot access archive: No such file or directory Errors were encountered while processing: critical-stack-intel-arm.deb *Please help* 0 Permalink Ricardo Almeida June 20, 2017 18:47 Hi, I am having the same problem as @orhiee, but with some other feeds. Output example: critical-stack 15:37:40 [DEBUG] %!(EXTRA string=%s - %s, string=hosts-file.net Phishing Domains, *errors.errorString=Feed missing: critical-stack-intel-82-hosts-file.net-Phishing-Domains.bro.dat, string=)critical-stack 15:37:40 [DEBUG] %!(EXTRA string=%s - %s, string=ET: Botnet Command and Control, *errors.errorString=Feed missing: critical-stack-intel-59-ET--Botnet-Command-and-Control.bro.dat, string=)critical-stack 15:37:41 [DEBUG] %!(EXTRA string=%s - %s, string=OpenPhish.com (Verified), *errors.errorString=Feed missing: critical-stack-intel-58-OpenPhish.com-(Verified).bro.dat, string=)critical-stack 15:37:42 [DEBUG] %!(EXTRA string=%s - %s, string=ET: Known Compromised Hosts, *errors.errorString=Feed missing: critical-stack-intel-25-ET--Known-Compromised-Hosts.bro.dat, string=)critical-stack 15:37:44 [DEBUG] Downloading file: Filename: critical-stack-intel-23-Malware-Domains.bro.dat Checksum: 5435c6da6269766ba010d69d6cf6599c critical-stack 15:37:45 [DEBUG] %!(EXTRA string=%s - %s, string=PhishTank Intel Feed (Verified), *errors.errorString=Feed missing: critical-stack-intel-18-PhishTank-Intel-Feed-(Verified).bro.dat, string=) In this example only the Malware-Domains worked properly. Any ideas? 0 Permalink Info-france December 13, 2016 21:43 by the way ... i currently working on the same project ... Bro + Critical-Stack-Intel + Elasticsearch / logstash / kibana :) 0 Permalink Richard Compton March 14, 2015 04:40 Thank you for the reply! 0 Permalink Karl Hart August 24, 2017 13:17 Also having the same issue updating some of the feeds. Seems that most of the feeds are missing. Anyone know of a work-around to be able to get the feeds? 0 Permalink Paul Owen April 30, 2018 19:37 That's it! I did wonder why the intel.log stopped being created a couple of weeks ago. After re-installing the client a few times and fiddling about with configs, deleting the master-public.bro.dat and re-creating it a few times I realised - Raspberry Pi 3 is no longer supported! That's sad because as they get more powerful with lower power draw, they seem ideal NSM sensors but alas - it seem its not to be! Here's the errors I'm getting: critical-stack 20:25:21 [INFO] Pulling feed list from the Intel Marketplace.critical-stack 20:25:22 [INFO] Downloading feed information. Run with the `--debug` flag for more information.critical-stack 20:25:23 [DEBUG] %!(EXTRA string=%s - %s, string=DShield Domain List (Low Sev), *errors.errorString=Feed missing: critical-stack-intel-20-DShield-Domain-List-(Low-Sev).bro.dat, string=)critical-stack 20:25:23 [DEBUG] %!(EXTRA string=%s - %s, string=DShield Domain List (Medium Sev), *errors.errorString=gzip: invalid header, string=)critical-stack 20:25:24 [DEBUG] %!(EXTRA string=%s - %s, string=IP Bad Reputation (Scan), *errors.errorString=Feed missing: critical-stack-intel-29-IP-Bad-Reputation-(Scan).bro.dat, string=)critical-stack 20:25:24 [DEBUG] %!(EXTRA string=%s - %s, string=ET: Botnet Command and Control, *errors.errorString=unexpected EOF, string=)critical-stack 20:25:25 [DEBUG] %!(EXTRA string=%s - %s, string=joxeankoret.com Malware URLs, *errors.errorString=Feed missing: critical-stack-intel-65-joxeankoret.com-Malware-URLs.bro.dat, string=)critical-stack 20:25:26 [DEBUG] %!(EXTRA string=%s - %s, string=danger.rulez.sk SSH Brute Force Report, *errors.errorString=Feed missing: critical-stack-intel-72-danger.rulez.sk-SSH-Brute-Force-Report.bro.dat, string=)critical-stack 20:25:26 [DEBUG] %!(EXTRA string=%s - %s, string=hosts-file.net Ad/Tracking Domains, *errors.errorString=unexpected EOF, string=)critical-stack 20:25:27 [DEBUG] %!(EXTRA string=%s - %s, string=sysctl.org Domain Blocklist (Ads), *errors.errorString=Feed missing: critical-stack-intel-86-sysctl.org-Domain-Blocklist-(Ads).bro.dat, string=)critical-stack 20:25:28 [DEBUG] %!(EXTRA string=%s - %s, string=mvps.org Domain Blocklist (Ads), *errors.errorString=Feed missing: critical-stack-intel-95-mvps.org-Domain-Blocklist-(Ads).bro.dat, string=)critical-stack 20:25:28 [DEBUG] %!(EXTRA string=%s - %s, string=malwareconfig.com APTnotes (Hashes), *errors.errorString=Feed missing: critical-stack-intel-96-malwareconfig.com-APTnotes-(Hashes).bro.dat, string=)critical-stack 20:25:29 [DEBUG] %!(EXTRA string=%s - %s, string=binarydefense.com IP Banlist, *errors.errorString=Feed missing: critical-stack-intel-99-binarydefense.com-IP-Banlist.bro.dat, string=)critical-stack 20:25:30 [DEBUG] %!(EXTRA string=%s - %s, string=autoshun.org IP Shunlist, *errors.errorString=Feed missing: critical-stack-intel-101-autoshun.org-IP-Shunlist.bro.dat, string=)critical-stack 20:25:30 [DEBUG] %!(EXTRA string=%s - %s, string=torproject.org Official Exit Node List, *errors.errorString=Feed missing: critical-stack-intel-104-torproject.org-Official-Exit-Node-List.bro.dat, string=)critical-stack 20:25:31 [DEBUG] %!(EXTRA string=%s - %s, string=shodan.io Remote Access Trojan (RAT) Controllers, *errors.errorString=Feed missing: critical-stack-intel-133-shodan.io-Remote-Access-Trojan-(RAT)-Controllers.bro.dat, string=)critical-stack 20:25:31 [INFO] Creating master file: master-public.bro.dat. Please wait.critical-stack 20:25:36 [INFO] Master file created successfully.critical-stack 20:25:38 [INFO] Intel files located at: /opt/critical-stack/frameworks/intelcritical-stack 20:25:38 [INFO] API Requests Remaining: 985 of 1000/minute 0 Permalink securid December 09, 2016 17:21 For the people running Bro on the Raspberry Pi how do you get your data into Security Onion? I want to run a few Pi's and push all the data to one central spot. Thanks 0 Permalink Wes December 14, 2016 21:31 Edited @Info-france I'm also working on the same setup as you. Just some questions in the hope you can answer them:Did you compiled critical stack intel from source or did you use - curl https://packagecloud.io/install/repositories/criticalstack/critical-stack- intel/script.deb.sh | sudo bash - sudo apt-get install critical-stack-intel When I tried to apt-get there was no file found on raspbian.After that I've compiled it from source but I continuously needed to use 'sudo -u critical-stack' before all commands which is a bit annoying. Besides, I'm not getting the intel.log you supposed to get in the current log folder. The 'normal' bro logs are all working. I've set my api and sudo -u critical-stack critical-stack-intel list shows my sensors. Could you please tell me the exact commands you used to get critical-stack to work? So from adding the API key onwards. 0 Permalink Raj Kumar May 09, 2017 14:26 Hi , Is there any another link to test it apart from https://csic01.taosecurity.com from tails to test critical stack intel.log Thanks, Raj 0 Permalink Laurencefield October 23, 2016 05:25 Chris, Thank you. The above solved my problems too. 0 Permalink Info-france December 19, 2016 15:47 Edited @Wes: Hello ! to install the package: sudo wget --no-check-certificate https://intel.criticalstack.com/client/critical-stack-intel-arm.deb sudo dpkg -i critical-stack-intel-arm.deb then, check your /etc/passwd. if critical stack is store in /bin/false you can switch to the user "critical-stack" from root: su critical-stack -s /bin/bash. From here, just put your api key with the command mentioned above :) => critical-stack-intel "api-key". then, just fetch your feeds : critical-stack-intel pull and, list them: critical-stack-intel list You can also edit the /etc/passwd and switch "critical" user to /bin/bash and add it to the sudoers :) 0 Permalink psy phii February 19, 2017 09:04 Yep since the merger alot of feeds are apparently being blocked by capital ones sophos firewall, judging by the thumbnail of a sophos utm block page that replaced alot of the broken feeds origianl thumbs. 0 Permalink Richard Compton March 13, 2015 20:52 Hi Richard, I am having issues installing the Intel Client. When I run "sudo critical-stack-intel fetch run" I'm getting the error: 2015/02/25 16:10:21 --- NOTICE ---------- 2015/02/25 16:10:21 Error: Unable to locate bro or configure permissions properly. 2015/02/25 16:10:21 Error: Bro not found. 2015/02/25 16:10:21 If you have a custom setup you can add your paths manually. 2015/02/25 16:10:21 $ sudo critical-stack-intel config bro.path /my/path/bro 2015/02/25 16:10:21 $ sudo critical-stack-intel config bro.include.path /my/path/local.bro 2015/02/25 16:10:21 $ sudo critical-stack-intel config bro.broctl.path /my/path/broctl 2015/02/25 16:10:21 --- NOTICE ---------- I have run: sudo critical-stack-intel config bro.path /nsm/bro/bin/bro sudo critical-stack-intel config bro.include.path /nsm/bro/share/bro/site/local.bro sudo critical-stack-intel config bro.broctl.path /nsm/bro/bin/broctl but I'm still getting the error message. Is there any documentation on how to configure this or any idea what I am doing wrong? I opened up a Zendesk ticket 3 weeks ago but I haven't heard back from anyone. Thanks in advance! 0 Permalink Tim Crothers May 29, 2016 04:13 The "apt-get install critical-stack-intel" returns an error no package found. I triple-checked the prior and "/var/lib/apt/lists/packagecloud.io_criticalstack_critical-stack-intel_raspbian_dists_jessie_main_binary-armhf_Packages" is empty. Perhaps the raspbian debian package isn't available at the moment? In case it's not obvious I'm attempting to build this on a Raspbian Pi 3 running Bro successfully already. Thanks! Tim 0 Permalink Info-france December 22, 2016 19:43 @Wes Same issue ... :'( no more "intel.log", the file disappeared few days after the full install of my setup when i will have a moment, to work on it, i will probably reinstall all the stuff from scratch 0 Permalink Corey January 26, 2015 21:24 Thank you for the guide Richard! One issue arose after installing the Critical Stack Intel Client in that another service (ELSA) on the server no longer functions correctly. Would the package install impact MySQL in any way or modify any additional bro .conf file? 0 Permalink Richard Bejtlich March 13, 2015 22:35 Hi RC, we're taking a look and will get back with you. Thank you for trying the client! Richard 0 Permalink Chris Brown October 17, 2016 07:07 Tim, I was having a similar issue. Here's what worked for me: sudo wget --no-check-certificate https://intel.criticalstack.com/client/critical-stack-intel-arm.deb sudo dpkg -i critical-stack-intel-arm.deb sudo -u critical-stack critical-stack-intel api [API-KEY-HERE] Hope this helps! 0 Permalink Richard Bejtlich January 26, 2015 21:36 Hi Corey, Thanks for trying the client. I'm using it on my SO system without issues. ELSA works as far as I can tell. Would you mind opening a ticket in Zendesk? You can do it at intel.criticalstack.com, in the upper right side of the page. I think this might be the direct link: https://criticalstack.zendesk.com/hc/en-us/requests/new Sincerely, Richard 0 Permalink Sorin Mustaca October 05, 2015 17:19 Hi Richard B., I have the same problem as described by RC above. Additionally, it seems I have issues to authenticate my API key. Strangely, last time i checked it a few weeks ago, everything worked like a charm. Now, I have a new system where i test and nothing seems to works anymore. Testing on Ubuntu 14.04. #sudo critical-stack-intel --debug api <api> 2015/10/05 18:56:43 This API key has already been added. sudo critical-stack-intel pull critical-stack 18:56:50 [ERROR] --- NOTICE ---------- critical-stack 18:56:50 [ERROR] Unable to locate bro or configure permissions properly. critical-stack 18:56:50 [ERROR] Unable to add sudoers access for bro binary. critical-stack 18:56:50 [INFO] If you have a custom setup you can add your paths manually. critical-stack 18:56:50 [INFO] $ sudo critical-stack-intel config --set bro.path=/my/path/bro critical-stack 18:56:50 [INFO] $ sudo critical-stack-intel config --set bro.include.path=/my/path/local.bro critical-stack 18:56:50 [INFO] $ sudo critical-stack-intel config --set bro.broctl.path=/my/path/broctl critical-stack 18:56:50 [ERROR] --- NOTICE ---------- critical-stack 18:56:50 [INFO] Pulling feed list from the Intel Marketplace. critical-stack 18:56:50 [ERROR] Error: Invalid API key. critical-stack 18:56:50 [INFO] Please verify the API key you are using is correct. critical-stack 18:56:50 [ERROR] Invalid API key Thanks Sorin 0 Permalink Wes December 22, 2016 19:05 Edited @info-france Thanks for your help! So, I finally got it working. Couple of problems I encountered. I hope I can help other people working on the same thing and save them some time. _____________________________________________________________________ @Mat Yes, I have the same problem, currently those feeds at my setup are: PhishTank-Intel-Feed-(Verified) Scam-Domains-(Fake/Malware/Drive-By) Don't know how to fix this. _____________________________________________________________________ I think the ticketsystem on this site doesn't work. When you submit your ticket you can't go back to the page where you're ticket is, and you're supposed to get an email when you've a reaction, but those emails never get delivered as I read other people also never got a response while the team of critical stack claimed they had responded. _____________________________________________________________________ After routing all my traffic through the RPi, I thought my critical stack intel wasn't working for a long time. When I had configured everything as it was supposed to be, I used tor on another device to reach certain (normal) sites. But there was no intel.log every time I tried. It made me crazy and I thought I had configured something wrong. But apparently the use of tor in your local network will go unnoticed, only if someone with a tor exit node connects to your local network, it will be noticed. Quite logical, but I hadn't expected it. After this, I used tails on a other device, pointed the gateway to the Pi and tried again, this time there was almost no traffic from the tails devices' IP logged in conn.log, even though all the webpages would load. The strangest thing is, just after I enable the internet connection on the tails device I see the tails' IP connecting to a couple external IP's, after that, nothing. Doesn't matter which site I go there is nothing visible in conn.log. I'm really curious how this is possible, because I've configured it just like my other devices, from which I do see all traffic in the conn.log. Maybe some recent security measure implemented in tails or something? So finally I decided to use a Virtual Machine to connect to some malicious domains, finally there was a intel.log created. So If others experience the same problem, use a VM. _____________________________________________________________________ I hope someone of the critical stack team knows how to fix the problems above or can give his opinion about this :) Last but not least,check https://github.com/TravisFSmith/SweetSecurity for some handy scripts. 0 Permalink Please sign in to leave a comment.