- An Internet Connection
- Docker
- VirusTotal Account (username and password)
- The following headers, which can be gathered by examining the headers in Burp or DevTools.
- X-Recaptcha-Response
- X-Vt-Anti-Abuse-Header
docker run -p8888:8888 azuretrustworthyml/counterfit-mlsecevasion:latest
Get into the interface and open a terminal. Counterfit should start automatically, if it doesn't you can run python counterfit.py
There is a basic use case in the Counterfit wiki
list targetslist frameworks
---------------------------------------------------
__ _____ __
_________ __ ______ / /____ _____/ __(_) /_
/ ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
/ /__/ /_/ / /_/ / / / / /_/ __/ / / __/ / /
\___/\____/\__,_/_/ /_/\__/\___/_/ /_/ /_/\__/
#ATML
---------------------------------------------------
[+] 28 attacks
[+] 8 targets
counterfit> list targets
Name Type Input Shape Location
----------------------------------------------------------------------------------------------------------------------------------------
mlsecmalware pe (1,) https://api.mlsec.io/api
moviereviews text (1,) counterfit/targets/moviereviews/movie_reviews_sentiment_analysis.pt
creditfraud numpy (30,) counterfit/targets/creditfraud/creditfraud_sklearn_pipeline.pkl
ember pe (1,) counterfit/targets/ember/ember_model.txt.gz
satelliteimages image (3, 256, 256) counterfit/targets/satelliteimages/satellite-image-params-airplane-stadium.h5
tutorial image (1, 28, 28) counterfit/targets/tutorial/mnist_sklearn_pipeline.pkl
mlsecphish html (1,) https://api.mlsec.io/api/phish
counterfit> list frameworks
Framework # of Attacks
----------------------------------------------------
mlsecevade 10
textattack 11
art 7
load mlsecevadelist attacks
counterfit> load mlsecevade
[+] Framework loaded successfully!
counterfit> list attacks
Name Type Category Tags Framework
----------------------------------------------------------------------------------------
hop_skip_jump-pe evasion blackbox pe mlsecevade
randdescent-html evasion blackbox html mlsecevade
zoo-html evasion blackbox html mlsecevade
hyperopt-pe evasion blackbox pe mlsecevade
zoo-pe evasion blackbox pe mlsecevade
hop_skip_jump-html evasion blackbox html mlsecevade
boundary-html evasion blackbox html mlsecevade
hyperopt-html evasion blackbox html mlsecevade
boundary-pe evasion blackbox pe mlsecevade
counterfit> new
? Target name: virustotal
? Which framework? art
? What data type? numpy
The target is written to a folder under /counterfit/targets/.
Note: There is a bug for the new framework. We will change some of this information in the next step.
from counterfit.core.targets import ArtTarget
class virustotal(ArtTarget):
model_name = "virustotal"
model_data_type = "pe"
model_endpoint = "https://www.virustotal.com"
model_input_shape = (1,)
model_output_classes = [0, 1]
X = []
def __init__(self):
self.X = []
def __call__(self, x):
return xYou should see the information about the model change.
counterfit> reload
counterfit> list targets
counterfit> list targets
Name Type Input Shape Location
----------------------------------------------------------------------------------------------------------------------------------------
...
virustotal pe (1,) https://virustotal.com
...
Here is the completed target for reference. Note: there was a revision that changed the way target interacts with VT. The API takes >10 minutes to process new files, but the UI takes ~75 seconds. As Counterfit is generating new malware algorithmically, this code change will save ALOT of time.
The process inside def __init__(self) is,
- Download malware samples.
- Load malware samples into the target (self.X)
- Manually gather the proper headers
- Login programatically, keep a session
After you have data loaded, it's time to send it to the target. The process inside def __call__(self, x) is,
- Extract the malware from the list of samples
- Upload the malware to VT
- Check the results with the file_id VT returns
- Process scores and return a list of probabilities
counterfit> interact virustotal
[-] scanning malware sample info from counterfit/targets/virustotal/mlsec_malware_samples.zip
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 214652.20it/s]
virustotal>
virustotal> use hyperopt-pe
[+] Using hyperopt-pe 3321e1d0
virustotal>hyperopt-pe> show options
Attack Parameter (type) Default Current
-----------------------------------------------------
max_evals (int) 10 10
sample_index (int) 0 0
target_class (int) 0 0
virustotal>hyperopt-pe>
You can explore what to print, save them off, etc. Be aware that as scores change the regex might not work any more for some products. For example, CrowdStrike will go from win/malicious_confidence_100% (W) to 'usafe'. This both breaks a regex looking for a score and obviously removes a score for an algo to use.
virustotal>hyperopt-pe> run
[+] Running hyperopt-pe on virustotal
[+] File id is: ZDJmYzQwZWZmZTQyZDFiODE5NDgwN2MyY2I0YmYxOGM6MTYyODI5NTI4MA==
Remember to reload the target after making changes!
{
'category': 'malicious',
'engine_name': 'CrowdStrike',
'engine_version': '1.0',
'result': 'win/malicious_confidence_100% (W)',
'method': 'blacklist',
'engine_update': '20210203'
},
{
'category': 'malicious',
'engine_name': 'eGambit',
'engine_version': None,
'result': 'Unsafe.AI_Score_94%',
'method': 'blacklist',
'engine_update': '20210806'
},
{
'category': 'malicious',
'engine_name': 'MAX',
'engine_version': '2019.9.16.1',
'result': 'malware (ai score=80)',
'method': 'blacklist',
'engine_update': '20210806'
},
{
'category': 'malicious',
'engine_name': 'Cynet',
'engine_version': '4.0.0.27',
'result': 'Malicious (score: 100)',
'method': 'blacklist',
'engine_update': '20210806'
}