How to use these guides
Choose the tasks your team needs to complete, then ask each vendor to demonstrate them using representative evidence. Agree on success measures first and include failures, missing data and legitimate exceptions. The same questions apply to ARRTECH.
Questions for every vendor
Where does our data live, and what leaves?
A complete answer lists every way data leaves, from support uploads to AI services, and who can turn each off.
What does it cost at twice our volume?
Get a quote at today's volume and twice it, including anything charged beyond the license.
Can we leave with our data?
Data, rules and reports should export in a form another tool can read.
Which actions are automated?
Ask which actions run automatically, which wait for approval and where each is recorded.
What does the product not do?
The vendor should state the product’s limits in writing.
How will we evaluate it?
Ask whose data it uses, who sets the pass criteria and who may tune the product mid-test.
Who supports it, and from where?
Ask where support staff work and whether any of them can reach the deployment.
SIEM
A security information and event management (SIEM) platform collects, stores, correlates and reports on logs.
Which of our sources are supported today?
Bring the list of priority sources and ask who writes any missing parser, at what cost.
How do we know a source has gone silent?
A source that stops sending is easy to miss. The platform should alert when one goes silent.
What is the pricing unit?
Pricing can be by data volume, events per second, agents or users. Ask which unit applies and what happens past any cap.
Where does it run?
Ask whether it runs on the buyer’s own servers or in the vendor’s cloud, and how it fails over.
Can a third party verify our logs?
Ask how stored logs are protected from change. Can an auditor verify an export without the vendor's software?
How is retention set?
Check whether retention can be set per source and whether archives stay searchable.
How do rules and alerts work?
Have the vendor show a rule that spans sources and time, and its MITRE ATT&CK mapping.
How do we test detection?
Use representative telemetry and known scenarios, including ordinary activity that resembles a threat. Review useful findings, misses, false positives and the time needed to assess them.
Can our saved work move with us?
Ask whether saved searches, rules and reports export in a form another tool can read.
What does support include?
Check whether support covers new parsers, regulation updates and help with migration.
What AI runs inside it?
Ask what each AI feature does, what it reads and where its model runs.
Our answers: ARRTECH SIEM
SOAR
A security orchestration, automation and response (SOAR) platform runs response playbooks across existing security tools.
What does it need from our SIEM?
SOAR acts on the alerts it receives. Ask which alert sources it accepts.
Which integrations exist, and who builds new ones?
Ask for the list, and how much code a playbook takes to build.
Which actions run without approval?
Can an approval sit at any step? Ask who receives it and what happens while it waits.
Who chose what, and when?
Look for each run and each human decision on record, by name. Ask how long it is kept.
How are playbooks tested?
See whether a playbook can run on sample data without acting.
How is a false positive kept from acting?
Ask what keeps a wrong alert from triggering an action that cannot be undone.
Which actions can be reversed?
Ask which have a rollback and which do not.
How are incidents created?
Ask what can open an incident, and how indicators are pulled out.
What is measured?
Ask which measures the dashboards compute, such as time to respond, and how.
Can we take our playbooks with us?
Check that playbooks export in a form another tool can read.
Where does it run?
Ask whether it runs on the buyer’s own servers and how it reaches the systems it acts on.
Our answers: ARRTECH SOAR
DLP
A data loss prevention (DLP) product finds sensitive data and controls how it leaves.
Which channels are covered?
List every exit, from email and USB drives to generative-AI services, and ask which are covered with and without an agent.
Is encrypted traffic inspected?
Most data leaves over encrypted connections. Ask whether inspection depends on the browser or application.
How is sensitive data recognized?
Ask what it uses beyond patterns, such as file fingerprints, and how false positives are measured.
What happens when a rule matches?
Ask what a rule can do besides block. Can a new policy run in monitoring mode first?
What does each incident record?
It should name the user, device, destination and rule. Ask who can see the matched content.
What reaches generative-AI services?
Ask whether prompts and uploads are inspected for content, or only blocked by domain.
Does it find data at rest?
Check whether it scans endpoints, file servers and databases, and what it does with findings.
How is employee privacy handled?
Monitoring staff means processing personal data. Ask what the product records about people and who sees it.
Where does it run?
Ask where content is inspected and stored in each deployment option.
Can a user stop the agent?
Ask how the agent resists being disabled, including by local administrators.
How does it reach our SIEM?
Check whether incidents reach the SIEM as a log source.
Our answers: ARRTECH DLP
AI detection
AI Detection identifies activity worth reviewing. AI Investigation gathers context and prepares an assessment. Evaluate the two jobs separately, including the evidence and human effort needed to use their results.
What does a finding show?
Each finding should show the baseline, score and events behind it, so an analyst can see why.
How does it report missing telemetry?
Ask how the product tells "nothing suspicious found" from "the data was incomplete".
How long from event to finding?
Ask for the full delay, including any batch wait.
How is a new detector introduced?
Look for a shadow mode before a detector alerts, with promotion and rollback on record.
How is it measured, and on whose data?
The test should use the buyer’s own telemetry and count false positives and misses. Ask how any published score was measured.
Is our data used to train models?
Ask for a written yes or no on whether customer data trains, tunes or evaluates models used for other customers.
Where does inference run?
Ask where the models run and what goes to any outside service.
Which models, from whom?
Ask for each model's source, license and version, and how updates are delivered.
What does the AI produce?
Ask what the AI produces, how analysts review and accept it and which tool runs the response.
How is prompt injection handled?
Logs and emails can contain text written to steer a model. Ask how the product handles it.
What does an investigation record?
Each conclusion should carry its evidence and open objections. Can an analyst rerun each query?
When can it conclude that nothing happened?
An empty result or a dead feed does not prove that nothing happened. Ask how the product reports each one.
How are changes announced?
Ask how much notice comes before a new model, and whether an update can be held or rolled back.
Our answers for Cyberdroid AI Detection (available now to select organizations) and Cyberdroid AI Investigation (early access): Cyberdroid SOC Platform · Evaluations
