How to use these guides
Ask every vendor the same questions, in writing, and ask for the record behind each answer. Then test the answers in an evaluation on your own data, against a scorecard you agree before the first case. A vendor that cannot answer a question has still told you something. These questions apply to every vendor, including us. Our answers are linked at the end of each guide.
Questions for every vendor
Where does our data live, and what leaves?
Ask for every path by which data leaves your environment: telemetry, diagnostics, license checks, support uploads and any AI service. A complete answer names each destination, who operates it and who can switch it off.
What does it cost at twice our volume?
Ask for the pricing unit and a quote at today's volume and at twice it. Ask what is charged beyond the license: retention, searches, agents, implementation and training.
Can we leave with our data?
Ask how data, rules, detections and reports export, in what form, and whether a third party can read the export without the vendor's software.
What acts without a person?
Ask for the actions the product takes on its own, the actions that wait for a person, and where each decision is recorded.
What does the product not do?
Ask for the limits in writing. A vendor that states none is leaving you to find them.
How will we evaluate it?
Ask whose data the evaluation uses, who agrees the success criteria and when, how long it runs, and who may tune the product during it.
Who supports it, and from where?
Ask where support staff work, in which time zones and languages, and whether any of them can reach your deployment.
SIEM
A security information and event management (SIEM) platform collects, stores, correlates and reports on logs. These questions test collection, cost, evidence and exit.
Which of our sources are supported today?
Bring your list of priority sources. Ask which are supported out of the box, which need a custom parser, and who writes it at what cost.
How do we know a source has gone silent?
A missing source looks like a quiet day. Ask how the platform alerts when a source stops sending, and what agents do when the server is unreachable.
What is the pricing unit?
Data volume, events per second, sources, agents and users each reward different habits. Ask which unit applies, whether there is a cap, and what happens when you exceed it.
Where does it run?
Ask whether it runs on your servers, in the vendor's cloud or both, which operating systems it supports, and how it scales and fails over.
Can a third party verify our logs?
Ask how stored logs are protected from change and deletion, how a change is detected, and whether an auditor can verify an export without the vendor's software.
How is retention set?
Ask whether retention is set per source, what archived data costs to keep, and whether archives stay searchable.
How do rules and alerts work?
Ask to see a correlation rule that spans sources and time, how duplicate alerts are suppressed, and how alerts map to MITRE ATT&CK.
How do we test detection?
Ask whether you can run known attack techniques and see which alert fires, if any. Ask for true and false positive results on your data, not a lab's.
Can our saved work move with us?
Ask whether searches, rules and reports use one language, and whether they export in a form another tool can read.
What does support include?
Ask what the support contract covers, including new parsers, updates when regulations change, and help with migration.
What AI runs inside it?
Ask what each AI feature does, where its model runs, what data it reads, and whether anything it drafts goes live without a person.
Our answers: ARRTECH SIEM
SOAR
A security orchestration, automation and response (SOAR) platform runs response playbooks across your tools. These questions test integration, control and testing.
What does it need from our SIEM?
A SOAR platform acts on alerts it receives. Ask which alert sources it accepts and whether it detects anything on its own.
Which integrations exist, and who builds new ones?
Ask for the integration list, how a new API integration is built, and how much code an analyst writes to build a playbook.
Which actions run without approval?
Ask whether an approval can sit at any step, who receives the request, and what happens while it waits.
Who chose what, and when?
Ask whether the history records every run, every human decision and who made it, and how long the history is kept.
How are playbooks tested?
Ask whether a playbook can run on sample data without acting, and how errors show in the run history.
How is a false positive kept from acting?
Automated containment on a wrong alert disrupts the business. Ask how the product stops a false positive from triggering an action that cannot be undone.
Which actions can be reversed?
Ask which actions have a rollback and which do not.
How are incidents created?
Ask whether incidents open from alerts, a mailbox, an API and by hand, and how indicators are extracted from them.
What is measured?
Ask which measures the dashboards compute, such as time to respond and SLA breaches, and how each is calculated.
Can we take our playbooks with us?
Ask whether playbooks and workflows export in a readable form.
Where does it run?
Ask whether it runs on your servers, standalone or distributed, and how it reaches the systems it must act on.
Our answers: ARRTECH SOAR
DLP
A data loss prevention (DLP) product finds sensitive data and controls how it leaves. These questions test coverage, accuracy, response and privacy.
Which channels are covered?
List your exits: endpoints by operating system, email, web, file shares, removable media, printers, generative-AI services and unmanaged devices. Ask which are covered, with and without an agent.
Is encrypted traffic inspected?
Most data leaves over encrypted connections. Ask whether the product inspects them, how, and whether that depends on the browser or application.
How is sensitive data recognized?
Ask which methods go beyond patterns: validated identifiers, file and database fingerprints, text inside archives and images. Ask how false positives are measured.
What happens when a rule matches?
Ask for every response: block, allow and log, ask a manager, ask the user for a reason, encrypt, quarantine. Ask whether a new policy can run in a monitoring mode first.
What does each incident record?
Ask whether the record holds user, device, destination, rule, action and time, and who can see the matched content.
What reaches generative-AI services?
Ask whether prompts, uploads and API requests sent to generative-AI services are inspected by content, not only blocked by domain.
Does it find data at rest?
Ask whether it scans endpoints, file servers and databases, with or without agents, and what it can do with what it finds.
How is employee privacy handled?
Monitoring staff is itself processing personal data. Ask what the product records about people, who can see it, and how access to those records is limited.
Where does it run?
Ask for on-premises, SaaS and hybrid options, and where content is inspected and stored in each.
Can a user stop the agent?
Ask how endpoint agents are protected from being disabled, including by local administrators.
How does it reach our SIEM?
Ask whether incidents feed your SIEM as a log source, and through which interface.
Our answers: ARRTECH DLP
AI detection
AI detection here means behavioral detection that learns normal activity and flags change, and AI that investigates alerts. These questions test evidence, measurement, data use and control.
What does a finding show?
Ask whether each finding shows the observed value, the baseline, the score and the threshold, with the events behind it. A score alone is not evidence.
How does it report missing telemetry?
Ask how the product tells "nothing suspicious found" from "the data was incomplete".
How long from event to finding?
Ask for the delay from an event to a finding, including any batch wait, stated plainly.
How is a new detector introduced?
Ask whether a detector can run in a shadow mode before it alerts, how it is measured there, and whether promotion and rollback are recorded.
How is it measured, and on whose data?
Ask for an evaluation on your telemetry, with labeled scenarios and criteria agreed first, that counts false positives, duplicates and misses. Ask how any published score was measured before you rely on it.
Is our data used to train models?
Ask for a yes or no in writing: whether your data trains, tunes or evaluates models used for other customers.
Where does inference run?
Ask where the models run, what is sent to any outside service, and who configures that.
Which models, from whom?
Ask for each model's source, license and version, and how updates reach you.
What can the AI change?
Ask whether the AI can block, change or delete anything, approve its own work, or choose who receives its output.
What stops a planted instruction?
Logs and emails can carry text written to steer a model. Ask how the product keeps text in evidence from becoming an instruction.
What does an investigation record?
Ask whether each conclusion carries its evidence references, confidence and open objections, and whether an analyst can rerun each query.
When can it conclude that nothing happened?
Ask how an empty result or a dead feed is treated. Neither is proof that nothing happened.
How are changes announced?
Ask how far ahead you hear of a new model or detector, whether you can hold an update, and how it is rolled back.
Our answers: Cyberdroid SOC Platform · Evaluations
Further reading
Australian Signals Directorate's ACSC, CISA and partners. Implementing SIEM and SOAR Platforms: Executive Guidance and Practitioner Guidance. 2025.
Australian Signals Directorate's ACSC and partners. Priority Logs for SIEM Ingestion: Practitioner Guidance. 2025.
NIST. SP 1800-28, Data Confidentiality: Identifying and Protecting Assets Against Data Breaches. 2024.
NIST. SP 800-55, Measurement Guide for Information Security. 2024.
US Office of Management and Budget. M-25-22, Driving Efficient Acquisition of Artificial Intelligence in Government. 2025.
NSA, CISA, FBI and partners. Deploying AI Systems Securely. 2024.
NIST. AI 100-1, Artificial Intelligence Risk Management Framework. 2023.
OWASP. LLM AI Cybersecurity and Governance Checklist, version 1.1. 2024.
MITRE Center for Threat-Informed Defense. Adversary Emulation Library.
