Understanding AI Safety Research Teams & Testing

Understanding how the team-based agentic and human-validated risk assessment process works helps you prepare better programs and respond more effectively to findings. This guide explains how AI Safety Research teams operate using The Consolidated AI Product Safety Pipeline, how they work with your programs, and how you can maximize engagement.

Who Are the Teams?

External Teams (Professional AIRTA Systems Teams)

• Professional AI Safety Research Teams: Teams of approved researchers with diverse skills and expertise who use agentic testing agents combined with human expertise
• AIRTA Systems AI Safety Professionals: Vetted security researchers, AI experts, and safety specialists
• Academic and Industry Teams: PhD researchers and industry professionals organized into collaborative teams
• Approval Process: External teams require AIRTA Systems approval (status changes from pending to active)
• Team Structure: Each team has a team lead, members with specific roles, and defined specializations
• Testing Methodology: Teams combine AI-powered agentic testing with human verification for comprehensive coverage

Internal Teams (Company-Managed Testers)

• Your Own Testing Team: Company employees or contractors you manage internally
• Pre-Approved: Internal teams are automatically approved when created by company owners/admins
• Full Control: You manage team membership, roles, and program access
• Confidential Testing: Ideal for sensitive projects requiring internal-only access
• Flexible Structure: Create multiple internal teams for different specializations or projects

Team Types Explained

External Teams

• Managed by AIRTA Systems: We vet, approve, and monitor external teams
• Invitation-Based: Companies invite specific teams to programs
• Reputation System: Teams build track records through quality project work
• Specializations: Teams showcase expertise (LLMs, Computer Vision, AI Safety, Prompt Engineering, etc.)
• Per-Project Compensation: Teams are paid per project engagement, not per finding

Internal Teams

• Managed by Your Company: You control all aspects of team operation
• Direct Access: Grant teams access to specific programs
• Company Resources: Your own employees working on your programs
• Privacy Focused: Keep sensitive testing completely internal

How Teams Work on Your Program

Invitation Model (Not Browsing)

Important: Teams do not browse and self-select programs. Instead, companies invite teams to specific projects.

• You review available teams or create internal teams
• You send invitations to teams for specific programs
• Teams receive invitations and decide whether to accept
• Accepted teams gain program access and begin structured engagement

Program Access Control

• Grant Access: Add teams to specific programs through your dashboard
• Access Levels: Control what teams can see and do within programs
• Revoke Access: Remove team access at any time
• Multiple Teams: Invite multiple teams to the same program if needed

Per-Project Engagement Structure

• Defined Scope: Each project has clear boundaries and objectives
• Timeline: Projects have specific start and completion targets
• Deliverables: Teams know exactly what outputs are expected
• Compensation: Per-project payment structure, not per-finding bounties
• Milestones: Track progress through defined project milestones

The Team-Based Risk Assessment Process

1. Team Invitation: You invite a team to your program with a detailed project brief
2. Project Brief Review: Team reviews scope, objectives, timelines, and deliverables
3. Team Planning: Team lead coordinates with members, assigns responsibilities
4. Internal Team Coordination: Team uses team chat to coordinate efforts and share findings
5. Collaborative Testing: Team members use agentic testing agents combined with human expertise to assess different aspects of your AI system for comprehensive coverage
6. Finding Compilation: Team consolidates individual findings into comprehensive reports
7. Report Submission: Team lead submits reports through the platform, crediting team members
8. Team Communication: Team lead serves as primary contact for questions and updates

What Teams Assess For

• Bias & Fairness: Unfair treatment based on demographics, systematic discrimination
• Security Testing & Risks: Comprehensive security vulnerability assessment including API key leaks, authentication bypasses, data poisoning attacks, prompt injection, model manipulation, adversarial attacks
• Hallucinations: False or misleading information generation, unreliable outputs
• Privacy Violations: Unintended data exposure, training data leaks, PII exposure
• Control Issues: AI taking unintended actions, unsafe autonomy, alignment failures
• Compliance Risks: EU AI Act violations, fundamental rights impacts, regulatory gaps

Team Reputation System

How Teams Build Reputation

• Per-Program Points: Teams earn points for each program they successfully complete
• Quality Metrics: Measured by finding quality, project completion, and client satisfaction
• Track Record: Teams display total members, points, submissions, and acceptance rates
• Specialization Evidence: Teams showcase expertise in specific AI domains

Benefits of Higher Reputation

• Premium Program Access: Higher reputation unlocks invitations to premium, high-value programs
• Increased Visibility: Top teams are prioritized when companies search for teams
• Trust Indicator: Reputation serves as quality signal for companies
• Team Growth: Successful teams can recruit more members and expand capabilities

How to Attract Quality Teams

Clear Project Briefs

• Detailed Scope: Specify exactly what needs to be tested and what's out of scope
• Clear Objectives: Define what success looks like for the project
• Timeline Expectations: Provide realistic timeframes for project completion
• Deliverable Requirements: Explain what reports and documentation you expect

Fair Per-Project Compensation

• Project-Based Pricing: Compensate teams for the entire engagement, not individual findings
• Milestone Payments: Consider structuring payments around project milestones
• Value Alignment: Ensure compensation reflects project complexity and scope
• Transparent Terms: Be upfront about budget and payment structure

Good Documentation

• System Documentation: Provide comprehensive information about your AI systems
• Architecture Diagrams: Help teams understand system components and data flows
• Access Requirements: Clearly document what access teams will have
• Testing Guidelines: Specify any testing constraints or special considerations

Responsive Communication

• Quick Responses: Reply to team questions within 24-48 hours
• Clear Answers: Provide detailed, actionable responses
• Open Dialogue: Encourage teams to ask questions and seek clarification
• Professional Tone: Maintain constructive, collaborative communication

Communication with Teams

Team Lead as Point of Contact

• Primary Contact: Team lead serves as main communication channel
• Team Coordination: Lead coordinates internally with team members
• Status Updates: Lead provides project progress updates
• Question Routing: Lead routes questions to appropriate team members

Communication Channels

• Platform Messages: Teams use the Messages system for formal communication
• Team Chat: Teams coordinate internally using team chat functionality
• Submission Comments: Discuss specific findings within submission threads
• Project Updates: Regular status updates through agreed channels

Best Practices for Team Communication

• Project-Focused: Keep communication centered on project objectives and scope
• Professional: Maintain respectful, constructive dialogue
• Documented: Use platform messages for important communications (audit trail)
• Timely: Respond to questions and updates promptly
• Clear Feedback: Provide specific, actionable feedback on deliverables

Managing Internal vs External Teams

When to Use Internal Teams

• Highly confidential or sensitive projects
• Ongoing, continuous testing needs
• Deep product knowledge required
• Regulatory requirements for internal-only testing

When to Use External Teams

• Need diverse, specialized expertise
• Fresh, independent perspective required
• Scale testing quickly for specific projects
• Access to cutting-edge AI safety techniques

Hybrid Approach

• Use internal teams for baseline testing
• Engage external teams for specialized audits
• Combine perspectives for comprehensive coverage
• Internal teams learn from external team methodologies