How to Build Scalable eCRF Systems

The eCRF system you build in the first sprint will serve a 3-site, 80-subject pilot. The one you need two years later will run an 80-site, 12,000-subject Phase III trial across 15 countries. These are not the same system — unless you make the right architectural decisions at the beginning. This guide shows you what those decisions are.

01. Metadata-Driven Schema Design: The Decision That Defines Everything

The single most consequential eCRF architecture decision is choosing between a monolithic protocol-specific schema and a metadata-driven schema. In a monolithic design, CRF structure is encoded directly in database tables — creating tables named VISIT_2_VITALS, ECG_WEEK12, SCREENING_LABS. Every protocol amendment (which happens in virtually every Phase II/III trial) requires a database migration, which in a regulated environment means a validation protocol, change control record, and partial or full revalidation of the system.

The core entity model of a metadata-driven eCRF separates the data model (static, validated once, never changes) from the study definition (CRF structure, visit schedule, edit checks — changeable without schema impact). This is the architecture used by every purpose-built EDC platform: Medidata Rave, Oracle Clinical One, OpenClinica, and Veeva Vault CDMS all implement this separation as their foundational design choice.

-- Core data model: stable, validated once, never changes on protocol amendment

CREATE TABLE study        (study_id UUID PRIMARY KEY, oid VARCHAR(64) UNIQUE, name TEXT, status TEXT);
CREATE TABLE subject      (subject_id UUID PRIMARY KEY, study_id UUID, subject_key VARCHAR(32), site_id UUID, status TEXT);
CREATE TABLE visit_def    (visit_def_id UUID PRIMARY KEY, study_id UUID, oid VARCHAR(64), label TEXT, order_num INT);
CREATE TABLE form_def     (form_def_id UUID PRIMARY KEY, visit_def_id UUID, oid VARCHAR(64), label TEXT, repeating BOOL);
CREATE TABLE item_def     (item_def_id UUID PRIMARY KEY, form_def_id UUID, oid VARCHAR(64), data_type TEXT, label TEXT, required BOOL);

-- ITEM_DATA: the single table that stores all clinical data for all studies
-- No protocol-specific columns, no visit-specific tables — just structured item data

CREATE TABLE item_data (
    item_data_id    UUID          DEFAULT gen_random_uuid() PRIMARY KEY,
    item_def_id     UUID          NOT NULL REFERENCES item_def,
    subject_id      UUID          NOT NULL REFERENCES subject,
    visit_def_id    UUID          NOT NULL REFERENCES visit_def,
    form_def_id     UUID          NOT NULL REFERENCES form_def,
    string_value    TEXT,          -- Most clinical data stored as string, typed in app layer
    numeric_value   NUMERIC,       -- Range checks applied at entry, not schema
    date_value      DATE,
    status          TEXT          NOT NULL DEFAULT 'incomplete',
    entered_at      TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    entered_by      UUID          NOT NULL            -- FK to users table
);

-- Protocol amendment = metadata config update, zero migration, zero revalidation

Read More: Why Many eCRF Systems Fail at Scale

02. 21 CFR Part 11 Audit Trail Architecture

21 CFR Part 11 mandates that any electronic system used to create, modify, or maintain records used in FDA-regulated submissions must maintain a computer-generated, time-stamped audit trail that captures the date, time, and identity of every operator entry and action that creates, modifies, or deletes electronic records. This is non-negotiable — an eCRF without a 21 CFR Part 11 compliant audit trail cannot support a regulatory submission.

21 CFR Part 11 — §11.10(e) — Audit Trail Requirement Use of computer-generated, time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify, or delete electronic records. Record changes shall not obscure previously recorded information and audit trail documentation shall be retained for a period at least as long as that required for the subject electronic records. Source: 21 CFR §11.10(e) · US FDA Electronic Records Regulation

-- Separate audit schema — never in same DB as live clinical data
CREATE TABLE audit.item_data_history (
    audit_id         UUID          DEFAULT gen_random_uuid() PRIMARY KEY,
    item_data_id     UUID          NOT NULL,
    study_oid        VARCHAR(64)   NOT NULL,
    subject_key      VARCHAR(32)   NOT NULL,
    form_oid         VARCHAR(64)   NOT NULL,
    item_oid         VARCHAR(64)   NOT NULL,
    old_value        TEXT,                        -- REQUIRED: what it was before
    new_value        TEXT,                        -- REQUIRED: what it became
    change_reason    TEXT,                        -- Required when protocol mandates it

    action           VARCHAR(16)   NOT NULL,    -- 'INSERT' | 'UPDATE' | 'DELETE'
    changed_by_id    UUID          NOT NULL,    -- System user UUID — never nullable
    changed_by_role  VARCHAR(64)   NOT NULL,
    client_ip        INET,
    changed_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    server_utc_ts    TIMESTAMPTZ   NOT NULL DEFAULT NOW()
);

-- Enforce immutability: INSERT only — no UPDATE, no DELETE for any role including DBA
REVOKE UPDATE, DELETE ON audit.item_data_history FROM ALL;
GRANT  INSERT, SELECT ON audit.item_data_history TO   ecrf_app_role;

-- Forward to WORM storage (AWS S3 Object Lock or Azure Immutable Blob) for long retention

03. Form Versioning & Protocol Amendment Architecture

Protocol amendments produce CRF amendments. A subject enrolled before Amendment 3 may have Week 4 data captured on CRF version 1.2. After Amendment 3, Week 4 adds a new safety assessment form — captured on CRF version 1.5. At database lock, your system must clearly identify which version of each form was used to collect each data point, for every subject. An eCRF that overwrites its active CRF definition and re-renders historical data against the new version destroys regulatory traceability.

• Form versions are immutable objects, not editable records. Each CRF version is a new, independently addressable object with its own UUID. Published versions are never modified — only new versions are created. Historical data always references the version active at collection time.

• Subject-version assignment is tracked per form instance. Each ItemData record carries a crf_version_id foreign key — not just a form_oid. This enables the system to render historical data exactly as it was collected, and the investigator to see what questions a subject answered at each visit.

• Amendment deployment is tenant-aware. When Amendment 3 is activated, the system determines which subjects are affected based on their enrollment date and applicable protocol version, and activates the correct CRF version for future visits without altering past data.

• Version comparison audit report is mandatory. When deploying a new CRF version, the system must produce a diff report showing exactly which items were added, removed, or modified — stored permanently as part of the study's regulatory documentation package.

04. Multi-Site Role Architecture That Scales to 100+ Sites

A naive RBAC implementation assigns permissions per-user, per-study, per-site. With 80 sites averaging 6 users each, across 3 active studies, you have 1,440 individual role assignments to maintain — before accounting for staff turnover running at 20–30% annually in clinical research. Maintaining this matrix is a full-time job for a data manager, and errors create either access gaps (sites cannot enter data) or access violations (users see data they shouldn't).

Peerbits Service: EHR Integration Services

05. Edit Check Rules Engine: Parameterized & Maintainable

Edit checks are the primary mechanism for catching data quality issues at the point of entry. A well-governed edit check library catches protocol deviations, out-of-range values, and missing mandatory data before they become queries. A poorly managed library with 200+ hardcoded checks — added one at a time by data managers over years — fires incorrectly on 15–20% of encounters, trains site staff to ignore warnings, and requires weeks of remediation for each protocol amendment that invalidates check logic.

The correct architecture implements edit checks as a parameterized rule engine: each check is an instance of a typed rule template with parameters stored in the study metadata layer. This enables firing rate monitoring per rule, automated regression testing on amendment, and site-level exception configuration without forking rule logic.

from dataclasses import dataclass
from enum        import Enum
from typing      import Optional, Any

class EditCheckType(Enum):
    RANGE_CHECK        = "range"       # value within [min, max]
    CROSS_FORM_CHECK   = "cross_form"  # value consistent across forms
    CONDITIONAL_REQ    = "cond_req"    # item required when condition is met
    TEMPORAL_CHECK     = "temporal"    # date sequence validation
    REGEX_CHECK        = "regex"       # format validation

@dataclass
class EditCheckRule:
    check_oid:   str
    check_type:  EditCheckType
    item_oid:    str               # Target item this check evaluates
    params:      dict              # Type-specific params — stored in metadata, not code
    message:     str               # Message shown to site when check fires
    severity:    str               # 'error' | 'warning' | 'query'

    site_exceptions: list[str]     # Site IDs exempt from this check

def evaluate_range_check(value: Optional[float], params: dict, site_id: str,
                         rule: EditCheckRule) -> Optional[str]:
    """Range check: params = {'min': float, 'max': float, 'unit': str}"""
    if value is None:
        return None   # Missing value handled by required-field check, not here
    if site_id in rule.site_exceptions:
        return None   # Site has documented exception for this check

    p = params
    if not (p['min'] <= value <= p['max']):
        return rule.message.format(value=value, min=p['min'], max=p['max'], unit=p['unit'])

    return None

# Protocol amendment: update params dict in metadata — no code change, no revalidation
# heart_rate_range = EditCheckRule(..., params={'min': 40, 'max': 180, 'unit': 'bpm'})

06. Query Management at Scale: Event-Sourced Workflow Service

A Phase III trial with 80 sites and 4,000 subjects may have 20,000 open queries at any point in the study. eCRF systems that store query state in the primary clinical database — requiring joins across clinical data, form metadata, user assignments, and query history for every dashboard load — become progressively slower as the trial scales. Query dashboards that take 8 seconds to load are abandoned by data managers who switch back to spreadsheets.

Queries are a workflow object, not a database column. Architect them as a dedicated service with an event-sourced state model: every state transition (Open → Answered → Closed / Reopened) is an immutable event, not an in-place update. Query state is computed from the event log — you get a complete query lifecycle history for free, and query load is isolated from clinical data entry performance.

07. Offline Data Capture for Low-Connectivity Sites

Clinical trial sites range from academic medical centers with enterprise IT to community clinics in rural Kenya and hospital networks in Southeast Asia where internet connectivity is intermittent. An eCRF with no offline capability forces these sites back to paper, which defeats the entire purpose of electronic data capture. But offline mode in a regulated context is not simply "cache the form and sync later" — it carries specific regulatory requirements around timestamp integrity, conflict resolution, and audit trail continuity.

• Device clock timestamp vs server receipt timestamp must both be captured. Offline entries are timestamped by device clock at collection time. The server timestamp is captured at sync time. Both must be stored in the audit trail. If device and server timestamps differ by more than 2 minutes, the system must flag the entry and require investigator attestation for the discrepancy.

• Conflicts require human resolution, never last-write-wins. When an offline session syncs and finds the same field was edited both offline and by a remote user (e.g., a data manager correction), a silent last-write-wins resolution is a data integrity violation under 21 CFR Part 11. Conflicts must surface as queries requiring human resolution with a documented rationale.

• Form submission must be atomic. All items in a form submission sync as a single transaction — all committed or none. Partial sync leaving some items pending produces incomplete records that fail edit checks unpredictably and create phantom query states.

• Offline edit checks must match online edit checks. If the offline client runs a different version of edit checks than the server, the same data entry will produce different query states online vs offline. Edit check logic must be delivered to offline clients as versioned configuration, not as compiled code embedded in the client application.

08. CTMS & Central Lab Integration: Preventing Silent Data Misalignment

A functioning clinical trial generates data from at least four concurrent systems: the eCRF (site-entered clinical data), the CTMS (site contacts, monitoring visits, IRB submissions), the central lab system (hematology, chemistry, PK samples via HL7 v2 ORU messages), and the IxRS/RTSM (randomization and trial supply management). Each integration that works at study launch has a 30–40% probability of breaking within 12 months as source systems are upgraded — and when they break, they typically break silently.

A lab integration that stops receiving HL7 v2 ORU messages doesn't produce errors — it produces missing lab values that appear as incomplete CRFs rather than integration failures. The eCRF team discovers this 3 months later during a site monitoring visit. The architectural fix is proactive integration health monitoring: track the expected message frequency from each lab system (a LIMS typically sends result batches within 24 hours of analysis), and alert when the frequency drops below threshold before site staff or monitors notice.

Read More: Common Challenges in FHIR Integrations

09. CDISC SDTM Export: Built-In, Not Bolted On

CDISC SDTM (Study Data Tabulation Model) is the FDA and PMDA required format for clinical trial data submissions. If your eCRF architecture doesn't map to SDTM domains at design time, database lock becomes a 6–12 month data transformation project. This is by far the most expensive mistake in clinical data management — a mistake that is entirely preventable with upfront CDASH-to-SDTM alignment.

The correct approach maps every eCRF item to its SDTM variable at the time the item is defined in the metadata layer. SDTM domain (DM, AE, LB, VS, CM, EX, etc.), variable name, controlled terminology, and origin flag are all metadata attributes of each item definition. SDTM export is then a query against the metadata-driven model — not a custom ETL pipeline developed post-hoc at database lock.

-- Item definition carries SDTM mapping at definition time — not added at lock
ALTER TABLE item_def ADD COLUMN sdtm_domain     VARCHAR(8);   -- e.g. 'LB', 'VS', 'AE', 'DM'
ALTER TABLE item_def ADD COLUMN sdtm_variable    VARCHAR(32);  -- e.g. 'LBSTRESN', 'VSRESN'
ALTER TABLE item_def ADD COLUMN sdtm_ct_codelist VARCHAR(64);  -- CDISC controlled terminology
ALTER TABLE item_def ADD COLUMN sdtm_origin     VARCHAR(16);  -- 'CRF' | 'Derived' | 'Assigned'
ALTER TABLE item_def ADD COLUMN cdash_variable   VARCHAR(32);  -- CDASH collection variable

-- SDTM LB (Laboratory) domain export — generated from metadata + item_data
-- No manual ETL at database lock — this view is always current
CREATE VIEW sdtm_lb AS
SELECT
    s.oid                        AS studyid,
    'LAB'                        AS domain,
    sub.subject_key              AS usubjid,
    id.sdtm_variable             AS lbtestcd,      -- From item metadata, not hardcoded
    id.label                     AS lbtest,
    d.numeric_value              AS lbstresn,      -- Standardized result (numeric)
    d.string_value               AS lbstresc,      -- Standardized result (character)

    d.entered_at::date           AS lbdtc
FROM  item_data d
JOIN  item_def   id  ON d.item_def_id  = id.item_def_id  AND id.sdtm_domain = 'LB'
JOIN  subject    sub ON d.subject_id   = sub.subject_id
JOIN  study      s   ON sub.study_id   = s.study_id;

10. Database Lock & Closeout: Orchestrated, Not Manual

Database lock — the formal process of freezing a clinical trial dataset for statistical analysis — should be an orchestrated, automated workflow. In practice, it is frequently a multi-week manual process of spreadsheet-based checklists, individual query reviews, and manual confirmations. An eCRF system built for scale makes database lock a machine-checkable precondition, not a human-reviewed milestone.

Read More: HIPAA by Design: The Engineering Blueprint for Compliant Healthcare Systems

The Complete eCRF Data Flow Architecture

The 10 pillars above work together as an integrated system. Here is how data flows through a production eCRF architecture from site entry to regulatory submission:

Table------------------