Arclight-FEBE v1.0
Executive Summary and Documentation
The Problem
Practical implementation of machine learning and artificial intelligence is significantly lacking on the front lines of public health. While many proposed solutions show promise in controlled academic settings, actual implementation is challenged by data fragmentation, data bias, privacy concerns, domain oversights, or a reliance on computing infrastructure unavailable to under-resourced local health departments.
Whether the health emergency is a new outbreak of an existing pathogen, a novel virus, a drug-resistant fungus, an environmental toxin, or a bioterror attack, the first contact between pathogen and the public health system will be the outbreak investigation conducted by local health departments. The impact generated by aiding this process with ML/AI tools is considerable, and currently unaddressed.
Our Solution
Arclight is an open-source suite of NLP models designed to empower epidemiologists and public health professionals with real-time decision support during outbreak investigations. The models are designed to run locally, on device via a cross-platform app, replicating the functionality of the latest foundation models, but without the possibility of hallucination, PHI leakage, or data contamination.
Design Goals and Exclusions
Arclight will assist field epidemiologists and public health professionals when investigating potential infectious disease outbreaks. Arclight will exclusively process information on-device to protect PHI, eliminate infrastructure costs, and add direct oversight by the end-user.
Specific Arclight functions will evolve over time, but will generally will be designed to support epidemiologists during the following phases of an outbreak investigation:
Integrating existing public health surveillance data
Processing case reports and investigative notes
Refining the case definition based on collected information and alternative diagnoses
Organizing case reports based on population, location, and time
Developing etiological hypotheses based on observed notes and reference data.
Arclight deployment will include only Android, Linux, and Windows devices. Model performance will be constrained by the demands of on-device processing, to include available modules in the required tech stack. Arclight will not replace established decision support tools used for policy development, outbreak forecasting, or other use cases requiring large data sets and server-grade processing power; rather, it fills a gap at the local level.
Detailed Documentation
For detailed information on the development schedule, data selection, pipeline, ingestion, custom tokenizer, model architecture, or other technical details, consult the following documents.
1. Development Schedule
This section will detail the key project timelines, critical engineering milestones, and phased rollout targets for field deployment.
2. Functional Requirements
This section will detail the core functional capabilities of the platform, including offline data ingestion, automated query processing, and localized reporting for field operations.
3. Data Pipeline and Curation
This section will detail the offline pipeline, specifically focusing on the integration of synthetic textbook pretraining data, offline Wikipedia parsing, and the use of MinHash LSH for text deduplication to ensure data quality.
4. Custom Tokenization
This section will outline the design and training of a custom Byte-Pair Encoding (BPE) tokenizer optimized specifically for epidemiological terminology, addressing the vocabulary limitations of off-the-shelf generalist models.
5. Model Architecture
This section will provide the neural network specifications engineered to keep the model under the 1B parameter threshold, ensuring stable offline execution on standard field epidemiology hardware.
6. Evaluation Framework
This section will define the benchmarking strategy prior to deployment, focusing on missingness diagnostics, hallucination rates, and clinical accuracy for human-in-the-loop validation.
7. Training and Optimization
This section will outline the distributed training schedule, loss functions, and hardware-specific quantization strategies used to optimize training convergence and inference speed under limited computational resources.