ECG is a non-invasive tool for arrhythmia detection. In recent years, wearable ECG-based ambulatory arrhythmia monitoring has gained increasing attention. However, arrhythmia detection algorithms trained on existing public arrhythmia databases show higher FPR when applied to such ambulatory ECG recordings. It is primarily because the existing public databases are relatively clean as they are recorded using clinical-grade ECG devices in controlled clinical environments. They may not represent the signal quality and artifacts present in ambulatory patient-operated ECG. To help build and evaluate arrhythmia detection algorithms that can work on wearable ECG from free-living conditions, we present the design and development of the CACHET-CADB, a multi-site contextualized ECG database from free-living conditions. The CACHET-CADB is subpart of the REAFEL study, which aims at reaching the frail elderly patient to optimize the diagnosis of atrial fibrillation. In contrast to the existing databases, along with the ECG, CACHET-CADB also provides the continuous recording of patients' contextual data such as activities, body positions, movement accelerations, symptoms, stress level, and sleep quality. These contextual data can aid in improving the machine/deep learning-based automated arrhythmia detection algorithms on patient-operated wearable ECG. Currently, CACHET-CADB has 259 days of contextualized ECG recordings from 24 patients and 1,602 manually annotated 10 s heart-rhythm samples. The length of the ECG records in the CACHET-CADB varies from 24 h to 3 weeks. The patient's ambulatory context information (activities, movement acceleration, body position, etc.) is extracted for every 10 s interval cumulatively. From the analysis, nearly 11% of the ECG data in the database is found to be noisy. A software toolkit for the use of the CACHET-CADB is also provided.