Problem.
Manual triage of pharmaceutical adverse-event reports is slow, inconsistent, and expensive when report volumes are high. The pharmacovigilance teams that classify these reports against standardised taxonomies are bottlenecked by volume, and the classification quality varies analyst by analyst.
The question this project asked: can a multi-class classifier match human-quality triage accurately enough to be worth deploying as a decision-support tool? And, since pharmacovigilance is a regulated domain, can the explainable baseline get close enough to the more complex model that you can ship the explainable one?
My contribution.
Solo, end to end:
- Built the feature pipeline. TF-IDF vectorisation of medicine name, composition, indications, and side-effect text, plus manufacturer ordinal encoding (with safe handling of unseen manufacturers at inference) and review-percentage features, all behind a single sklearn
ColumnTransformer+FeatureUnion. - Trained two complementary models on 11,825 marketed medicines mapped to a MedDRA-style taxonomy of ten clinical categories. Random Forest as the primary model, Logistic Regression as the interpretable comparison.
- Wrapped both in a serialisable sklearn
Pipelinesojoblib.dumpproduces one end-to-end artefact and inference does not require manual feature prep. - Externalised every hyperparameter to a YAML config loaded into frozen dataclasses. No magic numbers in code.
- Wrote the CLI (
train,predict) and the pytest suite (synthetic 60-row fixture so tests do not depend on the 4 MB xls). - Set up the GitHub Actions matrix CI on Python 3.10, 3.11, and 3.12 with ruff and black gating.
Architecture.
+------------------+ +-----------------------+ +------------------+
| Medicine_Details | -> | Category mapping | -> | Stratified split |
| (.xls, 11,825) | | (10 MedDRA-aligned | | (80/20, seed 42) |
+------------------+ | categories) | +------------------+
+-----------------------+ |
v
+----------------------------+
| sklearn Pipeline |
| ColumnTransformer |
| TF-IDF (name, comp, |
| uses, side fx) |
| OrdinalEncoder (mfr) |
| StandardScaler (rev %) |
| Classifier |
| RF or LR (registry) |
+----------------------------+
|
v
joblib artefact + metrics + plots
Results.
The training pipeline emits accuracy, macro F1, weighted F1, per-class precision and recall, and a confusion matrix figure. Random Forest reaches 98.5 percent accuracy on the held-out stratified test set; the Logistic Regression baseline reaches 97.2 percent. Random Forest stays above 95 percent F1 across every retained category.
Lessons.
The most useful finding was that the much simpler model came within a couple of percentage points of the more complex one. This shows that the categorical signal in the data is strong enough for a shallow interpretable model to perform almost as well, without a deep architecture. In a regulated domain like pharmacovigilance, where every model decision needs to be explainable, this finding affects which model to deploy. You ship the explainable model unless you absolutely need the marginal extra points.
The second lesson was about scope discipline. A first version of this had a Streamlit demo, a Hugging Face transformer baseline, and a deployment workflow. None of it was finished. I cut all three and shipped the production-grade core: feature pipeline, two models, evaluation, inference CLI, tests, CI matrix. The result has fewer features, and each one is complete.
The third lesson was about consistency between training and inference. The original prototype encoded manufacturer with a one-shot LabelEncoder outside the pipeline, which would crash at inference on any unseen manufacturer. I moved the encoding inside the pipeline as OrdinalEncoder(handle_unknown="use_encoded_value"); the fix does not change the metrics, but it stops the model from failing on unseen manufacturers in production data.
Stack.
Python · scikit-learn · pandas · numpy · TF-IDF · Random Forest · Logistic Regression · joblib · pytest · ruff · black · GitHub Actions