Machine Learning Without a GPU on an 8 GB Laptop
Running machine learning models on a laptop with only 8 GB of RAM and no GPU requires careful optimization. This post documents the workflow for training and experimenting with small models, using cloud GPU resources for heavier tasks, and optimizing the local environment.
Hardware Limitations
An 8 GB RAM laptop can comfortably run models up to approximately 2 billion parameters in 4-bit quantized mode. Beyond that, swapping to disk causes severe performance degradation. The CPU becomes the primary compute device, so operations that leverage vectorized instructions (SIMD) perform better.
Environment Setup
Python 3.11 with the following packages: torch (CPU-only build), transformers, datasets, and bitsandbytes for 8-bit optimization. A virtual environment is recommended:
python -m venv venv source venv/bin/activate pip install torch transformers datasets bitsandbytesData Preparation
Use small, curated datasets. Load data in batches that fit within the 8 GB limit. For the IMDB sentiment dataset, a batch size of 16 works well. Larger batch sizes trigger OOM (out-of-memory) errors on the CPU-only path.
Cloud GPU Workflow
For experiments requiring larger models, Google Colab Pro provides GPU access. The workflow: save model code and data to Google Drive, run the experiment on Colab, and download results. This allows quick iteration without maintaining GPU hardware.
Memory Optimization
- Use 4-bit or 8-bit quantization via
bitsandbytes. - Enable
torch.float32only where necessary; usefloat16for inference. - Clear cache between experiments:
torch.cuda.empty_cache()(even on CPU, this frees memory). - Batch size tuning: start small (8) and increase until the first OOM error, then settle one step below.
Python Examples
Loading a quantized model:
from transformers import AutoModelForSequenceClassification, AutoTokenizer import torch model_name = "distilbert-base-uncased" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name, device_map="cpu", torch_dtype=torch.float32)Smoke Tests & Small Experiments
Run a quick test to verify the environment:
import torch print(f"PyTorch version: {torch.__version__}") print(f"CUDA available: {torch.cuda.is_available()}") # Should be False on CPU-only