Understanding DeepSeek-V3: Architecture and Design
An exploration of the architecture behind modern language models, including Mixture of Experts, attention design, and efficient model computation.
Writing & Research
Technical writing on large language models, model architectures, PyTorch implementations, GPU computing, and machine learning systems. These articles document concepts I explore, implement, and learn.
Selected articles
An exploration of the architecture behind modern language models, including Mixture of Experts, attention design, and efficient model computation.
Understanding the MTP training objective, its transformer modules, shared embeddings, projection layers, and contribution to language-model training.
A practical look at transformer components, attention mechanisms, normalization, embeddings, and training a model using PyTorch.
How tensor parallelism distributes linear layers across GPUs, including row-parallel and column-parallel computation and communication between devices.
A hands-on explanation of dataset preparation, tokenization, training configuration, loss tracking, and evaluating language-model experiments.
No articles in this category yet.