A comprehensive Rust implementation comparing StandardNet vs RecursiveNet architectures to analyze space-for-time computational trade-offs in neural networks.
This project implements and compares two neural network architectures on the MNIST dataset:
- StandardNet: Traditional feedforward neural network with multiple layers
- RecursiveNet: Novel architecture using recursive computation with weight sharing
The experiment validates the hypothesis that recursive architectures can achieve competitive performance with fewer parameters through space-for-time trade-offs.
This implementation is inspired by and references the theoretical foundations presented in:
📖 Reference Paper: Space-Time Computational Trade-offs in Neural Networks
The paper explores fundamental trade-offs between memory usage (space) and computational steps (time) in neural architectures, providing the theoretical foundation for this empirical study.
- RecursiveNet Depth 4: 93.66% accuracy with 78,802 parameters
- StandardNet Medium: 93.07% accuracy with 109,386 parameters
- Result: RecursiveNet achieves better accuracy with 28% fewer parameters
- StandardNet: Predictable scaling (more parameters → better accuracy)
- RecursiveNet: Optimal depth window (depths 2-4 work well, 6+ fail due to gradient vanishing)
✅ Space-for-Time Hypothesis CONFIRMED
- Fewer parameters through weight sharing
- Sequential vs parallel computation trade-off
- Competitive performance at optimal configurations
- Rust (latest stable version)
- ~100MB disk space for MNIST dataset
# Clone or download the project
cd rust_net_demo
# Run the comprehensive experiment (10-30 minutes)
cargo run --release
# Run focused analysis (quick summary)
cargo run --bin focused_experiment
# Run validation test
cargo run --bin test_experimentThe experiment will automatically download the MNIST dataset on first execution.
StandardNet Configurations:
- Small: 784→64→10 (≈51K parameters)
- Medium: 784→128→64→10 (≈109K parameters)
- Large: 784→256→128→64→10 (≈235K parameters)
RecursiveNet Configurations:
- Depth 2: 784→120→10 with 2 recursive iterations (≈95K parameters)
- Depth 4: 784→88→10 with 4 recursive iterations (≈70K parameters)
- Depth 6: 784→70→10 with 6 recursive iterations (≈56K parameters)
- Depth 8: 784→60→10 with 8 recursive iterations (≈48K parameters)
- Learning Rates: 0.005, 0.01, 0.02
- Batch Sizes: 32, 64, 128
- Epochs: 8, 10, 15
- Training/Validation Split: 50K/10K from MNIST training set
- Accuracy: Final test set performance
- Training Time: Wall-clock time for complete training
- Inference Time: Forward pass speed measurement
- Memory Efficiency: Accuracy per parameter ratio
- Convergence Analysis: Epoch at which training stabilizes
- Loss Tracking: Cross-entropy loss throughout training
| Architecture | Accuracy | Parameters | Memory Efficiency | Use Case |
|---|---|---|---|---|
| RecursiveNet D4 | 93.66% | 78,802 | 0.012 | Memory-constrained |
| StandardNet Medium | 93.07% | 109,386 | 0.009 | Balanced |
| StandardNet Large | 94.54% | 242,762 | 0.004 | Maximum accuracy |
- Parameter Efficiency: RecursiveNet D4 achieves better accuracy with 28% fewer parameters
- Computational Pattern: StandardNet enables parallel computation; RecursiveNet requires sequential processing
- Scalability Limits: RecursiveNet depths >4 suffer from gradient vanishing problems
- Sweet Spot: Depth 4 balances recursive benefits with trainability
- Activation Functions: ReLU for hidden layers, Softmax for output
- Training Algorithm: Backpropagation with mini-batch SGD
- RecursiveNet Feature: Backpropagation Through Time (BPTT) for recursive layers
- Loss Function: Cross-entropy with gradient clipping
- Optimization: Manual parameter updates with configurable learning rates
src/
├── main.rs # Complete experiment implementation
├── focused_experiment.rs # Quick analysis summary
└── test_experiment.rs # Validation testing
Key Functions:
├── StandardNet # Traditional feedforward implementation
├── RecursiveNet # Recursive architecture with BPTT
├── ExperimentMetrics # Comprehensive metrics collection
└── Analysis Functions # Statistical comparison and reporting
- experiment_results.csv: Complete experimental data for further analysis
- Console Output: Real-time training progress and final comparisons
- Statistical Analysis: Winner identification by category and trade-off summaries
- Edge computing with memory constraints
- Mobile/embedded ML applications
- Parameter budget limitations
- Research into recursive architectures
- Maximum accuracy requirements
- Parallel computation advantages
- Predictable performance scaling
- Production systems with ample resources
This experiment demonstrates that:
- Recursive architectures can achieve parameter efficiency without sacrificing performance
- Optimal depth selection is critical for recursive networks (depth 4 optimal for MNIST)
- Space-for-time trade-offs are practically viable in real machine learning scenarios
- Gradient flow limitations impose fundamental constraints on recursive depth
- Primary Reference: Space-Time Computational Trade-offs in Neural Networks
- MNIST Dataset: http://yann.lecun.com/exdb/mnist/
- Implementation Framework: Rust with ndarray for numerical computing
This is a research experiment implementation. For questions or improvements:
- Review the experimental methodology in
src/main.rs - Check results validation in
focused_experiment.rs - Refer to the original paper for theoretical background
This implementation is provided for research and educational purposes. Please cite the original paper when using this work:
@article{space_time_tradeoffs_2025,
title={Space-Time Computational Trade-offs in Neural Networks},
author={[Authors from the paper]},
journal={arXiv preprint arXiv:2502.17779},
year={2025}
}🎉 Successfully demonstrates that recursive neural architectures can achieve competitive performance with significantly fewer parameters through intelligent space-for-time trade-offs!