A Full Sequence-Based Reasoning Example with Code Breakdown

Neural networks become far more interesting when they process information as sequences instead of fixed numerical inputs.

In the previous example (“Teaching a Neural Network to Add Two Numbers with PyTorch (Full Working Example)“), we trained a model like this:

[10, 15] → 25

and generate another sequence:

"25"

In this article, we will build a complete PyTorch sequence model that learns addition character-by-character.

The model will:

• Read arithmetic expressions as text
• Encode the sequence internally
• Generate answers token-by-token
• Learn from examples using gradient descent
• Generalize to new arithmetic expressions

This introduces important concepts used in:

• Sequence models
• Encoder-decoder architectures
• GRUs and LSTMs
• Language models
• Chain-of-thought systems
• Neural reasoning architectures

What We Are Building

We want the neural network to learn this transformation:

"12+7=" → "19"

Instead of directly supplying two numerical variables, we provide a sequence of characters.

The network must:

1. Read the digits
2. Understand the plus operator
3. Maintain internal memory
4. Generate the answer sequence

This is much closer to real-world AI reasoning systems.

Overall Architecture

The model follows a simple encoder-decoder structure:

Input Sequence      
↓Character Embeddings      
↓Encoder GRU      
↓Hidden Representation      
↓Decoder GRU      
↓Generated Output Sequence

The encoder compresses the input expression into an internal representation.

The decoder then generates the answer one token at a time.

Mathematical View

The model learns a sequence transformation function:

$P(y_1,y_2,…,y_n\mid x_1,x_2,…,x_m)$

where:

• $x$ = input tokens
• $y$ = output tokens

The hidden state evolves through time:

$h_t = \mathrm{GRU}(x_t,h_{t-1})$

This hidden state acts as the model’s working memory.

Full Working Code

Save this as:

sequence_addition_reasoning.py

👉 You can experiment with a practical Python implementation of this concept in the official GitHub repository for the Reasoning Systems examples: https://github.com/BenardoKemp/reasoningsystems/tree/main/practical-python/sequence_addition_reasoning

Step-by-Step Code Breakdown

1. Creating the Vocabulary

The model cannot process raw text directly.

We must convert characters into token IDs.

tokens = [
    "<PAD>", "<SOS>", "<EOS>",
    "0", "1", "2", ...
]

Special tokens:

The mappings:

stoi

and

itos

convert between characters and integers.

2. Generating Training Examples

We randomly create arithmetic problems:

a = random.randint(0, MAX_NUMBER)
b = random.randint(0, MAX_NUMBER)

Then convert them into text:

input_text = f"{a}+{b}="

Example:

"27+8="

The target becomes:

"35"

3. Encoding Sequences

Neural networks require numerical tensors.

Input text:

"27+8="

becomes something like:

[5, 10, 13, 11, 14]

The encoder transforms characters into integers.

4. Embedding Layer

The embedding layer converts token IDs into dense vectors.

self.embedding = nn.Embedding(...)

Instead of representing characters as simple integers, embeddings learn semantic representations.

This is the same principle used in language models.

5. GRU Encoder

The encoder reads the sequence one token at a time.

self.gru = nn.GRU(...)

The hidden state updates at every step:

$h_t = \mathrm{GRU}(x_t,h_{t-1})$

The final hidden state summarizes the entire arithmetic expression.

6. Decoder

The decoder generates the answer token-by-token.

Example generation:

<SOS> → "1" → "9" → <EOS>

The decoder uses:

• previous generated token
• hidden memory state

to predict the next token.

7. Teacher Forcing

During training, we feed the correct previous token into the decoder.

decoder_input = target[:, t].unsqueeze(1)

This stabilizes learning.

Without teacher forcing, training becomes harder and less stable.

8. Cross Entropy Loss

The model predicts probability distributions over vocabulary tokens.

We optimize using:

nn.CrossEntropyLoss()

The objective:

$\mathcal{L}=-\sum y\log(\hat{y})$

Lower loss means better token predictions.

9. Backpropagation

The core learning step:

loss.backward()

PyTorch automatically computes gradients through time.

This process is called:

• Backpropagation Through Time (BPTT)

for recurrent sequence models.

10. Inference

After training, we test on unseen expressions.

Example:

predict("45+32=")

Possible output:

77

The decoder generates one character at a time until:

<EOS>

is produced.

Example Output

You may see results like:

Epoch 1/30, Loss: 1.9213
Epoch 10/30, Loss: 0.1421
Epoch 20/30, Loss: 0.0329
Epoch 30/30, Loss: 0.0114
Test Results
--------------------
3+4= predicted: 7 | actual: 7
10+15= predicted: 25 | actual: 25
27+8= predicted: 35 | actual: 35
45+32= predicted: 77 | actual: 77
99+99= predicted: 198 | actual: 198

Why This Matters for AI Reasoning

This example introduces several foundational ideas behind reasoning systems.

The model:

• Maintains sequential memory
• Processes symbolic structures
• Generates outputs autoregressively
• Learns internal representations
• Handles variable-length inputs

These concepts directly relate to:

• Language models
• Transformers
• Neural reasoning systems
• Chain-of-thought prompting
• Agent architectures

Limitations of This Model

This is still a very small neural system.

It does not truly “understand” arithmetic symbolically.

Limitations include:

• Small training range
• Limited generalization
• Character-level processing only
• No attention mechanism
• No transformer architecture

However, it demonstrates the core mechanics clearly.

Possible Extensions

You can extend this project into:

More Operations

• subtraction
• multiplication
• division

Longer Numbers

Train on 5-digit or 10-digit arithmetic.

Transformer Architecture

Replace GRUs with transformers.

Chain-of-Thought Outputs

Generate intermediate reasoning steps:

12+7=
12 plus 7
19

Token-Based Arithmetic

Move from characters to tokenizer-based inputs.

Final Thoughts

Sequence-based reasoning is one of the most important ideas in modern AI.

Even advanced reasoning models ultimately process information as ordered sequences.

This small PyTorch project demonstrates:

• sequence encoding
• recurrent memory
• autoregressive generation
• neural reasoning foundations

Understanding these mechanics makes larger AI systems far easier to understand.