Jose Antonio Martin H.

Jose Antonio Martin H. PhD

Software Tools for Reinforcement Learning,
Artificial Neural Networks and Robotics (Matlab and Python)

Neural Networks and other utilities:

Python numpy  ndlinspace,  the n-dimensional linspace function:
Python Code of the n-dimensional linspace function nd-linspace (python and numpy)
Simple and very useful Multilayer Perceptron Neural Networks with Back Propagation training:
Python Code   (pure python)
Python Code  (python with numpy - fast for big networks)  Xbpnn.PY
Java Code  (pure java implementation)
Matlab simple and nice multilayer perceptron (MLP) with back-propagation training (pure Maltab/Octave implementation).  Matlab BackPropagation
This implementation is specially designed for neuro-evolution since all the weights are represented in a vector which is then automatically decoded in the evaluate function.   
Simple and very useful Kohonen's style Self Organized Maps SOM  Neural Networks:
Pure python Self Organized Map SOM
Pure python Fuzzy Gaussian Kernel Self Organized Map OSOM  (you must install Matplotlib only to see the example)
An Evolutionary Neural Network (Multilayer perceptron) with Differential Evolution Training Algorithm:
Python Code (pure python), please download both two files which are needed by the Neural Net.
This Network performs better than back propagation. Also it is implemented saving and reading weights to/from file, Enjoy it.

Reinforcement Learning:

Competition Records for JAMH Team in the Reinforcement Learning Competition
Reinforcement Learning Competition Results Event - problem - domain
1th First place Poly-Athlon 2008
2nd Second place Mountain-Car 2008
1th First place Helicopter Hovering 2009
2nd Second place Poly-Athlon 2009
A Reinforcement Learning Environment in Python: (QLearning and SARSA) Version 1.0
Download the Package RLearning for python
Also, a win32 installer is provided RLearning-1.0.0.win32.exe
It includes as examples a Mountain Car Problem  and Cart Pole Control Problem: 
Some pictures of the python implementation
Mountain Car CartPole
Python Mountain Car example Python Cart-pole example
A Reinforcement Learning Environment in Python: (NN, kNN-TD and Exa) Version 2.0 (22 may 2010)
Download the Package FAReinforcement for python: FAReinforcement
It includes as examples: Acrobot,  Mountain Car,  discrete and continuous Cart Pole and a predator/protector/prey game. You should install Python+Vpython+Numpy+SciPy
note: these are High Quality/Performance Reinforcement Learning implementations! do not think they are simple software just because they are public and free! I used this same software in the Reinforcement Learning Competitions and I have won!.
A Reinforcement Learning Environment in Matlab: (QLearning and SARSA)
Please note that in some versions of Matlab you should delete some empty parenthesis in order avoid some errors.
This should not happen with Matlab release from version 7. This code is a simple implementation of the SARSA Reinforcement Learning algorithm without eligibility traces, but you can easily extend it and add more features due to the simplicity and modularity of this implementation. Enjoy it!
Matlab SARSA implementation of the Mountain Car Problem SARSA Mountain
Matlab SARSA implemetation of the Cart Pole Control Problem SARSA
Matlab SARSA implementation of the Acrobot Control Problem SARSA
Matlab Multi-Agent SARSA implementation of a Three-Link Planar Robot SARSA
Matlab Multi-Agent SARSA implementation of a SCARA 3D Robot
Matlab Dyna-Q  implementation of Dyna Maze World DynaMaze.rar
Matlab SARSA() (sarsa-lambda) eligibility traces implementation for the Mountain Car Problem: SARSA lambda (eligibility traces) Mountain Car
Some pictures of the Matlab implementations:
Mountain Car: Cart Pole:
Matlab Mountain Car example

Matlab Cart Pole example

  Three-link Planar Robot:


  Matlab Three link planar robot example

Matlab Acrobot example 

Online Planning with  Dyna-Q Dyna Maze  
Matlab implementation of Dyna-Q Dyna Maze of Sutton & Barto  
Some more advanced algorithms:
Matlab kNN-TD(λ) implementation for the Mountain Car Problem: kNN-TD Mountain Car
Please note that this is a Matlab implementation, not the competition one (originally in python), and is made for academic purposes so it is not optimized for performance or software quality design. It is just an "easy to understand and easy to try" implementation. comments are welcome!.  
Matlab Ex<a> implementation for the continuous Cart-Pole Problem:
(updated 7/5/2010)
José Antonio Martin H. & Javier de Lope (2009), "Ex<a> : An Effective Algorithm for Continuous Actions Reinforcement Learning Problems" , In Proceedings of 35th Annual Conference of the IEEE Industrial Electronics Society (IECON 2009) . Oporto, Portugal. November 2009., pp. 2063 - 2068. IEEE.

[DOI] [URL] [IECON 2009]

Matlab Dyna-H implementation for path finding in a Maze problem: Dyna-H.rar
(updated 12/2/2011)
Matilde Santos, José Antonio Martin H., Victoria Lopez: Dyna-H: a heuristic planning reinforcement learning algorithm applied to role-playing-game strategy decision systems (2011), Knowledge based systems, Elsevier  
Dyna-H: Heuristic planning Reinforcement Learning Algorithm
Dyna-H heuristic planning reinforcement learning
Comparison between Q-Learning, Dyna-Q and Dyna-H
Partially Observable Markov Decision Processes:  
A collection of experiments to prove the hypothesis of the reduction of POMDPs to MDP by means of the use of Internal Clocks. I am sorry for not having at this time more theoretical material at hand but you can write me if you want to talk about or even better, join the rl-list at Google.
Matlab SARSA Iclock (sarsa plus internal clock) solution to Altered POMDP Mountain Car Problem: SARSA ICLOCK Mountain Car
Altered POMDP Mountain Car Problem. The Agent cannot see the position of the car but only its speed !!!. )  
Mountain Car internal clock experiment:  
Classifier System (XCS) in python:  1.0
A collection of neuro-evolution experiments in robotics  (CMA-ES and NEAT):
Matlab implementation of neuro-evolution learning for robot control:
NeuroEvolution Controll of three link planar robot NeuroEvolution Controll of a SCARA robot
Planar Robot SCARA Robot
This software is part of a research paper on neuro-evolutionary methods for multi-link robots, such as the three link planar robot and the SCARA robot.
Revised: .  
Copyright © José Antonio Martín Hernández.
eXTReMe Tracker