I have started recently a new project in which I am trying to visualize a neural network in 3D with Python, Blender and Tensorflow. This is a very interesting challenge and a great way to learn things that I didn’t know I needed to know. For example how to get access to the different layers in a neural network. How to be able to draw a fully connected layer with thousands of synapses in Blender without crashing it? How to calculate the angle between two vectors in 3D is needed to do a rotation and many more interesting challenges.
In this page you will be able to follow me on my journey, because I will record every single time I am coding on this project. I will put all the videos on this page, along with the timestamps, source code produced, and a summary of what I have achieved in each live coding session.
#1 Live Coding – Connecting two Neurons
In this coding session I, was able to work on the core components for connecting two dense layers of a fully connected Neural Network. It was hard work to find out the angles between two 3D vectors in order to work out the exact rotation of my synapses and in order to connect two neurons together with exact precision. At the end Stackoverflow came to the rescue!
Timestamps
0:00 Introduction – Visualising Tensorflow Neural Network
00:9:07 Flatten – Neuron Layer
00:9:51 Geometry Nodes 10:25 Particle Systems
00:11:10 Dense Layer
00:13:28 Neural Network with Array Modifier
00:17:55 working on connections between neurons – synapses
00:25:43 Creating connectNeurons function
00:31:47 Calculating distance between 2 points in 3D
00:40:13 bpy.ops.mesh.primitive_cylinder_add
00:45:00 Calculating the middle point between two points
1:00:00 Trying to get connectNeurons to work
2:03:00 I am sweating – 40 degrees in my room and my head spinning in geometry 101
2:39:00 Did I bite more than I can chew? I want to sleep…
2:40:06 It is late, and I want to sleep, Stackoverflow to the rescue?
2:42:00 I am trying to calculate the angle between two vectors
2:47:00 Mathematics at High School? I used to know all of this inside out.
2:58:58 What is that scribbling? 3:01:00 I fell asleep for a second??
3:01:47 Still trying to calculate that angle(turns out there were two!)
3:09:00 A viewer looking to help me out. At this stage, I can’t tell angles from straight lines.
3:15:00 A lifeline from StackOverflow – How to add a cube between 2 points in Blender
3:30:00 Seems like this solution will work!
Source Code
https://gist.github.com/armindocachada/88d3733d69a3318e0cde5c4514a8f390
#2 Live Coding – Connecting dense layers with Synapses
It was a very productive afternoon. I made good progress connecting two dense layers in a neural network using cylinders, which represent the synapses between neurons. There is a bug but it should be easy to fix!
Timestamps
0:00 Setup
2:00 Introduction
9:00 ArrayModifier in Blender
12:00 Explaining project
15;00 Creating connections between neurons using Cylinder
23:00 Function that calculates the position of each neuron
29:00 Reading documentation about ArrayModifier
36:00 Unexpected Blender Art
39:00 Experimenting with ArrayModifier
48:00 calculateNeuronGrid()
55:00 Visualising my algorithm using doodles
1:10:00 Fixing runtime and syntax errors in code
1:30:00 Testing function that connects neurons
2:56:00 Finally two fully connected layers! With a bug…
Source Code
https://gist.github.com/armindocachada/414f99127f0544f758970c103553e879
#3 Live Coding
Today I managed to do a bit more progress on visualizing fully connected layers in a neural network using Python and Blender. I fixed the bug I shared yesterday. The next stage will be to optimize my blender function so that it scales for thousands of neurons. The number of connections between neurons grows exponentially.
Source Code
https://gist.github.com/armindocachada/414f99127f0544f758970c103553e879
#4 Live Coding – Optimization, ArrayModifier and Materials
Optimised the creation of the connections between neurons. Instead of using a cylinder and using expensive bpy.ops operations, I replaced the cylinder with cuboids for synapses and used bpy from_pydata function to create cube from vertices, which is much faster. I added materials for the neurons and synapses for now these are quite random, and simply a placeholder for changing later.
Timestamps
0:00 Getting Started and plan
5:30 Adding a gap between adjacent neurons
7:46 Viewer suggests a video on using GANs inside Blender
9:46 Experiment with ArrayModifier
10:36 Where is the ArrayModifier???
12:21 Relative offset and constant offset
17:00 Enabling Python Tooltips
18:00 Modifying Python code to use ArrayModifier with both relative offset and constant offset
30:00 Showing how to subdivide object created with ArrayModifier into loose parts
36:00 Adding gap between neurons horizontally
57:54 Adding gap between neurons vertically
1:04:30 Replacing cylinders with cuboids for synapses
1:07:00 Defining cube using from_pydata
1:15:00 Defining createCube function
1:28:00 Testing bigger neural networks
1:39:00 adding materials – PrincipledBSDF
1:46:00 Adding material to cube
2:00:00 Adding materials to synapses
2:21:00 Visualising with materials
Source Code
https://gist.github.com/armindocachada/5d6edbf7b3ad6cb2c22948c289e6af34
#5 Live Coding – Installing Tensorflow inside Blender
In this coding session I spend most of my time installing Tensorflow and other Python libraries that I need inside Blender, fixing memory hogging issues with Tensorflow, refactoring code, modularizing my Python code, making changes in modules without refreshing and I also managed to work on the visualisation of the neural network with the two fully connected dense layers.
Timestamps
0:00 Introduction and plan for today’s live coding session
5:00 Draft version of neural network in 3D with Blender 2.93
7:30 Stream lagging because Tensorflow hogging my GPU ( I will fix this later)
14:30 Installing Tensorflow in Blender 3.2
25:00 Installing Tensorflow, Pillow, and other Libs with pip and virtualenv
47:00 Adding Python libraries with sys.path.append
50:00 Testing installation of Tensorflow inside Blender
57:00 Visualising neural network
1:00:00 Refactoring code into separate Python Modules
1:08:00 Tensorflow is hogging y GPU memory again
1:14:00 tf.config.experimental.set_memory_growth
1:22:00 Changing code in modules without restarting Blender
1:26:00 Trying System->Reload
1:30:00 Found solution to change code in modules without restarting Blender
1:38:00 More Refactoring
1:45:00 How to import function from python module ( visualise() )
1:48:00 Tensorflow hogging my GPU again ( I will eventually fix it)
2:02:00 More refactoring
2:12:00 Checking memory usage in my RTX 3070 GPU
2:20:00 Finally solved Tensorflow GPU memory hog issue
2:27:00 Merging code for visualising neural network
2:33:00 I get StructRNA of Material has been removed error?!
2:36:00 I fix the error => I kept references to deleted materials
2:44:00 Fixing more errors
3:02:00 Final visualisation of the neural network(for the evening)
Source Code
Coming soon!
