The variety in music has always fascinated me and I was curious if I could create an algorithm that could generate melodies based on existing music. After the initial phase in which I learned the basics of MATLAB, I realized that music theory would be an essential part of this project. So after reading about music theory, the Math behind sound waves and analyzing songs and melody structures, I created an algorithm that actually worked. All choices the algorithm made were based on statistics. I had exceeded my own expectations, since the generated melodies were in my opinion quite decent for a non-human creator. However, only a small portion of the melodies were pleasant to listen to.

The next question arose; How could I increase the success rate of pleasant melodies? There were two possible options I had in mind:

• Improve the algorithm iteratively, by listening to many melodies that are generated and compare them to existing melodies in the music industry.
• Be able to classify the generated melodies from 'bad' to 'good' and filter out the bad ones.

To challenge myself even more, I chose the second solution. I wanted people to rate the melodies and use their opinions with a neural network to be able to classify what melodies were considered good. There were two big challenges ahead:

• Building a website, which I had never done before. On this website people could listen and rate the melodies online.
• Coding a neural network.

I did both. I built a website from scratch with HTML, CSS and JavaScript and started posting it on all the places I could think of. In just 2 weeks, 10.000 melodies were rated at least 5 times by more than 5000 people. It was fun!

Next up was the neural network. I managed to create one from scratch in MATLAB with the help of a friend. However, the classifying gave inconsistent results. It could have different reasons; lack of experience in determining the variables for the neural network, the inconsistency in ratings from people or the lack of data. Whatever it was, I chose to leave Artificial intelligence out of it and went back to improving the algorithm based on statistics.

Rating Melodies

The first version of the JudgeMySound website was focused on collecting data for the neural network. I generated 10.000 melodies with the melody generating algorithm and put them online. People could listen to and rate them.

See the video: Click Here.

After the failed attempt at the neural network, I first started working on improving the success rate of generated melodies. I decided to change the algorithm from generating random melodies to melodies that were based on existing melodies. Successful songs are successful for a reason. Their melodies must be really good. The algorithm retrieved data and statistics from existing songs and used it to generate new ones. On the side I started thinking about possible business models with the current algorithm. How could generated melodies, that varied from good to bad, be used in any way? I could imagine that when someone creates a song, the initial version will never be perfect. Therefore, music producers were my next target.

I started asking around in my network if people knew any music producers and I found around 20 music producers that were willing to show their workflow and discuss how my algorithm could contribute or help them in their process. Over a couple of weeks and many meetings further I could conclude that many struggled to get anything worthy from scratch. Or at least it took quite some time to get anything to work with. Some even said they would pay for such services to spark their creativity.

During the same time, I was studying Mechanical Engineering at the TU Delft. In the third year, students have to get a Minor degree. A Minor degree refers to a supplemental second discipline of study and concentration that often complements the major. I managed to setup a custom Minor degree for myself in which I followed multiple programming courses and wrote a thesis on how the algorithm worked. In the meantime I got assistance on how to improve the algorithm and applied the changes directly. In February of 2017 I demonstrated the application that I wrote in Python and presented the thesis to my mentors.

Read the whole report here: Minor Thesis - JudgeMySound.

Melody inspiration tool for music producers

In this version music producers could generate 8 bar melodies and regenerate smaller parts. The melodies were separated into three sections; Chords, Bass and Melody. The chords were generated and chosen at first, after which the bass and melody were generated based on the chords that were chosen. Every note in every section could be edited. Notes could also be created and deleted.

See the video: Click Here.

After finishing my Minor thesis and Python application, I returned to some of the previous 20 music producers. They were impressed, but were also also major issues:

• They were unable to use it at home, since the Python application was only locally available on my computer.
• Sounds are a huge part of determining if a melody is catchy and usable. The bleep soundwaves from the application had to be replaced for the application to be useful.

The solution to both; rebuild the application for the web. Once it is on the web, it will be available from anywhere. After some testing in JavaScript, I managed to get some custom sounds working.

Since the algorithm was rewritten from MATLAB to Python when writing the Python application, Django was the ideal backend. After a steep learning curve, I managed to get the application working after months of hard work.

The first thing I did was contacting the music producers. They were impressed, again. However, they were not willing to pay for it... WHAT?! I was shocked. How, after so much work and them saying they would pay for the final product, were they not willing to pay?

After the initial shock I started asking more questions and came to the conclusion that music producers were not the right target audience from the start. Most music producers are proud of their work and once they feel a computer has created a melody or has contributed to it, it does not feel like it is their creation anymore. What a bummer; I was devastated.

Endless Inspiration

The final product of JudgeMySound. Music producers can import melodies that they like and the algorithm will create melodies based on the imported melodies. These imported songs can be (de)selected. The algorithm needs at least three existing melodies to be able to function. Custom sounds are implemented and multiple new features, like a metronome, are included to improve the experience for the user.

See the video: Click Here.

After the failed attempt at helping music producers, I kept replaying the previous two years in my head. What went wrong, how could I have done it differently?

People learn the most when they fail and I definitely felt that. I learned a lot, but I still had to learn even more. By chance, I spoke with someone at the rowing club. She studied Industrial Design and was doing her Masters. She asked me to join a course at the Industrial Design faculty called 'Build your Start-up'. I happily accepted. A friend of mine was also interested and I went from a one-man team to a team of three.

The course was an eye opener. Designers have such a different way of thinking in comparison to engineers. Engineers want to fix things. However, designers put their focus on the customer before doing anything. They seek for a problem and validate it before creating a solution.

The next step was to do the same kind of strategy with JudgeMySound. The only difference was that we already had a product without a problem it solved. Maybe we could find a problem in society where a modified version of JudgeMySound would be of meaning. And yes, there was. In today's society, the focus on prestige in primary schools is much higher than it used to be. Children need to be at a certain educational level, which is determined by the government. Therefore, there is much less time for music, drama and other artistic subjects. On top of that, the pressure on teachers due to the amount of administration is way too high. Whenever there is some free time in the classroom, the teacher decides what the children will do. What we found out is that most teachers do not feel comfortable singing or making music in the classroom. Therefore, music is not getting the attention it deserves, since music is really important in the development of a child's brain.

Problem found!

Over a weekend, I changed the layout of JudgeMySound to a more child friendly interface and we started testing immediately. The kids were excited and did not want to stop playing with it.

The next step was to make it profitable. The business model would be providing workshops at schools; learning the kids how to use the application without interference of the teacher. This way, children could develop their abilities in regards to music while the teacher could focus on other things. Whenever we called a school, they were willing to provide a class for an hour to validate our product. Everyone was enthusiastic about it and we saw in the web statistics that around 70% of the children came back to the platform after school.

However...

Although I have given multiple paid workshops, most schools could not afford to spend money on external workshops. They were already on a budget and had a tough time getting around.

After many attempts we decided to leave it for what it was. I have done everything I could and it was an amazing journey. After many failed attempts, I was ready for something new where I could use all the gained knowledge to make something successful!

Online song creation tool for kids

With Tringa, children could make a whole song interactively. They first had to listen to generated chords, basslines and melodies. After choosing those, they could add, change or delete notes. They could change sounds, BPM (beats per minute) and add drums.

See the video: Click Here.