A text-based instrument and ensemble language for better AI assistance

D. Vyd · 2023-03-21T15:58:40+00:00

There was an error rendering this rich post.

The visual programming interface for Reaktor is both its greatest strength and greatest weakness. I'd like to start a discussion about a feature request: adding a text-based language and file format as well. This would exist in parallel to the visual interface. Users could decide how they want to view and edit in Reaktor. An ensemble built using the text-based language could still be edited through the GUI and vice versa.

This would serve two major purposes:

In the long term, such a language and file format would allow users to receive help and ideas more easily through AI tools like ChatGPT because instead of cumbersome GUI instructions, the AI tool would just offer code, like it does for any other programming language (Python, Java, C#, etc). Of course, the AI would need to be trained on thousands, perhaps millions of examples first, but these could be generated from the vast library of ensembles already available.
Such a language and file format would allow programmers to immediately use their language of choice to generate Reaktor instruments and ensembles, including using loops to easily generate repetitive code without copy-pasting and then editing the copy.

Right now, ChatGPT can code well in text-based music notation like ABC and poorly in synthesis languages like CSOUND and Supercollider. This will improve over time. Reaktor should be part of that process. What do people think?

-d.vyd

Find more posts tagged with

Reaktor Building

Feature Request

Reaktor User Library

Comments

colB

There are hundreds of existing text based languages with lots of libraries and support. Some of them are specifically designed for developing audio, or have add on frameworks or libraries for audio development. Why not just use one of those?

------------------------

Also, I would guess that the reason why ChatGPT isn't so great a coding in CSound and Supercollider, is because it really doesn't and can't understand sound or dsp or synthesis, it just processes language. So the more levels of abstraction of understanding required, the more an AI would have to have a true understanding, rather than just guessing which word or phrase comes next based on glorified probability networks...

Even in Reaktor, there is plenty of code that you wouldn't be able to reverse engineer and understand unless you already had at least some basic understanding of the technique being implemented.

e.g. some filters are just a few arithmetic operation with a list of coefficients. The relationship between the coefficients is crucial, must be precise, and has been generated by some very complicated maths and probably software tools, but in the DSP code it's just a short list of numbers. There is literally no way an AI like ChatGPT could see a bunch of example lists of those coefficients, infer the rules behind them, and then generate a new set of coefficients that applies to a new/novel context.

Similarly with other DSP techniques, various anti-aliasing approaches, ZDF filters, basically any of the cool stuff. There is too much work that is not in the code for an understanding to be inferred from that code by an AI that is just a language processor.

So I think ChatGPT would be good for the simple stuff, but then the point of Reaktor is that most of the simple stuff is already built in - we don't have to write code to interface with the Operating System, we don't have to explicitly interact with memory management, we don't have to scratch build GUI code using widget libraries, we don't have to handle VST buffer chunks...

I guess I'm saying that the things that ChatGPT would be able to help us with if there was a text layer in Reaktor, are the things that are already trivial to do in Reaktor. The difficult stuff like product design and complex DSP, are exactly the things it will not be good at anyway. So what would be the point?

-------------------------

It's also a dangerous road to go down!

...imagine what happens when there's an AI that IS capable, and if NI and other platform vendors start redesigning everything to suit that AI...

The environments slowly evolve to be more and more AI friendly, and less and less human friendly... to the point where either all successful development tools incorporate licenced AI access engines ...or the AI company owns all the other companies...

The price of using the AI goes up and up

Then at some future point folk realise that they literally can't write software without paying the folk who own the AI, because all the tools are designed around the AI, not human creators.

Or worse, the ability to interact directly with the tools disappears, and we are left with an Amazon style shopfront, and just tell it what we want and it sells us some autogenerated application, that doesn't really do what we need, but we are stuck with it, because their competitors went out of business...

Half of school study becomes how to formulate requests to AI and get what you actually want... and the only jobs left are hairdressers, because the only thing that folk still absolutely refuse to let robots do is wield sharp scissor blades near their eyes and ears...

Yeah, have fun trying to build your own synths/samplers/FX in that future! ...and remember - all that happened because you wanted NI to make Reaktor more AI friendly... shame on you! lol

KoaN

https://community.native-instruments.com/discussion/comment/59340#Comment_59340

That last bit felt like a futuristic dystopian movie ,interesting perspective of the future 😎

If that happens all the old timers here will have to fight to save the human race from this terrifying Reaktor version...7 maybe? or 8? Considering the speed developement of Reaktor...probably 7.

🤖

Gee_Flat

https://community.native-instruments.com/discussion/comment/59400#Comment_59400

It's no movie. In 2024. AI operated killer mosquitos, drones and robots. No more websites or search engines. Just government controlled information operated by AI.

Utopia! 😎

After reading a discussion here yesterday I visited a website called 'Beatport' and it blew my mind. Not only at what I heard, but how out of touch I am with a subhuman, subculture. The current dystopia.

AI might actually be welcome there. It might inject some creativity into music that's been debased by a de-humanized culture of sheep that all use the same drum beat at the same tempo.

One thing is for sure. Opinionated people like me will be erased by a social credit score controlled by AI.

D. Vyd

@colB , ChatGPT is only as good as its training data. There is a lot of good Python programs on the Internet, so it is good at programming in Python. There are several orders of magnitude fewer examples of CSOUND and Supercollider code, so it is orders of magnitude worse at programming in those languages. To a language model, code is just another language.

Have you tried ChatGPT? I never thought a correlation-based model could ever appear intelligent. It was never fed a giant ontology or symbolic logic database of human knowledge, but responds as-if it understands what it is saying. Often, that is good enough.

However, these tools sometimes "hallucinate", meaning they speak convincingly and with authority--but are 100% wrong. When asked to double check a previous statement they usually apologize and say it was incorrect. Sometimes though they stand behind the erroneous statement. This is a known flaw and area of active research. I've noticed that the newest version of ChatGPT (GPT4) also knows its strength and weaknesses. It knows that it is better at Python code than C# code.

I believe AI will enter ("infiltrate" if unwelcomed) every aspect of our lives and become a co-producer at work and with our hobbies. If Reaktor is not compatible with AI, its user base will dwindle over time. Most people would prefer making or modifying an ensemble in hours or days rather than weeks or months. ChatGPT can already help teach users how to code by explaining the code it writes. It could eventually do the same for DSP.

I'd prefer to see Reaktor's user base grow because I like Reaktor. There are image-based AI's but I doubt those will work well with Reaktor because--although Reaktor is a visual programming language--ensembles are not pictures, they are code.

@KoaN , colB's vision of the future is not far fetched. I share those concerns about dumbing down jobs and life in general. I work at a university and professors don't know what to do. One approach is to hold in-class tests and prevent students from consulting AI. Another approach is to make the tests and projects much harder. I'm not sure what the answer is. But...I'm sure AI is here.

-d. vyd

colB

https://community.native-instruments.com/discussion/comment/59466#Comment_59466

I share those concerns about dumbing down jobs and life in general. I work at a university and professors don't know what to do. One approach is to hold in-class tests and prevent students from consulting AI. Another approach is to make the tests and projects much harder. I'm not sure what the answer is. But...I'm sure AI is here.

I think the Universities first need to work together to develop their own AI tools, so they (and their students) are not beholden to monopolistic corporate entities, then they need to completely redesign all their curricula to positively incorporate those AI tools in the same way other technological developments have been incorporated over time. Good luck with that though - there is so much hubris and arrogance in academia, and it is generally very slow to react to change, which has historically been a good thing, but the AI train is going to hit fast and hard. So it's all aboard sharpish, or get left behind and suffer the consequences!

...and remember - it doesn't feel pity, or remorse, or fear and it absolutely will not stop😁

Gee_Flat

https://community.native-instruments.com/discussion/comment/59492#Comment_59492

It'll never stop? I knew Jordan Peterson was an AI robot! 💡

salamanderanagram

NI can't even be bothered to back up a forum. we've been asking for iteration in core for nearly 2 decades. there are bugs in reaktor primary that i reported 10 years ago that straight up break the functionality of multiple modules, that are still there and will never be fixed.

i mention all of these things because if you think NI is going to invest in something like AI assisted programming in reaktor, you haven't been paying attention to the direction of this company.

the second point i would make is that LLMs have literally no understanding of anything. they are great at sounding authoritative, but they often cite non-existent papers or articles, make up "facts" and make other errors that render them basically unusable for a task such as this. certainly an LLM is not capable of creating a new, novel algorithm for anything. so at the absolute best, they'll basically plagiarize existing content. at worst, they'll spout a bunch of drivel that sounds right but is actually garbage, wasting your time and effort.

i asked chatGPT to suggest an algorithm for generating bandlimited oscillators and it suggested oversampling, which is the worst method i know for removing aliasing (and the best at wasting CPU).

lastly, if NI were capable of making something like this, which they aren't, there would be no reason for them to share it with us. it would make much more sense for them to use it as an internal tool.

"...and remember - it doesn't feel pity, or remorse, or fear and it absolutely will not stop😁"

my favorite movie since i was a little kid.

colB

https://community.native-instruments.com/discussion/comment/59466#Comment_59466

ChatGPT is only as good as its training data. There is a lot of good Python programs on the Internet, so it is good at programming in Python. There are several orders of magnitude fewer examples of CSOUND and Supercollider code, so it is orders of magnitude worse at programming in those languages. To a language model, code is just another language.

I don't think its so much because there are more Python programs, more that folk are using Python for the type of coding where the code really does express the problem and the solution, so if you look at enough programs to grok the language, you can then reproduce similar code - that works!

My point is that with DSP, often the code does not express the solution space at all, all the code has is abstract side effects of the solution space. The actual mechanisms involved are very opaque, buried under layers of abstraction and magic numbers. In that case an AI designed to process language cannot ever break through that barrier.

I'm not suggesting that general AI cannot solve that problem, just that a language processing AI cannot, because the solution mechanism is not expressed in the language. If you showed chatGPT the pages of math that were used to derive the resulting code, then maybe, but that stuff is rarely if ever available alongside the code it was used to generate, so the required connections still don't exist without a deeper understanding beyond the language.

Yes, I have, and I am very impressed, but I can also see where the limits of that approach will be, training data is one, but also the fact that it doesn't 'understand' any abstractions that are not expressed by the language. It is amazing that it can 'understand' implied meaning, but that is still something that is being expressed, just indirectly - it's still encoded in the language. But you don't even get implied meaning in more advanced DSP, and similarly abstract code. It's just meaningless unless you already have an understanding.

Maybe if you give it hints about what area of mathematics to use to derive specific parts of the code... ?

However, these tools sometimes "hallucinate", meaning they speak convincingly and with authority--but are 100% wrong.

I think that's likely a symptom of what I'm talking about. The AI understands the language well, but can only 'understand' the purpose when that purpose is encoded in the language - it can't infer the 'why' or the 'how' when all that is encoded is opaque instructions.. the 'do'

One of the reasons Core was developed (sinc modular) was because the visual graph representation more intuitively represents the meaning of the DSP code, the sequence and flow of the structure, that's something that does not work nearly as well in text based languages. It enables the creation of effective 'code' that does express some of those abstractions... however, that requires extra effort an results in somewhat less efficient code. I could see something like chatGPT learning how to use the ZDF framework (assuming it could process diagramatic languages), but it wouldn't so easily get the optimised versions where the process is abstracted away in some math that was solved elsewhere and is not expressed directly in the resulting code

I have no doubt they will get there, but I think it would need to be a more general purpose AI, with the ability to combine language processing with more abstract 'reasoning'... so a mix of chatGPT, DALII and Alpha Zero... with some other new ****** mixed in for good measure... or something :)

KoaN

About the future,for sure anything is possible.I was just smiling at some of the things Colb said.

In this world where most powers(or aspiring ones) try to get to some kind of monopoly on something...who knows how it will turn out.

I don't think it will turn apocalyptic but it can certainly create issues and hinder instead of helping.

Look at this guys...

https://www.youtube.com/watch?v=EggmA0g71xA

It is starting sooner than i thought with music and sounds.

It does sound lofi and low quality but the point here is the Ai actually generates the sound,makes it's own interpretation,no samples here or midi files etc.You can here the voices being a bit weird...lol.

But wow...that is the start,in 1-2 years the quality will go up exponentially and you will get any type of sounds,music possible and unpredictable surprises...i can easily envisionned out of this world sounds,music when it gets to a good quality of rendering.

As a guy who spent his own life trying to create new sounds it does feel threatening,but that's my ego.

I guess my fear is a lot of people (those who studied these things less or not at all) will even have less discernement for sounds,music and will be very hard to impressed...thinking they have heard it all.

Or some will get arrogant,pretentious thinking they are great artists while most of their work is done by AI.

This already exist at some level,some people think any electronic sound,music is just like a Dj pressing a few buttons and voila.I feel AI will make that line even more blurry for most people.

Pionners and creativity will always exist but the tools,way to do it will completely change for sure.

ANDREW221231

the way I see it, all the borders between different programming languages are dissolving. I think it's unlikely we'll see reaktor adapt code like supercollider CSOUND etc, and I think it's probably better to use those if you want something other than a visual programming language

Imo something more exciting is the ability to take code that is beyond my ability to understand and have chatgpt et all render it into something I can understand well enough to implement in a visual programming language

MvKeinen

I'd like the event table and core table arrays to have access to some spreadsheet format.

Gee_Flat

https://community.native-instruments.com/discussion/comment/59434#Comment_59434

That's right... UTOPIA!

I"m not much on coincidences ... Looks like Beatport has a new product to create more of, well you know 😎 the music I love hahaha. You guys crack me up!

https://www.native-instruments.com/en/products/komplete/play-series/utopia/

colB

Been thinking more about this.

So here's a thought :)

Rather than adding a new layer of text based coding to Reaktor, which I think would be a terrible idea for so many reasons that I've listed numerous times in the past. what about an update that uses a 'human readable' markup language to store core cells and core macros?

This would allow chatGPT to process core based Reaktor stuff, because it's a readable text language format, but it has another potentially huge additional benefit... assuming NI would go for it which I think is unlikely :)

The big idea is that NI would make core an open, standardised language. Not make Reaktor open source, and not release any compiler code or what have you. But allow other vendors to create their own visual platforms that can read and write the same markup language and develop their own compiler tech to implement it. There would have to be some major work on language specifications and definitions and suchlike, but there would be benefits all round.

While we might benefit from third party stuff, like maybe linux, or ios, or android apps that can process core, it would also future proof our existing Reaktor core skills. This is a two way, its great for us, but its also great for NI and Reaktor, because it takes away the fear that newbies have of investing money and time in a platform that looks like its on the way out - a narrative is pervasive in some areas within the wider digital audio community.

It might make Reaktor much more attractive in academia too, where it seems things like Max and Pure Data are more common.

It would also open up the options for someone to create translators to turn Reaktor core into c++, or c++ into core... then you would have the much requested text based Reaktor coding by the backdoor ;)

Quick links

Unanswered questions