Splitting Sysex for older synths

Home Forums General Programming Splitting Sysex for older synths

Viewing 20 posts - 1 through 20 (of 31 total)
  • Author
    Posts
  • #84239
    Anonymous
      • Topics: 2
      • Replies: 12
      • Total: 14

      I have a wavestation SR – and would like to use CTRLR to create an editor/librarian for it. So far I have found most things possible – excellent. But there seems to be a snag sending the syses dump of a bank to the wavestation. The bank is around 64K. Ctrlr is easily able to send it – and the monitor confirms it can send it all in one message. The screen on the wave station shows receiving sysex but then sticks on a timeout.
      My suspicion is the transmission is too much, too fast – and overruns the receiving buffer.

      I can’t find between any way from a lua programme to split the message into multiple buffers with a time delay between each successive buffer. Not even sure I could do it from the ctrlr C++ code. It looks as though it might even be missing from JUCE.

      Have I missed something – or is this a current limitation?

      #84240
      human fly
      Participant
        • Topics: 124
        • Replies: 1070
        • Total: 1194
        • ★★★★

        Goodweather and Possemo will be able to tell you all about that.
        am not familiar with it, but it will involve a timer callback.
        maybe you will find what you need in the Pro2 panel. or perhaps
        the matrix1000 panel.

        #84241
        Anonymous
          • Topics: 2
          • Replies: 12
          • Total: 14

          Thanks

          I appreciate the stuff about timers and callbacks – and whilst that gives me the periodic action I want – it doesn’t actually send sequence of midi messages. It appears sendMidiMessageNow expects to send a single message that starts with a F0 and ends with a F7 – it seems to have a problem with blocks that don’t end with a F7 and the next send doesn’t happen..

          #84242
          Possemo
          Participant
            • Topics: 14
            • Replies: 638
            • Total: 652
            • ★★★

            The synths I made panels for just have small sysex dumps so I never had to split messages. But maybe one day I will be faced with that problem too…

            I guess your dump is “one piece” beginning wiht F0 and ending wit F7

            IMHO the best way would be making a memoryblock out of your dump if not already done then selecting parts with getRange. I am not sure if it will work as a conform sysex dump needs the start and stop byte (F0/F7) but in general Ctrlr sends anything to MIDI-out when doing it by Lua scripts:

            DumpData = MemoryBlock()
            firstSplit = DumpData:getRange(0,256) — values explanation => (Starting Byte, Number of Bytes)
            secondSplit = DumpData:getRange(254,256) — values explanation => (Starting Byte, Number of Bytes)

            panel:sendMidiMessageNow(CtrlrMidiMessage(firstSplit))
            os.execute(sleep(20))
            panel:sendMidiMessageNow(CtrlrMidiMessage(secondSplit))

            Edit: ok. you already tried that… I was too late. How about the “Midi Global Delay”, have you tried it? It’s probably no solution either. This seems to be a real problem.

            • This reply was modified 5 years, 9 months ago by Possemo.
            #84244
            Anonymous
              • Topics: 2
              • Replies: 12
              • Total: 14

              Hi Possemo,

              Thanks. Tried that – the splitting works fine.. as does the loop. I’m pretty sure the first call to sendMidiMessage gets sent – the monitor confirms (and the synth says “Receiving Sysex” – but never sees the following packets – and finally fails with “Timeout”.
              My suspicion is that
              EITHER
              the wavestation can only cope with a few K in each lump, transferring each and assembling them elsewhere. A final confirmation of the checksum happens when the F7 is received..and controls whether the wavestation transfers it into the RAM bank and shows a success message following the transfer
              OR
              the wavestation wants the rate at which the data is sent to be much slower – and ctrlr outstrips the processing speed of the receiving circuitry.

              FWIW In the past I’ve had success with other software (Send SX BOME?) transmitting these banks to the wavestation. But I have no idea what went on inside the software.

              #84245
              Possemo
              Participant
                • Topics: 14
                • Replies: 638
                • Total: 652
                • ★★★

                Ok, so these sysex dumps are correct. It is strange as normally you cannot vary the speed MIDI data is transmitted. It is defined by the MIDI specifications (MIDI has a single baud rate: 31250 baud – taken form there: http://www.personal.kent.edu/~sbirch/Music_Production/MP-II/MIDI/midi_physical_layer.htm).

                Have you checked the data being sent with a MIDI monitor software like MIDIoX? Maybe the delay set with os.execute(sleep(20)) has to be right. It could be that the synth won’t wait forever for the data to continue.

                #84246
                Possemo
                Participant
                  • Topics: 14
                  • Replies: 638
                  • Total: 652
                  • ★★★

                  It tested it and indeed it does not work. sendMidiMessageNow wants the first byte to be f0 otherwise it won’t send it.
                  I will have a look at Sounddiver’s Wavestation panel (I guess it does have one). Sounddiver has showed me a lot of tricks in the past.

                  • This reply was modified 5 years, 9 months ago by Possemo.
                  #84249
                  Anonymous
                    • Topics: 2
                    • Replies: 12
                    • Total: 14

                    Hi,

                    Great test, your finding fits the symptoms I’ve observed. Initial packet starting with F0 sent – but no option to get anything afterwards sent.

                    Let me know if sound diver has any clues. I’ll have a look see at snoize – which has source code available.

                    On another tack I have been trying to find out whether I can find a way from LUA to execute sendMidiBuffer. This call appears to accept a set of messages. Don’t whether this tactic would be a viable workaround, or simply run into the same problem (if it actually is in JUCE).

                    Sorry about red herring of transmission speed – silly me..

                    Vernon

                    #84253
                    Possemo
                    Participant
                      • Topics: 14
                      • Replies: 638
                      • Total: 652
                      • ★★★

                      I have now looked into a wavestation dump (64k) with the help of a hex editor. This dump consists in fact of 3 sysex messages. So it is clear what you have to do. Seperate the messages and set a delay in-between. When sysex messages are sent without delay many synths/systems will make problems.

                      #84254
                      dasfaker
                      Keymaster
                        • Topics: 80
                        • Replies: 793
                        • Total: 873
                        • ★★★

                        The sysex file you are trying to send, has only one F0 and one F7 (one single block of data) or has several blocks? According to these links, it’s made of several blocks.
                        http://www.korgforums.com/forum/phpBB2/viewtopic.php?t=7550
                        https://www.soundonsound.com/techniques/korg-wavestation-tips-techniques

                        If that’s the case, you need to send them individually with a timer to allow a delay between them.

                        #84255
                        dasfaker
                        Keymaster
                          • Topics: 80
                          • Replies: 793
                          • Total: 873
                          • ★★★

                          You beat me Poss!!

                          #84256
                          Anonymous
                            • Topics: 2
                            • Replies: 12
                            • Total: 14

                            Hi Both,

                            I’ll try that and let you know.

                            Thanks

                            #84262
                            Possemo
                            Participant
                              • Topics: 14
                              • Replies: 638
                              • Total: 652
                              • ★★★

                              Ah. for once I was faster than dasfaker.. 🙂

                              I would prefer os.execute(sleep(value in milliseconds)) instead of a timer. sleep works fine – just be aware that the MIDI Monitor of Ctrlr will display wrong values. It looks like sleep has no effect but in fact when you look at it with MIDIOX you can see that it does work flawlessly.

                              #84366
                              Anonymous
                                • Topics: 2
                                • Replies: 12
                                • Total: 14

                                Hi,

                                Have managed to do some further tests. I succeeded in sending large sysex messages to the Wavestation SR using SX Send (Bome Software) having resurrected an old windows 2003 server machine.

                                Each Patch bank sysex message is 29K. The Performance Banks are 17K. and the wave sequences 17K (so 9 in total for the RAM banks of the SR). But the important lesson is how SX Send offers to send the message. SX Send offers a slider to adjust the speed of the message. I got it working sending by setting the slider to 1300 bytes/second. It counted the lines of 64 byte sysex (as shown on the screen) sent. That (as mentioned above) isn’t close to the 31,500 baud rate specified by the Midi standard. So SX Send would appear to be splitting the message into much smaller buffers and send these individually with a configurable pause between each message.

                                This fits with the way Snoize SYSEX librarian (a Mac program) appears to try and do it – and the source code refers to the MIDIOX practice of splitting the sysex message into 256 Byte buffers. This may have been driven by windows 16 bit nature..

                                I have yet to test sending these sysex messages using Snoize Sysex librarian which would be good – given that the source code can be inspected/changed – unlike SX Send which is a binary compiled app.

                                So earlier thoughts about splitting messages appear to be right. Ctrlr can’t yet split messages into sufficiently small lumps because it won’t send buffers that don’t start with F0. I think it’s a change to LUA/Ctrlr infrastructure that we need.

                                Cheers

                                #84367
                                m.tarenskeen
                                Participant
                                  • Topics: 30
                                  • Replies: 113
                                  • Total: 143
                                  • ★★

                                  Ah. for once I was faster than dasfaker.. ?

                                  I would prefer os.execute(sleep(value in milliseconds)) instead of a timer. sleep works fine – just be aware that the MIDI Monitor of Ctrlr will display wrong values. It looks like sleep has no effect but in fact when you look at it with MIDIOX you can see that it does work flawlessly.

                                  Thanks, I didn’t know that. I can use that in one of my homemade panels. For me the timer construction is difficult to grasp. Using os.execute(sleep()) is easier to understand. But I can imagine situations where sleep can not be used. For example when used in situations where a track is playing in a DAW simultaneously. Timing will be ruined then, or maybe the system will crash?

                                  #84368
                                  Possemo
                                  Participant
                                    • Topics: 14
                                    • Replies: 638
                                    • Total: 652
                                    • ★★★

                                    To vernonr: have you tried sending the dumps with a sleep command in-between? Until now it fixed all issues I encountered. I think synths with MIDI interfaces should be able to handle sysex data at the speed that meets MIDI specifications. Various MIDI-interfaces and quality of the MIDI cables may give different results.

                                    To m.tarenskeen: yea this is true, sleep is not a substitute for timers. There are many cases where you absolutely need a timer. Mainly because Ctrlr is single-threaded and therefore does not always do what you want when using Lua scripts.

                                    #84531
                                    m.tarenskeen
                                    Participant
                                      • Topics: 30
                                      • Replies: 113
                                      • Total: 143
                                      • ★★

                                      To m.tarenskeen: yea this is true, sleep is not a substitute for timers. There are many cases where you absolutely need a timer. Mainly because Ctrlr is single-threaded and therefore does not always do what you want when using Lua scripts.

                                      I have some questions about that os.execute stuff.
                                      I develop my panels mainly on a Linux system. Once in a while I start an old Windows laptop just to check compatibility issues. I don’t own an Apple computer.
                                      I guess (for example) os.execute(sleep(100)) executes a command that has to be available on the host operating system.
                                      Question one: shouldn’t it be os.execute("sleep(10)") (with quotes)?
                                      Question two: is the command working on Linux, Windows, and, OSX and working with the same syntax? My aim is to create panels that work on all these three OS’s

                                      #84535
                                      Possemo
                                      Participant
                                        • Topics: 14
                                        • Replies: 638
                                        • Total: 652
                                        • ★★★

                                        it is without quotes – like this: os.execute(sleep(dely)). It does work for windows and mac. Some time ago I tried Ubuntu for testing panels and… at least it does not throw an error but I have not tested if it really works. When I saw that you cannot make instances on linux and when I didn’t found a DAW host where the plugin does work I gave up on linux. Raspberry linux would be an interesting option but unfortunately there is no Ctrlr ARM-Linux port.

                                        #84551
                                        dnaldoog
                                        Participant
                                          • Topics: 4
                                          • Replies: 480
                                          • Total: 484
                                          • ★★

                                          m.tarenskeen – There’s an excellent explanation here by Goodweather on timers:

                                          basic timer setup

                                          I couldn’t get the concept of timers until I read the post carefully and realised a timer is a loop or rather a process that keeps running until told not to.

                                          If you set an external global variable as he does iCounter=0 it is picked up by the timer (which is a loop) – you can stop the timer when a condition is met; in this case an incremented variable.

                                          Within the timer you increment iCounter iCounter=iCounter+1if iCounter == 10 then timer:stopTimer(thisTimerId) end for example and the timer will return back to your code that triggered the timer.

                                          #114915
                                          Ethan Fawkes
                                          Participant
                                            • Topics: 0
                                            • Replies: 1
                                            • Total: 1

                                            Hello Vernonr,

                                            Have you had the possibility to complete your panel for the Wavestation SR?

                                          Viewing 20 posts - 1 through 20 (of 31 total)
                                          • The forum ‘Programming’ is closed to new topics and replies.
                                          There is currently 0 users and 36 guests online
                                          No users are currently active
                                          Forum Statistics
                                          Threads: 2,495, Posts: 17,374, Members: 77,605
                                          Most users ever online was 12 on January 22, 2019 3:47 pm
                                          Ctrlr