Splitting Sysex for older synths

[Possemo]

I don’t know if vernonr is still working on it.

I have to say that I was wrong on that topic. MIDI specifications do not clearly define a fixed transfer-speed. Old interfaces (RS-232 type) usually transferred data at 31250 bps. Todays usb-interfaces send data much faster which is a problem for old gear. Their old interfaces aren’t able to handle it. I did notice this now as I am working on a Korg 05R/W panel. A Korg bank-dump consists of one big sysex message (starting with F0 and ending with F7). No problem for Ctrlr to receive such big messages but it cannot send them at a throttled speed required for these 90’ies devices. I guess this is due to the Juce MIDI class not allowing speed manipulations. Like MIDIOX it should be able to insert a delay between MIDI transfer-blocks. AFAIK this is not the case so there is probably no way to add this feature in Ctrlr.

The solution for this is to send a bank by sending all patches one-by-one with a delay in-between of about 10 to 100ms. As long as one patch doesn’t execeed ~ 1KB it should be ok. If you get bigger patches, like e.g. from a Kawai K5000 you’d probably better forget about creating a comprehensive panel with Ctrlr.

[Anonymous]

Thanks for the mention – Sadly I’m haven’t actively worked on it for a good length of time.

FWIW I came to the conclusion (as Possemo notes) that mods are needed to the underlying software – can’t remember whether it was JUCE or even the libraries underneath that. I didn’t have the will/time at that stage to build the dev kits for all that stuff and start working the C/C++ necessary to do the task. There would be a good case for enhancing the MIDI send capability to allow for timing etc. and removing the restriction on what you can send – just needs to good amount of time and skill to achieve it.

Cheers

[Anonymous]

Hi there,
Just to say (having had the prompt) I have picked up the issue again…
I’m looking at what mods are required to JUCE that would enable a large sysex message to be split up and sent in a number of smaller messages to an older synth.
Please post if you have any detailed information on what the constraints on an older synth might be..
Thanks

[EnzoF04]

Vernonr,

Very nice. In my particular situation I need an method of checking the incoming sysex data real time.

When the panel (or lower processes) detect a specific byte (combination) in the incoming sysex string it is receiving from the hardware, the panel needs to send an acknowledge byte to the hardware back. When this not happens the hardware stops sending data to the panel. There is a certain time span where in between the panel needs to respond. Otherwise the hardware ‘forgets’ what part it was sending. The first sysex data string the panel is receiving from the hardware has a byte that tells how many ‘packages’ of sysex data will be send separately.

Hope this is helpful.

Best regards, if i can be of some sort of help, let me know!

[Anonymous]

Hi,
Brief update..
We ( I & Possemo) were right. The problem appears to be in JUCE and its midi transmit code. JUCE code currently sends the entire buffer in one go – which completely overwhelms an older synth.

I am experimenting with a small modification to JUCE that enables the sysex message I’m trying to send (211K) to be split into smaller blocks (I’m currently trying 256 bytes each) with a time delay ( around 200ms) between each block. After a small battle I’ve got both JUCE and the underlying ALSA (in linux) compiled with debug and linked so I can step through line by line if necessary and uncover what’s happening.

I can receive on an old windows machine with either BOME SX or MIDIOX.

Currently I’m getting about 80% of the information through. I’m believe there will be an explanation for this, and the problem is soluble.

My hope is to achieve a modification to the JUCE interface that is usable from Ctrlr / Other and allows the control panel developer to configure JUCE so that it can send big sysex messages to old synths.

After I’ve got the output path (computer to synth) working I will test the input path. I can see a potential issue that the JUCE code might get “bored” waiting for all the sysex information to be sent. @EnzoF04’s problem is even more subtle than this – I’m not sure how to test/mimic this without the kit?

Whilst this is positive – I don’t have a timescale for completing this – and don’t know what success I may have in persuading those responsible for JUCE of the need to make mods in their “standard” library. It may be that Ctrlr would have to make a patch to JUCE for its use.

Cheers

[Anonymous]

Positive Update.
I have got the output path through JUCE working (having made some slight mods) to another computer. I have sent and received without error a 211,559 byte SYSEX in lumps of 128 bytes with an approximate 125ms delay between – which were recorded on BOME SendSX / MIDIOX as arriving at around 1KB/second. For info I’m doing the transmit on Linux – and the receive on an old Windows machine. Since these bits of software actually sent/received info from the Korg wavestation this suggests the actual path to the synth will work. So final steps on this – are tests with real synth – and creating an interface that would be callable from LUA

Now looking at Input Path for Big Sysex.
To explain – I’m working on finding the solution using Linux. This means
CRTLR uses LUA script which relies on the JUCE library. I believe CTRLR uses this “stack” on all platforms (Mac, PC & Linux) – although the compilation differs. But beneath that the situation varies. On linux JUCE sits on top of the ALSA libraries.

I have managed to receive the 211,559 byte SYSEX back from BOME SendSX using the amidi utility (uses the ALSA libraries) and dump to file. So I suspect any problems in making large Sysex dumps available to Ctrlr will be in JUCE.

I don’t know in which area of the code the solution to @EnzoF04’s problem with AKAI sampler might lie yet…

Cheers

[Possemo]

Great, that sounds promising. A big challenge will be to compile Ctrlr with the changes made. Some time ago I tried to compile the standalone-windows-branch without success. Compiling the plugin must be even harder as it needs even more 3rd party components. Meanwhile I am not interested in the standalone version anymore. I wouldn’t want to miss the ability to save patches to the DAW-tracks they belong.

[Anonymous]

Hi,
Got the Sysex output working (at least to BOME SendSX / MIDIOX). Sending 211K Sysex – worked with chunking set at 128 bytes, with 125 msec (approx) delay between each chunk. Receiving software showed receiving rate (given USB/midi interface) of about 1KB/sec. As SendSX/MIDIOX are capable of working with Korg Wavestation synth – this is promising.
Will now look at input and see if 211K Sysex is receivable through JUCE & ALSA (linux).
Thanks @Possemo for your encouragement & caution about difficulty of building Ctrlr (the various platforms & plugins). Clearly that’s a place to focus on..and might take time to get working.

[EnzoF04]

Very nice work, sounds promising! Hope it will enable bi-directional communication with responding messages to and from both sides.
I’m a very novice developer. Keep it up, good work!

In response to Vernonr’s post:
It is all predictable ‘cause the first midi message that the S700 replies, tells the amount of data to be send by the S700. I’m aware of the fact that getting Ctrlr and the S700 ‘in sync’ for good and reliable two way communication is a hard thing to achieve.

I’m getting very enthusiastic and i’m Motivated to set up a testing. This will be in 4 weeks time. But no hurry. If I have to install a Linux machine, I will.

• This reply was modified 4 years, 10 months ago by EnzoF04.

[Anonymous]

OK,
Incoming path… I’ve now made some progress here too. The incoming interface that JUCE has allows for 2 “callbacks”
handleIncomingMidiMessage
AND
handlePartialSysexMessage

I believe the issue @EnzoF04 wants to solve is related to the second of these. This “partial” function gets called each time a buffer is transmitted from an external device.
So if the S700 transmits the sysex message as a number of blocks I believe that handlePartialSysexMessage will get notified of each one of these. Certainly I configured MIDIOX to send a number of 18108 bytes of Sysex message in 256 byte buffer. My command line utility based on JUCE got notified in handlePartialSysexMessage of each one of these 256 byte blocks.
After all of these blocks had arrived the end of the message (with the F7) got handed to handleIncomingMidiMessage.

What I’ve established is that Ctrlr deals with the handleIncomingMidiMessage. It does have a stub function which receives the handlePartialSysexMessage – but this simply seems to add the buffers into Memory and periodically clear them. I can’t see that these buffers are used – nor that there is any provision for anything (including LUA) to be told that a buffer of this sort has been received. So @EnzoF04 ‘s problem will affect anyone needing to send acknowledgements back to an external device from Ctrlr during the receipt of a Sysex message.

So what we have to establish is what changes do we need to make to the Ctrlr code to capture these “partial” messages in LUA script. Then I suspect we may need to add some utility functions to ease the calculation of checksums, CRCs etc. in order that the appropriate acknowledgements could be sent back to the external device to cause it to continue sending blocks.

Cheers

[EnzoF04]

This is very nice progress! For me a bit abstract but it makes sense. What I don’t understand is the stages of development; how does JUCE level or not level to LUA? Perhaps a question for an other time.

Concerning the checksums, this is all pretty predictable. My device (the S700) responds to sysex messages. I can request sample properties before I request a dump of a sample. The sample properties tell the data count. Adding header, length, play mode, ending byte (F7) to it and we have a total number of strings to be expected by CTRLR.

Thanks so far, @Vernonr
Am looking forward to get this working!

[Author]

Posts