Sysex dump with multiple separate messages

[memorysplice]

Here is where I am at. I can get a dump to trigger for my JD-Xi. I used the example code from this post to update the name for the program patch as proof of concept.

Due to it being a multi-timbral synth, I have 55 separate midi messages of various sizes how do I Utilize all 55 separate midi messages to update the panel?

[dasfaker]

Each message will have some byte that differentiate those of the same size (could be the channel byte, part byte…), this way you should know what’s inside each message and assign their values to the correct modulators.

[memorysplice]

Here are examples of the messages in question.

Message one

00  F0 41 10 00 00 00 0E 12  18 00 00 00 44 69 73 74  | A          Dist|
10  20 53 65 71 0D 00 00 00  00 00 00 00 6E 03 04 0B  | Seq        n   |
20  0C 00 00 01 01 01 01 00  00 00 00 6E F7           |           n |

Message two

00  F0 41 10 00 00 00 0E 12  18 00 01 00 7F 40 00 00  | A           @  |
10  02 01 00 01 00 00 0A 01  64 01 00 28 28 00 02 00  |        d  ((   |
20  30 00 32 00 00 F7                                 |0 2   |

Message three

00  F0 41 10 00 00 00 0E 12  18 00 02 00 00 7F 32 32  | A            22|
10  01 00 40 00 40 00 40 00  40 00 00 00 00 08 00 06  |  @ @ @ @       |
20  04 08 00 00 0A 08 00 00  05 08 00 07 0F 08 00 00  |                |
30  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
40  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
50  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
60  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
70  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
80  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
90  00 08 00 00 00 08 00 00  00 08 00 00 00 53 F7     |             S |

Message four

00  F0 41 10 00 00 00 0E 12  18 00 04 00 06 7F 7F 00  | A              |
10  01 00 40 00 40 00 40 00  40 00 00 00 00 08 00 00  |  @ @ @ @       |
20  00 08 00 00 00 08 00 00  0B 08 00 00 00 08 00 00  |                |
30  00 08 00 03 0E 08 00 04  02 08 00 00 00 08 00 00  |                |
40  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
50  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
60  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
70  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
80  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
90  00 08 00 00 00 08 00 00  00 08 00 00 00 3D F7     |             = |

.
.
.

Message 54

0  F0 41 10 00 00 00 0E 12  18 00 3F 00 00 7F 00 00  | A        ?     |
10  01 01 01 01 01 01 01 01  01 01 08 00 00 08 00 08  |                |
20  00 08 00 01 7F 40 40 00  00 00 00 00 00 00 00 00  |     @@         |
30  F7                                                | |

Message 55

00  F0 41 10 00 00 00 0E 12  18 00 40 00 00 05 03 00  | A        @     |
10  01 60 05 40 00 64 00 00  16 F7                    | 

@ d |
`

[Puppeteer]

You’ll need to have an understanding of what the dumped data actually means, whether that is through the sysex dump specification from the manufacturer or from reverse engineering the sysex dump format.

The Puppeteer
http://godlike.com.au

[memorysplice]

I do somewhat have a understanding of what the sysex messages mean due to process of elimination. My question is do I treat all 55 messages as one message and attempt to continue or is there a different way to deal with everything in regards to this post.

dasfaker wrote:

This is a very basic guide to manage sysex dumps.

In theory your panel is finished and you created a modulator for each byte from the dump.

Now you need to request a dump from the synth. You need to know the sysex command for this. This is an example dump request, that can be added to the LUA code of a button, for instance:

m = CtrlrMidiMessage({0xF0, 0x00, 0x20, 0x33, 0x01, 0x10, 0x30, 0x00, 0x00, 0xF7}) -- REQUEST PATCH MESSAGE
panel:sendMidiMessageNow(m) -- SENDS THE DUMP REQUEST MESSAGE

Once you send the dump request, the synth will respond sending a sysex message containing the data of the patch. This message usually contains a header followed by the parameters of the synth engine, and this is the part you need to manage. This is an reduced example of a sysex patch:

f0 00 20 33 01 00 10 00 00 0c 01 02 7d 00 00 00 00 00 00 40 00 00 00 00 00 00 40 00 00 40 60 40 00 00 40 f7

You need to know where the actual data of patch starts, in this case is byte 9 (0c)(starting from the first byte that is number 0).

In order to manage the sysex dump received, you need to create a LUA method for “Called when panel receives a MIDI message” panel property. This is a basic method for this:

midiMessageReceived = function(midiMessage)
	s = midiMessage:getSize() -- Size of the midi dump received
	if s == 524 then -- if size match the expected size of the dump requested
		PatchDataLoaded = midiMessage:getData() -- create a memoryblock with the data dump
		programData 	= midiMessage:getData():getRange(09,514) -- create a memory block with the synth engine data, leaving the header
		assignValues(midiMessage,false) -- call a script to assign each byte to each modulator.
	end
end

Now the assignValues script will look like this:

function assignValues(midiMessage,var)
	panel:getModulatorByName("Portamento"):setModulatorValue(programData:getByte(0), false,var,false) -- assign the first byte of programData memoryBlock to it's correspondent modulator
panel:getModulatorByName("Osc1 Volme"):setModulatorValue(programData:getByte(1), false,var,false) -- assign the second byte of programData memoryBlock to it's correspondent modulator. 
.
.
.
-- And so on till the last byte.
end

Of course, you need to know the target modulator for each byte of the dump.

Now to send a patch to the synth, you can do it in several ways: you can send each modulator individually, or collect all modulator values in a memoryBlock and send it to the synth in the same form the panel will receive a dump patch.

To do it individually, just create a button and add this lua method:

function sendPatch()	

	a = panel:getModulatorByName("Portamento"):getValue() -- get the value of this modulator
	panel:getModulatorByName("Portamento"):setModulatorValue(a,false,true,false) -- set the same value for this modulator, sending it to the synth
	a = panel:getModulatorByName("Osc1 Volume"):getValue() -- get the value of this modulator
	panel:getModulatorByName("Osc1 Volume"):setModulatorValue(a,false,true,false) -- set the same value for this modulator, sending it to the synth
.
.
.
-- and so on till the last modulator.
end

Your modulators could be arranged in some way to do this more easy. In the following case, modulators where created in the same order they have in a dump message (so modulator with index 0 corresponds to dump’s byte 0 and so on)

function sendPatch()
	for n = 0, 514, 1 do -- the amount of modulators to send 
		a = panel:getModulatorByIndex(n) -- get modulator with index n			
		b = a:getModulatorValue() -- get it's value
		a:setModulatorValue(b, false,true,false) -- set it's value, sending it to the synth.
	end
end

If you want to send the patch to the synth as a single dump the procedure is similar; collect modulator values, add them to a memoryBlock and send the memoryBlock as a sysex message:

function sendPatch()
	a = panel:getModulatorByName("Portamento"):getValue() -- get the value of this modulator
	PatchDataLoaded:setByte(9,a) -- write it in the memory block in the correct byte
	a = panel:getModulatorByName("Osc1 Volume"):getValue()
	PatchDataLoaded:setByte(10,a)
.
.
.

	m = CtrlrMidiMessage(PatchDataLoaded:toHexString(1)) -- convert the memory block in a midi message
	panel:sendMidiMessageNow(m) -- send the midi message to the synth
end

Hope that helps.

[dasfaker]

As you should know what each message if for, store them one by one when they are received as a memoryBlock and manage them separately. Later, when you want to assign values to modulators, use the stored memoryBlocks instead of midiMessage to read values. Those memoryBlocks can be used later to send data to the synth, instead of sending modulators individually.

[memorysplice]

Can you post a code example on how to do that please so I can follow along?

[dasfaker]

Following the above example, to create a memoryBlock with the midi data of single message:
memBlockName = midiMessage:getData()

Create as many memoryBlocks as messages you receive (remember that each incoming message will trigger the “Called when panel receives a MIDI message” method, so you need to filter each message to know what’s inside, either by size or from some byte inside:

msg = midiMessage:getData() -- incoming midi message

if msg:getSize() == 128 then -- if message size == 128 then is Tone data
    if msg:getByte(9) == 0 then -- if this byte is 0 then is Tone1 data
        Tone1Data = midiMessage:getData() -- create Tone1 memoryBlock
    else                                -- not Tone1 data, then it's Tone2 data
        Tone2Data = midiMessage:getData() -- create Tone2 memoryBlock
    end
elseif msg:getSize() == 20 then -- it's common data
    CommonData = midiMessage:getData()
end

It’s just about analyzing incoming messages.

[memorysplice]

Thank you! This is exactly what I needed!

[memorysplice]

Got to trying to utilize your example dasfaker. I am running into problems and I am not sure what questions to ask to move further. Any thoughts?

[daimondamps]

Do you have some specification of SYSEX that your device is sending?
You need something like:

1byte F0
2byte : device ID
3byte: command
4byte: length of data
5……..x data
F7

So with this info you could interpret incoming messages by command byte
or something else.

[memorysplice]

Yes! Here is a link for the JD-Xi sysex. Click to download sysex spec

I figured out how to trigger a smaller dump that only outputs the Digital Synth 1 and 2 and the analog synth with only 14 messages vs 55.

f0 41 10 00 00 00 0e 11 19 01 00 00 00 6e 7f 7f z5 f7`

Here is what I have been using before that works.

—
— Called when a panel receives a midi message (does not need to match any modulator mask)
— @midi CtrlrMidiMessage object
—

midiMessageReceived = function(midiMessage)

s = midiMessage:getSize()

if s == 45 then — if size match the expected size of the dump requested
PatchDataLoaded = midiMessage:getData() — create a memoryblock with the data dump
programData = midiMessage:getData():getRange(11,45-11) — create a memory block with the synth engine data, leaving the header
assignValues(midiMessage,false) — call a script to assign each byte to each modulator.
end
end

function assignValues(midiMessage,var)
panel:getModulatorByName(“Letter 1”):setModulatorValue(programData:getByte(1), false, false, false)
— panel:getModulatorByName(“Letter 2”):setModulatorValue(programData:getByte(2), false, false, false)
— panel:getModulatorByName(“Letter 3”):setModulatorValue(programData:getByte(3), false, false, false)
— panel:getModulatorByName(“Letter 4”):setModulatorValue(programData:getByte(4), false, false, false)
— panel:getModulatorByName(“Letter 5”):setModulatorValue(programData:getByte(5), false, false, false)
— panel:getModulatorByName(“Letter 6”):setModulatorValue(programData:getByte(6), false, false, false)
— panel:getModulatorByName(“Letter 7”):setModulatorValue(programData:getByte(7), false, false, false)
— panel:getModulatorByName(“Letter 8”):setModulatorValue(programData:getByte(8), false, false, false)
end`

I am not sure how to implement the code with what I have.

[daimondamps]

The beginning of the messages are alway the same but after 0x18 there are some bytes which may help you to identify message types.
F0 41 10 00 00 00 0E 12 18 00 02 00
So you must read those byte and then assign data to the correct variables.

I found this:http://www.synthforum.nl/forums/showthread.php?t=192504

and this panel :
http://www.stephenm.be/JDXI/Ctrlr-JD-Xi-R6-Steph-001.zip

• This reply was modified 8 years, 4 months ago by daimondamps.

[memorysplice]

Thank you for looking around, daimondamps. I know how to read the sysex dump. My problem is writhing out the lua code to accept a dump and update the panel from multiple sysex messages. As for the panel that stephen uploaded, he is building on what I started and he as far as I know does not know anything about coding in lua.

[daimondamps]

Ok, so if you have byte in sysex that will tell you what sysex is this you can construct your code like:

If (midiMessage:getData():getByte(yourmagical commad byte)==02) then
Modulator1=….
elseif(secod type of sysx message) then
Modulator2=…

Else
End

[memorysplice]

Is there a way to setup a ctrlr script to lets say I request 4 specific sysex dumps and get back 4 messages that reference the synth common and the the 3 parts that make up the synth. “the 3 partials that make up the 1st part of the JD-Xi.” How would I go about doing that?

[memorysplice]

The reason why I asked is because I just figured out how to transmit more than 1 sysex message with a button to trigger specific midi dump responses.

[daimondamps]

No.each time ctrlr recives a midi message your script is ivoked.. Is up to you how to handle icoming messages. Like i wroted before you could filter incoming messages with if else statment and the Command byte from incomming messages and assign the values in if else blocks.

[memorysplice]

Thank you for your guidance. I do understand everything you told me yet, but I will try to figure out where to go from here.

[daimondamps]

So I will try once more:)
No matter how do you invoke your device to sending sysex messages.
Panel recive callback is invoken once per sysex message. So you have to write a scripti MessageReceived = function(midiMessage) that will handle every midi sysex message that your device is sending.

So lets say you have following messages( and I am assuming that byte 11 of that message is unique for every group of parameters) :
1’st message
F0 41 10 00 00 00 0E 12 18 00 3F 00 00 7F 00 00 | A ? |
10 01 01 01 01 01 01 01 01 01 01 08 00 00 08 00 08 | |
20 00 08 00 01 7F 40 40 00 00 00 00 00 00 00 00 00 | @@ |
30 F7

and next message
2’nd message
F0 41 10 00 00 00 0E 12 18 00 40 00 00 05 03 00 | A @ |
10 01 60 05 40 00 64 00 00 16 F7

What you need to do is :

MessageReceived = function(midiMessage)

if  ( midiMessage:getData():getByte(11) == 0x3f) then
--you are processing 1'st message (if it arrives - this fragment will process this message)

     modulator1= midiMessage:getData():getByte(byte from 1'st message for your modulator)
     modulator2=.....

elseif  ( midiMessage:getData():getByte(11) == 0x40) then

--you are processing 2'cond message (if it arrives - this fragment will process this message)
     modulator3= midiMessage:getData():getByte(byte from 2'st message for your modulator)  
.
.
. 
elseif ( midiMessage:getData():getByte(n) == 0xXX)
--you are processing N'th message          
.
.
else
--you are processing any other message
end

• This reply was modified 8 years, 4 months ago by daimondamps.
• This reply was modified 8 years, 4 months ago by daimondamps.
• This reply was modified 8 years, 4 months ago by daimondamps.
• This reply was modified 8 years, 4 months ago by daimondamps.
• This reply was modified 8 years, 4 months ago by daimondamps.

[Author]

Posts