Sysex dump with multiple separate messages

Home Forums General Programming Sysex dump with multiple separate messages

Viewing 20 posts - 1 through 20 (of 39 total)
  • Author
    Posts
  • #64509
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Here is where I am at. I can get a dump to trigger for my JD-Xi. I used the example code from this post to update the name for the program patch as proof of concept.

    Due to it being a multi-timbral synth, I have 55 separate midi messages of various sizes how do I Utilize all 55 separate midi messages to update the panel?

    #64530
    dasfaker
    Keymaster
    • Topics: 80
    • Replies: 793
    • Total: 873
    • ★★★

    Each message will have some byte that differentiate those of the same size (could be the channel byte, part byte…), this way you should know what’s inside each message and assign their values to the correct modulators.

    #64614
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Here are examples of the messages in question.

    Message one

    00  F0 41 10 00 00 00 0E 12  18 00 00 00 44 69 73 74  | A          Dist|
    10  20 53 65 71 0D 00 00 00  00 00 00 00 6E 03 04 0B  | Seq        n   |
    20  0C 00 00 01 01 01 01 00  00 00 00 6E F7           |           n |
    

    Message two

    00  F0 41 10 00 00 00 0E 12  18 00 01 00 7F 40 00 00  | A           @  |
    10  02 01 00 01 00 00 0A 01  64 01 00 28 28 00 02 00  |        d  ((   |
    20  30 00 32 00 00 F7                                 |0 2   |
    

    Message three

    00  F0 41 10 00 00 00 0E 12  18 00 02 00 00 7F 32 32  | A            22|
    10  01 00 40 00 40 00 40 00  40 00 00 00 00 08 00 06  |  @ @ @ @       |
    20  04 08 00 00 0A 08 00 00  05 08 00 07 0F 08 00 00  |                |
    30  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    40  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    50  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    60  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    70  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    80  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    90  00 08 00 00 00 08 00 00  00 08 00 00 00 53 F7     |             S |
    

    Message four

    00  F0 41 10 00 00 00 0E 12  18 00 04 00 06 7F 7F 00  | A              |
    10  01 00 40 00 40 00 40 00  40 00 00 00 00 08 00 00  |  @ @ @ @       |
    20  00 08 00 00 00 08 00 00  0B 08 00 00 00 08 00 00  |                |
    30  00 08 00 03 0E 08 00 04  02 08 00 00 00 08 00 00  |                |
    40  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    50  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    60  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    70  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    80  00 08 00 00 00 08 00 00  00 08 00 00 00 08 00 00  |                |
    90  00 08 00 00 00 08 00 00  00 08 00 00 00 3D F7     |             = |
    

    .
    .
    .

    Message 54

    0  F0 41 10 00 00 00 0E 12  18 00 3F 00 00 7F 00 00  | A        ?     |
    10  01 01 01 01 01 01 01 01  01 01 08 00 00 08 00 08  |                |
    20  00 08 00 01 7F 40 40 00  00 00 00 00 00 00 00 00  |     @@         |
    30  F7                                                | |
    

    Message 55

    00  F0 41 10 00 00 00 0E 12  18 00 40 00 00 05 03 00  | A        @     |
    10  01 60 05 40 00 64 00 00  16 F7                    | 

    @ d |
    `

    #64701
    Puppeteer
    Participant
    • Topics: 16
    • Replies: 185
    • Total: 201
    • ★★

    You’ll need to have an understanding of what the dumped data actually means, whether that is through the sysex dump specification from the manufacturer or from reverse engineering the sysex dump format.

    The Puppeteer
    http://godlike.com.au

    #64714
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    I do somewhat have a understanding of what the sysex messages mean due to process of elimination. My question is do I treat all 55 messages as one message and attempt to continue or is there a different way to deal with everything in regards to this post.

    This is a very basic guide to manage sysex dumps.

    In theory your panel is finished and you created a modulator for each byte from the dump.

    Now you need to request a dump from the synth. You need to know the sysex command for this. This is an example dump request, that can be added to the LUA code of a button, for instance:

    m = CtrlrMidiMessage({0xF0, 0x00, 0x20, 0x33, 0x01, 0x10, 0x30, 0x00, 0x00, 0xF7}) -- REQUEST PATCH MESSAGE
    panel:sendMidiMessageNow(m) -- SENDS THE DUMP REQUEST MESSAGE

    Once you send the dump request, the synth will respond sending a sysex message containing the data of the patch. This message usually contains a header followed by the parameters of the synth engine, and this is the part you need to manage. This is an reduced example of a sysex patch:

    
    f0 00 20 33 01 00 10 00 00 0c 01 02 7d 00 00 00 00 00 00 40 00 00 00 00 00 00 40 00 00 40 60 40 00 00 40 f7

    You need to know where the actual data of patch starts, in this case is byte 9 (0c)(starting from the first byte that is number 0).

    In order to manage the sysex dump received, you need to create a LUA method for “Called when panel receives a MIDI message” panel property. This is a basic method for this:

    midiMessageReceived = function(midiMessage)
    	s = midiMessage:getSize() -- Size of the midi dump received
    	if s == 524 then -- if size match the expected size of the dump requested
    		PatchDataLoaded = midiMessage:getData() -- create a memoryblock with the data dump
    		programData 	= midiMessage:getData():getRange(09,514) -- create a memory block with the synth engine data, leaving the header
    		assignValues(midiMessage,false) -- call a script to assign each byte to each modulator.
    	end
    end

    Now the assignValues script will look like this:

    function assignValues(midiMessage,var)
    	panel:getModulatorByName("Portamento"):setModulatorValue(programData:getByte(0), false,var,false) -- assign the first byte of programData memoryBlock to it's correspondent modulator
    panel:getModulatorByName("Osc1 Volme"):setModulatorValue(programData:getByte(1), false,var,false) -- assign the second byte of programData memoryBlock to it's correspondent modulator. 
    .
    .
    .
    -- And so on till the last byte.
    end

    Of course, you need to know the target modulator for each byte of the dump.

    Now to send a patch to the synth, you can do it in several ways: you can send each modulator individually, or collect all modulator values in a memoryBlock and send it to the synth in the same form the panel will receive a dump patch.

    To do it individually, just create a button and add this lua method:

    
    function sendPatch()	
    
    	a = panel:getModulatorByName("Portamento"):getValue() -- get the value of this modulator
    	panel:getModulatorByName("Portamento"):setModulatorValue(a,false,true,false) -- set the same value for this modulator, sending it to the synth
    	a = panel:getModulatorByName("Osc1 Volume"):getValue() -- get the value of this modulator
    	panel:getModulatorByName("Osc1 Volume"):setModulatorValue(a,false,true,false) -- set the same value for this modulator, sending it to the synth
    .
    .
    .
    -- and so on till the last modulator.
    end

    Your modulators could be arranged in some way to do this more easy. In the following case, modulators where created in the same order they have in a dump message (so modulator with index 0 corresponds to dump’s byte 0 and so on)

    function sendPatch()
    	for n = 0, 514, 1 do -- the amount of modulators to send 
    		a = panel:getModulatorByIndex(n) -- get modulator with index n			
    		b = a:getModulatorValue() -- get it's value
    		a:setModulatorValue(b, false,true,false) -- set it's value, sending it to the synth.
    	end
    end

    If you want to send the patch to the synth as a single dump the procedure is similar; collect modulator values, add them to a memoryBlock and send the memoryBlock as a sysex message:

    
    function sendPatch()
    	a = panel:getModulatorByName("Portamento"):getValue() -- get the value of this modulator
    	PatchDataLoaded:setByte(9,a) -- write it in the memory block in the correct byte
    	a = panel:getModulatorByName("Osc1 Volume"):getValue()
    	PatchDataLoaded:setByte(10,a)
    .
    .
    .
    
    	m = CtrlrMidiMessage(PatchDataLoaded:toHexString(1)) -- convert the memory block in a midi message
    	panel:sendMidiMessageNow(m) -- send the midi message to the synth
    end

    Hope that helps.

    #64719
    dasfaker
    Keymaster
    • Topics: 80
    • Replies: 793
    • Total: 873
    • ★★★

    As you should know what each message if for, store them one by one when they are received as a memoryBlock and manage them separately. Later, when you want to assign values to modulators, use the stored memoryBlocks instead of midiMessage to read values. Those memoryBlocks can be used later to send data to the synth, instead of sending modulators individually.

    #64735
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Can you post a code example on how to do that please so I can follow along?

    #64776
    dasfaker
    Keymaster
    • Topics: 80
    • Replies: 793
    • Total: 873
    • ★★★

    Following the above example, to create a memoryBlock with the midi data of single message:
    memBlockName = midiMessage:getData()

    Create as many memoryBlocks as messages you receive (remember that each incoming message will trigger the “Called when panel receives a MIDI message” method, so you need to filter each message to know what’s inside, either by size or from some byte inside:

    msg = midiMessage:getData() -- incoming midi message
    
    if msg:getSize() == 128 then -- if message size == 128 then is Tone data
        if msg:getByte(9) == 0 then -- if this byte is 0 then is Tone1 data
            Tone1Data = midiMessage:getData() -- create Tone1 memoryBlock
        else                                -- not Tone1 data, then it's Tone2 data
            Tone2Data = midiMessage:getData() -- create Tone2 memoryBlock
        end
    elseif msg:getSize() == 20 then -- it's common data
        CommonData = midiMessage:getData()
    end

    It’s just about analyzing incoming messages.

    #64784
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Thank you! This is exactly what I needed!

    #65017
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Got to trying to utilize your example dasfaker. I am running into problems and I am not sure what questions to ask to move further. Any thoughts?

    #65021
    daimondamps
    Participant
    • Topics: 8
    • Replies: 80
    • Total: 88

    Do you have some specification of SYSEX that your device is sending?
    You need something like:

    1byte F0
    2byte : device ID
    3byte: command
    4byte: length of data
    5……..x data
    F7

    So with this info you could interpret incoming messages by command byte
    or something else.

    #65022
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Yes! Here is a link for the JD-Xi sysex. Click to download sysex spec

    I figured out how to trigger a smaller dump that only outputs the Digital Synth 1 and 2 and the analog synth with only 14 messages vs 55.

    f0 41 10 00 00 00 0e 11 19 01 00 00 00 6e 7f 7f z5 f7`

    Here is what I have been using before that works.


    — Called when a panel receives a midi message (does not need to match any modulator mask)
    — @midi CtrlrMidiMessage object

    midiMessageReceived = function(midiMessage)

    s = midiMessage:getSize()

    if s == 45 then — if size match the expected size of the dump requested
    PatchDataLoaded = midiMessage:getData() — create a memoryblock with the data dump
    programData = midiMessage:getData():getRange(11,45-11) — create a memory block with the synth engine data, leaving the header
    assignValues(midiMessage,false) — call a script to assign each byte to each modulator.
    end
    end

    function assignValues(midiMessage,var)
    panel:getModulatorByName(“Letter 1”):setModulatorValue(programData:getByte(1), false, false, false)
    — panel:getModulatorByName(“Letter 2”):setModulatorValue(programData:getByte(2), false, false, false)
    — panel:getModulatorByName(“Letter 3”):setModulatorValue(programData:getByte(3), false, false, false)
    — panel:getModulatorByName(“Letter 4”):setModulatorValue(programData:getByte(4), false, false, false)
    — panel:getModulatorByName(“Letter 5”):setModulatorValue(programData:getByte(5), false, false, false)
    — panel:getModulatorByName(“Letter 6”):setModulatorValue(programData:getByte(6), false, false, false)
    — panel:getModulatorByName(“Letter 7”):setModulatorValue(programData:getByte(7), false, false, false)
    — panel:getModulatorByName(“Letter 8”):setModulatorValue(programData:getByte(8), false, false, false)
    end`

    I am not sure how to implement the code with what I have.

    #65115
    daimondamps
    Participant
    • Topics: 8
    • Replies: 80
    • Total: 88

    The beginning of the messages are alway the same but after 0x18 there are some bytes which may help you to identify message types.
    F0 41 10 00 00 00 0E 12 18 00 02 00
    So you must read those byte and then assign data to the correct variables.

    I found this:http://www.synthforum.nl/forums/showthread.php?t=192504

    and this panel :
    http://www.stephenm.be/JDXI/Ctrlr-JD-Xi-R6-Steph-001.zip

    • This reply was modified 6 years, 6 months ago by daimondamps.
    #65117
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Thank you for looking around, daimondamps. I know how to read the sysex dump. My problem is writhing out the lua code to accept a dump and update the panel from multiple sysex messages. As for the panel that stephen uploaded, he is building on what I started and he as far as I know does not know anything about coding in lua.

    #65204
    daimondamps
    Participant
    • Topics: 8
    • Replies: 80
    • Total: 88

    Ok, so if you have byte in sysex that will tell you what sysex is this you can construct your code like:

    If (midiMessage:getData():getByte(yourmagical commad byte)==02) then
    Modulator1=….
    elseif(secod type of sysx message) then
    Modulator2=…

    Else
    End

    #65206
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Is there a way to setup a ctrlr script to lets say I request 4 specific sysex dumps and get back 4 messages that reference the synth common and the the 3 parts that make up the synth. “the 3 partials that make up the 1st part of the JD-Xi.” How would I go about doing that?

    #65207
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    The reason why I asked is because I just figured out how to transmit more than 1 sysex message with a button to trigger specific midi dump responses.

    #65212
    daimondamps
    Participant
    • Topics: 8
    • Replies: 80
    • Total: 88

    No.each time ctrlr recives a midi message your script is ivoked.. Is up to you how to handle icoming messages. Like i wroted before you could filter incoming messages with if else statment and the Command byte from incomming messages and assign the values in if else blocks.

    #65215
    memorysplice
    Participant
    • Topics: 14
    • Replies: 59
    • Total: 73

    Thank you for your guidance. I do understand everything you told me yet, but I will try to figure out where to go from here.

    #65224
    daimondamps
    Participant
    • Topics: 8
    • Replies: 80
    • Total: 88

    So I will try once more:)
    No matter how do you invoke your device to sending sysex messages.
    Panel recive callback is invoken once per sysex message. So you have to write a scripti MessageReceived = function(midiMessage) that will handle every midi sysex message that your device is sending.

    So lets say you have following messages( and I am assuming that byte 11 of that message is unique for every group of parameters) :
    1’st message
    F0 41 10 00 00 00 0E 12 18 00 3F 00 00 7F 00 00 | A ? |
    10 01 01 01 01 01 01 01 01 01 01 08 00 00 08 00 08 | |
    20 00 08 00 01 7F 40 40 00 00 00 00 00 00 00 00 00 | @@ |
    30 F7

    and next message
    2’nd message
    F0 41 10 00 00 00 0E 12 18 00 40 00 00 05 03 00 | A @ |
    10 01 60 05 40 00 64 00 00 16 F7

    What you need to do is :

    MessageReceived = function(midiMessage)
    
    if  ( midiMessage:getData():getByte(11) == 0x3f) then
    --you are processing 1'st message (if it arrives - this fragment will process this message)
    
         modulator1= midiMessage:getData():getByte(byte from 1'st message for your modulator)
         modulator2=.....
    
    elseif  ( midiMessage:getData():getByte(11) == 0x40) then
    
    --you are processing 2'cond message (if it arrives - this fragment will process this message)
         modulator3= midiMessage:getData():getByte(byte from 2'st message for your modulator)  
    .
    .
    . 
    elseif ( midiMessage:getData():getByte(n) == 0xXX)
    --you are processing N'th message          
    .
    .
    else
    --you are processing any other message
    end
    • This reply was modified 6 years, 6 months ago by daimondamps.
    • This reply was modified 6 years, 6 months ago by daimondamps.
    • This reply was modified 6 years, 6 months ago by daimondamps.
    • This reply was modified 6 years, 6 months ago by daimondamps.
    • This reply was modified 6 years, 6 months ago by daimondamps.
Viewing 20 posts - 1 through 20 (of 39 total)
  • The forum ‘Programming’ is closed to new topics and replies.
There is currently 0 users and 54 guests online
No users are currently active
Forum Statistics
Threads: 2,495, Posts: 17,374, Members: 77,605
Most users ever online was 12 on January 22, 2019 3:47 pm
Ctrlr