Large Files Transfers Between Parts of Chrome Extensions for Manifest V3

As you know, I am an author of the PerfectPixel browser extension. Recently, I transferred it from Manifest V2 to Manifest V3, and during the process, re-architected the messaging system due to the changes made in the new messaging API. The challenge is that the messaging protocol has a message size limit now, so we have to deal with that limitation. In this article, I'll outline Manifest V3 messaging fundamentals, overhaul the problem, and then share the solution I've made. The solution is published as an NPM package; the source code is available on GitHub. You can find the link at the end of the article. Chrome Extensions Messaging Fundamentals Chrome extension compliant with Manifest V3 consists of several parts: background service worker (replacement for background pages in Manifest V2), content scripts, and different web pages - popup, settings, offscreen. All of those pieces have a universal API for communication with each other: chrome.runtime.messaging. It uses a pub-sub model. chrome.runtime.sendMessage(message: any, callback: function) - method broadcast message to all parts of your extension. The second argument is used to handle the response. chrome.runtime.onMessage.addListener((message: any, sender: MessageSender, sendResponse: function) => boolean) method registers a listener. The third argument of the listener function is used to send a response to the sender, sync, or async. The return value of the listener is used to determine if the response is expected to be synced or asynced. Example content script: chrome.runtime.sendMessage({ type: 'foo-bar' content: 'foo' }, (response) => { // response === 'bar' ... }); service worker, sync listener: chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch(message.type) { case 'foo-bar': sendResponse('bar'); break; default: break; } return false; // sync }) service worker, async listener: chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch(message.type) { case 'foo-bar': somePromiseToExecute().then(() => { sendResponse('bar'); }) break; default: break; } return true; // async }) The Problem The message sent via sendMessage has a maximum length that cannot be exceeded or the message won't be sent; you will see. Uncaught Error: Message length exceeded maximum allowed length. In my tests, the max message size is slightly above 32Mb, which is not enough for some large images, especially in a serialized state. The Solution Let's divide the message into chunks and send them in a separate sendMessage calls. For the sender, I will be creating the sendChunkedMessage(message: any): Promise async function that mimics the sendMessage signature and hides chunking and transmitting under the hood. The original message is serialized and split into chunks. A group of messages is created to send individual chunks that share the generated requestId. The last message in the group contains done: true that signals the receiver that transmission is done. // To filter out chunked messages on receiver side const CHUNKED_MESSAGE_FLAG = 'CHUNKED_MESSAGE_FLAG' const MAX_CHUNK_SIZE = 32 * 1024 * 1024; // 32Mb const sendMessage = (message) => new Promise(resolve => chrome.runtime.sendMessage(message, response => { resolve(response); }); ); const sendChunkedMessage = (message) => { // Generating requestId for the message const requestId = self.crypto.randomUUID(); const messageSerialized = JSON.stringify(message); const len = messageSerialized.length; const step = MAX_CHUNK_SIZE; let ii = 0; // Sending messageSerialized in chunks in separate sendMessage calls while (ii < len) { const nextIndex = Math.min(ii + step, len); const substr = messageSerialized.substring(ii, nextIndex); await sendMessage({ [CHUNKED_MESSAGE_FLAG]: true, requestId, chunk: substr }); ii = nextIndex; } // At least 2 messages will be sent. Last one - with done: true const response = await sendMessage({ [CHUNKED_MESSAGE_FLAG]: true, requestId, done: true }); } On the receiver end, we need to create a handler that will be reconstructing messages from chunks based on requestId. I've created a function addOnChunkedMessageListener(handler: (request, sender, sendReponse) => boolean) that mimics chrome.runtime.onMessage.addListener signature hides implementation details under the hood. Received chunks are stored into requestsStorage hashmap by requestId. When the done: true message is received, the original message is reconstructed by combining chunks from requestsStorage[requestId]. Then, provided handler function is executed with a reconstructed message const requestsStorage = {}; const addOnChunkedMessageListener = (handler) => { const newListener = (request, sender, sendResponse) => { if (request && request[CHUNKED_MESSAGE_FLAG] && request.requestId) { const requestId = request.requestId; if (request.done) { const fullMessageSearialized = ''.concat.apply( '', requestsStorage[requestId] ); delete requestsStorage[requestId]; const fullMessage = JSON.parse(fullMessageSearialized); // async sendResponse can be enabled inside handler function return handler(fullMessage, sender, sendResponse); } else { if (!requestsStorage[requestId]) { requestsStorage[requestId] = []; } requestsStorage[requestId].push(request.chunk); sendResponse({ status: 'PENDING' }); } return false; // sync listener } } chrome.runtime.onMessage.addListener(newListener); return newListener; }; Example Usage content script: sendChunkedMessage(largeMessage) .then(response => { ... }) background service worker: addOnChunkedMessageListener((message, sender, sendResponse) => { // message === largeMessage ... }) Large Response The above solution works when we need to send a large message and receive a "normal size" response. But how can we send back the large response on the receiver side with sendResponse? The idea is to send back an indication large response will follow, and temporarily add the same addOnChunkedMessageListener on the sender side, receive a chunked response, then remove the temporary listener. To send a large response, sendChunkedResponse function should be used on the receiver side: const sendChunkedResponse = ({ sendMessageFn } = {}) => ( response, sendResponse, ) => { const requestId = self.crypto.randomUUID(); // Sending an indication that file will be sent as chunked messages sendResponse({ [CHUNKED_MESSAGE_FLAG]: true, requestId }); // At this point content script has added a listener with addOnMessageWithChunksListener // Sending file contents as chunked messages sendChunkedMessage(response, { sendMessageFn: sendMessageFn || sendMessage, requestId }); }; I've modified sendChunkedMessage function with adding options: the ability to override the sendMessage function, override requestId, and support for receiving chunked responses. const sendChunkedMessage = async ( message, { sendMessageFn, requestId: requestIdOverriden } = {} ) => { const sendMessage = sendMessageFn || sendMessage; // Generating requestId for the message const requestId = requestIdOverriden || self.crypto.randomUUID(); ... // If response indicates there will be a chunk message sent, adding a listener to retrieve full response if (response && response[CHUNKED_MESSAGE_FLAG]) { let listener; try { const fullResponse = await new Promise(resolve => { listener = addOnChunkedMessageListener( (fullResponseFromListener, _, sendResponse) => { sendResponse(); resolve(fullResponseFromListener); }, { requestIdToMonitor: response.requestId } ); }); return fullResponse; } finally { if (listener) { removeOnChunkedMessageListener(listener); } } } else { return response; } } You may notice addOnChunkedMessageListener is also slightly modified to filter out incoming requestId with requestIdToMonitor option. Example Usage With Large Response content script - same: sendChunkedMessage(largeMessage) .then(response => { ... }) background service worker: addOnChunkedMessageListener((message, sender, sendResponse) => { // message === largeMessage ... sendChunkedResponse({ sendMessageFn: message => chrome.tabs.sendMessage(sender.tab.id, message) })(largeResponse, sendResponse); return true; // async listener }) Conclusion The solution was created to overcome message length limitation for Chrome Extensions messaging in Manifest V3; the solution works and I am using it in a real project. Happy to hear advice on how to make it better, and feel free to contribute! Code with an example extension can be found on GitHub: https://github.com/abelozerov/ext-send-chunked-message Published to NPM. It can be installed with npm i ext-send-chunked-message Thank you for reading! As you know, I am an author of the PerfectPixel browser extension. Recently, I transferred it from Manifest V2 to Manifest V3, and during the process, re-architected the messaging system due to the changes made in the new messaging API. PerfectPixel The challenge is that the messaging protocol has a message size limit now, so we have to deal with that limitation. In this article, I'll outline Manifest V3 messaging fundamentals, overhaul the problem, and then share the solution I've made. The solution is published as an NPM package; the source code is available on GitHub. You can find the link at the end of the article. Chrome Extensions Messaging Fundamentals Chrome extension compliant with Manifest V3 consists of several parts: background service worker (replacement for background pages in Manifest V2), content scripts, and different web pages - popup, settings, offscreen. All of those pieces have a universal API for communication with each other: chrome.runtime.messaging . It uses a pub-sub model. chrome.runtime.messaging chrome.runtime.sendMessage(message: any, callback: function) - method broadcast message to all parts of your extension. The second argument is used to handle the response. chrome.runtime.sendMessage(message: any, callback: function) chrome.runtime.onMessage.addListener((message: any, sender: MessageSender, sendResponse: function) => boolean) method registers a listener. The third argument of the listener function is used to send a response to the sender, sync, or async. The return value of the listener is used to determine if the response is expected to be synced or asynced. chrome.runtime.onMessage.addListener((message: any, sender: MessageSender, sendResponse: function) => boolean) Example content script: chrome.runtime.sendMessage({ type: 'foo-bar' content: 'foo' }, (response) => { // response === 'bar' ... }); chrome.runtime.sendMessage({ type: 'foo-bar' content: 'foo' }, (response) => { // response === 'bar' ... }); service worker, sync listener: chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch(message.type) { case 'foo-bar': sendResponse('bar'); break; default: break; } return false; // sync }) chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch(message.type) { case 'foo-bar': sendResponse('bar'); break; default: break; } return false; // sync }) service worker, async listener: chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch(message.type) { case 'foo-bar': somePromiseToExecute().then(() => { sendResponse('bar'); }) break; default: break; } return true; // async }) chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch(message.type) { case 'foo-bar': somePromiseToExecute().then(() => { sendResponse('bar'); }) break; default: break; } return true; // async }) The Problem The message sent via sendMessage has a maximum length that cannot be exceeded or the message won't be sent; you will see. Uncaught Error: Message length exceeded maximum allowed length. Uncaught Error: Message length exceeded maximum allowed length. In my tests, the max message size is slightly above 32Mb, which is not enough for some large images, especially in a serialized state. The Solution Let's divide the message into chunks and send them in a separate sendMessage calls. sendMessage For the sender, I will be creating the sendChunkedMessage(message: any): Promise async function that mimics the sendMessage signature and hides chunking and transmitting under the hood. For the sender, sendChunkedMessage(message: any): Promise The original message is serialized and split into chunks. A group of messages is created to send individual chunks that share the generated requestId . The last message in the group contains done: true that signals the receiver that transmission is done. requestId done: true // To filter out chunked messages on receiver side const CHUNKED_MESSAGE_FLAG = 'CHUNKED_MESSAGE_FLAG' const MAX_CHUNK_SIZE = 32 * 1024 * 1024; // 32Mb const sendMessage = (message) => new Promise(resolve => chrome.runtime.sendMessage(message, response => { resolve(response); }); ); const sendChunkedMessage = (message) => { // Generating requestId for the message const requestId = self.crypto.randomUUID(); const messageSerialized = JSON.stringify(message); const len = messageSerialized.length; const step = MAX_CHUNK_SIZE; let ii = 0; // Sending messageSerialized in chunks in separate sendMessage calls while (ii < len) { const nextIndex = Math.min(ii + step, len); const substr = messageSerialized.substring(ii, nextIndex); await sendMessage({ [CHUNKED_MESSAGE_FLAG]: true, requestId, chunk: substr }); ii = nextIndex; } // At least 2 messages will be sent. Last one - with done: true const response = await sendMessage({ [CHUNKED_MESSAGE_FLAG]: true, requestId, done: true }); } // To filter out chunked messages on receiver side const CHUNKED_MESSAGE_FLAG = 'CHUNKED_MESSAGE_FLAG' const MAX_CHUNK_SIZE = 32 * 1024 * 1024; // 32Mb const sendMessage = (message) => new Promise(resolve => chrome.runtime.sendMessage(message, response => { resolve(response); }); ); const sendChunkedMessage = (message) => { // Generating requestId for the message const requestId = self.crypto.randomUUID(); const messageSerialized = JSON.stringify(message); const len = messageSerialized.length; const step = MAX_CHUNK_SIZE; let ii = 0; // Sending messageSerialized in chunks in separate sendMessage calls while (ii < len) { const nextIndex = Math.min(ii + step, len); const substr = messageSerialized.substring(ii, nextIndex); await sendMessage({ [CHUNKED_MESSAGE_FLAG]: true, requestId, chunk: substr }); ii = nextIndex; } // At least 2 messages will be sent. Last one - with done: true const response = await sendMessage({ [CHUNKED_MESSAGE_FLAG]: true, requestId, done: true }); } On the receiver end, we need to create a handler that will be reconstructing messages from chunks based on requestId. I've created a function addOnChunkedMessageListener(handler: (request, sender, sendReponse) => boolean) that mimics chrome.runtime.onMessage.addListener signature hides implementation details under the hood. On the receiver end, addOnChunkedMessageListener(handler: (request, sender, sendReponse) => boolean) Received chunks are stored into requestsStorage hashmap by requestId . When the done: true message is received, the original message is reconstructed by combining chunks from requestsStorage[requestId] . requestsStorage requestId done: true requestsStorage[requestId] Then, provided handler function is executed with a reconstructed message handler const requestsStorage = {}; const addOnChunkedMessageListener = (handler) => { const newListener = (request, sender, sendResponse) => { if (request && request[CHUNKED_MESSAGE_FLAG] && request.requestId) { const requestId = request.requestId; if (request.done) { const fullMessageSearialized = ''.concat.apply( '', requestsStorage[requestId] ); delete requestsStorage[requestId]; const fullMessage = JSON.parse(fullMessageSearialized); // async sendResponse can be enabled inside handler function return handler(fullMessage, sender, sendResponse); } else { if (!requestsStorage[requestId]) { requestsStorage[requestId] = []; } requestsStorage[requestId].push(request.chunk); sendResponse({ status: 'PENDING' }); } return false; // sync listener } } chrome.runtime.onMessage.addListener(newListener); return newListener; }; const requestsStorage = {}; const addOnChunkedMessageListener = (handler) => { const newListener = (request, sender, sendResponse) => { if (request && request[CHUNKED_MESSAGE_FLAG] && request.requestId) { const requestId = request.requestId; if (request.done) { const fullMessageSearialized = ''.concat.apply( '', requestsStorage[requestId] ); delete requestsStorage[requestId]; const fullMessage = JSON.parse(fullMessageSearialized); // async sendResponse can be enabled inside handler function return handler(fullMessage, sender, sendResponse); } else { if (!requestsStorage[requestId]) { requestsStorage[requestId] = []; } requestsStorage[requestId].push(request.chunk); sendResponse({ status: 'PENDING' }); } return false; // sync listener } } chrome.runtime.onMessage.addListener(newListener); return newListener; }; Example Usage content script: sendChunkedMessage(largeMessage) .then(response => { ... }) sendChunkedMessage(largeMessage) .then(response => { ... }) background service worker: addOnChunkedMessageListener((message, sender, sendResponse) => { // message === largeMessage ... }) addOnChunkedMessageListener((message, sender, sendResponse) => { // message === largeMessage ... }) Large Response The above solution works when we need to send a large message and receive a "normal size" response. But how can we send back the large response on the receiver side with sendResponse? The idea is to send back an indication large response will follow, and temporarily add the same addOnChunkedMessageListener on the sender side, receive a chunked response, then remove the temporary listener. addOnChunkedMessageListener To send a large response, sendChunkedResponse function should be used on the receiver side: To send a large response, sendChunkedResponse const sendChunkedResponse = ({ sendMessageFn } = {}) => ( response, sendResponse, ) => { const requestId = self.crypto.randomUUID(); // Sending an indication that file will be sent as chunked messages sendResponse({ [CHUNKED_MESSAGE_FLAG]: true, requestId }); // At this point content script has added a listener with addOnMessageWithChunksListener // Sending file contents as chunked messages sendChunkedMessage(response, { sendMessageFn: sendMessageFn || sendMessage, requestId }); }; const sendChunkedResponse = ({ sendMessageFn } = {}) => ( response, sendResponse, ) => { const requestId = self.crypto.randomUUID(); // Sending an indication that file will be sent as chunked messages sendResponse({ [CHUNKED_MESSAGE_FLAG]: true, requestId }); // At this point content script has added a listener with addOnMessageWithChunksListener // Sending file contents as chunked messages sendChunkedMessage(response, { sendMessageFn: sendMessageFn || sendMessage, requestId }); }; I've modified sendChunkedMessage function with adding options: the ability to override the sendMessage function, override requestId, and support for receiving chunked responses. sendChunkedMessage const sendChunkedMessage = async ( message, { sendMessageFn, requestId: requestIdOverriden } = {} ) => { const sendMessage = sendMessageFn || sendMessage; // Generating requestId for the message const requestId = requestIdOverriden || self.crypto.randomUUID(); ... // If response indicates there will be a chunk message sent, adding a listener to retrieve full response if (response && response[CHUNKED_MESSAGE_FLAG]) { let listener; try { const fullResponse = await new Promise(resolve => { listener = addOnChunkedMessageListener( (fullResponseFromListener, _, sendResponse) => { sendResponse(); resolve(fullResponseFromListener); }, { requestIdToMonitor: response.requestId } ); }); return fullResponse; } finally { if (listener) { removeOnChunkedMessageListener(listener); } } } else { return response; } } const sendChunkedMessage = async ( message, { sendMessageFn, requestId: requestIdOverriden } = {} ) => { const sendMessage = sendMessageFn || sendMessage; // Generating requestId for the message const requestId = requestIdOverriden || self.crypto.randomUUID(); ... // If response indicates there will be a chunk message sent, adding a listener to retrieve full response if (response && response[CHUNKED_MESSAGE_FLAG]) { let listener; try { const fullResponse = await new Promise(resolve => { listener = addOnChunkedMessageListener( (fullResponseFromListener, _, sendResponse) => { sendResponse(); resolve(fullResponseFromListener); }, { requestIdToMonitor: response.requestId } ); }); return fullResponse; } finally { if (listener) { removeOnChunkedMessageListener(listener); } } } else { return response; } } You may notice addOnChunkedMessageListener is also slightly modified to filter out incoming requestId with requestIdToMonitor option. addOnChunkedMessageListener requestId requestIdToMonitor Example Usage With Large Response content script - same: sendChunkedMessage(largeMessage) .then(response => { ... }) sendChunkedMessage(largeMessage) .then(response => { ... }) background service worker: addOnChunkedMessageListener((message, sender, sendResponse) => { // message === largeMessage ... sendChunkedResponse({ sendMessageFn: message => chrome.tabs.sendMessage(sender.tab.id, message) })(largeResponse, sendResponse); return true; // async listener }) addOnChunkedMessageListener((message, sender, sendResponse) => { // message === largeMessage ... sendChunkedResponse({ sendMessageFn: message => chrome.tabs.sendMessage(sender.tab.id, message) })(largeResponse, sendResponse); return true; // async listener }) Conclusion The solution was created to overcome message length limitation for Chrome Extensions messaging in Manifest V3; the solution works and I am using it in a real project. Happy to hear advice on how to make it better, and feel free to contribute! Code with an example extension can be found on GitHub: https://github.com/abelozerov/ext-send-chunked-message https://github.com/abelozerov/ext-send-chunked-message Published to NPM. It can be installed with npm i ext-send-chunked-message npm i ext-send-chunked-message Thank you for reading!