From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide gap between the performance of recent MLLM-based applications and the expectation of the broad public, even though the most powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper strives to enhance understanding of the gap through the lens of a qualitative study on the generalizability, trustworthiness, and causal reasoning capabilities of recent proprietary and open-source MLLMs across four modalities: ie, text, code, image, and video, ultimately aiming to improve the transparency of MLLMs. We believe these properties are several representative factors that define the reliability of MLLMs, in supporting various downstream applications. To be specific, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed cases, where the qualitative results are then summarized into 12 scores (ie, 4 modalities times 3 properties). In total, we uncover 14 empirical findings that are useful to understand the capabilities and limitations of both proprietary and open-source MLLMs, towards more reliable downstream multi-modal applications

Western blot analysis was used to compare and evaluate the expressing quantity of neuroligins in ICC-MY of different segments in HSCR.

<p>Neuroligins were expressed significantly in ICC-MY of ganglionic colonic segment, moderately in transitional segment, and obviously downed-regulated in aganglionic colonic segment.</p

Primary antibody.

<p>Primary antibody.</p

ICC-MY percentages of ganglionic, transitional and aganglionic segments from one case of HSCR were shown by Figure 3A, 3B and 3C, respectively.

<p>ICC-MY percentages of ganglionic, transitional and aganglionic segments from one case of HSCR were shown by <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0067205#pone-0067205-g003" target="_blank">Figure 3A, 3B and 3C</a>, respectively.</p

Immunohistochemical staining was performed on LMMP, light microscopy was used for observation.

<p>ICC-MY were identified by immunoreactivity for the tyrosine kinase receptor c-Kit(CD117) (brownish yellow, network structure) on LMMP. <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0067205#pone-0067205-g001" target="_blank">Figure 1A</a> represented ganglionic colonic segment and <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0067205#pone-0067205-g001" target="_blank">Figure 1B</a> represented aganglionic colonic segment. No obvious disruptions of ICC-MY network were showed in aganglionic segments. Magnification 40×.</p

ICC-MY percentages of ganglionic, transitional and aganglionic segments from all 50 cases HSCR and the numerical data are presented as the mean±standard deviation (90.98±3.24, 90.30±3.09, 90.01±3.11, n = 50).

<p>Statistical analysis was performed using T test, and P<0.05 was considered statistically significant.</p

Immunofluorescence labeling was used to identify ICC-MY.

<p>c-Kit was expressed in ICC-MY, whose cell bodies and processes were obviously expressed. These cells interact and interwove with each other, whose shape was fusiform or triangular. Scale bars:25 µm.</p

Secondary antibody.

<p>Secondary antibody.</p

The gray level of neuroligins expressed in Western blot analysis was 204.07±8.81 in ICC-MY of ganglionic colonic segments, 136.13±13.2 in transitional segments and 86.65±4.54 in aganglionic colonic segments, and the difference of gray level had statistical significance (204.07±8.81 vs 136.13±13.2, P<0.05; 136.13±13.2 vs 86.65, P<0.05; 204.07±8.81 vs 86.65±4.54, P<0.05).

<p><b>(<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0067205#pone-0067205-g007" target="_blank"><b>Figure 7A</b></a><b>).</b></b> The gray level of c-Kit expressed in Western blot analysis was 207.66±13.5 in ICC-MY of ganglionic colonic segments, 205.32±10.75 in transitional segments and 203.52±11.99 in aganglionic colonic segments and there was no statistical difference among aganglionic, transitional and ganglionic colonic segments (207.66±13.5 vs 205.32±10.75, P>0.05; 205.2±10.75 vs 203.52±11.99, P>0.05; 207.66±13.5 vs 203.52±11.99, P>0.05 ). (<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0067205#pone-0067205-g007" target="_blank">Figure 7B</a>).</p