Using AI in Transana

Introduction

To use AI well in Transana, researchers need to make several important decisions. This article highlights two major choices.

The choice between using external AI tools and embedded (internal) AI tools is fairly straight-forward, and this article lays out the advantages and disadvantages of each option.

The choice of which AI model to use when exploring research data is more challenging. While external AI tools generally offer limited choices and limited flexibility, embedded AI offers an overwhelming set of choices. The bulk of this article is dedicated to laying out these choices, describing the process I used narrow down the options, and sharing the results of my extensive testing of AI models..

External AI and Embedded AI

The first choice a researcher must make when using AI in Transana is between external AI and embedded (or internal) AI.

External AI uses an outside service such as ChatGPT, Gemini, Co-pilot, or Claude, for AI queries. Currently, Transana supports ChatGPT for external exploration of data. 

Embedded AI uses an AI service on a computer controlled by the researcher.  Currently, Transana supports the Ollama system, which offers hundreds of individual AI models to choose from.

Each of these systems has advantages and disadvantages:

Privacy and Confidentiality

Data privacy, security, and confidentiality are central issues for most research projects. When a researcher uses an external AI tool, they send their data to a server that is not under their control. There are a wide variety of company policies and legal issues that influence what happens to the data once is received by the external computer. It is imperative that researchers understand the privacy and confidentiality policies of the companies they work with for external AI exploration of their data. Researchers should never submit data for external AI exploration without the explicit approval of their Institutional Review Board or other ethics board overseeing their research.

With embedded AI processing in Transana, all AI processing is handled by the Ollama server select by the researcher. This Ollama Server may reside on their own computer, or they may configure Transana to use an Ollama server on a different computer under their control. When properly configured, the Ollama server does not share or retain any data. Researchers should only connect to Ollama servers on their own computer or that is controlled by someone they trust, such as their department, university, or organization IT department. Choosing a server they control is how they ensure the privacy and confidentiality of their data during the AI exploration phase of your analysis.

Selecting Models

For external AI, the main choice of model is made when selecting the tool; only ChatGPT is currently supported in Transana, and ChatGPT only offers a small number of model options. If other external options such as Claude, Co-Pilot, Gemini, and others, were available, the evaluation and selection of external model would take a bit more work. Thus, external model selection is falsely simplified when compared to embedded model selection.

For Embedded AI, there are many models so choose from, models that are generally not so well known, making this choice less obvious. The remainder of this article is devoted to the topic of selecting AI models for best results.

Embedded AI Models and Model Parameters

AI Models are algorithms that determine how an AI works with data. They are trained on (typically very large) data sets and are designed to handle certain types of tasks and achieve certain types of goals. Transana supports the external ChatGPT AI service from OpenAI. It also supports an embedded AI tool called Ollama to manage the download, selection, and use of pre-defined AI models to allow the embedded exploration of qualitative data.

ChatGPT offers limited options for users, effectively hiding a lot of complexity from end users. Ollama presents a more complex AI landscape, requiring more background knowledge of the researcher.

Models

ChatGPT offers a small handful of models, mostly different versions of the same set of algorithms. Recent options (as of this writing) include ChatGPT 5, ChatGPT 5.1, and ChatGPT 5.2.

Ollama offers a large catalog of models. See the Models page on Ollama’s web site for more inforation.

Parameters

Ollama AI models are built with a characteristic called parameters, which represents internal variables used by the AI model to map input data to outputs, influencing the model’s ability to see patterns in data. To over-simplify, the higher the value of “parameters” a model is built with, the more sophisticated a response the model should be able to generate. However, the number of parameters also affects AI processing factors such as memory requirements and processing speed.

Some Ollama models support several different parameter options, and we have determined that the parameters value is important in how models work within Transana. Models are always presented as “(model name):(parameters)” pairs, for example, “gemma3:12b” for the gemma3 model with the 12 billion parameter setting. We strongly recommend that Transana users avoid the use of “cloud” and “turbo” parameters, as these require external processing within Ollama and may compromise data confidentiality.

You can explore the full list of models available with Ollama with their parameter options through the “Models” section of the Ollama website.

Testing Ollama Models

Ollama offers a huge number of models. I wanted to test as many of these models as I could in an effort to determine which models did a good job at exploring qualitative data within Transana. The next several sections of this article provides a narrative of how I approached this task and what I found.

For testing the analysis of text (including transcripts), I used a transcript of the movie “12 Angry Men” as my initial testing data because it was long enough (90 minutes) and complex enough (12 major speakers) to represent a potential challenge to AI, and because it is non-confidential data that others can obtain if they want to explore, replicate, or extend my test results.

For testing still image analysis by AI, I used a photograph I took a few years ago while traveling. The image included several distinct elements that could be included in the analysis.

While this does not represent typical qualitative research data, the narrative of the movie provides information that can be analyzed qualitatively that leads to clear conclusions, making between-model comparisons of AI results easier than if actual qualitative data had been used.

I performed three rounds of tests.

First Tests

For my first attempt at testing, I selected a very simple prompt on my main development computer, paired with the 12 Angry Men transcript.

Summarize the following transcript

For these tests, I used the default settings for Context Size (Ollama defaults to 4k, but individual models can override this) and Temperature (often 0.8, but can vary by model). Context Size is a term that describes how much memory is allocated for the AI to store and process information. Temperature is a setting that determines the amount of “randomness” the AI engine introduces in forming a response to the prompt and data submitted.

There were three mistakes in this setup that doomed these test attempts to failure.

• First, it appears that summary of data is a default prompt for many AI models. When I tried more complex prompts with the default settings used here, a surprising number of models returned a summary rather than responding to the specific prompt they were given. So a summary request is not a good test of AI models when working with text-based data.
• Second, the transcript I submitted contained more than 20,000 tokens, and this was far too much data for Ollama’s default context size for many models. When using non-summary prompts, the ability of some models to respond to the submitted prompt rather than producing a summary required a large-enough context size setting. A large-enough context size allowed some models to “see” the submitted prompt. However, this was only true for some models.
• Third, since the computer I used for these first tests has 64 GB of RAM, it did not represent the computers that many Transana users have. When I tried this prompt and data on different computers with less RAM, some models that worked on the first computer could not run the same prompts with the same settings due, presumably, to RAM limitations.

Second Tests

Having learned these three points, I abandoned this data and started over.

For my second attempt at testing, I changed my prompt to explicitly indicate that the data I submitted was a transcript of a jury deliberation and to request a description of each of the jurors.

This is a transcript of a jury deliberation. Describe each juror in a separate paragraph, including juror number, name if known, occupation, and personality.

After a little experimentation, I settled on a context size of 32K, large enough to hold the 20K+ tokens of data I was exploring and a reasonable-sized response. For this test, I used a Temperature of 0.8. I ran each test on multiple computers. I was not able to test all models on all computers because these tests could take as long as 24 hours in some circumstances. I made the decision to skip some tests that were unlikely to succeed for the sake of efficiency and sanity.

A condensed summary of the result of this testing are presented in expandable tables below. The first table presents the the model and parameter combinations that produced “good” results on at least one computer. This table does not include the large number of models for which all tests failed. The second table is a more detailed description of the five computers used for testing.

There are several important points to keep in mind when reviewing this.

• First, new models are coming out all the time. This is a snapshot of a moving target.
• Second, I used a very broad prompt. In actual analysis, revising the prompt is an important step in AI exploration, and it is likely that changing the prompt will affect the AI output in both expected and unexpected ways.
• Third, this is one example of a prompt and data. The data is from a movie, so may not represent real-world research data in important ways.

Third Test

For the third round of testing, I submitted a still image to the full suite of Ollama models with the following query:

Describe the following image:

While asking for a description was not a good test for text data, it revealed something very interesting for these image tests. Some models described an image in response to this prompt that was clearly not the image I submitted. This description of a different image could sometimes be quite detailed. Thus, for images, this description prompt ends up revealing models that are not able to process images the way Transana submits them but that do not inform the researcher of this failure.

I settled on a context size of 48K for the image. For this test, I set the Temperature to 0.3, as I wanted to get more consistent results. I ran each test on multiple computers. I was not able to test all models on all computers because these tests could take as long as 24 hours in some circumstances and I made the decision to skip some tests that were unlikely to succeed for the sake of efficiency and sanity.

The results of this testing are presented in an expandable table below. This table presents the the 21 model and parameter combinations that produced “good” results on at least one computer. Note that there is limited overlap between these models and models that did well with text. (See both above and below.)

Models that failed

The models listed below failed to produce adequate results for either text and image exploration.

Lessons Learned

• I tested 128 combinations of models and model parameters for Ollama LLMs and 3 models for ChatGPT.
• Using 2 Windows computers, I tested all text analysis and image analysis for all 128 Ollama models for a total of 512 tests. (2 computers x 2 data types x 128 models = 512 tests.)
• I found a total of 38 “Good” results for text and 35 “Good” results for still images. That’s a 14.3% overall success rate. Over 85% of all of these tests failed when held to a quality standard of “good.” About 1 test in every 7 produced “good” results.
• Still focusing on these Windows tests, 40 models produced at least one “Good” response across text and image analyses.
• Over 2 out of every 3 models failed at both text and image analysis tasks.
• But that over-states the results. Over half (54.4%) of the 160 tests performed on just these 40 models on Windows failed.
• I could not test all 128 models on both of my macOS computers. Processing times were too long for this to be practical.
• I did, however, run all 40 models that produced at least 1 “Good” results in the Windows tests on macOS for both text analysis and image analysis.
• 60.3% of the 320 tests performed on these 40 models failed.
• Results on macOS were worse than results on WIndows, especially for text analysis. (This could be due to memory limitations on my macOS computers. It could also be that some models don’t work on Apple’s hardware and OS.)
• This suggests a challenging environment for qualitative researchers. The task of picking a model or set of models is complicated. (It is why I’ve included so much detail in this article.)
• With that caveat, I can also say that some Ollama models produced high quality results, rivaling, and in the case of image analysis, surpassing the results from the current ChatGPT model tests with the same prompts and data.

• Results are sometimes different with the same prompt and the same data and the same settings on different computers. Reducing the Temperature setting will likely address some of this. (In retrospect, I wish I’d chosen 0.3 for the Temperature setting for the Text tests, as I did for the Image tests, to increase consistency across different variations of the same model.)

• Hardware matters. Because I had only a few computers to test with, I can’t sort out all the factors. Your computer will likely differ from mine, so your results will differ from mine. Here are my speculations:

  • As a generalization, the more memory (RAM) a computer had, the more models ran successfully, and the better the quality was of those responses.
  • I still haven’t figured out why some tests failed sporadically, especially on the 32 GB Windows computer.
  • My Windows computers ran tests a lot faster than my macOS computers. There is a 16x difference in average processing time between the slower Mac and the faster Windows computer across all of the models tested that produced at least one “good” result. The fastest computer ran 16 times faster than the slowest.
  • My Windows computers both have Nvidia GPUs, which probably helped a lot.
  • It appears that, despite their relatively poor performance, the GPU features of the Apple M1 and M2 processors in my Macs were engaged in some processing. Newer Apple processors (M3, M4, and M5) might perform better. I don’t have a way to test this at this time.
  • The Windows computers also have more RAM than the Macs, which is likely a confounding variable here. Both Macs had only 8 GB of RAM, and, due to the infinite wisdom of Apple, neither is upgradable.

The following table summarizes which models produced “good” results for which kinds of data on which operating systems.

This is, of course, all part of a rapidly changing landscape. Different models have different designs and capabilities. New Ollama models come out frequently. New chips are announced regularly. I can speed up AI processing on my slowest Mac by linking it to the Ollama server on my fastest Windows computer. This page only scratches the surface.

Evaluating AI Results – Impressions

I am currently in the process of analyzing the “Good” results from the models presented in the data tables above.

So far, I have coded the responses from ChatGPT 5 and 10 of the “good” Ollama results. The majority of the responses are pretty good, and several findings seem to be emerging so far:

• I had expected the ChatGPT response to be clearly superior to the Ollama model responses. This has not been the case. Many of the responses I’ve evaluated so far seem roughly equivalent. In particular, some Ollama models identified features of the test image that ChatGPT did not, such as the location where the photo was taken.
• The text responses are surprisingly different from each other. As I code these responses, I have found less code re-use across models than I expected. Different model seems to find different salient features to highlight.
• One way that a few models differ in text responses is that they are more likely to make mistakes. Some conflate different people. Some make statements that I don’t feel the data supports. I’ve seen several instances of attributing quotes to the wrong person, and even an instance or two of making up quotes out of thin air.
• With image summaries, some models identified the location of the image. Some were more specific, naming a city. A few that mentioned locations that were incorrect or called the building a temple rather than an amphitheater.

These impressions are very preliminary. I am in no way approaching theoretical saturation at this time.  And to be fair, I’ve cherry-picked models that are most likely to have the best results so far, so these findings may not hold up. At this time, I must focus on getting the release containing embedded AI published, so am putting this analysis indefinitely on hold.

Asking AI to Evaluate AI

The task of evaluating and ranking the AI results produced by all of these tests, as described above, proved quite difficult and time-consuming. It is a task that I have not had adequate time to complete as of this time of this writing. Then it occurred to me that I could ask AI to handle this task.

Text

I created Quotes of the juror descriptions from the “Who Are the Jurors?” AI Summaries of transcripts described above. I then explored the resulting Collection using the following query:

The following are descriptions of the jurors in the movie “12 Angry Men” created by different AI models. Which 5 models do the best job? Please justify your response with quotes from the different descriptions.

I started with model qwen3-vl:8b and recorded the results from that model. I repeated the query with with the most highly rated of models in each of the summaries produced. I continued with this process until I found a consensus of the10 ranked “top” models according to the 10 top-ranked models. This produced the following rankings:

For our text prompt, ChatGPT produced the best results, according to our AI models. Interestingly, the models preferred ChatGPT 5 to the newer ChatGPT 5.1 and ChatGPT 5.2 models. Unfortunately, these ChatGPT models do not offer data security and privacy the way the remaining models do. qwen3:14b might be a good choice for Windows users, and qwen3-vl:8b, ministral-3:8b, and devstral:24b do reasonably well on both Windows and macOS.

Images

I then explored the still image descriptions, using the top 10 text models listed above. (That is, I used the best image models to generate text descriptions of the images, which were then evaluated using the best text models.) I used the following prompt:

The following are descriptions of a photo of Taormina, Italy created by different AI models. Which 5 models do the best job? Please justify your response with quotes from the different descriptions.

For our image prompt, gemma3:12b is rated highest by our AI models, followed by ChatGPT 5, qwen3-vl:30b, and devstral-small-2:24b. All of these models work well on both Windows and macOS when exploring images. As with text exploration, ChatGPT 5 does not offer the same confidentiality protections the other models here do. I also find it interesting that the newer ChatGPT 5.1 and ChatGPT-5.2 models did not make the cut while the older ChatGPT 5 model did.

The Last Word

I want to emphasize one last point as part of this discussion. For both text and for images, I asked 10 models to rank the AI results I had generated. Each model did so, assertively and confidently. And while some consensus emerged across the 10 models, these models disagreed far more than they agreed. No single model agreed with this consensus, and no two models agreed with each other.

At one point, I accidentally ran one of the image tests twice. The selections and rankings of the two identical test runs were quite different, even though the Temperature setting used (0.3) should have limited the amount of randomness coming out of the AI model. So the AI models don’t even agree with themselves!

This reinforces in my mind that all AI results must be reviewed and checked against the data by a researcher before being reported as a research finding, and that all use of AI in qualitative analysis must be carefully described in qualitative write-ups and presentations. AI comments on qualitative data may be interesting, and they may sometimes suggest useful ideas. It’s vital to recognize that there is no actual, real “intelligence” behind AI, even when the clever application of high-level mathematics makes it appear so.