On this page

1. Start with the answer you actually need
2. Choose a model that fits comfortably
3. Reduce work the model does not need
4. Let OGAD choose the starting speed settings
5. Change advanced settings carefully
6. Use a comparison that tells you where the wait occurs
7. Try changes in a useful order
8. Find your practical local model

A local model is useful when you can finish the work while the answer is still relevant. Waiting for an oversized model to write a long response is not always the best use of your computer.

OGAD (Off Grid AI Desktop) lets you run and compare local models in one app. The most useful first change is often a smaller model that can still do the job, followed by a shorter context and a more focused request. You keep local processing without assuming every model will be fast on every machine.

Download OGAD for Mac or Windows

---

What would you like to do with Off Grid AI?

Have a feature or use case you would like us to support? Tell us what you want to do and which device you use.

Write to [email protected], join our Slack community, or talk to us on Reddit.

Start with the answer you actually need

Compare models on a task from your normal work: rewrite one email, explain one function or extract three decisions from a short note. Keep the input and requested output the same for each trial.

Ask for a bounded result:

Rewrite this update in two short paragraphs. Keep the dates and unresolved questions. Return only the revised text.

That is easier to compare than an open-ended request for an exhaustive analysis. A shorter answer also requires fewer generated tokens, although the initial processing time can still depend on the model and input.

Choose a model that fits comfortably

1. Open Models and inspect the memory guidance for your current choice.
2. If it is tight on RAM, download a smaller compatible text model.
3. Run your sample task in a fresh chat.
4. Compare the time to a useful result and the quality of the answer.
5. Keep the smallest option that meets the task’s needs.

Do not choose on speed alone. A fast answer that needs extensive repair can cost more of your time than a slower accurate draft.

Reduce work the model does not need

Use a practical Context window in the model settings. More context is useful when the task needs long material; it can cost memory and prompt-processing work when it does not.

Start a new conversation with a short checked handoff when the old discussion contains many abandoned versions. Include the current facts, the latest draft and the next task. Do not depend on a summary to preserve every earlier detail.

Unload a model you no longer need from the model picker. Close other heavy work while making your comparison. The first request after loading may include setup time, so compare that separately from repeated requests with the model already available.

Let OGAD choose the starting speed settings

With a local text model selected, open chat settings and look for Use fastest settings. In desktop beta 122, this applies the app’s settings plan for the detected hardware and model. The app can keep that plan current as the hardware or model changes. If the settings are already applied, the button may not be offered.

1. Record the current model and use a short request that you can repeat.
2. Choose Use fastest settings when it is available.
3. Allow a model reload to finish if the chosen backend or load settings change.
4. Repeat the same request in a fresh chat and compare the answer, first-token wait, and total time.

A manual speed-setting change ends automatic management of those settings. Use the automatic control again when you want the app to choose them. The button name describes the app’s configuration choice; it is not a measured guarantee that every request will finish faster.

If the model still cannot fit in memory, choose a smaller model or a shorter context. Settings cannot add physical RAM. Keep the sample prompt and required answer length the same while comparing results.

Change advanced settings carefully

OGAD exposes CPU threads and Batch size controls. More threads or a larger batch is not a universal speed setting. The result depends on the processor, memory and workload.

Keep a note of the starting values. Change one setting, repeat the same task, and return to the earlier value if it does not help. Use the app’s normal settings as a starting point before attempting machine-specific tuning.

This guide does not promise a particular tokens-per-second figure. Your hardware, model, context and other running work all affect the result.

The controls described here are available in OGAD 0.0.51.

Use a comparison that tells you where the wait occurs

Choose one short task and run it several times with the same settings. Note the initial delay before the answer starts and how long the complete usable answer takes. These are different costs: a model can start slowly but then produce text quickly.

For example, use a 150-word meeting note and ask for three decisions plus any missing owner. Keep that same note and output request when trying a smaller model. Do not compare a three-bullet answer from one model with a long essay from another.

In the Text settings, Generation details can show context use, speed, token count and time beneath each answer. Read the result alongside the answer quality. A higher token rate is not a useful improvement if the output omits the decision you needed.

Try changes in a useful order

Start with the request: remove irrelevant history and ask for the output you will actually use. Next compare a smaller model that fits comfortably. Then adjust context if it is larger than the task needs. Advanced tuning comes after those checks.

Keep a short record of the model, context and setting changed. Restore the old value when a change does not help. This prevents several simultaneous changes from leaving you with no idea what caused the difference.

For a second test, use your normal work environment with the apps you usually keep open. A configuration that feels quick only after closing everything may not suit your day. Choose the setup that helps you finish the task with acceptable accuracy and enough room for the rest of your work.

Find your practical local model

Download OGAD and compare two sensible model choices on one real task. Keep the choice that helps you finish the work, with your data processed locally after setup.