AI General Thread

[Legend]

For some reason, running local models feels goofier than it should be. And I'm not sure why. Like for some reason, prompt adherence is worse when I'm using different Android app.

Not sure if they're formatting the requests differently, or passing different default values for the model.

Different sampling method/temperature?

[the-pi-guy]

Different sampling method/temperature?

Most of the apps let you change the temperature, top K, top P. 

I'm leaning towards formatting being different, which would be harder to fix as an end user.

[the-pi-guy]

LM Studio has so many options you can mess with. 

You can even set how the context window is managed (truncate middle, rolling).  

[Legend]

LM Studio has so many options you can mess with.

You can even set how the context window is managed (truncate middle, rolling). 

Is there a way to run two models in parallel so that the next token can be sampled using both their outputs? Like averaged or multiplied or whatever?